On 2022-02-09, Ben Bacarisse <ben.usenet@bsb.me.uk> wrote:
> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>
>> On 09.02.2022 08:49, Axel Reichert wrote:
>> [ about an ASCII data mangling Python script ]
>>> [....] I started immediately, cobbled something together (awk featured
>>> prominently among other usual suspects, such as tr, sed, cut, grep).
>>
>> Hmm.. - these four tools are amongst those where I usually say; instead
>> of connecting and running a lot of such processes use just one instance
>> of awk. The functions expressed in those tools are - modulo a few edge
>> cases - basics in Awk and part of its core.
>
> That sometimes works, but the trouble is that once you've used AWK's
> pattern/action once feature, you can't do so again -- you are stuck
> inside the action part. Just the other day I needed to split fields
> within a filed after finding the lines I wanted. This was, for me, an
> obvious case for two processes:
>
> awk -F: '/wanted/ { print $3 }' | awk -F, '...'

You can split $3 into fields by assigning its value to $0, after
tweaking FS for the inner field separator:

$ awk '/wanted/ { FS=","; $0=$3; OFS=":"; $1=$1; print }'
wanted two three,a,b,c <- input
three:a:b:c <- output

You have to save and restore FS to do this repeatedly for
different records of the outer file. Another approach is to
use the split function to populate an array, where the pattern
is an argument (only defaulting to FS if omitted).

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal

On 09.02.2022 22:05, Ben Bacarisse wrote:
> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>
>> On 09.02.2022 08:49, Axel Reichert wrote:
>> [ about an ASCII data mangling Python script ]
>>> [....] I started immediately, cobbled something together (awk featured
>>> prominently among other usual suspects, such as tr, sed, cut, grep).
>>
>> Hmm.. - these four tools are amongst those where I usually say; instead
>> of connecting and running a lot of such processes use just one instance
>> of awk. The functions expressed in those tools are - modulo a few edge
>> cases - basics in Awk and part of its core.
>
> That sometimes works,

My observation is that it usually works smoothly, and only sometimes
(the edge cases, I called them above) not obviously straightforward,
but usually just in a slightly different way. But it works generally.

> but the trouble is that once you've used AWK's
> pattern/action once feature, you can't do so again -- you are stuck
> inside the action part. Just the other day I needed to split fields
> within a filed after finding the lines I wanted.

You can always simply split() the fields, no need to invoke another
process just for another implicit loop that awk supports.

> This was, for me, an obvious case for two processes:
>
> awk -F: '/wanted/ { print $3 }' | awk -F, '...'

I understand the impulse to develop commands that way; that usually
leads to such horrible and inflexible cascades of the tools mentioned
above (cat, sed, grep, cut, head, tail, tr, wc, pr, or yet more awks).

And as soon as you need yet more information from the first instance
this approach needs more workarounds, e.g. passing state information
through the OS level.

Of course there's many ways to skin a cat. I just advocate to think
about one-process solutions before following the reflex to construct
inflexible pipeline constructs.

>
> but I could have used grep and cut in place of the first AWK. Maybe I'm
> just not good at remembering the details of all the key functions,

The nice thing about awk - actually already mentioned in context of
the features/complexity vs. power comments - is that you don't need
to memorize a lot;[*] I think awk is terse and compact enough. YMMV.

> but I find I use AWK in pipelines quite a lot.

That's how we learned it; pipelining through simple dedicated tools.
I also still do that. My observation is that whenever a more powerful
tool like awk gets into use, the more primitive tools in the pipeline
can be eliminated, the whole pipeline gets then refactored, typically
for efficiency, flexibility, robustness, and clarity in design.

I want to close my comment with another aspect; the primitive helper
tools are often restricted and incoherent.[*] In GNU context you have
additional options that I'm glad to be able to use, but if you want to
stay standard conforming the tools might not "suffice" or usage gets
more bulky. With awk the standard version supports already the powerful
core.

Janis

[*] If I'd have a remembering issue then it would be how options, e.g.
the delimiters, are (differently) defined in the various tools, since
options are incoherent and inconsistently named across the tools, and
such options have also different semantics. That results in a lot man
page lookups and more software maintenance issues.

Kaz Kylheku <480-992-1380@kylheku.com> writes:

> On 2022-02-09, Ben Bacarisse <ben.usenet@bsb.me.uk> wrote:
>> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>>
>>> On 09.02.2022 08:49, Axel Reichert wrote:
>>> [ about an ASCII data mangling Python script ]
>>>> [....] I started immediately, cobbled something together (awk featured
>>>> prominently among other usual suspects, such as tr, sed, cut, grep).
>>>
>>> Hmm.. - these four tools are amongst those where I usually say; instead
>>> of connecting and running a lot of such processes use just one instance
>>> of awk. The functions expressed in those tools are - modulo a few edge
>>> cases - basics in Awk and part of its core.
>>
>> That sometimes works, but the trouble is that once you've used AWK's
>> pattern/action once feature, you can't do so again -- you are stuck
>> inside the action part. Just the other day I needed to split fields
>> within a filed after finding the lines I wanted. This was, for me, an
>> obvious case for two processes:
>>
>> awk -F: '/wanted/ { print $3 }' | awk -F, '...'
>
> You can split $3 into fields by assigning its value to $0, after
> tweaking FS for the inner field separator:
>
> $ awk '/wanted/ { FS=","; $0=$3; OFS=":"; $1=$1; print }'
> wanted two three,a,b,c <- input
> three:a:b:c <- output

Sure, but you don't get to use pattern/action pairs on the result.

> You have to save and restore FS to do this repeatedly for
> different records of the outer file. Another approach is to
> use the split function to populate an array, where the pattern
> is an argument (only defaulting to FS if omitted).

I would much prefer to use split, but only if someone stopped me doing
it the natural way with a pipeline.

I suspected there would be a slew of replies about how to do it in one
command! However, I seriously doubt that there is any Unix programmer
or sysadmin who has not used AWK in a pipeline with comments that could,
relatively easily, be coded into the {} part of one or more actions. I
really don't think my point is very contentious.

--
Ben.

Janis Papanagnou <janis_papanagnou@hotmail.com> writes:

> On 09.02.2022 22:05, Ben Bacarisse wrote:
>> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>>
>>> On 09.02.2022 08:49, Axel Reichert wrote:
>>> [ about an ASCII data mangling Python script ]
>>>> [....] I started immediately, cobbled something together (awk featured
>>>> prominently among other usual suspects, such as tr, sed, cut, grep).
>>>
>>> Hmm.. - these four tools are amongst those where I usually say; instead
>>> of connecting and running a lot of such processes use just one instance
>>> of awk. The functions expressed in those tools are - modulo a few edge
>>> cases - basics in Awk and part of its core.
>>
>> That sometimes works,
>
> My observation is that it usually works smoothly, and only sometimes
> (the edge cases, I called them above) not obviously straightforward,
> but usually just in a slightly different way. But it works generally.
>
>> but the trouble is that once you've used AWK's
>> pattern/action once feature, you can't do so again -- you are stuck
>> inside the action part. Just the other day I needed to split fields
>> within a filed after finding the lines I wanted.
>
> You can always simply split() the fields, no need to invoke another
> process just for another implicit loop that awk supports.

Yes, there's no need, but why worry about it? Maybe I am alone in
thinking processes are cheap.

But more to the point, a pipeline is an elegant, easily understood, and
often natural way to organise a task. I will keep using them, even if
there is no need.

>> This was, for me, an obvious case for two processes:
>>
>> awk -F: '/wanted/ { print $3 }' | awk -F, '...'
>
> I understand the impulse to develop commands that way; that usually
> leads to such horrible and inflexible cascades of the tools mentioned
> above (cat, sed, grep, cut, head, tail, tr, wc, pr, or yet more awks).
>
> And as soon as you need yet more information from the first instance
> this approach needs more workarounds, e.g. passing state information
> through the OS level.

A pipeline is not the right structure for such tasks, but there are a
huge number of tasks where combining Unix tools is the simplest
solution.

> Of course there's many ways to skin a cat. I just advocate to think
> about one-process solutions before following the reflex to construct
> inflexible pipeline constructs.
>
>>
>> but I could have used grep and cut in place of the first AWK. Maybe I'm
>> just not good at remembering the details of all the key functions,
>
> The nice thing about awk - actually already mentioned in context of
> the features/complexity vs. power comments - is that you don't need
> to memorize a lot;[*] I think awk is terse and compact enough. YMMV.

But since I use pipelines so much, I rarely use split, patsplit, gsub or
gensub. I find myself checking their arguments pretty much every time I
use them.

>> but I find I use AWK in pipelines quite a lot.
>
> That's how we learned it; pipelining through simple dedicated tools.
> I also still do that.

Why? Serious question. It sound like a dreadful risk based on your
comments above. Doing is "usually leads to such horrible and inflexible
cascades of the tools" when there is no need "to invoke another
process". What makes you sometimes take the risk of horrible cascades
and pay the price of another process?

I ask because it's possible we disagree only on how frequently it should
be done, and about exactly what circumstances warrant it.

> My observation is that whenever a more powerful
> tool like awk gets into use, the more primitive tools in the pipeline
> can be eliminated,

I think we all agree that it /can/ be done.

> the whole pipeline gets then refactored, typically for efficiency,
> flexibility, robustness, and clarity in design.

That's where I disagree. I often choose a pipeline because it is the
most robust, flexible and clear design. (I rarely care about efficiency
when doing this sort of thing.)

I do it in other contexts too. In Haskell, because of it's lazy
evaluation, you can chain function calls that filter and process lists,
even potentially infinite ones. It often results in clear, easy to
modify code.

> I want to close my comment with another aspect; the primitive helper
> tools are often restricted and incoherent.[*] In GNU context you have
> additional options that I'm glad to be able to use, but if you want to
> stay standard conforming the tools might not "suffice" or usage gets
> more bulky. With awk the standard version supports already the powerful
> core.

I agree. That's a shame, but an inevitable cost of piecemeal historical
development.

--
Ben.

On 2022-02-10, Ben Bacarisse <ben.usenet@bsb.me.uk> wrote:
> Kaz Kylheku <480-992-1380@kylheku.com> writes:
>
>> On 2022-02-09, Ben Bacarisse <ben.usenet@bsb.me.uk> wrote:
>>> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>>>
>>>> On 09.02.2022 08:49, Axel Reichert wrote:
>>>> [ about an ASCII data mangling Python script ]
>>>>> [....] I started immediately, cobbled something together (awk featured
>>>>> prominently among other usual suspects, such as tr, sed, cut, grep).
>>>>
>>>> Hmm.. - these four tools are amongst those where I usually say; instead
>>>> of connecting and running a lot of such processes use just one instance
>>>> of awk. The functions expressed in those tools are - modulo a few edge
>>>> cases - basics in Awk and part of its core.
>>>
>>> That sometimes works, but the trouble is that once you've used AWK's
>>> pattern/action once feature, you can't do so again -- you are stuck
>>> inside the action part. Just the other day I needed to split fields
>>> within a filed after finding the lines I wanted. This was, for me, an
>>> obvious case for two processes:
>>>
>>> awk -F: '/wanted/ { print $3 }' | awk -F, '...'
>>
>> You can split $3 into fields by assigning its value to $0, after
>> tweaking FS for the inner field separator:
>>
>> $ awk '/wanted/ { FS=","; $0=$3; OFS=":"; $1=$1; print }'
>> wanted two three,a,b,c <- input
>> three:a:b:c <- output
>
> Sure, but you don't get to use pattern/action pairs on the result.

But that's largely just syntactic sugar for a glorified case statement.

Instead of

/abc/ { ... }
$2 > $3 { ... }

you have to write

if (/abc/) { ... }
if ($2 > $3) { ... }

kind of thing.

Janis Papanagnou <janis_papanagnou@hotmail.com> writes:

> I understand the impulse to develop commands that way; that usually
> leads to such horrible and inflexible cascades of the tools mentioned
> above (cat, sed, grep, cut, head, tail, tr, wc, pr, or yet more awks).
>
> And as soon as you need yet more information from the first instance
> this approach needs more workarounds, e.g. passing state information
> through the OS level.
>
> Of course there's many ways to skin a cat. I just advocate to think
> about one-process solutions before following the reflex to construct
> inflexible pipeline constructs.

It seems that like Ben I am a pipeliner, the igniting spark probably
"Opening the software toolbox":

https://www.gnu.org/software/coreutils/manual/html_node/Opening-the-software-toolbox.html

I know that a lot can be done within awk, but it often does not seem to
meet my way of thinking. For example, I might start with a grep. To my
surprise it finds many matches, so further processing is called for, say
awk '{print $3}' or similar. At that point, I will NOT replace the grep
with awk '/.../', because it is easier to just add another pipeline
after fetching the command from history using the up arrow. And so on,
adding pipeline after pipeline (which I also can easily relate to
functional programming). Once the whole dataflow is ready, I will
usually not "refactor" the beast, only in glaringly obvious
cases/optimizations. I might even have started with a (in hindsight)
Useless Use Of Cat. On the more ambitious side, I well remember how
proud I was when plumbing several xargs into a pipeline:

foo | bar | xargs -i baz {} 333 | quux | xargs fubar

By now this is a common idiom for me on the command line.

But full ACK on passing information from the first instance downstream,
at which point I tend to start using Python. But up to then pipelining
"just flows". That's what they were designed for. (-:

Axel

P. S.: I will keep your advice in memory, though, to avoid my worst
excesses. Point taken.

one-liner solution to that wanted-three question :

echo 'wanted two three,a,b,c' \
\
| [mg]awk '/^wanted/ && gsub(",", substr(":", ($0=$3)~"", 1)) + 1'

three:a:b:c

On Thursday, February 10, 2022 at 2:59:45 AM UTC-5, Kaz Kylheku wrote:
> On 2022-02-10, Ben Bacarisse <ben.u...@bsb.me.uk> wrote:
> > Kaz Kylheku <480-99...@kylheku.com> writes:
> >
> >> On 2022-02-09, Ben Bacarisse <ben.u...@bsb.me.uk> wrote:
> >>> Janis Papanagnou <janis_pa...@hotmail.com> writes:
> >>>
> >>>> On 09.02.2022 08:49, Axel Reichert wrote:
> >>>> [ about an ASCII data mangling Python script ]
> >>>>> [....] I started immediately, cobbled something together (awk featured
> >>>>> prominently among other usual suspects, such as tr, sed, cut, grep).
> >>>>
> >>>> Hmm.. - these four tools are amongst those where I usually say; instead
> >>>> of connecting and running a lot of such processes use just one instance
> >>>> of awk. The functions expressed in those tools are - modulo a few edge
> >>>> cases - basics in Awk and part of its core.
> >>>
> >>> That sometimes works, but the trouble is that once you've used AWK's
> >>> pattern/action once feature, you can't do so again -- you are stuck
> >>> inside the action part. Just the other day I needed to split fields
> >>> within a filed after finding the lines I wanted. This was, for me, an
> >>> obvious case for two processes:
> >>>
> >>> awk -F: '/wanted/ { print $3 }' | awk -F, '...'
> >>
> >> You can split $3 into fields by assigning its value to $0, after
> >> tweaking FS for the inner field separator:
> >>
> >> $ awk '/wanted/ { FS=","; $0=$3; OFS=":"; $1=$1; print }'
> >> wanted two three,a,b,c <- input
> >> three:a:b:c <- output
> >
> > Sure, but you don't get to use pattern/action pairs on the result.
> But that's largely just syntactic sugar for a glorified case statement.
>
> Instead of
>
> /abc/ { ... }
> $2 > $3 { ... }
>
> you have to write
>
> if (/abc/) { ... }
> if ($2 > $3) { ... }
>
> kind of thing.

two different one-liner solutions i managed to conjure up, neither of which requires dealing with patterns, or arrays, or patsplit, but both involve assigning back to $0

command 1 is

[ echo "wanted two three,a,b,c" | mawk2 '/wanted/ * gsub(",", substr(":",$_!=($_=$NF),_~_))' ]

three:a:b:c

command 2 is

[ echo "wanted two three,a,b,c" | mawk2 -F, '/wanted/ && ($!_=substr($!_,match($!_,/[^ \t]+$/) ) )' OFS=":" ]

three:a:b:c

On 2022-02-11, Kpop 2GM <jason.cy.kwan@gmail.com> wrote:
> one-liner solution to that wanted-three question :
>
> echo 'wanted two three,a,b,c' \
> \
> | [mg]awk '/^wanted/ && gsub(",", substr(":", ($0=$3)~"", 1)) + 1'
>
> three:a:b:c

Are you positively sure that you're taking my example literally enough?

Try this:

sed -e 's/wanted two //' -e 's/,/:/g'

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal

Kpop 2GM <jason.cy.kwan@gmail.com> writes:

> command 1 is
>
> [ echo "wanted two three,a,b,c" | mawk2 '/wanted/ * gsub(",", substr(":",$_!=($_=$NF),_~_))' ]
>
> three:a:b:c
>
> command 2 is
>
> [ echo "wanted two three,a,b,c" | mawk2 -F, '/wanted/ && ($!_=substr($!_,match($!_,/[^ \t]+$/) ) )' OFS=":" ]
>
> three:a:b:c

And both seem to me horrendously unelegant compared to

echo "wanted two three,a,b,c" | awk '{print $3}' | tr ',' ':'

But maybe I missed some detail from the original task and '/wanted/' has
to be added as awk pattern.

Best regards

Axel

On Friday, February 11, 2022 at 2:43:21 AM UTC-5, Kaz Kylheku wrote:
> On 2022-02-11, Kpop 2GM <jason....@gmail.com> wrote:
> > one-liner solution to that wanted-three question :
> >
> > echo 'wanted two three,a,b,c' \
> > \
> > | [mg]awk '/^wanted/ && gsub(",", substr(":", ($0=$3)~"", 1)) + 1'
> >
> > three:a:b:c
>
> Are you positively sure that you're taking my example literally enough?
>
> Try this:
>
> sed -e 's/wanted two //' -e 's/,/:/g'

i think yours doesn't enforce the filter criteria but simply cleaned it up, which could be circumvented as such :

echo $'wanted two threeA,a,b,c\nhi wanted two threeB,a,b,c' \
\ [mg]awk 'sub("^wanted.+ ","")*gsub(",",":")'
threeA:a:b:c

echo $'wanted two threeA,a,b,c\nhi wanted two threeB,a,b,c' | sed -e 's/wanted two //' -e 's/,/:/g'
threeA:a:b:c
hi threeB:a:b:c

>> echo "wanted two three,a,b,c" | awk '{print $3}' | tr ',' ':'

if elegance is a concern then no pattern and no print statement is more elegant, at least to me

echo "wanted two three,a,b,c" | gawk '($_=$NF)~_' | tr ',' ':'
three:a:b:c

On Thursday, February 10, 2022 at 12:33:11 PM UTC-5, Axel Reichert wrote:
> Janis Papanagnou <janis_pa...@hotmail.com> writes:
>
> > I understand the impulse to develop commands that way; that usually
> > leads to such horrible and inflexible cascades of the tools mentioned
> > above (cat, sed, grep, cut, head, tail, tr, wc, pr, or yet more awks).
> >
> > And as soon as you need yet more information from the first instance
> > this approach needs more workarounds, e.g. passing state information
> > through the OS level.
> >
> > Of course there's many ways to skin a cat. I just advocate to think
> > about one-process solutions before following the reflex to construct
> > inflexible pipeline constructs.
> It seems that like Ben I am a pipeliner, the igniting spark probably
> "Opening the software toolbox":
>
> https://www.gnu.org/software/coreutils/manual/html_node/Opening-the-software-toolbox.html
>
> I know that a lot can be done within awk, but it often does not seem to
> meet my way of thinking. For example, I might start with a grep. To my
> surprise it finds many matches, so further processing is called for, say
> awk '{print $3}' or similar. At that point, I will NOT replace the grep
> with awk '/.../', because it is easier to just add another pipeline
> after fetching the command from history using the up arrow. And so on,
> adding pipeline after pipeline (which I also can easily relate to
> functional programming). Once the whole dataflow is ready, I will
> usually not "refactor" the beast, only in glaringly obvious
> cases/optimizations. I might even have started with a (in hindsight)
> Useless Use Of Cat. On the more ambitious side, I well remember how
> proud I was when plumbing several xargs into a pipeline:
>
> foo | bar | xargs -i baz {} 333 | quux | xargs fubar
>
> By now this is a common idiom for me on the command line.
>

speaking of specialized tools for piping at the command line, how many open-source utilities you're aware of that could decode unsigned hex to arbitrary precision piping in from /dev/stdin with nothing more than :

gawk -nM '$!_=+$_'

or this variant, pre-negate it for you before outputting : gawk -nM '$!_=-$_'
or this variant, pre-double it for you before outputting : gawk -nM '$!_-=-$_'

or this best one yet :

gawk -nM '$(($!_-=-$_)~_)++'

decoding unsigned hex to arbitrary precision, and returning 2 n + 1 of that input. maybe it's even cleaner in perl i dunno

In article <20220209235134.861@kylheku.com>,
Kaz Kylheku <480-992-1380@kylheku.com> wrote:
....
>> Sure, but you don't get to use pattern/action pairs on the result.
>
>But that's largely just syntactic sugar for a glorified case statement.
>
>Instead of
>
> /abc/ { ... }
> $2 > $3 { ... }
>
>you have to write
>
> if (/abc/) { ... }
> if ($2 > $3) { ... }
>
>kind of thing.

Of course, it can be (and often has been) argued that everything in any
programming language is just "syntactic sugar" for the underling machine
code. Everything is really just for programmer convenience.

Personally, I tend to agree with Ben here, that not being able to use the
"automatic input loop" (aka, the "pattern/action" facility for which AWK is
justly famous) on non-traditional input is a Bad Thing. It would be nice
if this were not the case.

In fact, since I like to write my AWK programs as "true AWK programs"
rather than as shell scripts - i.e., my scripts start with:

#!/path/to/gawk -f

rather than:

#!/bin/bash
...
something | gawk ...
...

And I want to be able to use the pattern/action facility on piped-in input
(and also output - see below), I have written an extension library for GAWK
that enables that. That is, I can do:

@load "pipeline"
BEGIN { pipeline("in","Some Shell Command Here") }
/foo/ { bar...}

and it reads from the pipeline as if it had been piped-in from the shell.

I also find it annoying to have to put "| cmd" at the end of every command
that generates output - when I want the output to go to a pipe (e.g.,
"less"). So, I can do:

@load "pipeline"
BEGIN { pipeline("out","less") }
{ print }

With the expected result.

--
"He is exactly as they taught in KGB school: an egoist, a liar, but talented - he
knows the mind of the wrestling-loving, under-educated, authoritarian-admiring
white male populous."
- Malcolm Nance, p59. -

On 2022-02-11, Kenny McCormack <gazelle@shell.xmission.com> wrote:
> In article <20220209235134.861@kylheku.com>,
> Kaz Kylheku <480-992-1380@kylheku.com> wrote:
> ...
>>> Sure, but you don't get to use pattern/action pairs on the result.
>>
>>But that's largely just syntactic sugar for a glorified case statement.
>>
>>Instead of
>>
>> /abc/ { ... }
>> $2 > $3 { ... }
>>
>>you have to write
>>
>> if (/abc/) { ... }
>> if ($2 > $3) { ... }
>>
>>kind of thing.
>
> Of course, it can be (and often has been) argued that everything in any
> programming language is just "syntactic sugar"

Not by me.

> for the underling machine
> code.

When I say "syntactic sugar", I'm referring to a light transformation to
improve the taste. This is justified for the above example because the
"machine code" differs from the "HLL" counterpart only in that
"if (...)" has been wrapped around the test expressions.

I have a lot of experience implementing complicated language features
as code transformations, which I wouldn't call syntactic sugar.

> Personally, I tend to agree with Ben here, that not being able to use the
> "automatic input loop" (aka, the "pattern/action" facility for which AWK is

Ben didn't need a loop in the specific situation because he just wanted
to take a field and further split it as if it were a record.

You'd benefit from a loop if you wanted to take a string and treat it
as an entire file, separated into records and then fields.

I agree that having to implement that loop around your conditional
statements would start to get more than a little inconvenient.

I made a Lisp version of Awk as a macro. You can use it anywhere you can
use an expression, and cleanly nest it with itself.

E.g. we can use it to make a function which returns a list of objects:

This is the TXR Lisp interactive listener of TXR 273.
Quit with :quit or Ctrl-D on an empty line. Ctrl-X ? for cheatsheet.
This area is under 24 hour TTY surveillance.
1> (defun wrapped-awk (strings)
(build ;; establish sccope for procedural list construction
;; built-up list is returned when scope terminates
(awk (:inputs strings)
(:set fs ":") ;; field separator is colon (:)
(t (fconv - i i)) ;; convert second and third fields to integer
(#/plus/ (add (+ [f 1] [f 2])))
(#/minus/ (add (- [f 1] [f 2]))))))
wrapped-awk

Now we just pass a list of strings into this function: they become
records:

2> (wrapped-awk '("plus:1:2" "plus:3:4" "minus:5:5"))
(3 7 0)

The list of sums and differences is returned.

Now, let's use the wrapped-awk function inside another awk loop;
this time we will feed it input from the TTY interactively:

3> (awk (t (prn (wrapped-awk f))))
plus:1:2 plus:3:4 minus:5:5
3 7 0
minus:15:20
-5

"If the 't' condition is true, which is always, then print the list
obtained by passing the delimited fields f into wrapped-awk."

> And I want to be able to use the pattern/action facility on piped-in input
> (and also output - see below), I have written an extension library for GAWK
> that enables that. That is, I can do:

> @load "pipeline"
> BEGIN { pipeline("in","Some Shell Command Here") }
> /foo/ { bar...}

Let's pipe the above into a character string.

4> (with-out-string-stream (*stdout*)
(awk (t (prn (wrapped-awk f)))))
plus:10:20 minus:13:7
[Ctrl-D][Enter]
"30 6\n"

with-out-string-stream creates a scope in which it binds a variable
to a string stream. Everything sent to that stream is appended to
a string, and that string is returned when with-out-string-stream
terminates.

If we choose *stdout* for the variable name, then standard output
is temporary bound to this string stream. Everything that would go
to standard output goes into the string stream.

Already in the 1980s, people were able to use FFI to define bindings
to foreign libraries, and not have to write any glue code that had to be
compiled.

Here is my "extension lib" for calling size_t wcslen(const wchar_t *str):

5> (with-dyn-lib nil
(deffi wcslen "wcslen" size-t (wstr)))
wcslen
6> (wcslen "abcd")
4

How about the structure-returning function

ldiv_t lldiv(long numerator, long long denominator);

Define the foreign structure:

6> (deffi-struct lldiv
(quot longlong)
(rem longlong))
#<ffi-type (struct lldiv (quot longlong) (rem longlong))>

Define the foreign function binding:

7> (with-dyn-lib nil
(deffi lldiv "lldiv" lldiv (longlong longlong)))
lldiv

Test:

8> (lldiv 127 15)
#S(lldiv quot 8 rem 7)

This shit is unsafe, since we are calling C:

9> (lldiv 127 0)
Floating point exception (core dumped)

We could catch that signal. Let's try it again:

1> (deffi-struct lldiv (quot longlong) (rem longlong))
#<ffi-type (struct lldiv (quot longlong) (rem longlong))>
2> (with-dyn-lib nil
(deffi lldiv "lldiv" lldiv (longlong longlong)))
lldiv

Now install signal handler for SIGFPE, binding it to a
lambda function which throws the yikes symbol as an exception:

3> (set-sig-handler sig-fpe (lambda (signal async-p) (throw 'yikes)))
t

Now:

4> (lldiv 127 15)
#S(lldiv quot 8 rem 7)
5> (lldiv 127 0)
** yikes exception, args: nil
6>

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal

On 2022-02-11, Axel Reichert <mail@axel-reichert.de> wrote:
> Kpop 2GM <jason.cy.kwan@gmail.com> writes:
>
>> command 1 is
>>
>> [ echo "wanted two three,a,b,c" | mawk2 '/wanted/ * gsub(",", substr(":",$_!=($_=$NF),_~_))' ]
>>
>> three:a:b:c
>>
>> command 2 is
>>
>> [ echo "wanted two three,a,b,c" | mawk2 -F, '/wanted/ && ($!_=substr($!_,match($!_,/[^ \t]+$/) ) )' OFS=":" ]
>>
>> three:a:b:c
>
> And both seem to me horrendously unelegant compared to

How nice of you to provide company; now Kpop 2GM doesn't have to feel
he's the only one missing the point.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal

Kaz Kylheku <480-992-1380@kylheku.com> writes:

> Ben didn't need a loop in the specific situation because he just wanted
> to take a field and further split it as if it were a record.

I see this sketch of mine now has a life of it's own. I wrote '...' as
the second AWK source, so how anyone can know that I did not need a loop
I can't imagine.

The first AWK picked out lines of interest (they matched dates and AWK
would allow me to use date functions in the picking out though I had, on
this occasion, no need for that). The second AWK matched logged 'add'
and 'remove' actions that the second command counted -- anything added
more than it was removed remained for the END action to print.

Of course this could be one AWK command. What I don't get is why such a
simple use should raise so many people's hackles. It took about 30
seconds to write, and it did the job.

--
Ben.

It just occurred to me that we are discussing basically Unix/shell
issues at the moment, but since there's a strong relation to awk I
abstain from marking it [OT].

On 10.02.2022 18:33, Axel Reichert wrote:
> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>
>> I understand the impulse to develop commands that way; that usually
>> leads to such horrible and inflexible cascades of the tools mentioned
>> above (cat, sed, grep, cut, head, tail, tr, wc, pr, or yet more awks).
>>
>> [...]
>
> It seems that like Ben I am a pipeliner, the igniting spark probably
> "Opening the software toolbox":
>
> https://www.gnu.org/software/coreutils/manual/html_node/Opening-the-software-toolbox.html
>
> I know that a lot can be done within awk, but it often does not seem to
> meet my way of thinking. For example, I might start with a grep. To my
> surprise it finds many matches, so further processing is called for, say
> awk '{print $3}' or similar. At that point, I will NOT replace the grep
> with awk '/.../', because it is easier to just add another pipeline
> after fetching the command from history using the up arrow. And so on,
> adding pipeline after pipeline (which I also can easily relate to
> functional programming). Once the whole dataflow is ready, I will
> usually not "refactor" the beast, only in glaringly obvious
> cases/optimizations. I might even have started with a (in hindsight)
> Useless Use Of Cat. On the more ambitious side, I well remember how
> proud I was when plumbing several xargs into a pipeline:
>
> foo | bar | xargs -i baz {} 333 | quux | xargs fubar
>
> By now this is a common idiom for me on the command line.
>
> But full ACK on passing information from the first instance downstream,
> at which point I tend to start using Python. But up to then pipelining
> "just flows". That's what they were designed for. (-:
>
> Axel
>
> P. S.: I will keep your advice in memory, though, to avoid my worst
> excesses. Point taken.
>

Don't get me wrong. Pipelines of tools are not "bad" and I also wrote:

"That's how we learned it; pipelining through simple dedicated tools.
I also still do that. [...]"

Some application cases or programming patterns are also often not that
easily implementable by other "monolithic" languages (including awk).
Your double xargs programming pattern is certainly rare, but I used it
also at times, and it didn't occur to me to try to re-implement it by
awk just for the sake of praising the language or something like that.
The same with two instances of awk, it may make sense, there's no dogma.

It's not a black-or-white issue. And personal preferences vary anyway.
But sometimes the way we're used to use a tool-chest is in our way of
"finding" (actually just writing down) better solutions.

What I was addressing is the use of programs with primitive functions
that awk is providing in a simple and consistent way inherently! My
impression is that Unix folks today learn command-line patterns about
the same way I did decades ago; starting from cat, cut, grep, sed (and
so on), often not even reaching awk - because it's more complex than
the simpler tools dedicated to a task. The joined dedicated Unix tools
are not simpler, though, rather the opposite. Recognizing that let me
change the way how I start my tasks; for simple searches it wouldn't
occur to me to use awk, but as soon as a search task gets slightly
more complex, say searching for lines with two matches, /A/&&/B/, I'd
use awk. Or if it is clear from the beginning that the task will be
harder and more clumsy to implement with pipes, e.g. extracting keys
from a file to match records in another file; then I don't even think
about how that (maybe) could be implemented by function compositions
with primitive Unix programs, I'd immediately take awk if appropriate.

There are immediate gains, but also gains not obvious in the first
moment. You may observe that, say, /A/&&/B/ would return too many
results; no problem to restrict the results set simply with further
qualifications like, say, $1~/A/ && $1~/B/ , adding a FS definition,
and whatnot. Of course grep A | grep B isn't complex (less terse,
okay, but still not complex), but you're often in a dead end if your
demands get extended (e.g. by matching fields instead of lines). For
a one-shot ad hoc task the greps are okay, and I see that I sometimes
start with a command, then add another piped command to the previous
one (from shell history), and then a third one. But as soon as this
goes into a shell script these commands are getting optimized. Lately
I don't even start with pipes if I think the command might get bulky.

Note that here I am not advocating for smart awk code patterns as we
occasionally suggest here for fun, to keep things minimalistic terse.
It's just the mundane inherent features I am talking about. Features
that are consistently available with awk. And that's different to the
Unix tools set where I have a hammer, a nail, and a screwdriver, and
think about how to combine them to change the light bulb. - Yes, I'm
exaggerating (at least a bit), but you get the point.

Folks learn by examples, and we see (in shell newsgroups or in books)
often bad solutions based on published code that follows the historic
approach, copy/pasted without thinking, and copied as code pattern in
their future projects, posts, lectures, and own books.

It got a bit pathetic.

Janis

On 10.02.2022 02:37, Ben Bacarisse wrote:
> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>
>> You can always simply split() the fields, no need to invoke another
>> process just for another implicit loop that awk supports.
>
> Yes, there's no need, but why worry about it? Maybe I am alone in
> thinking processes are cheap.

I just try to avoid unnecessary processes. A dozen is not an issue,
but once you've embedded them in shell loops it might get an issue.

>
> But more to the point, a pipeline is an elegant, easily understood, and
> often natural way to organise a task.

Agreed.

>> [...]
>
> A pipeline is not the right structure for such tasks, but there are a
> huge number of tasks where combining Unix tools is the simplest
> solution.

Agreed.

>>
>> The nice thing about awk - actually already mentioned in context of
>> the features/complexity vs. power comments - is that you don't need
>> to memorize a lot;[*] I think awk is terse and compact enough. YMMV.
>
> But since I use pipelines so much, I rarely use split, patsplit, gsub or
> gensub. I find myself checking their arguments pretty much every time I
> use them.

(Some of the mentioned functions are non-standard, GNU Awk'ish.)

Well, if you don't use them regularly you'll have to look up the docs.
Personally I think the [standard] functions are easy to remember, but
okay. Myself I can easy remember them if only by thinking about their
finally placed default arguments; split (what, where [,by-what])
omitting the "by-what" will use the standard FS, and "what", "where"
I think is the natural order I'd expect, and similarly with gsub;
gsub (what, by-what [, where]) omitting the where will operate on
the whole line, and "what", "by-what" I think are the natural order.

The non-standard GNU functions patsplit() and gensub() I also have to
look up, but I think just because I rarely have a need to use these
functions.

>>
>> That's how we learned it; pipelining through simple dedicated tools.
>> I also still do that.
>
> Why? Serious question. It sound like a dreadful risk based on your
> comments above. Doing is "usually leads to such horrible and inflexible
> cascades of the tools" when there is no need "to invoke another
> process". What makes you sometimes take the risk of horrible cascades
> and pay the price of another process?

I think this is answered in my previous post, my reply to Axel's post.

It's certainly not something I'd call a risk, because I can control it,
I can make the decisions, based on application case, requirements, and
expertise.

>
> I ask because it's possible we disagree only on how frequently it should
> be done, and about exactly what circumstances warrant it.

I think we should not take the theme too dogmatic or too strict. To
quote from my other post:

>> What I was addressing is the use of programs with primitive functions
>> that awk is providing in a simple and consistent way inherently!

>
>> the whole pipeline gets then refactored, typically for efficiency,
>> flexibility, robustness, and clarity in design.
>
> That's where I disagree. I often choose a pipeline because it is the
> most robust, flexible and clear design. (I rarely care about efficiency
> when doing this sort of thing.)

We do not disagree concerning the clearness of the pipe concept. It is
just very _primitive_ (an advantage, and a restriction WRT flexibility).

>
>> I want to close my comment with another aspect; the primitive helper
>> tools are often restricted and incoherent.[*] In GNU context you have
>> additional options that I'm glad to be able to use, but if you want to
>> stay standard conforming the tools might not "suffice" or usage gets
>> more bulky. With awk the standard version supports already the powerful
>> core.
>
> I agree. That's a shame, but an inevitable cost of piecemeal historical
> development.

My resume is that we "mostly"[*] agree. :-)

Janis

[*] Term borrowed from the HHGTTG.

Janis Papanagnou <janis_papanagnou@hotmail.com> writes:

> On 10.02.2022 18:33, Axel Reichert wrote:
>>
>> P. S.: I will keep your advice in memory, though, to avoid my worst
>> excesses. Point taken.
>>
>
> Don't get me wrong. Pipelines of tools are not "bad" and I also wrote:
>
> "That's how we learned it; pipelining through simple dedicated tools.
> I also still do that. [...]"

Yes, sure, I noticed that. I do think that it's mostly about stylistic
matters at this point of the discussion. I like to compare
shell/Unix/awk/CLI issues with a language: A former boss was impressed
by what was feasible with these strange "words", which to him sounded
like Greek. He wanted me to save these utterances for others to benefit
from my "words of wisdom". I argued that they were not a "quote" to be
put into some anthology, but a spontaneous sentence formed during "live
talk". The point for me was not "memorizing" them (shell script), but
being able to speak.

Of course this analogy is valid only for ad hoc stuff, but that is how I
use them in almost all cases: These are throw-away command lines and
only very rarely I see the potential for them to be re-used. It is for
those instances that I will try to remember your words of
wisdom/warning. (-:

> Your double xargs programming pattern is certainly rare

Not for me, my guess is that I use it daily.

> as soon as a search task gets slightly more complex, say searching for
> lines with two matches, /A/&&/B/, I'd use awk.

grep A ... | grep B

or, often

grep -E '(A|B)' ...

Again, due to the ad-hoc nature of my shell usage, I will often not
notice that two matches are needed, only after the first grep command
has been executed: "Ah, I will need another match!"

> more clumsy to implement with pipes, e.g. extracting keys from a file
> to match records in another file; then I don't even think about how
> that (maybe) could be implemented by function compositions with
> primitive Unix programs

But you do know "join"? An often overlooked gem.

> a one-shot ad hoc task the greps are okay, and I see that I sometimes
> start with a command, then add another piped command to the previous
> one (from shell history), and then a third one. But as soon as this
> goes into a shell script these commands are getting optimized.

Yes. And it is at that point that I should try to reduce the number of
tools used. In fact I am a big fan of universal weapons.

Best regards

Axel

Janis Papanagnou <janis_papanagnou@hotmail.com> writes:

> On 10.02.2022 02:37, Ben Bacarisse wrote:
>> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>>
>>> You can always simply split() the fields, no need to invoke another
>>> process just for another implicit loop that awk supports.
>>
>> Yes, there's no need, but why worry about it? Maybe I am alone in
>> thinking processes are cheap.
>
> I just try to avoid unnecessary processes. A dozen is not an issue,
> but once you've embedded them in shell loops it might get an issue.

That does not really advance the discussion because almost no processes
(other than one to preform the task) are actually necessary. It's clear
that you attach a different weighting to the various considerations, but
I don't really know more than that. For my part, I can't remember the
last time I even thought about how many processes were involved in a
pipeline.

>> But more to the point, a pipeline is an elegant, easily understood, and
>> often natural way to organise a task.
>
> Agreed.
>
>>> [...]
>>
>> A pipeline is not the right structure for such tasks, but there are a
>> huge number of tasks where combining Unix tools is the simplest
>> solution.
>
> Agreed.
>
>>>
>>> The nice thing about awk - actually already mentioned in context of
>>> the features/complexity vs. power comments - is that you don't need
>>> to memorize a lot;[*] I think awk is terse and compact enough. YMMV.
>>
>> But since I use pipelines so much, I rarely use split, patsplit, gsub or
>> gensub. I find myself checking their arguments pretty much every time I
>> use them.
>
> (Some of the mentioned functions are non-standard, GNU Awk'ish.)
>
> Well, if you don't use them regularly you'll have to look up the docs.
> Personally I think the [standard] functions are easy to remember, but
> okay. Myself I can easy remember them if only by thinking about their
> finally placed default arguments; split (what, where [,by-what])
> omitting the "by-what" will use the standard FS, and "what", "where"
> I think is the natural order I'd expect, and similarly with gsub;
> gsub (what, by-what [, where]) omitting the where will operate on
> the whole line, and "what", "by-what" I think are the natural order.
>
> The non-standard GNU functions patsplit() and gensub() I also have to
> look up, but I think just because I rarely have a need to use these
> functions.
>
>>>
>>> That's how we learned it; pipelining through simple dedicated tools.
>>> I also still do that.
>>
>> Why? Serious question. It sound like a dreadful risk based on your
>> comments above. Doing is "usually leads to such horrible and inflexible
>> cascades of the tools" when there is no need "to invoke another
>> process". What makes you sometimes take the risk of horrible cascades
>> and pay the price of another process?
>
> I think this is answered in my previous post, my reply to Axel's post.
>
> It's certainly not something I'd call a risk, because I can control it,
> I can make the decisions, based on application case, requirements, and
> expertise.

That's an odd answer. Do you think I can't control the risk, or was the
advice about it "usually [leading] to such horrible and inflexible
cascades of the tools" aimed at some unstated group who are not as good
at controlling risk as you and I?

>> I ask because it's possible we disagree only on how frequently it should
>> be done, and about exactly what circumstances warrant it.
>
> I think we should not take the theme too dogmatic or too strict. To
> quote from my other post:
>
>>> What I was addressing is the use of programs with primitive functions
>>> that awk is providing in a simple and consistent way inherently!
>
>>
>>> the whole pipeline gets then refactored, typically for efficiency,
>>> flexibility, robustness, and clarity in design.
>>
>> That's where I disagree. I often choose a pipeline because it is the
>> most robust, flexible and clear design. (I rarely care about efficiency
>> when doing this sort of thing.)
>
> We do not disagree concerning the clearness of the pipe concept. It is
> just very _primitive_ (an advantage, and a restriction WRT
> flexibility).

But you also appear to consider it costly and risky. That's the
difference I think.

--
Ben.

On 11.02.2022 22:45, Axel Reichert wrote:
> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>
> Yes, sure, I noticed that. I do think that it's mostly about stylistic
> matters at this point of the discussion. I like to compare
> shell/Unix/awk/CLI issues with a language: A former boss was impressed
> by what was feasible with these strange "words", which to him sounded
> like Greek. He wanted me to save these utterances for others to benefit
> from my "words of wisdom". I argued that they were not a "quote" to be
> put into some anthology, but a spontaneous sentence formed during "live
> talk". The point for me was not "memorizing" them (shell script), but
> being able to speak.
>
> Of course this analogy is valid only for ad hoc stuff, but that is how I
> use them in almost all cases: These are throw-away command lines and
> only very rarely I see the potential for them to be re-used. [...]

I understand that analogy. And thinking about your and Ben's posts I
concluded that this must be the reason for your approach and view of
the topic. (But please correct me if I misinterpreted your reasons.)

I indeed have two types of tasks, ad hoc tools, and applications that
are supporting regular occurring tasks. And often it is the case that
the former evolve to the latter! I already described the former tasks
in my previous posts by the (often incremental) development of tools;
starting with a command and appending pipe-connected other commands
to refine the task, to make the output better usable, and whatnot. At
the other side there's the functionality that goes in the direction
of software development; I'm therefore also reluctant to call what I
am creating here as "shell script", I often call it "shell program".
How these two types of code are written/designed depends on the type.

I'd like to jump in with anecdotes from Real Life (as you've done as
well), since what I wrote can probably be better understood that way
of how I am thinking.

Quite some years ago I had an interview for a job, and since it was
a Unix/Linux running company they asked me about writing code for a
shell task; basically log file analysis (as far as my faint memories
serve). I heard the task and quickly typed a simple 1-liner of awk;
because that was what appeared to be the right tool for the given
task, and for the perspective that the requirements for such tools
or tool-applications will typically quickly get extended and grow
("implement this", "now add that", "and this function would help",
"we also missed to consider that", etc.). The interviewing person
was (positively!) astonished, and he told me that he expected some
solution with 'cut', 'uniq', 'wc', and the like. So I provided him
also the piped variant (luckily even remembering uniq's option -c),
but also pointed him out the inherent drawbacks and restrictions.

On another occasion we quickly needed an analysis of mobile phone
customer data from a production database. My boss sat adjacent and
I told him to just get and produce this data instantly; one minute
and an awk 1-liner later we had that information. His expectation
was that we'd have needed some high level programming environment,
and so he was quite fascinated. With piped commands that wouldn't
have been possible to achieve, and with a high-level language it
would have required a lot more time.

It all depends on the specific task, and the perspective whereto
the application will likely evolve (and of course also personal
preference and specific experience what makes more or less sense).

>
>> as soon as a search task gets slightly more complex, say searching for
>> lines with two matches, /A/&&/B/, I'd use awk.
>
> grep A ... | grep B

I anticipated that pattern already in my previous post to that you
are replying here (but didn't quote), so I refer to my original text.

>
>> more clumsy to implement with pipes, e.g. extracting keys from a file
>> to match records in another file; then I don't even think about how
>> that (maybe) could be implemented by function compositions with
>> primitive Unix programs
>
> But you do know "join"? An often overlooked gem.

I know the 'join' command but don't see what that has to do with what I
wrote here. By "function composition" I meant that programs represent
functions; tool x does f, tool y does g, and combining tool x and y by,
say, x|y does g o f, where o is the function connector, a composition
of functionality (and code).

Janis

On 12.02.2022 00:32, Ben Bacarisse wrote:
> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>> On 10.02.2022 02:37, Ben Bacarisse wrote:
>>>
>>> Why? Serious question. It sound like a dreadful risk based on your
>>> comments above. Doing is "usually leads to such horrible and inflexible
>>> cascades of the tools" when there is no need "to invoke another
>>> process". What makes you sometimes take the risk of horrible cascades
>>> and pay the price of another process?
>>
>> I think this is answered in my previous post, my reply to Axel's post.
>>
>> It's certainly not something I'd call a risk, because I can control it,
>> I can make the decisions, based on application case, requirements, and
>> expertise.
>
> That's an odd answer. Do you think I can't control the risk,

No, Ben. You were (IMO unnecessarily) introducing the (IMO also
inappropriate) term "risk". And I pointed out that I see risks
only in cases where one cannot control the situation or where I
am restricted in any ways in my decisions. I was neither saying
nor implying anything about you. (This discussion got a spin in
an [emotional] direction that I don't want to follow.)

In a reply to Axel (a few minutes ago) I've just extended on my
view; it may (or may not) shed some light on some open questions
or maybe give some insights about possible differences of personal
approaches or mindsets.

Janis

Janis Papanagnou <janis_papanagnou@hotmail.com> writes:

> On 12.02.2022 00:32, Ben Bacarisse wrote:
>> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>>> On 10.02.2022 02:37, Ben Bacarisse wrote:
>>>>
>>>> Why? Serious question. It sound like a dreadful risk based on your
>>>> comments above. Doing is "usually leads to such horrible and inflexible
>>>> cascades of the tools" when there is no need "to invoke another
>>>> process". What makes you sometimes take the risk of horrible cascades
>>>> and pay the price of another process?
>>>
>>> I think this is answered in my previous post, my reply to Axel's post.
>>>
>>> It's certainly not something I'd call a risk, because I can control it,
>>> I can make the decisions, based on application case, requirements, and
>>> expertise.
>>
>> That's an odd answer. Do you think I can't control the risk,
>
> No, Ben. You were (IMO unnecessarily) introducing the (IMO also
> inappropriate) term "risk".

It seems to me reasonable to characterise doing something that "usually
leads to such horrible and inflexible cascades of the tools" as taking a
risk. A practice (that you describe as an impulse!) that usually leads
to anything horrible and inflexible is surely risky, isn't it?

> And I pointed out that I see risks
> only in cases where one cannot control the situation or where I
> am restricted in any ways in my decisions.

I accept that this is what you intended to say, but it's not what you
said. There was no inclusive "one" in your justification for your use
of pipelines of basic tools.

You had warned me of where following the "impulse" I had demonstrated in
my example usually leads, but when I was surprised that you "still do
that" you tell me it's fine because "I can control it, I can make the
decisions, based on application case, requirements, and expertise". No
"where one can make the decisions", no "where one can control it".

> I was neither saying nor implying anything about you.

If you did not intend your remarks imply and say things about me, you
phrased it badly. Talking about developing commands "that way",
immediately after an example of mine inevitably includes me in the
remark about it being an (understandable) impulse.

> (This discussion got a spin in
> an [emotional] direction that I don't want to follow.)

Telling someone they are acting on an impulse that usually leads to
horrible and inflexible code is obviously somewhat personal.

--
Ben.

Janis Papanagnou <janis_papanagnou@hotmail.com> writes:

> On 11.02.2022 22:45, Axel Reichert wrote:
>> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>>
>>> more clumsy to implement with pipes, e.g. extracting keys from a file
>>> to match records in another file; then I don't even think about how
>>> that (maybe) could be implemented by function compositions with
>>> primitive Unix programs
>>
>> But you do know "join"? An often overlooked gem.
>
> I know the 'join' command but don't see what that has to do with what I
> wrote here. By "function composition" I meant that programs represent
> functions; tool x does f, tool y does g, and combining tool x and y by,
> say, x|y does g o f, where o is the function connector, a composition
> of functionality (and code).

foo-1.txt:
foo 1 2 3
Foo 4 5 6 7
FOO 8 9

foo-2.txt:
foo 456
Foo 45 67
FOO 89

To me, the first column seems like a key and the whole line like a
record. To get something like

foo-joined.txt:
foo 1 2 456
Foo 4 5 45
Foo 8 9 89

would be a typical job for join. Hence my question. But we digress from
awk. (-:

Axel