Hi, thank you for your outstanding contributions and discussions on awk.

Working with it from more than 20 years now and still amazed at the power of this wonderful language !

This is my modest contribution to shed light on a usage too little documented on the Internet, I named "Record Separator" : https://rosettacode.org/wiki/Search_in_paragraph%27s_text

Olivier Gabathuler

On 13.02.2022 22:00, Axel Reichert wrote:
> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>
>> On 11.02.2022 22:45, Axel Reichert wrote:
>>> Janis Papanagnou <janis_papanagnou@hotmail.com> writes:
>>>
>>>> more clumsy to implement with pipes, e.g. extracting keys from a file
>>>> to match records in another file; then I don't even think about how
>>>> that (maybe) could be implemented by function compositions with
>>>> primitive Unix programs
>>>
>>> But you do know "join"? An often overlooked gem.
>>
>> I know the 'join' command but don't see what that has to do with what I
>> wrote here. By "function composition" I meant that programs represent
>> functions; tool x does f, tool y does g, and combining tool x and y by,
>> say, x|y does g o f, where o is the function connector, a composition
>> of functionality (and code).
>
> foo-1.txt:
> foo 1 2 3
> Foo 4 5 6 7
> FOO 8 9
>
> foo-2.txt:
> foo 456
> Foo 45 67
> FOO 89
>
> To me, the first column seems like a key and the whole line like a
> record.

Sure. You join two data sets identified by a common key. But so what?

You have probably been triggered by the formulation of a sample use
("extracting keys from a file to match records in another file") in
my post where I was more aiming at patterns like

awk '
NR==FNR { map[$1] ; next }
$1 in map
' keys data

(i.e. a filtering task - in a simple form also doable by grep) or like

awk '
NR==FNR { map[$1] = $2 ; next }
{ for (i in map) gsub (i, map[i]) }
1
' mapping data

(i.e. a simple replacement task).

One point was that function compositions by piped commands may (or not)
work for reductions of original data but if you need context from a
previous stage you are lost (or rather; need workarounds).

> To get something like
>
> foo-joined.txt:
> foo 1 2 456
> Foo 4 5 45
> Foo 8 9 89
>
> would be a typical job for join. Hence my question.

Yes, as I said in previous posts as well, I do know join but that was
not what I had been speaking about.

> But we digress from awk. (-:

I hope I dragged the thread back on topic with the awk samples. ;-)

Janis

>
> Axel
>

On 16.02.2022 23:11, olivier gabathuler wrote:
> Hi, thank you for your outstanding contributions and discussions on
> awk.
>
> Working with it from more than 20 years now and still amazed at the
> power of this wonderful language !
>
> This is my modest contribution to shed light on a usage too little
> documented on the Internet, I named "Record Separator" :
> https://rosettacode.org/wiki/Search_in_paragraph%27s_text

I had a peek view into the awk code and (unstructured) data sample.

The task is described not very specific as:
"The goal is to verify the presence of a word or regular expression
within several paragraphs of text (structured or not) and to print
the relevant paragraphs on the standard output."

When I saw the code I first wondered about the definition of a two
newlines output record separator just to define the same as input
separator to the next awk stage. (An indication for a candidate to
be refactored.)

It seems that your code basically extracts from records of blocks
those blocks that contain a specific string. In addition it changes
the data in a subtle way beyond the formulated task description.

Personally my first attempt for such a task would have been simpler
(using awk's multi-line data blocks feature), something like

awk 'BEGIN { RS = "" ; ORS = "\n----------------\n" }
/Traceback/ && /SystemError/
' Traceback.txt

with possible extensions to test for the patterns in specific fields
(by adding FS = "\n") so that the patterns if appearing in the data
won't compromise the correct function.

(Note that the output of above code keeps the matched data intact.)

Yes, features relying on the separators allow interesting solutions.
(In the given case it's arguable whether they've been used sensibly.)

Janis

>
> Olivier Gabathuler
>

Janis Papanagnou <janis_papanagnou@hotmail.com> writes:

> Sure. You join two data sets identified by a common key. But so what?

Hey, I was very proud when I discovered this 15 years ago. (-:

> You have probably been triggered by the formulation of a sample use
> ("extracting keys from a file to match records in another file") in my
> post

Yes.

> I hope I dragged the thread back on topic with the awk samples. ;-)

You did, and I am happy to learn more here from, it seems, much more
advanced awk usage than I am used to so far. Thanks!

Axel

Le jeudi 17 février 2022 à 03:36:11 UTC+1, Janis Papanagnou a écrit :
> On 16.02.2022 23:11, olivier gabathuler wrote:
> > Hi, thank you for your outstanding contributions and discussions on
> > awk.
> >
> > Working with it from more than 20 years now and still amazed at the
> > power of this wonderful language !
> >
> > This is my modest contribution to shed light on a usage too little
> > documented on the Internet, I named "Record Separator" :
> > https://rosettacode.org/wiki/Search_in_paragraph%27s_text
> I had a peek view into the awk code and (unstructured) data sample.
>
> The task is described not very specific as:
> "The goal is to verify the presence of a word or regular expression
> within several paragraphs of text (structured or not) and to print
> the relevant paragraphs on the standard output."
>
> When I saw the code I first wondered about the definition of a two
> newlines output record separator just to define the same as input
> separator to the next awk stage. (An indication for a candidate to
> be refactored.)
>
> It seems that your code basically extracts from records of blocks
> those blocks that contain a specific string. In addition it changes
> the data in a subtle way beyond the formulated task description.
>
> Personally my first attempt for such a task would have been simpler
> (using awk's multi-line data blocks feature), something like
>
> awk 'BEGIN { RS = "" ; ORS = "\n----------------\n" }
> /Traceback/ && /SystemError/
> ' Traceback.txt
>
> with possible extensions to test for the patterns in specific fields
> (by adding FS = "\n") so that the patterns if appearing in the data
> won't compromise the correct function.
>
> (Note that the output of above code keeps the matched data intact.)
>
> Yes, features relying on the separators allow interesting solutions.
> (In the given case it's arguable whether they've been used sensibly.)
>
> Janis
>
> >
> > Olivier Gabathuler
> >

Hi Janis,
thanks for your response :-)

Just to understand, the output with
> awk 'BEGIN { RS = "" ; ORS = "\n----------------\n" }
> /Traceback/ && /SystemError/
> ' Traceback.txt
is :
...
----------------
[Tue Jan 21 16:16:19.250245 2020] [wsgi:error] [pid 6515:tid 3041002528] [remote 10.0.0.12:50757] Traceback (most recent call last):
[Tue Jan 21 16:16:19.252221 2020] [wsgi:error] [pid 6515:tid 3041002528] [remote 10.0.0.12:50757] SystemError: unable to access /home/dir
[Tue Jan 21 16:16:19.249067 2020] [wsgi:error] [pid 6515:tid 3041002528] [remote 10.0.0.12:50757] mod_wsgi (pid=6515): Failed to exec Python script file '/home/pi/RaspBerryPiAdhan/www/sysinfo.wsgi'.
[Tue Jan 21 16:16:19.249609 2020] [wsgi:error] [pid 6515:tid 3041002528] [remote 10.0.0.12:50757] mod_wsgi (pid=6515): Exception occurred processing WSGI script '/home/pi/RaspBerryPiAdhan/www/sysinfo.wsgi'.
----------------
12/01 19:24:57.726 ERROR| log:0072| post-test sysinfo error: 11/01 18:24:57..727 ERROR| traceback:0013| Traceback (most recent call last): 11/01 18:24:57.728 ERROR| traceback:0013| File "/tmp/sysinfo/autoserv-0tMj3m/common_lib/log.py", line 70, in decorated_func 11/01 18:24:57.729 ERROR| traceback:0013| fn(*args, **dargs) 11/01 18:24:57.730 ERROR| traceback:0013| File "/tmp/sysinfo/autoserv-0tMj3m/bin/base_sysinfo.py", line 286, in log_after_each_test 11/01 18:24:57.731 ERROR| traceback:0013| old_packages = set(self._installed_packages) 11/01 18:24:57.731 ERROR| traceback:0013| SystemError: no such file or directory
----------------
...
-> not exactly the output I expect, but as you said, I was not specific enough in the description of the output formatting.
I will fix that.

In fact I took this example, but in my working life on +10k Linux boxes as sysadmin, I used RS extensively to parse a lot of logs, so..

Olivier G.

On 18.02.2022 19:08, olivier gabathuler wrote:
> Le jeudi 17 février 2022 à 03:36:11 UTC+1, Janis Papanagnou a écrit :
>> On 16.02.2022 23:11, olivier gabathuler wrote:
>>> Hi, thank you for your outstanding contributions and discussions on
>>> awk.
>>>
>>> Working with it from more than 20 years now and still amazed at the
>>> power of this wonderful language !
>>>
>>> This is my modest contribution to shed light on a usage too little
>>> documented on the Internet, I named "Record Separator" :
>>> https://rosettacode.org/wiki/Search_in_paragraph%27s_text
>> I had a peek view into the awk code and (unstructured) data sample.
>>
>> The task is described not very specific as:
>> "The goal is to verify the presence of a word or regular expression
>> within several paragraphs of text (structured or not) and to print
>> the relevant paragraphs on the standard output."
>>
>> When I saw the code I first wondered about the definition of a two
>> newlines output record separator just to define the same as input
>> separator to the next awk stage. (An indication for a candidate to
>> be refactored.)
>>
>> It seems that your code basically extracts from records of blocks
>> those blocks that contain a specific string. In addition it changes
>> the data in a subtle way beyond the formulated task description.
>>
>> Personally my first attempt for such a task would have been simpler
>> (using awk's multi-line data blocks feature), something like
>>
>> awk 'BEGIN { RS = "" ; ORS = "\n----------------\n" }
>> /Traceback/ && /SystemError/
>> ' Traceback.txt
>>
>> with possible extensions to test for the patterns in specific fields
>> (by adding FS = "\n") so that the patterns if appearing in the data
>> won't compromise the correct function.
>>
>> (Note that the output of above code keeps the matched data intact.)
>>
>> Yes, features relying on the separators allow interesting solutions.
>> (In the given case it's arguable whether they've been used sensibly.)
>>
>> Janis
>>
>>>
>>> Olivier Gabathuler
>>>
>
> Hi Janis,
> thanks for your response :-)
>
> Just to understand, the output with
>> awk 'BEGIN { RS = "" ; ORS = "\n----------------\n" }
>> /Traceback/ && /SystemError/
>> ' Traceback.txt
> is :
> ..
> ----------------
> [Tue Jan 21 16:16:19.250245 2020] [wsgi:error] [pid 6515:tid 3041002528] [remote 10.0.0.12:50757] Traceback (most recent call last):
> [Tue Jan 21 16:16:19.252221 2020] [wsgi:error] [pid 6515:tid 3041002528] [remote 10.0.0.12:50757] SystemError: unable to access /home/dir
> [Tue Jan 21 16:16:19.249067 2020] [wsgi:error] [pid 6515:tid 3041002528] [remote 10.0.0.12:50757] mod_wsgi (pid=6515): Failed to exec Python script file '/home/pi/RaspBerryPiAdhan/www/sysinfo.wsgi'.
> [Tue Jan 21 16:16:19.249609 2020] [wsgi:error] [pid 6515:tid 3041002528] [remote 10.0.0.12:50757] mod_wsgi (pid=6515): Exception occurred processing WSGI script '/home/pi/RaspBerryPiAdhan/www/sysinfo.wsgi'.
> ----------------
> 12/01 19:24:57.726 ERROR| log:0072| post-test sysinfo error: 11/01 18:24:57.727 ERROR| traceback:0013| Traceback (most recent call last): 11/01 18:24:57.728 ERROR| traceback:0013| File "/tmp/sysinfo/autoserv-0tMj3m/common_lib/log.py", line 70, in decorated_func 11/01 18:24:57.729 ERROR| traceback:0013| fn(*args, **dargs) 11/01 18:24:57.730 ERROR| traceback:0013| File "/tmp/sysinfo/autoserv-0tMj3m/bin/base_sysinfo.py", line 286, in log_after_each_test 11/01 18:24:57.731 ERROR| traceback:0013| old_packages = set(self._installed_packages) 11/01 18:24:57.731 ERROR| traceback:0013| SystemError: no such file or directory
> ----------------
> ..
> -> not exactly the output I expect, but as you said, I was not specific enough in the description of the output formatting.
> I will fix that.

Actually I was much more saying and implying. To expand on it...

From the code and the task description it was unclear whether the
output of your script was just by accident or deliberately beyond
the description on the web page.

If it was by accident differing - as a consequence of a convoluted
design based on the field separators - then above simple code is an
immediate improvement (in more than one aspect).

If your task was actually to output the matching lines, but these
matching lines should start from the keyword "Traceback" (and the
leading time stamps suppressed), then you can and should formulate
that in a clean way; not only the description but also the code
should be clearly formulated.

A clean awk function is simply substr($0,index($0,"Traceback"))
and the resulting code still clean and comprehensible; instead of
printing the whole record

awk 'BEGIN { RS = "" ; ORS = "\n----------------\n" }

/Traceback/ && /SystemError/ ## this implies: print $0

' Traceback.txt

you just print the desired part

awk 'BEGIN { RS = "" ; ORS = "\n----------------\n" }

/Traceback/ && /SystemError/ {
print substr($0,index($0,"Traceback"))
}
' Traceback.txt

A simple straightforward addition without side-effects or any hard
to follow program logic. No unnecessary awk instances, FS-fiddling,
or anything.

And this code prints at least the same output as the code you posted
on that web page. That code of yours was

awk -v ORS='\n\n' '/SystemError/ { print RS $0 }'
RS="Traceback" Traceback.txt |\
awk -v ORS='\n----------------\n' '/Traceback/' RS="\n\n"

If you think this code is in any way to prefer I'd be interested in
your explanations. - No, not really, that was just rhetorical.

>
> In fact I took this example, but in my working life on +10k Linux boxes as sysadmin, I used RS extensively to parse a lot of logs, so..

In fact, if that posted code you showed here is a characteristic code
sample, then I doubt that it's a good idea to spread it to +10k Linux
systems.

But that taunt aside; there's nothing wrong in using the awk separators,
it's a basic feature any proficient awk authority will [sensibly] use.
It's its pathological or unnecessary use I consider to be problematic.

YMMV.

Janis

>
> Olivier G.
>

On 18.02.2022 21:57, Janis Papanagnou wrote:
> It's its pathological or unnecessary use I consider to be problematic.

It just occurred to me that the circle closes. The thread started
with a book called "The Art of Unix Programming", resembling the
classic books title "The Art of Computer Programming" from Donald
Knuth, from an epoch where there was a clear path away from hacking
code together individually by dedicated computer experts, towards
taking a more systematic and scientific approach. My feeling is that
we're still balancing on the bleeding edge of software development,
between hacks and - whatever.

Janis

Le vendredi 18 février 2022 à 21:57:14 UTC+1, Janis Papanagnou a écrit :
> On 18.02.2022 19:08, olivier gabathuler wrote:
> > Le jeudi 17 février 2022 à 03:36:11 UTC+1, Janis Papanagnou a écrit :
> >> On 16.02.2022 23:11, olivier gabathuler wrote:
> >>> Hi, thank you for your outstanding contributions and discussions on
> >>> awk.
> >>>
> >>> Working with it from more than 20 years now and still amazed at the
> >>> power of this wonderful language !
> >>>
> >>> This is my modest contribution to shed light on a usage too little
> >>> documented on the Internet, I named "Record Separator" :
> >>> https://rosettacode.org/wiki/Search_in_paragraph%27s_text
> >> I had a peek view into the awk code and (unstructured) data sample.
> >>
> >> The task is described not very specific as:
> >> "The goal is to verify the presence of a word or regular expression
> >> within several paragraphs of text (structured or not) and to print
> >> the relevant paragraphs on the standard output."
> >>
> >> When I saw the code I first wondered about the definition of a two
> >> newlines output record separator just to define the same as input
> >> separator to the next awk stage. (An indication for a candidate to
> >> be refactored.)
> >>
> >> It seems that your code basically extracts from records of blocks
> >> those blocks that contain a specific string. In addition it changes
> >> the data in a subtle way beyond the formulated task description.
> >>
> >> Personally my first attempt for such a task would have been simpler
> >> (using awk's multi-line data blocks feature), something like
> >>
> >> awk 'BEGIN { RS = "" ; ORS = "\n----------------\n" }
> >> /Traceback/ && /SystemError/
> >> ' Traceback.txt
> >>
> >> with possible extensions to test for the patterns in specific fields
> >> (by adding FS = "\n") so that the patterns if appearing in the data
> >> won't compromise the correct function.
> >>
> >> (Note that the output of above code keeps the matched data intact.)
> >>
> >> Yes, features relying on the separators allow interesting solutions.
> >> (In the given case it's arguable whether they've been used sensibly.)
> >>
> >> Janis
> >>
> >>>
> >>> Olivier Gabathuler
> >>>
> >
> > Hi Janis,
> > thanks for your response :-)
> >
> > Just to understand, the output with
> >> awk 'BEGIN { RS = "" ; ORS = "\n----------------\n" }
> >> /Traceback/ && /SystemError/
> >> ' Traceback.txt
> > is :
> > ..
> > ----------------
> > [Tue Jan 21 16:16:19.250245 2020] [wsgi:error] [pid 6515:tid 3041002528] [remote 10.0.0.12:50757] Traceback (most recent call last):
> > [Tue Jan 21 16:16:19.252221 2020] [wsgi:error] [pid 6515:tid 3041002528] [remote 10.0.0.12:50757] SystemError: unable to access /home/dir
> > [Tue Jan 21 16:16:19.249067 2020] [wsgi:error] [pid 6515:tid 3041002528] [remote 10.0.0.12:50757] mod_wsgi (pid=6515): Failed to exec Python script file '/home/pi/RaspBerryPiAdhan/www/sysinfo.wsgi'.
> > [Tue Jan 21 16:16:19.249609 2020] [wsgi:error] [pid 6515:tid 3041002528] [remote 10.0.0.12:50757] mod_wsgi (pid=6515): Exception occurred processing WSGI script '/home/pi/RaspBerryPiAdhan/www/sysinfo.wsgi'.
> > ----------------
> > 12/01 19:24:57.726 ERROR| log:0072| post-test sysinfo error: 11/01 18:24:57.727 ERROR| traceback:0013| Traceback (most recent call last): 11/01 18:24:57.728 ERROR| traceback:0013| File "/tmp/sysinfo/autoserv-0tMj3m/common_lib/log.py", line 70, in decorated_func 11/01 18:24:57.729 ERROR| traceback:0013| fn(*args, **dargs) 11/01 18:24:57.730 ERROR| traceback:0013| File "/tmp/sysinfo/autoserv-0tMj3m/bin/base_sysinfo.py", line 286, in log_after_each_test 11/01 18:24:57.731 ERROR| traceback:0013| old_packages = set(self._installed_packages) 11/01 18:24:57.731 ERROR| traceback:0013| SystemError: no such file or directory
> > ----------------
> > ..
> > -> not exactly the output I expect, but as you said, I was not specific enough in the description of the output formatting.
> > I will fix that.
> Actually I was much more saying and implying. To expand on it...
>
> From the code and the task description it was unclear whether the
> output of your script was just by accident or deliberately beyond
> the description on the web page.
>
> If it was by accident differing - as a consequence of a convoluted
> design based on the field separators - then above simple code is an
> immediate improvement (in more than one aspect).
>
> If your task was actually to output the matching lines, but these
> matching lines should start from the keyword "Traceback" (and the
> leading time stamps suppressed), then you can and should formulate
> that in a clean way; not only the description but also the code
> should be clearly formulated.
>
> A clean awk function is simply substr($0,index($0,"Traceback"))
> and the resulting code still clean and comprehensible; instead of
> printing the whole record
> awk 'BEGIN { RS = "" ; ORS = "\n----------------\n" }
> /Traceback/ && /SystemError/ ## this implies: print $0
>
> ' Traceback.txt
>
> you just print the desired part
> awk 'BEGIN { RS = "" ; ORS = "\n----------------\n" }
> /Traceback/ && /SystemError/ {
> print substr($0,index($0,"Traceback"))
> }
> ' Traceback.txt
>
> A simple straightforward addition without side-effects or any hard
> to follow program logic. No unnecessary awk instances, FS-fiddling,
> or anything.
>
> And this code prints at least the same output as the code you posted
> on that web page. That code of yours was
>
> awk -v ORS='\n\n' '/SystemError/ { print RS $0 }'
> RS="Traceback" Traceback.txt |\
> awk -v ORS='\n----------------\n' '/Traceback/' RS="\n\n"
>
> If you think this code is in any way to prefer I'd be interested in
> your explanations. - No, not really, that was just rhetorical.
> >
> > In fact I took this example, but in my working life on +10k Linux boxes as sysadmin, I used RS extensively to parse a lot of logs, so..
> In fact, if that posted code you showed here is a characteristic code
> sample, then I doubt that it's a good idea to spread it to +10k Linux
> systems.
>
> But that taunt aside; there's nothing wrong in using the awk separators,
> it's a basic feature any proficient awk authority will [sensibly] use.
> It's its pathological or unnecessary use I consider to be problematic.
>
> YMMV.
>
> Janis
>
> >
> > Olivier G.
> >

Hi Janis,
thank you very munch for your explanations and apologize for my mistake.
Yes you stated it properly, this is exaclty what I wanted (and with using index function).
I will fix that on rosettacode.org
Have a nice day !

Olivier G.

@Axel :

how many scripting languages you know of that can calculate the modulo of 3 for a single hexadecimal, 2.07 GB in size, with a log2(x) value of approx. 8.9 billion (~8,898,328,444),

in just 14.3 seconds :

echo; ( time ( nice echo '0x888889999888888888877765432CC111111111111111111111188888888CCCCCCCCC88888888899998088888888877765432111111111111111110000000000000011111888888888DDDDDDDD8888888899998888888FFFFFFFFFFFFFFFFFFFFFFFFFfFFFFFFFFFFf88877765432111111111111111111111188888888888888888999988888888887776543211111111111111111111118888AAA88888888888889B9998888888888777654321111111111111111111111' | mawk2 'sub(/^0[Xx]/,"")+gsub(//,$0,$0)+gsub(/........................../,$0,$0)+sub("^",("0x1")($0)($0),$0)+1' | pvE0 | mawk2 'function mod3(_) {
return \
(sub("^0[Xx]","",_)<(substr(_,76,1)==""))\
? (length(_)<16?(+_):substr(_,+1,15)+substr(_,16,15)+\
substr(_,31,15)+substr(_,46,15)+substr(_,61,15) \
) % 3 \
: gsub("[ -0369CFILORUXcfx_\n]+","",_)*0+\
(length(_)*+2 + \
gsub("[258BEHKNQTWZbe]+","",_)*0-\
length(_)) % 3 } BEGIN { FS=ORS; RS="^$" } END { print mod3($1) }' ) | pvE9 | ggP '[0-9]*'| lgp3 ) |ecp

in0: 0.00 B 0:00:00 [0.00 B/s] [0.00 B/s] [<=> ]
out9: 2.00 B 0:00:14 [ 143miB/s] [ 143miB/s] [<=> ]
in0: 2.07GiB 0:00:01 [1.72GiB/s] [1.72GiB/s] [ <=> ]
( nice echo | mawk2 | pvE 0.1 in0 | mawk2 ; ) 12.21s user 2.25s system 101% cpu 14.299 total
pvE 0.1 out9 0.01s user 0.02s system 0% cpu 14.298 total
ggrep --text -P --color=always '[0-9]*' 0.00s user 0.00s system 0% cpu 14.296 total

1

On Wednesday, February 9, 2022 at 2:50:01 AM UTC-5, Axel Reichert wrote:
> Kpop 2GM <> writes:
>
> > i'm an ultra late-comer to awk - only discovering it in 2017-2018. and
> > the moment i found it, i realized nearly all else - perl R python java
> > C# - can be thrown straight into the toilet, if performance is a key
> > criteria for the task at a hand
> I would rather go for TCW (Total Cost of Wizardry): A competent Python
> programmer once consulted me on performance tuning for an (ASCII data
> mangling) script he had written (which took him about 30 min). It was
> running since 10 min, and no end in sight according to a monitor on the
> (transformed) output. After he had explained the task at hand, I replied
> that I would not use Python, but rather some Unix command line tools. I
> started immediately, cobbled something together (awk featured
> prominently among other usual suspects, such as tr, sed, cut, grep). It
> delivered the desired results before his Python script was finished. So
> the final tally was "10 min" versus "> 30 min + 10 min + 10 min".
>
> Once the logic becomes more intricate, I will usually go for Python
> though, so I will use awk mostly for command line use, rarely as a file
> to be run by "awk -f".
>
> I was also a later-comer to this tool. When I started to learn Perl in
> the late 90s, I learned that it was a superset to sed and awk (coming
> even with conversion scripts), and so I gave the older tools another try
> (the "man" pages were completely incomprehensible to me before, I could
> not wrap my head around stream processing). Once it clicked, I rarely
> used Perl anywmore.
>
> Same goes for spreadsheet tools, for which I also seldom feel the need.
>
> Best regards
>
> Axel

@Axel :

interesting that you brought up unix CLI utilities : gnu-sed is almost unbelievably slow in this very basic replacement task -

- replacing all odd-digits with a 1, and
- replacing all even digits with a 0

Maybe you can tell me what I'm doing wrong in gnu-sed, cuz I can't seem to be able to even get it to within 3x slower than mawk-1.3.4.

f='jwengowengonoewgnwoegn.txt'; gwc -lcm "${f}";
echo;
( time ( pv -q < "${f}" | mawk 'BEGIN {FS=(FS0="[3579]")(FS0);OFS="11"; ORS=""; RS="^$" } (NF=NF)+gsub("[2468][2468]","00")+gsub("[2468]","0")+gsub(FS0,"1")' | pvE 0.25 mid | xxh128sum ) ) | lgp3 ;
sleep 1;
( time ( pv -q < "${f}" | gsed -zE 's/[3579]{2}/11/g;s/[2468]{2}/00/g;s/[3579]/1/g;s/[2468]/0/g' | pvE 0.25 mid | xxh128sum ) ) | lgp3

1 275150613 275150613 jwengowengonoewgnwoegn.txt

mid: 262MiB 0:00:02 [94.8MiB/s] [94.8MiB/s] [<=> ]
( pv -q < "${f}" | mawk | pvE 0.25 mid | xxh128sum; ) 2.52s user 0.35s system 102% cpu 2.788 total
1fee31c387358cc9b13eea846746dfbc stdin

mid: 262MiB 0:00:08 [30.1MiB/s] [30.1MiB/s] [<=> ]
( pv -q < "${f}" | gsed -zE | pvE 0.25 mid | xxh128sum; ) 8.57s user 0.29s system 101% cpu 8.745 total
1fee31c387358cc9b13eea846746dfbc stdin

The 4Chan Teller