On 3/7/24 18:09, Keith Thompson wrote:
> I know that's what awk does, but I don't think I would have expected
> it if I didn't know about it.

Okay. I think that's a fair observation.

> $0 is the current input line.

Or $0 is the current /record/ in awk parlance.

> If you don't change anything, or if you modify $0 itself, whitespace
> betweeen fields is preserved.

> If you modify any of the fields, $0 is recomputed and whitespace
> between tokens is collapsed.

I don't agree with that.

% echo 'one two three' | awk '{print $0; print $1,$2,$3}'
one two three
one two three

I didn't /modify/ anything and awk does print the fields with different
white space.

> awk *could* have been defined to preserve inter-field whitespace even
> when you modify individual fields,

I question the veracity of that. Specifically when lengthening or
shortening the value of a field. E.g. replacing "two" with "fifteen".
This is particularly germane when you look at $0 as a fixed width
formatted output.

> and I think I would have found that more intuitive.

I don't agree.

> (And ideally there would be a way to refer to that inter-field
> whitespace.)

Remember, awk is meant for working on fields of data in a record. By
default, the fields are delimited by white space characters. I'll say
it this way, awk is meant for working on the non-white space characters.
Or yet another way, awk is not meant for working on white space charters.

> The fact that modifying a field has the side effect of messing up $0
> seems counterintuitive.

Maybe.

But I think it's one that is acceptable for what awk is intended to do.

> Perhaps the behavior matches your intuition better than it matches
> mine.

I sort of feel like you are wanting to / trying to use awk in places
where sed might be better. sed just sees a string of text and is
ignorant of any structure without a carefully crafted RE to provide it.

Conversely awk is quite happy working with an easily identified field
based on the count with field separators of one or more white space
characters.

Consider the output of `netstat -an` wherein you have multiple columns
of IP addresses.

Please find a quick way, preferably that doesn't involve negation
(because what needs to be negated may bey highly dynamic) that lists
inbound SMTP connections on an email server but doesn't list outbound
SMTP connections.

awk makes it trivial to identify and print records that have the SMTP
port in the local IP column, thus ignoring outbound connections with
SMTP in the remote column.

Aside: Yes, I know that ss and the likes have more features for this,
but this is my example and ss is not installed everywhere.

I sort of view awk as somewhat akin to SQL wherein fields in awk are
like columns in SQL.

I'd be more than a little bit surprised to find an SQL interface that
preserved white space /between/ columns. -- Many will do it /within/
columns.

awk makes it trivial to take field oriented output from commands and
apply some logic / parsing / action on specific fields in records.

> (And perhaps this should be moved to comp.lang.awk if it doesn't die
> out soon.

comp.lang.awk added and followup pointed there.

> Though both sed and awk are both languages in their own right
> and tools that can be used from the shell, so I'd argue there's a
> topicality overlap.)

;-)

--
Grant. . . .

Grant Taylor <gtaylor@tnetconsulting.net> writes:
> On 3/7/24 18:09, Keith Thompson wrote:
>> I know that's what awk does, but I don't think I would have expected
>> it if I didn't know about it.
>
> Okay. I think that's a fair observation.
>
>> $0 is the current input line.
>
> Or $0 is the current /record/ in awk parlance.

Yes.

>> If you don't change anything, or if you modify $0 itself, whitespace
>> betweeen fields is preserved.
>
>> If you modify any of the fields, $0 is recomputed and whitespace
>> between tokens is collapsed.
>
> I don't agree with that.
>
> % echo 'one two three' | awk '{print $0; print $1,$2,$3}'
> one two three
> one two three
>
> I didn't /modify/ anything and awk does print the fields with
> different white space.

That's just the semantics of print with comma-delimited arguments, just
like:

% awk 'BEGIN{a="foo"; b="bar"; print a, b}'
foo bar

Printing the values of $1, $2, and $3 doesn't change $0. Writing to any
of $1, $2, $3, even with the same value, does change $0.

$ echo 'one two three' | awk '{print $0; print $1,$2,$3; print $0; $2 = $2; print $0}'
one two three
one two three
one two three
one two three

>> awk *could* have been defined to preserve inter-field whitespace
>> even when you modify individual fields,
>
> I question the veracity of that. Specifically when lengthening or
> shortening the value of a field. E.g. replacing "two" with
> "fifteen". This is particularly germane when you look at $0 as a fixed
> width formatted output.

But awk doesn't work with fixed-width data. The length of each field,
and the length of $0, is variable.

If awk *purely* dealt with input lines only as lists of tokens, then
this:

echo 'one two three' | awk '{print $0}'

would print "one two three" rather than "one two three" (and awk would
lose the ability to deal with arbitrarily formatted input). The fact
that the inter-field whitespace is reset only when individual fields are
touched feels arbitrary to me.

>> and I think I would have found that more intuitive.
>
> I don't agree.
>
>> (And ideally there would be a way to refer to that inter-field
>> whitespace.)
>
> Remember, awk is meant for working on fields of data in a record. By
> default, the fields are delimited by white space characters. I'll say
> it this way, awk is meant for working on the non-white space
> characters. Or yet another way, awk is not meant for working on
> white space charters.

Awk has strong builtin support for working on whitespace-delimited
fields, and that support tends to ignore the details of that whitespace.
But you can also write awk code that just deals with $0.

One trivial example:

awk '{ count += length + 1 } END { print count }'

behaves similarly to `wc -l`, and counts whitespace characters just like
any other characters.

>> The fact that modifying a field has the side effect of messing up $0
>> seems counterintuitive.
>
> Maybe.
>
> But I think it's one that is acceptable for what awk is intended to do.

It's also the existing behavior, and changing it would break things, so
I wouldn't suggest changing it.

>> Perhaps the behavior matches your intuition better than it matches
>> mine.
>
> I sort of feel like you are wanting to / trying to use awk in places
> where sed might be better. sed just sees a string of text and is
> ignorant of any structure without a carefully crafted RE to provide it.

Not really. I'm just remarking on one particular awk feature that I
find a bit counterintuitive.

Awk is optimized for working on records consisting of fields, and not
caring much about how much whitespace there is between fields. But it's
flexible enought to do *lots* of other things.

[...]

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

On 8/3/2024 12:06 pm, Keith Thompson wrote:
>
> Not really. I'm just remarking on one particular awk feature that I
> find a bit counterintuitive.
>
> Awk is optimized for working on records consisting of fields, and not
> caring much about how much whitespace there is between fields. But it's
> flexible enought to do *lots* of other things.

The original Awk doesn't support regular expressions, right? Because
regex was not yet talked about back then??

On 2024-03-08, Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:
> But awk doesn't work with fixed-width data. The length of each field,
> and the length of $0, is variable.

GNU Awk, however, can. It has a FIELDWIDTHS variable where you can
specify column widths that then work instead of the FS field separator.

There is also FPAT (see below).

> If awk *purely* dealt with input lines only as lists of tokens, then
> this:
>
> echo 'one two three' | awk '{print $0}'
>
> would print "one two three" rather than "one two three" (and awk would
> lose the ability to deal with arbitrarily formatted input). The fact
> that the inter-field whitespace is reset only when individual fields are
> touched feels arbitrary to me.

There is no inter-field whitespace.

There is the original record in $0, and parsed out fields in $1, $2, ..

The fields don't have any space.

The space comes from the value of the OFS variable.

$ echo 'one two three' | awk -v OFS=: '{ $1=$1; print }'
one:two:three

GNU Awk has a FPAT mechanism by which we can specify the positive
regex for recognizing fields as tokens. By means of that, we can
save the separating whitespace, turning it into a field:

$ echo 'one two three' | \
awk -v FPAT='[^ ]+|[ ]+' -v OFS= \
'{ $1=$1; print; print NF }'
one two three
5

There you go. We now have 5 fields. The interstitial space is a field.
We set OFS to empty and so $1=$1 doesn't collapse the separation.

> Not really. I'm just remarking on one particular awk feature that I
> find a bit counterintuitive.

The proposed feature of preserving the whitespace separation is a niche
use case in relation to Awk's orientation toward tabular data.

In tabular data that is not formatted into nice columns for a monospaced
font, the whitespace doesn't matter. Awk's behavior is that it will
normalize the separation.

In tabular data that is aligned visually, preserving the whitespace
will not work, if any of your field edits change a field width.

I'm believe that your niche use case has a value though.

That's why, in the TXR Lisp Awk macro, I implemented something which
helps with that use case: the "kfs" variable (keep field separators).
This Boolean variable, if true, causes the separating character
sequences to be retained, and turned into fields.

$ echo 'one two three' | txr -e '(awk (:set kfs t) (t))'
one two three

We can see the list f instead, printed machine readably:

$ echo 'one two three' | txr -e '(awk (:set kfs t) (t (prinl f)))'
("" "one" " " "two" " " "three" "")

There is a leading and trailing empty separator. There is a
very good reason for that, in that the default strategy in Awk
is that, for instance " a b c " produces three fields.
For consistency, if we retain separation, we should always have
five fields where there woudl be three.

My FPAT approach above in the GNU Awk example won't do this correctly;
more work is needed.

This kfs variable is not so recent; I implemented in it 2016.

On 08.03.2024 10:03, Mr. Man-wai Chang wrote:
>
> The original Awk doesn't support regular expressions, right?

Where did you get that from? - Awk without regexps makes little sense;
mind that the basic syntax of Awk programs is described as
/pattern/ { action }
What would remain if there's no regexp patterns; string comparisons?

> Because regex was not yet talked about back then??

Stable Awk (1985) was released 1987. The (initial) old Awk (1977) was
released 1979. Before that tool we had Sed (1974), and before that we
had Ed and Grep (1973). My perception is that regexps were there as a
basic concept of UNIX in all these tools, so why should Awk be exempt.
According to the authors Awk was designed to see how Sed and Grep could
be generalized.

Janis

On 3/8/24 08:46, Janis Papanagnou wrote:
> Awk without regexps makes little sense;

I think this comes down to what is a regular expression and what is not
a regular expression.

> mind that the basic syntax of Awk programs is described as
> pattern { action }

I'm guessing that 40-60% of the awk that I use doesn't use what I would
consider to be regular expressions.

(NF == 5){print $3}
(NF == 8){print $4}

Or:

{total+=$5}
END{print total}

I usually think of regular expressions when I'm doing a sub(/re/, ...)
type thing or a (... ~ /re/) type conditional. More specifically things
between the // in both of those statements are the REs.

Maybe I have an imprecise understanding / definition.

--
Grant. . . .

On 8/3/2024 10:46 pm, Janis Papanagnou wrote:
> On 08.03.2024 10:03, Mr. Man-wai Chang wrote:
>>
>> The original Awk doesn't support regular expressions, right?
>
> Where did you get that from? - Awk without regexps makes little sense;
> mind that the basic syntax of Awk programs is described as
> /pattern/ { action }
> What would remain if there's no regexp patterns; string comparisons?
>
>> Because regex was not yet talked about back then??
>
> Stable Awk (1985) was released 1987. The (initial) old Awk (1977) was
> released 1979. Before that tool we had Sed (1974), and before that we
> had Ed and Grep (1973). My perception is that regexps were there as a
> basic concept of UNIX in all these tools, so why should Awk be exempt.
> According to the authors Awk was designed to see how Sed and Grep could
> be generalized.

That part of history is beyond me. Sorry... my fault for not doing a check.

On 8/3/2024 11:12 pm, Grant Taylor wrote:
>
> I usually think of regular expressions when I'm doing a sub(/re/, ...)
> type thing or a (... ~ /re/) type conditional. More specifically things
> between the // in both of those statements are the REs.
>
> Maybe I have an imprecise understanding / definition.

Do Linux and Unix have a ONE AND ONLY ONE STANDARD regex library?

It seemed that tools and programming languages have their own
implementions, let alone different versions among them.

On 08.03.2024 16:12, Grant Taylor wrote:
> On 3/8/24 08:46, Janis Papanagnou wrote:
>> Awk without regexps makes little sense;
>
> I think this comes down to what is a regular expression and what is not
> a regular expression.
>
>> mind that the basic syntax of Awk programs is described as
>> pattern { action }
>
> I'm guessing that 40-60% of the awk that I use doesn't use what I would
> consider to be regular expressions.
> [...]
> Maybe I have an imprecise understanding / definition.

Your definition matches the common naming, where I deliberately
deviate from. (I think that "pattern" is an inferior naming and
"condition" should better be used, where a 'condition' can also
be a regexp that I regularly write as '/regexp/' or '/pattern/'
in explanations.) So I agree that it's likely that this alone
doesn't serve well as explanation for the existence of regexps
in Awk. The rationale is better seen in the statement "Awk was
designed to see how Sed and Grep could be generalized." that I
quoted (not literally, but from the original Awk book).

Janis

On 08.03.2024 16:56, Mr. Man-wai Chang wrote:
> On 8/3/2024 10:46 pm, Janis Papanagnou wrote:
>>
>> Stable Awk (1985) was released 1987. The (initial) old Awk (1977) was
>> released 1979. Before that tool we had Sed (1974), and before that we
>> had Ed and Grep (1973). My perception is that regexps were there as a
>> basic concept of UNIX in all these tools, so why should Awk be exempt.
>> According to the authors Awk was designed to see how Sed and Grep could
>> be generalized.
>
> That part of history is beyond me. Sorry... my fault for not doing a check.

The mistake may stem from a myth (I heard it before already); it may
have been misinterpreted where it's said that in the first Awk there
was no match function (which is true, but it means the concrete match()
function not the abstract function of a (regexp) pattern match).

Janis

Mr. Man-wai Chang wrote:

> Do Linux and Unix have a ONE AND ONLY ONE STANDARD regex library?
>
> It seemed that tools and programming languages have their own
> implementions, let alone different versions among them.

In the POSIX/UNIX standard the functions used for handling regular
expressions are regcomp() and regexec() (and regerror() and regfree()).
They are in the C library, not a separate "regex library".

They support different RE flavours via flags. The standard requires
that "basic regular expressions" (default) and "extended regular
expressions" (with REG_EXTENDED flag) are supported. Implementations
can support other flavours with non-standard flags.

POSIX requires that awk uses extended regular expressions (i.e. the
same as regcomp() with REG_EXTENDED).

--
Geoff Clare <netnews@gclare.org.uk>

In article <tv26ck-3qt.ln1@ID-313840.user.individual.net>,
Geoff Clare <netnews@gclare.org.uk> wrote:
>POSIX requires that awk uses extended regular expressions (i.e. the
>same as regcomp() with REG_EXTENDED).

There is the additional requirement that \ inside [....] can
be used to escape characters, so that [abc\]def] is valid in
awk but not in other uses of REG_EXTENDED.

Aharon Robbins wrote:

> In article <tv26ck-3qt.ln1@ID-313840.user.individual.net>,
> Geoff Clare <netnews@gclare.org.uk> wrote:
>>POSIX requires that awk uses extended regular expressions (i.e. the
>>same as regcomp() with REG_EXTENDED).
>
> There is the additional requirement that \ inside [....] can
> be used to escape characters,

Yes, awk effectively has an extra "layer" of backslash escaping
before the ERE rules kick in, both inside and outside [....].
I didn't mention this so as not to overload the OP with
information - he seemed more interested in the different flavours
of RE than in nitty gritty details like that.

--
Geoff Clare <netnews@gclare.org.uk>