On 01/02/2024 02:29, bart wrote:
> On 01/02/2024 00:47, Scott Lurndal wrote:
>> bart <bc@freeuk.com> writes:
>>> On 31/01/2024 23:36, Ben Bacarisse wrote:
>>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>
>>>> According to you, these tools are poorly designed.  I don't think so.
>>>> How would you design them?  Endless input and output file names to be
>>>> juggled and tidied up afterwards?
>>>
>>> I think they're poorly designed too.
>>
>> Of course you do.   They're not bart programs.
>>
>>>
>>>  From the POV of interactive console programs, they /are/ poor.
>>
>> You don't provide any reason why - do elucidate!
>
> They only do one thing, like you can't first do A, then B. They don't
> give any prompts. They often apparently do nothing (so you can't tell if
> they're busy, waiting for input, or hanging). There is no dialog.

That's the whole point!

If you want to do A, then B, then you do "A | B", or "A; B", or "A && B"
or "A || B". And if you want to do A, then B twice, then C, then A
again, you write "A | B | B | C | A". Other operator choices let you
say "do this then that", or "do this, and if successful do that", etc.

Your monolithic AB program fails when you want to do C, or want to do A
and B in a way the AB author didn't envisage.

You have a Transformer - a toy that can be either a car or a robot.
I've got a box of Lego. Sometimes I need instructions and a bit of
time, but I can have a car, a robot, a plane, an alien, a house, and
anything else I might want.

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>On 31/01/2024 23:34, Keith Thompson wrote:
>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>> On 31/01/2024 20:14, Keith Thompson wrote:
>> [...]
>>>> Terminal control sequences (almost always based on VT100 these days)
>>>> are typically not printable, but tend to avoid null characters, which
>>>> means you can very probably use printf to print them (assuming you're
>>>> on a POSIX-like system).
>>>> [...]
>>>>
>>> In ASCII, 0 means NUL, or "ignore". So an ASCII sequence may contain
>>> any number of embedded zero bytes, which the receiver ignores. That's
>>> because for technical reasons some communications channels have to
>>> send data every cycle, and if there is no data, they will send a
>>> signal indistinguishable from all bits zero.
>>
>> Not particularly relevant. A quick experiment with xterm indicates that
>> embedding null bytes in a control sequence prevents it from being
>> recognized. There may be some standards that require embedded zero
>> bytes to be ignored, but xterm doesn't any such standard. Similarly, if
>> you embed null bytes in text written to a file, the result is corrupted
>> text file.
>>
>The standard is ASCII. (American standard for computer information
>interchange). Byte zero is NUL, which means "ignore".

ASCII does not define a semantic for
the NUL byte other than grouping it as a "control character" with
the function NO-OP. It was often used as a padding character.

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
>On 01.02.2024 06:24, Malcolm McLean wrote:
>> On 31/01/2024 23:36, Ben Bacarisse wrote:
>>>
>>> Maybe you are not used to a system where it's trivial to inspect such
>>> data. When "some_prog" produces data that are not compatible with the
>>> current terminal settings, "some_prog | hd" shows a hex dump instead.
>>> The need to do this does not make "some_prog" poorly designed. It may
>>> simply mean that the output is /intended/ for further processing.
>>>
>> Well almost by definition binary output is intended for further
>> processing. Binary audio files must ultimately be converted to analogue
>> if anyone is to listen to them, for example.
>
>Well, not necessarily. Let's leave the typical use case for a moment...
>
>It might also be analyzed and converted to a digitally represented
>formula, say some TeX code, or e.g. like the formal syntax that the
>lilypond program uses.
>
>> I had to check how to do a hex dump on the system I'm typing this on.
>> The name of the hex dumper is xxd instead of hd, but otherwise it works
>> the same way and will accept piped data. But the fact I had to look it
>> up tells you that I've never actually used it.
>
>Well, there's always the old Unix standard tool, 'od'.
>
>I use that without thinking or looking it up, since it was ever there,
>despite I only rarely use it.
>
>And you observed correctly that nowadays there's typically even more
>than one tool available. (And Bart will probably write his own tool. :-)
>
>> The two problems with hex
>> dumps are that you've got to do mental arithmetic to convert 8 bit hex
>> values into 16 or 32 bit fields,
>
>Hmm.. - have you inspected the man pages of the tools?
>
>At least for 'od' I know it's easy per option...
> od -c file # characters (or escapes and octals)
> od -t x1 file # hex octets
> od -t x2 file # words (two octets)
> od -c -t x1 file # characters and octets

Likewise, with xxd, use the -g flag. But Malcolm can't be
troubled to read the man page before complaining.

On 01/02/2024 02:53, Malcolm McLean wrote:
> On 31/01/2024 23:36, Ben Bacarisse wrote:
>>
>> An example where it's really useful not to care: I have a suite of tools
>> for doing toy cryptanalysis.  Some apply various transformations and/or
>> filters to byte streams and others collect and output (on stderr)
>> various statistics.  Plugging them together in various pipelines is very
>> handy when investigating an encrypted text.  The output is almost always
>> "binary" in the sense that there would be not point in looking at on a
>> terminal.
>>
>> According to you, these tools are poorly designed.  I don't think so.
>> How would you design them?  Endless input and output file names to be
>> juggled and tidied up afterwards?
>>
> I'd write a monolithic program.

It's very strange to me to see people that consider themselves
programmers talk about having multiple small functions to do specific
tasks and combining them into bigger functions to solve bigger problems,
yet are reduced to quivering jellies at the thought of multiple small
programs to do specific tasks that can be combined to solve bigger tasks.

Do you think the C standard library would be improved by a single
function "flubadub" that takes 20 parameters and can calculate
logarithms, print formatted text, allocate memory and write it all to a
file?

On 01/02/2024 14:45, Scott Lurndal wrote:
> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>> On 31/01/2024 23:36, Ben Bacarisse wrote:
>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>
>>>> On 30/01/2024 07:27, Tim Rentsch wrote:
>>>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>>>
>>>>>> On 29/01/2024 20:10, Tim Rentsch wrote:
>>>>>>
>>>>>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>>>>>
>>>>>>> [...]
>>>>>>>
>>>>>>>> I've never used standard output for binary data.
>>>>>>>> [...] it strikes me as a poor design decision.
>>>>>>>
>>>>>>> How so?
>>>>>>
>>>>>> Because the output can't be inspected by humans, and because it might
>>>>>> have unusual effects if passed though systems designed to handle
>>>>>> human-readable text.
>>>
>>> Maybe you are not used to a system where it's trivial to inspect such
>>> data. When "some_prog" produces data that are not compatible with the
>>> current terminal settings, "some_prog | hd" shows a hex dump instead.
>>> The need to do this does not make "some_prog" poorly designed. It may
>>> simply mean that the output is /intended/ for further processing.
>>>
>>>> For instance in some systems designed to receive
>>>>>> ASCII text, there is no distinction between the nul byte and "waiting
>>>>>> for next data byte". Obviously this will cause difficuties if the data
>>>>>> is binary.
>>>>>> Also many binary formats can't easily be extended, so you can pass one
>>>>>> image and that's all. While it is possible to devise a text format
>>>>>> which is similar, in practice text formats usually have enough
>>>>>> redundancy to be easily extended.
>>>>>>
>>>>>> So it's harder to correct errors, more prone to errors, and harder to
>>>>>> extend.
>>>>> Your reasoning is all gobbledygook. Your comments reflect only
>>>>> limitations in your thinking, not any essential truth about using
>>>>> standard out for binary data.
>>>>>
>>>> I must admit that it's nothing I have ever done or considered doing.
>>>>
>>>> However standard output is designed for text and not binary ouput.
>>>
>>> What is your evidence? stdout was just designed for output (as far as I
>>> can tell) and, anyway, what is the distinction you are making between
>>> binary and text? iconv --from ACSII --to EBCDIC-UK will produce
>>> something that is "logically" text on stdout, but it might look like
>>> binary to you.
>>>
>>> An example where it's really useful not to care: I have a suite of tools
>>> for doing toy cryptanalysis. Some apply various transformations and/or
>>> filters to byte streams and others collect and output (on stderr)
>>> various statistics. Plugging them together in various pipelines is very
>>> handy when investigating an encrypted text. The output is almost always
>>> "binary" in the sense that there would be not point in looking at on a
>>> terminal.
>>>
>>> According to you, these tools are poorly designed. I don't think so.
>>> How would you design them? Endless input and output file names to be
>>> juggled and tidied up afterwards?
>>>
>> I'd write a monolithic program.
>
> Even a monolithic program is decomposed into subroutines (or malcolm functions).
>
> A pipeline is the same concept at a higher level.
>
Exactly. So whilst it might have some advantages, they aren't going to
be very large, because as you say, it;s the same basic concept.
>> Load the encryoted text into memory, and then pass it to subroutines to
>> do the various analyses.
>
> So your program is arbitrarily large and needs to be recompiled
> to add new subroutines. Advantage to the pipeline, again.
>
The computer I'm typing this on isn't especially expensive and is just
my personal machine. But it has six cores each of which execute 3.7
billion instructions per second. So programs have to be very large
indeed before compile time is a significant problem.
--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
>On 31.01.2024 20:28, Scott Lurndal wrote:
>> Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
>>> On 31.01.2024 18:20, Scott Lurndal wrote:
>>
>> <snip commentary about POSIX adopting most of ksh88>
>>
>>> On Linux you got only a ksh clone, first basically based on ksh88
>>> (but I anyway never used it but downloaded the original AT&T ksh93).
>>> Meanwhile, I think, you should get a ksh93 on Linux per default.
>>> But I suggest to use the ksh93u+m (the version maintained by Martijn
>>> Dekker), instead; it has a lot errors fixed.
>>
>> The AT&T ksh93 has been distributed for a long time. My 2012
>> install of Fedora Core 20, OSX, and my recent Ubuntu install use:
>>
>> $ Version AJM 93u+ 2012-08-01
>
>This is what I regularly see. - Also on my system. But I replaced it
>with Version AJM 93u+m/1.0.8 2024-01-01.
>
>(I suppose you use bash, but if you're using ksh I suggest to switch
>to "ksh93u+m".)

No, I don't use bash. I've used ksh since 1989.

I'm perfectly happy with ksh93u.

On 01/02/2024 13:24, Tim Rentsch wrote:
> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>
>> You admit this with "not tested". Says it all. '"Understandig Unix" is
>> an intellectually useless achievement. You might have to do it if you
>> have to use the system and debug and trouble shoot. But it's nothing
>> to be proud about.
>
> You're an idiot. As usual trying to have a useful discussion
> with you has turned out to be a complete waste of time.
>
Some things are interesting in themselves and worth talking about at
lenght. Like how Haskell builds up functions of functions. Other things
really aren't. And how to set up a Unix pipeline is one of those that
really aren't (unless actually faced with a such a system and with a
practical need to do it).

I think you have the intelligence to understand this, if you'd just
understand where I am coming from. This arrogant and dismissive attitude
does not become you.
--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

On 01.02.2024 14:26, Malcolm McLean wrote:
> On 01/02/2024 13:02, Janis Papanagnou wrote:
>> Well, not necessarily. Let's leave the typical use case for a moment...
>>
>> It might also be analyzed and converted to a digitally represented
>> formula, say some TeX code, or e.g. like the formal syntax that the
>> lilypond program uses.
>>
> And ultimately converted to a non binary form. A list of 1s and 0s is
> seldom any use to the final consumer of the data.

No, I was speaking about an application that creates lilypond _input_,
which is a formal language to write notes, e.g. for evaluation by the
lilypond software, but not excluding other usages.

>
>>> The two problems with hex
>>> dumps are that you've got to do mental arithmetic to convert 8 bit hex
>>> values into 16 or 32 bit fields,
>>
>> Hmm.. - have you inspected the man pages of the tools?
>>
> I just ran "man xxd". The man page contains this statement.
>
> The tool's weirdness matches its creator's brain. Use entirely at your
> own risk. Copy files. Trace it. Become a wizard.

This statement repelled you? (Can't help you here.)

>> At least for 'od' I know it's easy per option...
>> od -c file # characters (or escapes and octals)
>> od -t x1 file # hex octets
>> od -t x2 file # words (two octets)
>> od -c -t x1 file # characters and octets
>>
> So a JPEG file starts with
> FF D8
> FF E0
> hi lo (length of the FF E0 segment)
>
> So we want the output
>
> FF D8 FF E0 [1000] to check that the segment markers are correct and FF
> E0 segment is genuinely a thousand bytes (or whatever it is). This isn't
> easy to achieve with a hex dump utility.

I don't know binary format details about jpg, so I cannot help you here.

I was responding to your question where you wanted entities larger than
a single octet. I showed you some examples what I can do with 'od'.
(Just open it's man page to find all sorts of possible options and
option combinations.)

Yes, you can use entities of length four. Guess how? By 'od -t x4 file'
Or if you need decimal numbers use 'od -t d4 file' or 'od -t d2 file'.

And I already answered that for specific binary structures you'll need
something data specific. You can also generalize that to some degree...

For example I recently wrote a shell script that supports binary data
definitions in a very primitive declarative form; it's allowing me to
specify the field lengths, output type, and identification for readable
output. It allows hex, bin, dec, text, hex-seq, data to skip. Length of
fields many be constants, 0-terminated, or defined by another preceding
field. It also handles endianness.

You see even some primitive "generic" thing needs a couple features.

Janis

On 01.02.2024 16:04, Scott Lurndal wrote:
> Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
>> On 31.01.2024 20:28, Scott Lurndal wrote:
>>>
>>> $ Version AJM 93u+ 2012-08-01
>>
>> This is what I regularly see. - Also on my system. But I replaced it
>> with Version AJM 93u+m/1.0.8 2024-01-01.
>>
>> (I suppose you use bash, but if you're using ksh I suggest to switch
>> to "ksh93u+m".)
>
> No, I don't use bash. I've used ksh since 1989.
>
> I'm perfectly happy with ksh93u.

Oh, that's okay then. (I just wonder that you haven't yet stumbled
over any of the countless and partly very annoying bugs. But okay,
if it works reliably for you...)

Janis

On 01/02/2024 14:35, David Brown wrote:
> On 01/02/2024 01:21, bart wrote:

>> * Have versions of these tools for use as filters with no UI, just
>>    a default input and output, and versions for interactive use with
>>    helpful prompts. Or even just a sensibly named output! Instead of
>>    every program writing a.out.
>>
>
> Sometimes a "front end" with a nice UI /is/ useful.  So people write
> front ends with nice UI's - text-based or gui.  Typically these are
> great for common tasks, and are cumbersome or useless for rarer or more
> advanced stuff.  And that's fine - use whatever works best for you at
> the time.  If you think "ssh" is complicated from the command line, use
> "putty" - but that won't handle all the uses that some people need.
>
>
> But I most certainly don't want interactive and "helpful" prompts for
> most of my command-line tools.  It's fine on occasion if it is
> necessary, useful if something out of the ordinary happens, and
> appropriate for things like passwords.  But when I know what I am doing,
> why would I want to see help messages repeated?  And if I don't know
> what I am doing - it's the first time using a command, or I've forgotten
> some details, then I have "prog --help", "man prog", or "google prog"
> that will all give much more useful information than interactive prompts
> ever could.
>
> Can you imagine typing "ls" and being asked :
>
> * Did you want to list files in the current directory, or elsewhere?
> * Did you want to list all files, including hidden files?
> * Did you want to use colour?
>
> and then twenty more questions for the other common options for ls?
>
>
> What would be the benefits of "sort" using command line options for
> "filter" or "script" usage and then asking a dozen questions for
> "interactive" use?

I'd classify command-line programs into categories:

* Filters, as I explained before, which are designed to work with piped data

* Programs that can meaningfully be run with no command line options
(such as your 'ls' example), because a default action is defined

* Programs that expect command line parameters, which can generate an
error message or usage info if missing

* Programs that launch into an on-going session, which may or may not
take command line parameters (eg. python interpreter)

* Everything else, programs which are launched from the command line but
do not otherwise have a CLI, eg. a GUI text editor.

The troublesome ones are those I've called filters. Some programs I
expect to behaviour conventionally, unexpectedly behave as filters, such
as 'as'.

>
>> * Have a concept of a current block of data, analogous to a clipboard.
>>    Then separate commands can load data, sort it, count it, display it,
>>    write it etc, with no need for intermediate named files.
>
> You mean, a convenient way of moving data between programs?  Sort of
....
>
>>
>> But I'd be happier if this was all contained within a separate
>> application from an OS shell program.
>
> Yes, because it is /so/ much better if it is limited to a few commands
> that you think of when writing this special application, than having a
> general system that works with any commands.
>
> Basically, all you are saying is that you'd like command line utilities
> to work with a default file name "/tmp/clipboard" - something you didn't
> want earlier on.

No. Someone said that it is convenient to run a sequence of programs,
where each processes the output of the other, without having to
explicitly named intermediate files.

I'm exploring other ways of doing that ...

> Let's use /tmp/x for convenience.
>
> /tmp$ cat > fred
> one
> two
> three
> four
> <ctrl-D>
>
> >      > load fred
> >      Data loaded
>
> $ cp fred x
....
> $ cat -n x
>      1    FOUR
>      2    ONE
>      3    THREE
>      4    TWO
>
> >
> >      > save bill
> >      Written to bill
> >
> >      > q
> >
>
> $ cp x bill
>
>

> With pipes, this is all vastly simpler:
>
> $ cat fred | tac | sort | awk '{print toupper($0)}' > bill

.... one way, which follows on from my previous suggestion to have two
versions of filter utilities, might be to have versions which work on
that blob of current data as I demonstrated.

Then on the shell command line you'd have:

> cload fred
> ctac
> csort
> cawk ...
> csave bill

Any inputs will always come from that blob; it will not just do nothing
waiting for input, unless the command is specifically for that.

And if so, it can prompt for it. For that matter, it can also write
messages confirming what it's just done, or show a progress report.

If you like, you can also have a stack of such data blobs, so:

> cload fred
> cdupl
> csave bill
> crev
> csave llib

(I think I've just invented shell-Forth!)

I've just realised why it is that your filter programs don't show
prompts or any kinds of messages: because those are sent to stdout, and
therefore will screw up any data that is being sent there as the primary
output.

THAT'S why sending what ought to be packaged data to stdout is Wrong.

> The "sponge" utility reads all of its stdin, then writes the file.
> Otherwise, since Unix is inherently multi-tasking and runs the programs
> in parallel (unlike your utility), trying to redirect output back into
> the same file you use for output is a race condition. Utilities are
> generally designed for pipes, not destructive changes to a single file.

My utility is like pretty much any application that loads a file, does
something with it, and writes it out. (Eg. text or image editors.)

So that kind of pattern is well understand. The difference is that your
filters tend to bluntly work on the whole blob of data.

The kind of REPL utility I outlined is far more user-friendly. The data
it deals with isn't as transient as the data in a pipe either.

Halfway through my session, the data is still there. You can examine it
so far, and decide where to go next, or perhaps reverse the last stup.

When you write a|b|c>d, it whizzes through it all at once. I can write
more on this but I simply don't think that any 'Unix-head' here is going
to get it; it's too ingrained.

On 01/02/2024 14:55, David Brown wrote:
> On 01/02/2024 02:53, Malcolm McLean wrote:
>> On 31/01/2024 23:36, Ben Bacarisse wrote:
>>>
>>> An example where it's really useful not to care: I have a suite of tools
>>> for doing toy cryptanalysis.  Some apply various transformations and/or
>>> filters to byte streams and others collect and output (on stderr)
>>> various statistics.  Plugging them together in various pipelines is very
>>> handy when investigating an encrypted text.  The output is almost always
>>> "binary" in the sense that there would be not point in looking at on a
>>> terminal.
>>>
>>> According to you, these tools are poorly designed.  I don't think so.
>>> How would you design them?  Endless input and output file names to be
>>> juggled and tidied up afterwards?
>>>
>> I'd write a monolithic program.
>
> It's very strange to me to see people that consider themselves
> programmers talk about having multiple small functions to do specific
> tasks and combining them into bigger functions to solve bigger problems,
> yet are reduced to quivering jellies at the thought of multiple small
> programs to do specific tasks that can be combined to solve bigger tasks.
>
> Do you think the C standard library would be improved by a single
> function "flubadub" that takes 20 parameters and can calculate
> logarithms, print formatted text, allocate memory and write it all to a
> file?
>
By breaking down the problem into several parts e.g. "collect
statistical data, analyse statistics, form hypothesis, attempt
decryption, check decrypt for plausible plaintext" we can usually attack
it better. And you're right, there's not a fundamental difference
between writing one program with five subroutines, or five programs
which pass data to each other via pipelines.
But that doesn't mean that decision must not be made, or that you can't
give reasons for and against each option.

So could you list one or two reasons why you might prefer a program with
five subroutines, and one or two reasons why you might prefer to write
five programs which communicate via piped data?
--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

On 01.02.2024 15:57, Malcolm McLean wrote:
> On 01/02/2024 14:45, Scott Lurndal wrote:
>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>> I'd write a monolithic program.

I certainly want re-usability of functions, modules, components,
commands, and systems. (And I want no duplication on any level.)

>> Even a monolithic program is decomposed into subroutines (or malcolm
>> functions).

There are various ways to organize software. Some supported by the
language, some by the OS mechanisms, some specifically implemented.

>> A pipeline is the same concept at a higher level.

Functions is a way, communicating processes is a way, etc. etc. etc.
All can be taken to combine various entities. All mechanisms can be
used together or some omitted.

> Exactly. So whilst it might have some advantages, they aren't going to
> be very large, because as you say, it;s the same basic concept.

I think that you draw the wrong conclusion (on a statement that is
prone to misunderstandings or even wrong).

Pipelines are a very useful method to let processes communicate in
a one-way direction (as the name already suggests). From that it's
immediately recognizable that filters are a natural element in that
OS-architectural glue.

One original Unix philosophy was to have specialized commands that
do one thing well, and to combine such tasks as necessary. (To some
degree there was as similar statement concerning C function design.)
Unfortunately some popular GNU tools deviate from that. Features get
incorporated (as duplicates) in many tools (instead of using the
existing specialized one).

Janis

On 01/02/2024 14:26, Malcolm McLean wrote:
> I just ran "man xxd". The man page contains this statement.
>
> The tool's weirdness matches its creator's brain.  Use entirely at your
>        own risk. Copy files. Trace it. Become a wizard.

If you don't like xxd, use one of the dozens of other hex dump programs
that are easily available. I use hexdump myself.

On 01/02/2024 15:07, Janis Papanagnou wrote:
> On 01.02.2024 14:26, Malcolm McLean wrote:
>> On 01/02/2024 13:02, Janis Papanagnou wrote:
>>> Well, not necessarily. Let's leave the typical use case for a moment...
>>>
>>> It might also be analyzed and converted to a digitally represented
>>> formula, say some TeX code, or e.g. like the formal syntax that the
>>> lilypond program uses.
>>>
>> And ultimately converted to a non binary form. A list of 1s and 0s is
>> seldom any use to the final consumer of the data.
>
> No, I was speaking about an application that creates lilypond _input_,
> which is a formal language to write notes, e.g. for evaluation by the
> lilypond software, but not excluding other usages.
>
>>
>>>> The two problems with hex
>>>> dumps are that you've got to do mental arithmetic to convert 8 bit hex
>>>> values into 16 or 32 bit fields,
>>>
>>> Hmm.. - have you inspected the man pages of the tools?
>>>
>> I just ran "man xxd". The man page contains this statement.
>>
>> The tool's weirdness matches its creator's brain. Use entirely at your
>> own risk. Copy files. Trace it. Become a wizard.
>
> This statement repelled you? (Can't help you here.)
>
>>> At least for 'od' I know it's easy per option...
>>> od -c file # characters (or escapes and octals)
>>> od -t x1 file # hex octets
>>> od -t x2 file # words (two octets)
>>> od -c -t x1 file # characters and octets
>>>
>> So a JPEG file starts with
>> FF D8
>> FF E0
>> hi lo (length of the FF E0 segment)
>>
>> So we want the output
>>
>> FF D8 FF E0 [1000] to check that the segment markers are correct and FF
>> E0 segment is genuinely a thousand bytes (or whatever it is). This isn't
>> easy to achieve with a hex dump utility.
>
> I don't know binary format details about jpg, so I cannot help you here.
>
JPEG is an extremely common binary file format and JPEG files will be
found on most general purpose computers.
All you need to know for the purposes of the discussion is that the
first four bytes are segment identifiers and must have the values I
gave, whilst bytes five and six are a big endian 16 bit number that
represents a segment length, and that potentially any of those values
could be unexpected and you might want to inspect them.

So how would you achieve that in a convenient and non-error prone way?

--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

On 01/02/2024 14:52, Scott Lurndal wrote:
> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>> On 31/01/2024 23:34, Keith Thompson wrote:
>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>> On 31/01/2024 20:14, Keith Thompson wrote:
>>> [...]
>>>>> Terminal control sequences (almost always based on VT100 these days)
>>>>> are typically not printable, but tend to avoid null characters, which
>>>>> means you can very probably use printf to print them (assuming you're
>>>>> on a POSIX-like system).
>>>>> [...]
>>>>>
>>>> In ASCII, 0 means NUL, or "ignore". So an ASCII sequence may contain
>>>> any number of embedded zero bytes, which the receiver ignores. That's
>>>> because for technical reasons some communications channels have to
>>>> send data every cycle, and if there is no data, they will send a
>>>> signal indistinguishable from all bits zero.
>>>
>>> Not particularly relevant. A quick experiment with xterm indicates that
>>> embedding null bytes in a control sequence prevents it from being
>>> recognized. There may be some standards that require embedded zero
>>> bytes to be ignored, but xterm doesn't any such standard. Similarly, if
>>> you embed null bytes in text written to a file, the result is corrupted
>>> text file.
>>>
>> The standard is ASCII. (American standard for computer information
>> interchange). Byte zero is NUL, which means "ignore".
>
> ASCII does not define a semantic for
> the NUL byte other than grouping it as a "control character" with
> the function NO-OP. It was often used as a padding character.
>
Yes. No op. So when you receive it, you do not perform an operation.
You're just saying the same thing in different words.

And yes, the reason is that with some systems you have to send data
every cycle regardless of whether you have it or not, and with others
you can have device not ready / communications not working conditions
which can be hard to distinguish from sending all zeroes. Another reason
might that you need to pad it a transmission to a certain length because
a protocol passes fixed length blocks. So NUL is often important.

--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

On 01.02.2024 16:41, Malcolm McLean wrote:
>
> So could you list one or two reasons why you might prefer a program with
> five subroutines, and one or two reasons why you might prefer to write
> five programs which communicate via piped data?

A quite appealing and naturally appearing task (from the past) to use
pipes was to model communication cascades. Something like (off the top
of my head)...

data-source | sign | compress | crc | encrypt | channel-enc |
interleaver | channel-simulator | deinterleaver | channel-dec |
decrypt | crc-check | uncompress | check-sign | data-sink

Component-pairs can be omitted, say you may leave out the un-/compress
function. And every component may be either special purpose or general.
A special purpose entity could be BCH-enc and RCPC-enc, or it can also
be (if better suited) a combined module, say 'crc -16' vs. 'crc -32'
with the function realized as option argument.

Reasons to not use pipelines are when you don't have a linear flow.
In some circumstances you can bypass the pipe (opening a side-channel)
on other cases you can't or it's overly messy to do so.

Reasons not to use in-memory processing are of course if you have huge
amounts of data. Then you need filtering and pipeline processing.
(A former fellow student who worked for the ESO told me remarkable
things about the amounts of data they continuously receive and that
must on the fly be processed.) Another more recent example can be
processing of real time data for Digital Twins (e.g. city models).

Janis

On 01/02/2024 15:48, David Brown wrote:
> On 01/02/2024 14:26, Malcolm McLean wrote:
>> I just ran "man xxd". The man page contains this statement.
>>
>> The tool's weirdness matches its creator's brain.  Use entirely at your
>>         own risk. Copy files. Trace it. Become a wizard.
>
> If you don't like xxd, use one of the dozens of other hex dump programs
> that are easily available.  I use hexdump myself.
>

I just thought that man page section was funny enough to be worth
sharing. I don't want to put people off using xxd.
--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
>On 01.02.2024 15:57, Malcolm McLean wrote:
>> On 01/02/2024 14:45, Scott Lurndal wrote:

>> Exactly. So whilst it might have some advantages, they aren't going to
>> be very large, because as you say, it;s the same basic concept.
>
>I think that you draw the wrong conclusion (on a statement that is
>prone to misunderstandings or even wrong).
>
>Pipelines are a very useful method to let processes communicate in
>a one-way direction (as the name already suggests). From that it's
>immediately recognizable that filters are a natural element in that
>OS-architectural glue.
>
>One original Unix philosophy was to have specialized commands that
>do one thing well, and to combine such tasks as necessary. (To some
>degree there was as similar statement concerning C function design.)
>Unfortunately some popular GNU tools deviate from that. Features get
>incorporated (as duplicates) in many tools (instead of using the
>existing specialized one).

I believe the classic example is the documenters workbench.

Troff converts a set of page layout directives into a typeset
document. Originally for the CAT typesetter, but I'll address
that later.

So, now you want to add tables. You can modify troff to add
new macros that use existing markup directives, or you can
add a filter that converts a 'table description' language
into troff markup directives.

$ < document.tr | tbl | troff -mm > typesetter_output

Now, you want to add support for arbitrary mathematical
formulae. You can modify troff to add more macros, or
you can write a filter that converts an 'equation description'
to troff and add that to your pipeline;

$ <document.tr | eqn | tbl | troff -mm > typesetter_output

Now, you realize that you want to support multiple typesetters,
so you can either modify troff to support all possible typesetters,
or you can add post processing filters.

$ <document.tr | eqn | tbl | ditroff -mm | dit2cat > typesetter_output
$ <document.tr | eqn | tbl | ditroff -mm | dit2ps > postscript output

Then you might want to be able to include pictures in your
document.

$ <document.tr | pic | eqn | tbl | ditroff -mm | dit2ps | ps2pdf > document.pdf.

Or you might want ascii text output

$ <document.tr | pic | eqn | tbl | nroff -mm > document.txt

On 01.02.2024 11:34, David Brown wrote:
> On 31/01/2024 19:35, Keith Thompson wrote:
>> David Brown <david.brown@hesbynett.no> writes:
>>>
>>> I regularly see it as more symmetrical and clearer to push data left
>>> to right. So I might write "cat infile | grep foo | sort > outfile".
>>> Of course I could use "<" redirection, but somehow it seems more
>>> natural to me to have this flow. I'll use "<" for simpler cases.
>>>
>>> But perhaps this is just my habit, and makes little sense to other
>>> people.

I completely understand that.

>>
>> You can also use:
>>
>> < infile grep foo | sort > outfile
>>
>> Redirections don't have to be written after a command.

Indeed. And if we also respect that 'grep' accepts arguments,
then it's even more compact and yet probably better legible... :-)

grep foo infile | sort > outfile

> I did not know you could write it that way - thanks for another
> off-topic, but useful, tip.

Yes. We certainly should instead have written

grep foo iso646.h | sort > outfile

Janis

On 2024-02-01, Scott Lurndal <scott@slp53.sl.home> wrote:
> Kaz Kylheku <433-929-6894@kylheku.com> writes:
>>On 2024-01-31, Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:
>>> Kaz Kylheku <433-929-6894@kylheku.com> writes:
>>But ESC is related text; it's a character described in ASCII used for
>>signaling in the middle of text, which is what it's doing here.
>
> So are most of the other ASCII codes less than 0x20. Including
> file and record delimiters and shift-in/shift-out.

ESC is such an important piece of text, that it has a dedicated key on
your keyboard.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>On 01/02/2024 15:07, Janis Papanagnou wrote:
>> On 01.02.2024 14:26, Malcolm McLean wrote:
>>> On 01/02/2024 13:02, Janis Papanagnou wrote:
>>>> Well, not necessarily. Let's leave the typical use case for a moment...
>>>>
>>>> It might also be analyzed and converted to a digitally represented
>>>> formula, say some TeX code, or e.g. like the formal syntax that the
>>>> lilypond program uses.
>>>>
>>> And ultimately converted to a non binary form. A list of 1s and 0s is
>>> seldom any use to the final consumer of the data.
>>
>> No, I was speaking about an application that creates lilypond _input_,
>> which is a formal language to write notes, e.g. for evaluation by the
>> lilypond software, but not excluding other usages.
>>
>>>
>>>>> The two problems with hex
>>>>> dumps are that you've got to do mental arithmetic to convert 8 bit hex
>>>>> values into 16 or 32 bit fields,
>>>>
>>>> Hmm.. - have you inspected the man pages of the tools?
>>>>
>>> I just ran "man xxd". The man page contains this statement.
>>>
>>> The tool's weirdness matches its creator's brain. Use entirely at your
>>> own risk. Copy files. Trace it. Become a wizard.
>>
>> This statement repelled you? (Can't help you here.)
>>
>>>> At least for 'od' I know it's easy per option...
>>>> od -c file # characters (or escapes and octals)
>>>> od -t x1 file # hex octets
>>>> od -t x2 file # words (two octets)
>>>> od -c -t x1 file # characters and octets
>>>>
>>> So a JPEG file starts with
>>> FF D8
>>> FF E0
>>> hi lo (length of the FF E0 segment)
>>>
>>> So we want the output
>>>
>>> FF D8 FF E0 [1000] to check that the segment markers are correct and FF
>>> E0 segment is genuinely a thousand bytes (or whatever it is). This isn't
>>> easy to achieve with a hex dump utility.
>>
>> I don't know binary format details about jpg, so I cannot help you here.
>>
>JPEG is an extremely common binary file format and JPEG files will be
>found on most general purpose computers.
>All you need to know for the purposes of the discussion is that the
>first four bytes are segment identifiers and must have the values I
>gave, whilst bytes five and six are a big endian 16 bit number that
>represents a segment length, and that potentially any of those values
>could be unexpected and you might want to inspect them.
>
>So how would you achieve that in a convenient and non-error prone way?

$ if file /tmp/garage.jpg | grep JPEG > /dev/null^Jthen^Jecho "it is a jpeg"^Jfi
it is a jpeg

On 01/02/2024 14:50, David Brown wrote:
> On 01/02/2024 02:29, bart wrote:
>> On 01/02/2024 00:47, Scott Lurndal wrote:
>>> bart <bc@freeuk.com> writes:
>>>> On 31/01/2024 23:36, Ben Bacarisse wrote:
>>>>> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>>>
>>>>> According to you, these tools are poorly designed.  I don't think so.
>>>>> How would you design them?  Endless input and output file names to be
>>>>> juggled and tidied up afterwards?
>>>>
>>>> I think they're poorly designed too.
>>>
>>> Of course you do.   They're not bart programs.
>>>
>>>>
>>>>  From the POV of interactive console programs, they /are/ poor.
>>>
>>> You don't provide any reason why - do elucidate!
>>
>> They only do one thing, like you can't first do A, then B. They don't
>> give any prompts. They often apparently do nothing (so you can't tell
>> if they're busy, waiting for input, or hanging). There is no dialog.
>
> That's the whole point!
>
> If you want to do A, then B, then you do "A | B", or "A; B", or "A && B"
> or "A || B".  And if you want to do A, then B twice, then C, then A
> again, you write "A | B | B | C | A".  Other operator choices let you
> say "do this then that", or "do this, and if successful do that", etc.
>
> Your monolithic AB program fails when you want to do C, or want to do A
> and B in a way the AB author didn't envisage.
>
> You have a Transformer - a toy that can be either a car or a robot. I've
> got a box of Lego.  Sometimes I need instructions and a bit of time, but
> I can have a car, a robot, a plane, an alien, a house, and anything else
> I might want.

You can only do one thing, as you can only have one unbroken byte
sequence as output sent to stdout.

You can't send output A to stdout, then B to stdout, and certainly can't
interleave messages to the console on stdout, as that would then be all
mixed up with the possibly binary data, and if redirected, you won't see it.

I can see the idea of having one permanently open channel, but call it
stdbinout or stdpipeout. But you still won't be able to generate a
sequence of distinct data blocks along that one channel because it is
continuous.

This why 'as' only ever produces one object file, even for multiple
input source files.

And explains why 'as' treats multiple .s input files as though they were
all part of the same single source file: you can take one .s file, chop
it up into multiple .s files, and submit them all to 'as' (keeping the
right order).

It's a feature! It's also the whackiest assembler I've encountered, this
century anyway. That fact that it's implemented as a crude filter with
one input stream and one output streams helps explain it.

Although it works differently from most such filters, because if its
output is not piped, and not redirected, it is sent to a file (always
called a.out). It's not quite crazy enough to send binary object file
data to the termimal; I wonder why not?

On 01.02.2024 11:30, David Brown wrote:
> On 31/01/2024 14:35, Janis Papanagnou wrote:
>>
>> First; the EU publishes in all languages of the member states,
>> for example. (There's no single lingua franca.)
>
> Weirdly, while Norway is not in the EU but Sweden and Denmark are, they
> publish (for some things at least) in Norwegian but not in Swedish or
> Danish. [...]

Hmm.. - in my ears this sounds strange. I've looked it up and found...

"The EU has 24 official languages:

Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish,
French, German, Greek, Hungarian, Irish, Italian, Latvian, Lithuanian,
Maltese, Polish, Portuguese, Romanian, Slovak, Slovenian, Spanish and
Swedish."

[
https://european-union.europa.eu/principles-countries-history/languages_en ]

Searching for a specific regulation, e.g. the GDPR, there's documents
in these languages:

BG ES CS DA DE ET EL EN FR GA HR IT
LV LT HU MT NL PL PT RO SK SL FI SV

"We aim to provide information on our websites in all 24 EU official
languages. If content is not available in your chosen EU language,
more and more websites offer eTranslation, the Commission’s machine
translation service."

"All content is published in at least English, because research has
shown that with English we can reach around 90% of visitors to our
sites in either their preferred foreign language or their native
language."

[ https://european-union.europa.eu/languages-our-websites_en ]

It's interesting that they have an extra explanation about English:

"English remains an official EU language, despite the United Kingdom
having left the EU. It remains an official and working language of the
EU institutions as long as it is listed as such in Regulation No 1.
English is also one of Ireland’s and Malta’s official languages."

[
https://european-union.europa.eu/principles-countries-history/languages_en ]

Janis

On 01/02/2024 16:30, Scott Lurndal wrote:
> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>> On 01/02/2024 15:07, Janis Papanagnou wrote:
>>> On 01.02.2024 14:26, Malcolm McLean wrote:
>>>> On 01/02/2024 13:02, Janis Papanagnou wrote:
>>>>> Well, not necessarily. Let's leave the typical use case for a moment...
>>>>>
>>>>> It might also be analyzed and converted to a digitally represented
>>>>> formula, say some TeX code, or e.g. like the formal syntax that the
>>>>> lilypond program uses.
>>>>>
>>>> And ultimately converted to a non binary form. A list of 1s and 0s is
>>>> seldom any use to the final consumer of the data.
>>>
>>> No, I was speaking about an application that creates lilypond _input_,
>>> which is a formal language to write notes, e.g. for evaluation by the
>>> lilypond software, but not excluding other usages.
>>>
>>>>
>>>>>> The two problems with hex
>>>>>> dumps are that you've got to do mental arithmetic to convert 8 bit hex
>>>>>> values into 16 or 32 bit fields,
>>>>>
>>>>> Hmm.. - have you inspected the man pages of the tools?
>>>>>
>>>> I just ran "man xxd". The man page contains this statement.
>>>>
>>>> The tool's weirdness matches its creator's brain. Use entirely at your
>>>> own risk. Copy files. Trace it. Become a wizard.
>>>
>>> This statement repelled you? (Can't help you here.)
>>>
>>>>> At least for 'od' I know it's easy per option...
>>>>> od -c file # characters (or escapes and octals)
>>>>> od -t x1 file # hex octets
>>>>> od -t x2 file # words (two octets)
>>>>> od -c -t x1 file # characters and octets
>>>>>
>>>> So a JPEG file starts with
>>>> FF D8
>>>> FF E0
>>>> hi lo (length of the FF E0 segment)
>>>>
>>>> So we want the output
>>>>
>>>> FF D8 FF E0 [1000] to check that the segment markers are correct and FF
>>>> E0 segment is genuinely a thousand bytes (or whatever it is). This isn't
>>>> easy to achieve with a hex dump utility.
>>>
>>> I don't know binary format details about jpg, so I cannot help you here.
>>>
>> JPEG is an extremely common binary file format and JPEG files will be
>> found on most general purpose computers.
>> All you need to know for the purposes of the discussion is that the
>> first four bytes are segment identifiers and must have the values I
>> gave, whilst bytes five and six are a big endian 16 bit number that
>> represents a segment length, and that potentially any of those values
>> could be unexpected and you might want to inspect them.
>>
>> So how would you achieve that in a convenient and non-error prone way?
>
> $ if file /tmp/garage.jpg | grep JPEG > /dev/null^Jthen^Jecho "it is a jpeg"^Jfi
> it is a jpeg

That doesn't work for me:

root@xxx:/mnt/c/mx# ls card2.jpg
card2.jpg
root@xxx:/mnt/c/mx# if file card2.jpg | grep JPEG >
/dev/null^Jthen^Jecho "it is a jpeg"^Jfi
>
>

I just get a lone ">". If press Enter, I get more. If I press Ctrl=D, it
says:

> -bash: syntax error: unexpected end of file
logout

I think anyway that you need to grep for JFIF not JPEG, but that is a
really poor way to check for a JPEG file. Any text or binary file can
have a JFIF byte sequence.

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
>On 01.02.2024 16:41, Malcolm McLean wrote:
>>
>> So could you list one or two reasons why you might prefer a program with
>> five subroutines, and one or two reasons why you might prefer to write
>> five programs which communicate via piped data?
>
>A quite appealing and naturally appearing task (from the past) to use
>pipes was to model communication cascades. Something like (off the top
>of my head)...
>
> data-source | sign | compress | crc | encrypt | channel-enc |
> interleaver | channel-simulator | deinterleaver | channel-dec |
> decrypt | crc-check | uncompress | check-sign | data-sink
>
>Component-pairs can be omitted, say you may leave out the un-/compress
>function. And every component may be either special purpose or general.
>A special purpose entity could be BCH-enc and RCPC-enc, or it can also
>be (if better suited) a combined module, say 'crc -16' vs. 'crc -32'
>with the function realized as option argument.

There was also the widely used netpbm package for translating
between different image formats.

https://en.wikipedia.org/wiki/Netpbm

$ giftopnm somepic.gif | ppmtobmp > somepic.bmp
$ for i in *.png; do pngtopam $i | ppmtojpeg >`basename $i .png`.jpg; done