On 2022-01-21 12:46, Bart wrote:
> On 20/01/2022 21:25, Dmitry A. Kazakov wrote:
>> On 2022-01-20 21:13, Bart wrote:
>
>>> I notice you didn't respond to my bookshelf analogy.
>>
>> Because it is a wrong analogy. The right one is the poles in the Tower
>> of Hanoi game with the difference that instead of moving rings one
>> moves the poles. Though one could move the rings instead, in some
>> rather inefficient implementation.
>
> OK. But if I've got a pile of books on one pole (using a hole near one
> corner to minimise the inconvenience), how to do I remove a book in the
> middle of the pile?

Simple, do not pile up your books!

"Algorithms + Data Structures = Programs"
-- Niklaus Wirth

> Here's a very similar one which is about actual code:

[...]

Irrelevant stuff. The case is about string expressions and passing
parameters.

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

On 21/01/2022 12:27, Dmitry A. Kazakov wrote:
> On 2022-01-21 12:46, Bart wrote:
>> On 20/01/2022 21:25, Dmitry A. Kazakov wrote:
>>> On 2022-01-20 21:13, Bart wrote:
>>
>>>> I notice you didn't respond to my bookshelf analogy.
>>>
>>> Because it is a wrong analogy. The right one is the poles in the
>>> Tower of Hanoi game with the difference that instead of moving rings
>>> one moves the poles. Though one could move the rings instead, in some
>>> rather inefficient implementation.
>>
>> OK. But if I've got a pile of books on one pole (using a hole near one
>> corner to minimise the inconvenience), how to do I remove a book in
>> the middle of the pile?
>
> Simple, do not pile up your books!
>
> "Algorithms + Data Structures = Programs"
>    -- Niklaus Wirth
>
>> Here's a very similar one which is about actual code:
>
> [...]
>
> Irrelevant stuff. The case is about string expressions and passing
> parameters.

Sounds like you are admitting the need for heap-like allocations in some
cases.

Which means that if they exist, then objects that could be conditionally
heap-allocated could be passed to functions and specifically to Print.

On 2022-01-21 19:29, Bart wrote:

> Sounds like you are admitting the need for heap-like allocations in some
> cases.

You must be joking. Surely there exist data structures and algorithms
that rely on the heap. That never was a discussion point.

> Which means that if they exist, then objects that could be conditionally
> heap-allocated could be passed to functions and specifically to Print.

Why should that impose any problems? You doubt you could allocate an
integer or a string or whatever (except for a very limited set of data
types) in the heap? In any decent language you can.

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

On 21/01/2022 00:28, Bart wrote:
>>> The sort of tagged unions /I/ would want, would need a tag value that
>>> is a global enum.
>>      That makes it [unnecessarily] hard to compile independent
>> modules.  But perhaps I've misunderstood your notion of "global"?
> Typically it might be used like this:
>     record unitrec_etc =
>         byte tag              # or opcode, opndtype etc
>         ...
>     end
> 'tag' is an instance of some global enumeration: token type, AST mode
> type, IL or target language opcode.

Yes, but does "global" mean "throughout this module" or
"throughout this program" [including perhaps independent modules
that were compiled a decade ago] or "global throughout the entire
visible universe" [IOW, a tag that can be derived universally
from the declaration of the type]? Separately, I don't see what
legitimate use there is for "tag" for union types that makes

case fred.tag in
...
tagforreals: somecode,
...
esac

interestingly different from

case fred in
...
(real): somecode,
...
esac

[...]
> However I've done a test, and the results are interesting. This is
> with dynamic code with 1 million iterations of this, directed to a
> file:
>    println a, b, c, d      # took 1.8 seconds
[...]
>    F((a, b, c, d))        # F iterates over a list and prints
>                           # one at a time + discrete space
> was 2.8 seconds.

IOW, the extra cost, on your system, of using some sort of
cleverly-formatted user-code transput instead of the default [which
I would call unformatted, as you have not provided the "mould" into
which "a, b, c, d" are to be cast] was one second per million lines,
call it per 15000 pages, ~50 reasonably-sized books. How does that
compare with the typical time taken to generate those four million
variables in the first place? Does anyone care about that extra
microsecond per line in real practical use? [Note that we're here
talking about transput for human consumption, not things like file
copying, which would normally be buffered and be much faster.]

--
Andy Walker, Nottingham.
Andy's music pages: www.cuboid.me.uk/andy/Music
Composer of the day: www.cuboid.me.uk/andy/Music/Composers/Schubert

On 22/01/2022 22:40, Andy Walker wrote:
> On 21/01/2022 00:28, Bart wrote:
>>>> The sort of tagged unions /I/ would want, would need a tag value that
>>>> is a global enum.
>>>      That makes it [unnecessarily] hard to compile independent
>>> modules.  But perhaps I've misunderstood your notion of "global"?
>> Typically it might be used like this:
>>      record unitrec_etc =
>>          byte tag              # or opcode, opndtype etc
>>          ...
>>      end
>> 'tag' is an instance of some global enumeration: token type, AST mode
>> type, IL or target language opcode.
>
>     Yes, but does "global" mean "throughout this module" or
> "throughout this program" [including perhaps independent modules
> that were compiled a decade ago] or "global throughout the entire
> visible universe" [IOW, a tag that can be derived universally
> from the declaration of the type]?

Does it matter? It's an enum that lives in the same kinds of scopes as
anything else; how global is up to the language, the programmer and the
app's needs, whether it's between functions, modules, subprograms or
programs (the 4 main scopes of my languages).

>  Separately, I don't see what
> legitimate use there is for "tag" for union types that makes
>
>   case fred.tag in
>     ...
>     tagforreals:  somecode,
>     ...
>   esac
>
> interestingly different from
>
>   case fred in
>     ...
>     (real): somecode,
>     ...
>   esac

Try using more useful tags as:

case p.opcode
when add then ...
when sub then ...
when mul then ...

which causes other elements of p to be interpreted differently depending
on that tag.

>     IOW, the extra cost, on your system, of using some sort of
> cleverly-formatted user-code transput instead of the default [which
> I would call unformatted, as you have not provided the "mould" into
> which "a, b, c, d" are to be cast]  was one second per million lines,
> call it per 15000 pages, ~50 reasonably-sized books.  How does that
> compare with the typical time taken to generate those four million
> variables in the first place?  Does anyone care about that extra
> microsecond per line in real practical use?  [Note that we're here
> talking about transput for human consumption, not things like file
> copying, which would normally be buffered and be much faster.]

This is all nonsense. When you need the speed, and most people apart
from you do want it, then 50% is a big deal:

* The Z80 used to run at 4MHz, until eventually there was a 6MHz part
that could run programs 50% faster.

* How much effort have Intel, AMD and ARM put into speeding up programs
on their CPUs by 50% at the same clock speed. /If only/ they could just
run at a 50% faster clock speed.

* When I did all the work to convert my 16-bit application 20 years ago,
into a 32-bit one, the new program ran 50% faster (and helped my client
get continuing support contracts)

* The difference in generated code between my toy, homemade compilers
and giant LLVM-based ones with dozens of optimisation passes like Clang
is typically 50%.

* My new PC runs programs 50% faster then the 12-year-old one I had before.

And here you're just dismissing it because it merely means that some
program runs in 3 seconds instead of 2.

The fact is that when you are implementing a language feature, you don't
know what use a programmer will put it to; it might be speed critical,
or maybe not.

On 22/01/2022 22:40, Andy Walker wrote:
> On 21/01/2022 00:28, Bart wrote:

>> However I've done a test, and the results are interesting. This is
>> with dynamic code with 1 million iterations of this, directed to a
>> file:
>>     println a, b, c, d      # took 1.8 seconds
> [...]
>>     F((a, b, c, d))        # F iterates over a list and prints
>>                            # one at a time + discrete space
>> was 2.8 seconds.
>
>     IOW, the extra cost, on your system, of using some sort of
> cleverly-formatted user-code transput instead of the default [which
> I would call unformatted, as you have not provided the "mould" into
> which "a, b, c, d" are to be cast]  was one second per million lines,
> call it per 15000 pages, ~50 reasonably-sized books.  How does that
> compare with the typical time taken to generate those four million
> variables in the first place?  Does anyone care about that extra
> microsecond per line in real practical use?

For a long while I paid little attention to the speed of Print, thinking
as you do that Print, turning numbers into text, was slow anyway. Until
I measured it.

For a test like this:

to 10 million do
println a, b, c, d
end

with respective values of 1, 2, 3, 4, with spaces and newline, it used
to take 11.3 seconds. A C program using a single printf took up to 3-4
seconds. So what was the problem?

My runtime eventually also calls printf, to print a character or string,
but it calls it several times for each println statement: 4 times for
the stringified value of each number; 3 times printing a single space,
and once to print a newline character. So 8 calls in all.

I spent an hour or so adding some buffering, so that it most cases it
would also end up calling printf only once.

Now the benchmark took only 2.8 seconds; faster than C, and /4 times as
fast/ as originally.

And yes, I do often write large text files using just Print, it /can/
take a long time for big files.

These are figures for a selection of languages; in all cases, capturing
output to the console, and redirecting to a file:

Millions of lines per second

M (mine) 3.5
LuaJIT 3.1
C 2.5-3.0M
Q (mine) 0.5 (2.5M if it just calls printf...)
PyPy 0.4
Lua 0.2
CPython 0.2
A68G 0.03 (Extrapolated from 30K lines)

Remember this is not a demanding task computationally.

On 22/01/2022 23:41, Bart wrote:
[I wrote:]
>>      IOW, the extra cost, on your system, of using some sort of
>> cleverly-formatted user-code transput instead of the default [which
>> I would call unformatted, as you have not provided the "mould" into
>> which "a, b, c, d" are to be cast]  was one second per million lines,
>> call it per 15000 pages, ~50 reasonably-sized books.  How does that
>> compare with the typical time taken to generate those four million
>> variables in the first place?  Does anyone care about that extra
>> microsecond per line in real practical use?  [Note that we're here
>> talking about transput for human consumption, not things like file
>> copying, which would normally be buffered and be much faster.]
> This is all nonsense. When you need the speed, and most people apart
> from you do want it, then 50% is a big deal:
> * The Z80 used to run at 4MHz, until eventually there was a 6MHz part
> that could run programs 50% faster.
[snip further examples of *programs* and *computers* running 50% faster]
> And here you're just dismissing it because it merely means that some
> program runs in 3 seconds instead of 2.
> The fact is that when you are implementing a language feature, you
> don't know what use a programmer will put it to; it might be speed
> critical, or maybe not.

Did you read what I wrote? Which was nothing about computers
or programs running faster, but about "transput for human consumption".
I don't know how fast you read; I normally manage ~100 pages/hour.
Nor how fast your lineprinter is; mine does 36 pages/minute. IOW,
your 1000000 lines would take me around 150 hours [call it three
working weeks] to read or around 7 hours to print. You would have
to be able to read or print 1000x faster for the saved second to be
of even the slightest interest.

*Note again*: this is for /human/ use; obviously an extra
50% in the total time taken to do a multi-terabyte back up would be
different [and I personally would not use fancy string formatting on
every line to implement that].

Even that extra second is not comparing like with like; you
were comparing a default "print a b c d" with a procedure call that
formatted "a", "b", "c", "d" first; we have to assume that the
programmer wanted something different from the default, and it's not
surprising that it takes extra time to compute that. You're also
still ignoring what would normally be the dominant time, which would
be that taken to compute the four million variables in the first
place; it's 50% faster only if that time is negligible.

[I note your second reply, which seems to have the same
misconception and to be equally irrelevant.]

--
Andy Walker, Nottingham.
Andy's music pages: www.cuboid.me.uk/andy/Music
Composer of the day: www.cuboid.me.uk/andy/Music/Composers/Hause

On 24/01/2022 18:41, Andy Walker wrote:
> On 22/01/2022 23:41, Bart wrote:
> [I wrote:]
>>>      IOW, the extra cost, on your system, of using some sort of
>>> cleverly-formatted user-code transput instead of the default [which
>>> I would call unformatted, as you have not provided the "mould" into
>>> which "a, b, c, d" are to be cast]  was one second per million lines,
>>> call it per 15000 pages, ~50 reasonably-sized books.  How does that
>>> compare with the typical time taken to generate those four million
>>> variables in the first place?  Does anyone care about that extra
>>> microsecond per line in real practical use?  [Note that we're here
>>> talking about transput for human consumption, not things like file
>>> copying, which would normally be buffered and be much faster.]
>> This is all nonsense. When you need the speed, and most people apart
>> from you do want it, then 50% is a big deal:
>> * The Z80 used to run at 4MHz, until eventually there was a 6MHz part
>> that could run programs 50% faster.
>   [snip further examples of *programs* and *computers* running 50% faster]
>> And here you're just dismissing it because it merely means that some
>> program runs in 3 seconds instead of 2.
>> The fact is that when you are implementing a language feature, you
>> don't know what use a programmer will put it to; it might be speed
>> critical, or maybe not.
>
>     Did you read what I wrote?  Which was nothing about computers
> or programs running faster, but about "transput for human consumption".
> I don't know how fast you read;  I normally manage ~100 pages/hour.
> Nor how fast your lineprinter is;  mine does 36 pages/minute.

I think you're just not getting it.

Imagine if your editor only updated the display at 1200 cps because
that's the speed at which humans can read text. So it would take several
seconds per page; a long time to scroll through a file!

The fact is that most text generated by a computer is not read; maybe a
lot of output is produced which is selectively read. A lot of it is for
machine consumption.

For example, when gcc compiles a C program, it produces a .s file
containing assembly code, code that no human is ever going to read. And
yes, you will still need formatting to get it right.

Text files are also read and written for use as data, and for data
formats (such as DXF or some PGM formats).

So when you are implementing Print, although it's not the fastest kind
of computation because it may be dealing with binary/text conversions
and working with strings, you don't want it to be unnecessarily slow.

> IOW,
> your 1000000 lines would take me around 150 hours [call it three
> working weeks] to read or around 7 hours to print.  You would have
> to be able to read or print 1000x faster for the saved second to be
> of even the slightest interest.
>
>     *Note again*:  this is for /human/ use;

No, it isn't. A tiny, tiny fraction of such output will actually be
processed by a human.

>     Even that extra second is not comparing like with like;  you
> were comparing a default "print a b c d" with a procedure call that
> formatted "a", "b", "c", "d" first;  we have to assume that the
> programmer wanted something different from the default, and it's not
> surprising that it takes extra time to compute that.  You're also
> still ignoring what would normally be the dominant time, which would
> be that taken to compute the four million variables in the first
> place;  it's 50% faster only if that time is negligible.

That's not generally the case either. The data may already exist, or the
generation is neglible compared printing it to a text file. IME that
latter part is the bottleneck.

This is not Asimov's 1 cubic mile Multivac that takes days to do a
calculation then prints the single result on a card.

I'm done another experiment: I took my compiler, and a 700Kloc test file
that normally takes 1.1 seconds to turn into an 8MB executable.

I got it to generate an ASM file instead; now it took 3.7 seconds in all
(and the result still needs to be assembled into a binary). Text
processing is slow!

This is normally done into an internal file buffer. I changed it to use
Print directly to a file. This was only done superficially, taking 1-3
strings per line (opcode and 1-2 operands, still generated internally),
and printing those.

Now it takes 5.1 seconds in all. Remember that lexing, parsing, name
resolving, type analysis, IL code generation, x64 code generation,
native code binary image generation, writing EXE - the 'computation' -
took only 1.2 seconds.

Doing only part of that and writing it out as text, partially using
Print, took 4 times as long.

Yes ideally you would avoiding using text, but sometimes it the easiest
way to get things done; it's portable, and also human readable when you
need to debug something on line 873,900 out of 2M lines. But you're not
going to read all of it! (My test above was 2.3M lines.)

On 24/01/2022 20:14, Bart wrote:

>> Nor how fast your lineprinter is;  mine does 36 pages/minute.
>
> I think you're just not getting it.
>
> Imagine if your editor only updated the display at 1200 cps because
> that's the speed at which humans can read text. So it would take several
> seconds per page; a long time to scroll through a file!

Human reading speed is 250wpm, not per second! So update speed would be
20cps - double that of an ASR33 - if it was only necessary to keep up
with someone reading as it printed.

(So updating a page on my 100x60 display would take, what, 5 minutes?)

Also, WHY can your printer do 36 pages per minute, if you cannot read
that fast? Isn't the output for human consumption too?

The answer to that will help in understanding why a language might have
output facilities that not only work faster than 20cps, but magnitudes
faster.

On 24/01/2022 20:14, Bart wrote:
> For example, when gcc compiles a C program, it produces a .s file
> containing assembly code, code that no human is ever going to read.
> And yes, you will still need formatting to get it right. [...]

The question wasn't about whether you need formatting, but
whether you need formatting beyond what "print" can produce. I
don't dispute that some programs produce output by the megaline;
whether they need fancy output [which was the root cause of the
"wasted" extra second per megaline] is another matter.

>> IOW,
>> your 1000000 lines would take me around 150 hours [call it three
>> working weeks] to read or around 7 hours to print.  You would have
>> to be able to read or print 1000x faster for the saved second to be
>> of even the slightest interest.
>>      *Note again*:  this is for /human/ use;
> No, it isn't. A tiny, tiny fraction of such output will actually be
> processed by a human.

Of course! I'm not going to spend three weeks actually
reading all that fancy output. It makes more sense to spend a
tiny, tiny fraction of that time writing an extra condition so
that just [an approximation] to that tiny, tiny fraction of
useful information is produced in the first place. If you want
more than appeared at first sight, in your scenario it takes
only three seconds to reconstruct it "ab initio". Or to save
it to disc, for analytic dissection at leisure.

>> [...] You're also
>> still ignoring what would normally be the dominant time, which would
>> be that taken to compute the four million variables in the first
>> place;  it's 50% faster only if that time is negligible.
> That's not generally the case either. The data may already exist, or
> the generation is neglible compared printing it to a text file. IME
> that latter part is the bottleneck.

Your computing practice obviously differs from mine. I
write programs to do things, such as solve equations, search for
solutions to problems, find out whether [some number] is prime,
..... Yes, of course there are some problems where finding out
that you want to print "1 2 3 4" is as fast as or faster than
converting that to a string and copying that string to a buffer;
but that's not the general case.

> I'm done another experiment: I took my compiler, and a 700Kloc test
> file that normally takes 1.1 seconds to turn into an 8MB executable.

And you're off again! How long does it take you to write
700Kloc? A year? How often do you turn it into an executable?
Ten times a day? Wow, ...

> I got it to generate an ASM file instead; now it took 3.7 seconds in
> all (and the result still needs to be assembled into a binary). Text
> processing is slow!

..., you could have saved ~30s per day. You could have used
that to get a cup of coffee. Or perhaps not. It really, really is
not worth worrying about. Half an hour per day might be worthwhile,
half a minute isn't.

[...]
> Yes ideally you would avoiding using text, but sometimes it the
> easiest way to get things done; it's portable, and also human
> readable when you need to debug something on line 873,900 out of 2M
> lines. But you're not going to read all of it! (My test above was
> 2.3M lines.)

Leading to another bugbear. There shouldn't be a line
873,900. Large projects are normally broken down into separate
modules. They're easier to debug that way. You can't possibly
understand 2.3M lines; as a matter of practicality, not only
will you not understand them, they will also contain dozens or
hundreds of bugs of varying seriousness. We see the results in
the almost daily updates to our systems, in the bloat that is
our browsers, mailers, compilers, ...., in the malware that gets
installed and has dire effects on our computers and so on.

I'm not sure what the modern equivalent is of Dijkstra's
proposal that programmers should be charged for the punched cards
they use [which would make 2.3Mloc prohibitively expensive], but
something equivalent would be a Good Thing. The latest versions
of major programs always tell us about the new facilities, never
about the unused facilities that have been abandoned. Of course,
that's because there is someone somewhere who might be using that
facility. But the effect over the decades is that the browser on
my computer is 1225 times as big as its predecessor ~20 years ago
for only very marginal improvements in utility.

--
Andy Walker, Nottingham.
Andy's music pages: www.cuboid.me.uk/andy/Music
Composer of the day: www.cuboid.me.uk/andy/Music/Composers/Kotzwara

On 24/01/2022 22:25, Bart wrote:
> Also, WHY can your printer do 36 pages per minute, if you cannot read
> that fast? Isn't the output for human consumption too?

(a) Because it produces hard copy, which I can store away and
read at leisure. (b) Because it also prints pictures/diagrams, which
can be adequately scanned much more quickly [picture worth 1000 words
and all that]. (c) Because I often want several, or even many, copies
[eg for circulation to friends, colleagues or our Christmas card list].

None of those have any sensible connexion with whether a program
produces a page in 0.14ms or 0.2ms. Either is perfectly fast enough
for any normal user who wants programs of a few hundred lines to run
in somewhere between a second and a few minutes, depending on the
complexity of the task being tackled, and to produce normally between
one and a thousand lines of output.

> The answer to that will help in understanding why a language might
> have output facilities that not only work faster than 20cps, but
> magnitudes faster.

No-one is proposing anything different.

--
Andy Walker, Nottingham.
Andy's music pages: www.cuboid.me.uk/andy/Music
Composer of the day: www.cuboid.me.uk/andy/Music/Composers/Simpson

On 28/01/2022 00:29, Andy Walker wrote:
> On 24/01/2022 20:14, Bart wrote:
>> For example, when gcc compiles a C program, it produces a .s file
>> containing assembly code, code that no human is ever going to read.
>> And yes, you will still need formatting to get it right. [...]
>
>     The question wasn't about whether you need formatting, but
> whether you need formatting beyond what "print" can produce.  I
> don't dispute that some programs produce output by the megaline;
> whether they need fancy output [which was the root cause of the
> "wasted" extra second per megaline] is another matter.
>
>>> IOW,
>>> your 1000000 lines would take me around 150 hours [call it three
>>> working weeks] to read or around 7 hours to print.  You would have
>>> to be able to read or print 1000x faster for the saved second to be
>>> of even the slightest interest.
>>>      *Note again*:  this is for /human/ use;
>> No, it isn't. A tiny, tiny fraction of such output will actually be
>> processed by a human.
>
>     Of course!  I'm not going to spend three weeks actually
> reading all that fancy output.  It makes more sense to spend a
> tiny, tiny fraction

No, you still don't get it. The rest of the output isn't discarded, it's
just not read by a human. MOST TEXT OUTPUT IS FOR MACHINE CONSUMPTION.

Usually it's never read at all by a human (who reads the billions of
lines of XML, HTML or JS code, or billions of lines of .s files produced
by gcc?)

>> I'm done another experiment: I took my compiler, and a 700Kloc test
>> file that normally takes 1.1 seconds to turn into an 8MB executable.
>
>     And you're off again!  How long does it take you to write
> 700Kloc?  A year?  How often do you turn it into an executable?
> Ten times a day?  Wow, ...

Ten times a day, seriously? Who runs a compiler only once an hour?!
Unless the compiler takes 59 minutes to build the program...

I can run it every few seconds. (Most times in response to compile
errors, but mainly to very large numbers of incremental changes. Also, I
might compile anyway, if I can't remember if it needs to. Not a big
deal, as it finishes in the time it takes to press and release the Enter
key.)

But since you've asked, I've just now put a counter into my main
compiler, and a counter into my current project which is a new compiler.
Since when you compile a new version of the new compiler (count A), you
will then run that version at least once (count B).

I'll report back in the next day or so.

>> I got it to generate an ASM file instead; now it took 3.7 seconds in
>> all (and the result still needs to be assembled into a binary). Text
>> processing is slow!
>
>     ..., you could have saved ~30s per day.  You could have used
> that to get a cup of coffee.  Or perhaps not.  It really, really is
> not worth worrying about.  Half an hour per day might be worthwhile,
> half a minute isn't.

You're missing the point. I choose an example with 0.7M/2/3Mloc input
and output to magnify the differences between using text processing and not.

I could have done that with 70/230-line data if I could have measured
accurately enough.

(Which wouldn't have cut any ice, as then you'd just belittle the 400us
saving. But you did anyway, with 3.7 seconds!)

The point was that, with the text option this app spent 75% of its
runtime doing text output. A counter-example to your claim that
formatted text output was an insignificant factor in most programs.

As to whether saving those 3.7 extra seconds, ie. 75% shorter runtime,
was worthwhile, you can't measure that in coffee breaks:

* The software might run on an unattended computer

* It might do 1000s of such jobs a day (so saving buying more h/w)

* A copy of the program might be used by 1000000 people who each save
3.7 seconds each times they run it. Or 37 seconds or 37 minutes; we
don't know what they're running.

* It might use 4 times as much power, if battery-operated

You don't seem to get that fact that you can't isolate one feature in a
language, eg. Print, and proclaim that it will only ever be used to
generate textual output to be consumed in real time by a single human,
therefore it doesn't matter how slow it is so long as it meets that
requirement.

It does matter, as we don't know what it will be used for.

>     Leading to another bugbear.  There shouldn't be a line
> 873,900.  Large projects are normally broken down into separate
> modules.  They're easier to debug that way.  You can't possibly
> understand 2.3M lines;

This is the output of a compiler.

My compilers are whole-program ones which will always produce a single
monolithic output file, be it EXE or ASM (or OBJ or PCL or C or DLL...).

I guess a 2.3MB EXE is OK with you? Those are a LOT harder to debug than
ASM!

(The largest ASM output produced by any of my compilers for a real
program is 0.3Mloc, corresponding to a 1MB EXE.)

> facility.  But the effect over the decades is that the browser on
> my computer is 1225 times as big as its predecessor ~20 years ago
> for only very marginal improvements in utility.

Yes, there are some monster applications about. I specialise in
relatively tiny (for modern PCs) and very fast language applications.

By making throughput a priority, that opened the door to new possibilities:

* Whole-program compilation
* A trivial build process
* The ability to run directly from source

In other words, allow rapid development techniques just like a script
language.

On 28/01/2022 17:29, Bart wrote:
> On 28/01/2022 00:29, Andy Walker wrote:

>>      And you're off again!  How long does it take you to write
>> 700Kloc?  A year?  How often do you turn it into an executable?
>> Ten times a day?  Wow, ...

> I can run it every few seconds.

(This is for normal projects not of the 700Kloc test.)

> But since you've asked, I've just now put a counter into my main
> compiler, and a counter into my current project which is a new compiler.
> Since when you compile a new version of the new compiler (count A), you
> will then run that version at least once (count B).
>
> I'll report back in the next day or so.

In less than 2 days, and only working part-time and not as intensively
as normal, my production compiler has been run nearly 500 times, and the
one under development some 300 times.

So it's quite possible that in one very intensive day's work, I could
invoke a compiler up to 1000 times.

Invocations of a test compiler often involve 1000s of lines of
diagnostic output (eg. AST or ST listings), which can slow things down,
so I try to minimise that as much as possible. (It of course is sent to
a file, otherwise it would take forever.)

But it's clear than large scale textual output, even to a file, is slow.
For that reason, you don't want it any slower than necessary.

On 28/01/2022 17:29, Bart wrote:
[I wrote:]
>>>>      *Note again*:  this is for /human/ use;
>>> No, it isn't. A tiny, tiny fraction of such output will actually be
>>> processed by a human.
>>      Of course!  I'm not going to spend three weeks actually
>> reading all that fancy output.  It makes more sense to spend a
>> tiny, tiny fraction
> No, you still don't get it. The rest of the output isn't discarded,
> it's just not read by a human. MOST TEXT OUTPUT IS FOR MACHINE
> CONSUMPTION.

Yes, of course. But why does that output [the stuff /not/
for human consumption] need fancy formatting? Which, as you seem
to have forgotten, was the point. If no-one is going to look at
your million-line "*.s" file, then you don't care what it looks
like. Only output that you may be going to read needs to be laid
out "just so". Or, if perchance your programs /do/ care about the
precise layout of the "*.s" file, then you need to print them
carefully anyway, and it's silly to complain about the extra time
needed to do so.

> Usually it's never read at all by a human (who reads the billions of
> lines of XML, HTML or JS code, or billions of lines of .s files
> produced by gcc?)

[I don't read any more "XML" (spit!) or JS than I absolutely
have to, but almost all of my HTML is hand- or script-produced (by
scripts of my own). As for "*.s" files, it's more than 30 years
since I looked at one, or even found one on my computer; I have no
idea what "gcc" does behind the scenes, but if it does produce a
"*.s" file it zaps it before completing the compilation of programs
on my machine, and I see no reason at all why anyone who is not
maintaining "gcc" should want to have it.]

>>> I'm done another experiment: I took my compiler, and a 700Kloc test
>>> file that normally takes 1.1 seconds to turn into an 8MB executable.
>>      And you're off again!  How long does it take you to write
>> 700Kloc?  A year?  How often do you turn it into an executable?
>> Ten times a day?  Wow, ...
> Ten times a day, seriously? Who runs a compiler only once an hour?!
> Unless the compiler takes 59 minutes to build the program...

The question wasn't about running the compiler, but about
creating the executable of a 700Kloc test program. Why do you need
to keep on creating that executable? Only when the compiler changes,
surely, and not every time then? There is point to running a series
of tests when a compiler under development reaches a semi-stable
state, but mostly you just need to confirm that the most recent typo
or other bug has been cured. Even then, most test programs are just
to confirm specific features, perhaps 20 lines typically, and don't
usually take even 0.01s to compile.

> I can run it every few seconds. (Most times in response to compile
> errors, but mainly to very large numbers of incremental changes.
> Also, I might compile anyway, if I can't remember if it needs to. Not
> a big deal, as it finishes in the time it takes to press and release
> the Enter key.)

There are morals there. See below. It could well be a big
deal if you persistently re-create files, as they occupy space [and
time] in your back-ups. That's why there are programs like Make to
decide what needs to be done.

> But since you've asked, I've just now put a counter into my main
> compiler, and a counter into my current project which is a new
> compiler. Since when you compile a new version of the new compiler
> (count A), you will then run that version at least once (count B).
> I'll report back in the next day or so.

I would have replied "Don't bother", but as it turned out
the results were interesting. You claim in a nearby article to be
running "up to 1000" compilations per working day, many of them
producing thousands of lines of diagnostics. What? That's around
a compilation every 30s, hour in, hour out. That's enough of a gap
to permit the correction of an obvious typo, but it doesn't allow
much, if any, time for reflexion and careful analysis. See above
re "mainly to very large numbers of incremental changes". Well, of
course, if you make very large numbers of changes without thinking
about them, you are going to get large numbers of infelicities as
well as typos, and a consequent need to make lots more changes. I
wonder if anyone else here programs that way. I certainly don't.

> The point was that, with the text option this app spent 75% of its
> runtime doing text output. A counter-example to your claim that
> formatted text output was an insignificant factor in most programs.

You can't disprove a claim about "most programs" by giving
examples! But again, you seem to be thinking that the problem was
with output, as opposed to /formatted/ output. I have absolutely
no objection to you saving 3.7 seconds by producing boring default
output by the megaline that no-one will read instead of producing
fancy formatted output, and I'm amazed that you think otherwise.
How much does anyone care about the extra 0.0037s wasted by using
carefully-formatted output for the 1000 lines that someone /might/
read?

> You don't seem to get that fact that you can't isolate one feature in
> a language, eg. Print, and proclaim that it will only ever be used to
> generate textual output to be consumed in real time by a single
> human, therefore it doesn't matter how slow it is so long as it meets
> that requirement.

I have made no such [pro]claim. But if formatted output
is too slow, then use unformatted with my blessing. In my normal
practice, it's my default anyway.

> By making throughput a priority, that opened the door to new possibilities:
>   * Whole-program compilation
>   * A trivial build process
>   * The ability to run directly from source

/New/ possibilities? That's straight back to the first 13
years or so of my career in computing. Load-and-go was all there
was in those days. No file systems, no editors, no storage available
to users unless you were one of the prominenti who managed to qualify
for a mag tape. We developed from there as proper operating systems,
file systems, editors, ... allowed modern project management.

> In other words, allow rapid development techniques just like a script
> language.

Well, yeah. I've taken to using A68G as a scripting language
for the jobs that are a little too much for Sh and Sed. OTOH, that
caused problems when the latest A68G came out and all my composers
[as per ".signature"] came out as being called "Warning:". Grr.

[The Shulz-Evler "Blue Danube" -- see below --, btw (Strauss
as you've never heard him before!) was reputedly the most difficult
piece ever put onto piano rolls, ~100 years ago.]

--
Andy Walker, Nottingham.
Andy's music pages: www.cuboid.me.uk/andy/Music
Composer of the day: www.cuboid.me.uk/andy/Music/Composers/Schulz-Evler

On 03/02/2022 21:33, Andy Walker wrote:
> On 28/01/2022 17:29, Bart wrote:

>     The question wasn't about running the compiler, but about
> creating the executable of a 700Kloc test program.  Why do you need
> to keep on creating that executable?

It's a benchmark that helps compare the compilers of different language
implementations. Such programs are pointless in themselves but they can
highlight issues in a compiler that are not otherwise obvious.

It's interesting to see how various compilers cope with such inputs; a
kind of QoI measure. It had to be that big otherwise the fastest
compilers were too quick to measure.

Example: https://github.com/sal55/langs/blob/master/Compilertest3.md

The fastest was just under 1 second; the slowest was an estimated 79000
seconds.

I used that program in this subthread just to give me something
substantial to easily measure.

>     I would have replied "Don't bother", but as it turned out
> the results were interesting.  You claim in a nearby article to be
> running "up to 1000" compilations per working day, many of them
> producing thousands of lines of diagnostics.  What?  That's around
> a compilation every 30s, hour in, hour out.

Actually the average over last few days has been 400 invocations of
either compiler per day. One problem is that there really haven't been
any bugs of the sort that can take hours of intensive work to track
down. Everything has gone /too/ well, and too quickly. So more time was
spent thinking about direction.

>  That's enough of a gap
> to permit the correction of an obvious typo, but it doesn't allow
> much, if any, time for reflexion and careful analysis.

When you have a build system that effectively works instantly, then you
just don't think about it. In fact it's so quick that it can appear that
nothing happened or it went wrong: you might need to build again and
check the time on the EXE to make sure!

>     /New/ possibilities?  That's straight back to the first 13
> years or so of my career in computing.  Load-and-go was all there
> was in those days.  No file systems, no editors, no storage available
> to users unless you were one of the prominenti who managed to qualify
> for a mag tape.  We developed from there as proper operating systems,
> file systems, editors, ... allowed modern project management.

That sounds just like my first machine. But it would have been done like
that out of necessity.

I'm talking about how things have traditionally worked, for languages
compiled to native code, over at least the last half century on machines
with the usual requisites, like a file system, keyboard and display.

I think I've mentioned the C++ programs that took from minutes to many
hours to build (the latter for a mere 100K lines). New languages like
Rust, as you can see from my link, are not really that nippy either.

Then you have makefiles, configure scripts, auto-conf-generated scripts,
Cmake, solution files with MSVC, Docker (whatever that is), SVN, GIT,
MSBuild tools, LLVM backends ...

I'm discarding all of that /because/ I can remember how simple it could be.

This is my current compiler set to an experimental run-from-source mode:

c:\mx>mm hello
Compiling hello.m to memory
Hello, World! 11:55:49

No intermediate executable; it just runs it directly. Of course, hello.m
is a 3-line program. I'll try it on mm.m, a compiler (35Kloc in 38
modules), and give it hello.m as input:

c:\mx>mm mm hello
Compiling mm.m to memory
Compiling hello.m to memory
Hello, World! 11:55:53

So it's first building the whole compiler from source then, then evoking
it on hello.m. Try that with gcc! Or Building Clang+LLVM (tcc however,
it can do it too).

Here, I build the compiler once, run it on itself again, /then/ compile
hello.m:

c:\mx>mm mm mm hello
Compiling mm.m to memory
Compiling mm.m to memory
Compiling hello.m to memory
Hello, World! 11:55:56

These commands were all typed manually one after the other. You can see
from the time stamps that it didn't take long (the last took 0.3 seconds
in all).

I'm trying to make this stuff completely effortless, and more or less
instant, where the scale of the project allows it.

On Thursday, February 3, 2022 at 3:33:07 PM UTC-6, Andy Walker wrote:

> [The Shulz-Evler "Blue Danube" -- see below --, btw (Strauss
> as you've never heard him before!) was reputedly the most difficult
> piece ever put onto piano rolls, ~100 years ago.]
[...]
> Composer of the day: www.cuboid.me.uk/andy/Music/Composers/Schulz-Evler

Woah. It's like Paganini and Liszt had a baby, godfathered by Prokofiev and
Conlon Nancarrow. Vintage black midi! Oh so pretty.

On 05/02/2022 03:50, luserdroog wrote:
[I wrote:]
>> [The Shulz-Evler "Blue Danube" -- see below --, btw (Strauss
>> as you've never heard him before!) was reputedly the most difficult
>> piece ever put onto piano rolls, ~100 years ago.]
> [...]
>> Composer of the day: www.cuboid.me.uk/andy/Music/Composers/Schulz-Evler
> Woah. It's like Paganini and Liszt had a baby, godfathered by Prokofiev and
> Conlon Nancarrow. Vintage black midi! Oh so pretty.

Yes, but did you like it?

Prokofiev is still in copyright [for another 13 months!],
and Nancarrow won't be freely available for decades. So I can't
do anything much with them. But Paganini and Liszt are an astute
observation. But the Schulz-Evler is a factor 1000 or so short of
being "Black MIDI"! Actually, I had forgotten about CN; I should
perhaps have said "the most difficult piece ever /performed/ onto
piano rolls"? Of course it's easy to write "unplayable" MIDI.

Personally, I'm primarily interested in /real/ music.

--
Andy Walker, Nottingham.
Andy's music pages: www.cuboid.me.uk/andy/Music
Composer of the day: www.cuboid.me.uk/andy/Music/Composers/Joplin

On 06/02/2022 20:50, Andy Walker wrote:
> On 05/02/2022 03:50, luserdroog wrote:
> [I wrote:]
>>> [The Shulz-Evler "Blue Danube" -- see below --, btw (Strauss
>>> as you've never heard him before!) was reputedly the most difficult
>>> piece ever put onto piano rolls, ~100 years ago.]
>> [...]
>>> Composer of the day: www.cuboid.me.uk/andy/Music/Composers/Schulz-Evler
>> Woah. It's like Paganini and Liszt had a baby, godfathered by
>> Prokofiev and
>> Conlon Nancarrow. Vintage black midi! Oh so pretty.
>
>     Yes, but did you like it?

Anyone hoping to hear the actual waltz will likely be disappointed. You
can only just about discern it in parts.

It helps if you play it at double the speed (it's also over more quickly!).

>     Prokofiev is still in copyright [for another 13 months!],
> and Nancarrow won't be freely available for decades.  So I can't
> do anything much with them.  But Paganini and Liszt are an astute
> observation.  But the Schulz-Evler is a factor 1000 or so short of
> being "Black MIDI"!  Actually, I had forgotten about CN;  I should
> perhaps have said "the most difficult piece ever /performed/ onto
> piano rolls"?  Of course it's easy to write "unplayable" MIDI.
>
>     Personally, I'm primarily interested in /real/ music.
>

On Sunday, February 6, 2022 at 2:50:21 PM UTC-6, Andy Walker wrote:
> On 05/02/2022 03:50, luserdroog wrote:
> [I wrote:]
> >> [The Shulz-Evler "Blue Danube" -- see below --, btw (Strauss
> >> as you've never heard him before!) was reputedly the most difficult
> >> piece ever put onto piano rolls, ~100 years ago.]
> > [...]
> >> Composer of the day: www.cuboid.me.uk/andy/Music/Composers/Schulz-Evler
> > Woah. It's like Paganini and Liszt had a baby, godfathered by Prokofiev and
> > Conlon Nancarrow. Vintage black midi! Oh so pretty.
> Yes, but did you like it?

Yes, very much so. Thanks for sharing.
> Prokofiev is still in copyright [for another 13 months!],
> and Nancarrow won't be freely available for decades. So I can't
> do anything much with them. But Paganini and Liszt are an astute
> observation. But the Schulz-Evler is a factor 1000 or so short of
> being "Black MIDI"! Actually, I had forgotten about CN; I should
> perhaps have said "the most difficult piece ever /performed/ onto
> piano rolls"? Of course it's easy to write "unplayable" MIDI.
>
> Personally, I'm primarily interested in /real/ music.

I get what you mean. But it can be difficult to establish a rigorous
definition. I didn't understand My Life In The Bush of Ghosts when
I first bought it. But squeeze it in between some Tricky and Bjork
and it's refreshing and just delightful.

The Paganini piece I was particularly reminded of is Caprice No. 6,
"the Trill". I've been trying to play it on the guitar with limited success.

On 08/02/2022 04:30, luserdroog wrote:
[I wrote:]
>> Yes, but did you like it?
> Yes, very much so. Thanks for sharing.

Thanks, and you're welcome.

[...]
>> Personally, I'm primarily interested in /real/ music.
> I get what you mean. But it can be difficult to establish a rigorous
> definition. [...]

It can, but I don't really need one. I just sequence up things I
like, or think I might like when I start, and although I've done a few
very easy pieces, mostly they're pieces that are somewhat beyond my skill
[or /way/ beyond!]. The advantage of computerisation.

To make this article at least somewhat relevant to "c.l.m",
Cakewalk [which I use for sequencing] comes with a language, CAL, in
which to write macros to be applied to the MIDI sequence. It's quite
neat, though rather limited, and I have rather a lot of such macros to
achieve special effects. [These don't always transfer to MIDI files, so
the MIDI on my site is quite different in detail from the MP3s there.]
It would be quite nice to have CAL expanded to a full computer language;
perhaps more fruitful than trying to invent a new C++? But I don't have
full specs either for Cakewalk files or even for MIDI [and too much like
hard work to chase them down any further]. The emphasis these days seems
to be on providing a "home studio" rather than playing music.

> The Paganini piece I was particularly reminded of is Caprice No. 6,
> "the Trill". I've been trying to play it on the guitar with limited success.

Violin music is particularly hard to sequence up. Partly this
is because I don't really understand some of the instructions, nor how
to implement them in MIDI, but mostly it's because my keyboard under-
stands them even less -- any long note disappears into a vibrato that
can't be switched off and short notes come out in a horrible staccato
tone. There are a few violin/chamber/orchestral pieces on my site, but
I'm not proud of the results. The best I've managed, occasionally, is
to replace the violin by a clarinet, which sounds much better on my
keyboard. Yeah, pieces like "The Trill" would probably come out OK in
a guitar version, and would be much easier to play on the keyboard! I
may give it a go.

--
Andy Walker, Nottingham.
Andy's music pages: www.cuboid.me.uk/andy/Music
Composer of the day: www.cuboid.me.uk/andy/Music/Composers/Morel

On 09/01/2022 22:05, Andy Walker wrote:
> On 08/01/2022 17:15, James Harris wrote:
>>>> If you are not convinced by formatted io then what kind of io do
>>>> you prefer?
>>>      Unformatted transput, of course.  Eg,
>>>    print this, that, these, those and the many other things
>> OK. I would say that that has /default/ formatting but is still
>> formatted.
>
>     You can say that if you insist, but ISTM to be an abuse of
> language.  The point about /formatted/ transput is that the user
> prescribes some "mould" into which objects such as integers [for
> output] or strings [for input] are "poured" for conversion.  So
> we get calls like "printf (some_mould, some_value)".  It's a
> useful distinction from unformatted transput, where the user does
> not specify any mould.

You seem to be thinking about printing a number using some character set
- perhaps even some particular base, such as decimal. All of those are,
surely, kinds of formatting. If the default formatting is decimal, with
no leading zeroes and left justified then that's still a way to format a
number.

--
James Harris

On 12/12/2023 16:10, James Harris wrote:
> On 09/01/2022 22:05, I wrote:
>> [...] The point about /formatted/ transput is that the user
>> prescribes some "mould" into which objects such as integers [for
>> output] or strings [for input] are "poured" for conversion.  So
>> we get calls like "printf (some_mould, some_value)".  It's a
>> useful distinction from unformatted transput, where the user does
>> not specify any mould.
> You seem to be thinking about printing a number using some character
> set - perhaps even some particular base, such as decimal. All of
> those are, surely, kinds of formatting. If the default formatting is
> decimal, with no leading zeroes and left justified then that's still
> a way to format a number.

Yes, of course it is, but you have [again] lost the distinction
between a format specified by the programmer and a default. Sometimes
you just want to see the numbers [or whatever]; other times, it matters
to you how they are formatted. In the one case, you don't want to have
to spend time thinking about what format to use [esp while debugging];
in the other, the layout of the output is important. It's more stark on
input; a default read should be able to read what's there if /you/ can,
whereas a formatted read can [eg] split up "123abc0456hello" into a
6-digit hex integer, a 4-digit decimal integer and a 5-char string.
OTOH, an unformatted read could split "123abc 456 hello" into a hex
integer, a decimal integer and a string without having to be told how
many characters to allow for each. It's not an /essential/ difference,
just a convenience for real-life users. So it's a reason for a language
to provide both.
--
Andy Walker, Nottingham.
Andy's music pages: www.cuboid.me.uk/andy/Music
Composer of the day: www.cuboid.me.uk/andy/Music/Composers/Mendelssohn

Andy Walker <anw@cuboid.co.uk> schrieb:
> Sometimes
> you just want to see the numbers [or whatever]; other times, it matters
> to you how they are formatted. In the one case, you don't want to have
> to spend time thinking about what format to use [esp while debugging];
> in the other, the layout of the output is important.

In Fortran, the former is list-formatted output. Using

write (10,*) asdf

will write the value of the expression asdf to unit 10 in a format
of the compiler's choosing, terminating that record (i.e. writing
a newline).

write (10,'(I4)') asdf

will write the value of asdf, if asdf is an integer, to
unit 10 with a width of four characters, right-justified
and filled up with spaces. If the value does not fit, four
asterisks are written instead. This statement will also terminate

If asdf is not an integer, then this is an error, which is usually
caught by the compiler at either compile or run-time.