On 09/12/2021 13:16, David Brown wrote:
> On 09/12/2021 12:01, Bart wrote:
>> On 09/12/2021 08:28, David Brown wrote:
>>> On 09/12/2021 08:46, Dmitry A. Kazakov wrote:
>>
>>>> Yes, a modern medium-sized project is extremely large. In a language
>>>> without polymorphism it would take years to design and the code would be
>>>> tenfold bigger and complex.
>>>
>>> Exactly.  Developer time is more important than computer time.
>>
>> Exactly. That is why having developers twiddle their thumbs for five
>> minutes every time they compile is a joke.
>
> When the alternative is spending weeks or months writing the code by
> hand rather than using templates?

I can think of half a dozen alternatives. Plenty of time to ponder
during the 7 hours out of 8 each day that I would be spending waiting!

One would be to develop first using a rapid-development environment (eg.
scripting). Once it works, then you port it to C++, but now it's a
working program; you will not need to build it so many times.

(But maintenance will still be a pig.)

>> Scripting languages have all the benefits of C++ and then some. Except
>> for execution speed.
>>

> Spoken like someone who does not know C++, or have any concept of
> scripting languages.

I created my first scripting languages in the late 80s (so I course I
have no concept of what they are).

When I first found out about C++ a decade later, and its 'OOP', I wanted
to find out what it could do.

The first examples I saw concerned first-class string handling; stuff
I'd already been doing for years in my script language!

> There are different languages with different balances of features and
> emphasises, and that's good. But you'd have to be completely ignorant
> to think that there are any scripting languages with /all/ the benefits
> of C++

So, what does C++ specifically offer that would be impossible in
scripting code? I don't mean things that only work with native code.

>> But the C++ in these examples is so slow to build, that a scripting
>> language is likely to be faster.
>>
>
> That would seem highly unlikely.

If the reason for slowness is some degenerate cases of template
expansion, then it is a possibility.

Then a script language that provides generics anyway won't have that
problem. It will just be a fixed factor slower for the more conventional
parts of a compiler, not exponentially slower.

On 2021-12-09 14:39, Bart wrote:

> Get me an example of polymorphism, as there seem to be various kinds,
> and I'll see if I can express that in my language.

This again shows lack of understanding. Types of polymorphism are
irrelevant as they all serve the same goal. The difference between
ad-hoc, parametric, dynamic polymorphism are in their
capabilities/limitations to express certain types of classes.

> But, on top of all the various issues involved in bigger projects, you
> don't want build times to be unnecessarily slow for no good reason.

Nobody cares about build times, because this is not how the software
development process works. People even agree to suffer nightly builds,
while calling it agile... (:-))

> For that, you need someone whose job it is to oversee that part, to
> identify such problems, and to do something to fix it if possible.

Yes, you do not know how testing and software quality assurance work either.

>> On your part we see an active denial of very existence of such
>> problems. Things you see important are just silly from the point of
>> view of a practicing software developer. Things you reject are key
>> tools for managing complexity of modern software.
>
> OK. Carry on waiting 5 minutes between each run.

Yep, like the James Webb telescope, 30 years before the first and only
run...

Again, properties of advanced programming languages such as static
verification, modularization, separation of interface and
implementation, reuse, generic programming are focused on risk analysis
and understanding that 50% of cases are fundamentally untestable, 50% of
the rest is economically, technically, timely infeasible to test, 50% of
the rest of the rest require in-depth analysis of the requirements, the
underlying physical processes and the implementation algorithms in order
to test properly.

> Hopefully it will be
> for a more worthwhile reason than over-eager, inappropriate use of
> templates.

Nobody is really fond of templates or generics. The problem is that no
practical alternative to completely supersede parametric polymorphism
was ever shown. Dynamic polymorphism is better than the parametric one
in all aspects except that it cannot handle many cases of the former and
inconsistent with regard to multiple dispatch etc.

My interest in new crazy languages is if they address these issues.
Silly suggestions about non-existing unsigned arithmetic or summing
bitfields are not.

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

On 08/12/2021 22:49, Bart wrote:
[I wrote:]
>>      Meanwhile, my I suggest a little test for you?  Instead
>> of "fannkuch(10)", please get your collection of programs and
>> compilers to produce "fannkuch(2)".  Any compilers that return
>> "2, 2" [without comment and from code equivalent to what you
>> supplied] should be kept well away from any serious computation.
> What should it produce for fannkuch(2)?
> I downloaded versions in 4 languages from the original benchmarks
> site. Three of them (Lua, Python, Julia) go wrong in giving a program
> error.
That's not "wrong"; with the code [as supplied to me in the
Algol version], there /is/ a program error. I deduce that the authors
either didn't test their code or didn't care that there was a bug.
Neither case is commendable. Given a function of "int n", it would
seem natural to me to test with at least something like 0, 1, 2, 3,
something bigger, and something negative. It costs zilch to add a
test "if n < 3 then ..." to the code.

> The fourth, in C, completes, but fannkuch(2) returns -1. I don't know
> if that is correct or not, but it sounds like it might be undefined.

The problem is perfectly well defined in the cases 1 and 2;
for 0 or -1, you get to decide what it means. But if the C is "the
same as" the Algol you supplied, then it has failed to detect the
errors that A68G detects. See below.

> The presence of these lines:
>     swap(p[2],p[3])
>     signx:=1
>     for i:=3 to n do
> plus perm1[2] in the 0-based Python, and p[3] in Julia, suggests n
> needs to be at least 3.

In my universe, it "suggests" that the code ought to check
that "n" is at least 3. If the arrays in fact have size 2, then
using "p[3]" is already an error. If that error is disguised, in
languages where "dynamic arrays" are impossible/difficult, by
declaring the arrays to have size [say] 100 [prompting another
sanity check in good code!], then there is a second error in that
"p[3]" now exists but is uninitialised. A68G detects both of
these; over to you to test other compilers.

I'm not claiming that these are the worst and most serious
bugs ever perpetrated. But they are examples of the sorts of bug
that Black Hats exploit in creating malware. They look at source
code with a keen eye on potential buffer over-runs, on dumping
naughty instructions/data in places where they may be stumbled
across by code using uninitialised storage, on using null pointers
and the like [inc, of course, race conditions, not relevant here].
Languages and compilers that don't help detect these, even in the
presence of good testing, are a menace. [Note that C (eg) /could/
detect these; but not if it's somehow essential to shave every
last microsecond off the run time.]

>>> I would expect an optimising compiler to iron out such minor
>>> differences.
>>      I suspect you're being optimistic if you expect even a good
>> optimiser to detect that [in a reasonably large and complicated
>> program] element 0 is never accessed, or systemically to reduce all
>> indexes by 1.  But you never know.
> Element 0 can stay there; it's not doing any harm with just 3 arrays.

Yes, but it was your claim that an optimising compiler should
"iron out such minor differences". My point was that what start out
as minor differences soon turn into differences that permeate the
entire program. For the sorts of toy program that we tend to give
as examples here, that barely happens. But over something bigger,
say a compiler, you make decisions early on about [eg] whether an
index should be pre- or post-incremented, that influences whether
you use the array element and /then/ change the index or vv, that
changes the value of "i" /after/ the loop, that influences the
next stage of the code and so on. Then you perhaps need to switch
from a 0-based to a 1-based language, and whereas you can perhaps
ignore an element 0 that you don't need, you can't ignore one that
has been used, so you have to change every array access.

> Being 1-based may simply mean an extra offset on some address modes;
> is that what you think is causing a slow-down? With local arrays,
> that offset should just be absorbed into the stack offset of the
> array.

I wasn't thinking about slow-downs at all; I just thought
you were being optimistic in thinking that "gcc -O3" might magically
turn some 1-based code into equivalent 0-based code or vv.

--
Andy Walker, Nottingham.
Andy's music pages: www.cuboid.me.uk/andy/Music
Composer of the day: www.cuboid.me.uk/andy/Music/Composers/Widor

On 09/12/2021 14:49, Dmitry A. Kazakov wrote:
> On 2021-12-09 14:39, Bart wrote:
>
>> Get me an example of polymorphism, as there seem to be various kinds,
>> and I'll see if I can express that in my language.
>
> This again shows lack of understanding. Types of polymorphism are
> irrelevant as they all serve the same goal. The difference between
> ad-hoc, parametric, dynamic polymorphism are in their
> capabilities/limitations to express certain types of classes.
>
>> But, on top of all the various issues involved in bigger projects, you
>> don't want build times to be unnecessarily slow for no good reason.
>
> Nobody cares about build times,
Yeah, like nobody cares about stopping at a red light!

> Yes, you do not know how testing and software quality assurance work
> either.

I guess quality assurance doesn't extend to keeping a development
environment productive.

>>> On your part we see an active denial of very existence of such
>>> problems. Things you see important are just silly from the point of
>>> view of a practicing software developer. Things you reject are key
>>> tools for managing complexity of modern software.
>>
>> OK. Carry on waiting 5 minutes between each run.
>
> Yep, like the James Webb telescope, 30 years before the first and only
> run...
>
> Again, properties of advanced programming languages such as static
> verification, modularization, separation of interface and
> implementation, reuse, generic programming are focused on risk analysis
> and understanding that 50% of cases are fundamentally untestable, 50% of
> the rest is economically, technically, timely infeasible to test, 50% of
> the rest of the rest require in-depth analysis of the requirements, the
> underlying physical processes and the implementation algorithms in order
> to test properly.
Do you think is reasonable to spend 5 minutes across 8 cores doing that
on a 25,000-line program on every routine build?

I don't. Most people would agree that that is over the top. Even with
those 'polymorphic' features (which, if the cost is to slow things down
by 100 times, are non-viable).

Perhaps you genuinely think that's how long such things should take.

Or you are just being scornful of products where it can done instantly.

Bart <bc@freeuk.com> wrote:
> On 04/12/2021 09:32, antispam@math.uni.wroc.pl wrote:
> > Bart <bc@freeuk.com> wrote:
>
> >> * Inline assemby
> >> * Inlining
> >> * User-defined operator overloads
> >> * The equivalent of C++'s constexpr
> >> * A gcc-class optimiser
> >> ...
> >
> > I use a language like this. You do need inlining and it is useful
> > only due to overloading (_not_ limited to operators). However,
> > other position on your list above are not needed. More precisely,
> > there is small core language which implements basic types and
> > corresponding operators. Basic library types simply ofer "view"
> > of basic types and operators from core language. Due to inlining
> > there is no runtime loss of efficiency compared to using core
> > language directly. But there are also fancy library types
> > which are infeasible to have as part of core language.
> >
> >> Too many advanced compilers features, just to keep the language 'small'.
> >
> > You did not get it: purpose is to have managable implementation
> > of _very large_ language.
>
> Source code for gcc was 45,000 source files last I looked. I don't even
> know if that's just for C, or for C++ too.
>
> That's a thousand times a larger scale than what I do. So I guess I'm in
> less need of such methods.

gcc is in different game: "how many % of runtime efficiency can we
gain if we throw 10 times more resources at compilation". But
you can have much smaller implementation of large language.
> > Low-level error prone parts are
> > small. Library types can be written by ordinary application
> > programmers.
> >
> > Overloading can be implemented with quite small extra code.
> > Inlining is also easy to implement if you have internal
> > representation of larger code fragments (as opposed to
> > generating machine code directly after recognizing given
> > syntactic construct). Both put limits on maximal possible
> > compiler speed: with overloading you have more complex
> > symbol table and need more lookups. Intermediate representation
> > takes time to build (on top of time needed to generate code).
> > Also, inlining allows large object code from small sources,
> > so if you measure time per source line compier speed will
> > likely be worse than speed of compiler that forces you to
> > write each source line separately. Still, compiler can
> > be reasonably fast.
> >
> > Anyway, increase in compiler complexity is relatively
> > small. Even for small languages in total balance
> > approach using overloading may give you saving in
> > effort spent on implementation.
>
> I could do overloading. I have an approach I've developed, I just
> haven't got round to it (there are a dozen more things more useful ahead
> of it, which I haven't done yet either!).
>
> But, this purely user-defined overloading of existing operators, with
> user-defined types, or applying extra operations to standard types.
>
> It is not for implementing the core language, which I like to handle by
> dedicated code in the compiler, keeping it tight, fast and allowing such
> enhancements as above to be optional (and on a waiting list).
>
> For example, there is already a backend IL instruction, MUL.I64, to do
> signed multiply. The main part, MUL, is attached to the binary * op
> during parsing. The .I64 part is assigned two passes further on.
>
> With only mechanisms with user-defined overloads, it's a lot harder to
> make those associations. For one thing, those IL instructions are not
> exposed in the language.

I do not think exposing IL is good idea. However, logically all you
have is small number of "special forms", like 'if', loops, assignments
and functions calls. Compiler needs to understand that '+' for
machine integer type should be replaced by single instruction.
In other words, each operation in IL represents optimization for
specific type. This can be as simple as table of tens or hundreds
of entries like:

["+", I64, I64, I64, ADD.I64]

where first position is name of operation, second is result type,
third and fourth are argument types and last one is corresponging
operation in IL. This table is part of compiler, users only knows
that "+" will add integers.

> > Just as a simple
> > example let me mention Pascal. In Pascal there is
> > a buch of "builtin" operations (mostly on files).
> > Most of them are too complex to do by inline code
> > so practical compilers have runtime library implementing
> > them. But compiler still needs to parse those
> > "builtins" and convert them to runtime calls.
> > In a sense this is trivial task, but with realistic
> > compiler structure you need several lines of compiler
> > code to generate corresponding runtime call.
>
> I have a similar set of operations for I/O. They are implemented as a
> set of function calls, designed to be called implicitly from code, so
> have funny names. So:
>
> print a,b
>
> is equivalent to:
>
> m$print_startcon()
> m$print_i64_nf(a) # when a,b are i64
> m$print_i64_nf(b)
> m$print_end
>
> (The bracketing between start/end calls allows separator logic within
> the whole print statement; and allows calls to functions within the
> print list that might themselves use print, to files etc.)
>
> So, print is already implemented in user code. I just do not see the
> advantage of replacing 110 or 50 lines of code within the compiler's
> code generator for print, with a likely more complicated solution.
>
> (110 lines of code for static language which needs to generate different
> calls for various types; 50 lines for dynamic language.)

OK, if you do not need more "builtin functions", then ad-hoc approach
is fine. GNU Pascal supports hundreds of "builtin functions" (when
you count diffecences due to different types of arguments). This
forces more systematic approach. AFAIK for Cobol you will need
more buitins...

> > Bunch
> > of older languages like Cobol, Fortran, PL/I had
> > special "builtin" operations, some of them much
> > more than Pascal.
> >
> > Let me add that there are more radical approaches:
> > there are languages in which "users" can redefine
> > scanner and/or parser. So users can add on the
> > fly new operators and define their meaning. As
> > a possible example consider you 'stringimage'
> > extention:
>
> Do you mean my 'strinclude' which incorporates a text file (or
> text/binary file in dynamic code), as string data within the program?

Yes.

> > in extensible language user simply
> > declares that 'stringimage' is a new keyword that
> > should invoke its handler.
>
> That itself is not so easy. Keywords are built-in, scopeless, and known
> program-wide.

no, no, no.

> With a user-define keyword, could two modules define that
> keyword differently?

Yes, keywords are scoped and recognized only within their scope.
More precisely, each compilation stream has it own keywords which
can be added or retracted at essentially any time.

> Could another part of the program use that same
> name as an ordinary variable?

Yes.

> If so then it's not known as a new keyword after the next pass after
> parsing. So it will need to have a suitable 'shape' to be parsed as
> /something/. (My language also has out-of-order definitions.)
>
>
> Handler looks at
> > following program text, extracts filename,
> > reads the file and tells compiler to use resulting
> > string as a string constant.
>
> How? String literals are detected fairly early, they have a certain AST
> node, length etc. At what pass will this be done?

Apparently you have your compiler in mind and do not think about
alternatives. String literals produce string tokens. There
is no law of nature that forbids alternative ways of creating
string tokens. Once handler created sting token and passed it
to normal compilation mechanizm, there is no difference between
such string token and string tokens coming from string literals
in source file.

> > The whole thing is
> > probably 10 lines for handler and a single
> > line to register new keyword.
>
> In language that
> > allows syntax extentions instead of overloading
> > you can just define new operators, so overloading
> > strictly speaking is not necessary. Similarly
> > you have some way for syntax handlers to generate
> > code, which is similar to inlining, but stricly
> > speaking inlining is not needed.
>
> Do you have an example of such a language where implementing my
> 'strimport' is 10 lines of code? For example, C++. Or the one you
> mentioned. If so, how does it look?

Bart <bc@freeuk.com> wrote:
> On 09/12/2021 14:49, Dmitry A. Kazakov wrote:
> > On 2021-12-09 14:39, Bart wrote:
<snip>
> >>> On your part we see an active denial of very existence of such
> >>> problems. Things you see important are just silly from the point of
> >>> view of a practicing software developer. Things you reject are key
> >>> tools for managing complexity of modern software.
> >>
> >> OK. Carry on waiting 5 minutes between each run.
> >
> > Yep, like the James Webb telescope, 30 years before the first and only
> > run...
> >
> > Again, properties of advanced programming languages such as static
> > verification, modularization, separation of interface and
> > implementation, reuse, generic programming are focused on risk analysis
> > and understanding that 50% of cases are fundamentally untestable, 50% of
> > the rest is economically, technically, timely infeasible to test, 50% of
> > the rest of the rest require in-depth analysis of the requirements, the
> > underlying physical processes and the implementation algorithms in order
> > to test properly.
> Do you think is reasonable to spend 5 minutes across 8 cores doing that
> on a 25,000-line program on every routine build?
>
> I don't. Most people would agree that that is over the top. Even with
> those 'polymorphic' features (which, if the cost is to slow things down
> by 100 times, are non-viable).

Well, in my main project I spent more than 5 minutes on each full build
(it in not 25000, wc counts is closer to 350000). However, full build
is relatively rare thing, normally I do hundreds of compile-edit
cycles between full builds (recompiling single file). Sure,
faster build time would be nice, and I spent some time decreasing
it (it went down from about 2 hours to few minutes, partly due to
faster muchine, partly due to efficiency improvements). But I am
not going to sacrifice features for faster build time.

> Perhaps you genuinely think that's how long such things should take.

Well, health experts says that on should take regular breaks from
programming. 5 minutes is somewhat short, but better than nothing
(10 minutes probably would be optimal).

--
Waldek Hebisch

On 10/12/2021 23:28, antispam@math.uni.wroc.pl wrote:
> Bart <bc@freeuk.com> wrote:

>> Source code for gcc was 45,000 source files last I looked. I don't even
>> know if that's just for C, or for C++ too.
>>
>> That's a thousand times a larger scale than what I do. So I guess I'm in
>> less need of such methods.
>
> gcc is in different game: "how many % of runtime efficiency can we
> gain if we throw 10 times more resources at compilation".

gcc-O3 manages some 50% better performance on the typical programs /I
write/. Example (the BB-opt column, from 2020, is roughly same as my
current compiler):

https://github.com/sal55/langs/blob/master/benchsumm.md

But in general getting 2-3 times as slow as gcc-O3 is easy. Tcc manages
that, and it is a 0.18MB compiler. It will still be faster than most
interpreted languages!

This is for fairly sensible input files. When input is crazier, perhaps
machine-generated with lots of redundant code and variables [or nested
template expansions!], then you will need better optimisation methods to
clean up the mess.

>> (110 lines of code for static language which needs to generate different
>> calls for various types; 50 lines for dynamic language.)
>
> OK, if you do not need more "builtin functions", then ad-hoc approach
> is fine. GNU Pascal supports hundreds of "builtin functions" (when
> you count diffecences due to different types of arguments).

If they have function syntax (note my Print statements don't) without
too many special features, then a systematic approach is easy.

For example, for most binary ops with symmetric operand types, there is
just one AST node type, and translation to IL uses four lines of code:

proc do_bin(unit p,a,b) =
evalunit(a)
evalunit(b)
pcl_gen(p.pclop)
setmode_u(p)
end

You tend to do this when you can find patterns, to avoid duplication.

>> That itself is not so easy. Keywords are built-in, scopeless, and known
>> program-wide.
>
> no, no, no.
>
>> With a user-define keyword, could two modules define that
>> keyword differently?
>
> Yes, keywords are scoped and recognized only within their scope.
> More precisely, each compilation stream has it own keywords which
> can be added or retracted at essentially any time.

Those aren't what I call keywords! More like user-defined syntax. But
with custom syntax, you don't want it changing in different parts of the
program.

Or are you thinking about Lisp?

>> Do you have an example of such a language where implementing my
>> 'strimport' is 10 lines of code? For example, C++. Or the one you
>> mentioned. If so, how does it look?
>
> Your 'stringimage' can be done as Lisp macro:
>
> ;;; ---<cut here>-------------
> (defun slurp_file (fn)
> (with-open-file (f fn)
> (let ((seq (make-string (file-length f))))
> (read-sequence seq f)
> (coerce seq 'string)))
> )
>
> (defmacro string_include (f)
> (slurp_file f))
> ;;; ---<cut here>-------------
>
> Note: in source file use of 'string_include' looks like function
> call (almost any Lisp construct looks like function call).
> However, since it is a Lisp macro, it is executed at compile
> time and result is handled like part of source.

So this approach can also be used by C++'s constexpr feature? (Assuming
there are ways to read files into strings.)

I notice you say 'compile-time'; does Lisp have a compilation stage with
some intermediate or binary output? I thought it was run from source.
Actually I thought that that was the big deal about it, that you can
manipulate Lisp code as though it was runtime data.

The thing about a discrete compilation stage is that the resulting
binary output contains embedded versions of those string includes.

(My current dynamic language is run-from-source only. However there is a
feature to package all source/support files into one text file; the
string include file goes there.)

>
>> I think that 20 lines of code for 'strinclude' inside a compiler is
>> reasonable.
>
> Sure. However note the difference: you could add 'strinclude' because
> you are compiler author. With extensible language any user can add
> constructs that she/he needs.

No, my language isn't extendable that way. People can only add things in
the usual way: user-defined functions, records, types, etc.

I think it makes for a simpler, easier-to-implement and faster-to-run
language.

On 10/12/2021 23:46, antispam@math.uni.wroc.pl wrote:
> Bart <bc@freeuk.com> wrote:

>
>> Perhaps you genuinely think that's how long such things should take.
>
> Well, health experts says that on should take regular breaks from
> programming. 5 minutes is somewhat short, but better than nothing
> (10 minutes probably would be optimal).

I take plenty of breaks, like going on forums like these!

But when I am working, then I don't want to hang about waiting until I
see the results of any change.

Bart <bc@freeuk.com> wrote:
> On 10/12/2021 23:28, antispam@math.uni.wroc.pl wrote:
> > Bart <bc@freeuk.com> wrote:
<snip>
> >> That itself is not so easy. Keywords are built-in, scopeless, and known
> >> program-wide.
> >
> > no, no, no.
> >
> >> With a user-define keyword, could two modules define that
> >> keyword differently?
> >
> > Yes, keywords are scoped and recognized only within their scope.
> > More precisely, each compilation stream has it own keywords which
> > can be added or retracted at essentially any time.
>
> Those aren't what I call keywords! More like user-defined syntax. But
> with custom syntax, you don't want it changing in different parts of the
> program.
>
> Or are you thinking about Lisp?

Not only Lisp. The following is slightly redacted part of Pop11 code:

external declare XEvents in c;

typedef struct {
int type; ;;; of event */
unsigned long serial; ;;; # of last request processed by server */
Bool send_event; ;;; true if this came from a SendEvent request */
Display *display; ;;; Display the event was read from */
Window window; ;;; "event" window it is reported relative to */
Window root; ;;; root window that the event occured on */
Window subwindow; ;;; child window */
Time time; ;;; milliseconds */
int x, y; ;;; pointer x, y coordinates in event window */
int x_root, y_root; ;;; coordinates relative to root */
unsigned int state; ;;; key or button mask */
unsigned int button; ;;; detail */
Bool same_screen; ;;; same screen flag */
} XButtonEvent;

....
....
....

int *XSynchronize(display, onoff)
Display *display;
int onoff;
{}

endexternal;

define global QLength(dpy);
lvars dpy;
dpy #-> qlen
enddefine;

Part between 'external' and 'endexternal' essentially uses C syntax.
I declares C routines callable from Pop11. To properly declare C
routines 'external' part also declares bunch of C types (needed for
proper handling of arguments. After 'endexternal' you have more
normal Pop11 code. 'external' part is quite different than other
parts so it is natural to use distinct syntax. Note that Pop11
has convention that each construct has clearly visible terminator,
so there is well-defined moment to switch off syntax extention.

'external' is provided as part of Pop11, but could be defined
by user. ATM there are constructs that are considerd part of
"standard" Pop11, but which were originaly created as user
extentions.

Concerning "you don't want it changing in different parts of the
program", most extentions are intended to blend nicely into
whole language. In particular you want ability to combine
extentions. But some extentions naturally work on delimited
fragments of code and withing fragment (like 'external' above)
you may wich to use syntax which is quite different than rest
of language. Another situation is "legacy code". There may be
extention which is better than older way of doing things and
you may intend for new way to replace the old one. In particular
new way may be incompatible with old way. If you have old source
files that were not updated to new way, it is natural to switch
on old syntax in old files (or in old fragments of new files
if you want to cram a lot of things into single file), while
using new (incompatible) syntax in new code.

Anyway, AFAICS Lisp tradition is to stick to "standard" Lisp
syntax and define extentions mostly as macros (in principle
you could redefine Lisp syntax so that is accepts your
language as input, but such things seem to be relatively
infreqent). OTOH in Pop11 each extention usually has its
own syntax. Note that normal use case it to create extentions,
that is leave core language unchanged. But if you wish
you can redefine or elliminate part of core language
(in Lisp macros can not change core language, you need to
redefine syntax to effect such change).

> >> Do you have an example of such a language where implementing my
> >> 'strimport' is 10 lines of code? For example, C++. Or the one you
> >> mentioned. If so, how does it look?
> >
> > Your 'stringimage' can be done as Lisp macro:
> >
> > ;;; ---<cut here>-------------
> > (defun slurp_file (fn)
> > (with-open-file (f fn)
> > (let ((seq (make-string (file-length f))))
> > (read-sequence seq f)
> > (coerce seq 'string)))
> > )
> >
> > (defmacro string_include (f)
> > (slurp_file f))
> > ;;; ---<cut here>-------------
> >
> > Note: in source file use of 'string_include' looks like function
> > call (almost any Lisp construct looks like function call).
> > However, since it is a Lisp macro, it is executed at compile
> > time and result is handled like part of source.
>
> So this approach can also be used by C++'s constexpr feature? (Assuming
> there are ways to read files into strings.)

IIUC currently C++ constexpr has rather strong restrictions of what
can be computed and I think it can not read from files. But
assuming that such restrictions are lifted, then the approach
would work.

Note that Lisp macros can do more: they are not limited to
producind new constants, they can produce arbitrary code.

Similarly, syntax handler in Pop11 can produce arbitrary code.

> I notice you say 'compile-time'; does Lisp have a compilation stage with
> some intermediate or binary output? I thought it was run from source.

I know one "run from source" Lisp implementation (it actually compiles
to memory and you can dump memory and later restart program from
dumped memory image) but this is non-standard. Most Lisp implmentation
have some intermediate format for compiled files. Also, all
Lisp implmentations that I know can run from dumped memory image.
Most can create stanalone executables, but some insits that you
run loader program which loads previously dumped memory image
and runs from it.

> Actually I thought that that was the big deal about it, that you can
> manipulate Lisp code as though it was runtime data.

You can, as long as it source code. There are limitations of what
you can do with compiled code. If you want most (all???) Lisp
implementations will keep source together with binary code, so
you can examine, transform and maybe recompile any routine.
Due to binary format in use normally at any time you can replace
given top-level routine by different one and rest of code will use
new routine. I wrote normally, because you can _not_ replace
inlined routines in this way (you need to replace all users).

--
Waldek Hebisch