On 29/10/2022 18:42, James Harris wrote:

> As above, the language is meant to treat strings as arrays. So AISI it should
> not ascribe any particular meaning to their contents.

In that case you should have a look at Algol68, where string are arrays of
characters. Every array has an associated descriptor containing its bounds etc.
But more importantly a REF to an array is best implemented by including the
descriptor in the REF (being a strongly typed language, there is no necessity
for REFs to assorted other types to have the same length - they are not
necessarily just address values). This makes it easy to construct slices
(immutable) and REFs to slices (so the slice is mutable), thus providing many of
the features discussed in this thread. However there is no provision for
extending (or shortening) an array, other than to create a new space to copy it
into; one could provide library routines with smart features to avoid actual
copying in some cases, and with a friendly interface which did not expose the
messiness inside.

--
Charles H. Lindsey ---------At my New Home, still doing my own thing------
Tel: +44 161 488 1845 Web: https://www.clerew.man.ac.uk
Email: chl@clerew.man.ac.uk Snail-mail: Apt 40, SK8 5BF, U.K.
PGP: 2C15F1A9 Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

On 29/10/2022 22:25, Bart wrote:
> On 29/10/2022 15:01, James Harris wrote:
>> On 27/10/2022 12:14, Bart wrote:
>
>>> Most strings are fixed-length once created; strings that can grow are
>>> rare. You don't need a 'capacity' field for example (like C++'s
>>> Vector type).
>>
>> Having watched some videos on string storage recently I now think I
>> know what you mean by the capacity field - basically that a string
>> descriptor would consist of these fields:
>>
>>    start
>>    length
>>    capacity
>>
>> so that the string could be extended at the end (up to the capacity).
>> That may be a bit restrictive. A programmer might want to remove or
>> add characters at the beginning rather than just at the end, even
>> though such would be done less often.
>
> Doing a prepend is not a problem. What's critical is whether the new
> length is still within the current allocation. (Prepend requires
> shifting of the old string so is less efficient anyway.)

Well, see below.

>
> If a new allocation is needed, you may be copying data for both prepend
> and append.

Yes.

>
> With delete however, you may need to think about whether to /reduce/ the
> allocation size.

Agreed.

>
>> So what do you think of having a string descriptor more like
>>
>>    first
>>    past
>>    memfirst
>>    mempast
>>
>> where memfirst and mempast would define the allocated space in which
>> the string body would sit.
>
> What's the difference between 'first' and 'memfirst'?

memfirst would point to the start of the block in which the string body
existed. (first would point at the same address or a later address.)

> Would you have a
> string that doesn't start at the beginning of its allocated block?

Yes, that would be useful in some cases. For example, if deleting the
first part of a string one wouldn't want to be forced to copy the rest
of it down.

And there are cases where strings may be built by prepending. A classic
example is construction of a network frame. Each layer adds a header
which, naturally, has to go on the front.

--
James Harris

On 30/10/2022 14:20, Dmitry A. Kazakov wrote:
> On 2022-10-30 12:24, James Harris wrote:

...

>> I don't see where you see a multitude of representations of the null
>> string. AISI the empty string would simply have past equal to first in
>> all cases.
>
>    ...
>    (0..0)
>    (1..1)
>    (2..2)
>    ...
>    (n..n)
>    ...
>
> With pointers it becomes even worse as some of them might point to
> invalid addresses.

In the general case strings would live at arbitrary addresses so no
meaning could be inferred from any address. In all cases

past - first

would define the length of the string. If the length was zero then it
would be an empty string.

That said, if the string could be extended then both past and first
would have to point to allocated memory into which the extension could
take place.

...

>> What is your definition of a slice? Is it /part/ of an underlying
>> string or is it a /copy/ of part of a string? For example, if
>>
>>    string S = "abcde"
>>    slice T = S[1..3]  ;"bcd"
>>
>> then changes to T would do what to S?
>
> No idea. It depends. Is slice in your example an independent object?

A slice, at the moment, at least, would be a view of part of a string.
Extending the earlier example,

!_a_!_b_!_c_!_d_!

^ ^
! !
s_first s_past

The string in the example would be "abcd" and the slice, delimited by
s_first and s_past, would be the "bc" in the middle of the string. Note
that the contents of the slice would not include the element at s_past.

The slice would appear to be a string with constraints. By default its
contents could be updated in place but it could not be made longer or
shorter.

A callee which wanted only to read a string (a common case) or to update
a string in place should not have to care whether it was passed a string
or a slice. For such a case, strings and slices would be interchangeable.

>
> But considering this:
>
> declare
>    S : String := "abcde";
> begin
>    S (1..3) := "x"; -- Illegal in Ada

In Ada would the following be legal?

S (1..3) := "xxx"; --replacement same size as what it is replacing

I'd be happy with that.

>
> But should it be legal, then the result would be
>
>   "xde"
>
> Many implementations make this illegal because it would require either
> bounded or dynamically allocated unbounded string.
>
> You can consider make it legal for these, but then you would have
> different semantics of slices for different strings. And this would
> contradict the design principle of having all strings interchangeable
> regardless the implementation method.

I don't mind there being differences along the lines of 'constraints'
where a less-constrained object can be passed to a callee which expects
an object with such constraints or imposes more constraints, but not one
which needs fewer constraints.

...

>> I presume such constraints would be specified when objects are declared.
>
> Objects and/or subtypes. Depending on the language preferences. Note
> also that you can have constrained views of the same object. E.g. you
> have a mutable variable passed down as in-argument. That would be an
> immutable view of the same object.

Yes, and an immutable object could not be passed to a callee which
wanted a mutable object.

>
>> As a programmer how would you want to specify such constraints? Would
>> each have a reserved word, for example?
>
> In some cases constraints might be implied. But usually language have
> lots of [sub]type modifiers like
>
>    in, in out, out, constant
>    atomic, volatile, shared
>    aliased (can get pointers to)
>    external, static
>    public, private, protected (visibility constraints)
>    range, length, bounds
>    parameter AKA discriminant (general purpose constraint)
>    specific type AKA static/dynamic up/downcast (view as another type)
>    class-wide (view as a class of types rooted in this one)
>    ...
>    measurement unit

So you wouldn't have a keyword to indicate a constraint such as
"Non-sliding lower bound" which you mentioned before but IIUC you might
have some qualification of the 'bounds' keyword as in

bounds(^..)

to indicate an unchangeable lower bound (with ^ meaning the start of the
string)?

--
James Harris

On 29/10/2022 22:02, Bart wrote:
> On 29/10/2022 18:42, James Harris wrote:
>> On 29/10/2022 16:30, Bart wrote:

>> Further, functions which /return/ a string would create the string and
>> return it whole.
>
> Not necessarily. My dynamic language can return a string which is a
> slice into another. (Slices are not exposed in this language; they are
> in the static one, where slices are distinct types.)
>
> Example:
>
>     func trim(s) =
>         if s.len=2 then return "" fi
>         return s[2..$-1]
>     end
>
> This trims the first and last character of string. But here it returns a
> slice into the original string. If I wanted a fresh copy, I'd have to
> use copy() inside the function, or copy() (or a special kind of
> assignment) outside it.

That's a challenging example. In a sense it returns either of two
different types: the caller could be handed a string or a slice.

--
James Harris

On 30/10/2022 16:21, luserdroog wrote:
> On Monday, October 24, 2022 at 5:31:14 AM UTC-5, James Harris wrote:
>> Do you guys have any thoughts on the best ways for strings of characters
>> to be stored?
>>
>> 1. There's the C way, of course, of reserving one value (zero) and using
>> it as a terminator.
>>
>> 2. There's the 'length prefix' option of putting the length of the
>> string in a machine word before the characters.
>>
>> 3. There's the 'double pointer' way of pointing at, say, first and past
>> (where 'past' means first plus length such that the second pointer
>> points one position beyond the last character).
>>
>> Any others?

...

> I think an exhaustive list of options would be very large if you're not
> pre-judging and filtering as you're adding options.
>
> 4) [List|Array|Tuple|Iterator] of character objects

You mean where the characters are stored individually (one per node)?

>
> 5) Use 7 bits for data, 8th bit for terminator. Either ASCII7 or UTF-7
> can be used to format the data to squeeze it into 7 bits.

Interesting idea. It's certainly one I hadn't thought of.

>
> 6) Use UCS4 codes (24bit) padded out to 32 bits, and then you get a
> whole byte for metadata attached to each character.

That's definitely thinking outside the box. I can see it working if the
user (the programmer) wanted a string of 24-bit values but it could be
awkward in other cases such as if he wanted a string of 32-bit or 8-bit
values. I don't think I mentioned it but I'd like the programmer to be
able to choose what the elements of the string would be.

--
James Harris

On 2022-10-30 18:46, James Harris wrote:
> On 30/10/2022 14:20, Dmitry A. Kazakov wrote:
>> On 2022-10-30 12:24, James Harris wrote:
>
> ..
>
>>> I don't see where you see a multitude of representations of the null
>>> string. AISI the empty string would simply have past equal to first
>>> in all cases.
>>
>>     ...
>>     (0..0)
>>     (1..1)
>>     (2..2)
>>     ...
>>     (n..n)
>>     ...
>>
>> With pointers it becomes even worse as some of them might point to
>> invalid addresses.
>
> In the general case strings would live at arbitrary addresses so no
> meaning could be inferred from any address.

As I said, that is a problem which will preclude some implementations
and make others inefficient.

And in general using pointers is wasting space as strings are much shorter.

> In all cases
>
>   past - first
>
> would define the length of the string.

Not really. Usually negative length is considered equivalent to zero,
e.g. when iterating substrings. Other choices may consider iteration in
reverse, when bounds are inverted.

> If the length was zero then it would be an empty string.

Ergo, a special case to treat in many operations.

>> But considering this:
>>
>> declare
>>     S : String := "abcde";
>> begin
>>     S (1..3) := "x"; -- Illegal in Ada
>
> In Ada would the following be legal?

Yes, in Ada slice length is constrained as the string length is.

>   S (1..3) := "xxx";  --replacement same size as what it is replacing
>
> I'd be happy with that.

It is still not fully defined. You need to consider the issue of sliding
bounds. E.g.

S (2..4) (2) := 'x'; -- Assign a character

Now with sliding:

S (2..4) (2) := 'x' gives "abxde", x is second in the slice

without sliding

S (2..4) (2) := 'x' gives "axcde", x is at 2 in the original string

In Ada the right side slides, the left does not. Sliding the right side
allows doing logical things like:

S1 (1..5) := S1 (5..9); -- 5..9 slides to 1..5

>> But should it be legal, then the result would be
>>
>>    "xde"
>>
>> Many implementations make this illegal because it would require either
>> bounded or dynamically allocated unbounded string.
>>
>> You can consider make it legal for these, but then you would have
>> different semantics of slices for different strings. And this would
>> contradict the design principle of having all strings interchangeable
>> regardless the implementation method.
>
> I don't mind there being differences along the lines of 'constraints'
> where a less-constrained object can be passed to a callee which expects
> an object with such constraints or imposes more constraints, but not one
> which needs fewer constraints.

No, the problem is with semantics. E.g. let in a subprogram you do

S (1..100) := ""; -- Remove the first 100 characters

S is a formal parameter. Then, depending on the actual parameter's
subtype it may succeed (for a bounded length string) or fail (for a
fixed length string). Such things are big no-no in language design,
because they become a nightmare for programmers to track.

>>> I presume such constraints would be specified when objects are declared.
>>
>> Objects and/or subtypes. Depending on the language preferences. Note
>> also that you can have constrained views of the same object. E.g. you
>> have a mutable variable passed down as in-argument. That would be an
>> immutable view of the same object.
>
> Yes, and an immutable object could not be passed to a callee which
> wanted a mutable object.

Yes, lifting a constraint is not possible in most cases. However,
dynamic cast is a counterexample. You can move the view up the
inheritance tree. But this is frowned upon since it enforces certain
implementations.

>>> As a programmer how would you want to specify such constraints? Would
>>> each have a reserved word, for example?
>>
>> In some cases constraints might be implied. But usually language have
>> lots of [sub]type modifiers like
>>
>>     in, in out, out, constant
>>     atomic, volatile, shared
>>     aliased (can get pointers to)
>>     external, static
>>     public, private, protected (visibility constraints)
>>     range, length, bounds
>>     parameter AKA discriminant (general purpose constraint)
>>     specific type AKA static/dynamic up/downcast (view as another type)
>>     class-wide (view as a class of types rooted in this one)
>>     ...
>>     measurement unit
>
> So you wouldn't have a keyword to indicate a constraint such as
> "Non-sliding lower bound" which you mentioned before but IIUC you might
> have some qualification of the 'bounds' keyword as in
>
>   bounds(^..)
>
> to indicate an unchangeable lower bound (with ^ meaning the start of the
> string)?

I am not sure if sliding constraint might be usable. It is a different
issue to constraining bounds because it involves operations like
assignment. And it is not clear how to implement such a constraint
effectively. Most constraints are either static (compile time), or
simple to represent, like bounds or type tags. Sliding might be
implemented as a flag, but then you will have to check it all the time.
Maybe not worth having it as a choice. And it is unclear what is the
unconstrained state, sliding or non-sliding? (:-))

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

On 30/10/2022 17:52, James Harris wrote:
> On 29/10/2022 22:02, Bart wrote:
>> On 29/10/2022 18:42, James Harris wrote:
>>> On 29/10/2022 16:30, Bart wrote:
>
>>> Further, functions which /return/ a string would create the string
>>> and return it whole.
>>
>> Not necessarily. My dynamic language can return a string which is a
>> slice into another. (Slices are not exposed in this language; they are
>> in the static one, where slices are distinct types.)
>>
>> Example:
>>
>>      func trim(s) =
>>          if s.len=2 then return "" fi
>>          return s[2..$-1]
>>      end
>>
>> This trims the first and last character of string. But here it returns
>> a slice into the original string. If I wanted a fresh copy, I'd have
>> to use copy() inside the function, or copy() (or a special kind of
>> assignment) outside it.
>
> That's a challenging example. In a sense it returns either of two
> different types: the caller could be handed a string or a slice.
>

In this language, it only has a String type, not a Slice. Slicing is an
operation you apply on strings to yield another String object.

(Internally, it has to distinguish between owned strings and slices into
strings owned by other objects, but as I said that aspect is not exposed.)

On Sunday, October 30, 2022 at 1:13:44 PM UTC-5, James Harris wrote:
> On 30/10/2022 16:21, luserdroog wrote:
> > On Monday, October 24, 2022 at 5:31:14 AM UTC-5, James Harris wrote:
> >> Do you guys have any thoughts on the best ways for strings of characters
> >> to be stored?
> >>
> >> 1. There's the C way, of course, of reserving one value (zero) and using
> >> it as a terminator.
> >>
> >> 2. There's the 'length prefix' option of putting the length of the
> >> string in a machine word before the characters.
> >>
> >> 3. There's the 'double pointer' way of pointing at, say, first and past
> >> (where 'past' means first plus length such that the second pointer
> >> points one position beyond the last character).
> >>
> >> Any others?
> ..
> > I think an exhaustive list of options would be very large if you're not
> > pre-judging and filtering as you're adding options.
> >
> > 4) [List|Array|Tuple|Iterator] of character objects
> You mean where the characters are stored individually (one per node)?

Yep. Fat characters, or whatever other scaffolding helps for the operations
you want to support.

> > 5) Use 7 bits for data, 8th bit for terminator. Either ASCII7 or UTF-7
> > can be used to format the data to squeeze it into 7 bits.
> Interesting idea. It's certainly one I hadn't thought of.

It's used in the Classical Forth Dictionary header for the name field which
can be variable length. Often it's followed by a length byte and you store
the pointer to the length byte, using

(length - *length)

to actually get the pointer to the start.

> > 6) Use UCS4 codes (24bit) padded out to 32 bits, and then you get a
> > whole byte for metadata attached to each character.
> That's definitely thinking outside the box. I can see it working if the
> user (the programmer) wanted a string of 24-bit values but it could be
> awkward in other cases such as if he wanted a string of 32-bit or 8-bit
> values. I don't think I mentioned it but I'd like the programmer to be
> able to choose what the elements of the string would be.

This is what I was using in my unfinished APL-like language. The 8 bits
of tag meant it was easy for a node to be a character or an integer (25 bit)
or a nested subarray or whatever ... mpfp number... symbol table.

So I didn't really need a string type *per se* because there's an array type
whose data elements are these 32bits of encoded whatevs. A string is
just a 1D array, or maybe an array of arrays or a 2D array padded out with
spaces on each line. You'd read it in or receive it initially as a 1D array probably
from a file. Oh, an element could also be a file.

On 30/10/2022 19:13, James Harris wrote:
> On 30/10/2022 16:21, luserdroog wrote:
>> On Monday, October 24, 2022 at 5:31:14 AM UTC-5, James Harris wrote:
>>> Do you guys have any thoughts on the best ways for strings of characters
>>> to be stored?
>>>
>>> 1. There's the C way, of course, of reserving one value (zero) and using
>>> it as a terminator.
>>>
>>> 2. There's the 'length prefix' option of putting the length of the
>>> string in a machine word before the characters.
>>>
>>> 3. There's the 'double pointer' way of pointing at, say, first and past
>>> (where 'past' means first plus length such that the second pointer
>>> points one position beyond the last character).
>>>
>>> Any others?
>
> ..
>
>> I think an exhaustive list of options would be very large if you're not
>> pre-judging and filtering as you're adding options.
>>
>> 4) [List|Array|Tuple|Iterator] of character objects
>
> You mean where the characters are stored individually (one per node)?
>
>>
>> 5) Use 7 bits for data, 8th bit for terminator. Either ASCII7 or UTF-7
>> can be used to format the data to squeeze it into 7 bits.
>
> Interesting idea. It's certainly one I hadn't thought of.

Nor should you - that is a crazy idea. It is massively inefficient, as
well as being inconsistent with everything else.

>
>>
>> 6) Use UCS4 codes (24bit) padded out to 32 bits, and then you get a
>> whole byte for metadata attached to each character.
>
> That's definitely thinking outside the box. I can see it working if the
> user (the programmer) wanted a string of 24-bit values but it could be
> awkward in other cases such as if he wanted a string of 32-bit or 8-bit
> values. I don't think I mentioned it but I'd like the programmer to be
> able to choose what the elements of the string would be.
>

UCS4 is 31 bit, not 24 bit. Perhaps luserdroog is mixing it up with
UTF-32, which can be covered by 21 bits. (The extra 10 bits in UCS4
have never been anything but 0, but if you want to refer to a long
out-dated and obsolete standard, it should still be done so accurately.)

Do not make a new string or character storage system based around
anything obsolete - that includes UTF-7 and UCS4. Making lots of extra
work for yourself to support something that everyone else rejected as
complicated, unnecessary and unused decades ago, is just silly.

There are only two character encodings that make any kind of sense in a
modern language (i.e., anything from this century). UTF-8 and
/possibly/ UTF-32 for internal usage, if you find it more convenient for
the operations you want.

If you are using UTF-32 only for internal usage within the language, and
not for export (external function calls, file IO, etc.), then the high
byte is always unused - and therefore free for metadata if that's what
you want. I'm not convinced it is a good idea, but it's certainly possible.

For any kind of interaction with anything else, UTF-8 is the standard.
There are a few other encodings that haven't died off completely yet,
but they are all on the way out.

I would also recommend treating characters and character strings as
something very different from raw bytes and binary blobs. Users want to
do very different things with them, and many of the useful operations
are completely different. Some languages have made the mistake of
conflating the two concepts - it's difficult to fix once that design
flaw is set into a language.

On 31/10/2022 16:58, David Brown wrote:
> On 30/10/2022 19:13, James Harris wrote:
>> On 30/10/2022 16:21, luserdroog wrote:
>>> On Monday, October 24, 2022 at 5:31:14 AM UTC-5, James Harris wrote:
>>>> Do you guys have any thoughts on the best ways for strings of
>>>> characters
>>>> to be stored?
>>>>
>>>> 1. There's the C way, of course, of reserving one value (zero) and
>>>> using
>>>> it as a terminator.
>>>>
>>>> 2. There's the 'length prefix' option of putting the length of the
>>>> string in a machine word before the characters.
>>>>
>>>> 3. There's the 'double pointer' way of pointing at, say, first and past
>>>> (where 'past' means first plus length such that the second pointer
>>>> points one position beyond the last character).
>>>>
>>>> Any others?
>>
>> ..
>>
>>> I think an exhaustive list of options would be very large if you're not
>>> pre-judging and filtering as you're adding options.
>>>
>>> 4) [List|Array|Tuple|Iterator] of character objects
>>
>> You mean where the characters are stored individually (one per node)?
>>
>>>
>>> 5) Use 7 bits for data, 8th bit for terminator. Either ASCII7 or UTF-7
>>> can be used to format the data to squeeze it into 7 bits.
>>
>> Interesting idea. It's certainly one I hadn't thought of.
>
> Nor should you - that is a crazy idea.  It is massively inefficient, as
> well as being inconsistent with everything else.

It's a perfectly fine idea - for the 1970s.

(Now the 8th bit is better put to use to represent UTF8.)

On 31/10/2022 18:31, Bart wrote:
> On 31/10/2022 16:58, David Brown wrote:
>> On 30/10/2022 19:13, James Harris wrote:
>>> On 30/10/2022 16:21, luserdroog wrote:
>>>> On Monday, October 24, 2022 at 5:31:14 AM UTC-5, James Harris wrote:
>>>>> Do you guys have any thoughts on the best ways for strings of
>>>>> characters
>>>>> to be stored?
>>>>>
>>>>> 1. There's the C way, of course, of reserving one value (zero) and
>>>>> using
>>>>> it as a terminator.
>>>>>
>>>>> 2. There's the 'length prefix' option of putting the length of the
>>>>> string in a machine word before the characters.
>>>>>
>>>>> 3. There's the 'double pointer' way of pointing at, say, first and
>>>>> past
>>>>> (where 'past' means first plus length such that the second pointer
>>>>> points one position beyond the last character).
>>>>>
>>>>> Any others?
>>>
>>> ..
>>>
>>>> I think an exhaustive list of options would be very large if you're not
>>>> pre-judging and filtering as you're adding options.
>>>>
>>>> 4) [List|Array|Tuple|Iterator] of character objects
>>>
>>> You mean where the characters are stored individually (one per node)?
>>>
>>>>
>>>> 5) Use 7 bits for data, 8th bit for terminator. Either ASCII7 or UTF-7
>>>> can be used to format the data to squeeze it into 7 bits.
>>>
>>> Interesting idea. It's certainly one I hadn't thought of.
>>
>> Nor should you - that is a crazy idea.  It is massively inefficient,
>> as well as being inconsistent with everything else.
>
> It's a perfectly fine idea - for the 1970s.
>
> (Now the 8th bit is better put to use to represent UTF8.)

Indeed.

UTF-7 was invented in attempt to make an encoding for Unicode that would
work for email, since some email servers did not handle 8-bit characters
correctly, long ago in the dark ages. It was never formally a Unicode
encoding, and almost never used in practice.

Using 7-bit characters and the eighth bit for a terminator would be
pretty inefficient - 12.5% wasted space to get a single bit of useful
information per string. Not a good trade-off!

On 31/10/2022 16:58, David Brown wrote:
> On 30/10/2022 19:13, James Harris wrote:
>> On 30/10/2022 16:21, luserdroog wrote:
>>> On Monday, October 24, 2022 at 5:31:14 AM UTC-5, James Harris wrote:

>>>> Do you guys have any thoughts on the best ways for strings of
>>>> characters
>>>> to be stored?

...

>>> 5) Use 7 bits for data, 8th bit for terminator. Either ASCII7 or UTF-7
>>> can be used to format the data to squeeze it into 7 bits.
>>
>> Interesting idea. It's certainly one I hadn't thought of.
>
> Nor should you - that is a crazy idea.  It is massively inefficient, as
> well as being inconsistent with everything else.

The model I have chosen (at least, for now) is to have a string indexed
logically from zero (so indices do not need to be stored) and, for
implementation, delimited by two pointers.

The one downside I am aware of is that it will, at times, require
creation and destruction of a small descriptor. I'll have to see how the
approach works out in practice.

...

> I would also recommend treating characters and character strings as
> something very different from raw bytes and binary blobs.  Users want to
> do very different things with them, and many of the useful operations
> are completely different.  Some languages have made the mistake of
> conflating the two concepts - it's difficult to fix once that design
> flaw is set into a language.

That sounds interesting but I cannot tell what you have in mind.

One could consider strings as having two categories of operation: those
which involve only the memory used by strings such as allocation,
concatenation, insertion, deletion, etc; and those which care about the
contents of a string such as capitalisation, comparison, whitespace
recognition, parsing, etc. Why could the mechanics not apply to raw
bytes and blobs?

--
James Harris

On 30/10/2022 18:28, Dmitry A. Kazakov wrote:
> On 2022-10-30 18:46, James Harris wrote:

...

>> In Ada would the following be legal?
>
> Yes, in Ada slice length is constrained as the string length is.
>
>>    S (1..3) := "xxx";  --replacement same size as what it is replacing
>>
>> I'd be happy with that.
>
> It is still not fully defined. You need to consider the issue of sliding
> bounds. E.g.
>
>   S (2..4) (2) := 'x'; -- Assign a character
>
> Now with sliding:
>
>   S (2..4) (2) := 'x'  gives "abxde", x is second in the slice
>
> without sliding
>
>   S (2..4) (2) := 'x'  gives "axcde", x is at 2 in the original string

AISI the elements of strings and slices would always be accessed by
offset. That appears to be the 'sliding' model you mention.

>
> In Ada the right side slides, the left does not. Sliding the right side
> allows doing logical things like:
>
>   S1 (1..5) := S1 (5..9); -- 5..9 slides to 1..5

I don't see any sliding there, only that chars 5 to 9 are copied over
chars 1 to 5 of the same string.

...

>
> I am not sure if sliding constraint might be usable. It is a different
> issue to constraining bounds because it involves operations like
> assignment. And it is not clear how to implement such a constraint
> effectively. Most constraints are either static (compile time), or
> simple to represent, like bounds or type tags. Sliding might be
> implemented as a flag, but then you will have to check it all the time.
> Maybe not worth having it as a choice. And it is unclear what is the
> unconstrained state, sliding or non-sliding? (:-))
>

Always sliding (if I understand the term)!

--
James Harris

On 30/10/2022 22:37, Bart wrote:
> On 30/10/2022 17:52, James Harris wrote:
>> On 29/10/2022 22:02, Bart wrote:
>>> On 29/10/2022 18:42, James Harris wrote:
>>>> On 29/10/2022 16:30, Bart wrote:
>>
>>>> Further, functions which /return/ a string would create the string
>>>> and return it whole.
>>>
>>> Not necessarily. My dynamic language can return a string which is a
>>> slice into another. (Slices are not exposed in this language; they
>>> are in the static one, where slices are distinct types.)
>>>
>>> Example:
>>>
>>>      func trim(s) =
>>>          if s.len=2 then return "" fi
>>>          return s[2..$-1]
>>>      end
>>>
>>> This trims the first and last character of string. But here it
>>> returns a slice into the original string. If I wanted a fresh copy,
>>> I'd have to use copy() inside the function, or copy() (or a special
>>> kind of assignment) outside it.
>>
>> That's a challenging example. In a sense it returns either of two
>> different types: the caller could be handed a string or a slice.
>>
>
> In this language, it only has a String type, not a Slice. Slicing is an
> operation you apply on strings to yield another String object.
>
> (Internally, it has to distinguish between owned strings and slices into
> strings owned by other objects, but as I said that aspect is not exposed.)

As most uses of slices would be read-only but some would be read-write,
and there are various potential ways to implement a slice, it might be
sensible for me to do something like:

1. Have the default slice as read-only and the simplest to construct. If
s is a string or another slice then a slice of part of that string could
be constructed as

s(1..4)

2. Use keywords to effect other, more specialised, operations. For example,

s.slice_cow(1..4) ;a copy-on-write slice
s.slice_rw(1..4) ;a slice via which the base string can be changed
s.copy(1..4) ;copy the designated chars to a new string

--
James Harris

On 03/11/2022 16:48, James Harris wrote:
> On 31/10/2022 16:58, David Brown wrote:
>> On 30/10/2022 19:13, James Harris wrote:
>>> On 30/10/2022 16:21, luserdroog wrote:
>>>> On Monday, October 24, 2022 at 5:31:14 AM UTC-5, James Harris wrote:
>
>>>>> Do you guys have any thoughts on the best ways for strings of
>>>>> characters
>>>>> to be stored?
>
> ..
>
>>>> 5) Use 7 bits for data, 8th bit for terminator. Either ASCII7 or UTF-7
>>>> can be used to format the data to squeeze it into 7 bits.
>>>
>>> Interesting idea. It's certainly one I hadn't thought of.
>>
>> Nor should you - that is a crazy idea.  It is massively inefficient,
>> as well as being inconsistent with everything else.
>
> The model I have chosen (at least, for now) is to have a string indexed
> logically from zero (so indices do not need to be stored) and, for
> implementation, delimited by two pointers.
>
> The one downside I am aware of is that it will, at times, require
> creation and destruction of a small descriptor. I'll have to see how the
> approach works out in practice.

There is no universally ideal way to store a string - every method is
inefficient for some uses and operations. You just have to find
something that is good enough for typical uses in your language, add
support for connecting to external code (such as exporting C-style
strings), and make it possible for users to implement other types when
it makes more sense for the user code.

>
> ..
>
>> I would also recommend treating characters and character strings as
>> something very different from raw bytes and binary blobs.  Users want
>> to do very different things with them, and many of the useful
>> operations are completely different.  Some languages have made the
>> mistake of conflating the two concepts - it's difficult to fix once
>> that design flaw is set into a language.
>
> That sounds interesting but I cannot tell what you have in mind.
>

I mean you should consider a "string" to be a way of holding a sequence
of "character" units which can hold a code unit of UTF-8 (since any
other choice of character encoding is madness).

This should be different from a "byte", which would be an 8-bit unit of
raw memory (ignore the existence of machines that can't address 8-bit
memory units directly - they will never use your language). Arrays of
this type should always be contiguous, and used for raw memory access.
(You may also want larger types - memory16, memory32, memory64, etc., if
that is convenient for efficient usage.)

So when you read a block of data from a file, or send a block to a
network socket, it is an array of bytes - not a string. (You can have
high-level abstractions for a "text file" wrapper that can read and
write strings, but that's not fundamental.) If you have an equivalent
of C's "memcpy" function it should use bytes, not any kind of character
type. If you have something like C's "type-based aliasing rules", then
it is bytes that should have the special exception, not characters.

Neither "byte" nor "character" should have any kind of arithmetic
operators - they are not integers. But you will need cast or conversion
operations on them.

The concept of "signed char" and "unsigned char" in C is a serious
design flaw. A type designed to hold letters should not have a sign,
and should not be used to hold arbitrary raw, low-level data.

You might also consider not having a character type at all. Python 3
has no character types - "a" is a string, not a character.

> One could consider strings as having two categories of operation: those
> which involve only the memory used by strings such as allocation,
> concatenation, insertion, deletion, etc; and those which care about the
> contents of a string such as capitalisation, comparison, whitespace
> recognition, parsing, etc. Why could the mechanics not apply to raw
> bytes and blobs?
>

Operations on strings are those that are relevant to strings. It makes
no sense to capitalise a raw binary blob. It makes no sense to have
methods for linking or chaining sets of blobs - these are direct handles
into memory, and chaining, allocation, etc., are higher level operations.

Raw binary buffers require nothing more than an address and a size to
describe them - anything more, and it is too high level. (Again,
there's nothing wrong with providing higher level features and
interfaces, but they have to build on the fundamental ones.)

On 04/11/2022 13:58, David Brown wrote:

> Neither "byte" nor "character" should have any kind of arithmetic
> operators - they are not integers.  But you will need cast or conversion
> operations on them.

Bytes are small integers, typically of u8 type.

I can't see why arithmetic can't be done with them, unless you want a
purer kind of language where arithmetic is only allowed on signed
numbers, and bitwise ops only on unsigned numbers, which is usually
going to be a pain for all concerned.

> The concept of "signed char" and "unsigned char" in C is a serious
> design flaw.  A type designed to hold letters should not have a sign,
> and should not be used to hold arbitrary raw, low-level data.

Signed and unsigned chars are not so bad; presumably C intended these to
do the job of a 'byte' type for small integers. So it was just a poor
choice of name. (After all there is no separate type in C for bytes
holding character data.)

What's bad is that third kind: a 'plain char' type, which is
incompatible with both signed and unsigned char, even though it
necessarily needs to be one of the other on a specific platform. It
occurs in no other language, and causes problems within FFI APIs.

On 04/11/2022 17:50, Bart wrote:
> On 04/11/2022 13:58, David Brown wrote:
>
>> Neither "byte" nor "character" should have any kind of arithmetic
>> operators - they are not integers.  But you will need cast or
>> conversion operations on them.
>
> Bytes are small integers, typically of u8 type.
>
> I can't see why arithmetic can't be done with them, unless you want a
> purer kind of language where arithmetic is only allowed on signed
> numbers, and bitwise ops only on unsigned numbers, which is usually
> going to be a pain for all concerned.

I think what David means is that arithmetic operations don't apply to
characters (even though some languages permit such operations). For
example, neither

'a' * 5

nor even

'R' + 1

have any meaning over the set of characters. Prohibiting arithmetic on
them could be dome but would make classifying and manipulating
characters difficult unless one had a comprehensive set of library
functions such as

is_digit(char)
is_alphanum(locale, char)
is_lower(locale, char)
upper(locale, char)

and many more.

--
James Harris

On 04/11/2022 21:35, James Harris wrote:
> On 04/11/2022 17:50, Bart wrote:
>> On 04/11/2022 13:58, David Brown wrote:
>>
>>> Neither "byte" nor "character" should have any kind of arithmetic
>>> operators - they are not integers.  But you will need cast or
>>> conversion operations on them.
>>
>> Bytes are small integers, typically of u8 type.
>>
>> I can't see why arithmetic can't be done with them, unless you want a
>> purer kind of language where arithmetic is only allowed on signed
>> numbers, and bitwise ops only on unsigned numbers, which is usually
>> going to be a pain for all concerned.
>
> I think what David means is that arithmetic operations don't apply to
> characters

I was picking on the 'byte' type; it seems extraordinary that you
shouldn't be allowed to do arithmetic with them. If you can initialise a
byte value with a number like this:

byte a = 123

then it's a number!

> (even though some languages permit such operations). For
> example, neither
>
>   'a' * 5
>
> nor even
>
>   'R' + 1
>
> have any meaning over the set of characters.

I actually had such a restriction for a while: char*5 wasn't allowed,
but char+1 was. After all why on earth shouldn't you want the next
character in that alphabet? Why should code like this be made illegal:

a := a * 10 + (c - '0')

Then I realised I shouldn't be telling the programmer what they can and
can't do with characters, as there might be some perfectly valid
use-case that I simply hadn't thought of.

Maybe 'a' * 5 yields the value 'aaaaa' or the string "aaaaa", or this is
some kind on encryption algorithm.

So now they are treated like integers, other than printing an array of
char or pointer to char assumes they are strings.

> Prohibiting arithmetic on
> them could be dome but would make classifying and manipulating
> characters difficult unless one had a comprehensive set of library
> functions such as
>
>   is_digit(char)
>   is_alphanum(locale, char)
>   is_lower(locale, char)
>   upper(locale, char)
>
> and many more.

As I said, you and I don't know all the possibilites. Of course there
would need to be conversions between char and int, but this can become a
nuisance.

On 04/11/2022 22:28, Bart wrote:
> On 04/11/2022 21:35, James Harris wrote:
>> On 04/11/2022 17:50, Bart wrote:
>>> On 04/11/2022 13:58, David Brown wrote:
>>>
>>>> Neither "byte" nor "character" should have any kind of arithmetic
>>>> operators - they are not integers.  But you will need cast or
>>>> conversion operations on them.
>>>
>>> Bytes are small integers, typically of u8 type.
>>>
>>> I can't see why arithmetic can't be done with them, unless you want a
>>> purer kind of language where arithmetic is only allowed on signed
>>> numbers, and bitwise ops only on unsigned numbers, which is usually
>>> going to be a pain for all concerned.
>>
>> I think what David means is that arithmetic operations don't apply to
>> characters
>
> I was picking on the 'byte' type; it seems extraordinary that you
> shouldn't be allowed to do arithmetic with them. If you can initialise a
> byte value with a number like this:
>
>     byte a = 123
>
> then it's a number!
>
>> (even though some languages permit such operations). For example, neither
>>
>>    'a' * 5
>>
>> nor even
>>
>>    'R' + 1
>>
>> have any meaning over the set of characters.
>
> I actually had such a restriction for a while: char*5 wasn't allowed,
> but char+1 was. After all why on earth shouldn't you want the next
> character in that alphabet?

That's because 'R' + 1 may not be the next character in all alphabets.
Defining 'next' is more than difficult. It depends on intended collation
order which varies in different parts of the world and can even change
over time as authorities choose different collation orders. Some
plausible meanings of 'R' + 1:

'S' (as in ASCII)
'r' (what the user may want as sort order)
's' (what the user may want as sort order)
a non-character (as in EBCDIC)

Perhaps a pseudo-call would be better such as

char_plus(collation, 'R', 1)

where 'collation' would be used to determine what was the specified
number of characters away from 'R'.

> Why should code like this be made illegal:
>
>     a := a * 10 + (c - '0')

I'm not saying it should. I am in two minds about what's best. The
alternative is something like

a := a * 10 + digit_value(c)

>
> Then I realised I shouldn't be telling the programmer what they can and
> can't do with characters, as there might be some perfectly valid
> use-case that I simply hadn't thought of.

Agreed. Although the point of prohibiting arithmetic on characters is to
make multilingual programming easier, not harder.

...

>> Prohibiting arithmetic on them could be dome but would make
>> classifying and manipulating characters difficult unless one had a
>> comprehensive set of library functions such as
>>
>>    is_digit(char)
>>    is_alphanum(locale, char)
>>    is_lower(locale, char)
>>    upper(locale, char)
>>
>> and many more.
>
> As I said, you and I don't know all the possibilites.

Yes, the challenge of making multilingual programming easier is
providing a comprehensive and convenient set of conversions.

> Of course there
> would need to be conversions between char and int, but this can become a
> nuisance.

Aside from converting digits (and any other characters used as digits in
a higher number base) is there's any meaning to converting chars to/from
ints?

--
James Harris

On 04/11/2022 13:58, David Brown wrote:
> On 03/11/2022 16:48, James Harris wrote:
>> On 31/10/2022 16:58, David Brown wrote:

...

>>> I would also recommend treating characters and character strings as
>>> something very different from raw bytes and binary blobs.  Users want
>>> to do very different things with them, and many of the useful
>>> operations are completely different.  Some languages have made the
>>> mistake of conflating the two concepts - it's difficult to fix once
>>> that design flaw is set into a language.
>>
>> That sounds interesting but I cannot tell what you have in mind.
>>
>
> I mean you should consider a "string" to be a way of holding a sequence
> of "character" units which can hold a code unit of UTF-8 (since any
> other choice of character encoding is madness).

We've discussed before that (IMO) Unicode is useful for physical
printing to paper or electronic rendering such as to PDF but that it's a
nightmare for programmers and users when it is used for any kind of
input so I won't go over that again except to say that AISI Unicode
should be handled by library functions rather than a language.

What I do have in mind is strings of 'containers' where a string might
be declared as of type

string of char8 -- meaning a string of char8 containers
string of char32 -- meaning a string of char32 containers

What goes in each 8-bit or 32-bit 'container' would be another matter.

That agnostic ideal is somewhat in tension with the desire to include
string literals in a program text. For that, as I've mentioned before,
my preference is to have the program text and any literals within it
written in ASCII and American English; supplementary files would express
the string literals in other languages.

For example,

print "Hello world"

would be accompanied by a file for French which included

"Hello world" --> "Bonjour le monde"

Naturally, multilingual programming is much more complex than that
simple example but it shows the basic idea. The compiler would be able
to check that language files had everything required for a given piece
of source code.

...

> Neither "byte" nor "character" should have any kind of arithmetic
> operators - they are not integers.  But you will need cast or conversion
> operations on them.

The char8 and char32 containers could omit support for arithmetic **if**
enough support routines were provided. But, as Bart says, it's difficult
to anticipate all such support routines that programmers might need.

>
> The concept of "signed char" and "unsigned char" in C is a serious
> design flaw.  A type designed to hold letters should not have a sign,
> and should not be used to hold arbitrary raw, low-level data.

OK.

> You might also consider not having a character type at all.  Python 3
> has no character types - "a" is a string, not a character.

I can see that as being possible. I keep coming across examples of where
a 1-character string would do just as well as a character. ATM, though,
I have a separate character type.

...

> Raw binary buffers require nothing more than an address and a size to
> describe them - anything more, and it is too high level.  (Again,
> there's nothing wrong with providing higher level features and
> interfaces, but they have to build on the fundamental ones.)

Maybe that's where I am going with this. What you describe does sound
rather like my idea of using chars as containers and putting char and
string handling in libraries.

Incidentally, AISI I could have strings of any data type, not just
chars. For example,

string of int16
string of struct {x: float, y: float}
string of function(int, bool) -> uint

I'll have to see how it works out in practice but the idea is to
separate the concept of a string (basically, storage layout) from the
concept of whatever type of element the string is made from.

--
James Harris

On 05/11/2022 10:36, James Harris wrote:
> On 04/11/2022 22:28, Bart wrote:

>> I actually had such a restriction for a while: char*5 wasn't allowed,
>> but char+1 was. After all why on earth shouldn't you want the next
>> character in that alphabet?
>
> That's because 'R' + 1 may not be the next character in all alphabets.
> Defining 'next' is more than difficult. It depends on intended collation
> order which varies in different parts of the world and can even change
> over time as authorities choose different collation orders. Some
> plausible meanings of 'R' + 1:
>
>   'S' (as in ASCII)

You said elsewhere that you want to use ASCII within programs. Which is
it happens, corresponds to the first 128 points of Unicode. Here:

char c
for c in 'A'..'Z' do
print c
od

this displays 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'. The for-loop works by adding
+1 to 'c'; it doesn't care about collating order!

(This also illustrates the difference between `byte` and `char`; using a
byte type, the output would be '656667...90'.)

>   'r' (what the user may want as sort order)
>   's' (what the user may want as sort order)
>   a non-character (as in EBCDIC)

My feeling is that it is these diverse requirements that require
user-supplied functions.

My 'char' is still a thinly veiled numeric type, so ordinary integer
arithmatic can be used. Otherwise even something like this becomes
impossible:

['A'..'Z']int histogram

midpoint := (histogram.upb - histogram.lwb)/2

++histogram[midpoint+1]

This requires certain properties of array indicates, like being able to
do arithmetic, as well as being consecutive ordinal values.

> Perhaps a pseudo-call would be better such as
>
>   char_plus(collation, 'R', 1)
>
> where 'collation' would be used to determine what was the specified
> number of characters away from 'R'.

Sure, as I said, you can provide any interpretation you like. But if you
do C+1, you expect to get the code of the next character (or next
codepoint if venturing outside ASCII).

>
> Aside from converting digits (and any other characters used as digits in
> a higher number base) is there's any meaning to converting chars to/from
> ints?

My static language makes byte and char slightly different types. (Types
involving char may get printed differently.)

That meant that `ref byte` and `ref char` were incompatible, which
rapidly turned into a nightmare: I might have a readfile() routine that
returned a `ref byte` type, a pointer to a block of memory.

But I wanted to interpret that block as `ref char` - a string. So this
meant loads of casts to either `ref byte` or `ref char` to get things to
work, but it got too much (a bit like 'const poisoning' in C where it
just propagates everywhere). That was clearly the wrong approach.

In the end I relaxed the type rules so that `ref byte` and `ref char`
are compatible, and everything is now SO much simpler.

On 05/11/2022 11:14, James Harris wrote:
> On 04/11/2022 13:58, David Brown wrote:

>> I mean you should consider a "string" to be a way of holding a
>> sequence of "character" units which can hold a code unit of UTF-8
>> (since any other choice of character encoding is madness).
>
> We've discussed before that (IMO) Unicode is useful for physical
> printing to paper or electronic rendering such as to PDF but that it's a
> nightmare for programmers and users when it is used for any kind of
> input so I won't go over that again except to say that AISI Unicode
> should be handled by library functions rather than a language.
>
> What I do have in mind is strings of 'containers' where a string might
> be declared as of type
>
>   string of char8      -- meaning a string of char8 containers
>   string of char32     -- meaning a string of char32 containers
>
> What goes in each 8-bit or 32-bit 'container' would be another matter.
>
> That agnostic ideal is somewhat in tension with the desire to include
> string literals in a program text. For that, as I've mentioned before,
> my preference is to have the program text and any literals within it
> written in ASCII and American English; supplementary files would express
> the string literals in other languages.
>
> For example,
>
>   print "Hello world"
>
> would be accompanied by a file for French which included
>
>   "Hello world" --> "Bonjour le monde"
>
> Naturally, multilingual programming is much more complex than that
> simple example but it shows the basic idea. The compiler would be able
> to check that language files had everything required for a given piece
> of source code.

Is it? This pretty much all I did when I used to write internationalised
applications. Although that was only done for French, German and Dutch.

But that print example would be written like this:

print /"Hello World"

The "/" was a translation operator, so only certain strings were
translated. This also made it easy to scan source code to build a list
of messages, used to maintain the dictionary as entries were added,
deleted or modified.

The scheme did need some hints sometimes, written like this, to get
around ambiguities:

print /"Green!colour"
print /"Green!fresh"

The hint was usually filtered out.

But this is little to do with how strings are represented. Even in
English, messages may include characters like "£" (pound sign) which is
not part of ASCII.

So a way to represent Unicode within literals is still needed (didn't we
discuss this a couple of years ago?).

On 05/11/2022 13:38, Bart wrote:
> On 05/11/2022 10:36, James Harris wrote:
>> On 04/11/2022 22:28, Bart wrote:
>
>>> I actually had such a restriction for a while: char*5 wasn't allowed,
>>> but char+1 was. After all why on earth shouldn't you want the next
>>> character in that alphabet?
>>
>> That's because 'R' + 1 may not be the next character in all alphabets.
>> Defining 'next' is more than difficult. It depends on intended
>> collation order which varies in different parts of the world and can
>> even change over time as authorities choose different collation
>> orders. Some plausible meanings of 'R' + 1:
>>
>>    'S' (as in ASCII)
>
> You said elsewhere that you want to use ASCII within programs. Which is
> it happens, corresponds to the first 128 points of Unicode. Here:
>
>     char c
>     for c in 'A'..'Z' do
>         print c
>     od
>
> this displays 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'. The for-loop works by adding
> +1 to 'c'; it doesn't care about collating order!

That piece of code is fine for an English-speaking user but to a
Spaniard the alphabet has a character missing, and Greeks wouldn't agree
with it at all.

Where L is the locale why not allow something like

for c in L.alpha_first..L.alpha_last
print c
od

?

That should work in English or Spanish or Greek etc, shouldn't it?

>
> (This also illustrates the difference between `byte` and `char`; using a
> byte type, the output would be '656667...90'.)
>
>
>
>>    'r' (what the user may want as sort order)
>>    's' (what the user may want as sort order)
>>    a non-character (as in EBCDIC)
>
> My feeling is that it is these diverse requirements that require
> user-supplied functions.

Functions, yes, (or what appear to be functions) though surely they
should be part of a library that comes with the language.

>
> My 'char' is still a thinly veiled numeric type, so ordinary integer
> arithmatic can be used. Otherwise even something like this becomes
> impossible:
>
>     ['A'..'Z']int histogram
>
>     midpoint := (histogram.upb - histogram.lwb)/2
>
>     ++histogram[midpoint+1]
>
> This requires certain properties of array indicates, like being able to
> do arithmetic, as well as being consecutive ordinal values.

Why not

[L.alpha_first..L.alpha_last] int histogram

?

As for the calculations what about using L.ord and L.chr to convert
between chars and integers?

>
>> Perhaps a pseudo-call would be better such as
>>
>>    char_plus(collation, 'R', 1)
>>
>> where 'collation' would be used to determine what was the specified
>> number of characters away from 'R'.
>
> Sure, as I said, you can provide any interpretation you like. But if you
> do C+1, you expect to get the code of the next character (or next
> codepoint if venturing outside ASCII).

If you use codepoints then you might not get the next character in
sequence - as in the case of 'R' in ebcdic (you'd get a non-printing
character) or 'N' in Spanish (you'd get 'O' rather than the N with a hat
that a Spaniard would expect).

If the programmer wants "the next character in the alphabet" then
shouldn't the programming language or a standard library help him get
that irrespective of the human language the program is meant to be
processing?

>
>
>>
>> Aside from converting digits (and any other characters used as digits
>> in a higher number base) is there's any meaning to converting chars
>> to/from ints?
>
>
> My static language makes byte and char slightly different types. (Types
> involving char may get printed differently.)
>
> That meant that `ref byte` and `ref char` were incompatible, which
> rapidly turned into a nightmare: I might have a readfile() routine that
> returned a `ref byte` type, a pointer to a block of memory.
>
> But I wanted to interpret that block as `ref char` - a string. So this
> meant loads of casts to either `ref byte` or `ref char` to get things to
> work, but it got too much (a bit like 'const poisoning' in C where it
> just propagates everywhere). That was clearly the wrong approach.

C's const propagation sounds like Java with its horrible, and sticky,
exception propagation.

>
> In the end I relaxed the type rules so that `ref byte` and `ref char`
> are compatible, and everything is now SO much simpler.

Would there have been any value in defining a layout for the untyped
area of bytes (or parts thereof)? That's where I think I am headed.

--
James Harris

On 05/11/2022 13:46, Bart wrote:
> On 05/11/2022 11:14, James Harris wrote:

...

>> For example,
>>
>>    print "Hello world"
>>
>> would be accompanied by a file for French which included
>>
>>    "Hello world" --> "Bonjour le monde"
>>
>> Naturally, multilingual programming is much more complex than that
>> simple example but it shows the basic idea. The compiler would be able
>> to check that language files had everything required for a given piece
>> of source code.
>
> Is it? This pretty much all I did when I used to write internationalised
> applications. Although that was only done for French, German and Dutch.

I imagine multilingual programming would be very difficult and that it's
something a language should help with but it's not something I have had
to do as yet.

>
> But that print example would be written like this:
>
>     print /"Hello World"
>
> The "/" was a translation operator, so only certain strings were
> translated. This also made it easy to scan source code to build a list
> of messages, used to maintain the dictionary as entries were added,
> deleted or modified.
>
> The scheme did need some hints sometimes, written like this, to get
> around ambiguities:
>
>     print /"Green!colour"
>     print /"Green!fresh"

That's cool. I have something similar which is trailing identifiers but
I had them down as specific rather than hints. For example,

print "Green" :GreenColor
print "Green" :GreenFresh

Then the accompanying language files could distinguish between the strings.

>
> The hint was usually filtered out.
>
> But this is little to do with how strings are represented. Even in
> English, messages may include characters like "£" (pound sign) which is
> not part of ASCII.

I have two ways to deal with that. If the program is to use a pound sign
in England but a Dollar sign in America, say, then the source would have
a dollar sign and there'd be a file to translate for use in England.

On the other hand, if the program was to use a pound sign in all cases
then I'd do as below.

>
> So a way to represent Unicode within literals is still needed (didn't we
> discuss this a couple of years ago?).
>

Yes. You may remember my preference was for a named character something like

pound_currency_string = "\PoundSterling/"

--
James Harris

On 04/11/2022 18:50, Bart wrote:
> On 04/11/2022 13:58, David Brown wrote:
>
>> Neither "byte" nor "character" should have any kind of arithmetic
>> operators - they are not integers.  But you will need cast or
>> conversion operations on them.
>
> Bytes are small integers, typically of u8 type.

That's true in some languages. In other languages they are character
types. (In C, they are both.)

My suggestion is that they should not be considered integers or
characters, but a low-level "raw data" type. A "byte" should represent
a byte of memory, or an octet in a network packet, or a byte of a file.
It doesn't make sense to do any arithmetic on it because it might be
part of some completely different data type.

A given "byte" might be part of the storage of a floating point number,
or the storage of an address or pointer, or anything else.

I've said this before - the strength of a programming language is mainly
determined by what you /cannot/ do, rather than what you /can/ do.
Design a language to make it as hard as possible to get things wrong,
while still being easy to do the things you want it to do. Keeping
integer types, character types, strings, and raw data independent can
help with that.

>
> I can't see why arithmetic can't be done with them, unless you want a
> purer kind of language where arithmetic is only allowed on signed
> numbers, and bitwise ops only on unsigned numbers, which is usually
> going to be a pain for all concerned.
>

Whether you have signed and unsigned integer types, what sizes you have,
whether you have an abstract "integer" type and explicitly ranged
subtypes (as in Ada) is another matter.

>> The concept of "signed char" and "unsigned char" in C is a serious
>> design flaw.  A type designed to hold letters should not have a sign,
>> and should not be used to hold arbitrary raw, low-level data.
>
> Signed and unsigned chars are not so bad;

Yes, they are bad. A "character" is a letter or other visual symbol.
Can you explain the difference between "a positive letter X" and "a
negative letter X" ? Of course you can't - it is utter nonsense. The
same goes for adding 6 to the Greek letter µ, or multiplying Ð by Å.
Even operations that might appear sensible in code, such as adding 1 to
a char, don't actually make sense - it is the operation of taking the
next letter in the alphabet that makes sense.

> presumably C intended these to
> do the job of a 'byte' type for small integers. So it was just a poor
> choice of name. (After all there is no separate type in C for bytes
> holding character data.)
>
> What's bad is that third kind: a 'plain char' type, which is
> incompatible with both signed and unsigned char, even though it
> necessarily needs to be one of the other on a specific platform. It
> occurs in no other language, and causes problems within FFI APIs.
>

Certainly having three different "char" types in C is bad. But the only
sensible choice is to have nothing but a plain "char" and use /integer/
types for numeric data. (Call them "u8" and "i8" if you prefer, rather
than the C names "uint8_t" and "int8_t".)

(These days I would consider not having any kind of character type at
all, unless it was a language targeting small embedded systems that need
maximal efficiency - by the time you have something that can hold any
UTF-8 character, you might as well just call it a "string".)