On Tue, 24 Jan 2023 16:25:00 +0100
Johnny Billquist <bqt@softjar.se> wrote:

> Unicode is such a mixture of everything that nothing is clear.

It is a lot less unclear if you ignore the glyphs completely and
simply ask which code point should I use to represent this entity - if
there is a specialised code point (eg units) then use that otherwise
fallback to the nearest available option.

There is a lot of stuff in Unicode that seems to me to have no
reason for being there but I consider that a small price to pay for having
pretty much every character I might ever need available and usually even
being able to be semantically correct in my choice of code point. We've
come a long way from backspacing over an o and typing a /.

Oh and yes I agree typographical differences have no place in
unicode, I have no idea why the fullwidth m is there.

--
Steve O'Hara-Smith
Odds and Ends at http://www.sohara.org/

Rich Alderson <news@alderson.users.panix.com> writes:

> Andreas Kohlbach <ank@spamfence.net> writes:
>
>> On 17 Jan 2023 19:57:05 -0500, Rich Alderson wrote:
>
>>> You're missing an umlaut, or an "e": His new name is Oetzi, not
>>> Otzi.
>
>> Actually _tzi.
>
> I assume that that's umlaut-o, but I've never gotten GNUS to display
> Unicode as anything other than question marks or \ooo octals. I was
> told in German classes 50 years ago that "ae", "oe", and "ue" were
> acceptable alternatives, if somewhat outmoded.

Ötzi. The trick is to know what it is called "LATIN CAPITAL LETTER O
WITH DIAERESIS" and then you can use ctrl-x 8 <return> LATIN<tab> etc.

On 2023-01-24 18:30, Ahem A Rivet's Shot wrote:
> On Tue, 24 Jan 2023 16:25:00 +0100
> Johnny Billquist <bqt@softjar.se> wrote:
>
>> Unicode is such a mixture of everything that nothing is clear.
>
> It is a lot less unclear if you ignore the glyphs completely and
> simply ask which code point should I use to represent this entity - if
> there is a specialised code point (eg units) then use that otherwise
> fallback to the nearest available option.

Which is sortof unclear. :-)

> There is a lot of stuff in Unicode that seems to me to have no
> reason for being there but I consider that a small price to pay for having
> pretty much every character I might ever need available and usually even
> being able to be semantically correct in my choice of code point. We've
> come a long way from backspacing over an o and typing a /.

I'm not sure we actually progressed that much.

> Oh and yes I agree typographical differences have no place in
> unicode, I have no idea why the fullwidth m is there.

I sortof know, which is why I'm also aware of it, and annoyed by it. In
for example, Japanese, when they write latin characters, they
traditionally wanted them to use the same kind of spacing as hiragana,
katakana and so on, so they already had these full width latin. So when
Unicode was created, it was decided to preserve this as well.

And then you also have superscript and subscript of all the latin
letters and numbers, and a few more duplicates, not to mention similar
looking letters in different alphabets, making it extremely hard to know
which Unicode code point you actually should use. Not to mention
understanding if two strings are actually equal, if we talk about the
word they represent, as the actual codepoints used can be very
different, with no semantical difference.

I could go on... But I get the feeling you get it. :-)

Johnny

On Wed, 25 Jan 2023 15:05:38 +0100
Johnny Billquist <bqt@softjar.se> wrote:

> On 2023-01-24 18:30, Ahem A Rivet's Shot wrote:
> > On Tue, 24 Jan 2023 16:25:00 +0100
> > Johnny Billquist <bqt@softjar.se> wrote:
> >
> >> Unicode is such a mixture of everything that nothing is clear.
> >
> > It is a lot less unclear if you ignore the glyphs completely and
> > simply ask which code point should I use to represent this entity - if
> > there is a specialised code point (eg units) then use that otherwise
> > fallback to the nearest available option.
>
> Which is sortof unclear. :-)
>
> > There is a lot of stuff in Unicode that seems to me to have no
> > reason for being there but I consider that a small price to pay for having
> > pretty much every character I might ever need available and usually even
> > being able to be semantically correct in my choice of code point. We've
> > come a long way from backspacing over an o and typing a /.
>
> I'm not sure we actually progressed that much.
>
> > Oh and yes I agree typographical differences have no place in
> > unicode, I have no idea why the fullwidth m is there.
>
> I sortof know, which is why I'm also aware of it, and annoyed by it. In
> for example, Japanese, when they write latin characters, they
> traditionally wanted them to use the same kind of spacing as hiragana,
> katakana and so on, so they already had these full width latin. So when
> Unicode was created, it was decided to preserve this as well.
>
> And then you also have superscript and subscript of all the latin
> letters and numbers, and a few more duplicates, not to mention similar
> looking letters in different alphabets, making it extremely hard to know
> which Unicode code point you actually should use. Not to mention
> understanding if two strings are actually equal, if we talk about the
> word they represent, as the actual codepoints used can be very
> different, with no semantical difference.
>
> I could go on... But I get the feeling you get it. :-)
>
> Johnny

TL;DR - it's a mess.

--
Bah, and indeed Humbug.

On Wed, 25 Jan 2023 17:13:26 +0000
"Kerr-Mudd, John" <admin@127.0.0.1> wrote:

> TL;DR - it's a mess.

But it's the least awful available solution to international text
encoding.

--
Steve O'Hara-Smith
Odds and Ends at http://www.sohara.org/

On 2023-01-25 19:19, Ahem A Rivet's Shot wrote:
> On Wed, 25 Jan 2023 17:13:26 +0000
> "Kerr-Mudd, John" <admin@127.0.0.1> wrote:
>
>> TL;DR - it's a mess.
>
> But it's the least awful available solution to international text
> encoding.

Well, it's the only one, and it's unlikely we'll see any other. It's
established, and we'll have to live with it, just like the x86. (I know
ARM exists... :-P )

Johnny

On Thu, 26 Jan 2023 22:05:35 +0100
Johnny Billquist <bqt@softjar.se> wrote:

> On 2023-01-25 19:19, Ahem A Rivet's Shot wrote:
> > On Wed, 25 Jan 2023 17:13:26 +0000
> > "Kerr-Mudd, John" <admin@127.0.0.1> wrote:
> >
> >> TL;DR - it's a mess.
> >
> > But it's the least awful available solution to international
> > text encoding.
>
> Well, it's the only one, and it's unlikely we'll see any other. It's

Nah, we had international text encoding before Unicode, with all
the iso-8859 8 bit codes, the DOS code pages and WIN-12* 8 bit codes along
with several dozen wider encodings such as shift-JIS and of course HTML
and XML entities. All you had to do was encode each script separately and
keep track of which encoding you were using for which text at all times.

Unicode is way less awful than that mess.

--
Steve O'Hara-Smith
Odds and Ends at http://www.sohara.org/

On 2023-01-27 08:13, Ahem A Rivet's Shot wrote:
> On Thu, 26 Jan 2023 22:05:35 +0100
> Johnny Billquist <bqt@softjar.se> wrote:
>
>> On 2023-01-25 19:19, Ahem A Rivet's Shot wrote:
>>> On Wed, 25 Jan 2023 17:13:26 +0000
>>> "Kerr-Mudd, John" <admin@127.0.0.1> wrote:
>>>
>>>> TL;DR - it's a mess.
>>>
>>> But it's the least awful available solution to international
>>> text encoding.
>>
>> Well, it's the only one, and it's unlikely we'll see any other. It's
>
> Nah, we had international text encoding before Unicode, with all
> the iso-8859 8 bit codes, the DOS code pages and WIN-12* 8 bit codes along
> with several dozen wider encodings such as shift-JIS and of course HTML
> and XML entities. All you had to do was encode each script separately and
> keep track of which encoding you were using for which text at all times.

If you consider a coding which actually don't tell what it encodes
without additional meta-data as an international text encoding. I think
I do not.

> Unicode is way less awful than that mess.

Well, truth be told, the way Unicode turned out, I would have preferred
staying with 8859-xyzzy and have additional meta-data to tell which
encoding was used.

But that's not happening. And the amount of encodings possible with any
8859 variant is really not close to enough to cover all needs. And noone
is adding any more.

Johnny

On Sun, 29 Jan 2023 00:25:24 +0100
Johnny Billquist <bqt@softjar.se> wrote:

> Well, truth be told, the way Unicode turned out, I would have preferred
> staying with 8859-xyzzy and have additional meta-data to tell which
> encoding was used.

Throw in all the other encodings for CJK scripts etc. too and then
remember that the meta data and text can and will get separated even
assuming the meta data is correct which it often isn't.

One of the problems we had with the Yahoo! news feed was WIN-1252
text being supplied when the meta data said ISO-8859-1 - WIN-1252 has
printable characters where ISO-8859-1 has control characters but is
otherwise almost identical, there are similar WIN/ISO mismatches for the
other scripts too.

The upshot of this discovery was some horrible heuristic code in
the path from live news feed to web page that tries to guess the the correct
encoding when the text is invalid under the declared encoding. All that
because someone wanted to print a Euro sign and couldn't get the meta data
right.

So I disagree - *any* universal encoding no matter how messed up by
politics and bike shedding is an infinite improvement over the chaos of
metadata and an ever expanding mess of encodings none of which cover
everything making some forms of mixed text completely impossible. The
ability to take a string of bytes and display it as text in a form
guaranteed to match the original intent is *priceless*.

--
Steve O'Hara-Smith
Odds and Ends at http://www.sohara.org/