Hi all,

I'm trying to sync up the active and newsgroups file from 15 peers and
it's proving to be a bit of a challenge.

The first bit is done, which is mainly getting rid of groups that have
invalid names (those that end in a period, contain illegal characters,
and the like).

Next is a little more of a challenge. Trying to sync the descriptions.
It wouldn't be so bad if everyone used the same encoding, however the
majority are using ISO-8859-1, a couple are using UTF-8, some using
ASCII and one is Non-ISO extended-ASCII.

This becomes a problem when trying to do a diff or other operations
trying to match group names.

I know there isn't a standard encoding for the newsgroup file but that
may have been a bit of an oversight now that some people are trying to
run a clean server.

Going forward, maybe the powers that be can get their heads together
and enforce a certain coding standard for innd (and whatever else is
out there) that is at least maintained. Personally, I don't care which
one we end up with, ISO-8859 seems to be the far more popular (7
servers) followed by ASCII (4) then UTF-8 (3) .

I guess we'd need all the servers to want to agree and update their
files accordingly.

Somehow, I feel I'll be shot down here since it's been like this since
1986.

Thoughts?
--
End Of The Line BBS - Plano, TX
telnet endofthelinebbs.com 23

Nigel Reed <sysop@endofthelinebbs.com> writes:
> I know there isn't a standard encoding for the newsgroup file but that
> may have been a bit of an oversight now that some people are trying to
> run a clean server.
>
> Going forward, maybe the powers that be can get their heads together
> and enforce a certain coding standard for innd (and whatever else is
> out there) that is at least maintained. Personally, I don't care which
> one we end up with, ISO-8859 seems to be the far more popular (7
> servers) followed by ASCII (4) then UTF-8 (3) .

If there’s going to be a global choice of encoding then it has to be
UTF-8.

> I guess we'd need all the servers to want to agree and update their
> files accordingly.

That’s the hard bit...

--
https://www.greenend.org.uk/rjk/

Hello,

> I'm trying to sync up the active and newsgroups file from 15 peers and
> it's proving to be a bit of a challenge.

Hum, I rather say : "It's a f***ing challenge"!

> Next is a little more of a challenge. Trying to sync the descriptions.

> I know there isn't a standard encoding for the newsgroup file but that
> may have been a bit of an oversight now that some people are trying to
> run a clean server.
Since my server has a graphical interface that displays group names and
descriptions, I need to be able to know which encoding is used.

To solve the problem, I use a small file that records the hierarchies
with the encoding used. Either I get it from the cmsg checkgroup of some
hierarchies that mention it (notably fr), or I fix it by hand.

This file is very simple. It mentions the hierarchy and the encoding
used, separated by a TAB.

Maybe adding an identical file to "active" and "newsgroups" ones would
do the trick?

##
## This file lists the charsets used by hierarchies.
##
## Format: hierarchie<TAB>charset
cn Big5
fr UTF-8

And so on.

?

Franck

Hello,

> If there’s going to be a global choice of encoding then it has to be
> UTF-8.

+1

Hello,

> Since my server has a graphical interface that displays group names and
> descriptions, I need to be able to know which encoding is used.

It looks like : https://i.ibb.co/LYcsQVQ/Console.png

Hi Nigel,

> Next is a little more of a challenge. Trying to sync the descriptions.
> It wouldn't be so bad if everyone used the same encoding, however the
> majority are using ISO-8859-1, a couple are using UTF-8, some using
> ASCII and one is Non-ISO extended-ASCII.

FWIW, the descriptions encoded in UTF-8 from the ftp.isc.org newsgroup
file are here:
http://usenet.trigofacile.com/hierarchies/data/newsgroups.utf8

It may facilitate your life :-)

The conversions I found out to work are:
- cn.* and han.* are encoded in gb18030;
- fido7.*, medlux.* and relcom.* in koi8-r;
- ukr.* in koi8-u;
- nctu.*, ncu.* and tw.* in big5;
- scout.forum.chinese and scout.forum.korean in big5;
- eternal-september.*, fido.* and fr.* in utf-8;
- all the others fit well in cp1252.

> Going forward, maybe the powers that be can get their heads together
> and enforce a certain coding standard for innd (and whatever else is
> out there) that is at least maintained.

UTF-8 is the expected encoding for the descriptions returned by a LIST
NEWSGROUPS command in the NNTP protocol.

--
Julien ÉLIE

« Et maintenant, la balle est dans le camp des slalomeurs. »

Salut Julien.

> FWIW, the descriptions encoded in UTF-8 from the ftp.isc.org newsgroup
> file are here:
>   http://usenet.trigofacile.com/hierarchies/data/newsgroups.utf8

Et tu le dis maintenant!?!?!? ;-)

Why not to put it at ftp.isc.org?

I had looked on your site, notably the List of Usenet public managed
hierarchies, but I had not found this list, reason why I coded it in
SNS. Maybe I not looked so well...

> It may facilitate your life :-)

I think so, I'll use it instead of the one listed at ftp.isc.org and
will remove some lines of code in SNS.

Franck

Julien ÉLIE <iulius@nom-de-mon-site.com.invalid> wrote:

> The conversions I found out to work are:

In Poland (pl.*) we traditionally used iso-8859-2 for posting (now I think
utf-8 has become a de facto standard, but iso-8859-2 is still accepted),
but I can see that group descriptions for pl.* are just transliterated
(there are no national characters used, all are 7-bit, or us-ascii).

The main question is if currently used readers can handle utf-8 in group
descriptions. If yes, I'd stick with utf-8. If not, then I think it would
be safest to transliterate the descriptions to us-ascii (if it can be done
for all encodings; for Polish national characters it's perfectly fine, but
I don't know how it works with non-Latin alphabets like Russian or
Chinese).

Hello Adam,

Le 11/04/2023 17:15, Adam H. Kerman a écrit :
>
>>If there’s going to be a global choice of encoding then it has to be
>>UTF-8.
>
> ASCII's advantage over UTF-8 is its universality.

I would have said exactly the opposite : UTF-8's advantage over ASCII
is its universality, because UTF-8 can express any character from any
language.

But of course ASCII's advantage over UTF-8 is that it is recognized by
all usenet softwares.

> [...]
>
> Gee whiz. The ASCII apostrophe was used ambiguously as single close quote
> AND as the combining diacritical mark for the acute accent since 1967.

Oh? Which software does that weird thing? Surely it is not a standard use
of ASCII.

--
Olivier Miakinen

On Tue, 11 Apr 2023 12:12:03 +0200
Julien ÉLIE <iulius@nom-de-mon-site.com.invalid> wrote:

> > Going forward, maybe the powers that be can get their heads together
> > and enforce a certain coding standard for innd (and whatever else is
> > out there) that is at least maintained.
>
> UTF-8 is the expected encoding for the descriptions returned by a
> LIST NEWSGROUPS command in the NNTP protocol.

That's good to know. I've been converting everything to fit into my
newsgroups file which is ISO-8859 so it looks like I've been going the
wrong way. Back to the drawing board now that my scripts are almost
done lol.

--
End Of The Line BBS - Plano, TX
telnet endofthelinebbs.com 23

Olivier Miakinen <om+news@miakinen.net> wrote:
>04/11/2023 17:15, Adam H. Kerman wrote:

>>>If there’s going to be a global choice of encoding then it has to be
>>>UTF-8.

>>ASCII's advantage over UTF-8 is its universality.

>I would have said exactly the opposite : UTF-8's advantage over ASCII
>is its universality, because UTF-8 can express any character from any
>language.

Well, yes, if one's set up displays UTF-8, but every setup can use
ASCII.

>But of course ASCII's advantage over UTF-8 is that it is recognized by
>all usenet softwares.

>>[...]

>>Gee whiz. The ASCII apostrophe was used ambiguously as single close quote
>>AND as the combining diacritical mark for the acute accent since 1967.

>Oh? Which software does that weird thing? Surely it is not a standard use
>of ASCII.

ASCII was the 7-bit encoding used for teletypewriters, an improvement
over 5-bit Baudot code. Backspace/overstrike sequences were the way
diacritic marks were combined with the alphabetic character on a
teletypewriter. No, generally this wasn't implemented in computer
software.

But the notion that ASCII wasn't intended for Latin alphabets beyond
English used in America was always wrong.

On 2023-04-11, Nigel Reed <sysop@endofthelinebbs.com> wrote:
> On Tue, 11 Apr 2023 12:12:03 +0200
> Julien ÉLIE <iulius@nom-de-mon-site.com.invalid> wrote:
>
>> UTF-8 is the expected encoding for the descriptions returned by a
>> LIST NEWSGROUPS command in the NNTP protocol.
>
> That's good to know. I've been converting everything to fit into my
> newsgroups file which is ISO-8859 so it looks like I've been going the
> wrong way. Back to the drawing board now that my scripts are almost
> done lol.

At least the conversion from ISO-8859 to UTF-8 will be much more
straightforward than the conversion from <whatever encoding> to ISO-8859
;)

Cheers,
Tom

"Adam H. Kerman" <ahk@chinet.com> writes:

> If you do that, may I request that ASCII equivalents be substituted for
> UTF-8 punctuation in brief descriptions? Pretty please?

The goal of all of that machinery is that the hierarchy administrators
should be canonical for the newsgroups entries for their hierarchy.
Encoding is one of those things where we need to standardize in order to,
say, comply with the NNTP standard, but I'm not willing to make any other
editorial judgments because it gets into too much annoying work. So this
is something you should take up with the hierarchy administrators.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

Russ Allbery <eagle@eyrie.org> wrote:
>"Adam H. Kerman" <ahk@chinet.com> writes:

>>If you do that, may I request that ASCII equivalents be substituted for
>>UTF-8 punctuation in brief descriptions? Pretty please?

>The goal of all of that machinery is that the hierarchy administrators
>should be canonical for the newsgroups entries for their hierarchy.
>Encoding is one of those things where we need to standardize in order to,
>say, comply with the NNTP standard, but I'm not willing to make any other
>editorial judgments because it gets into too much annoying work. So this
>is something you should take up with the hierarchy administrators.

I apologize for suggesting additional programming work for you. I change
my request to asking for an amendment to your README in which you might
urge a proponent or hierarchy administrator not to use UTF-8 punctuation
for which ASCII punctuation would suffice, to avoid needlessly turning a
description into UTF-8.

Richard Kettlewell <invalid@invalid.invalid> wrote:
>Nigel Reed <sysop@endofthelinebbs.com> writes:

>>I know there isn't a standard encoding for the newsgroup file but that
>>may have been a bit of an oversight now that some people are trying to
>>run a clean server.

>>Going forward, maybe the powers that be can get their heads together
>>and enforce a certain coding standard for innd (and whatever else is
>>out there) that is at least maintained. Personally, I don't care which
>>one we end up with, ISO-8859 seems to be the far more popular (7
>>servers) followed by ASCII (4) then UTF-8 (3) .

>If there’s going to be a global choice of encoding then it has to be
>UTF-8.

ASCII's advantage over UTF-8 is its universality.

You yourself just used the UTF-8 character code for single close quote
ambiguously as an apostrophe. The character is has been used ambiguously
in the current version of Unicode, replacing another character code that
was used to indicate a glottal stop as a letter modifier.

Gee whiz. The ASCII apostrophe was used ambiguously as single close quote
AND as the combining diacritical mark for the acute accent since 1967.
Where is the UTF-8 advantage if there continues to be ambiguously-used
character codes for such common punctuation marks?

If there's going to be a global choice, then stop using UTF-8 character
codes to substitue for ASCII in plain text communication. Use open and
close single and double quotes ONLY in typography, not email and not
Usenet. This thwarts communication. It makes a difference as ASCII is
universal and UTF-8 is not.

I'm pointing out again that ASCII had combining characters but it didn't
include all possible diacritical marks like umlaut, but it has acute,
grave, circumflex, tilde, slash, cedilla, and I'm sure I've forgotten.

Teletypewriters could perform the combining action with a backspace/
overstrike sequence but terminals didn't usually display them.

>>I guess we'd need all the servers to want to agree and update their
>>files accordingly.

>That’s the hard bit...

Russ Allbery <eagle@eyrie.org> wrote:

> It definitely cannot. It's rare to find a language where that can be done
> without losing information (only in Europe, essentially).

Well, to be honest, you lose some information, but it's very rare and can
usually be deduced from context.

Nigel Reed <sysop@endofthelinebbs.com> wrote:
> This becomes a problem when trying to do a diff or other operations
> trying to match group names.

My opinion is that newsgroup names should be purely ASCII (there are many
benefits to this, and using non-ASCII characters in newsgroup names and
domain names and commands and configuration files can cause many problems,
including security issues (especially if any Unicode-based encoding is
used; non-Unicode has less security issues, but still is not worth it to
use non-ASCII in these cases), comparisons, input, etc).

Descriptions for non-English newsgroups, and non-English articles in such
newsgroups, should probably use the appropriate encodings for those
languages, rather than ASCII only (it is not as problematic to use
non-ASCII characters in newsgroup descriptions, and there are clearly
benefits to doing so in some cases, so it should be permitted in
descriptions and in articles; however, if the text is English only then
it is almost always more beneficial to stay to ASCII only, I think).

> Going forward, maybe the powers that be can get their heads together
> and enforce a certain coding standard for innd (and whatever else is
> out there) that is at least maintained. Personally, I don't care which
> one we end up with, ISO-8859 seems to be the far more popular (7
> servers) followed by ASCII (4) then UTF-8 (3) .

Well, fortunately, ISO-8859-1 and UTF-8 are both supersets of ASCII, so
if you use ASCII as much as possible then it will still work. (But, I
really hate Unicode; it is full of problems, including Han unification
and other complications; and it is a stateful character set even though
the encoding is stateless. TRON character code is better in some ways
(especially for Japanese text), and I have done some work using this.)

However, also, enforcing a certain coding standard (regardless of what it
might be, whether it is Unicode or TRON or something else) can be a problem
when you will need other encodings for a reason not previously known by
whoever enforced them. Making recommendations can be helpful though, but I
think that ASCII should be used when possible, and in some contexts (e.g.
the names of the commands, etc, in the computer programming) should be
required to be ASCII only.

It might also be worth to mention what character encodings it uses in the
CAPABILITIES, on servers where that is applicable. (The RFC says that it
should be UTF-8, but I think that this is a mistake in the design of the
protocol. Capabilities and commands should be pure ASCII, but this should
not mean that any text in articles, descriptions, MOTD, etc has to be pure
ASCII; it can use other character sets, including the possibility of ones
which might be incompatible with Unicode, and including TRON codes too.)

Many people, they just want to put Unicode in everything, without actually
understanding Unicode or international text or security or anything else,
and this just makes a mess (especially since Unicode itself is messy, but
even if using something else, just putting it in without any consideration,
does not substitute for actual understanding). So, don't do that, please.

(And, for Usenet client programs intended for PC (if using DOS or other
text-mode programs), use of PC character set may be beneficial.)

(I run a NNTP server with my own newsgroups, which are not (currently)
considered part of Usenet, and currently have no need for non-ASCII
descriptions, but in future if it does, then I will consider what to do.
However, I also don't use INN, anyways.)

--
Don't laugh at the moon when it is day time in France.

Adam W. <gof-cut-this-news@cut-this-chmurka.net.invalid> wrote:
>Russ Allbery <eagle@eyrie.org> wrote:

>>It definitely cannot. It's rare to find a language where that can be done
>>without losing information (only in Europe, essentially).

>Well, to be honest, you lose some information, but it's very rare and can
>usually be deduced from context.

In a language that doesn't use the Latin alphabet? C'mon.

[Please do not mail me a copy of your followup]

Richard Kettlewell <invalid@invalid.invalid> spake the secret code
<wwvh6tmg8oa.fsf@LkoBDZeT.terraraq.uk> thusly:

>If there's going to be a global choice of encoding then it has to be
>UTF-8.

You mean, so you can use a gratuitously fancy apostrophe character
instead of the ASCII ' character that serves exactly the same purpose
with fewer problems?

UTF-8 is great for non-Latin codepoints like Asian languages and
Klingon.

Where UTF-8 fails is in using fancy codepoints for the functional
equivalent of the same ASCII character.
--
"The Direct3D Graphics Pipeline" free book <http://tinyurl.com/d3d-pipeline>
The Terminals Wiki <http://terminals-wiki.org>
The Computer Graphics Museum <http://computergraphicsmuseum.org>
Legalize Adulthood! (my blog) <http://legalizeadulthood.wordpress.com>

Nigel Reed <sysop@endofthelinebbs.com> writes:

> Going forward, maybe the powers that be can get their heads together and
> enforce a certain coding standard for innd (and whatever else is out
> there) that is at least maintained. Personally, I don't care which one
> we end up with, ISO-8859 seems to be the far more popular (7 servers)
> followed by ASCII (4) then UTF-8 (3) .

It's been on my list for years to encode the ftp.isc.org newsgroups file
uniformly in UTF-8, which I think is a prerequisite for enforcing
something in innd, but it's a bunch of tedious work and I haven't found
the time yet.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

gof-cut-this-news@cut-this-chmurka.net.invalid (Adam W.) writes:

> The main question is if currently used readers can handle utf-8 in group
> descriptions. If yes, I'd stick with utf-8. If not, then I think it would
> be safest to transliterate the descriptions to us-ascii (if it can be done
> for all encodings;

It definitely cannot. It's rare to find a language where that can be done
without losing information (only in Europe, essentially).

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

Russ Allbery <eagle@eyrie.org> wrote:
>Nigel Reed <sysop@endofthelinebbs.com> writes:

>>Going forward, maybe the powers that be can get their heads together and
>>enforce a certain coding standard for innd (and whatever else is out
>>there) that is at least maintained. Personally, I don't care which one
>>we end up with, ISO-8859 seems to be the far more popular (7 servers)
>>followed by ASCII (4) then UTF-8 (3) .

>It's been on my list for years to encode the ftp.isc.org newsgroups file
>uniformly in UTF-8, which I think is a prerequisite for enforcing
>something in innd, but it's a bunch of tedious work and I haven't found
>the time yet.

If you do that, may I request that ASCII equivalents be substituted for
UTF-8 punctuation in brief descriptions? Pretty please?

Hi all,

> The RFC says that it
> should be UTF-8, but I think that this is a mistake in the design of the
> protocol. Capabilities and commands should be pure ASCII, but this should
> not mean that any text in articles, descriptions, MOTD, etc has to be pure
> ASCII; it can use other character sets, including the possibility of ones
> which might be incompatible with Unicode, and including TRON codes too.

We need an interoperable way to provide texts.
Please note RFC 2277 (BCP 18) about charsets:

Protocols MUST be able to use the UTF-8 charset, which consists of
the ISO 10646 coded character set combined with the UTF-8 character
encoding scheme, as defined in [10646] Annex R (published in
Amendment 2), for all text.

Protocols MAY specify, in addition, how to use other charsets or
other character encoding schemes for ISO 10646, such as UTF-16, but
lack of an ability to use UTF-8 is a violation of this policy; such a
violation would need a variance procedure ([BCP9] section 9) with
clear and solid justification in the protocol specification document
before being entered into or advanced upon the standards track.

For existing protocols or protocols that move data from existing
datastores, support of other charsets, or even using a default other
than UTF-8, may be a requirement. This is acceptable, but UTF-8
support MUST be possible.

> (I run a NNTP server with my own newsgroups, which are not (currently)
> considered part of Usenet, and currently have no need for non-ASCII
> descriptions, but in future if it does, then I will consider what to do.
> However, I also don't use INN, anyways.)

FWIW, INN does not enforce UTF-8 in the descriptions of newsgroups. You
can use any encoding you want for them.

--
Julien ÉLIE

« Ils ont refusé une offre de Normand ?!? » (Astérix)

"Adam H. Kerman" <ahk@chinet.com> writes:
> Adam W. <gof-cut-this-news@cut-this-chmurka.net.invalid> wrote:
>> Adam H. Kerman <ahk@chinet.com> wrote:

>>>> Well, to be honest, you lose some information, but it's very rare and
>>>> can usually be deduced from context.

>>> In a language that doesn't use the Latin alphabet? C'mon.

>> No, I'm only talking about Polish.

> You are. Russ wasn't.

Yeah, but I understood Adam was only talking about Polish in his reply.

It's common in a lot of European languages to be able to transliterate to
ASCII using various schemes without losing *too* much information. German
has a standard scheme, some Scandinavian languages have an old scheme that
used to be used when it was hard to find anything other than ASCII, some
other European languages are still comprehensible if all the diacritic
marks are stripped even though it looks weird, etc. Apparently Polish is
one of those (I know very little about Polish, sadly).

Arguably, English is itself a case of being able to transliterate to ASCII
without losing too much information, depending on how you feel about the
correct spelling of Zoë and naïve (I think everyone but the New Yorker has
given up on coöperate), or how much you care about reproducing English
poetry containing words like learnèd.

But if one gets too far beyond Europe, or even farther into eastern Europe
and non-Romance languages, the transliterations get more and more dubious
or simply nonexistent.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

Russ Allbery <eagle@eyrie.org> wrote:

> Yeah, but I understood Adam was only talking about Polish in his reply.

Yes, exactly. That's the only non-English language I know.

> German has a standard scheme,

Do you mean substituting umlauts with their Latin equivalents and adding
"e"?

ä = ae
ö = oe
ü = ue

At least that's what I found:

https://blogs.transparent.com/german/writing-the-letters-%E2%80%9Ca%E2%80%9D-%E2%80%9Co%E2%80%9D-and-%E2%80%9Cu%E2%80%9D-without-a-german-keyboard/

I also know that their ß (scharfes S) can be substituted with ss.

> some other European languages are still comprehensible if all the
> diacritic marks are stripped even though it looks weird, etc.
> Apparently Polish is one of those (I know very little about Polish,
> sadly).

It is. There are some word plays, because some words have different
meanings with and without diacritics (for example, "łaska" and "laska"
mean different things), but they're rare and correct meaning can be
deduced from context.