Hi Retro Guy,

> I'm having some trouble where after inn2 runs for a few hours I
> get the error 'File exists writing SMstore file -- throttling'

Do you happen to use tradspool and some newsgroup names have components
with only digits?
For instance, if you have a newsgroup named net.test.17 or
net.test.17.help and another named net.test, I believe this error will
come up when receiving article number 17 for net.test. INN will try to
write the article into the file <patharticles>/net/test/17 whereas it is
a directory (belonging to the net.test.17 newsgroup or net.test.17.help).

Or the inverse is possible: having net.test and trying to insert article
1 for the net.test.17 newsgroup whereas net.test already has 17 articles.

You should either remove the <patharticles>/net/test/17 file or the
net.test.17 newsgroup. Or use another storage method.

--
Julien ÉLIE

« – Je t'ai préparé une bonne soupe dont tu me diras des nouvelles, mon
garçon !
– Pour moi, ça, ce n'est pas des bonnes nouvelles ! » (Astérix)

Julien_ÉLIE wrote:

> Hi Retro Guy,

>> I'm having some trouble where after inn2 runs for a few hours I
>> get the error 'File exists writing SMstore file -- throttling'

> Do you happen to use tradspool and some newsgroup names have components
> with only digits?
> For instance, if you have a newsgroup named net.test.17 or
> net.test.17.help and another named net.test, I believe this error will
> come up when receiving article number 17 for net.test. INN will try to
> write the article into the file <patharticles>/net/test/17 whereas it is
> a directory (belonging to the net.test.17 newsgroup or net.test.17.help).

> Or the inverse is possible: having net.test and trying to insert article
> 1 for the net.test.17 newsgroup whereas net.test already has 17 articles.

> You should either remove the <patharticles>/net/test/17 file or the
> net.test.17 newsgroup. Or use another storage method.

Thank you for the pointer. This appears to be exactly the problem. I found
net.micro, net.micro432 and net.micro6809. Following your advice and the
problem appears to be resolved.

--
Retro Guy

On Jun 8, 2023 at 11:13:44 AM CDT, "Retro Guy" <Retro Guy> wrote:

> Julien_ÉLIE wrote:
>
>> Hi Retro Guy,
>
>>> I'm having some trouble where after inn2 runs for a few hours I
>>> get the error 'File exists writing SMstore file -- throttling'
>
>> Do you happen to use tradspool and some newsgroup names have components
>> with only digits?
>> For instance, if you have a newsgroup named net.test.17 or
>> net.test.17.help and another named net.test, I believe this error will
>> come up when receiving article number 17 for net.test. INN will try to
>> write the article into the file <patharticles>/net/test/17 whereas it is
>> a directory (belonging to the net.test.17 newsgroup or net.test.17.help).
>
>> Or the inverse is possible: having net.test and trying to insert article
>> 1 for the net.test.17 newsgroup whereas net.test already has 17 articles.
>
>> You should either remove the <patharticles>/net/test/17 file or the
>> net.test.17 newsgroup. Or use another storage method.
>
> Thank you for the pointer. This appears to be exactly the problem. I found
> net.micro, net.micro432 and net.micro6809. Following your advice and the
> problem appears to be resolved.

How's your effort coming along?

Jesse Rehmer wrote:

> On Jun 8, 2023 at 11:13:44 AM CDT, "Retro Guy" <Retro Guy> wrote:

>> Julien_ÉLIE wrote:
>>
>>> Hi Retro Guy,
>>
>>>> I'm having some trouble where after inn2 runs for a few hours I
>>>> get the error 'File exists writing SMstore file -- throttling'
>>
>>> Do you happen to use tradspool and some newsgroup names have components
>>> with only digits?
>>> For instance, if you have a newsgroup named net.test.17 or
>>> net.test.17.help and another named net.test, I believe this error will
>>> come up when receiving article number 17 for net.test. INN will try to
>>> write the article into the file <patharticles>/net/test/17 whereas it is
>>> a directory (belonging to the net.test.17 newsgroup or net.test.17.help).
>>
>>> Or the inverse is possible: having net.test and trying to insert article
>>> 1 for the net.test.17 newsgroup whereas net.test already has 17 articles.
>>
>>> You should either remove the <patharticles>/net/test/17 file or the
>>> net.test.17 newsgroup. Or use another storage method.
>>
>> Thank you for the pointer. This appears to be exactly the problem. I found
>> net.micro, net.micro432 and net.micro6809. Following your advice and the
>> problem appears to be resolved.

> How's your effort coming along?

I currently have 1.49 million posts on novalink.us:119. Some visible by web
browser at http://novalink.us .

Most of the articles were able to be imported after running a script to modify
some older headers, and they are on the inn server. The oldest ones, which I
have a few, have not yet been modified. They need some work and I haven't had
the tme, but I still have them.

--
Retro Guy

On 23.08.23 14:31, Retro Guy wrote:
> I currently have 1.49 million posts on novalink.us:119. Some visible by web

this is the utzoo archive?

i scanned novalink vs my server and sucked only few missing messages.

Billy G. (go-while) wrote:

> On 23.08.23 14:31, Retro Guy wrote:
>> I currently have 1.49 million posts on novalink.us:119. Some visible by web

> this is the utzoo archive?

> i scanned novalink vs my server and sucked only few missing messages.

Yes, it is utzoo.

I don't know what differences there are between what is available from archive.org
and what is in utzoo. Just that the source of the posts on novalink is utzoo.

--
Retro Guy

On 02.06.23 23:36, Julien ÉLIE wrote:
>
> Actually, you need adding an additional dot to lines *beginning* with a
> dot, not only lines containing only a dot.
>

are you sure?

if a line contains a dot at the beginning and any text following can't
be a <CRLF>.
the beginning dot in for example ".anytext" or ". any text" does not
need to be escaped.

nntp protocol defines end-of-message with a <CRLF> <DOT> <CRLF>?

https://www.rfc-editor.org/rfc/rfc3977.txt

3.1.1. Multi-line Data Blocks

A multi-line data block is used in certain commands and responses.
It MUST adhere to the following rules:

1. The block consists of a sequence of zero or more "lines", each
being a stream of octets ending with a CRLF pair. Apart from
those line endings, the stream MUST NOT include the octets NUL,
LF, or CR.

2. In a multi-line response, the block immediately follows the CRLF
at the end of the initial line of the response. When used in any
other context, the specific command will define when the block is
sent.

3. If any line of the data block begins with the "termination octet"
("." or %x2E), that line MUST be "dot-stuffed" by prepending an
additional termination octet to that line of the block.

4. The lines of the block MUST be followed by a terminating line
consisting of a single termination octet followed by a CRLF pair
in the normal way. Thus, unless it is empty, a multi-line block
is always terminated with the five octets CRLF "." CRLF
(%x0D.0A.2E.0D.0A).

5. When a multi-line block is interpreted, the "dot-stuffing" MUST
be undone; i.e., the recipient MUST ensure that, in any line
beginning with the termination octet followed by octets other
than a CRLF pair, that initial termination octet is disregarded.

6. Likewise, the terminating line ("." CRLF or %x2E.0D.0A) MUST NOT
be considered part of the multi-line block; i.e., the recipient
MUST ensure that any line beginning with the termination octet
followed immediately by a CRLF pair is disregarded. (The first
CRLF pair of the terminating CRLF "." CRLF of a non-empty block
is, of course, part of the last line of the block.)

Note that texts using an encoding (such as UTF-16 or UTF-32) that may
contain the octets NUL, LF, or CR other than a CRLF pair cannot be
reliably conveyed in the above format (that is, they violate the MUST
requirement above). However, except when stated otherwise, this
specification does not require the content to be UTF-8, and therefore
(subject to that same requirement) it MAY include octets above and
below 128 mixed arbitrarily.

This document does not place any limit on the length of a line in a
multi-line block. However, the standards that define the format of
articles may do so.

"Billy G. (go-while)" <no-reply@no.spam> writes:
> On 02.06.23 23:36, Julien ÉLIE wrote:

>> Actually, you need adding an additional dot to lines *beginning* with a
>> dot, not only lines containing only a dot.

> are you sure?

Yes. :)

> if a line contains a dot at the beginning and any text following can't
> be a <CRLF>. the beginning dot in for example ".anytext" or ". any
> text" does not need to be escaped.

But see the bit that you quoted:

> 3. If any line of the data block begins with the "termination octet"
> ("." or %x2E), that line MUST be "dot-stuffed" by prepending an
> additional termination octet to that line of the block.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

On 11.09.23 17:42, Russ Allbery wrote:
> "Billy G. (go-while)" <no-reply@no.spam> writes:
>> On 02.06.23 23:36, Julien ÉLIE wrote:
>
>>> Actually, you need adding an additional dot to lines *beginning* with a
>>> dot, not only lines containing only a dot.
>
>> are you sure?
>
> Yes. :)
>
> But see the bit that you quoted:
>
>> 3. If any line of the data block begins with the "termination octet"
>> ("." or %x2E), that line MUST be "dot-stuffed" by prepending an
>> additional termination octet to that line of the block.
>

++thanks Russ!
++thanks Julien!

sometimes you can't see the forest for the trees!

found this and looks like GO does it correctly...
if you use dotreader/dotwriter.

i don't. the basic GO way does not stop reading.
in order to have an incoming article size limit, i read by lines, count
bytes and break out if an article is too large.
basic dotreader returns whenever client sends a closing dot which could
be somewhere near infinite.

to send data use "dotwriter": add a dot to every leading dot.

to read data use "dotreader": cut any first dot if line is not only a
(closing) dot.

i hope this is correct or did i miss something?

1) a) server sends article via IHAVE/TAKETHIS OR
b) client sends article via POST to server OR
c) server sends ARTICLE/BODY to a client:
--> use "dotwriter"

2) server receives article via IHAVE/TAKETHIS/POST
--> use "dotreader"
+ server writes data to storage
+ client requests ARTICLE/BODY: jump to 1) c).

3) client receives (reads) ARTICLE/BODY:
--> use "dotreader" and print it.

dotreader
https://cs.opensource.google/go/go/+/refs/tags/go1.21.1:src/net/textproto/reader.go;drc=1e43cfa15b4b618812e85c00c9e92c2615b324c8;l=448

// Dot by itself marks end; otherwise cut one dot.
if len(line) > 0 && line[0] == '.' {
if len(line) == 1 {
break
}
line = line[1:]
}

dotwriter
https://cs.opensource.google/go/go/+/refs/tags/go1.21.1:src/net/textproto/writer.go;drc=2580d0e08d5e9f979b943758d3c49877fb2324cb;l=67,

case wstateBegin, wstateBeginLine:
d.state = wstateData
if c == '.' {
// escape leading dot
bw.WriteByte('.')
}

Hi Billy,

> to send data use "dotwriter": add a dot to every leading dot.
>
> to read data use "dotreader": cut any first dot if line is not only a
> (closing) dot.

Exactly.

> i hope this is correct or did i miss something?
>
> 1) a) server sends article via IHAVE/TAKETHIS OR
>    b) client sends article via POST to server OR
>    c) server sends ARTICLE/BODY to a client:
>    --> use "dotwriter"
>
> 2) server receives article via IHAVE/TAKETHIS/POST
>    --> use "dotreader"
>        + server writes data to storage
>        + client requests ARTICLE/BODY: jump to 1) c).
>
> 3) client receives (reads) ARTICLE/BODY:
>    --> use "dotreader" and print it.

Also for HEAD when you say ARTICLE/BODY.

Though not related to articles, there would also be dot-stuffing to deal
with when reading/sending HELP and LIST MOTD.
Any time you're reading/sending a multi-line data block, dot-stuffing
applies (though it should not occur in other commands than those listed
above, as the syntax of newsgroup names, distributions, etc. does not
allow a leading dot).

--
Julien ÉLIE

« Sum, ergo bibo ; bibo, ergo sum. »

On 12.09.23 19:39, Julien ÉLIE wrote:
>
> Also for HEAD when you say ARTICLE/BODY.
>

hm but first char of a Header Line should by either a space to indicate
a continuing line or [A-Z] (maybe [a-z] for some strange (old) clients
which should not exist nowadays, in theory)?

any leading dot in the header should break when server receives it?

thanks!

"Billy G. (go-while)" <no-reply@no.spam> writes:
> On 12.09.23 19:39, Julien ÉLIE wrote:

>> Also for HEAD when you say ARTICLE/BODY.

> hm but first char of a Header Line should by either a space to indicate
> a continuing line or [A-Z] (maybe [a-z] for some strange (old) clients
> which should not exist nowadays, in theory)?

> any leading dot in the header should break when server receives it?

Nope, there is no such requirement on RFC 5322 header fields, and thus no
such requirement on RFC 5536 header fields because this is not one of the
places where netnews is stricter. RFC 5322 section 2.2:

Header fields are lines beginning with a field name, followed by a
colon (":"), followed by a field body, and terminated by CRLF. A
field name MUST be composed of printable US-ASCII characters (i.e.,
characters that have values between 33 and 126, inclusive), except
colon. A field body may be composed of printable US-ASCII characters
as well as the space (SP, ASCII value 32) and horizontal tab (HTAB,
ASCII value 9) characters (together known as the white space
characters, WSP). A field body MUST NOT include CR and LF except
when used in "folding" and "unfolding", as described in section
2.2.3. All field bodies MUST conform to the syntax described in
sections 3 and 4 of this specification.

So it's allowed to have a header field name that starts with a period, as
well as all sorts of other exotic and fascinating stuff that you don't see
in practice.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

On 12.09.23 20:49, Russ Allbery wrote:
> So it's allowed to have a header field name that starts with a period, as
> well as all sorts of other exotic and fascinating stuff that you don't see
> in practice.
>

great thanks your help is priceless!

Hi Russ,

> So it's allowed to have a header field name that starts with a period, as
> well as all sorts of other exotic and fascinating stuff that you don't see
> in practice.

Just tried, but looks like there's a bug in INN as headers are not
dot-stuffed when retrieved via for instance ARTICLE:

POST
[...]
...header-test: valid

Adding a dot-stuffed .header-test header field in headers.
... as well a a dot-stuffed line in the body.
..

ARTICLE
[...]
..header-test: valid

Adding a dot-stuffed .header-test header field in headers.
... as well a a dot-stuffed line in the body.
..

HDR .header-test 729-
225 Header information for .header-test follows (from articles)
729 valid
..

I'll have a look, as well as the computation of the :bytes metadata in
that case.

FWIW, Thunderbird correctly shows the .header-test header field.

--
Julien ÉLIE

« L'éternité, c'est long, surtout vers la fin. » (Woody Allen)

"Billy G. (go-while)" <no-reply@no.spam> writes:
> On 12.09.23 20:49, Russ Allbery wrote:

>> So it's allowed to have a header field name that starts with a period,
>> as well as all sorts of other exotic and fascinating stuff that you
>> don't see in practice.

> great thanks your help is priceless!

I will never have time to write this personally, but if someone with a
love of pedantic nit-picks ever felt like writing an NNTP and netnews
compliance test suite that tried all sorts of edge conditions like this
that could be run as a read/write NNTP client against a server (and
presumably also the target of a feed), that would be a real service to
everyone writing NNTP and netnews software.

I have from time to time thought about rewriting some of the random tools
I use, like tinyleaf, in Rust, but my thought process gets as far as
header parsing and then I groan and find other hobbies.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

In addition to my previous article:

> Just tried, but looks like there's a bug in INN as headers are not
> dot-stuffed when retrieved via for instance ARTICLE:
>
>
> POST
> [...]
> ..header-test: valid
>
> Adding a dot-stuffed .header-test header field in headers.
> .. as well as a dot-stuffed line in the body.
> .
>
>
>
> ARTICLE
> [...]
> .header-test: valid
>
> Adding a dot-stuffed .header-test header field in headers.
> .. as well as a dot-stuffed line in the body.
> .

Issue found, and appearing only with nnrpd. It uses two different
output methods for the headers (Towire) and the body (NNTPsendarticle).
Only the second one dot-stuffs lines when appropriate. I'll fix that.

innd (IHAVE/TAKETHIS) already correctly handles dot-stuffed header lines.

> FWIW, Thunderbird correctly shows the .header-test header field.

flnews too BTW.
And both Thunderbird and flnews are resilient with ".header-test"
(invalid in an ARTICLE response per RFC - but INN currently sends it
without doubling the initial dot) and "..header-test" (the valid syntax
in wire format). They display the ".header-test" header field in both
cases.

--
Julien ÉLIE

« Vinum bonum laetificat cor hominis. »