I've been on auto-pilot for a bit, but started checking logs/reports today and
notice an increasing number of articles rejected due to the following:

439 No colon-space in "User-Agent:" header field

When reviewing headers of the those messages there is a User-Agent: header,
but its blank and does not contain a space.

It is probably against a RFC to accept these messages, but I'm not concerned
about the User-Agent header. I went looking for a way to relax the check, but
is strict for all headers in innd/art.c:

/* Find first colon */
if ((colon = memchr(header, ':', size)) == NULL || !ISWHITE(colon[1])) {
if ((p = memchr(header, '\r', size)) != NULL)
*p = '\0';
snprintf(cp->Error, sizeof(cp->Error),
"%d No colon-space in \"%s\" header field",
ihave ? NNTP_FAIL_IHAVE_REJECT : NNTP_FAIL_TAKETHIS_REJECT,
MaxLength(header, header));
if (p != NULL)
*p = '\r';
return;
}

I know there are potential implications for accepting messages with empty
headers, but do we need to treat all headers so strictly?

Regards,

Jesse Rehmer

Jesse Rehmer <jesse.rehmer@blueworldhosting.com> writes:

> It is probably against a RFC to accept these messages, but I'm not
> concerned about the User-Agent header. I went looking for a way to relax
> the check, but is strict for all headers in innd/art.c:

> /* Find first colon */
> if ((colon = memchr(header, ':', size)) == NULL || !ISWHITE(colon[1])) {
> if ((p = memchr(header, '\r', size)) != NULL)
> *p = '\0';
> snprintf(cp->Error, sizeof(cp->Error),
> "%d No colon-space in \"%s\" header field",
> ihave ? NNTP_FAIL_IHAVE_REJECT : NNTP_FAIL_TAKETHIS_REJECT,
> MaxLength(header, header));
> if (p != NULL)
> *p = '\r';
> return;
> }

> I know there are potential implications for accepting messages with
> empty headers, but do we need to treat all headers so strictly?

We do need to be pretty strict, since otherwise you can get into really
nasty situations with ambiguous parses where two servers may disagree
about something fundamental like the message ID of the article.

I think if someone wanted to send a patch that added the ability to accept
articles with headers that (a) weren't part of the protocol (so not
Message-ID, Path, etc.), (b) specifically ended in a colon and a newline
and no other variation, and (c) did not have a continuation, and it was
configurable with the syntaxchecks setting and was off by default, that
would be worth considering. I wouldn't want to relax the checks any
farther than that. (It may be a bit difficult to wedge that into innd's
code, though, since the above error happens at a fairly low level of the
parse.)

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

On Wed, 01 Feb 2023 08:00:45 -0800
Russ Allbery <eagle@eyrie.org> wrote:

> Jesse Rehmer <jesse.rehmer@blueworldhosting.com> writes:
>
> > It is probably against a RFC to accept these messages, but I'm not
> > concerned about the User-Agent header. I went looking for a way to
> > relax the check, but is strict for all headers in innd/art.c:
>
> > /* Find first colon */
> > if ((colon = memchr(header, ':', size)) == NULL ||
> > !ISWHITE(colon[1])) { if ((p = memchr(header, '\r', size)) != NULL)
> > *p = '\0';
> > snprintf(cp->Error, sizeof(cp->Error),
> > "%d No colon-space in \"%s\" header field",
> > ihave ? NNTP_FAIL_IHAVE_REJECT :
> > NNTP_FAIL_TAKETHIS_REJECT, MaxLength(header, header));
> > if (p != NULL)
> > *p = '\r';
> > return;
> > }
>
> > I know there are potential implications for accepting messages with
> > empty headers, but do we need to treat all headers so strictly?
>
> We do need to be pretty strict, since otherwise you can get into
> really nasty situations with ambiguous parses where two servers may
> disagree about something fundamental like the message ID of the
> article.
>
> I think if someone wanted to send a patch that added the ability to
> accept articles with headers that (a) weren't part of the protocol
> (so not Message-ID, Path, etc.), (b) specifically ended in a colon
> and a newline and no other variation, and (c) did not have a
> continuation, and it was configurable with the syntaxchecks setting
> and was off by default, that would be worth considering. I wouldn't
> want to relax the checks any farther than that. (It may be a bit
> difficult to wedge that into innd's code, though, since the above
> error happens at a fairly low level of the parse.)

I wouldn't be happy about relaxing the checks or having a server feed
in that does. They're there for a reason. Most legitimate users are
going to follow the RFC. Spammers and script kiddies are less likely to
adhere to the requirements, which is where we get them.

The proper course of action is to identify if the messages are
legitimate and then refer the poster to their software's author to
correct the issue.

Once you start making exceptions for one, then you're going to be
making exceptions for others and, as you say, this could lead to a
nasty mess. Let's please not fix innd which isn't broken to fix a
client that is.

--
End Of The Line BBS - Plano, TX
telnet endofthelinebbs.com 23

On Feb 1, 2023 at 12:03:50 PM CST, "Nigel Reed" <sysop@endofthelinebbs.com>
wrote:

> On Wed, 01 Feb 2023 08:00:45 -0800
> Russ Allbery <eagle@eyrie.org> wrote:
>
>> Jesse Rehmer <jesse.rehmer@blueworldhosting.com> writes:
>>
>>> It is probably against a RFC to accept these messages, but I'm not
>>> concerned about the User-Agent header. I went looking for a way to
>>> relax the check, but is strict for all headers in innd/art.c:
>>
>>> /* Find first colon */
>>> if ((colon = memchr(header, ':', size)) == NULL ||
>>> !ISWHITE(colon[1])) { if ((p = memchr(header, '\r', size)) != NULL)
>>> *p = '\0';
>>> snprintf(cp->Error, sizeof(cp->Error),
>>> "%d No colon-space in \"%s\" header field",
>>> ihave ? NNTP_FAIL_IHAVE_REJECT :
>>> NNTP_FAIL_TAKETHIS_REJECT, MaxLength(header, header));
>>> if (p != NULL)
>>> *p = '\r';
>>> return;
>>> }
>>
>>> I know there are potential implications for accepting messages with
>>> empty headers, but do we need to treat all headers so strictly?
>>
>> We do need to be pretty strict, since otherwise you can get into
>> really nasty situations with ambiguous parses where two servers may
>> disagree about something fundamental like the message ID of the
>> article.
>>
>> I think if someone wanted to send a patch that added the ability to
>> accept articles with headers that (a) weren't part of the protocol
>> (so not Message-ID, Path, etc.), (b) specifically ended in a colon
>> and a newline and no other variation, and (c) did not have a
>> continuation, and it was configurable with the syntaxchecks setting
>> and was off by default, that would be worth considering. I wouldn't
>> want to relax the checks any farther than that. (It may be a bit
>> difficult to wedge that into innd's code, though, since the above
>> error happens at a fairly low level of the parse.)
>
> I wouldn't be happy about relaxing the checks or having a server feed
> in that does. They're there for a reason. Most legitimate users are
> going to follow the RFC. Spammers and script kiddies are less likely to
> adhere to the requirements, which is where we get them.
>
> The proper course of action is to identify if the messages are
> legitimate and then refer the poster to their software's author to
> correct the issue.
>
> Once you start making exceptions for one, then you're going to be
> making exceptions for others and, as you say, this could lead to a
> nasty mess. Let's please not fix innd which isn't broken to fix a
> client that is.

They are legimately posted articles in my eyes, but maybe not others.
Switching to Diablo may be best to acommodate my unique needs. INN with
cleanfeed or pyclean does not scale well with my traffic, so this is another
push to the dark side.

Hi Jesse,

> I've been on auto-pilot for a bit, but started checking logs/reports today and
> notice an increasing number of articles rejected due to the following:
>
> 439 No colon-space in "User-Agent:" header field
>
> When reviewing headers of the those messages there is a User-Agent: header,
> but its blank and does not contain a space.
>
> It is probably against a RFC to accept these messages

Indeed, they shouldn't have been accepted by injecting agents.
Some posting agents send empty header fields, which are usually stripped
off by injecting agents before injecting the article.

> but I'm not concerned
> about the User-Agent header. I went looking for a way to relax the check, but
> is strict for all headers in innd/art.c:
>
> /* Find first colon */
> if ((colon = memchr(header, ':', size)) == NULL || !ISWHITE(colon[1])) {
> if ((p = memchr(header, '\r', size)) != NULL)
> *p = '\0';
> snprintf(cp->Error, sizeof(cp->Error),
> "%d No colon-space in \"%s\" header field",
> ihave ? NNTP_FAIL_IHAVE_REJECT : NNTP_FAIL_TAKETHIS_REJECT,
> MaxLength(header, header));
> if (p != NULL)
> *p = '\r';
> return;
> }

If you're looking for a quick-and-dirty hack for that very header field,
I think the following check would do the trick (not tested):

@@ -644,10 +644,12 @@ ARTcheckheader(CHANNEL *cp, int size)
if ((colon = memchr(header, ':', size)) == NULL ||
!ISWHITE(colon[1])) {
if ((p = memchr(header, '\r', size)) != NULL)
*p = '\0';
+ if (strcasecmp(header, "User-Agent:") != 0 || colon[1] != '\0') {
snprintf(cp->Error, sizeof(cp->Error),
"%d No colon-space in \"%s\" header field",
ihave ? NNTP_FAIL_IHAVE_REJECT :
NNTP_FAIL_TAKETHIS_REJECT,
MaxLength(header, header));
+ }
if (p != NULL)
*p = '\r';
return;

--
Julien ÉLIE

« – Ce n'était pas ma question.
– C'était p'têt pas vot'question, oui, mais c'est ma réponse ! »
(Georges Marchais répondant à Alain Duhamel)

On Feb 3, 2023 at 2:03:14 PM CST, "Julien ÉLIE"
<iulius@nom-de-mon-site.com.invalid> wrote:

> Hi Jesse,
>
> If you're looking for a quick-and-dirty hack for that very header field,
> I think the following check would do the trick (not tested):
>
> @@ -644,10 +644,12 @@ ARTcheckheader(CHANNEL *cp, int size)
> if ((colon = memchr(header, ':', size)) == NULL ||
> !ISWHITE(colon[1])) {
> if ((p = memchr(header, '\r', size)) != NULL)
> *p = '\0';
> + if (strcasecmp(header, "User-Agent:") != 0 || colon[1] != '\0') {
> snprintf(cp->Error, sizeof(cp->Error),
> "%d No colon-space in \"%s\" header field",
> ihave ? NNTP_FAIL_IHAVE_REJECT :
> NNTP_FAIL_TAKETHIS_REJECT,
> MaxLength(header, header));
> + }
> if (p != NULL)
> *p = '\r';
> return;

Thank you, Julien. After examining a larger sampling of these articles, they
appear to be noise and I'm not concerned about dropping them.

I will keep this code stashed away though, it may come in handy in the future
as lazy developers write more client software that isn't compliant.

On 2/1/23 11:03 AM, Nigel Reed wrote:
> I wouldn't be happy about relaxing the checks or having a server feed
> in that does. They're there for a reason. Most legitimate users are
> going to follow the RFC. Spammers and script kiddies are less likely
> to adhere to the requirements, which is where we get them.
>
> The proper course of action is to identify if the messages are
> legitimate and then refer the poster to their software's author to
> correct the issue.
>
> Once you start making exceptions for one, then you're going to be
> making exceptions for others and, as you say, this could lead to a
> nasty mess. Let's please not fix innd which isn't broken to fix a
> client that is.

+10 to everything that Nigel said.

--
Grant. . . .
unix || die

On 2/1/23 9:00 AM, Russ Allbery wrote:
> We do need to be pretty strict, since otherwise you can get into really
> nasty situations with ambiguous parses where two servers may disagree
> about something fundamental like the message ID of the article.

Agreed.

> I think if someone wanted to send a patch that added the ability to
> accept articles with headers that (a) weren't part of the protocol
> (so not Message-ID, Path, etc.), (b) specifically ended in a colon and
> a newline and no other variation, and (c) did not have a continuation,
> and it was configurable with the syntaxchecks setting and was off by
> default, that would be worth considering.

I think my biggest hang up is the idea that the header is defined
(present) but doesn't have a value. -- My personal opinion is that if
a header is there, then there should be /something/ in it.

I don't know if a single space following the colon is as important to
me. I'd be willing to accept -- what SMTP RFCs define as -- CFWS
sequences between the delimiting colon and header contents.

> I wouldn't want to relax the checks any farther than that. (It may
> be a bit difficult to wedge that into innd's code, though, since the
> above error happens at a fairly low level of the parse.)

Other than possibly allowing CFWS, I think that the headers should all
be well formed.

--
Grant. . . .
unix || die

Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:
> I've been on auto-pilot for a bit, but started checking logs/reports today and
> notice an increasing number of articles rejected due to the following:
>
> 439 No colon-space in "User-Agent:" header field
>
> When reviewing headers of the those messages there is a User-Agent: header,
> but its blank and does not contain a space.
>
> It is probably against a RFC to accept these messages,

The colon is merely a separator between field name and field value. The
space afterward is only for human readability and is not mandated by the
RFCs. The whitespace following a colon is part of the field value and is
typically trimmed out.

Check the RFCs; people assume the space following a colon is mandatory, but
it's not.

On 2023-02-07, D Finnigan <dog_cow@macgui.com> wrote:
> Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:
>> I've been on auto-pilot for a bit, but started checking logs/reports today and
>> notice an increasing number of articles rejected due to the following:
>>
>> 439 No colon-space in "User-Agent:" header field
>>
>> When reviewing headers of the those messages there is a User-Agent: header,
>> but its blank and does not contain a space.
>>
>> It is probably against a RFC to accept these messages,
>
> The colon is merely a separator between field name and field value. The
> space afterward is only for human readability and is not mandated by the
> RFCs. The whitespace following a colon is part of the field value and is
> typically trimmed out.
>
> Check the RFCs; people assume the space following a colon is mandatory, but
> it's not.

I think you should check the RFCs before telling people off for not
doing so. Here is what the current RFC, RFC 5536, says (sec. 2.2):

o All agents MUST generate header fields so that at least one space
immediately follows the ':' separating the header field name and
the header field body (for compatibility with deployed software,
including NNTP [RFC3977] servers). News agents MAY accept header
fields that do not contain the required space.

Julian Bradfield <jcb@inf.ed.ac.uk> wrote:
> On 2023-02-07, D Finnigan <dog_cow@macgui.com> wrote:
>> Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:
>>> I've been on auto-pilot for a bit, but started checking logs/reports today and
>>> notice an increasing number of articles rejected due to the following:
>>>
>>> 439 No colon-space in "User-Agent:" header field
>>>
>>> When reviewing headers of the those messages there is a User-Agent: header,
>>> but its blank and does not contain a space.
>>>
>>> It is probably against a RFC to accept these messages,
>>
>> The colon is merely a separator between field name and field value. The
>> space afterward is only for human readability and is not mandated by the
>> RFCs. The whitespace following a colon is part of the field value and is
>> typically trimmed out.
>>
>> Check the RFCs; people assume the space following a colon is mandatory, but
>> it's not.
>
> I think you should check the RFCs before telling people off for not
> doing so. Here is what the current RFC, RFC 5536, says (sec. 2.2):
>

Ah, nicely done. I'd only looked at RFC 5322. Didn't know the NNTP one
differed.

D Finnigan <dog_cow@macgui.com> writes:

> Ah, nicely done. I'd only looked at RFC 5322. Didn't know the NNTP one
> differed.

In general, the Usenet article format RFC is compatible with email but
more restrictive, in some cases substantially more so. The email RFCs
still allow for a bunch of legacy syntax that historically was never
permitted by Usenet software and therefore is not allowed by the Usenet
article format RFC (like comments in the middle of email addresses, or no
space after the colon in headers).

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

Hi Jesse,

>> They are legimately posted articles in my eyes, but maybe not
>> others.
[...]

> After examining a larger sampling of these articles, they
> appear to be noise and I'm not concerned about dropping them.

OK, thanks for the confirmation!
Seems like to be a spam-filtering feature :)

I won't therefore take time to integrate a proper patch to allow empty
header fields for a few set of header fields that are parsed by servers
and clients (as proposed by Russ - not Path, Message-ID, Date,
Cancel-Lock, Distribution, Control, etc.).

> I will keep this code stashed away though, it may come in handy in the future
> as lazy developers write more client software that isn't compliant.

Do not hesitate to tell if you happen to find legitimate articles
rejected because of that check.

Also, does it concern many posters or always the same ones? Maybe there
are all using the same newsreader, which should directly be fixed.
If that's the case, they should be made aware of that bug in their
messages (by responding to them in the newsgroup or by mail if they
provide one).

--
Julien ÉLIE

« – Où vous croyez-vous ici ?
– Où je me trouve, je sais. » (Astérix)