I recently switched to Diablo on the transit layer, but am observing it is
rejecting messages that contain PGP as binary articles. This is preventing
valid messages, such as NoCeM notices and messages that aren't encrypted but
contain a PGP Signature, to be rejected in some cases, or not propagated to
peers (due to "arttypes !binary" in the dnewsfeed definition).

In lib/arttype.c, I see there is a specific check to identify an article with
an article type of pgp, but this only checks for "-----BEGIN PGP
MESSAGE-----".

Looking at the contents of articles I am rejecting or not propagating, I
believe the check should also look the following:

"-----BEGIN PGP SIGNED MESSAGE-----"
and
"-----BEGIN PGP SIGNATURE-----"

but I am not a developer and can't quite cobble anything together from the
existing code that works to add these additional values.

If anyone out there is willing to help me with a patch, I will be forever
grateful!

This is the portion of lib/arttype.c that is causing the misidentification:

/*
* We look for binary formats first, since these eat
* up CPU like there is no tomorrow. We may miss
* some flags if we detect a binary file, the
* alternative is to remove the return(Type)'s and
* eat CPU for the entire binary.
*/

linetype = classifyLineAsTypes(ptr, len);

if (linetype & UUE) {
Base64 = 0;
BinHex = 0;
UUencode++;
if (UUencode > 8) {
Type |= ARTTYPE_UUENCODE | ARTTYPE_BINARY;
return(Type);
}
} else if (linetype & BHX) {
UUencode = 0;
Base64 = 0;
BinHex++;
if (BinHex > 8) {
Type |= ARTTYPE_BINHEX | ARTTYPE_BINARY;
return(Type);
}
} else if (linetype & B64) {
UUencode = 0;
BinHex = 0;
Base64++;
if (Base64 > 8) {
Type |= ARTTYPE_BASE64 | ARTTYPE_BINARY;
return(Type);
}
} else {
if (*ptr == '-' &&
strncmp(ptr, "-----BEGIN PGP MESSAGE-----", 27) == 0)
Type |= ARTTYPE_PGPMESSAGE;
UUencode = 0;
Base64 = 0;
BinHex = 0;
}
}

Warm Regards,

Jesse Rehmer

Jesse Rehmer wrote:

> I recently switched to Diablo on the transit layer, but am observing it is
> rejecting messages that contain PGP as binary articles. This is preventing
> valid messages, such as NoCeM notices and messages that aren't encrypted but
> contain a PGP Signature, to be rejected in some cases, or not propagated to
> peers (due to "arttypes !binary" in the dnewsfeed definition).

> In lib/arttype.c, I see there is a specific check to identify an article with
> an article type of pgp, but this only checks for "-----BEGIN PGP
> MESSAGE-----".

> Looking at the contents of articles I am rejecting or not propagating, I
> believe the check should also look the following:

> "-----BEGIN PGP SIGNED MESSAGE-----"
> and
> "-----BEGIN PGP SIGNATURE-----"

> but I am not a developer and can't quite cobble anything together from the
> existing code that works to add these additional values.

> If anyone out there is willing to help me with a patch, I will be forever
> grateful!

> This is the portion of lib/arttype.c that is causing the misidentification:

> /*
> * We look for binary formats first, since these eat
> * up CPU like there is no tomorrow. We may miss
> * some flags if we detect a binary file, the
> * alternative is to remove the return(Type)'s and
> * eat CPU for the entire binary.
> */

> linetype = classifyLineAsTypes(ptr, len);

> if (linetype & UUE) {
> Base64 = 0;
> BinHex = 0;
> UUencode++;
> if (UUencode > 8) {
> Type |= ARTTYPE_UUENCODE | ARTTYPE_BINARY;
> return(Type);
> }
> } else if (linetype & BHX) {
> UUencode = 0;
> Base64 = 0;
> BinHex++;
> if (BinHex > 8) {
> Type |= ARTTYPE_BINHEX | ARTTYPE_BINARY;
> return(Type);
> }
> } else if (linetype & B64) {
> UUencode = 0;
> BinHex = 0;
> Base64++;
> if (Base64 > 8) {
> Type |= ARTTYPE_BASE64 | ARTTYPE_BINARY;
> return(Type);
> }
> } else {
> if (*ptr == '-' &&
> strncmp(ptr, "-----BEGIN PGP MESSAGE-----", 27) == 0)
> Type |= ARTTYPE_PGPMESSAGE;
> UUencode = 0;
> Base64 = 0;
> BinHex = 0;
> }
> }

Maybe just change:

if (*ptr == '-' &&
strncmp(ptr, "-----BEGIN PGP MESSAGE-----", 27) == 0)
Type |= ARTTYPE_PGPMESSAGE;

to:

if (*ptr == '-' &&
strncmp(ptr, "-----BEGIN PGP ", 15) == 0)
Type |= ARTTYPE_PGPMESSAGE;

--
Retro Guy

On Mar 19, 2023 at 8:30:32 PM CDT, "Retro Guy" <Retro Guy> wrote:

> Maybe just change:
>
> if (*ptr == '-' &&
> strncmp(ptr, "-----BEGIN PGP MESSAGE-----", 27) == 0)
> Type |= ARTTYPE_PGPMESSAGE;
>
> to:
>
> if (*ptr == '-' &&
> strncmp(ptr, "-----BEGIN PGP ", 15) == 0)
> Type |= ARTTYPE_PGPMESSAGE;

Thanks for that, not being a C developer I didn't even put together that "27"
was the number of characters in the string, didn't know how to account for
that, and now feel like a n00b.

I will recompile and give this a shot, I really appreciate it!

-Jesse

On Mar 19, 2023 at 9:05:25 PM CDT, "Jesse Rehmer"
<jesse.rehmer@blueworldhosting.com> wrote:

> On Mar 19, 2023 at 8:30:32 PM CDT, "Retro Guy" <Retro Guy> wrote:
>
>> Maybe just change:
>>
>> if (*ptr == '-' &&
>> strncmp(ptr, "-----BEGIN PGP MESSAGE-----", 27) == 0)
>> Type |= ARTTYPE_PGPMESSAGE;
>>
>> to:
>>
>> if (*ptr == '-' &&
>> strncmp(ptr, "-----BEGIN PGP ", 15) == 0)
>> Type |= ARTTYPE_PGPMESSAGE;
>
> Thanks for that, not being a C developer I didn't even put together that "27"
> was the number of characters in the string, didn't know how to account for
> that, and now feel like a n00b.
>
> I will recompile and give this a shot, I really appreciate it!
>
> -Jesse

Hrm, I guess another portion of the code higher up is taking precedence as
this didn't change the behavior with messages like:
<tv8i7j$q8m$1@rasp.pasdenom.info>

On Mar 19, 2023 at 10:25:12 PM CDT, "Jesse Rehmer"
<jesse.rehmer@blueworldhosting.com> wrote:

> On Mar 19, 2023 at 9:05:25 PM CDT, "Jesse Rehmer"
> <jesse.rehmer@blueworldhosting.com> wrote:
>
>> On Mar 19, 2023 at 8:30:32 PM CDT, "Retro Guy" <Retro Guy> wrote:
>>
>>> Maybe just change:
>>>
>>> if (*ptr == '-' &&
>>> strncmp(ptr, "-----BEGIN PGP MESSAGE-----", 27) == 0)
>>> Type |= ARTTYPE_PGPMESSAGE;
>>>
>>> to:
>>>
>>> if (*ptr == '-' &&
>>> strncmp(ptr, "-----BEGIN PGP ", 15) == 0)
>>> Type |= ARTTYPE_PGPMESSAGE;
>>
>> Thanks for that, not being a C developer I didn't even put together that "27"
>> was the number of characters in the string, didn't know how to account for
>> that, and now feel like a n00b.
>>
>> I will recompile and give this a shot, I really appreciate it!
>>
>> -Jesse
>
> Hrm, I guess another portion of the code higher up is taking precedence as
> this didn't change the behavior with messages like:
> <tv8i7j$q8m$1@rasp.pasdenom.info>

I'm thinking it is the portion of code right above the PGP stuff:

} else if (linetype & B64) {
UUencode = 0;
BinHex = 0;
Base64++;
if (Base64 > 8) {
Type |= ARTTYPE_BASE64 | ARTTYPE_BINARY;
return(Type);
}

I'm assuming this is counting any article with more than 8 lines of Base64
encoded text as binary?

On 2023-03-20, Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:
> On Mar 19, 2023 at 10:25:12 PM CDT, "Jesse Rehmer"
>>
>> Hrm, I guess another portion of the code higher up is taking precedence as
>> this didn't change the behavior with messages like:
>> <tv8i7j$q8m$1@rasp.pasdenom.info>
>
> I'm thinking it is the portion of code right above the PGP stuff:
>
> } else if (linetype & B64) {
> UUencode = 0;
> BinHex = 0;
> Base64++;
> if (Base64 > 8) {
> Type |= ARTTYPE_BASE64 | ARTTYPE_BINARY;
> return(Type);
> }
>
> I'm assuming this is counting any article with more than 8 lines of Base64
> encoded text as binary?

That certainly seems to be the case. I'd be tempted to switch the order
of this whole 'if-else' sequence so the check for "-----BEGIN PGP ..."
comes before the binary tests.

Cheers,
Tom

On Mar 20, 2023 at 5:36:15 AM CDT, "Tom Furie" <tom@furie.org.uk> wrote:

> On 2023-03-20, Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:
>> On Mar 19, 2023 at 10:25:12 PM CDT, "Jesse Rehmer"
>>>
>>> Hrm, I guess another portion of the code higher up is taking precedence as
>>> this didn't change the behavior with messages like:
>>> <tv8i7j$q8m$1@rasp.pasdenom.info>
>>
>> I'm thinking it is the portion of code right above the PGP stuff:
>>
>> } else if (linetype & B64) {
>> UUencode = 0;
>> BinHex = 0;
>> Base64++;
>> if (Base64 > 8) {
>> Type |= ARTTYPE_BASE64 | ARTTYPE_BINARY;
>> return(Type);
>> }
>>
>> I'm assuming this is counting any article with more than 8 lines of Base64
>> encoded text as binary?
>
> That certainly seems to be the case. I'd be tempted to switch the order
> of this whole 'if-else' sequence so the check for "-----BEGIN PGP ..."
> comes before the binary tests.
>
> Cheers,
> Tom

I may give this a try. I ended up disabling the IFILTER feed/label and it
stopped the behavior.

I'm a bit confused by it's intended behavior, as I thought it was only for
rejecting misplaced binaries, but looking at the logs I see it is identifying
these particular messages as some other type, but according to what is in
lib/defs.h, I can't quite make heads or tails of how these translate.

Take these two entries for example:

2023-03-20 01:00:03.737 somehost - <tv8sp3$fql$1@rasp.pasdenom.info> 2642
020b00 IncomingFilter
2023-03-20 01:00:03.738 somehost - <tv8sp3$fql$1@rasp.pasdenom.info> 2656
020b00 IncomingFilter

The article type logged is "020b00", which that string does not appear
anywhere in the source code. In lib/defs.h there is the following block:

/*
* Article types that diablo can determine when receiving an article
*/

#define ARTTYPE_NONE 0x000000

#define ARTTYPE_DEFAULT 0x000001
#define ARTTYPE_CONTROL 0x000002
#define ARTTYPE_CANCEL 0x000004

#define ARTTYPE_MIME 0x000100
#define ARTTYPE_BINARY 0x000200
#define ARTTYPE_UUENCODE 0x000400
#define ARTTYPE_BASE64 0x000800
#define ARTTYPE_MULTIPART 0x001000
#define ARTTYPE_HTML 0x002000
#define ARTTYPE_PS 0x004000
#define ARTTYPE_BINHEX 0x008000
#define ARTTYPE_PARTIAL 0x010000
#define ARTTYPE_PGPMESSAGE 0x020000
#define ARTTYPE_YENC 0x040000
#define ARTTYPE_BOMMANEWS 0x080000
#define ARTTYPE_UNIDATA 0x100000

#define ARTTYPE_ALL 0xFFFFFF

Other articles that are misplaced binaries show up in the logs like this:

2023-03-20 02:50:35.991 somehost -
<001d0260a48f41ff95a5dd1ff1830a4d@lfxlxgf.ytv> 1737 050200 IncomingFilter

Where the article type is logged as "050200" but again, I'm having a hard time
understanding what this is truly supposed to indicated as its detected type.

I may be completely misunderstanding how the article type detection works and
how it is used.

On 2023-03-20, Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:
> I'm a bit confused by it's intended behavior, as I thought it was only for
> rejecting misplaced binaries, but looking at the logs I see it is
> identifying these particular messages as some other type, but according to
> what is in lib/defs.h, I can't quite make heads or tails of how these
> translate.
>
> Take these two entries for example:
>
> 2023-03-20 01:00:03.737 somehost - <tv8sp3$fql$1@rasp.pasdenom.info> 2642
> 020b00 IncomingFilter
> 2023-03-20 01:00:03.738 somehost - <tv8sp3$fql$1@rasp.pasdenom.info> 2656
> 020b00 IncomingFilter
>
> The article type logged is "020b00", which that string does not appear
> anywhere in the source code. In lib/defs.h there is the following block:
>
> /*
> * Article types that diablo can determine when receiving an article
> */
>
> #define ARTTYPE_NONE 0x000000
>
> #define ARTTYPE_DEFAULT 0x000001
> #define ARTTYPE_CONTROL 0x000002
> #define ARTTYPE_CANCEL 0x000004
>
> #define ARTTYPE_MIME 0x000100
> #define ARTTYPE_BINARY 0x000200
> #define ARTTYPE_UUENCODE 0x000400
> #define ARTTYPE_BASE64 0x000800
> #define ARTTYPE_MULTIPART 0x001000
> #define ARTTYPE_HTML 0x002000
> #define ARTTYPE_PS 0x004000
> #define ARTTYPE_BINHEX 0x008000
> #define ARTTYPE_PARTIAL 0x010000
> #define ARTTYPE_PGPMESSAGE 0x020000
> #define ARTTYPE_YENC 0x040000
> #define ARTTYPE_BOMMANEWS 0x080000
> #define ARTTYPE_UNIDATA 0x100000
>
> #define ARTTYPE_ALL 0xFFFFFF

This looks like a bitfield map, so the 020b00 you see above equates to a sum
of PGPMESSAGE + BASE64 + BINARY + MIME. At least it's recognised the PGP
element. The size of the signature (greater than the 8 line limit in the
earlier check) seems to be what triggers the filter to flag the article as
binary.

> Other articles that are misplaced binaries show up in the logs like this:
>
> 2023-03-20 02:50:35.991 somehost -
> <001d0260a48f41ff95a5dd1ff1830a4d@lfxlxgf.ytv> 1737 050200 IncomingFilter
>
> Where the article type is logged as "050200" but again, I'm having a hard
> time understanding what this is truly supposed to indicated as its
> detected type.

That 050200 says the article is a partial binary in yenc encoding.

Cheers,
Tom

On Mar 20, 2023 at 11:37:46 AM CDT, "Tom Furie" <tom@furie.org.uk> wrote:

> On 2023-03-20, Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:
>> I'm a bit confused by it's intended behavior, as I thought it was only for
>> rejecting misplaced binaries, but looking at the logs I see it is
>> identifying these particular messages as some other type, but according to
>> what is in lib/defs.h, I can't quite make heads or tails of how these
>> translate.
>>
>> Take these two entries for example:
>>
>> 2023-03-20 01:00:03.737 somehost - <tv8sp3$fql$1@rasp.pasdenom.info> 2642
>> 020b00 IncomingFilter
>> 2023-03-20 01:00:03.738 somehost - <tv8sp3$fql$1@rasp.pasdenom.info> 2656
>> 020b00 IncomingFilter
>>
>> The article type logged is "020b00", which that string does not appear
>> anywhere in the source code. In lib/defs.h there is the following block:
>>
>> /*
>> * Article types that diablo can determine when receiving an article
>> */
>>
>> #define ARTTYPE_NONE 0x000000
>>
>> #define ARTTYPE_DEFAULT 0x000001
>> #define ARTTYPE_CONTROL 0x000002
>> #define ARTTYPE_CANCEL 0x000004
>>
>> #define ARTTYPE_MIME 0x000100
>> #define ARTTYPE_BINARY 0x000200
>> #define ARTTYPE_UUENCODE 0x000400
>> #define ARTTYPE_BASE64 0x000800
>> #define ARTTYPE_MULTIPART 0x001000
>> #define ARTTYPE_HTML 0x002000
>> #define ARTTYPE_PS 0x004000
>> #define ARTTYPE_BINHEX 0x008000
>> #define ARTTYPE_PARTIAL 0x010000
>> #define ARTTYPE_PGPMESSAGE 0x020000
>> #define ARTTYPE_YENC 0x040000
>> #define ARTTYPE_BOMMANEWS 0x080000
>> #define ARTTYPE_UNIDATA 0x100000
>>
>> #define ARTTYPE_ALL 0xFFFFFF
>
> This looks like a bitfield map, so the 020b00 you see above equates to a sum
> of PGPMESSAGE + BASE64 + BINARY + MIME. At least it's recognised the PGP
> element. The size of the signature (greater than the 8 line limit in the
> earlier check) seems to be what triggers the filter to flag the article as
> binary.

Ahhh thank you, this makes sense.

I did move the PGP related if statements above the others as you had suggested
earlier. So far, so good, but may take a while to know for sure. The changes
didn't break the binary detection, so there's that.

I'm curious if there's any known reason why the developers chose 8 lines for
the checks? I assume this is somewhat arbitrary, but not certain, and thinking
if I'm still having trouble raising that value may be a better solution than
trying to make the code do things I don't fully understand.

On Mar 20, 2023 at 12:22:43 PM CDT, "Jesse Rehmer"
<jesse.rehmer@blueworldhosting.com> wrote:

> On Mar 20, 2023 at 11:37:46 AM CDT, "Tom Furie" <tom@furie.org.uk> wrote:
>
>> On 2023-03-20, Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:
>>> I'm a bit confused by it's intended behavior, as I thought it was only for
>>> rejecting misplaced binaries, but looking at the logs I see it is
>>> identifying these particular messages as some other type, but according to
>>> what is in lib/defs.h, I can't quite make heads or tails of how these
>>> translate.
>>>
>>> Take these two entries for example:
>>>
>>> 2023-03-20 01:00:03.737 somehost - <tv8sp3$fql$1@rasp.pasdenom.info> 2642
>>> 020b00 IncomingFilter
>>> 2023-03-20 01:00:03.738 somehost - <tv8sp3$fql$1@rasp.pasdenom.info> 2656
>>> 020b00 IncomingFilter
>>>
>>> The article type logged is "020b00", which that string does not appear
>>> anywhere in the source code. In lib/defs.h there is the following block:
>>>
>>> /*
>>> * Article types that diablo can determine when receiving an article
>>> */
>>>
>>> #define ARTTYPE_NONE 0x000000
>>>
>>> #define ARTTYPE_DEFAULT 0x000001
>>> #define ARTTYPE_CONTROL 0x000002
>>> #define ARTTYPE_CANCEL 0x000004
>>>
>>> #define ARTTYPE_MIME 0x000100
>>> #define ARTTYPE_BINARY 0x000200
>>> #define ARTTYPE_UUENCODE 0x000400
>>> #define ARTTYPE_BASE64 0x000800
>>> #define ARTTYPE_MULTIPART 0x001000
>>> #define ARTTYPE_HTML 0x002000
>>> #define ARTTYPE_PS 0x004000
>>> #define ARTTYPE_BINHEX 0x008000
>>> #define ARTTYPE_PARTIAL 0x010000
>>> #define ARTTYPE_PGPMESSAGE 0x020000
>>> #define ARTTYPE_YENC 0x040000
>>> #define ARTTYPE_BOMMANEWS 0x080000
>>> #define ARTTYPE_UNIDATA 0x100000
>>>
>>> #define ARTTYPE_ALL 0xFFFFFF
>>
>> This looks like a bitfield map, so the 020b00 you see above equates to a sum
>> of PGPMESSAGE + BASE64 + BINARY + MIME. At least it's recognised the PGP
>> element. The size of the signature (greater than the 8 line limit in the
>> earlier check) seems to be what triggers the filter to flag the article as
>> binary.
>
> Ahhh thank you, this makes sense.
>
> I did move the PGP related if statements above the others as you had suggested
> earlier. So far, so good, but may take a while to know for sure. The changes
> didn't break the binary detection, so there's that.
>
> I'm curious if there's any known reason why the developers chose 8 lines for
> the checks? I assume this is somewhat arbitrary, but not certain, and thinking
> if I'm still having trouble raising that value may be a better solution than
> trying to make the code do things I don't fully understand.

I backed out several of the changes I had made that weren't successful. I
believe moving the if statement for the PGP stuff at the top of this block and
adding "return(Type);" is working as expected. I was able to re-send some
articles that were previously rejected and they made it through.

How syntactically correct it is, I don't know, but I'm still seeing it filter
misplaced binaries and other useless Base64 encoded articles like I want, but
so far no PGP messages have been filtered.

/*
* We look for binary formats first, since these eat
* up CPU like there is no tomorrow. We may miss
* some flags if we detect a binary file, the
* alternative is to remove the return(Type)'s and
* eat CPU for the entire binary.
*/

linetype = classifyLineAsTypes(ptr, len);

if (*ptr == '-' &&
strncmp(ptr, "-----BEGIN PGP ", 15) == 0)
Type |= ARTTYPE_PGPMESSAGE;
return(Type);
UUencode = 0;
Base64 = 0;
BinHex = 0;
if (linetype & UUE) {
Base64 = 0;
BinHex = 0;
UUencode++;
if (UUencode > 8) {
Type |= ARTTYPE_UUENCODE | ARTTYPE_BINARY;
return(Type);
}
} else if (linetype & BHX) {
UUencode = 0;
Base64 = 0;
BinHex++;
if (BinHex > 8) {
Type |= ARTTYPE_BINHEX | ARTTYPE_BINARY;
return(Type);
}

Jesse Rehmer <jesse.rehmer@blueworldhosting.com> writes:

> linetype = classifyLineAsTypes(ptr, len);

> if (*ptr == '-' &&
> strncmp(ptr, "-----BEGIN PGP ", 15) == 0)
> Type |= ARTTYPE_PGPMESSAGE;
> return(Type);

Just FYI, by doing it this way you are allowing people to send you
arbitrarily large binaries (up to your article size limit) by just adding
a line starting with "-----BEGIN PGP " before the encoded binary. This is
likely why the code didn't short-circuit analysis of the rest of the
article in that case and still flagged messages as binary if there were
more than 8 lines of encoded text. (No PGP *signature* should be that
long.)

Back in the day, if any change like this were used on a bunch of servers,
the people trading binaries that for legal reasons you really do not want
on your server would probably quickly start doing that. You're only
making this change on one server, and hopefully there isn't as much abuse
of binaries on Usenet these days, but you may still want to keep that in
mind.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

Jesse Rehmer wrote:

looks like missing braces

> if (*ptr == '-' &&
> strncmp(ptr, "-----BEGIN PGP ", 15) == 0)
{
> Type |= ARTTYPE_PGPMESSAGE;
> return(Type);
}

On Mar 20, 2023 at 8:05:58 PM CDT, "Russ Allbery" <eagle@eyrie.org> wrote:

> Jesse Rehmer <jesse.rehmer@blueworldhosting.com> writes:
>
>> linetype = classifyLineAsTypes(ptr, len);
>
>> if (*ptr == '-' &&
>> strncmp(ptr, "-----BEGIN PGP ", 15) == 0)
>> Type |= ARTTYPE_PGPMESSAGE;
>> return(Type);
>
> Just FYI, by doing it this way you are allowing people to send you
> arbitrarily large binaries (up to your article size limit) by just adding
> a line starting with "-----BEGIN PGP " before the encoded binary. This is
> likely why the code didn't short-circuit analysis of the rest of the
> article in that case and still flagged messages as binary if there were
> more than 8 lines of encoded text. (No PGP *signature* should be that
> long.)

Most PGP Signatures I'm seeing in NoCeM notices are 11 lines long. See
<tvatqb$d4n$1@rasp.pasdenom.info> as just one example. These were being
rejected before.

My personal PGP Signature is > 8 lines as well.

> Back in the day, if any change like this were used on a bunch of servers,
> the people trading binaries that for legal reasons you really do not want
> on your server would probably quickly start doing that. You're only
> making this change on one server, and hopefully there isn't as much abuse
> of binaries on Usenet these days, but you may still want to keep that in
> mind.

I understand your points, and need to think this through a little more.

Jesse Rehmer <jesse.rehmer@blueworldhosting.com> writes:
> On Mar 20, 2023 at 8:05:58 PM CDT, "Russ Allbery" <eagle@eyrie.org> wrote:

>> Just FYI, by doing it this way you are allowing people to send you
>> arbitrarily large binaries (up to your article size limit) by just
>> adding a line starting with "-----BEGIN PGP " before the encoded
>> binary. This is likely why the code didn't short-circuit analysis of
>> the rest of the article in that case and still flagged messages as
>> binary if there were more than 8 lines of encoded text. (No PGP
>> *signature* should be that long.)

> Most PGP Signatures I'm seeing in NoCeM notices are 11 lines long. See
> <tvatqb$d4n$1@rasp.pasdenom.info> as just one example. These were being
> rejected before.

> My personal PGP Signature is > 8 lines as well.

Ah, yes, indeed, I'm wrong about how long the RSA signatures are (I assume
that's what that is; I didn't check) with a sufficiently large key. It
looks like, back in the day, I used a 40 line cutoff in my binary filter.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

On Mar 20, 2023 at 10:09:41 PM CDT, "Russ Allbery" <eagle@eyrie.org> wrote:

> Jesse Rehmer <jesse.rehmer@blueworldhosting.com> writes:
>> On Mar 20, 2023 at 8:05:58 PM CDT, "Russ Allbery" <eagle@eyrie.org> wrote:
>
>>> Just FYI, by doing it this way you are allowing people to send you
>>> arbitrarily large binaries (up to your article size limit) by just
>>> adding a line starting with "-----BEGIN PGP " before the encoded
>>> binary. This is likely why the code didn't short-circuit analysis of
>>> the rest of the article in that case and still flagged messages as
>>> binary if there were more than 8 lines of encoded text. (No PGP
>>> *signature* should be that long.)
>
>> Most PGP Signatures I'm seeing in NoCeM notices are 11 lines long. See
>> <tvatqb$d4n$1@rasp.pasdenom.info> as just one example. These were being
>> rejected before.
>
>> My personal PGP Signature is > 8 lines as well.
>
> Ah, yes, indeed, I'm wrong about how long the RSA signatures are (I assume
> that's what that is; I didn't check) with a sufficiently large key. It
> looks like, back in the day, I used a 40 line cutoff in my binary filter.

I did take your advice, reverted to the original code, and increased the
base64 line count. Should have realized it was that simple in the first place,
but I wasn't sure if that 8 line reference was a standard or arbitrary from
back in the day.

Really appreciate everyone's help here on this, it frustrated the shit out of
me, and felt like it should be simple, and naturally it was.

Thanks all!

Jesse Rehmer <jesse.rehmer@blueworldhosting.com> writes:

> I did take your advice, reverted to the original code, and increased the
> base64 line count. Should have realized it was that simple in the first
> place, but I wasn't sure if that 8 line reference was a standard or
> arbitrary from back in the day.

I'm fairly sure it's just arbitrary. Definitely not a formal standard.

My theory was that you can't pack a binary that would cause any real
trouble into only 40 lines of base64. I'm not 100% sure that's true
depending on whether implementations allow very long lines, but meh, this
stuff is all kind of fiddly anyway.

I suspect that Diablo code may be too old to detect yEnc, so you may still
be getting binaries. (I assume yEnc is still a thing people use; I am
very out of touch with anything that's happening over in binary land.)

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

On Mar 20, 2023 at 10:39:35 PM CDT, "Russ Allbery" <eagle@eyrie.org> wrote:

> Jesse Rehmer <jesse.rehmer@blueworldhosting.com> writes:
>
>> I did take your advice, reverted to the original code, and increased the
>> base64 line count. Should have realized it was that simple in the first
>> place, but I wasn't sure if that 8 line reference was a standard or
>> arbitrary from back in the day.
>
> I'm fairly sure it's just arbitrary. Definitely not a formal standard.
>
> My theory was that you can't pack a binary that would cause any real
> trouble into only 40 lines of base64. I'm not 100% sure that's true
> depending on whether implementations allow very long lines, but meh, this
> stuff is all kind of fiddly anyway.
>
> I suspect that Diablo code may be too old to detect yEnc, so you may still
> be getting binaries. (I assume yEnc is still a thing people use; I am
> very out of touch with anything that's happening over in binary land.)

It's very good at detecting yEnc (still the most popular format).

Not sure how much I care about alt.anonymous.messages and similar groups, but
I see quite a few PGP messages that I accepted while running pyClean and
Cleanfeed in my spool that are >1,000 lines. Not sure how safe it would be to
modify the Diablo base64 filter to allow that many lines. I guess if I cared
about getting those messages (currently I do not) I could add a delgroupany
statements for those specific groups to the IFILTER feed label.

Jesse Rehmer <jesse.rehmer@blueworldhosting.com> writes:
> On Mar 20, 2023 at 10:39:35 PM CDT, "Russ Allbery" <eagle@eyrie.org> wrote:

>> I suspect that Diablo code may be too old to detect yEnc, so you may
>> still be getting binaries. (I assume yEnc is still a thing people use;
>> I am very out of touch with anything that's happening over in binary
>> land.)

> It's very good at detecting yEnc (still the most popular format).

Oh, good!

> Not sure how much I care about alt.anonymous.messages and similar
> groups, but I see quite a few PGP messages that I accepted while running
> pyClean and Cleanfeed in my spool that are >1,000 lines.

My personal opinion is that those groups are silly and I don't feel the
need to use my resources to support them, but that's just me. I think it
takes some real mental contortions to convince yourself that the best
solution to communicating privately with someone else without being
vulnerable to traffic analysis is to convince a bunch of people all over
the world to pass all the messages around to everyone, almost none of whom
can read anything there. It has very strong https://xkcd.com/538/ energy.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

On Mar 20, 2023 at 11:14:00 PM CDT, "Russ Allbery" <eagle@eyrie.org> wrote:

> Jesse Rehmer <jesse.rehmer@blueworldhosting.com> writes:
>> On Mar 20, 2023 at 10:39:35 PM CDT, "Russ Allbery" <eagle@eyrie.org> wrote:
>
>>> I suspect that Diablo code may be too old to detect yEnc, so you may
>>> still be getting binaries. (I assume yEnc is still a thing people use;
>>> I am very out of touch with anything that's happening over in binary
>>> land.)
>
>> It's very good at detecting yEnc (still the most popular format).
>
> Oh, good!
>
>> Not sure how much I care about alt.anonymous.messages and similar
>> groups, but I see quite a few PGP messages that I accepted while running
>> pyClean and Cleanfeed in my spool that are >1,000 lines.
>
> My personal opinion is that those groups are silly and I don't feel the
> need to use my resources to support them, but that's just me. I think it
> takes some real mental contortions to convince yourself that the best
> solution to communicating privately with someone else without being
> vulnerable to traffic analysis is to convince a bunch of people all over
> the world to pass all the messages around to everyone, almost none of whom
> can read anything there. It has very strong https://xkcd.com/538/ energy.

I got a good chuckle and share the sentiment. That said, I offer my spool up
for reading to anyone without authentication, so who knows, but it seems
unlikely anyone actually reading or exchanging those messages would complain
to me if they weren't there.