Hi,

Just to make sure, because I can't find this information anywhere...

When is filter_innd.pl used? Is it used only for articles coming from
other servers (via a newsfeed), or also for locally posted articles?

If it's the latter, then where in this picture filter_nnrpd.pl fits in?
What if both are enabled, and someone posts the article (via nnrpd)?
filter_nnrpd.pl is executed, but is filter_innd.pl executed also?

Why am I asking...

I have cleanfeed installed. filter_innd.pl is a symlink to cleanfeed. I
would expect cleanfeed to reject certain posts posted to my server via
nnrpd (for example, binary postings to non-binary groups), and I vaguely
remember that it worked this way, but now it doesn't happen. Server
happily accepts these posts.

In logs I can only see that filtering is enabled.

Jun 8 16:16:16 kvm innd: SERVER perl filtering enabled

cleanfeed directory contains logs subdirectory, but it's empty.

What can be wrong here?

Adam W. <gof-cut-this-news@cut-this-chmurka.net.invalid> wrote:

> I have cleanfeed installed. filter_innd.pl is a symlink to cleanfeed. I
> would expect cleanfeed to reject certain posts posted to my server via
> nnrpd (for example, binary postings to non-binary groups), and I vaguely
> remember that it worked this way, but now it doesn't happen. Server
> happily accepts these posts.

An update.

When I try to post an article with a forbidden subject (simpbiz.software),
it gets rejected, so the filtering is enabled. But still, binary postings
get through.

I'm testing it with (prefixing here with : so it doesn't get flagged as a
binary posting):

: begin 664 test
: `
: end

gof-cut-this-news@cut-this-chmurka.net.invalid (Adam W.) writes:

> When is filter_innd.pl used? Is it used only for articles coming from
> other servers (via a newsfeed), or also for locally posted articles?

It's used for both.

> If it's the latter, then where in this picture filter_nnrpd.pl fits in?
> What if both are enabled, and someone posts the article (via nnrpd)?
> filter_nnrpd.pl is executed, but is filter_innd.pl executed also?

filter_nnrpd.pl is invoked first, inside nnrpd, and then filter_innd.pl is
invoked when nnrpd passes the article to innd.

(I don't know why Cleanfeed isn't working for you.)

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

On Thu, 08 Jun 2023 16:27:32, Adam W. <gof-cut-this-news@cut-this-chmurka.net.invalid> wrote:
> When is filter_innd.pl used? Is it used only for articles coming from

On my installation, /etc/news/filters/filter_innd.pl calls
/etc/news/cleanfeed/cleanfeed.local and is called for every article
going through the server, I would think also for locally posted articles
once they have passed through /etc/news/filters/filter_nnrpd.pl.

> If it's the latter, then where in this picture filter_nnrpd.pl fits in?

The main difference in my usage is that you can reject the article in a
way that the news client sees it (e.g. for bad quoting, for example),
and you can add headers or modify headers (I tend to modify
Injection-Info and add/update Cancel-Lock, since I don't
use an INN version which has internal support for it).

/etc/news/filters/filter_innd.pl and cleanfeed are not allowed to modify
articles (for good reasons).

> What if both are enabled, and someone posts the article (via nnrpd)?
> filter_nnrpd.pl is executed, but is filter_innd.pl executed also?

I think so, but I am not sure 100%.

> I have cleanfeed installed. filter_innd.pl is a symlink to cleanfeed. I
> would expect cleanfeed to reject certain posts posted to my server via
> nnrpd (for example, binary postings to non-binary groups), and I vaguely
> remember that it worked this way, but now it doesn't happen. Server
> happily accepts these posts.

In /etc/news/cleanfeed/cleanfeed.local there are some toggles you can
set to modify the behaviour.

--
Attention: limitez le nombre de lignes de citation à l'essentiel, sinon
je ne verrai pas votre réponse. Et si vous écrivez souvent des bobards,
je ne vous lirai plus et je recommanderai (NoCeM) de ne plus vous lire.

Marc SCHAEFER <schaefer@alphanet.ch> wrote:

> On my installation, /etc/news/filters/filter_innd.pl calls
> /etc/news/cleanfeed/cleanfeed.local and is called for every article

/etc/news/cleanfeed/cleanfeed I guess?

> /etc/news/filters/filter_innd.pl and cleanfeed are not allowed to modify
> articles (for good reasons).

Ok, now clear :)

> In /etc/news/cleanfeed/cleanfeed.local there are some toggles you can
> set to modify the behaviour.

It gets even weirder. I had the default configuration:

block_binaries => 1, # block misplaced binaries
block_all_binaries => 0, # Reject all binary regardless of distribution

I changed block_all_binaries to 1 just to be sure. Now:

- yEd-encoded binaries are rejected (Misplaced binary). Example:

: =ybegin line=128 size=0 name=test
:
: =yend size=0 crc32=00000000

- Images encoded with base64 are rejected (Misplaced jpg). Example:

: begin 664 test.jpg
: `
: end

- File with no extension encoded with base64 is accepted. Example:

: begin 664 test
: `
: end

- MIME-encoded image is accepted. Example (full post):

: From: test@test.test
: Newsgroups: chmurka.test
: Subject: test mime
: MIME-Version: 1.0
: Content-Type: multipart/mixed;
: boundary="------------AOXUEIFuRgZjEKO0fa0IFPxL"
: : This is a multi-part message in MIME format.
: --------------AOXUEIFuRgZjEKO0fa0IFPxL
: Content-Type: text/plain; charset=UTF-8; format=flowed
: Content-Transfer-Encoding: 8bit
: : test
: : --------------AOXUEIFuRgZjEKO0fa0IFPxL
: Content-Type: image/png; name="test.png"
: Content-Disposition: attachment; filename="test.png"
: Content-Transfer-Encoding: base64
: : test
: : --------------AOXUEIFuRgZjEKO0fa0IFPxL--

To me, it looks like a bug in cleanfeed, but is it possible that such
basic loophole in a filter used by most news servers went unnoticed?

On Thu, 08 Jun 2023 17:53:09, Adam W. <gof-cut-this-news@cut-this-chmurka.net.invalid> wrote:
>> On my installation, /etc/news/filters/filter_innd.pl calls
>> /etc/news/cleanfeed/cleanfeed.local and is called for every article
>
> /etc/news/cleanfeed/cleanfeed I guess?

Not in my installation, but I guess it was manually installed.

> - MIME-encoded image is accepted. Example (full post):

Maybe this is just not handled by this version?

I don't think mine does it either, however I have not encountered,
recently, anyone posting that kind of stuff. If it's in the big8 and you
have a Message-ID I could look for it.

There are also bots, for example in the USENET-fr hierarchy that will
kill that kind of stuff automatically (using a global cyberspam cancel
and/or a NoCeM).

I guess rejecting Content-Type: multipart/mixed in the header checks
could be enough. It would be even better to do it in filter_nnrpd.pl so
that the user gets some info (if it's a local post).

I have some experience in tweaking cleanfeed for my needs (the last
changed I am trying soon is to transform ALL rejects into NoCeM so that
my users can, through my NNTP/NNRP proxy, select which one they want to
apply, see https://nnrp.alphanet.ch/config for a web demo).

> To me, it looks like a bug in cleanfeed, but is it possible that such
> basic loophole in a filter used by most news servers went unnoticed?

Could well be either an old version, or a bug, yes.

Doing a quick search shows me that the original author has not released
anything since 1998, and that there are various patches floating around
and a mixmin version on GitHub, dating > 10 years ago.

Anyone has a more recent version?

Marc SCHAEFER <schaefer@alphanet.ch> wrote:

>>> /etc/news/cleanfeed/cleanfeed.local and is called for every article
>>
>> /etc/news/cleanfeed/cleanfeed I guess?
>
> Not in my installation, but I guess it was manually installed.

Hmm, but these files are different. There's a large file (cleanfeed, 95
KiB) and smaller cleanfeed.local.sample to be edited (~7 KiB). cleanfeed
uses cleanfeed.local, it has path configured in $config_dir.

> Maybe this is just not handled by this version?

Might be, yes...

I'm using this one: http://www.mixmin.net/cleanfeed/cleanfeed.tar.gz

I just grabbed latest from git:

https://github.com/crooks/cleanfeed

And all files are identical.

> I don't think mine does it either, however I have not encountered,
> recently, anyone posting that kind of stuff. If it's in the big8 and you
> have a Message-ID I could look for it.

It's not in big8, I made some tests on my local chmurka.test newsgroup
(accessible on news.chmurka.net for reading; if you're willing to do
some tests then I can create the account for posting).

Example Message-IDs in this group:

Message-ID: <u5ssht$n3c$6$arnold@news.chmurka.net> (with begin)
Message-ID: <u5snv1$m0v$1$pk@news.chmurka.net> (with MIME)

> I guess rejecting Content-Type: multipart/mixed in the header checks
> could be enough. It would be even better to do it in filter_nnrpd.pl so
> that the user gets some info (if it's a local post).

When I'm trying to post yEnc-encoded data, I'm getting the info in a
rejection message, so I guess filter_innd.pl is enough for that.

> Doing a quick search shows me that the original author has not released
> anything since 1998, and that there are various patches floating around
> and a mixmin version on GitHub, dating > 10 years ago.
>
> Anyone has a more recent version?

mixmin version (from that mentioned github) seems to be somewhat
maintained, but there are very few commits during the last years.

Date: Mon Mar 2 22:56:18 2020 +0000
Date: Sun Jun 2 11:24:47 2019 +0100
Date: Sat Jun 1 16:26:53 2019 +0100
Date: Fri Oct 28 14:15:09 2016 +0100
Date: Sun Sep 30 17:40:31 2012 +0100

And there are a few issues created (most by Julien), but they seem to be
ignored by the maintainer.

https://github.com/crooks/cleanfeed/issues

If there's any other cleanfeed-like tool (or cleanfeed fork) that's still
maintained I'll be happy to switch.

On Thu, 08 Jun 2023 21:25:49, Adam W. <gof-cut-this-news@cut-this-chmurka.net.invalid> wrote:
> When I'm trying to post yEnc-encoded data, I'm getting the info in a
> rejection message, so I guess filter_innd.pl is enough for that.

Rejection message to the NNTP/NNRP client? that's filter_nnrpd.pl.

> If there's any other cleanfeed-like tool (or cleanfeed fork) that's still
> maintained I'll be happy to switch.

There was also a tool used by AIOE (postfilter) [1], some french news server
installed it recently apparently with good success [2], although he
asked for help for configuration details.

[1] https://github.com/Aioe/postfilter
[2] https://usenet-fr.alphanet.ch/search/message-id/%3Ctvjqtq%2468o%241%40ns507557.dodin.fr.nf%3E/0

--
Attention: limitez le nombre de lignes de citation à l'essentiel, sinon
je ne verrai pas votre réponse. Et si vous écrivez souvent des bobards,
je ne vous lirai plus et je recommanderai (NoCeM) de ne plus vous lire.

Marc SCHAEFER <schaefer@alphanet.ch> wrote:

>> When I'm trying to post yEnc-encoded data, I'm getting the info in a
>> rejection message, so I guess filter_innd.pl is enough for that.
>
> Rejection message to the NNTP/NNRP client? that's filter_nnrpd.pl.

I got a rejection message during posting from filter_innd.pl.

: $ telnet news.chmurka.net nntp
: Trying 176.56.237.216...
: Connected to kvm.chmurka.net.
: Escape character is '^]'.
: 200 news.chmurka.net InterNetNews NNRP server INN 2.7.0 ready (no posting)
: authinfo user (...)
: 381 Enter password
: authinfo pass (...)
: 281 Authentication succeeded
: post
: 340 Ok, recommended Message-ID <u5tf3a$4b1$1@news.chmurka.net>
: from: test@test.test
: subject: test
: newsgroups: chmurka.test
: : =ybegin line=128 size=0 name=test
: .
: 441 437 Binary: misplaced binary

> There was also a tool used by AIOE (postfilter) [1], some french news server
> installed it recently apparently with good success [2], although he
> asked for help for configuration details.
>
> [1] https://github.com/Aioe/postfilter
> [2] https://usenet-fr.alphanet.ch/search/message-id/%3Ctvjqtq%2468o%241%40ns507557.dodin.fr.nf%3E/0

Merci, I'll take a look.

On Thu, 08 Jun 2023 22:56:38, Adam W. <gof-cut-this-news@cut-this-chmurka.net.invalid> wrote:
> I got a rejection message during posting from filter_innd.pl.

Ok, this contradicts what I thought was happening, aka filter_innd.pl is
run too late to be able to tell anything to the posting client, but
filter_nnrpd.pl is not.

But, indeed:

> : 441 437 Binary: misplaced binary

this message comes from /etc/news/filter/filter_innd.pl also in my
installation.

So, maybe I was wrong, and filter_nnrpd.pl AND filter_innd.pl (cleanfeed
here) can transmit error messages to the user.

gof-cut-this-news@cut-this-chmurka.net.invalid (Adam W.) writes:

> Adam W. <gof-cut-this-news@cut-this-chmurka.net.invalid> wrote:
>
>> I have cleanfeed installed. filter_innd.pl is a symlink to cleanfeed. I
>> would expect cleanfeed to reject certain posts posted to my server via
>> nnrpd (for example, binary postings to non-binary groups), and I vaguely
>> remember that it worked this way, but now it doesn't happen. Server
>> happily accepts these posts.
>
> An update.
>
> When I try to post an article with a forbidden subject (simpbiz.software),
> it gets rejected, so the filtering is enabled. But still, binary postings
> get through.
>
> I'm testing it with (prefixing here with : so it doesn't get flagged as a
> binary posting):
>
> : begin 664 test
> : `
> : end

Try to do bigger file, at leat couple of lines long.

Neodome Admin <admin@neodome.net> wrote:

>> : begin 664 test
>> : `
>> : end
>
> Try to do bigger file, at leat couple of lines long.

It looks like it's triggered when the extension is .jpg.

The most important issue is that MIME-encoded binaries are accepted...

gof-cut-this-news@cut-this-chmurka.net.invalid (Adam W.) writes:

> Neodome Admin <admin@neodome.net> wrote:
>
>>> : begin 664 test
>>> : `
>>> : end
>>
>> Try to do bigger file, at leat couple of lines long.
>
> It looks like it's triggered when the extension is .jpg.

I think it will be triggered when you'll try to post actual file, not
something that is 0 bytes long, because it's looking for actual
UU-encoded string. I might be wrong, though.

> The most important issue is that MIME-encoded binaries are accepted...

I don't remember if I ever had issues with that, however, I always had
two more filters in filter_nnrpd.pl, I called them whitelisted headers
and blacklisted headers. "Blacklisted headers" were checks for any
headers that my server wouldn't allow, and "whitelisted headers" would
be headers that, if exist, should conform to some kind of regex. If you
don't want to see any kind of MIME messages with attachments, you can
try to "blacklist" any Content-Type header except of "text/plain", or,
perharps, you can "whitelist" Content-Type header that only says
"text/plain".

Take a look at "Perl Hooks" page:

<https://www.eyrie.org/~eagle/software/inn/docs/hook-perl.html>

Apparently, Content-Type is one of the "standard" headers that can be
processed by filter_innd.pl. As far as I know, absolute most of
MIME-conforming text messages would have "text/plain" string in
Content-Type header. Usually it's something like "text/plain;
charset=us-ascii".

Salut Marc,

>> 441 437 Binary: misplaced binary
>
> So, maybe I was wrong, and filter_nnrpd.pl AND filter_innd.pl (cleanfeed
> here) can transmit error messages to the user.

Yes, nnrpd transmits the very error message innd gives when trying to
inject the article. When POSTing, nnrpd just sends the article to local
innd or whichever server is specified in nnrpdposthost in inn.conf.

441 is a reject code for POST, followed with a comment. The comment is
the response from innd (437 reject code for IHAVE with the reason coming
from the Perl filter for innd).

--
Julien ÉLIE

« – Debout paresseux ! Le coq a déjà salué le soleil levant !
– Ben il n'a plus besoin de moi alors… laissez-moi dormir barbares ! »
(Astérix)

Hi Adam,

> It gets even weirder. I had the default configuration:
>
> block_binaries => 1, # block misplaced binaries
> block_all_binaries => 0, # Reject all binary regardless of distribution
>
> I changed block_all_binaries to 1 just to be sure.

It should have worked with only block_binaries set to 1. Are you sure
you correctly reloaded the Perl filter after the change?
It may be worthwhile retrying.

> - File with no extension encoded with base64 is accepted. Example:

[...]

> - MIME-encoded image is accepted. Example (full post):
>
> : --------------AOXUEIFuRgZjEKO0fa0IFPxL
> : Content-Type: image/png; name="test.png"
> : Content-Disposition: attachment; filename="test.png"
> : Content-Transfer-Encoding: base64
> :
> : test
> :
> : --------------AOXUEIFuRgZjEKO0fa0IFPxL--

The attachment is not large enough, as Neodome Admin already spot.

Cleanfeed would have considered the article as a binary if there had
been more than max_base64_lines lines (150 by default).

--
Julien ÉLIE

« Ce qui m'intéresse chez une femme, c'est ce que je n'ose pas lui
demander. » (Wolinski)