Grant Taylor <gtaylor@tnetconsulting.net> writes:

> On 3/16/23 3:43 AM, Neodome Admin wrote:
>> I don't doubt that.
>
> So you agree that the content of the articles does have some value to
> some people.

Same as binary MIME attachments to legit Usenet messages written by real
people. They have some value for me if they add to the conversation. Is
there really a reason to avoid them now when I literally use more memory
on my 256 GB iPhone to store pictures of random dogs and cats than I use
on my server to store 2 years of unfiltered text Usenet? By unfiltered I
mean completely unfiltered, all Google Groups spam and other junk
included.

I just find it technically much simpler to differentiate by the article
size. Bigger than some value - binary. Smaller - text Usenet.

Thus my advice.

>> I doubt that regular posting of 700+KB FAQ is doing any good.
>
> What's your primary objection? The frequency or the size of the posts?
>
>> I doubt that anything in those FAQs is more useful than information
>> that can be found with Google or DuckDuckGo. We're not living in an
>> era of Altavista, after all. And if there is some kind of gem hidden
>> there, one simply don't need to post it to newsgroup regularly with
>> 700+KB of irrelevant text.
>
> I think that there is some value in having some unrequested
> information put in front of you.
>
> I've seen many things that I didn't know that I wanted to know put in
> front of me.
>
> I've also been mildly interested in something and seen something new
> (to me) done with it that really peaks my interest and causes me to
> actively investigate it.
>
> I believe there is some value in things being put in front of me for
> my perusal.

FAQs are little bit different story than other messages. Like I said, my
main problem with them is that they're not written by the people, and
thus I don't see the need to treat them any different than spam and
binaries. After all, all those binary messages also can be useful for
someone, maybe even bigger amount of people will find them more useful
compared to FAQs.

I think that legit text conversations in binary newsgroups bring more to
the Usenet as communication platform than bi-weekly FAQs in dead text
newsgroups, thus they are the ones that deserve to be preserved for the
future readers. BTW, currently it's not being done by text Usenet
servers.

>> Plus, I'm pretty sure that if there are any questions, one can just
>> ask a question in retro-computing group and expect an
>> answer... unless that group is dead, of course.
>
> It's really hard to ask a question about something if you don't know
> that said something exists.
>
> I don't mind quarterly or even monthly posting of FAQs. I do have an
> objection to super large FAQs. -- I think I have my server
> configured to accept 128 kB articles.
>
> Even at 1 MB, this is only a few seconds worth of audio / video as --
> purportedly -- admin@Neodome pointed out in a different message.
> These messages really are not much to sneeze at. -- My news server
> sees 50 or more of these messages worth of traffic per day. So, one
> of these per month, much less quarter, not even worth complaining
> about.

You are correct. If there are FAQs bigger than 64 Kb, the amount of data
they consume is miniscule compared even to the Google Groups
spam. Actually, thinking of it, I might receive them anyway from one of
the peers who set their newsfeeds incorrectly, and probably still didn't
fix it. I just never complained about it because it's not a problem from
technical point of view.

On 3/22/23 1:18 AM, Neodome Admin wrote:
> Same as binary MIME attachments to legit Usenet messages written
> by real people. They have some value for me if they add to the
> conversation.

Fair.

> Is there really a reason to avoid them now when I literally use more
> memory on my 256 GB iPhone to store pictures of random dogs and cats
> than I use on my server to store 2 years of unfiltered text Usenet? By
> unfiltered I mean completely unfiltered, all Google Groups spam and
> other junk included.
ACK

> I just find it technically much simpler to differentiate by the
> article size. Bigger than some value - binary. Smaller - text Usenet.

I agree that the size is a likely indicator of binary or not.

Though, I wonder if we are now in the day & age that we could create
filters that either:

- detect multiple strings of text with white space between them, thus
words.
- detect the standard encoding methods; e.g. 76 x [A-Za-z0-0+/=] for
base64

> Thus my advice.

Fair enough. Your server, your rules.

> FAQs are little bit different story than other messages. Like I said,
> my main problem with them is that they're not written by the people,
> and thus I don't see the need to treat them any different than spam
> and binaries. After all, all those binary messages also can be useful
> for someone, maybe even bigger amount of people will find them more
> useful compared to FAQs.

I understand what you're saying.

But is there anything to differentiate the FAQs posted by automation and
an FAQ copied from a template and pasted into the news reader by a
human? ;-)

The original text was almost certainly written by a human. Even if the
current form it is in is an amalgamation of copy & paste et al.

> I think that legit text conversations in binary newsgroups bring more
> to the Usenet as communication platform than bi-weekly FAQs in dead
> text newsgroups, thus they are the ones that deserve to be preserved
> for the future readers.

I can agree with that.

> BTW, currently it's not being done by text Usenet servers.

Agreed.

I suspect that's based on older methods of identifying / handling binary
attachments.

> You are correct. If there are FAQs bigger than 64 Kb, the amount of
> data they consume is miniscule compared even to the Google Groups
> spam. Actually, thinking of it, I might receive them anyway from one
> of the peers who set their newsfeeds incorrectly, and probably still
> didn't fix it. I just never complained about it because it's not a
> problem from technical point of view.

We all have things that we could improve on. I choose to focus on the
things with bigger impact.

--
Grant. . . .
unix || die

On Mar 22, 2023 at 1:09:28 PM CDT, "Grant Taylor" <gtaylor@tnetconsulting.net>
wrote:

> Though, I wonder if we are now in the day & age that we could create
> filters that either:
>
> - detect multiple strings of text with white space between them, thus
> words.
> - detect the standard encoding methods; e.g. 76 x [A-Za-z0-0+/=] for
> base64

Diablo has this article type detection built in and allows you to filter based
on types in newsfeed definitions. Cleanfeed and pyClean do the same for INN.
it's not perfect, but pretty damn effective.

On 2023-03-22, Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:
> On Mar 22, 2023 at 1:09:28 PM CDT, "Grant Taylor" <gtaylor@tnetconsulting.net>
> wrote:
>
>> Though, I wonder if we are now in the day & age that we could create
>> filters that either:
>>
>> - detect multiple strings of text with white space between them, thus
>> words.
>> - detect the standard encoding methods; e.g. 76 x [A-Za-z0-0+/=] for
>> base64
>
> Diablo has this article type detection built in and allows you to filter based
> on types in newsfeed definitions. Cleanfeed and pyClean do the same for INN.
> it's not perfect, but pretty damn effective.

While Cleanfeed is effective enough at what it does, there's no "smarts"
to it and it can be a chore coming up with effective patterns that work
but don't get in the way of legitimate posts that happen to contain some
of the "trouble" words or phrases. I've been wondering if it might be
possible to use something like spamassassin with bayesian learning on a
newsfeed though I haven't got to the point of trying to implement
anything yet.

Cheers,
Tom

On 3/22/23 1:33 PM, Tom Furie wrote:
> While Cleanfeed is effective enough at what it does, there's no
> "smarts" to it and it can be a chore coming up with effective patterns
> that work but don't get in the way of legitimate posts that happen to
> contain some of the "trouble" words or phrases.

Please elaborate and share some examples.

In the context of detecting encoded binary attachments, I feel like that
should be relatively easy to do.

> I've been wondering if it might be possible to use something like
> spamassassin with bayesian learning on a newsfeed though I haven't
> got to the point of trying to implement anything yet.

I don't know what SpamAssassin will think of news articles.

I wonder if it would be possible to leverage something like the milter
interface to SpamAssassin so that you don't need to integrate and or
fork SpamAssassin.

--
Grant. . . .
unix || die

On Mar 22, 2023 at 2:33:13 PM CDT, "Tom Furie" <tom@furie.org.uk> wrote:

> On 2023-03-22, Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:
>> On Mar 22, 2023 at 1:09:28 PM CDT, "Grant Taylor" <gtaylor@tnetconsulting.net>
>> wrote:
>>
>>> Though, I wonder if we are now in the day & age that we could create
>>> filters that either:
>>>
>>> - detect multiple strings of text with white space between them, thus
>>> words.
>>> - detect the standard encoding methods; e.g. 76 x [A-Za-z0-0+/=] for
>>> base64
>>
>> Diablo has this article type detection built in and allows you to filter based
>> on types in newsfeed definitions. Cleanfeed and pyClean do the same for INN.
>> it's not perfect, but pretty damn effective.
>
> While Cleanfeed is effective enough at what it does, there's no "smarts"
> to it and it can be a chore coming up with effective patterns that work
> but don't get in the way of legitimate posts that happen to contain some
> of the "trouble" words or phrases. I've been wondering if it might be
> possible to use something like spamassassin with bayesian learning on a
> newsfeed though I haven't got to the point of trying to implement
> anything yet.
>
> Cheers,
> Tom

I agree, and I decided that Diablo's duplicate article detection is good
enough for me in regards to spam filtering. Interestingly, all the default
documentation and examples only talk about using this duplicate detection for
binary articles, but I changed the feed definition to include everything and
seems about as effective as Cleanfeed/pyClean. It's binary detection seems
really good, and I'm not chewing up any noticeable CPU with filtering now.

I do use pyClean with some bad_from and bad_subject filters on my spool server
for finer granularity there.

Seems like I remember efforts in the past, perhaps not specific to INN or
Diablo, but other tools to implement SpamAssassin for filtering articles, but
off hand can't recall where that conversation occurred.

On 2023-03-22, Grant Taylor <gtaylor@tnetconsulting.net> wrote:
>
> In the context of detecting encoded binary attachments, I feel like that
> should be relatively easy to do.

Oh, there's no problem for it catching binaries, that's a non-issue. I'm
talking about methods for catching the still ever prevelant text spam..

>> I've been wondering if it might be possible to use something like
>> spamassassin with bayesian learning on a newsfeed though I haven't
>> got to the point of trying to implement anything yet.
> I don't know what SpamAssassin will think of news articles.

I don't imagine it will have any problem with the bodies, but the
headers will likely be a different matter since I doubt spamassassin
knows anything about them. Maybe some custom rulesets to inform it what
to look at...

> I wonder if it would be possible to leverage something like the milter
> interface to SpamAssassin so that you don't need to integrate and or
> fork SpamAssassin.

Yes, I was thinking of interfacing that way, or feeding everything off to
spamd.

Cheers,
Tom

On 2023-03-22, Grant Taylor <gtaylor@tnetconsulting.net> wrote:
> On 3/22/23 1:33 PM, Tom Furie wrote:
>> While Cleanfeed is effective enough at what it does, there's no
>> "smarts" to it and it can be a chore coming up with effective patterns
>> that work but don't get in the way of legitimate posts that happen to
>> contain some of the "trouble" words or phrases.
>
> Please elaborate and share some examples.

Here are a few that I think illustrate the "effective pattern" problem.
Now, this sample is all Google - which is already tagged as a known spam
source - but still they made it through. Sure, I could just block the
sender, but that seems a bit of "blunt instrument" approach to me. And
what happens in the potential situation where a spammer forges an
otherwise legitimate poster's email address, etc?

There's also the posts whereby the originals get caught by the filter,
but the fully quoted replies including full headers posted into the body
of the "complaint", make it through. That's one poster I'm incredibly
close to outright banning since he's effectively simply a reflector of
the original spam.

<c8eac7b9-bcf8-4c74-98a8-69cee7dfe9a3n@googlegroups.com>
<1273d23c-0256-4317-97d4-3eeb7bcd74b2n@googlegroups.com>
<bda61489-0096-4ff2-9a66-d1959824a1bbn@googlegroups.com>

Cheers,
Tom

Hi all,

>> I don't know what SpamAssassin will think of news articles.
>>
>> I wonder if it would be possible to leverage something like the
>> milter interface to SpamAssassin so that you don't need to
>> integrate and>> or fork SpamAssassin. >
> Seems like I remember efforts in the past, perhaps not specific to INN or
> Diablo, but other tools to implement SpamAssassin for filtering articles, but
> off hand can't recall where that conversation occurred.

From: yamo' <yamo@beurdin.invalid>
Newsgroups: news.software.nntp
Subject: Re: Google Groups spam - INN/Cleanfeed/etc solutions?
Date: Sun, 19 Sep 2021 10:11:24 -0000 (UTC)
Message-ID: <si72cc$ko9$1@pi2.pasdenom.info>

:-)

--
Julien ÉLIE

« Ta remise sur pied lui a fait perdre la tête ! » (Astérix)

On 2023-03-22, Julien ÉLIE <iulius@nom-de-mon-site.com.invalid> wrote:
>
> From: yamo' <yamo@beurdin.invalid>
> Newsgroups: news.software.nntp
> Subject: Re: Google Groups spam - INN/Cleanfeed/etc solutions?
> Date: Sun, 19 Sep 2021 10:11:24 -0000 (UTC)
> Message-ID: <si72cc$ko9$1@pi2.pasdenom.info>

Ooh, nice! That's going to be well worth a look into.

Cheers,
Tom

On 3/22/23 1:54 PM, Tom Furie wrote:
> Oh, there's no problem for it catching binaries, that's a
> non-issue. I'm talking about methods for catching the still ever
> prevelant text spam..

Oh. Okay. That makes more sense.

> I don't imagine it will have any problem with the bodies, but the
> headers will likely be a different matter since I doubt spamassassin
> knows anything about them. Maybe some custom rulesets to inform it
> what to look at...

I wouldn't be surprised if SpamAssassin did know what to do with a news
post.

I also would be surprised if it couldn't be taught how to deal with news
posts.

> Yes, I was thinking of interfacing that way, or feeding everything
> off to spamd.

:-)

--
Grant. . . .
unix || die

On 3/22/23 2:43 PM, Tom Furie wrote:
> Here are a few that I think illustrate the "effective pattern" problem.

Thank you for the message IDs. Unfortunately Thunderbird is treating
them as email addresses. I'll have to find a way to look them up.

> Now, this sample is all Google - which is already tagged as a known
> spam source - but still they made it through. Sure, I could just
> block the sender, but that seems a bit of "blunt instrument" approach
> to me. And what happens in the potential situation where a spammer
> forges an otherwise legitimate poster's email address, etc?

Ya. I'm not a fan of blocking Google carte blanche like some advocate for.

> There's also the posts whereby the originals get caught by the filter,
> but the fully quoted replies including full headers posted into
> the body of the "complaint", make it through. That's one poster I'm
> incredibly close to outright banning since he's effectively simply
> a reflector of the original spam.

Oh ya. I hear you on that one. I'm a single digit number of such
examples away from banning a user like that too. I sort of suspect we
are talking about the same user. Possibly one with a professional
sounding name?

--
Grant. . . .
unix || die

On 3/22/23 18:57, Grant Taylor wrote:
> Ya.  I'm not a fan of blocking Google carte blanche like some advocate for.

Phuque gewghul.

--

Gew Ghul Suques

On 3/22/23 9:09 PM, Gew Ghul Suques wrote:
> Phuque gewghul.

If you're going to insult / slander someone, please do it properly
instead of hiding in a homonym.

--
Grant. . . .
unix || die

Hi Grant,

>> Here are a few that I think illustrate the "effective pattern" problem.
>
> Thank you for the message IDs.  Unfortunately Thunderbird is treating
> them as email addresses.  I'll have to find a way to look them up.

http://al.howardknight.net/

--
Julien ÉLIE

« Nous autres communistes, nous avons une position claire : nous n'avons
jamais changé, nous ne changerons jamais et nous sommes pour le
changement. » (Georges Marchais)

On 3/23/23 12:57 PM, Julien ÉLIE wrote:
> Hi Grant,

Hi Julien,

> http://al.howardknight.net/

Thank you for that. That is a very neat tool. I've saved that for
future use.

I don't know how to go about fighting that spam.

I have a few hints of ideas that I'd chase. But I suspect they would
peter out and not help much.

For a while I thought about filtering out problematic posting-accounts
in the Injection-Info: header. But there were too many and I couldn't
keep up.

I feel like this is going the direction of statistical counts of various
words and improper relationships with multiple other normally
accompanying words. This becomes a statistics game that I am not
qualified to play.

--
Grant. . . .
unix || die