Hi, this is on Betterbird 115.4.2-bb17 but I guess this will be the same
on stock Thunderbird.

I am trying to filter out the usual trolls on uk.legal and I added this
filter:

https://i.ibb.co/PNLNjgC/Screenshot-at-2023-11-09-09-57-06.png

It's in German, "enthält" means "contains".

The first two filters don't work, the remaining ones, based purely on
the email address, do work but they are useless as the troll change the
"From:" header all the time.

Are this sorts of filters supported by TB or what have I done wrong?

--
Ottavio Caruso

On 09.11.23 11:03, Ottavio Caruso wrote:
> Hi, this is on Betterbird 115.4.2-bb17 but I guess this will be the same
> on stock Thunderbird.
>
> I am trying to filter out the usual trolls on uk.legal and I added this
> filter:
>
> https://i.ibb.co/PNLNjgC/Screenshot-at-2023-11-09-09-57-06.png
>
> It's in German, "enthält" means "contains".
>
> The first two filters don't work, the remaining ones, based purely on
> the email address, do work but they are useless as the troll change the
> "From:" header all the time.
>
> Are this sorts of filters supported by TB or what have I done wrong?

All of them or
any of them?

Ich bilde Adressbücher für die Trolle. Der Filter heisst dann nur noch:

If adress is in addressbook "Usenet Trolle"
then
Ignore subthread

Can be set directly in the header of a Troll-posting.

--
Gutta cavat lapidem (Ovid)

On 11/9/2023 4:03 AM, Ottavio Caruso wrote:
> Hi, this is on Betterbird 115.4.2-bb17 but I guess this will be the same
> on stock Thunderbird.
>
> I am trying to filter out the usual trolls on uk.legal and I added this
> filter:
>
> https://i.ibb.co/PNLNjgC/Screenshot-at-2023-11-09-09-57-06.png
>
> It's in German, "enthält" means "contains".
>
> The first two filters don't work, the remaining ones, based purely on
> the email address, do work but they are useless as the troll change the
> "From:" header all the time.
>
> Are this sorts of filters supported by TB or what have I done wrong?
>
>

This may not work in your case, but several of my filters use the custom
filter "Message-ID".
If the Message-ID contains googlegroups.com or vipernews.com, I delete
those.
For another filter, I delete the message if the Message-ID contains a
certain IP Address.

Am 09/11/2023 um 11:37 schrieb Jörg Lorenz:
> On 09.11.23 11:03, Ottavio Caruso wrote:
>> Hi, this is on Betterbird 115.4.2-bb17 but I guess this will be the same
>> on stock Thunderbird.
>>
>> I am trying to filter out the usual trolls on uk.legal and I added this
>> filter:
>>
>> https://i.ibb.co/PNLNjgC/Screenshot-at-2023-11-09-09-57-06.png
>>
>> It's in German, "enthält" means "contains".
>>
>> The first two filters don't work, the remaining ones, based purely on
>> the email address, do work but they are useless as the troll change the
>> "From:" header all the time.
>>
>> Are this sorts of filters supported by TB or what have I done wrong?
>
> All of them or
> any of them?
>
> Ich bilde Adressbücher für die Trolle. Der Filter heisst dann nur noch:
>
> If adress is in addressbook "Usenet Trolle"
> then
> Ignore subthread
>
> Can be set directly in the header of a Troll-posting.
>

The problem with address book is that they forge their identity on a
regular basis.

--
Ottavio Caruso

Am 09/11/2023 um 12:01 schrieb AllanH:
> On 11/9/2023 4:03 AM, Ottavio Caruso wrote:
>> Hi, this is on Betterbird 115.4.2-bb17 but I guess this will be the
>> same on stock Thunderbird.
>>
>> I am trying to filter out the usual trolls on uk.legal and I added
>> this filter:
>>
>> https://i.ibb.co/PNLNjgC/Screenshot-at-2023-11-09-09-57-06.png
>>
>> It's in German, "enthält" means "contains".
>>
>> The first two filters don't work, the remaining ones, based purely on
>> the email address, do work but they are useless as the troll change
>> the "From:" header all the time.
>>
>> Are this sorts of filters supported by TB or what have I done wrong?
>>
>>
>
> This may not work in your case, but several of my filters use the custom
> filter "Message-ID".
> If the Message-ID contains googlegroups.com or vipernews.com, I delete
> those.
> For another filter, I delete the message if the Message-ID contains a
> certain IP Address.
>

Eternal September already filters google groups. These trolls use real
newsclients.

--
Ottavio Caruso

On 11/9/2023 8:20 AM, Ottavio Caruso wrote:
> Am 09/11/2023 um 12:01 schrieb AllanH:
>> On 11/9/2023 4:03 AM, Ottavio Caruso wrote:
>>> Hi, this is on Betterbird 115.4.2-bb17 but I guess this will be the
>>> same on stock Thunderbird.
>>>
>>> I am trying to filter out the usual trolls on uk.legal and I added
>>> this filter:
>>>
>>> https://i.ibb.co/PNLNjgC/Screenshot-at-2023-11-09-09-57-06.png
>>>
>>> It's in German, "enthält" means "contains".
>>>
>>> The first two filters don't work, the remaining ones, based purely on
>>> the email address, do work but they are useless as the troll change
>>> the "From:" header all the time.
>>>
>>> Are this sorts of filters supported by TB or what have I done wrong?
>>>
>>>
>>
>> This may not work in your case, but several of my filters use the
>> custom filter "Message-ID".
>> If the Message-ID contains googlegroups.com or vipernews.com, I delete
>> those.
>> For another filter, I delete the message if the Message-ID contains a
>> certain IP Address.
>>
>
> Eternal September already filters google groups. These trolls use real
> newsclients.
>

ES does NOT filter out ALL posting from google groups as Ray has
clarified. He filters them for spam like he does every other group
presenting articles.

What you have to do is find one thing in the posts you can filter on
that works. Sometimes, injection-info works for example. It matters
not what client they use, there is always a way to rid yourself of them.
Post a full header if you can't decide and I'm sure people will give
you ideas on what to try.

--
WWG1WGA
NOTE: If you use Google Groups I don't see you,
unless you're whitelisted and that's doubtful.

Am 09/11/2023 um 15:08 schrieb sticks:
> On 11/9/2023 8:20 AM, Ottavio Caruso wrote:
>> Am 09/11/2023 um 12:01 schrieb AllanH:
>>> On 11/9/2023 4:03 AM, Ottavio Caruso wrote:
>>>> Hi, this is on Betterbird 115.4.2-bb17 but I guess this will be the
>>>> same on stock Thunderbird.
>>>>
>>>> I am trying to filter out the usual trolls on uk.legal and I added
>>>> this filter:
>>>>
>>>> https://i.ibb.co/PNLNjgC/Screenshot-at-2023-11-09-09-57-06.png
>>>>
>>>> It's in German, "enthält" means "contains".
>>>>
>>>> The first two filters don't work, the remaining ones, based purely
>>>> on the email address, do work but they are useless as the troll
>>>> change the "From:" header all the time.
>>>>
>>>> Are this sorts of filters supported by TB or what have I done wrong?
>>>>
>>>>
>>>
>>> This may not work in your case, but several of my filters use the
>>> custom filter "Message-ID".
>>> If the Message-ID contains googlegroups.com or vipernews.com, I
>>> delete those.
>>> For another filter, I delete the message if the Message-ID contains a
>>> certain IP Address.
>>>
>>
>> Eternal September already filters google groups. These trolls use real
>> newsclients.
>>
>
> ES does NOT filter out ALL posting from google groups as Ray has
> clarified.  He filters them for spam like he does every other group
> presenting articles.
>
> What you have to do is find one thing in the posts you can filter on
> that works.  Sometimes, injection-info works for example.  It matters
> not what client they use, there is always a way to rid yourself of them.
>  Post a full header if you can't decide and I'm sure people will give
> you ideas on what to try.
>
>

Ok, so this for example:

### start

Path:
eternal-september.org!news.eternal-september.org!feeder2.eternal-september.org!eternal-september.org!news.gegeweb.eu!gegeweb.org!news.mb-net.net!open-news-network.org!weretis.net!feeder8.news.weretis.net!news.imp.ch!usenet.csail.mit.edu!nntp.TheWorld.com!.POSTED!not-for-mail
From: NefeshBarYochai <void@invalid.noy>
Newsgroups: uk.legal,soc.culture.jewish,alt.abortion,uk.comp.sys.mac
Subject: Even more important than Palestine: National Socialist Leader
Michael Kuhnen explains that it is OK to be GAY
Date: Wed, 8 Nov 2023 21:20:00 +0000 (UTC)
Organization: The World : www.TheWorld.com : Since 1989
Lines: 13
Message-ID: <uigu20$n9u$1@pcls7.std.com>
NNTP-Posting-Host: shell02.theworld.com
X-Trace: pcls7.std.com 1699478400 23870 192.74.137.72 (8 Nov 2023
21:20:00 GMT)
X-Complaints-To: abuse@TheWorld.com
NNTP-Posting-Date: Wed, 8 Nov 2023 21:20:00 +0000 (UTC)
X-Newsreader: trn 4.0-test77 (Sep 1, 2010)
Originator: rjac@shell02.TheWorld.com (spamming/forging nazoid PAEDO
Andrew 'Andrzej' Baron aka SHEIN)
Xref: news.eternal-september.org uk.legal:1098039
soc.culture.jewish:492010 alt.abortion:399093 uk.comp.sys.mac:187440

### end

Let's say I want to filter out all posts that have this header:
Newsgroups: uk.legal,soc.culture.jewish,alt.abortion,uk.comp.sys.mac

I create a new filter and a new rule (header "Newsgroups", contains
"uk.legal,soc.culture.jewish,alt.abortion,uk.comp.sys.mac"), then I
press "run filter now" but Betterbird doesn't remove anything.

However I have noticed that uk.legal is now much quieter on my side. So
it is possible that the filter actually does something but it will only
filter next messages. I need to wait and see.

--
Ottavio Caruso

On 11/9/2023 9:29 AM, Ottavio Caruso wrote:
> Am 09/11/2023 um 15:08 schrieb sticks:
>> On 11/9/2023 8:20 AM, Ottavio Caruso wrote:
>>> Am 09/11/2023 um 12:01 schrieb AllanH:
>>>> On 11/9/2023 4:03 AM, Ottavio Caruso wrote:
>>>>> Hi, this is on Betterbird 115.4.2-bb17 but I guess this will be the
>>>>> same on stock Thunderbird.
>>>>>
>>>>> I am trying to filter out the usual trolls on uk.legal and I added
>>>>> this filter:
>>>>>
>>>>> https://i.ibb.co/PNLNjgC/Screenshot-at-2023-11-09-09-57-06.png
>>>>>
>>>>> It's in German, "enthält" means "contains".
>>>>>
>>>>> The first two filters don't work, the remaining ones, based purely
>>>>> on the email address, do work but they are useless as the troll
>>>>> change the "From:" header all the time.
>>>>>
>>>>> Are this sorts of filters supported by TB or what have I done wrong?
>>>>>
>>>>>
>>>>
>>>> This may not work in your case, but several of my filters use the
>>>> custom filter "Message-ID".
>>>> If the Message-ID contains googlegroups.com or vipernews.com, I
>>>> delete those.
>>>> For another filter, I delete the message if the Message-ID contains
>>>> a certain IP Address.
>>>>
>>>
>>> Eternal September already filters google groups. These trolls use
>>> real newsclients.
>>>
>>
>> ES does NOT filter out ALL posting from google groups as Ray has
>> clarified.  He filters them for spam like he does every other group
>> presenting articles.
>>
>> What you have to do is find one thing in the posts you can filter on
>> that works.  Sometimes, injection-info works for example.  It matters
>> not what client they use, there is always a way to rid yourself of
>> them.   Post a full header if you can't decide and I'm sure people
>> will give you ideas on what to try.
>>
>>
>
> Ok, so this for example:
>
> ### start
>
> Path:
> eternal-september.org!news.eternal-september.org!feeder2.eternal-september.org!eternal-september.org!news.gegeweb.eu!gegeweb.org!news.mb-net.net!open-news-network.org!weretis.net!feeder8.news.weretis.net!news.imp.ch!usenet.csail.mit.edu!nntp.TheWorld.com!.POSTED!not-for-mail
> From: NefeshBarYochai  <void@invalid.noy>
> Newsgroups: uk.legal,soc.culture.jewish,alt.abortion,uk.comp.sys.mac
> Subject: Even more important than Palestine: National Socialist Leader
> Michael Kuhnen explains that it is OK to be GAY
> Date: Wed, 8 Nov 2023 21:20:00 +0000 (UTC)
> Organization: The World : www.TheWorld.com : Since 1989
> Lines: 13
> Message-ID: <uigu20$n9u$1@pcls7.std.com>
> NNTP-Posting-Host: shell02.theworld.com
> X-Trace: pcls7.std.com 1699478400 23870 192.74.137.72 (8 Nov 2023
> 21:20:00 GMT)
> X-Complaints-To: abuse@TheWorld.com
> NNTP-Posting-Date: Wed, 8 Nov 2023 21:20:00 +0000 (UTC)
> X-Newsreader: trn 4.0-test77 (Sep 1, 2010)
> Originator: rjac@shell02.TheWorld.com (spamming/forging nazoid PAEDO
> Andrew 'Andrzej' Baron aka SHEIN)
> Xref: news.eternal-september.org uk.legal:1098039
> soc.culture.jewish:492010 alt.abortion:399093 uk.comp.sys.mac:187440
>
> ### end
>
> Let's say I want to filter out all posts that have this header:
> Newsgroups: uk.legal,soc.culture.jewish,alt.abortion,uk.comp.sys.mac
>
> I create a new filter and a new rule (header "Newsgroups", contains
> "uk.legal,soc.culture.jewish,alt.abortion,uk.comp.sys.mac"), then I
> press "run filter now" but Betterbird doesn't remove anything.
>
> However I have noticed that uk.legal is now much quieter on my side. So
> it is possible that the filter actually does something but it will only
> filter next messages. I need to wait and see.

Correct, it won't filter what you already have on your side. I close
it, reopen and check if it works.

If you don't visit any one of the groups the spammers regularly use,
filtering on just posts that use that group might be a way. I've done
that for certain groups who always post their nonsense to a particular
group because nothing that ever goes there is anything I want to see.

You could also try creating a custom header and filter on the
Originator: line. I would use includes rjac@shell02.

You could filter on the Orgainzation: header and nuke TheWorld.com.

--
WWG1WGA
NOTE: If you use Google Groups I don't see you,
unless you're whitelisted and that's doubtful.

On 11/9/2023 10:29 AM, Ottavio Caruso wrote:
> Am 09/11/2023 um 15:08 schrieb sticks:
>> On 11/9/2023 8:20 AM, Ottavio Caruso wrote:
>>> Am 09/11/2023 um 12:01 schrieb AllanH:
>>>> On 11/9/2023 4:03 AM, Ottavio Caruso wrote:
>>>>> Hi, this is on Betterbird 115.4.2-bb17 but I guess this will be the same on stock Thunderbird.
>>>>>
>>>>> I am trying to filter out the usual trolls on uk.legal and I added this filter:
>>>>>
>>>>> https://i.ibb.co/PNLNjgC/Screenshot-at-2023-11-09-09-57-06.png
>>>>>
>>>>> It's in German, "enthält" means "contains".
>>>>>
>>>>> The first two filters don't work, the remaining ones, based purely on the email address, do work but they are useless as the troll change the "From:" header all the time.
>>>>>
>>>>> Are this sorts of filters supported by TB or what have I done wrong?
>>>>>
>>>>>
>>>>
>>>> This may not work in your case, but several of my filters use the custom filter "Message-ID".
>>>> If the Message-ID contains googlegroups.com or vipernews.com, I delete those.
>>>> For another filter, I delete the message if the Message-ID contains a certain IP Address.
>>>>
>>>
>>> Eternal September already filters google groups. These trolls use real newsclients.
>>>
>>
>> ES does NOT filter out ALL posting from google groups as Ray has clarified.  He filters them for spam like he does every other group presenting articles.
>>
>> What you have to do is find one thing in the posts you can filter on that works.  Sometimes, injection-info works for example.  It matters not what client they use, there is always a way to rid yourself of them.   Post a full header if you can't decide and I'm sure people will give you ideas on what to try.
>>
>>
>
> Ok, so this for example:
>
> ### start
>
> From: NefeshBarYochai  <void@invalid.noy>
> Newsgroups: uk.legal,soc.culture.jewish,alt.abortion,uk.comp.sys.mac
>
> Let's say I want to filter out all posts that have this header:
> Newsgroups: uk.legal,soc.culture.jewish,alt.abortion,uk.comp.sys.mac
>
> I create a new filter and a new rule (header "Newsgroups", contains
> "uk.legal,soc.culture.jewish,alt.abortion,uk.comp.sys.mac"), then
> I press "run filter now" but Betterbird doesn't remove anything.
>
> However I have noticed that uk.legal is now much quieter on my side.
> So it is possible that the filter actually does something but it will
> only filter next messages. I need to wait and see.

This illustrates what happens when you use the GUI.

[Picture]

https://i.postimg.cc/QxYvGKsm/filter-test.gif

it's unclear, what kind of Boolean expression and nested parenthesis
this tool supports. Obviously it switches to OR instead of AND, by using
the appropriate radio button for it. Can you do an AND-OR compound expression ?
Not a clue. If you edited the .dat file by hand, I doubt the code contains
a boolean math processor.

At least you can see in my example, how I picked the data apart into fields.
I did that, because I know it parses on "," and breaks up the item into
four units. I wrote my filter, based on my best guess of the unit in question.

Notice that, when people craft replies to the individual, the order of the
newsgroups can get swapped, plus some tools might even add 0x20 spaces
next to stuff. Writing "one line filters" then, is not breaking the job
down enough. As a result, I selected four lines to do it.

Now, I would not even write a filter like this, but that's besides the point.

I've already explained to you, to filter on newsgroups "," and simply
eliminate all crossposts. Any Newsgroups line with a "," in it, has been
crossposted. Because you know the yoyo community will always
cross into soc.* . And of course, everyone in soc.* is using filters.
Filters that are better than I could write.

But generally speaking, the person who wrote this filter code, was not
there to tweak the job and improve it. What the guy did do, is
he drew a diagram of how data flowed inside Thunderbird, he
figured out some things he could do. He learned how Mork works (no small feat).
The finished work then, is a *demo* for the staff, on "how to write a filter".

In a normal organization, you would get someone with Regex experience, to re-write
the rules section. The datapath code would be fine as is. You cannot
do more than the datapath provides. (Overview.fmt server limitation.)

For the second person to work on the code, they don't have to know
how Mork works. The engine below the line is fine, for a start.
But a filter specialist should work on the sugary top layer.
Make the math part support more general boolean expressions with parenthesis.
There is Regex and PCRE as filter methods, Regex being unbounded,
PCRE being bounded (won't rail a CPU core when expanded). The person
fixing the filter, would use the subset of features known as PCRE.
To reduce complaints to a minimum, concerning filter construct.
(If the CPU railed, some clever individuals would file a bug, rather
than look at the filter they designed.) Regex/PCRE is outside my pay
scale, and that's the barest description.

Paul

On 11/9/2023 9:20 AM, Ottavio Caruso wrote:
> Am 09/11/2023 um 12:01 schrieb AllanH:
>> On 11/9/2023 4:03 AM, Ottavio Caruso wrote:
>>> Hi, this is on Betterbird 115.4.2-bb17 but I guess this will be the same on stock Thunderbird.
>>>
>>> I am trying to filter out the usual trolls on uk.legal and I added this filter:
>>>
>>> https://i.ibb.co/PNLNjgC/Screenshot-at-2023-11-09-09-57-06.png
>>>
>>> It's in German, "enthält" means "contains".
>>>
>>> The first two filters don't work, the remaining ones, based purely on the email address, do work but they are useless as the troll change the "From:" header all the time.
>>>
>>> Are this sorts of filters supported by TB or what have I done wrong?
>>>
>>>
>>
>> This may not work in your case, but several of my filters use the custom filter "Message-ID".
>> If the Message-ID contains googlegroups.com or vipernews.com, I delete those.
>> For another filter, I delete the message if the Message-ID contains a certain IP Address.
>>
>
> Eternal September already filters google groups. These trolls use real newsclients.
>

Start by filtering out all crossposts, and see what is left.
The very idea of writing a half dozen filters, that's ludicrous.

newsgroups contains ,
delete

Use the power of the Force, Luke. Don't dick around. If someone
nymshifts, you could be writing filters pretty well continuously.

You start with a noise filter.

If it gets personal with you, you filter individuals by name.
But only once the noise in the group is gone.

To start with a name-based filter, that is not profitable.

if you end up with a thousand filters in uk.legal, you are doing it wrong.

And this is also why the Administrator of the server, does not write
a thousand filters for uk.legal. He could do that... if an AI
could provide good service. But that is never going to work.
(Look at the Thai Spam for inspiration.)

Paul

On Thu, 9 Nov 2023 15:29:40 +0000, Ottavio Caruso wrote:

> Ok, so this for example:
>
> ### start
>
> Path:
> eternal-september.org!news.eternal-september.org!feeder2.eternal-september.org!eternal-september.org!news.gegeweb.eu!gegeweb.org!news.mb-net.net!open-news-network.org!weretis.net!feeder8.news.weretis.net!news.imp.ch!usenet.csail.mit.edu!nntp.TheWorld.com!.POSTED!not-for-mail
> From: NefeshBarYochai <void@invalid.noy>
> Newsgroups: uk.legal,soc.culture.jewish,alt.abortion,uk.comp.sys.mac
> Subject: Even more important than Palestine: National Socialist Leader
> Michael Kuhnen explains that it is OK to be GAY
> Date: Wed, 8 Nov 2023 21:20:00 +0000 (UTC)
> Organization: The World : www.TheWorld.com : Since 1989
> Lines: 13
> Message-ID: <uigu20$n9u$1@pcls7.std.com>
> NNTP-Posting-Host: shell02.theworld.com
> X-Trace: pcls7.std.com 1699478400 23870 192.74.137.72 (8 Nov 2023
> 21:20:00 GMT)
> X-Complaints-To: abuse@TheWorld.com
> NNTP-Posting-Date: Wed, 8 Nov 2023 21:20:00 +0000 (UTC)
> X-Newsreader: trn 4.0-test77 (Sep 1, 2010)
> Originator: rjac@shell02.TheWorld.com (spamming/forging nazoid PAEDO
> Andrew 'Andrzej' Baron aka SHEIN)
> Xref: news.eternal-september.org uk.legal:1098039
> soc.culture.jewish:492010 alt.abortion:399093 uk.comp.sys.mac:187440
>
> ### end
>
> Let's say I want to filter out all posts that have this header:
> Newsgroups: uk.legal,soc.culture.jewish,alt.abortion,uk.comp.sys.mac
>
> I create a new filter and a new rule (header "Newsgroups", contains
> "uk.legal,soc.culture.jewish,alt.abortion,uk.comp.sys.mac"), then I
> press "run filter now" but Betterbird doesn't remove anything.
>
> However I have noticed that uk.legal is now much quieter on my side. So
> it is possible that the filter actually does something but it will only
> filter next messages. I need to wait and see.

“Newsgroups” is not an ‘overview’ header. This means:

a) When your newsreader gets headers from the news server,
“Newsgroups” is _not_ one of the headers which the news server
provides.

b) When you try to run filters manually _after_ the header download,
the “Newsgroups” header is not there in Thunderbird’s (or
Betterbird’s) database. So the filter will not work when you run
it manually.

c) When filters are run automatically while downloading headers,
Thunderbird (or Betterbird) is online and talking to the server.
Thunderbird (or Betterbird) will _try_ to ask the server for any
extra ‘non-overview’ headers such as “Newsgroups” which are
needed by filters. Whether this will work or not, depends on
your news server.

FYI the ‘overview’ headers are these. The first 7 apply to
all news servers, the last one applies to most news servers.

Subject:
From:
Date:
Message-ID:
References:
Bytes:
Lines:
Xref:full --- this last one on most news servers including Eternal-September

--
Kind regards
Ralph Fox

🪔️ 🪔️ 🪔️ 🪔️ 🪔️ 🪔️ 🪔️ 🪔️ 🪔️ 🪔️ 🪔️ 🪔️ 🪔️

On 09.11.23 15:19, Ottavio Caruso wrote:
> Am 09/11/2023 um 11:37 schrieb Jörg Lorenz:
>> All of them or
>> any of them?
>>
>> Ich bilde Adressbücher für die Trolle. Der Filter heisst dann nur noch:
>>
>> If adress is in addressbook "Usenet Trolle"
>> then
>> Ignore subthread
>>
>> Can be set directly in the header of a Troll-posting.
>>
>
> The problem with address book is that they forge their identity on a
> regular basis.

I can handle that then you can too.

For the difficult cases (2 or 3) I introduced multi-factor filters where
the identity (name; email) is not included.

--
Gutta cavat lapidem (Ovid)

On 11/9/2023 12:59 PM, Ralph Fox wrote:
> On Thu, 9 Nov 2023 15:29:40 +0000, Ottavio Caruso wrote:
>
>> Ok, so this for example:
>>
>> ### start
>>
>> Path:
>> eternal-september.org!news.eternal-september.org!feeder2.eternal-september.org!eternal-september.org!news.gegeweb.eu!gegeweb.org!news.mb-net.net!open-news-network.org!weretis.net!feeder8.news.weretis.net!news.imp.ch!usenet.csail.mit.edu!nntp.TheWorld.com!.POSTED!not-for-mail
>> From: NefeshBarYochai <void@invalid.noy>
>> Newsgroups: uk.legal,soc.culture.jewish,alt.abortion,uk.comp.sys.mac
>> Subject: Even more important than Palestine: National Socialist Leader
>> Michael Kuhnen explains that it is OK to be GAY
>> Date: Wed, 8 Nov 2023 21:20:00 +0000 (UTC)
>> Organization: The World : www.TheWorld.com : Since 1989
>> Lines: 13
>> Message-ID: <uigu20$n9u$1@pcls7.std.com>
>> NNTP-Posting-Host: shell02.theworld.com
>> X-Trace: pcls7.std.com 1699478400 23870 192.74.137.72 (8 Nov 2023
>> 21:20:00 GMT)
>> X-Complaints-To: abuse@TheWorld.com
>> NNTP-Posting-Date: Wed, 8 Nov 2023 21:20:00 +0000 (UTC)
>> X-Newsreader: trn 4.0-test77 (Sep 1, 2010)
>> Originator: rjac@shell02.TheWorld.com (spamming/forging nazoid PAEDO
>> Andrew 'Andrzej' Baron aka SHEIN)
>> Xref: news.eternal-september.org uk.legal:1098039
>> soc.culture.jewish:492010 alt.abortion:399093 uk.comp.sys.mac:187440
>>
>> ### end
>>
>> Let's say I want to filter out all posts that have this header:
>> Newsgroups: uk.legal,soc.culture.jewish,alt.abortion,uk.comp.sys.mac
>>
>> I create a new filter and a new rule (header "Newsgroups", contains
>> "uk.legal,soc.culture.jewish,alt.abortion,uk.comp.sys.mac"), then I
>> press "run filter now" but Betterbird doesn't remove anything.
>>
>> However I have noticed that uk.legal is now much quieter on my side. So
>> it is possible that the filter actually does something but it will only
>> filter next messages. I need to wait and see.
>
>
> “Newsgroups” is not an ‘overview’ header. This means:
>
> a) When your newsreader gets headers from the news server,
> “Newsgroups” is _not_ one of the headers which the news server
> provides.
>
> b) When you try to run filters manually _after_ the header download,
> the “Newsgroups” header is not there in Thunderbird’s (or
> Betterbird’s) database. So the filter will not work when you run
> it manually.
>
> c) When filters are run automatically while downloading headers,
> Thunderbird (or Betterbird) is online and talking to the server.
> Thunderbird (or Betterbird) will _try_ to ask the server for any
> extra ‘non-overview’ headers such as “Newsgroups” which are
> needed by filters. Whether this will work or not, depends on
> your news server.
>
>
> FYI the ‘overview’ headers are these. The first 7 apply to
> all news servers, the last one applies to most news servers.
>
> Subject:
> From:
> Date:
> Message-ID:
> References:
> Bytes:
> Lines:
> Xref:full --- this last one on most news servers including Eternal-September

On Thunderbird, you can use "offline sync" to pull in enough
information to remove on any field.

File : Offline : Offline Settings # Select one group for synchronization
# The process is so slow, multiple selections would be confusing.

File : Offline : Download/Sync Now # It will take many hours to download a
# group, at two messages per second

Now run your filter on what was collected.

The speed of sync, is suggestive of some sort of mail-like protocol.
Downloads are not at overview speed.

*******

A message, looks like this, to give an idea what ThaiSpam looks like in CLC.
The Newsgroups: item is now in view, and available for a manual filter run.
You can run a sync, once daily, to keep a group synchronization up-to-date
with smaller sync runs as a result. The first sync run will be very large.
The spam run from CLC, this file reached 450MB before I stopped syncing.

You can back up the profile folder, before running filters, and then
you can re-run scenarios until you're happy. Filtering on Message-ID
containing googlegroups.com is how you would get rid of Google Groups
postings, without needing sync.

From - Tue Oct 24 02:05:13 2023
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
X-Received: by 2002:a05:622a:5517:b0:41c:e1f2:f718 with SMTP id fj23-20020a05622a551700b0041ce1f2f718mr157351qtb.6.1698127507808;
Mon, 23 Oct 2023 23:05:07 -0700 (PDT)
X-Received: by 2002:a05:6870:f622:b0:1ea:1bc1:4084 with SMTP id
ek34-20020a056870f62200b001ea1bc14084mr5196627oab.4.1698127507456; Mon, 23
Oct 2023 23:05:07 -0700 (PDT)
Path: news.netfront.net!newsfeed.endofthelinebbs.com!newsfeed.xs3.de!callisto.xs3.de!gandalf.srv.welterde.de!news.mixmin.net!Xbb.tags.giganews.com!border-2.nntp.ord.giganews.com!nntp.giganews.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.lang.c
Date: Mon, 23 Oct 2023 23:05:07 -0700 (PDT)
Injection-Info: google-groups.googlegroups.com; posting-host=2a0e:a942:8c0a:ec0e:609e:b867:c55c:1883;
posting-account=FL6PjgoAAAAbjvVjBXZtsKUfYfISwsQW
NNTP-Posting-Host: 2a0e:a942:8c0a:ec0e:609e:b867:c55c:1883
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <aef1dee4-d86d-4ce0-b4de-119fa4243918n@googlegroups.com>
Subject: =?UTF-8?B?4Liq4Lih4Lix4LiE4LijIHNhZ2FtZTY2IOC4quC4peC5h+C4reC4leC4reC4reC4meC5hA==?=
=?UTF-8?B?4Lil4LiZ4LmMIOC5hOC4oeC5iCDguIHguKPguLDguJXguLjguIEgOTE4a2lzcyAyMDE5ICA=?=
From: =?UTF-8?B?4Lit4Lix4LiZ4LiU4Lix4LiaMSDguJfguY3guLLguIHguY3guLLguYTguKPguYTguJTguYk=?=
=?UTF-8?B?4LiX4Li44LiB4LmA4Lin4Lil4Liy?= <nimboongrinkevychmm@gmail.com>
Injection-Date: Tue, 24 Oct 2023 06:05:07 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: base64
Lines: 48
Xref: news.netfront.net comp.lang.c:192662

4Liq4Lih4Lix4LiE4LijIHNhZ2FtZTY2IOC4quC4peC5h+C4reC4leC4reC4reC4meC5hOC4peC4
....
guC4oeC4iuC4seC5iOC4mQpodHRwczovL2JpdC5seS9jdXA4OHBiPzI5Zm9nPWJvbnVz

*******

GROUP uk.legal # I captured this using Wireshark, while using E-S.
211 266322 4122 1098142 uk.legal # This is info about how big the E-S cache of articles is.

Syntax
GROUP group

Responses
211 number low high group Group successfully selected
411 No such newsgroup

Parameters
group Name of newsgroup
number Estimated number of articles in the group <=== 266322
low Reported low water mark
high Reported high water mark

That could take a while. Maybe a day and a half for synchronization to finish.

E-S has this limit, on concurrent connections.

The number of concurrent connections to the server is limited to 4

Thunderbird can also limit how many connections it will make at a maximum.
There is a setting for that. The sync would use one connection, while
you do other things with the other connections.

Paul

Am 09/11/2023 um 10:03 schrieb Ottavio Caruso:
> Hi, this is on Betterbird 115.4.2-bb17 but I guess this will be the same
> on stock Thunderbird.
>
> I am trying to filter out the usual trolls on uk.legal and I added this
> filter:
>
> https://i.ibb.co/PNLNjgC/Screenshot-at-2023-11-09-09-57-06.png
>
> It's in German, "enthält" means "contains".
>
> The first two filters don't work, the remaining ones, based purely on
> the email address, do work but they are useless as the troll change the
> "From:" header all the time.
>
> Are this sorts of filters supported by TB or what have I done wrong?
>
>

Apparently, I checked the filter logs, and the above filter seems to
have caught the offending messages, but the emails haven't been deleted
as requested. So, IMHO the filter works, only that Betterbird didn't
delete them. I haven't tried on vanilla TB, though.

--
Ottavio Caruso

On 11/11/2023 5:25 AM, Ottavio Caruso wrote:
> Am 09/11/2023 um 10:03 schrieb Ottavio Caruso:
>> Hi, this is on Betterbird 115.4.2-bb17 but I guess this will be the same on stock Thunderbird.
>>
>> I am trying to filter out the usual trolls on uk.legal and I added this filter:
>>
>> https://i.ibb.co/PNLNjgC/Screenshot-at-2023-11-09-09-57-06.png
>>
>> It's in German, "enthält" means "contains".
>>
>> The first two filters don't work, the remaining ones, based purely on the email address, do work but they are useless as the troll change the "From:" header all the time.
>>
>> Are this sorts of filters supported by TB or what have I done wrong?
>>
>>
>
> Apparently, I checked the filter logs, and the above filter seems to have caught the offending messages, but the emails haven't been deleted as requested. So, IMHO the filter works, only that Betterbird didn't delete them. I haven't tried on vanilla TB, though.
>

Similar to how some file systems work, thunderbird/betterbird keep
metadata in at least two places.

Inbox
From - Wed Nov 26 06:13:35 2008
X-Mozilla-Status: 0015 \
X-Mozilla-Status2: 00800000 \___ These fields provide storage space for booleans or the like.
X-Mozilla-Keys: /

Inbox.msf [Mork database header summary of Inbox messages, auto-generated]

This is an example, of the information stored in the added Status fields in Inbox.

http://forums.mozillazine.org/viewtopic.php?f=39&t=484449

Some even suggest that the status codes "0000" were received,
not yet read and undeleted,

"8001" after reading the message but not deleted, and
"8009" when the message was read and marked for delete.

When you Delete a message, at the very least, the Mork file should
have the Delete bit set on the message. When you "compact" a box,
the message body should be physically removed from storage.
(I think I noticed my ThaiSpam removal test in CLC, did not cause
the comp.lang.c file to be reduced in size, which is something I
have to finish checking later.)

A message (stored entirely) in the Inbox, has several fields added
artificially by the software. The flags in the Status field, may
indicate some similar things to the Mork database file.

So if you wanted to learn about deletion, you might compare
messages in Inbox (load Inbox into text editor), versus what you
know about their deleted status.

At the very least, in an email tool, the "deleted" status means
the message header should no longer be displayed (whether the message
is read or unread). Removal of all traces of the message, is typically
left for later. In this case, "Compact" is an opportunity to do the work.

Really, if the Filter Log says something was done, well, it
was done. But you can always attempt to prove it. Exit the tool,
to ensure all the Profile data is at rest, before studying.

Paul