Is there a way of proactively removing expired articles?

I've noticed more issues recently where messages are being removed by a
NoCeM listing. However, in some cases I will have downloaded the header
before NoCeM has removed the message. TB then pops up an error message
that the newsgroup server reports that it can't find the article. At the
end of the error message TB offers to "Remove All Expired Articles".

I tried TB's Help, but couldn't see a way to remove the expired articles
prior to the popup appearing. Any ideas if it can be done?

--

Jeff

Jeff Layman <jeff@invalid.invalid> wrote:

> Is there a way of proactively removing expired articles?
>
> I've noticed more issues recently where messages are being removed by a
> NoCeM listing. However, in some cases I will have downloaded the header
> before NoCeM has removed the message. TB then pops up an error message
> that the newsgroup server reports that it can't find the article. At the
> end of the error message TB offers to "Remove All Expired Articles".
>
> I tried TB's Help, but couldn't see a way to remove the expired articles
> prior to the popup appearing. Any ideas if it can be done?

In my newsreader, after selecting a message, I hit Delete to remove it
from the newsreaders message store. Maybe Delete also works in Tbird.

On 12/26/2023 3:07 PM, VanguardLH wrote:
> Jeff Layman <jeff@invalid.invalid> wrote:
>
>> Is there a way of proactively removing expired articles?
>>
>> I've noticed more issues recently where messages are being removed by a
>> NoCeM listing. However, in some cases I will have downloaded the header
>> before NoCeM has removed the message. TB then pops up an error message
>> that the newsgroup server reports that it can't find the article. At the
>> end of the error message TB offers to "Remove All Expired Articles".
>>
>> I tried TB's Help, but couldn't see a way to remove the expired articles
>> prior to the popup appearing. Any ideas if it can be done?
>
> In my newsreader, after selecting a message, I hit Delete to remove it
> from the newsreaders message store. Maybe Delete also works in Tbird.

Delete does work in TBird/BBird. There was some talk about using it
awhile back and I'm trying to remember what the problem that got brought
up as a possibility was. It wasn't something that broke anything, IIRC.
I had been marking filtered posts for deletion, but after the thread
decided to just mark as read.

This caught my attention as I had been meaning to ask about it again in
hopes of understanding why I chose to do that because lately I've had
times when it seemed to me deletion would be a better option. Perhaps
someone can remember the thread and it could be rehashed?

As far as removing expired articles in TBird, you have to find one in
your group that has expired and click on it. If it is expired, a
dialogue box will pop up that says, "A News (NNTP) error occurred: No
such article number XXXXXX." You then click OK and it closes, and
behind it where the message would have appeared is another message that
explains there is no such message and then also has a button you can
click on that says, "Remove all expired Articles." HTH

--
Stand With Israel!
NOTE: If you use Google Groups I don't see you,
unless you're whitelisted and that's doubtful.

On 2023-12-26 22:07, VanguardLH wrote:
> Jeff Layman <jeff@invalid.invalid> wrote:
>
>> Is there a way of proactively removing expired articles?
>>
>> I've noticed more issues recently where messages are being removed by a
>> NoCeM listing. However, in some cases I will have downloaded the header
>> before NoCeM has removed the message. TB then pops up an error message
>> that the newsgroup server reports that it can't find the article. At the
>> end of the error message TB offers to "Remove All Expired Articles".
>>
>> I tried TB's Help, but couldn't see a way to remove the expired articles
>> prior to the popup appearing. Any ideas if it can be done?
>
> In my newsreader, after selecting a message,

At this instant, TB will say the article has expired and offer to purge
all expired messages in the current group.

> I hit Delete to remove it
> from the newsreaders message store. Maybe Delete also works in Tbird.

--
Cheers, Carlos.

"Carlos E.R." <robin_listas@es.invalid> wrote:

> On 2023-12-26 22:07, VanguardLH wrote:
>> Jeff Layman <jeff@invalid.invalid> wrote:
>>
>>> Is there a way of proactively removing expired articles?
>>>
>>> I've noticed more issues recently where messages are being removed by a
>>> NoCeM listing. However, in some cases I will have downloaded the header
>>> before NoCeM has removed the message. TB then pops up an error message
>>> that the newsgroup server reports that it can't find the article. At the
>>> end of the error message TB offers to "Remove All Expired Articles".
>>>
>>> I tried TB's Help, but couldn't see a way to remove the expired articles
>>> prior to the popup appearing. Any ideas if it can be done?
>>
>> In my newsreader, after selecting a message,
>
> At this instant, TB will say the article has expired and offer to purge
> all expired messages in the current group.
>
>> I hit Delete to remove it
>> from the newsreaders message store. Maybe Delete also works in Tbird.

I don't get nuisanced with a popup. A red-X'ed box shows up alongside
the message in the header list pane, I select it (no popup), and hit
Delete. Seems the popup is an interferring nuisance, like deleting a
file and a popup appears asking if you really want to delete the file;
i.e., a safety step to prevent accidental deletion of something you can
[no longer] retrieve.

If you first select a message that is not deleted on the server, and use
the arrow keys to move up/down the headers pane, do you get the Tbird
popup when the inaccessible message is highlighted?

Carlos E.R. wrote:

> VanguardLH wrote:
>
>> In my newsreader, after selecting a message,
>
> At this instant, TB will say the article has expired and offer to purge
> all expired messages in the current group.

Varies per server, e.g.

E-S filters out the spam on his transit server, before forwarding the
residual non-spam to the reader server, so TB never sees the spam.

N-I-N delivers all messages to the reader server, so TB sees them, then
the spam is cancelled leaving TB unable to retrieve it, and offering to
clean up the deletions.

Andy Burns <usenet@andyburns.uk> wrote:

> Carlos E.R. wrote:
>
>> VanguardLH wrote:
>>
>>> In my newsreader, after selecting a message,
>>
>> At this instant, TB will say the article has expired and offer to purge
>> all expired messages in the current group.
>
> Varies per server, e.g.
>
> E-S filters out the spam on his transit server, before forwarding the
> residual non-spam to the reader server, so TB never sees the spam.
>
> N-I-N delivers all messages to the reader server, so TB sees them, then
> the spam is cancelled leaving TB unable to retrieve it, and offering to
> clean up the deletions.

When are peered NOCEMs processed by ES? Seems they could come any time:
before or after Tbird has retrieved article headers from ES.

https://www.eternal-september.org/index.php?showpage=nocem

That your client retrieves an article does not force it to remain on the
server. Yes, you have a local copy of the article that was later
removed, but most NNTP clients try to synchronize with the server, not
just do a 1-way download.

Plus retrieving an article from just the Overview database on the server
doesn't get you the message from the Articles database until you tell
your client to retrieve the message. The client gets the overview
headers, you see them listed in the client, you click on an article, and
then your client tries to get the message from the Articles database.
The article may no longer be on the server between when you got the
overview headers to when you get the article. Pass 1: Get overview
headers for articles. Pass 2: Get the articles matching the overview
headers you previously downloaded.

If you configure your client to not just download overview headers, but
also download articles at the same time, then you're assured to have
both overview headers and article. I only visit text-only newsgroups,
so I configure my NNTP client to get overview and article in the same
poll (pass 1 gets the overview, pass 2 gets the articles, in the same
poll, and filters that only used overview headers can run in pass 1
while filters that look at non-overview headers which are retrieved when
the article gets download will run in pass 2). I don't do binary
newsgroups, so I'm not concerned about keeping the message store with
just overview headers for huge articles, and download the huge articles
when I select them. For me, each poll gets both overview headers and
message (i.e., full article). My client will skip filters in pass 2
that already fired in pass 1.

Maybe if you configure Tbird to work in offline line then it gets both
overview headers and non-overview headers (along with the article), or
its a separate option in Tbird. However, I thought offline mode in
Tbird simply disconnected from the network, so whatever you got so far
is what you get to look at. If all you got were overview headers, you
won't have the articles when offline.

Other than the offline option in Tbird, is there an option to force it
to download complete messages (instead of just overview headers)? I
thought that option was under:

Account Settings > Synchronization & Storage > Message Synchronising

What's the default? Immediately? Or is that setting navpath only for
e-mail accounts?

On 12/27/2023 2:52 AM, Andy Burns wrote:
> Carlos E.R. wrote:
>
>> VanguardLH wrote:
>>
>>> In my newsreader, after selecting a message,
>>
>> At this instant, TB will say the article has expired and offer to purge all expired messages in the current group.
>
> Varies per server, e.g.
>
> E-S filters out the spam on his transit server, before forwarding the residual non-spam to the reader server, so TB never sees the spam.
>
> N-I-N delivers all messages to the reader server, so TB sees them, then the spam is cancelled leaving TB unable to retrieve it, and offering to clean up the deletions.
>

Expired articles are also cleaned up when you "compact" a folder.

Or, if you run in offline mode, and download an entire group for
later consumption, the expired headers also get removed in that case.

Paul

VanguardLH wrote:

> When are peered NOCEMs processed by ES? Seems they could come any time:
> before or after Tbird has retrieved article headers from ES.

E-S is generating its own NOCEMs with spamassasin, so probably most
cancels happen locally before the messages pass from the transit server
to the receiver server, any NOCEMs that arrive from elsewhere may occur
after they reach the reader server ...

Andy Burns wrote:
> VanguardLH wrote:
>
>> When are peered NOCEMs processed by ES?  Seems they could come any time:
>> before or after Tbird has retrieved article headers from ES.
>
> E-S is generating its own NOCEMs with spamassasin, so probably most
> cancels happen locally before the messages pass from the transit server
> to the receiver

^^^^^^
reader
> server, any NOCEMs that arrive from elsewhere may occur
> after they reach the reader server ...

Andy Burns <usenet@andyburns.uk> wrote:

> VanguardLH wrote:
>
>> When are peered NOCEMs processed by ES? Seems they could come any time:
>> before or after Tbird has retrieved article headers from ES.
>
> E-S is generating its own NOCEMs with spamassasin, so probably most
> cancels happen locally before the messages pass from the transit server
> to the receiver server, any NOCEMs that arrive from elsewhere may occur
> after they reach the reader server ...

APD in the eternal-september.support newsgroup started a thread "NoCEM
nuttiness" (Message-ID: <umcvgg$36dhk$1@apd.eternal-september.org>) who
posted via ES about a NoCEM issued by ovh which presumably had a message
disappear from ES.

From the link that I gave, here again:

https://www.eternal-september.org/index.php?showpage=nocem

ES is picking up NoCEMs from several lists generated elsewhere. In the
ES nocem.ctl file are: ovh, i2pn2, httrack, berlin.de (perhaps
news.individual.net), and killfile.org. I couldn't find mention of
nocem.ctl at http://www.cm.org/faq.html, but did find mention at:

https://manpages.debian.org/testing/inn2/nocem.ctl.5.en.html
"Create a nocem.ctl config file in pathetc indicating the NoCeM issuers
and notices you want to follow."
"This will remove all articles for which the issuer (first part of the
line, before the colon ":") has issued NoCeM notices corresponding to
the criteria specified after the colon."

APD was complaining about a false positive in the ovh NoCEM list that ES
includes to remove articles from ES. ES might be creating its own
NoCEMs via its SpamAssassin filter, but it is also using NoCEMs
generated elsewhere (some NoCEMS for spam, some for forgeries, and some
based on some admin's decision at berlin.de).

On 27/12/2023 02:45, VanguardLH wrote:
> "Carlos E.R." <robin_listas@es.invalid> wrote:
>
>> On 2023-12-26 22:07, VanguardLH wrote:
>>> Jeff Layman <jeff@invalid.invalid> wrote:
>>>
>>>> Is there a way of proactively removing expired articles?
>>>>
>>>> I've noticed more issues recently where messages are being removed by a
>>>> NoCeM listing. However, in some cases I will have downloaded the header
>>>> before NoCeM has removed the message. TB then pops up an error message
>>>> that the newsgroup server reports that it can't find the article. At the
>>>> end of the error message TB offers to "Remove All Expired Articles".
>>>>
>>>> I tried TB's Help, but couldn't see a way to remove the expired articles
>>>> prior to the popup appearing. Any ideas if it can be done?
>>>
>>> In my newsreader, after selecting a message,
>>
>> At this instant, TB will say the article has expired and offer to purge
>> all expired messages in the current group.
>>
>>> I hit Delete to remove it
>>> from the newsreaders message store. Maybe Delete also works in Tbird.
>
> I don't get nuisanced with a popup. A red-X'ed box shows up alongside
> the message in the header list pane, I select it (no popup), and hit
> Delete. Seems the popup is an interferring nuisance, like deleting a
> file and a popup appears asking if you really want to delete the file;
> i.e., a safety step to prevent accidental deletion of something you can
> [no longer] retrieve.

Indeed. None of the expired articles are there anyway. It would be
useful to have a checkbox in "Settings" which could be ticked so that
the removal of all expired articles is automatic.

> If you first select a message that is not deleted on the server, and use
> the arrow keys to move up/down the headers pane, do you get the Tbird
> popup when the inaccessible message is highlighted?

No idea. Once the first error message appears I click on "Remove all
expired articles". What I was hoping for was to find a way for that to
be done automatically, perhaps every 30 minutes or so, or when I ask TB
to download new headers. If TB can find all expired articles once I
click on that button, why can't it be done for the first expired article
which appears to be required to start the process in motion?

--

Jeff

On 27/12/2023 09:37, Paul wrote:
> On 12/27/2023 2:52 AM, Andy Burns wrote:
>> Carlos E.R. wrote:
>>
>>> VanguardLH wrote:
>>>
>>>> In my newsreader, after selecting a message,
>>>
>>> At this instant, TB will say the article has expired and offer to purge all expired messages in the current group.
>>
>> Varies per server, e.g.
>>
>> E-S filters out the spam on his transit server, before forwarding the residual non-spam to the reader server, so TB never sees the spam.
>>
>> N-I-N delivers all messages to the reader server, so TB sees them, then the spam is cancelled leaving TB unable to retrieve it, and offering to clean up the deletions.
>>
>
> Expired articles are also cleaned up when you "compact" a folder.

For many years I have had TB set up to automatically compact folders
when it saves over 10MB in total. Perhaps I should change that to 1MB to
make it more frequent.

> Or, if you run in offline mode, and download an entire group for
> later consumption, the expired headers also get removed in that case.

I don't use offline mode.

--

Jeff

On 2023-12-27 13:50, Jeff Layman wrote:
> On 27/12/2023 02:45, VanguardLH wrote:
>> "Carlos E.R." <robin_listas@es.invalid> wrote:
>>
>>> On 2023-12-26 22:07, VanguardLH wrote:
>>>> Jeff Layman <jeff@invalid.invalid> wrote:
>>>>
>>>>> Is there a way of proactively removing expired articles?
>>>>>
>>>>> I've noticed more issues recently where messages are being
>>>>> removed by a NoCeM listing. However, in some cases I will
>>>>> have downloaded the header before NoCeM has removed the
>>>>> message. TB then pops up an error message that the newsgroup
>>>>> server reports that it can't find the article. At the end of
>>>>> the error message TB offers to "Remove All Expired Articles".
>>>>>
>>>>> I tried TB's Help, but couldn't see a way to remove the
>>>>> expired articles prior to the popup appearing. Any ideas if
>>>>> it can be done?
>>>>
>>>> In my newsreader, after selecting a message,
>>>
>>> At this instant, TB will say the article has expired and offer to purge
>>> all expired messages in the current group.
>>>
>>>> I hit Delete to remove it
>>>> from the newsreaders message store.  Maybe Delete also works in Tbird.
>>
>> I don't get nuisanced with a popup.  A red-X'ed box shows up alongside
>> the message in the header list pane, I select it (no popup), and hit
>> Delete.  Seems the popup is an interferring nuisance, like deleting a
>> file and a popup appears asking if you really want to delete the file;
>> i.e., a safety step to prevent accidental deletion of something you can
>> [no longer] retrieve.
>
> Indeed. None of the expired articles are there anyway. It would be
> useful to have a checkbox in "Settings" which could be ticked so that
> the removal of all expired articles is automatic.

The purge is a slow operation, can take even half a minute if the group
has many thousands of messages. Having TB unresponsive when opening a
group would have the human think that TB crashed and restarting TB.

And a result, several messages (can be even hundreds) would disappear,
so the user would ask "what has just happened"?

>
>> If you first select a message that is not deleted on the server, and use
>> the arrow keys to move up/down the headers pane, do you get the Tbird
>> popup when the inaccessible message is highlighted?

Yes, but it is not a popup. It is text in what would be the body of the
article.

>
> No idea. Once the first error message appears I click on "Remove all
> expired articles". What I was hoping for was to find a way for that to
> be done automatically, perhaps every 30 minutes or so, or when I ask TB
> to download new headers. If TB can find all expired articles once I
> click on that button, why can't it be done for the first expired article
> which appears to be required to start the process in motion?
>

--
Cheers, Carlos.

On 12/27/23 14:10, Carlos E.R. wrote:
> And a result, several messages (can be even hundreds) would disappear,
> so the user would ask "what has just happened"?

That could easily be addressed by displaying a message saying that the
newsgroup / folder is being compacted.

--
Grant. . . .

On 2023-12-28 00:24, Grant Taylor wrote:
> On 12/27/23 14:10, Carlos E.R. wrote:
>> And a result, several messages (can be even hundreds) would disappear,
>> so the user would ask "what has just happened"?
>
> That could easily be addressed by displaying a message saying that the
> newsgroup / folder is being compacted.

Or, you could answer "no" to the message about the purge. Less damage,
less surprise, route.

--
Cheers, Carlos.