When I Get New Headers from a particular newsgroup. It keeps finding
600+ unavailable headers. How can I stop these headers from being
attempted every time?

On Tue, 27 Sep 2022 13:14:57 +0100, Martin Harran wrote:

> When I Get New Headers from a particular newsgroup. It keeps finding
> 600+ unavailable headers. How can I stop these headers from being
> attempted every time?

When you write "600+ unavailable headers", what exactly do you mean?
For example:
A) The status bar says "Retrieving up to 642 new headers",
but only 42 new headers appear in the message list.
B) 642 new headers appear in the message list, but bodies
are only available for 42 of the 642 headers.

Also:
1) In your server settings, do you have a check-mark in
"The server creates messages pout of order"?
2) Is that in a binary group with multi-part messages?
3) Is this a binary newsgroup which may be subject to
EUCD/DMCA takedowns?

--
Kind regards
Ralph
🦊

On Wed, 28 Sep 2022 07:03:49 +1300, Ralph Fox <-rf-nz-@-.invalid>
wrote:

>On Tue, 27 Sep 2022 13:14:57 +0100, Martin Harran wrote:
>
>> When I Get New Headers from a particular newsgroup. It keeps finding
>> 600+ unavailable headers. How can I stop these headers from being
>> attempted every time?
>
>
>When you write "600+ unavailable headers", what exactly do you mean?
>For example:
> A) The status bar says "Retrieving up to 642 new headers",
> but only 42 new headers appear in the message list.
> B) 642 new headers appear in the message list, but bodies
> are only available for 42 of the 642 headers.

Sorry, Ralph, I got this wrong - it's marked bodies that are the
problem. A typical result is:

Completed Task: Get marked bodies for [newsgroup]

Result: Success: Bodies retrieved:6 complete, 675 unavailable, 0
cancelled

I have tried selecting all messages in my list pane, marking them all
for Body Retrieval and then Cancel Retrieval but it has no effect.

I think this goes back to me retrieving all headers and bodies from
the group when I did a new installation of agent.

"No" to your other 3 questions.

>Also:
> 1) In your server settings, do you have a check-mark in
> "The server creates messages pout of order"?
> 2) Is that in a binary group with multi-part messages?
> 3) Is this a binary newsgroup which may be subject to
> EUCD/DMCA takedowns?

On Wed, 28 Sep 2022 09:23:25 +0100, Martin Harran wrote:
> On Wed, 28 Sep 2022 07:03:49 +1300, Ralph Fox <-rf-nz-@-.invalid>
> wrote:
>> On Tue, 27 Sep 2022 13:14:57 +0100, Martin Harran wrote:
>>
>>> When I Get New Headers from a particular newsgroup. It keeps finding
>>> 600+ unavailable headers. How can I stop these headers from being
>>> attempted every time?
>>
>>
>> When you write "600+ unavailable headers", what exactly do you mean?
>> For example:
>> A) The status bar says "Retrieving up to 642 new headers",
>> but only 42 new headers appear in the message list.
>> B) 642 new headers appear in the message list, but bodies
>> are only available for 42 of the 642 headers.
>
>
> Sorry, Ralph, I got this wrong - it's marked bodies that are the
> problem. A typical result is:
>
> Completed Task: Get marked bodies for [newsgroup]
>
> Result: Success: Bodies retrieved:6 complete, 675 unavailable, 0
> cancelled
>
> I have tried selecting all messages in my list pane, marking them all
> for Body Retrieval and then Cancel Retrieval but it has no effect.
>
> I think this goes back to me retrieving all headers and bodies from
> the group when I did a new installation of agent.

If you are retrieving bodies from the same news server as the headers,
then it may be that you are trying to retrieve bodies for older posts
which have now exceeded the server's retention period and have now
been expired off the server.

You cannot make your news server have the unavailable bodies, but
what you can do is configure Agent to purge any headers without bodies
when the message is no longer available on the server.

To configure this for all newsgroups: go to
"Folder >> Default Properties >> Purging Messages >> What to Purge"

To configure this for a single newsgroup: go to
"(select folder) >> Folder >> Properties >> Purging Messages >> What to Purge"

In either case, set both "Without Bodies" settings to "When message is
no longer available" and put a check-mark in "Usenet messages".

Each time you get new headers, Agent will also check with the news
server which old messages are still available on the server so it
can identify which headers to purge.

There are two ways Agent can check with the server, depending on the
setting 'Always get complete list of server's available messages'.

A) If this setting is check-marked, Agent will use a more accurate
but much more time-consuming method: each time you get new headers
in the newsgroup Agent will also asks the server for a complete
list of *all* messages in that newsgroup. If your news server has
a long retention, this can add a lot of time to getting new
headers.

B) If this setting is not check-marked, Agent will use a quick but
rough-and-ready method: Agent asks the server only for the lowest
and highest article numbers in the group. Any header with a lower
article number than the lowest, is considered to be "no longer
available".

Either way, Agent has to rely on what the server tells it (accurate or
not).

> "No" to your other 3 questions.
>
>> Also:
>> 1) In your server settings, do you have a check-mark in
>> "The server creates messages pout of order"?
>> 2) Is that in a binary group with multi-part messages?
>> 3) Is this a binary newsgroup which may be subject to
>> EUCD/DMCA takedowns?

--
Kind regards
Ralph

Agent Unicode test: 🕵️📰️ 👉️ 🎚️ 👉️ 🧹️📄️🗑️

Il Fri, 30 Sep 2022 19:04:59 +1300, Ralph Fox <-rf-nz-@-.invalid> ha
scritto:

>On Wed, 28 Sep 2022 09:23:25 +0100, Martin Harran wrote:
>> On Wed, 28 Sep 2022 07:03:49 +1300, Ralph Fox <-rf-nz-@-.invalid>
>> wrote:
>>> On Tue, 27 Sep 2022 13:14:57 +0100, Martin Harran wrote:
>>>
>>>> When I Get New Headers from a particular newsgroup. It keeps finding
>>>> 600+ unavailable headers. How can I stop these headers from being
>>>> attempted every time?
>>>
>>>
>>> When you write "600+ unavailable headers", what exactly do you mean?
>>> For example:
>>> A) The status bar says "Retrieving up to 642 new headers",
>>> but only 42 new headers appear in the message list.
>>> B) 642 new headers appear in the message list, but bodies
>>> are only available for 42 of the 642 headers.
>>
>>
>> Sorry, Ralph, I got this wrong - it's marked bodies that are the
>> problem. A typical result is:
>>
>> Completed Task: Get marked bodies for [newsgroup]
>>
>> Result: Success: Bodies retrieved:6 complete, 675 unavailable, 0
>> cancelled
>>
>> I have tried selecting all messages in my list pane, marking them all
>> for Body Retrieval and then Cancel Retrieval but it has no effect.
>>
>> I think this goes back to me retrieving all headers and bodies from
>> the group when I did a new installation of agent.
>
>
>If you are retrieving bodies from the same news server as the headers,
>then it may be that you are trying to retrieve bodies for older posts
>which have now exceeded the server's retention period and have now
>been expired off the server.
>
>You cannot make your news server have the unavailable bodies, but
>what you can do is configure Agent to purge any headers without bodies
>when the message is no longer available on the server.
>
>To configure this for all newsgroups: go to
>"Folder >> Default Properties >> Purging Messages >> What to Purge"
>
>To configure this for a single newsgroup: go to
>"(select folder) >> Folder >> Properties >> Purging Messages >> What to Purge"
>
>In either case, set both "Without Bodies" settings to "When message is
>no longer available" and put a check-mark in "Usenet messages".
>
>Each time you get new headers, Agent will also check with the news
>server which old messages are still available on the server so it
>can identify which headers to purge.
>
>There are two ways Agent can check with the server, depending on the
>setting 'Always get complete list of server's available messages'.
>
> A) If this setting is check-marked, Agent will use a more accurate
> but much more time-consuming method: each time you get new headers
> in the newsgroup Agent will also asks the server for a complete
> list of *all* messages in that newsgroup. If your news server has
> a long retention, this can add a lot of time to getting new
> headers.
>
> B) If this setting is not check-marked, Agent will use a quick but
> rough-and-ready method: Agent asks the server only for the lowest
> and highest article numbers in the group. Any header with a lower
> article number than the lowest, is considered to be "no longer
> available".
>
>Either way, Agent has to rely on what the server tells it (accurate or
>not).
>
>
>> "No" to your other 3 questions.
>>
>>> Also:
>>> 1) In your server settings, do you have a check-mark in
>>> "The server creates messages pout of order"?
>>> 2) Is that in a binary group with multi-part messages?
>>> 3) Is this a binary newsgroup which may be subject to
>>> EUCD/DMCA takedowns?

hi,

same problem here.

trying to purge messages read and not read "when message is no longher
available".

after purging the message switch from "retrieving 532 bodies for
it.comp.hardware" to *485* bodies but don't complete task untill i press
'stop tasks' button.

note that i have some messages from 2008 and several servers;)

any idea ?

thank you
--
Rado

On Fri, 30 Sep 2022 19:04:59 +1300, Ralph Fox <-rf-nz-@-.invalid>
wrote:

>On Wed, 28 Sep 2022 09:23:25 +0100, Martin Harran wrote:
>> On Wed, 28 Sep 2022 07:03:49 +1300, Ralph Fox <-rf-nz-@-.invalid>
>> wrote:
>>> On Tue, 27 Sep 2022 13:14:57 +0100, Martin Harran wrote:
>>>
>>>> When I Get New Headers from a particular newsgroup. It keeps finding
>>>> 600+ unavailable headers. How can I stop these headers from being
>>>> attempted every time?
>>>
>>>
>>> When you write "600+ unavailable headers", what exactly do you mean?
>>> For example:
>>> A) The status bar says "Retrieving up to 642 new headers",
>>> but only 42 new headers appear in the message list.
>>> B) 642 new headers appear in the message list, but bodies
>>> are only available for 42 of the 642 headers.
>>
>>
>> Sorry, Ralph, I got this wrong - it's marked bodies that are the
>> problem. A typical result is:
>>
>> Completed Task: Get marked bodies for [newsgroup]
>>
>> Result: Success: Bodies retrieved:6 complete, 675 unavailable, 0
>> cancelled
>>
>> I have tried selecting all messages in my list pane, marking them all
>> for Body Retrieval and then Cancel Retrieval but it has no effect.
>>
>> I think this goes back to me retrieving all headers and bodies from
>> the group when I did a new installation of agent.
>
>
>If you are retrieving bodies from the same news server as the headers,
>then it may be that you are trying to retrieve bodies for older posts
>which have now exceeded the server's retention period and have now
>been expired off the server.
>
>You cannot make your news server have the unavailable bodies, but
>what you can do is configure Agent to purge any headers without bodies
>when the message is no longer available on the server.
>
>To configure this for all newsgroups: go to
>"Folder >> Default Properties >> Purging Messages >> What to Purge"
>
>To configure this for a single newsgroup: go to
>"(select folder) >> Folder >> Properties >> Purging Messages >> What to Purge"
>
>In either case, set both "Without Bodies" settings to "When message is
>no longer available" and put a check-mark in "Usenet messages".
>
>Each time you get new headers, Agent will also check with the news
>server which old messages are still available on the server so it
>can identify which headers to purge.
>
>There are two ways Agent can check with the server, depending on the
>setting 'Always get complete list of server's available messages'.
>
> A) If this setting is check-marked, Agent will use a more accurate
> but much more time-consuming method: each time you get new headers
> in the newsgroup Agent will also asks the server for a complete
> list of *all* messages in that newsgroup. If your news server has
> a long retention, this can add a lot of time to getting new
> headers.
>
> B) If this setting is not check-marked, Agent will use a quick but
> rough-and-ready method: Agent asks the server only for the lowest
> and highest article numbers in the group. Any header with a lower
> article number than the lowest, is considered to be "no longer
> available".
>
>Either way, Agent has to rely on what the server tells it (accurate or
>not).
>
>
>> "No" to your other 3 questions.
>>
>>> Also:
>>> 1) In your server settings, do you have a check-mark in
>>> "The server creates messages pout of order"?
>>> 2) Is that in a binary group with multi-part messages?
>>> 3) Is this a binary newsgroup which may be subject to
>>> EUCD/DMCA takedowns?

Just to update, I have eventually solved this. It seems to have been a
server issue. I use two servers news.individual.net and
news.eternal-september.org and both were showing the same number of
unavailable headers. I took out a trial account with newshosting.com
and it downloaded the missing headers. Looks like newshosting is
better than my two existing servers at retention but not worth $10+
per month for me.