I'm noticing very slow XOVER behavior with some of the larger groups on my
server. When I look at what nnrpd is doing it appears to be touching every
article on the spool for that group during XOVER. I *thought* the data
provided by the XOVER command came from the overview database. What I see with
truss is that it is accessing every article and while I'm watching never
accesses the overview database.

Is this expected behavior from XOVER? This is a large group with 1.3 million
articles, so I don't expect instant response, but surprised to see the nnrpd
process doesn't appear to be using overview data from what I can see with
truss:

<snipped millions of similar lines>
access("/usr/local/news/spool/articles/rec/arts/tv/1279573",R_OK) = 0 (0x0)
access("/usr/local/news/spool/articles/rec/arts/tv/1279574",R_OK) = 0 (0x0)
access("/usr/local/news/spool/articles/rec/arts/tv/1279575",R_OK) = 0 (0x0)
access("/usr/local/news/spool/articles/rec/arts/tv/1279576",R_OK) = 0 (0x0)
access("/usr/local/news/spool/articles/rec/arts/tv/1279577",R_OK) = 0 (0x0)
access("/usr/local/news/spool/articles/rec/arts/tv/1279578",R_OK) = 0 (0x0)
access("/usr/local/news/spool/articles/rec/arts/tv/1279579",R_OK) = 0 (0x0)
access("/usr/local/news/spool/articles/rec/arts/tv/1279580",R_OK) = 0 (0x0)
access("/usr/local/news/spool/articles/alt/fan/rush-limbaugh/521666",R_OK) = 0
(0x0)
access("/usr/local/news/spool/articles/alt/politics/usa/127302",R_OK) = 0
(0x0)
access("/usr/local/news/spool/articles/alt/fan/rush-limbaugh/521668",R_OK) = 0
(0x0)
access("/usr/local/news/spool/articles/alt/fan/rush-limbaugh/521669",R_OK) = 0
(0x0)
access("/usr/local/news/spool/articles/alt/fan/rush-limbaugh/521670",R_OK) = 0
(0x0)
access("/usr/local/news/spool/articles/rec/arts/tv/1321380",R_OK) = 0 (0x0)
access("/usr/local/news/spool/articles/rec/arts/tv/1321381",R_OK) = 0 (0x0)
access("/usr/local/news/spool/articles/alt/fan/rush-limbaugh/521671",R_OK) = 0
(0x0)
access("/usr/local/news/spool/articles/alt/fan/rush-limbaugh/521672",R_OK) = 0
(0x0)
access("/usr/local/news/spool/articles/alt/politics/usa/127303",R_OK) = 0
(0x0)
access("/usr/local/news/spool/articles/rec/arts/tv/1321385",R_OK) = 0 (0x0)
access("/usr/local/news/spool/articles/alt/survival/84115",R_OK) = 0 (0x0)
access("/usr/local/news/spool/articles/rec/arts/tv/1321387",R_OK) = 0 (0x0)
write(1,"\^W\^C\^C@\^X_\M-'\M-$>\M^M\M^Y"...,16413) = 16413 (0x401d)
write(1,"\^W\^C\^C@\^X_\M-'\M-$>\M^M\M^Y"...,16413) = 16413 (0x401d)
write(1,"\^W\^C\^C@\^X_\M-'\M-$>\M^M\M^Y"...,16413) = 16413 (0x401d)
write(1,"\^W\^C\^C@\^X_\M-'\M-$>\M^M\M^Y"...,16413) = 16413 (0x401d)
write(1,"\^W\^C\^C@\^X_\M-'\M-$>\M^M\M^Y"...,16413) = 16413 (0x401d)
write(1,"\^W\^C\^C@\^X_\M-'\M-$>\M^M\M^Y"...,16413) = 16413 (0x401d)
write(1,"\^W\^C\^C@\^X_\M-'\M-$>\M^M\M^Y"...,16413) = 16413 (0x401d)
write(1,"\^W\^C\^C@\^X_\M-'\M-$>\M^M\M^Y"...,16413) = 16413 (0x401d)
write(1,"\^W\^C\^C\^TT_\M-'\M-$>\M^M\M^Y"...,5209) = 5209 (0x1459)
sigprocmask(SIG_SETMASK,{
SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGEMT|SIGFPE|SIGKILL|SIGBUS|SIG
SEGV|SIGSYS|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SI
GTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR
1|SIGUSR2 },{ }) = 0 (0x0)
sigaction(SIGALRM,{ 0x80192ba00 SA_RESTART|SA_SIGINFO ss_t },{ SIG_DFL
SA_RESTART ss_t }) = 0 (0x0)
sigprocmask(SIG_SETMASK,{ },0x0) = 0 (0x0)
setitimer(0,{ 0.000000, 1800.000000 },{ 0.000000, 0.000000 }) = 0 (0x0)

False alarm... Just realized at some point I set nnrpdcheckart to true in
inn.conf. Not sure why I thought I needed that or was a good idea. :)

Hi Jesse,

> False alarm... Just realized at some point I set nnrpdcheckart to true in
> inn.conf. Not sure why I thought I needed that or was a good idea.:)

"true" is the default value of nnrpdcheckart, so you did nothing wrong!
In your case, it could indeed be set to "false".
Did it greatly improve the response time of the command?

Also, note that using virtualhost in readers.conf slows down a bit
(X)OVER responses as the Xref header field present in overview data will
be rewritten. Not sure it really takes much time, though!

--
Julien ÉLIE

« C'est comme chercher une aiguille dans du foin en bottes ! »
(Jolitorax)

On Jul 8, 2022 at 10:08:01 AM CDT, "Julien ÉLIE" in
<ta9h8h$1eo15$1@news.trigofacile.com> wrote:

> Hi Jesse,
>
>> False alarm... Just realized at some point I set nnrpdcheckart to true in
>> inn.conf. Not sure why I thought I needed that or was a good idea.:)
>
> "true" is the default value of nnrpdcheckart, so you did nothing wrong!
> In your case, it could indeed be set to "false".
> Did it greatly improve the response time of the command?

Ah, my first thought was I turned it on when I was having trouble with
expireover, but this makes sense.

> Also, note that using virtualhost in readers.conf slows down a bit
> (X)OVER responses as the Xref header field present in overview data will
> be rewritten. Not sure it really takes much time, though!

Setting it to false *dramatically* improved XOVER performance, hard to
quantify but certainly orders of magnitude faster (few seconds for fetching
50,000 records vs. minutes).

I did read that note about a performance hit using virtualhost, but I can't
say I notice any difference after configuring virtualhost. I happened to
notice the slow XOVER response because I was subscribing to the largest group
in my newsreader to look at something else.

Hi Jesse,

>> "true" is the default value of nnrpdcheckart, so you did nothing wrong!
>> In your case, it could indeed be set to "false".
>> Did it greatly improve the response time of the command?
>
> Setting it to false *dramatically* improved XOVER performance, hard to
> quantify but certainly orders of magnitude faster (few seconds for fetching
> 50,000 records vs. minutes).

Checking the existence of articles (nnrpdcheckart) is indeed really
time-consuming for 50,000 records!

> I happened to
> notice the slow XOVER response because I was subscribing to the largest group
> in my newsreader to look at something else.

In such cases, the COMPRESS extension is also useful to speed up the
transmission of overview data. Unfortunately, I'm not aware of any news
client other than flnews that implemented it :-/

--
Julien ÉLIE

« Piscis primum a capite foetet. »

On Jul 8, 2022 at 10:59:52 AM CDT, "Julien ÉLIE" in
<ta9k9o$1eqj6$1@news.trigofacile.com> wrote:

> Hi Jesse,
>
>>> "true" is the default value of nnrpdcheckart, so you did nothing wrong!
>>> In your case, it could indeed be set to "false".
>>> Did it greatly improve the response time of the command?
>>
>> Setting it to false *dramatically* improved XOVER performance, hard to
>> quantify but certainly orders of magnitude faster (few seconds for fetching
>> 50,000 records vs. minutes).
>
> Checking the existence of articles (nnrpdcheckart) is indeed really
> time-consuming for 50,000 records!
>
>
>> I happened to
>> notice the slow XOVER response because I was subscribing to the largest group
>> in my newsreader to look at something else.
>
> In such cases, the COMPRESS extension is also useful to speed up the
> transmission of overview data. Unfortunately, I'm not aware of any news
> client other than flnews that implemented it :-/

I bet there are some newsreaders more focused around downloading binaries that
implement it. I sent a question to the Usenapp Support team asking if their
client does. Usenapp is relatively new and has zero documentation at the
moment.

The client seems well rounded for newsreading and binary downloads and from
what I've observed, more conformant to standards than similar predecessors
(Unison, for example). I prefer it for reading and composing over Thunderbird.
It has more of the 'old school' newsreader features that have disappeared
from many clients (killfiles, for example).