Last-modified: 2023-04-17
Posted-by: postfaq 1.17 (Perl 5.28.1)
Archive-name: usenet/software/inn2-faq
URL: https://www.eyrie.org/~eagle/faqs/inn.html
Posting-frequency: monthly

This FAQ is intended to answer frequently asked questions concerning the
current versions of INN (INN 2.x and later) seen on news.software.nntp.
It should be referred to in preference to the old INN FAQ, which only
documents versions up to 1.7. It mostly covers INN 2.3 and later; earlier
versions of INN may behave differently or use different configuration
files.

If you're reading this on Usenet, this FAQ is formatted as a minimal
digest, so if your news or mail reader has digest handling capabilities
you can use them to navigate between sections. In rn variants, you can
use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
each section into a separate article.

Please send any comments, suggestions, or updates to <eagle@eyrie.org>.
Bear in mind when sending me e-mail that I receive upwards of 800 mail
messages a day and have unanswered personal e-mail dating back six months
or more, so please don't expect an immediate response. You may receive
quicker responses by posting to news.software.nntp (even, due to the
quirky way in which I read mail and news, from me).

This FAQ is posted monthly to news.software.nntp, and is available on the
web at <https://www.eyrie.org/~eagle/faqs/inn.html>.

------------------------------

Subject: Contents

1. General Questions
1.1. What is INN?
1.2. What is the current version?
1.3. Where can I get INN?
1.4. Where can I find documentation?
1.5. What newsgroups are there for INN?
1.6. What mailing lists are there for INN?
1.7. How can I support INN development?
1.8. How can I contribute to INN?

2. Terms
2.1. What is tradspool (traditional spool)?
2.2. What is CNFS?
2.3. What are timehash and timecaf?
2.4. What is overview?
2.5. What are deferrals (NNTP code 431)?

3. Specific Problems
3.1. INN won't start after a new installation
3.3. The news server isn't keeping up with incoming news
3.4. news.notice is empty and the nightly report is missing things
3.5. INN is running out of file descriptors
3.6. Can't get debugging information out of INN
3.7. Articles aren't being sent to remote peers
3.8. sendmail isn't installed

4. Error Messages
4.1. innd: SERVER cant store article
4.2. innd: SERVER internal no control and/or junk group
4.3. Modification of read-only value attempted (Cleanfeed)
4.4. tradspool: could not open ... File exists
4.5. Binary posting to non-binary group (Cleanfeed)

5. Problems on Specific Systems
5.1. INN won't compile on SCO OpenServer / UnixWare / OpenUNIX
5.2. Using raw devices on Solaris destroys the partition table
5.3. Will INN run on Windows?
5.4. Why aren't INN's files where the documentation says they are?
5.5. Running INN on macOS

6. How Do I...
6.1. Set up a server with no external feeds, just local groups
6.2. Process a single control message
6.4. Feed all articles on a server to another server
6.5. Rename a newsgroup
6.6. Change the domain used for message IDs
6.7. Use INN without a direct news feed
6.8. Generate MRTG graphs for INN
6.9. Hide the junk and control groups from users
6.10. Modify the body of posts made through my server
6.11. Hide the Injection-Info header field
6.12. Run innd and nnrpd on separate ports
6.13. Back up and restore an INN installation
6.14. Find external feeds and set up peering

(Note that some numbers have been skipped. When questions are removed,
the remaining questions are not renumbered to avoid breaking links in
Usenet and mailing list archives.)

------------------------------

Subject: 1. General Questions

Contained in this section are general questions about INN, where to find
it, and things of that sort. It is aimed at the person who is not yet
running INN, or who has general questions about how it works.

------------------------------

Subject: 1.1. What is INN?

The README that comes with INN has this to say (in part):

INN (InterNetNews), originally written by Rich Salz, is an extremely
flexible and configurable Usenet / Netnews news server. For a
complete description of the protocols behind Usenet and Netnews, see
RFC 3977 (NNTP), RFC 4642 updated by RFC 8143 (TLS/NNTP), RFC 4643 (NNTP
authentication), RFC 4644 (streaming NNTP feeds), RFC 5536 (USEFOR), RFC
5537 (USEPRO), RFC 6048 (NNTP LIST additions), RFC 8054 (NNTP compression)
and RFC 8315 (Cancel-Lock) or their replacements.

In brief, Netnews is a set of protocols for exchanging messages between
a decentralized network of news servers. News articles are organized
into newsgroups, which are themselves organized into hierarchies.
Each individual news server stores locally all articles it has received
for a given newsgroup, making access to stored articles extremely fast.
Netnews does not require any central server; instead, each news server
passes along articles it receives to all of the news servers it peers
with, those servers pass the articles along to their peers, and so on,
resulting in "flood fill" propagation of news articles.

INN is free software, supported by Internet Systems Consortium and
volunteers around the world.

For a more complete answer, see that file. A full description of what
Usenet and Netnews are is beyond the scope of this document; for a
beginner's introduction, see the news.newusers.questions home page at
<http://www.tokak.us/nnq/>.

------------------------------

Subject: 1.2. What is the current version?

The most recently released version of INN is 2.7.1.

INN development proceeds in two branches, as with many other free software
projects. The STABLE branch is maintenance of the most recently released
stable version, and only bug fixes are added to it. The CURRENT branch is
the development version of the next release of INN.

As mentioned in the next section, when installing a new INN server, you
may wish to download the latest snapshot of the STABLE branch rather than
the current full release.

Note that the previous STABLE series for INN 2.6 terminated in the release
of INN 2.6.5 and current STABLE snapshots are based on INN 2.7. You
should therefore read the upgrade instructions in NEWS when upgrading from
a STABLE snapshot before July 11th, 2022 to one dated after that.

------------------------------

Subject: 1.3. Where can I get INN?

The download site for INN is <https://ftp.isc.org/isc/inn/>. In that
directory are the various releases of INN, some additional documentation
(particularly of security holes), the original INN Usenix paper.

There is also a snapshots subdirectory, in which you will find two sets
of snapshots: ones at the top level, which are updated only when the
code changes, and ones in the daily subdirectory, which are generated
every day and retained for seven days. The daily snapshots with
STABLE in the name are the latest versions of the STABLE branch and
may have some additional bug fixes over the current released version.
The daily snapshots with CURRENT in the name are of the current
development version.

Please note: There is no guarantee that a snapshot will even compile, let
alone function well as a news server. In particular, the CURRENT branch
is under active development, and all sorts of things may be broken at any
given point in time. Use snapshots with caution, and don't use snapshots
from the CURRENT branch on any production system unless you're prepared to
debug the inevitable problems in code that's actively changing and not yet
thoroughly tested. (The STABLE snapshots should be fairly reliable,
however.)

------------------------------

Subject: 1.4. Where can I find documentation?

INN comes with extensive documentation. See the files INSTALL and README
at the top level of the source tree, for starters. In addition, nearly
every program and configuration file has its own Unix man page. The best
place to start is by reading the entire INSTALL file and then from there
discovering which configuration files and programs do what you want to do
and reading their individual man pages.

There are HTML conversions of the documentation that comes with recent
versions of INN available at:

<https://www.eyrie.org/~eagle/software/inn/>

For additional documentation beyond what is distributed with INN, follow
the links suggested in the above page.

The documentation that comes with INN is fairly technical in nature and
lacking in some more general details on configuring news servers. Some of
the links off of the INN home page have additional overview documentation
or documentation on how to set up servers for specific roles.

Another good resource is the newsgroup news.software.nntp (and the Google
archives thereof) and the archive of the inn-workers mailing list. A link
to the latter is off the INN page referenced above.

Hi all,

> Archive-name: usenet/software/inn2-faq
> URL: https://www.eyrie.org/~eagle/faqs/inn.html

For the recall, contributions are welcome to update the document. Do
not hesitate to send modifications, or new questions to add to the FAQ.

Nigel also recently asked for more interaction and search features with
the use of wikis. If people want to contribute to existing wikis or
installation guides, do not hesitate to do so likewise!

http://www.dodin.org/wiki/pmwiki.php?n=Doc.ConfigurerINN-2021

https://th-h.de/net/usenet/servers/inn/overview/ (in German)

https://web.archive.org/web/20230901182332/https://git.alphanet.ch/gitweb/?p=inn-install;a=blob_plain;f=README.html
(in French, to be revived)

In the next posting of the FAQ:

> 4.4. tradspool: could not open ... File exists

scanspool will be mentioned as a useful program to run.

> 6.7. Use INN without a direct news feed

pullnews will be referenced as a possible program to use (only suck and
newsx were mentioned).

> 6.14. Find external feeds and set up peering

A sentence about importing old articles will be added, pointing to the
manual page of pullnews which explains how to do that, as well as how
not to propagate these pulled articled to external feeds.

--
Julien ÉLIE

« If you lie to the compiler, it will get its revenge. » (Henry Spencer)

On Wed, 27 Dec 2023 00:29:07 +0100
Julien ÉLIE <iulius@nom-de-mon-site.com.invalid> wrote:

> Hi all,
>
> > Archive-name: usenet/software/inn2-faq
> > URL: https://www.eyrie.org/~eagle/faqs/inn.html
>
> For the recall, contributions are welcome to update the document. Do
> not hesitate to send modifications, or new questions to add to the
> FAQ.
>
> Nigel also recently asked for more interaction and search features
> with the use of wikis. If people want to contribute to existing
> wikis or installation guides, do not hesitate to do so likewise!
>
> http://www.dodin.org/wiki/pmwiki.php?n=Doc.ConfigurerINN-2021
>
> https://th-h.de/net/usenet/servers/inn/overview/ (in German)
>
>
> https://web.archive.org/web/20230901182332/https://git.alphanet.ch/gitweb/?p=inn-install;a=blob_plain;f=README.html
> (in French, to be revived)
>
>
>
> In the next posting of the FAQ:
>
>
> > 4.4. tradspool: could not open ... File exists
>
> scanspool will be mentioned as a useful program to run.
>
>
> > 6.7. Use INN without a direct news feed
>
> pullnews will be referenced as a possible program to use (only suck
> and newsx were mentioned).
>
>
> > 6.14. Find external feeds and set up peering
>
> A sentence about importing old articles will be added, pointing to
> the manual page of pullnews which explains how to do that, as well as
> how not to propagate these pulled articled to external feeds.
>

Thanks for the additions. I don't want to appear like a complete noob,
but I perform operations on the usenet server so rarely that it's not
something I keep in the front of my mind. I should probably keep better
notes for myself.

--
End Of The Line BBS - Plano, TX
telnet endofthelinebbs.com 23

Hi Nigel,

> Thanks for the additions. I don't want to appear like a complete noob

Far be it from me to have that thought. There was no offense in my
article and did not intend to make you feel like that. You already know
many things about INN and Usenet!

My message was just to ask for ideas of questions to add to the FAQ or
more examples in manual pages.

> but I perform operations on the usenet server so rarely that it's not
> something I keep in the front of my mind.

That's naturally fine.

> I should probably keep better notes for myself.

.... or write them into a wiki for other people to take benefit of them.
Maybe Grant and other people here could even complete with their own
notes gathered year after year :)

--
Julien ÉLIE

« Quae longo tempore extenuentur corpora, lente reficere oportet. »
(Hippocrate)

Julien ÉLIE <iulius@nom-de-mon-site.com.invalid> composa la prose suivante:

>Hi all,
>
>> Archive-name: usenet/software/inn2-faq
>> URL: https://www.eyrie.org/~eagle/faqs/inn.html
>
>For the recall, contributions are welcome to update the document. Do
>not hesitate to send modifications, or new questions to add to the FAQ.
>
>Nigel also recently asked for more interaction and search features with
>the use of wikis. If people want to contribute to existing wikis or
>installation guides, do not hesitate to do so likewise!
>
> http://www.dodin.org/wiki/pmwiki.php?n=Doc.ConfigurerINN-2021
>
> https://th-h.de/net/usenet/servers/inn/overview/ (in German)
>
>
>https://web.archive.org/web/20230901182332/https://git.alphanet.ch/gitweb/?p=inn-install;a=blob_plain;f=README.html
>(in French, to be revived)
>
>
>
>In the next posting of the FAQ:
>
>
>> 4.4. tradspool: could not open ... File exists
>
>scanspool will be mentioned as a useful program to run.
>
>
>> 6.7. Use INN without a direct news feed
>
>pullnews will be referenced as a possible program to use (only suck and
>newsx were mentioned).
>
>
>> 6.14. Find external feeds and set up peering
>
>A sentence about importing old articles will be added, pointing to the
>manual page of pullnews which explains how to do that, as well as how
>not to propagate these pulled articled to external feeds.

Oh yes, a clear manual for importing one or more article be nice.
I've already done it for the entire "fr" hierarchie with "suck" and
your help for inn2 configuration ;-)

- I'd also like to see a point for transferring a server to another machine with
the least possible hassle for users (no renumbering of items or other hassles).
I have an ulterior motive: I'm going to change server soon.

- The same question when changing storage method (I plan to switch from
tradspool to overdb)

Thank's for your job, here, and on the french hierarchie.

On Wed, 27 Dec 2023 21:25:06 +0100
Julien ÉLIE <iulius@nom-de-mon-site.com.invalid> wrote:

> > I should probably keep better notes for myself.
>
> ... or write them into a wiki for other people to take benefit of
> them. Maybe Grant and other people here could even complete with
> their own notes gathered year after year :)

Maybe I'll wait for them to do it first and then I can add my tiny bits
:)

--
End Of The Line BBS - Plano, TX
telnet endofthelinebbs.com 23

Bonjour llp,

> - I'd also like to see a point for transferring a server to another machine with
> the least possible hassle for users (no renumbering of items or other hassles).
> I have an ulterior motive: I'm going to change server soon.

Such a transfer is normally described in "6.4. Feed all articles on a
server to another server" of the FAQ.
I wish you the best for your upcoming change of server.

> - The same question when changing storage method (I plan to switch from
> tradspool to overdb)

I suggest that you do the change while changing your server. This way,
there's no forced rebuild to do as your new server will recreate
overview data when receiving articles. You just have to configure it
with the wanted overview storage method in the ovmethod parameter of
inn.conf.

--
Julien ÉLIE

« Donner un sens plus pur aux mots de la tribu. » (Stéphane Mallarmé)

Thus spake Julien ÉLIE <iulius@nom-de-mon-site.com.invalid>

Just two remarks to avoid some issues that can lead to problems and
unpalnned outages of the server:

> Bonjour llp,
>> - I'd also like to see a point for transferring a server to another machine with
>> the least possible hassle for users (no renumbering of items or other hassles).
>> I have an ulterior motive: I'm going to change server soon.
> Such a transfer is normally described in "6.4. Feed all articles on a
> server to another server" of the FAQ.
> I wish you the best for your upcoming change of server.

Always use xrefslave: true on the target server during the transfer to
keep the article numbers the same as on the source server to avoid
coofusing your clients after you switch to the new server.

>> - The same question when changing storage method (I plan to switch from
>> tradspool to overdb)
> I suggest that you do the change while changing your server. This
> way, there's no forced rebuild to do as your new server will recreate
> overview data when receiving articles. You just have to configure it
> with the wanted overview storage method in the ovmethod parameter of
> inn.conf.

I strongly recommend not to use ovdb (Berkeley DB) as it can cause database
corruption after a crash and the rebuild is painfully slow. It can also
create absurdly large database files if you have large gaps in your
article numbers which directly affects the memory use of your nnrpd
processes.

It also lacks utilities like tdx-util or ovsqlite-util. Depending on the number
of clients it also requires tweaking the DB_CONFIG to improve
performance. I used ovdb for almost 2 years and eventually went back to
tradindexed after the Linux kernel bug that killed servers in full
flight at midnight on New Years Day 2012 and completey destroyed my
overview database.

I'm currently testing a migration to ovsqlite and so far it looks much
more reliable than ovdb.

--
Пу́тін — хуйло́
https://www.eternal-september.org

Den 2023-12-26 skrev Julien ÉLIE <iulius@nom-de-mon-site.com.invalid>:
> Hi all,
>
>> Archive-name: usenet/software/inn2-faq
>> URL: https://www.eyrie.org/~eagle/faqs/inn.html
>
> For the recall, contributions are welcome to update the document. Do
> not hesitate to send modifications, or new questions to add to the FAQ.
>

I don't know if this is something that belongs in the FAQ, but, I
think, a year or so back, we had the issue that our INN process on
FreeBSD suddenly became unresponsive and started consuming 100 % CPU.
After debugging, it turned out that we hit the max number of fds
select() could handle. Switching overview method to ovsqlite solved
the problem.

Ray Banana <rayban@raybanana.net> composa la prose suivante:

>Thus spake Julien ÉLIE <iulius@nom-de-mon-site.com.invalid>
>
>Just two remarks to avoid some issues that can lead to problems and
>unpalnned outages of the server:
>
>> Bonjour llp,
>>> - I'd also like to see a point for transferring a server to another machine with
>>> the least possible hassle for users (no renumbering of items or other hassles).
>>> I have an ulterior motive: I'm going to change server soon.
>> Such a transfer is normally described in "6.4. Feed all articles on a
>> server to another server" of the FAQ.
>> I wish you the best for your upcoming change of server.
>
>Always use xrefslave: true on the target server during the transfer to
>keep the article numbers the same as on the source server to avoid
>coofusing your clients after you switch to the new server.

Ok.

>>> - The same question when changing storage method (I plan to switch from
>>> tradspool to overdb)
>> I suggest that you do the change while changing your server. This
>> way, there's no forced rebuild to do as your new server will recreate
>> overview data when receiving articles. You just have to configure it
>> with the wanted overview storage method in the ovmethod parameter of
>> inn.conf.
>
>I strongly recommend not to use ovdb (Berkeley DB) as it can cause database
>corruption after a crash and the rebuild is painfully slow. It can also
>create absurdly large database files if you have large gaps in your
>article numbers which directly affects the memory use of your nnrpd
>processes.
>
>It also lacks utilities like tdx-util or ovsqlite-util. Depending on the number
>of clients it also requires tweaking the DB_CONFIG to improve
>performance. I used ovdb for almost 2 years and eventually went back to
>tradindexed after the Linux kernel bug that killed servers in full
>flight at midnight on New Years Day 2012 and completey destroyed my
>overview database.
>
>I'm currently testing a migration to ovsqlite and so far it looks much
>more reliable than ovdb.

Good to know, i'll try ovsqlite.
The aim is to have an "up-to-date" overview even if messages are deleted during
the day (I don't know if I'm making myself clear?).

Bonsoir llp,

> Good to know, i'll try ovsqlite.
> The aim is to have an "up-to-date" overview even if messages are deleted during
> the day (I don't know if I'm making myself clear?).

When using tradindexed (your current overview method if I understand
well), the contents of the returned overview data is updated as soon as
a message is cancelled or removed by a NoCeM notice.
Do you currently have a different behaviour?

--
Julien ÉLIE

« Il buvait toutes mes paroles, et comme je parlais beaucoup, à un
moment, je le vois qui titubait… » (Raymond Devos)

Hi Wolfgang,

> I strongly recommend not to use ovdb (Berkeley DB) as it can cause database
> corruption after a crash and the rebuild is painfully slow. It can also
> create absurdly large database files if you have large gaps in your
> article numbers which directly affects the memory use of your nnrpd
> processes.

Thanks for your feedback, notably about the resilience. I'll mention
that in the documentation.

> It also lacks utilities like tdx-util or ovsqlite-util. Depending on the number
> of clients it also requires tweaking the DB_CONFIG to improve
> performance.

Do you have recommendations of tweaks in DB_CONFIG?
We only speak about putting the transaction logs on a different disk in
the DB_CONFIG section of ovdb(5). If you have suggestions to share, I
can add them to the manual page.

--
Julien ÉLIE

« Il vaut mieux avoir de l'avenir que du passé. »

Hi Andreas,

> I don't know if this is something that belongs in the FAQ, but, I
> think, a year or so back, we had the issue that our INN process on
> FreeBSD suddenly became unresponsive and started consuming 100 % CPU.
> After debugging, it turned out that we hit the max number of fds
> select() could handle. Switching overview method to ovsqlite solved
> the problem.

Interesting to know. Did you change the overcachesize setting in
inn.conf? (The default value is 128.)

This issue is probably fixed in the upcoming 2.7.2 release (in a few
months) as I recently added checks in innd not to open more file
descriptors than the size of FD_SET. I bet the 100 % CPU consumption is
related to that problem (channels may have been wrongly marked at a
wrong state).
From the changelog: "innd no longer malfunctions nor throttles when the
maximum number of file descriptors supported by the system is reached.
If needing to use more file descriptors than the default system limit, a
new LARGE_FD_SETSIZE option can be set at build time. See the
documentation for rlimitnofile in inn.conf for more information. Thanks
to Jesse Rehmer for the bug report."

--
Julien ÉLIE

« XXII ! Voilà les Romains ! » (Astérix)

Il se trouve que Julien ÉLIE a formulé :
> Bonsoir llp,
>
>> Good to know, i'll try ovsqlite.
>> The aim is to have an "up-to-date" overview even if messages are deleted
>> during
>> the day (I don't know if I'm making myself clear?).
>
> When using tradindexed (your current overview method
> if I understand well),

Absolutely.

> the contents of the returned overview data is updated as soon
> as a message is cancelled or removed by a NoCeM notice.
> Do you currently have a different behaviour?

When I read the list of new messages on a group, the message headers
are loaded, but when I ask for the message body, this is of course not
available and my news reader (mesnews) marks the message as deleted.

This is why I wanted to switch to ovsqlite.

Bonsoir llp,

>> the contents of the returned overview data is updated as soon
>> as a message is cancelled or removed by a NoCeM notice.
>> Do you currently have a different behaviour?
>
> When I read the list of new messages on a group, the message headers
> are loaded, but when I ask for the message body, this is of course not
> available and my news reader (mesnews) marks the message as deleted.

Strange. When an article is cancelled, its entry is normally
immediately blanked out in the tradindexed index file of the
corresponding newsgroup(s). Overview data for cancelled articles should
not be returned to news clients

> This is why I wanted to switch to ovsqlite.

I hope you'll have a better experience with ovsqlite, though I do not
understand why tradindexed returns cancelled articles in overview data.

--
Julien ÉLIE

« Life is short… so eat dessert first! »

Dans son message précédent, Julien ÉLIE a écrit :
> Bonsoir llp,
>
>>> the contents of the returned overview data is updated as soon
>>> as a message is cancelled or removed by a NoCeM notice.
>>> Do you currently have a different behaviour?
>>
>> When I read the list of new messages on a group, the message headers
>> are loaded, but when I ask for the message body, this is of course not
>> available and my news reader (mesnews) marks the message as deleted.
>
> Strange. When an article is cancelled, its entry is normally immediately
> blanked out in the tradindexed index file of the corresponding newsgroup(s).
> Overview data for cancelled articles should not be returned to news clients

Does the inn2 version matter for this?
I use version 2.6.4

>> This is why I wanted to switch to ovsqlite.
>
> I hope you'll have a better experience with ovsqlite, though I do not
> understand why tradindexed returns cancelled articles in overview data.

Is it the same problem ?
<news:8m8r7nmoop.fsf@raybanana.net>

Bonsoir llp,

>> When an article is cancelled, its entry is normally
>> immediately blanked out in the tradindexed index file of the
>> corresponding newsgroup(s). Overview data for cancelled articles
>> should not be returned to news clients
>
> Does the inn2 version matter for this?
> I use version 2.6.4

The version does not matter.

>>> This is why I wanted to switch to ovsqlite.
>>
>> I hope you'll have a better experience with ovsqlite, though I do not
>> understand why tradindexed returns cancelled articles in overview data.
>
> Is it the same problem ?
> <news:8m8r7nmoop.fsf@raybanana.net>

I don't know what the underlying problem was in that other thread. I
thought from past experience that the overview data for cancelled
articles was no longer returned to news clients when using tradindexed,
but Wolfgang indeed said the run of expireover was needed.
I must have been wrong then.

Looking at the code, overview entries of cancelled articles are
immediately blanked out. Seems like it no longer works; I shall retest
it then.

--
Julien ÉLIE

« Sol lucet omnibus. »

Den 2024-01-07 skrev Julien ÉLIE <iulius@nom-de-mon-site.com.invalid>:
> Hi Andreas,
>
>> I don't know if this is something that belongs in the FAQ, but, I
>> think, a year or so back, we had the issue that our INN process on
>> FreeBSD suddenly became unresponsive and started consuming 100 % CPU.
>> After debugging, it turned out that we hit the max number of fds
>> select() could handle. Switching overview method to ovsqlite solved
>> the problem.
>
> Interesting to know. Did you change the overcachesize setting in
> inn.conf? (The default value is 128.)
>

Checking our current configuration, it is set to 1024. I don't think
we have touched that setting since the server was installed so that's
probably the value it had when we were having issues.

> This issue is probably fixed in the upcoming 2.7.2 release (in a few
> months)

That's good to hear! Apologies for never getting around to reporting
the issue ourselves.

Den 2024-01-09 skrev Andreas Kempe <kempe@lysator.liu.se>:
> Den 2024-01-07 skrev Julien ÉLIE <iulius@nom-de-mon-site.com.invalid>:
>> Hi Andreas,
>>
>>> I don't know if this is something that belongs in the FAQ, but, I
>>> think, a year or so back, we had the issue that our INN process on
>>> FreeBSD suddenly became unresponsive and started consuming 100 % CPU.
>>> After debugging, it turned out that we hit the max number of fds
>>> select() could handle. Switching overview method to ovsqlite solved
>>> the problem.
>>
>> Interesting to know. Did you change the overcachesize setting in
>> inn.conf? (The default value is 128.)
>>
>
> Checking our current configuration, it is set to 1024. I don't think
> we have touched that setting since the server was installed so that's
> probably the value it had when we were having issues.
>

Checking the man page for inn.conf, it's coming back to me and you're
probably spot on with your assesment that overcachesize is what caused
the issue. 1024 fds is the default limit for select() on FreeBSD,
something we didn't realise at the time so we only checked the maximum
allowed open files limit. We bumped the setting because, back then,
the server was on storage that was really slow when it came to opening
files.

Either way, ovsqlite has been working really well for us so I see no
need to switch back even if we could.

Hi Andreas,

> Checking the man page for inn.conf, it's coming back to me and you're
> probably spot on with your assesment that overcachesize is what caused
> the issue. 1024 fds is the default limit for select() on FreeBSD,
> something we didn't realise at the time so we only checked the maximum
> allowed open files limit.

OK, thanks for your confirmation.
I am therefore pretty confident that the sanity-check fix in INN 2.7.2
solves your 100 % CPU problem.

Suppose you have 1000 open cache tradindexed slots, 24 fds used for innd
channels, and overcachesize set to 1024, then:
- more cache slots can be opened (up to 1024 slots, and therefore fd
number 1048, which is fine as there's no select() on these fds);
- if for whatever reason innd wants to create a new channel, then it
won't create it, refuse the connection, and log a warning ("SERVER file
descriptor 1049 too high in CHANcreate (see rlimitnofile in inn.conf)").
It won't become unresponsive, and the show goes on.

Let's just hope the news admin will one day catch these warnings. But
at least innd won't do weird things.

Thanks to your message, I note that the description of rlimitnofile does
not mention overcachesize, and the description of overcachesize does not
mention rlimitnofile. As they are somehow linked, I'll add a sentence
in both of them to mention the other one :)

https://www.eyrie.org/~eagle/software/inn/docs/inn.conf.html

> Either way, ovsqlite has been working really well for us so I see no
> need to switch back even if we could.

ovsqlite is indeed an efficient and reliable overview method. Thanks
again to Bo Lindbergh for having written it!

> Apologies for never getting around to reporting the issue ourselves.
No problem.
Thanks for mentioning it now, as it permits improving the wording of the
rlimitnofile and overcachesize descriptions.

--
Julien ÉLIE

« O fortunatos nimium, sua si bona norint, agricolas. » (Virgile)

On 2024-01-08 20:54, Julien ÉLIE wrote:
> Bonsoir llp,
>
>>> When an article is cancelled, its entry is normally immediately
>>> blanked out in the tradindexed index file of the corresponding
>>> newsgroup(s). Overview data for cancelled articles should not be
>>> returned to news clients
>>
> ...
>
>
>>>> This is why I wanted to switch to ovsqlite.
>>>
>>> I hope you'll have a better experience with ovsqlite, though I do not
>>> understand why tradindexed returns cancelled articles in overview data.
>>
>> Is it the same problem ?
>> <news:8m8r7nmoop.fsf@raybanana.net>
>
> I don't know what the underlying problem was in that other thread.  I
> thought from past experience that the overview data for cancelled
> articles was no longer returned to news clients when using tradindexed,
> but Wolfgang indeed said the run of expireover was needed.
> I must have been wrong then.
>
> Looking at the code, overview entries of cancelled articles are
> immediately blanked out.  Seems like it no longer works; I shall retest
> it then.
>

One common pattern is that some time passes between the client
retrieving the headers (often as an automated background task) and the
same client retrieving the article (often postponed to the moment the
user views the article offered by the cleint user interface).

The problem here is that if a spam article is removed between those two
network activities, the user is presented with an error message rather
than protected from seeing the spam. And often times, the ability to
make the client actively test for this has the negative side effect of
deleting old articles from the client's local message archive as soon as
upstream fails to retain them, thus users wanting retention of
previously read non-spam turn off those options and suffer from the
half-blanked messages in their user interface.

It would be better for client software implementation and use if NNTP
servers could return a result "this article deliberately cancelled, not
just expired" and this was sufficiently standardized for client software
to honour it without corrupting local archives.

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

Thus spake Jakob Bohm <jb-usenet@wisemo.invalid>

>> I thought from past experience that the overview data for cancelled
>> articles was no longer returned to news clients when using
>> tradindexed, but Wolfgang indeed said the run of expireover was
>> needed.
[...]
>> Looking at the code, overview entries of cancelled articles are
>> immediately blanked out.  Seems like it no longer works; I shall
>> retest it then.
> One common pattern is that some time passes between the client
> retrieving the headers (often as an automated background task) and the
> same client retrieving the article (often postponed to the moment the
> user views the article offered by the cleint user interface).
> The problem here is that if a spam article is removed between those two
> network activities, the user is presented with an error message rather
> than protected from seeing the spam. And often times, the ability to
> make the client actively test for this has the negative side effect of
> deleting old articles from the client's local message archive as soon as
> upstream fails to retain them, thus users wanting retention of
> previously read non-spam turn off those options and suffer from the
> half-blanked messages in their user interface.

After more testing I can confirm Jakob's explanation. Immediately after
locally cancelling articles, the overview data for the article had been
removed from tradindexed and sqlite overview databases. Clients
configured to automatically retrieve new overview data every 10 minutes
still showed the overview data and displayed an error message when the
user tried to open the article.

--
Пу́тін — хуйло́
https://www.eternal-september.org

Hi Jacob, Wolfgang,

>> One common pattern is that some time passes between the client
>> retrieving the headers (often as an automated background task) and the
>> same client retrieving the article (often postponed to the moment the
>> user views the article offered by the cleint user interface).
>> The problem here is that if a spam article is removed between those two
>> network activities, the user is presented with an error message rather
>> than protected from seeing the spam.

The client asks for article 42, and the server responds that this
article number does not exist at the time he asks. I am not under the
impression that the news client needs giving an error message. For a
better user experience, it should just go on and silently skip the
article (and records that in the logs, if any).

>> And often times, the ability to
>> make the client actively test for this has the negative side effect of
>> deleting old articles from the client's local message archive as soon as
>> upstream fails to retain them

I do not understand why a news client needs retrieving again an old
article (sending an ARTICLE command to the server) when it already has
it in its local message archive. Why not just display the already
downloaded article?
Besides, even if it does that, it doesn't have to delete it from its
local message archive.

>> thus users wanting retention of
>> previously read non-spam turn off those options and suffer from the
>> half-blanked messages in their user interface.

This indeed is not very user-friendly. There should be an option to
prevent already downloaded old messages to be deleted.

> After more testing I can confirm Jakob's explanation. Immediately after
> locally cancelling articles, the overview data for the article had been
> removed from tradindexed and sqlite overview databases.

Yes, that's normal. Cancelled articles should not be returned to news
clients (and therefore should not appear in the overview database).

> Clients configured to automatically retrieve new overview data every
> 10 minutes still showed the overview data
The client usually asks for new overview data since its last retrieval.
If an article is cancelled but its overview data has already been
downloaded by the previous automatical retrieval, yes I believe it still
shows up.

> and displayed an error message when the user tried to open the article.

If it had been configured by the user to show an error message, then it
is normal. I do not believe the client should explicitly return an
error. Retrieving an article by its number after having downloaded the
overview data is a common case, and the article may not be available. I
think the client should just skip it if it had not already downloaded
the article, and of course do not delete it if it had already downloaded
it and has it in its local archive.

--
Julien ÉLIE

« Passion is inversely proportional to the amount of real information
available. » (Benford's law)