I have used giganews for grabbing old articles, but they only reach
2004. Does anyone have older text retention available over NNTP (i.e not
google newsgroups or web archives). I would love to slurp/archive
anything not stored on the major commercial providers.

If so, can you give a rough disk usage and storage backend?

I have seen people mention 50mb/day recently based on eternal-september
stats, so assuming the average daily usage is static since 1980, it
should be under 1TB.

If not, I am planning to inject articles from archive.org and anywhere
else I can find them.

Are there any issues with injecting posts from 30 years ago? I don't
peer with anyone but if I can get everything imported and renumbered
correctly for my local reader to understand, I might consider peering or
making a public NNTP connection available.

-------------

ZMarkGC

On May 14, 2023 at 4:56:39 PM CDT, "ZMarkGC" <ZMarkGC@example.com> wrote:

> I have used giganews for grabbing old articles, but they only reach
> 2004. Does anyone have older text retention available over NNTP (i.e not
> google newsgroups or web archives). I would love to slurp/archive
> anything not stored on the major commercial providers.
>
> If so, can you give a rough disk usage and storage backend?
>
> I have seen people mention 50mb/day recently based on eternal-september
> stats, so assuming the average daily usage is static since 1980, it
> should be under 1TB.
>
> If not, I am planning to inject articles from archive.org and anywhere
> else I can find them.
>
> Are there any issues with injecting posts from 30 years ago? I don't
> peer with anyone but if I can get everything imported and renumbered
> correctly for my local reader to understand, I might consider peering or
> making a public NNTP connection available.
>
> -------------
>
> ZMarkGC

The oldest available on-spool articles I've been able to obtain are from 2003,
not much farther back than GigaNews.

I've grabbed the Big8, de.*, it.*, and most of uk.* from ~2003 and I'm at
1.4TB with overview.

ZMarkGC wrote:

> I have used giganews for grabbing old articles, but they only reach
> 2004. Does anyone have older text retention available over NNTP (i.e not
> google newsgroups or web archives). I would love to slurp/archive
> anything not stored on the major commercial providers.

> If so, can you give a rough disk usage and storage backend?

> I have seen people mention 50mb/day recently based on eternal-september
> stats, so assuming the average daily usage is static since 1980, it
> should be under 1TB.

> If not, I am planning to inject articles from archive.org and anywhere
> else I can find them.

> Are there any issues with injecting posts from 30 years ago? I don't
> peer with anyone but if I can get everything imported and renumbered
> correctly for my local reader to understand, I might consider peering or
> making a public NNTP connection available.

It's been a while since I looked at them, but I grabbed some old archives
and took a look. The oldest ones I found (some were uni's sending their first
test article) had some differences in headers.

I can't remember right now the specifics, but it would take some (probably
simple) scripting to modify them to work correctly with current news servers.

--
Retro Guy

Retro Guy wrote:

> ZMarkGC wrote:

>> I have used giganews for grabbing old articles, but they only reach
>> 2004. Does anyone have older text retention available over NNTP (i.e not
>> google newsgroups or web archives). I would love to slurp/archive
>> anything not stored on the major commercial providers.

>> If so, can you give a rough disk usage and storage backend?

>> I have seen people mention 50mb/day recently based on eternal-september
>> stats, so assuming the average daily usage is static since 1980, it
>> should be under 1TB.

>> If not, I am planning to inject articles from archive.org and anywhere
>> else I can find them.

>> Are there any issues with injecting posts from 30 years ago? I don't
>> peer with anyone but if I can get everything imported and renumbered
>> correctly for my local reader to understand, I might consider peering or
>> making a public NNTP connection available.

> It's been a while since I looked at them, but I grabbed some old archives
> and took a look. The oldest ones I found (some were uni's sending their first
> test article) had some differences in headers.

> I can't remember right now the specifics, but it would take some (probably
> simple) scripting to modify them to work correctly with current news servers.

I found an example:

----------
Autzoo.101
test
utzoo!henry
Fri Feb 6 00:19:47 1981
first_test
This is the first U of T test of the Duke news program.
Here is some more text.
And some more.
----------

The newer the article, the less work the header needs to work properly.

--
Retro Guy

retro.guy@rocksolidbbs.com (Retro Guy) writes:

> I found an example: ----------
> Autzoo.101
> test
> utzoo!henry
> Fri Feb 6 00:19:47 1981
> first_test
> This is the first U of T test of the Duke news program.
> Here is some more text.
> And some more.
> ----------

> The newer the article, the less work the header needs to work properly.

This is the "A News" format (named after the software in use at the time).
There is an example in RFC-850 but not a specification (RFC-850 documents
the B News format). I think there may be a specification for it somewhere
in old software, but I'm not sure where off-hand.

RFC-1036 documents the modern format, and everything from that point
forward is *mostly* compatible. The B News format (RFC-850) looks more
like the modern format but has some interesting variations, such as Title
instead of Subject, Article-I.D. instead of Message-ID, and UUCP bang
paths for From addresses.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

On Sun, 14 May 2023 22:56:39 +0100
ZMarkGC <ZMarkGC@example.com> wrote:
> I have used giganews for grabbing old articles, but they only reach
> 2004. Does anyone have older text retention available over NNTP (i.e not
> google newsgroups or web archives). I would love to slurp/archive
> anything not stored on the major commercial providers.
>
> If so, can you give a rough disk usage and storage backend?
>
> I have seen people mention 50mb/day recently based on eternal-september
> stats, so assuming the average daily usage is static since 1980, it
> should be under 1TB.
>
> If not, I am planning to inject articles from archive.org and anywhere
> else I can find them.

https://www.xach.com/naggum/articles/notes.html has a link to a
comp.lang.lisp archive , http://data.xach.com.s3.amazonaws.com/cll.txt.gz .
This I think is close to what you're asking but specific to one newsgroup.
Earliest posts are from 1987. The moderator of comp.compilers also keeps a
comprehensive archive going back to the 1990s. You can find it with a bit of
googling.

> Are there any issues with injecting posts from 30 years ago? I don't
> peer with anyone but if I can get everything imported and renumbered
> correctly for my local reader to understand, I might consider peering or
> making a public NNTP connection available.

A public NNTP connection to such an archive would be amazing.

Spiros Bousbouras wrote:

> On Sun, 14 May 2023 22:56:39 +0100
> ZMarkGC <ZMarkGC@example.com> wrote:
>> I have used giganews for grabbing old articles, but they only reach
>> 2004. Does anyone have older text retention available over NNTP (i.e not
>> google newsgroups or web archives). I would love to slurp/archive
>> anything not stored on the major commercial providers.
>>
>> If so, can you give a rough disk usage and storage backend?
>>
>> I have seen people mention 50mb/day recently based on eternal-september
>> stats, so assuming the average daily usage is static since 1980, it
>> should be under 1TB.
>>
>> If not, I am planning to inject articles from archive.org and anywhere
>> else I can find them.

> https://www.xach.com/naggum/articles/notes.html has a link to a
> comp.lang.lisp archive , http://data.xach.com.s3.amazonaws.com/cll.txt.gz .
> This I think is close to what you're asking but specific to one newsgroup.
> Earliest posts are from 1987. The moderator of comp.compilers also keeps a
> comprehensive archive going back to the 1990s. You can find it with a bit of
> googling.

>> Are there any issues with injecting posts from 30 years ago? I don't
>> peer with anyone but if I can get everything imported and renumbered
>> correctly for my local reader to understand, I might consider peering or
>> making a public NNTP connection available.

> A public NNTP connection to such an archive would be amazing.

I've taken some time to modify some articles so that inn2 will accept them.
These are all from the 1980s.

I needed to change the Date: format, so all the articles now end up with
my timezone (MST), but the date/times are correct, just wrong timezone.
Removed 'Relay-Version', 'Posting-Version' and 'Date-Received' headers.

Now they post except for one exception. I still get '441 Can't set system Xref header field'
on some articles, but it is a minority of them.

I've started with the can.* hierarchy, and will continue through the rest of
what I have (which is a lot), but it will take me a long time to complete.

You are free to view and/or pull the articles from news.novalink.us:119 if
you are interested. It will probably take me most of the summer to get it all
done as I don't have a ton of free time to work on it, but I want to complete
at some point.

If anyone has suggestions on the above error (Xref), I'd be glad to try to get
those articles to post also.

No account required to read at news.novalink.us:119

--
Retro Guy

Hi Retro Guy,

> Removed 'Relay-Version', 'Posting-Version' and 'Date-Received' headers.
>
> Now they post except for one exception. I still get '441 Can't set
> system Xref header field' on some articles, but it is a minority of
> them.
>
> If anyone has suggestions on the above error (Xref), I'd be glad to try
> to get those articles to post also.

I would just suggest to remove existing Xref header fields, like you did
for Relay-Version & al.

I bet you'll find out that the more recent the articles are, the more
header fields you'll need adding in the list to remove as they are not
supposed to be present in posted articles.
Like X-Trace, X-Complaints-To, NNTP-Posting-Host, Injection-Info, etc.

--
Julien ÉLIE

« Je préfère glisser ma peau sous des draps pour le plaisir des sens que
de la risquer sous les drapeaux pour le prix de l'essence. » (Raymond
Devos)

Julien_ÉLIE wrote:

> Hi Retro Guy,

>> Removed 'Relay-Version', 'Posting-Version' and 'Date-Received' headers.
>>
>> Now they post except for one exception. I still get '441 Can't set
>> system Xref header field' on some articles, but it is a minority of
>> them.
>>
>> If anyone has suggestions on the above error (Xref), I'd be glad to try
>> to get those articles to post also.

> I would just suggest to remove existing Xref header fields, like you did
> for Relay-Version & al.

> I bet you'll find out that the more recent the articles are, the more
> header fields you'll need adding in the list to remove as they are not
> supposed to be present in posted articles.
> Like X-Trace, X-Complaints-To, NNTP-Posting-Host, Injection-Info, etc.

Thank you for the hints. I will go ahead and add these headers for deletion
as they don't need to be there anyway when posting as a READER.

Let's see how it goes :)

--
Retro Guy

On Jun 2, 2023 at 1:48:43 PM CDT, "Retro Guy" <Retro Guy> wrote:

> Spiros Bousbouras wrote:
>
>> On Sun, 14 May 2023 22:56:39 +0100
>> ZMarkGC <ZMarkGC@example.com> wrote:
>>> I have used giganews for grabbing old articles, but they only reach
>>> 2004. Does anyone have older text retention available over NNTP (i.e not
>>> google newsgroups or web archives). I would love to slurp/archive
>>> anything not stored on the major commercial providers.
>>>
>>> If so, can you give a rough disk usage and storage backend?
>>>
>>> I have seen people mention 50mb/day recently based on eternal-september
>>> stats, so assuming the average daily usage is static since 1980, it
>>> should be under 1TB.
>>>
>>> If not, I am planning to inject articles from archive.org and anywhere
>>> else I can find them.
>
>> https://www.xach.com/naggum/articles/notes.html has a link to a
>> comp.lang.lisp archive , http://data.xach.com.s3.amazonaws.com/cll.txt.gz .
>> This I think is close to what you're asking but specific to one newsgroup.
>> Earliest posts are from 1987. The moderator of comp.compilers also keeps a
>> comprehensive archive going back to the 1990s. You can find it with a bit of
>> googling.
>
>>> Are there any issues with injecting posts from 30 years ago? I don't
>>> peer with anyone but if I can get everything imported and renumbered
>>> correctly for my local reader to understand, I might consider peering or
>>> making a public NNTP connection available.
>
>> A public NNTP connection to such an archive would be amazing.
>
> I've taken some time to modify some articles so that inn2 will accept them.
> These are all from the 1980s.
>
> I needed to change the Date: format, so all the articles now end up with
> my timezone (MST), but the date/times are correct, just wrong timezone.
> Removed 'Relay-Version', 'Posting-Version' and 'Date-Received' headers.
>
> Now they post except for one exception. I still get '441 Can't set system Xref
> header field'
> on some articles, but it is a minority of them.
>
> I've started with the can.* hierarchy, and will continue through the rest of
> what I have (which is a lot), but it will take me a long time to complete.
>
> You are free to view and/or pull the articles from news.novalink.us:119 if
> you are interested. It will probably take me most of the summer to get it all
> done as I don't have a ton of free time to work on it, but I want to complete
> at some point.
>
> If anyone has suggestions on the above error (Xref), I'd be glad to try to get
> those articles to post also.
>
> No account required to read at news.novalink.us:119

Are you going to take a crack at the net.* stuff that's available in various
archives? That stuff I will definitely suck off of your server, if you do. :)

Keep us updated as you progress. If you come up with a scriptable or easily
repeatable process and need another machine to help munge/inject articles let
me know, I'd be happy to offer some assistance.

I'm still pulling stuff available on public spools and would love to get stuff
from archives, but this work is slow and time consuming. Took a break from
sucking because I need to switch out INN's article storage subsystem to CNFS
from tradspool. Starting to run into some stupid things with performance when
doing certain maintenance operations that is annoying (like expireover taking
days to a week or more to complete). Currently feeding my spool into another
machine at home with one large CNFS buffer and will see if it resolves some
annoyances of a large spool.

On Jun 2, 2023 at 2:27:28 PM CDT, "Julien ÉLIE"
<iulius@nom-de-mon-site.com.invalid> wrote:

>
> Hi Retro Guy,
>
>> Removed 'Relay-Version', 'Posting-Version' and 'Date-Received' headers.
>>
>> Now they post except for one exception. I still get '441 Can't set
>> system Xref header field' on some articles, but it is a minority of
>> them.
>>
>> If anyone has suggestions on the above error (Xref), I'd be glad to try
>> to get those articles to post also.
>
> I would just suggest to remove existing Xref header fields, like you did
> for Relay-Version & al.
>
> I bet you'll find out that the more recent the articles are, the more
> header fields you'll need adding in the list to remove as they are not
> supposed to be present in posted articles.
> Like X-Trace, X-Complaints-To, NNTP-Posting-Host, Injection-Info, etc.

When I use suck/pullnews, articles with these headers come in with no issue,
is this due to a difference in the way the message gets to INN?

Hi Jesse,

>> I bet you'll find out that the more recent the articles are, the more
>> header fields you'll need adding in the list to remove as they are not
>> supposed to be present in posted articles.
>> Like X-Trace, X-Complaints-To, NNTP-Posting-Host, Injection-Info, etc.
>
> When I use suck/pullnews, articles with these headers come in with no issue,
> is this due to a difference in the way the message gets to INN?

Yes, you've configured in incoming.conf your suck/pullnews connections
to be handled by innd.
Retro Guy uses nnrpd. He may want to try to feed innd, that's a good
idea (hoping it won't complain of missing headers).

--
Julien ÉLIE

« Hey, I had to let awk be better at *something*… » (Larry Wall)

On Jun 2, 2023 at 3:26:20 PM CDT, "Julien ÉLIE"
<iulius@nom-de-mon-site.com.invalid> wrote:

> Hi Jesse,
>
>>> I bet you'll find out that the more recent the articles are, the more
>>> header fields you'll need adding in the list to remove as they are not
>>> supposed to be present in posted articles.
>>> Like X-Trace, X-Complaints-To, NNTP-Posting-Host, Injection-Info, etc.
>>
>> When I use suck/pullnews, articles with these headers come in with no issue,
>> is this due to a difference in the way the message gets to INN?
>
> Yes, you've configured in incoming.conf your suck/pullnews connections
> to be handled by innd.
> Retro Guy uses nnrpd. He may want to try to feed innd, that's a good
> idea (hoping it won't complain of missing headers).

I never added anything to incoming.conf, but I'm running the tools on the same
server as INN. I never paid attention to how the tools actually 'post' the
articles to be honest.

Retro Guy wrote:

> Julien_ÉLIE wrote:

>> Hi Retro Guy,

>>> Removed 'Relay-Version', 'Posting-Version' and 'Date-Received' headers.
>>>
>>> Now they post except for one exception. I still get '441 Can't set
>>> system Xref header field' on some articles, but it is a minority of
>>> them.
>>>
>>> If anyone has suggestions on the above error (Xref), I'd be glad to try
>>> to get those articles to post also.

>> I would just suggest to remove existing Xref header fields, like you did
>> for Relay-Version & al.

>> I bet you'll find out that the more recent the articles are, the more
>> header fields you'll need adding in the list to remove as they are not
>> supposed to be present in posted articles.
>> Like X-Trace, X-Complaints-To, NNTP-Posting-Host, Injection-Info, etc.

> Thank you for the hints. I will go ahead and add these headers for deletion
> as they don't need to be there anyway when posting as a READER.

> Let's see how it goes :)

That helps. I actually did try to remove the Xref header previously, but I
must have had a typo or something. That error is gone now.

One other thing I forgot to mention is that I needed to remove lines of
just '.', so I converted them to '..', same as a newsreader should.

--
Retro Guy

Julien_ÉLIE wrote:

> Hi Jesse,

>>> I bet you'll find out that the more recent the articles are, the more
>>> header fields you'll need adding in the list to remove as they are not
>>> supposed to be present in posted articles.
>>> Like X-Trace, X-Complaints-To, NNTP-Posting-Host, Injection-Info, etc.
>>
>> When I use suck/pullnews, articles with these headers come in with no issue,
>> is this due to a difference in the way the message gets to INN?

> Yes, you've configured in incoming.conf your suck/pullnews connections
> to be handled by innd.
> Retro Guy uses nnrpd. He may want to try to feed innd, that's a good
> idea (hoping it won't complain of missing headers).

Yes, I'm using nnrpd. The uploading is easy, and I've written a script to
modify the headers so that is easy also.

One thing that would really make a difference is not needing to create the
groups by hand. Is it possible for inn2 to create groups on demand? That would
make all the difference.

--
Retro Guy

Jesse Rehmer wrote:

> On Jun 2, 2023 at 1:48:43 PM CDT, "Retro Guy" <Retro Guy> wrote:

>> Spiros Bousbouras wrote:
>>
>>> On Sun, 14 May 2023 22:56:39 +0100
>>> ZMarkGC <ZMarkGC@example.com> wrote:
>>>> I have used giganews for grabbing old articles, but they only reach
>>>> 2004. Does anyone have older text retention available over NNTP (i.e not
>>>> google newsgroups or web archives). I would love to slurp/archive
>>>> anything not stored on the major commercial providers.
>>>>
>>>> If so, can you give a rough disk usage and storage backend?
>>>>
>>>> I have seen people mention 50mb/day recently based on eternal-september
>>>> stats, so assuming the average daily usage is static since 1980, it
>>>> should be under 1TB.
>>>>
>>>> If not, I am planning to inject articles from archive.org and anywhere
>>>> else I can find them.
>>
>>> https://www.xach.com/naggum/articles/notes.html has a link to a
>>> comp.lang.lisp archive , http://data.xach.com.s3.amazonaws.com/cll.txt.gz .
>>> This I think is close to what you're asking but specific to one newsgroup.
>>> Earliest posts are from 1987. The moderator of comp.compilers also keeps a
>>> comprehensive archive going back to the 1990s. You can find it with a bit of
>>> googling.
>>
>>>> Are there any issues with injecting posts from 30 years ago? I don't
>>>> peer with anyone but if I can get everything imported and renumbered
>>>> correctly for my local reader to understand, I might consider peering or
>>>> making a public NNTP connection available.
>>
>>> A public NNTP connection to such an archive would be amazing.
>>
>> I've taken some time to modify some articles so that inn2 will accept them.
>> These are all from the 1980s.
>>
>> I needed to change the Date: format, so all the articles now end up with
>> my timezone (MST), but the date/times are correct, just wrong timezone.
>> Removed 'Relay-Version', 'Posting-Version' and 'Date-Received' headers.
>>
>> Now they post except for one exception. I still get '441 Can't set system Xref
>> header field'
>> on some articles, but it is a minority of them.
>>
>> I've started with the can.* hierarchy, and will continue through the rest of
>> what I have (which is a lot), but it will take me a long time to complete.
>>
>> You are free to view and/or pull the articles from news.novalink.us:119 if
>> you are interested. It will probably take me most of the summer to get it all
>> done as I don't have a ton of free time to work on it, but I want to complete
>> at some point.
>>
>> If anyone has suggestions on the above error (Xref), I'd be glad to try to get
>> those articles to post also.
>>
>> No account required to read at news.novalink.us:119

> Are you going to take a crack at the net.* stuff that's available in various
> archives? That stuff I will definitely suck off of your server, if you do. :)

That's the hierarchy I'm working on now. Only 504,091 articles to handle :)

> Keep us updated as you progress. If you come up with a scriptable or easily
> repeatable process and need another machine to help munge/inject articles let
> me know, I'd be happy to offer some assistance.

Thank you, I'll keep you in mind if needed. Right now I should be able to handle
it as this is an inn2 install specifically dedicated to this.

--
Retro Guy

Jesse Rehmer wrote:

> On Jun 2, 2023 at 3:26:20 PM CDT, "Julien ÉLIE"
> <iulius@nom-de-mon-site.com.invalid> wrote:

>> Hi Jesse,
>>
>>>> I bet you'll find out that the more recent the articles are, the more
>>>> header fields you'll need adding in the list to remove as they are not
>>>> supposed to be present in posted articles.
>>>> Like X-Trace, X-Complaints-To, NNTP-Posting-Host, Injection-Info, etc.
>>>
>>> When I use suck/pullnews, articles with these headers come in with no issue,
>>> is this due to a difference in the way the message gets to INN?
>>
>> Yes, you've configured in incoming.conf your suck/pullnews connections
>> to be handled by innd.
>> Retro Guy uses nnrpd. He may want to try to feed innd, that's a good
>> idea (hoping it won't complain of missing headers).

> I never added anything to incoming.conf, but I'm running the tools on the same
> server as INN. I never paid attention to how the tools actually 'post' the
> articles to be honest.

Just to explain how I'm doing it. I dump all the file names (with path) to a big
file (using find), then run my script to modify all the headers at one time and dump those files
to another dir. Then I run a script to upload all the files in that dir using rpost.

I do notice that the newer files (later 80s or so) do not contain the headers I need
to remove, but earlier files do. I just run them all through my script.
second script to read

--
Retro Guy

Hi Retro Guy,

> One other thing I forgot to mention is that I needed to remove lines of
> just '.', so I converted them to '..', same as a newsreader should.

Actually, you need adding an additional dot to lines *beginning* with a
dot, not only lines containing only a dot.

--
Julien ÉLIE

« La bête aux douze pieds qui marche sur la tête. » (Nougaro)

Hi Retro Guy,

> One thing that would really make a difference is not needing to create the
> groups by hand. Is it possible for inn2 to create groups on demand? That
> would make all the difference.

No, it does not create groups on-the-fly.
Note that the logtrash parameter in inn.conf can be used to have a list
of newsgroups not present on the server but which received an attempt of
post.

As you're parsing all the articles before feeding them, why not parse
the Newsgroups header field and create a list of newsgroups you then
make unique and run "ctlinnd newgroup xxx" on all of them? (INN will
then create missing newsgroups)

--
Julien ÉLIE

« Il n'y a que le premier pas qui coûte. » (Mme du Deffand)

Julien_ÉLIE wrote:

> Hi Retro Guy,

>> One thing that would really make a difference is not needing to create the
>> groups by hand. Is it possible for inn2 to create groups on demand? That
>> would make all the difference.

> No, it does not create groups on-the-fly.
> Note that the logtrash parameter in inn.conf can be used to have a list
> of newsgroups not present on the server but which received an attempt of
> post.

> As you're parsing all the articles before feeding them, why not parse
> the Newsgroups header field and create a list of newsgroups you then
> make unique and run "ctlinnd newgroup xxx" on all of them? (INN will
> then create missing newsgroups)

That's an excellent idea. My brain was getting bit weak trying to come up
with a plan. That's when you miss the obvious :)

--
Retro Guy

Retro Guy wrote:

> Julien_ÉLIE wrote:

>> Hi Retro Guy,

>>> One thing that would really make a difference is not needing to create the
>>> groups by hand. Is it possible for inn2 to create groups on demand? That
>>> would make all the difference.

>> No, it does not create groups on-the-fly.
>> Note that the logtrash parameter in inn.conf can be used to have a list
>> of newsgroups not present on the server but which received an attempt of
>> post.

>> As you're parsing all the articles before feeding them, why not parse
>> the Newsgroups header field and create a list of newsgroups you then
>> make unique and run "ctlinnd newgroup xxx" on all of them? (INN will
>> then create missing newsgroups)

> That's an excellent idea. My brain was getting bit weak trying to come up
> with a plan. That's when you miss the obvious :)

Much better! Thanks to Julien's brain (better than mine), I got the groups
created in about 15 minutes of work (including writing the script to extract
the group names and split the multiple groups in a line).

Also, thanks to wed for providing a simple bash script to create groups:

#/bin/bash
for WORD in `cat ./newsgroups.txt`
do
echo $WORD
ctlinnd newgroup $WORD
done
echo "Done."

from: https://news.novabbs.org/rocksolid/article-flat.php?id=162&group=rocksolid.shared.linux#162

--
Retro Guy

Julien ÉLIE wrote:

> As you're parsing all the articles before feeding them, why not parse
> the Newsgroups header field and create a list of newsgroups you then
> make unique and run "ctlinnd newgroup xxx" on all of them? (INN will
> then create missing newsgroups)

.... including all typos. :)

Thomas Hochstein wrote:

> Julien ÉLIE wrote:

>> As you're parsing all the articles before feeding them, why not parse
>> the Newsgroups header field and create a list of newsgroups you then
>> make unique and run "ctlinnd newgroup xxx" on all of them? (INN will
>> then create missing newsgroups)

> .... including all typos. :)

Very true! I'll try to clean those up later.

Currently uploading net.* and it's been running now for about 24 hours.
Let's see if inn2 recovers after this is done, it's throttling right now,
but accepting the posts.

--
Retro Guy

Retro Guy wrote:

> Thomas Hochstein wrote:

>> Julien ÉLIE wrote:

>>> As you're parsing all the articles before feeding them, why not parse
>>> the Newsgroups header field and create a list of newsgroups you then
>>> make unique and run "ctlinnd newgroup xxx" on all of them? (INN will
>>> then create missing newsgroups)

>> .... including all typos. :)

> Very true! I'll try to clean those up later.

> Currently uploading net.* and it's been running now for about 24 hours.
> Let's see if inn2 recovers after this is done, it's throttling right now,
> but accepting the posts.

Finally have net.* on the server. I needed to rebuild history when complete
due probably to all the messing around I was doing with the server.

I'll clean up the typo group names at some point, but for now I plan to
put can.* back on, then move to some more hierarchies.

The fact it's working is nice to see.

--
Retro Guy

Retro Guy wrote:

> Retro Guy wrote:

>> Thomas Hochstein wrote:

>>> Julien ÉLIE wrote:

>>>> As you're parsing all the articles before feeding them, why not parse
>>>> the Newsgroups header field and create a list of newsgroups you then
>>>> make unique and run "ctlinnd newgroup xxx" on all of them? (INN will
>>>> then create missing newsgroups)

>>> .... including all typos. :)

>> Very true! I'll try to clean those up later.

>> Currently uploading net.* and it's been running now for about 24 hours.
>> Let's see if inn2 recovers after this is done, it's throttling right now,
>> but accepting the posts.

> Finally have net.* on the server. I needed to rebuild history when complete
> due probably to all the messing around I was doing with the server.

> I'll clean up the typo group names at some point, but for now I plan to
> put can.* back on, then move to some more hierarchies.

> The fact it's working is nice to see.

Or is it? I'm having some trouble where after inn2 runs for a few hours I
get the error 'File exists writing SMstore file -- throttling'

I then shut it down, rebuild the history 'makehistory -b -f history.n -O -l 30000 -I',
copy the .h files over as directed in the man page, then start inn2 up again. (I note
some duplicate Message-ID messages when it runs)

After a few hours the error returns. I'm not posting any messages at all, just
letting it run.

How can I fix this?

--
Retro Guy