Hello World!

i'm writing a usenet server in go from scratch and v0.1-p1 is online =)

got it more or less stable running many weeks, testing and importing.
performance is brutally good and very low to no cpu usage.

Host: 213.227.135.51
Port: 7119
User: reader2023
Pass: nopassword

also available in yggdrasil: [201:feda:24c0:4544:c808:42c5:f053:9fac]

the server holds loads of spam in 1.1b messages, 400k groups
about 3.3TB data at the moment
uncompressed, non-binary (=yenc filter)
and is still importing while writing.

0.44T heads @ compressratio 3.2x
1.13T bodys @ compressratio 1.7x

Supported Commands:
AUTHINFO
LIST
GROUP
ARTICLE, HEAD, BODY, STAT
IHAVE
MODE
XOVER
QUIT

Features:
basecode ~4k lines of chaos
go's concurrency model
dynamic creation of groups
user authentication
BW shaper
easy config in .json
peers via .json
users via .json

Todo:
+ peering
+ expiration
+ other commands
+ history + dbz or *sql?

the message numbering is not really ordered by date due to importing
from different sources.
groups may start with very old articles but may be mixed up later.
a re-ordering is planned later when fully backfilled.

you can walk around the server and read/suck/hack/crack/crash!

i'd like to see someone crashing the process!

but please don't use ddos botnets... can't fight physics... thank you!

i'm sure there are still many bugs in this early stage of development.

i appreciate your feedback and bugs if you encounter any!

the "active" file is static, numbers may be far behind "group" values.
because test server runs on a snapshot, but can read latest overview and
storage if joining a group with "group" command.

speed is limited and reading articles via msgnum (in groups) is quite
slow because it scans the overview to find the msgid&hash needed to
fetch data from storage.

interested in how the configs look like?

{
"Settings": {
"Hostname": "s01.host.name.local",
"Reload_CFG": 60,
"Daemon_Host": "[::1]",
"Daemon_TCP": "tcp",
"PortsTCP": "1119,1433,8080",
"PortsSSL": "1563,8433,8443",
"Logs_File": "/path/filename.log",
"SSL_CRT": "!/var/lib/news/fullchain.pem",
"SSL_KEY": "!/var/lib/news/privkey.pem",
"Json_AuthFile": "users.json",
"MinPWLen": 8,
"LIST_NeedsAuth": true,
"ActiveDir": "active",
"Overview": true,
"OverviewDir": "ov",
"StorageDir": "storage",
"StorageXrefLinker": false
},
"Peers": [
{
"Enabled": true,
"Hostname": "localhost1",
"Addr4": "127.0.0.17",
"Addr6": "::17",
"Speedlimit": 1024
},
{
"Enabled": true,
"Hostname": "news.local",
"Port": 1119,
"Addr4": "192.168.1.119",
"L_Auth_User": "user1972",
"L_Auth_Pass": "notapass",
"R_SSL": false,
"ReadOnlyAccess": true
},
{
"!Enabled": true,
"Hostname": "news.test.net.local",
"Port": 433,
"Addr4": "192.168.119.1",
"Addr6": "fd01:abcd:1234:ffff::abcd",
"Cidr4": "10.0.1.0/24",
"Cidr6": "fd01:abcd:1234:ffff::/64",
"R_Auth_User": "user42",
"R_Auth_Pass": "nopass",
"R_SSL": true,
"R_SSL_Insecure": true
}
]
}

EOF config.json

users.json:
{ "Users": [
{
"Username": "testuser1",
"Password": "testpass1",
"Expire": 1790539435,
"MaxConns": 1,
"Speedlimit": 8
},
{
"Username": "testuser2",
"Password": "testpass2",
"Expire": 1790539435,
"MaxConns": 2
},
{
"Username": "testuser3",
"Password": "testpass3",
"Expire": 1690539435,
"MaxConns": 3
},
{
"Username": "testuser42",
"Password": "testpass42",
"Expire": -1,
"MaxConns": 42
}
]
}

EOF users.json

will go opensource when code is more beautified =)

over and out!
Billy G.

Am 04.09.2023 um 03:20:39 Uhr schrieb Billy G. (go-while):

> i'm writing a usenet server in go from scratch and v0.1-p1 is online

Great!

> speed is limited and reading articles via msgnum (in groups) is quite
> slow because it scans the overview to find the msgid&hash needed to
> fetch data from storage.

Is that to be changed?
I tried it, it took ~10 minutes to receive 300 articles.

On 04.09.23 12:10, Marco Moock wrote:
>> speed is limited and reading articles via msgnum (in groups) is quite
>> slow because it scans the overview to find the msgid&hash needed to
>> fetch data from storage.
> Is that to be changed?
> I tried it, it took ~10 minutes to receive 300 articles.
>

cool! at least something works
speed was limited to 8 Kbits :D is 1 Mbit now.
latency is high because storage is still at home
my line is only 100M both ways.

On 2023-09-04, Billy G. (go-while) <no-reply@no.spam> wrote:
> Hello World!
>
> i'm writing a usenet server in go from scratch and v0.1-p1 is online =)
>
> got it more or less stable running many weeks, testing and importing.
> performance is brutally good and very low to no cpu usage.
>

Very cool, please let us know when you upload it to a git repo or
something so we can check and try it out and maybe submit code.

ReK2
Happy Hacking

--
- {gemini,https}://{,rek2.}hispagatos.org - mastodon: @rek2@hispagatos.space
- [https|gemini]://2600.Madrid - https://hispagatos.space/@rek2
- https://keyoxide.org/A31C7CE19D9C58084EA42BA26C0B0D11E9303EC5

On 15.09.23 01:51, rek2 hispagatos wrote:
> On 2023-09-04, Billy G. (go-while) <no-reply@no.spam> wrote:
>> Hello World!
>>
>> i'm writing a usenet server in go from scratch and v0.1-p1 is online =)
>>
>
> Very cool, please let us know when you upload it to a git repo or
> something so we can check and try it out and maybe submit code.
>
> ReK2
> Happy Hacking
>
Don't know if this will happen this year.
want it sable and free of bugs as much as possible.
peering and expiration is missing too.
it can receive, store and deliver to clients but can not talk to other
servers and without expiration who wants to use it at this stage?

If anyone has time to test reading functions feel free to hack.

Host: 213.227.135.51
Port: 7119
User: reader2023
Pass: nopassword

I've hidden groups with only 1 article
results in 289k groups now.
Thunderbird has issues receiving the LIST, sometimes.
Takes forever... maybe size problem?
LIST is 14MB at the moment.
Was >20MB unfiltered (442k groups).

Added an Auto Index to Overview.
Fetching articles via msg-num in groups should be much faster now.
Fetching via message-id is as fast as possible.

Command LISTGROUP should work now.

Almost all archive.org backups imported.
Some of the mbox files are buggy and need manual interaction.
archive.org backups date some groups to 2020, others 2014-2016 and
older, 80s-90s.
some groups are sync to 2023 but none is up to date.

Anyone with a long retention is willing to share the spool with me?
You can suck what you need from Host 213.227.135.51.
Give me (reader) access to same ip and post your hostname. thanks!
The ip won't send any articles to you.

I can scan whole spool with 1 connection and LISTGROUP
check vs. my spool and fetch missing articles.
But before, I'll fill the gap with a (or N) paid provider, before
scanning peoples spools.

Wrote some code to re-order the overview by dates and first test looks
good. This will be done after import, maybe soon.

over and out

On 2023-09-18, Billy G. (go-while) <no-reply@no.spam> wrote:
> I've hidden groups with only 1 article
> results in 289k groups now.
> Thunderbird has issues receiving the LIST, sometimes.
> Takes forever... maybe size problem?
> LIST is 14MB at the moment.
> Was >20MB unfiltered (442k groups).

Have you created the newsgroups list by just assuming that everything
mentioned in any article's Newsgroups: line is a valid group, or
something? 442k is an order of magnitude higher than the actual number
of newsgroups.

On 18.09.23 15:48, Jon Ribbens wrote:
> Have you created the newsgroups list by just assuming that everything
> mentioned in any article's Newsgroups: line is a valid group, or
> something? 442k is an order of magnitude higher than the actual number
> of newsgroups.

yes. catch all. we can filter later.

On 2023-09-18, Billy G. (go-while) <no-reply@no.spam> wrote:
> On 18.09.23 15:48, Jon Ribbens wrote:
>> Have you created the newsgroups list by just assuming that everything
>> mentioned in any article's Newsgroups: line is a valid group, or
>> something? 442k is an order of magnitude higher than the actual number
>> of newsgroups.
>
> yes. catch all. we can filter later.

That's not a reasonable way of managing the newsgroups list.

On 18.09.23 16:56, Jon Ribbens wrote:
>
> That's not a reasonable way of managing the newsgroups list.

what is reasonable if you just want to import all backups from earth?

On 2023-09-18, Billy G. (go-while) <no-reply@no.spam> wrote:
> On 18.09.23 16:56, Jon Ribbens wrote:
>> That's not a reasonable way of managing the newsgroups list.
>
> what is reasonable if you just want to import all backups from earth?

To get an active file from an existing well-run full-feed server
and use that I suppose, and have some code to validate and process
rmgroups, newgroups, and checkgroups. Making it so every typo ever
made creates a new newsgroup just completely destroys the concept
of the well-organised and managed hierarchical newsgroup structure.

Jon Ribbens wrote:

> On 2023-09-18, Billy G. (go-while) <no-reply@no.spam> wrote:
>> On 18.09.23 16:56, Jon Ribbens wrote:
>>> That's not a reasonable way of managing the newsgroups list.
>>
>> what is reasonable if you just want to import all backups from earth?

> To get an active file from an existing well-run full-feed server
> and use that I suppose, and have some code to validate and process
> rmgroups, newgroups, and checkgroups. Making it so every typo ever
> made creates a new newsgroup just completely destroys the concept
> of the well-organised and managed hierarchical newsgroup structure.

That's a great way to do it, but it won't work if trying to import old articles.

There were a lot of newsgroup names that no longer exist that are used in the old articles, and won't be in the active file of most modern servers.

--
Retro Guy

On 2023-09-18, Retro Guy <retro.guy@rocksolidbbs.com> wrote:
> Jon Ribbens wrote:
>> On 2023-09-18, Billy G. (go-while) <no-reply@no.spam> wrote:
>>> On 18.09.23 16:56, Jon Ribbens wrote:
>>>> That's not a reasonable way of managing the newsgroups list.
>>>
>>> what is reasonable if you just want to import all backups from earth?
>
>> To get an active file from an existing well-run full-feed server
>> and use that I suppose, and have some code to validate and process
>> rmgroups, newgroups, and checkgroups. Making it so every typo ever
>> made creates a new newsgroup just completely destroys the concept
>> of the well-organised and managed hierarchical newsgroup structure.
>
> That's a great way to do it, but it won't work if trying to import old
> articles.
>
> There were a lot of newsgroup names that no longer exist that are used
> in the old articles, and won't be in the active file of most modern
> servers.

If you have an archive of old messages you're importing, you might be
lucky enough for it to include the old control messages, enabling you
to construct a timeline of newsgroup creation and deletion...

If it doesn't then I suppose one could have a rule such as "if over
100 posts were made to group X (perhaps within a certain period of
time) then it presumably at least used to exist". But if Billy has
ended up with over 400k newsgroups then I don't think he's being
that selective.

What's reasonable may differ of course depending on whether the
purpose is "a read-only historical archive of past newsgroups",
or "a live news server for reading and posting new articles".
If it's a hybrid of the two then perhaps a hybrid approach is
required.

On 18.09.23 18:46, Jon Ribbens wrote:

> What's reasonable may differ of course depending on whether the
> purpose is "a read-only historical archive of past newsgroups",
> or "a live news server for reading and posting new articles".
> If it's a hybrid of the two then perhaps a hybrid approach is
> required.

maybe this is a good way to test my new code, while i'm writing a new
nntp-server software in GO from scratch.
that's the topic and it ate it all.

choose the first and the latter once import is finished.
as a peer, you tell/decide which groups you want in your configuration.
but if one would post to any group here, better check before if it's
still accepted worldwide.
whats the problem or difference for you? what do i miss?

On 18.09.23 18:46, Jon Ribbens wrote:
> If you have an archive of old messages you're importing, you might be
> lucky enough for it to include the old control messages, enabling you
> to construct a timeline of newsgroup creation and deletion...

what's the point in importing old messages only to delete their groups
later? this doesn't make sense.

On 2023-09-18, Billy G. (go-while) <no-reply@no.spam> wrote:
> On 18.09.23 18:46, Jon Ribbens wrote:
>> If you have an archive of old messages you're importing, you might be
>> lucky enough for it to include the old control messages, enabling you
>> to construct a timeline of newsgroup creation and deletion...
>
> what's the point in importing old messages only to delete their groups
> later? this doesn't make sense.

It could be homework for a job application at Google, in the Groups team?

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca
NOTE: If you use Google Groups, I don't see you, unless you're whitelisted.

On 2023-09-18, Billy G. (go-while) <no-reply@no.spam> wrote:
> On 18.09.23 18:46, Jon Ribbens wrote:
>> If you have an archive of old messages you're importing, you might be
>> lucky enough for it to include the old control messages, enabling you
>> to construct a timeline of newsgroup creation and deletion...
>
> what's the point in importing old messages only to delete their groups
> later? this doesn't make sense.

That's not at all what I suggested.