A while back you said downloading Usenet post info from glorb.com was
really fast.

glorb.com doesn't seem to be around any more.

How and where are you downloading Usenet messages?

I use blocknews and python nntplib and SuckMT (Windows), but they're not
quick enough any more. blocknews has suddenly gone downhill.

On 4/8/2024 11:06 PM, Relf wrote:
> DFS asked me:
>> A while back you said downloading Usenet post info from glorb.com was really fast.
>> glorb.com doesn't seem to be around any more.
>
> Yes, Glorb threw in the towel many years ago (2018?).
>
>> How and where are you downloading Usenet messages?
>> I use blocknews and python nntplib and SuckMT (Windows), but they're not
>> quick enough any more. blocknews has suddenly gone downhill.
>
> I switch servers on the fly, willy nilly.
> When BlockNews fails, I use UsenetNews, & vice versa.
> Oddly, they don't both fail at the same time.
>
> Instead of switching servers, you could just wait it out.
>
> MessageID (instead of Article Number) "bookmarking" is tricky.
> I list MessageIDs (100 at a time), starting from the highest Article Number,
> until I find one of the 10 MessageID "bookmarks" I've saved.

Can you give me the code you run to retrieve articles? How were you
getting 1000 articles in 4 seconds from glorb?

The python code I've relied on for years has become very slow and
unstable when running it against blocknews. Connections are dropped all
the time.

SuckMT is OK fast, but it drops connections too.

I want to stay with blocknews because it has articles going back to
2003, and they might recover from their current woes.

On 4/9/2024 12:24 AM, Relf wrote:
> You (DFS) replied ( to me ):
>>> MessageID (instead of Article Number) "bookmarking" is tricky.
>>> I list MessageIDs (100 at a time), starting from the highest Article Number,
>>> until I find one of the 10 Message-ID "bookmarks" I've saved.
>>
>> Can you give me the code you run to retrieve articles?
>> How were you getting 1000 articles in 4 seconds from glorb?
>
> Glorb was lightning fast.
> When possible, I sent it 100 commands at a time, instead of individually.
>
> From "z1.CPP" in "Jeff-Relf.Me/z1.ZIP":
>
> typedef __int64 i64 ; i64 _LowArtNum, _HighArtNum, HighArtNum ;
>
> const int max_BookMarks = 10, MIDsPerPage = 100, Max_szMID = 200, Max_Sz_BookMark = max_BookMarks * Max_szMID ;
>
> OpenGroup₍₎( LnP NG ) { // 211 303,341,573 1,844,605,951 2,147,947,523 Comp.OS.Linux.Advocacy
> if ( !Go( L"Group %s", LowerCaseNG( NG ) ) ) { Sh( L"No Such NewsGroup, “ %c%s%c ” ?", ChNew, NG, ChText ); return 0 ; }
> strCpy( _T, *Ln.PP ), P = _T, gTok, gTok, gTok, LowArtNum = AtoI( Tok ) - 1, gTok, HighArtNum = AtoI( Tok );
> Sh( L"%s Articles; From: %s To: %s .", Commas( HighArtNum - LowArtNum ), Commas( LowArtNum + 1 ), Commas( HighArtNum ) ); return HighArtNum ; }
>
> _LowArtNum = HighArtNum - MIDsPerPage - 1 ;
> Go( L"xHdr Message-ID %I64d-%I64d", _LowArtNum, _HighArtNum );
>
>> The python code I've relied on for years has become very slow and
>> unstable when running it against blocknews. Connections are dropped all
>> the time. SuckMT is OK fast, but it drops connections too.
>>
>> I want to stay with blocknews because it has articles going back to
>> 2003, and they might recover from their current woes.
>
> Try BlockNews and/or UsenetNews again, an hour later.

It's been slow for weeks... well, my previously quick python code, anyway.

I used to do head and body commands (via Python nntplib), but they don't
work very well lately. The 'article' command is pretty quick, though.

This code prints the 211 response for the group, then sends the
'article' command one at a time, parses and displays the response. It's
pretty fast - up to ~9 articles retrieved per second late in the evening.

==============================================================================================
import nntplib, time
from datetime import datetime

startStr = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
startTime = time.time()

srvName = 'usnews.blocknews.net'
grpName = 'comp.os.linux.advocacy'
news = nntplib.NNTP(srvName,119,'dfsblocknews','hogsblocknews')
r,a,b,e,g = news.group(grpName)
print(r)

cnt, found, notfound = 0,0,0
for artID in range(904779, 904785):
print("------------------------------------------------------------------------------")
print(artID)
msgID, msg = 0,''
try:
response, info = news.article(artID)
artID = info[0]
msgID = info[1]
msg = info[2]

for line in msg:
line = line.decode('latin-1')
if line != "Path: not-for-mail":
print(line)
found += 1
except (nntplib.NNTPTemporaryError, nntplib.NNTPPermanentError) as err:
print('Error:',err)
notfound += 1
cnt += 1
print("------------------------------------------------------------------------------")
print("done")

endStr = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
totTime = time.time() - startTime
print("\nStart: " + startStr)
print("End : " + endStr)
print("%.1fs to request %d messages (%.f per sec)" % (totTime, cnt, cnt
/ totTime))
print("%d messages downloaded" % (found))
print("%d messages not found" % (notfound))
==============================================================================================

Save it as a .py file and run it:
$ python prog.py

a minute ago, from blocknews:

211 1795999 899518 2695516 comp.os.linux.advocacy

But blocknews recently screwed up their article numbering, and there are
a lot of gaps in the sequence, so there aren't 1795999 articles
available on their server.

On 4/11/2024 11:07 AM, Relf wrote:
> You (DFS) replied ( to me ):
>>> I list MessageIDs (100 at a time), starting from the highest Article Number,
>>> until I find one of the 10 MessageID "bookmarks" I've saved.
>>
>> I used to do head and body commands (via Python nntplib), but they don't
>> work very well lately. The 'article' command is pretty quick, though.
>> info = news.article(artID) artID = info[0] msgID = info[1] msg = info[2]
>
> Yes, instead of "head" and "body", I use the "article" command.

The SuckMT program is a multithreaded Usenet downloader for Windows, but
it continually disconnects after around 4K posts are downloaded.

>> a minute ago, from blocknews:
>> 211 1795999 899518 2695516 comp.os.linux.advocacy
>>
>> But blocknews recently screwed up their article numbering,
>> and there are a lot of gaps in the sequence,
>> so there aren't 1795999 articles available on their server.
>
> At times, blocknews is missing some articles and such;
> some seconds later, they're back.

Did that just start? For years BN has been rock solid and fast, but in
the past few months it's not working too well (for me).

> My code actually checks that blocknews has the 10 MessageIDs
> that makes up my "bookmark", in the order expected;
> if it doesn't, it pays to try again later.

Which server(s) do you use to build a bookmark of 10 Msg-IDs in a
specific order? What's the order?

Note also that eternal-september blocks a variety of nyms. Using ES, I
never see anything from alt.checkmate in cola (which is good).

blocknews allows crossposting from alt.checkmate, so their data often
won't match with eternal-september.

On 4/11/2024 6:51 PM, Relf wrote:
> You (DFS) replied ( to me ):

>> Did that just start? For years BN has been rock solid and fast,
>> but in the past few months it's not working too well (for me).
>
> Yes, it's new. It took 10+ minutes for my
> "Jeff-Relf.Me@Apr.11--8.27am.Seattle.2024" post to show up.
>
> 3:16pm, Seattle, BlockNews reported:
> 29 Requested posts, got all 29 in 6 Secs.
>
> CntNew / 30_DayCnt
> 7 / 513 DFS

Is that 513 posts by me in the last 30 days? If so, it should be closer
to 170.

>> Which server(s) do you use to build a bookmark of 10 Msg-IDs in a
>> specific order? What's the order?
>
> Any server, every server; on the fly, willy-nilly.

How many and which news servers do you subscribe to? Why more than one?

I have a block acct at blocknews, and the free eternal-september.

> Censorship is the price you pay for using EternalSeptember, nigga.

Crossposting is the price you pay for using Usenet. I'm glad ES drops
some (all?) of the alt.checkmate crossposts into cola.