Web forums keep annoying me more and more with bloated interfaces,
so I'm using them less and less, yet there's less and less to read
on Usenet too. So lately I've been considering, not entirely
seriously, writing a program to scape specific web forums and
generate a news spool with the content of the forum's latest posts
for me to read (either locally, or perhaps remotely via NNTP).

Has anyone done this before? I know there are various web forum
platforms that support NNTP server-side, but I'm talking about web
forums hosted by other people who I have no association or
influence with. Has anyone done something that's purely a
client-side implementation?

Simple Machines Forum and Discourse are prime targets for me, maybe
phpBB too. Most don't have RSS enabled, or the feed only shows the
start of new posts. The ideal would be a system supporting scrapers
for multiple forum platforms which can be easily extended.

Support for posting would be nice, but read-only access in a news
reader (Tin) would be better than nothing.

--
__ __
#_ < |\| |< _#

On 2023-10-02, Computer Nerd Kev <not@telling.you.invalid> wrote:
> Web forums keep annoying me more and more with bloated interfaces,
> so I'm using them less and less, yet there's less and less to read
> on Usenet too. So lately I've been considering, not entirely
> seriously, writing a program to scape specific web forums and
> generate a news spool with the content of the forum's latest posts
> for me to read (either locally, or perhaps remotely via NNTP).

this is a good idea, but consider doing it from ethical forums
so let me explain, instead of reddit, use lemmy that is more interesting
and is the open/libre reddit and part of the fediverse.
since is a forum/threads style will be much easier I think...

>
> Has anyone done this before? I know there are various web forum
> platforms that support NNTP server-side, but I'm talking about web
> forums hosted by other people who I have no association or
> influence with. Has anyone done something that's purely a
> client-side implementation?

I just finished beta of a bot that will check
my mastodon feed and if I tag #2usenet it will post also to
the newsgroups I told him in the config.ini file .

PD: Writing as I speak a simple usenet client
for Android in GO/GOLANG

if we had a go newsgroup I will had posted my bot there and such :/

Happy Hacking
ReK2

--
- {gemini,https}://{,rek2.}hispagatos.org - mastodon: @rek2@hispagatos.space
- [https|gemini]://2600.Madrid - https://hispagatos.space/@rek2
- https://keyoxide.org/A31C7CE19D9C58084EA42BA26C0B0D11E9303EC5

rek2 hispagatos <rek2@hispagatos.org.invalid> wrote:
> On 2023-10-02, Computer Nerd Kev <not@telling.you.invalid> wrote:
>> Web forums keep annoying me more and more with bloated interfaces,
>> so I'm using them less and less, yet there's less and less to read
>> on Usenet too. So lately I've been considering, not entirely
>> seriously, writing a program to scape specific web forums and
>> generate a news spool with the content of the forum's latest posts
>> for me to read (either locally, or perhaps remotely via NNTP).
>
> this is a good idea, but consider doing it from ethical forums
> so let me explain, instead of reddit, use lemmy that is more interesting
> and is the open/libre reddit and part of the fediverse.
> since is a forum/threads style will be much easier I think...

I don't have much interest in the Reddit sorts of things, although
I've noticed projects along similar lines for them. It's dedicated
web forums for specific software projects or enthusiest groups that
I'd like to access and combine into one interface through my news
reader. They've got the content that I'd like to read.

Note that whereas Reddit, and presumably alternatives, apparantly
has/had a public API, the forum implementations that I'm looking at
will probably need to have all content scraped from the HTML. That
makes the process more difficult and fragile. This is much of the
reason why I'm not really committed to doing it all myself, but if
there's an existing project to build from then that might make the
amount of effort required more attractive.

--
__ __
#_ < |\| |< _#

Computer Nerd Kev <not@telling.you.invalid> wrote:
> Note that whereas Reddit, and presumably alternatives, apparantly
> has/had a public API, the forum implementations that I'm looking at
> will probably need to have all content scraped from the HTML. That
> makes the process more difficult and fragile.

Actually Discourse does have an API, so it might be a good one to
start out with:
https://docs.discourse.org/

Then again it doesn't seem easy (possible?) to get a list of recent
posts unless you have "an API key from the admin panel", which I
won't have.

--
__ __
#_ < |\| |< _#

On Tue, 03 Oct 2023 01:03:48 Computer Nerd Kev wrote:
> Web forums keep annoying me more and more with bloated interfaces,
> so I'm using them less and less, yet there's less and less to read
> on Usenet too. So lately I've been considering, not entirely
> seriously, writing a program to scape specific web forums and
> generate a news spool with the content of the forum's latest posts
> for me to read (either locally, or perhaps remotely via NNTP).
>
> Has anyone done this before? I know there are various web forum
> platforms that support NNTP server-side, but I'm talking about web
> forums hosted by other people who I have no association or
> influence with. Has anyone done something that's purely a
> client-side implementation?
>[...]
> Support for posting would be nice, but read-only access in a news
> reader (Tin) would be better than nothing.

Would news.gmane.io be suitable for you? I use it mainly for
mailing list (groups starting with gmane.*), but it also carries
articles from web-based blogs (groups starting with gwene.*).
The latter often only contain only teasers and links to the full
article, but I find that reading with Tin and invoking
w3m on the posting is sufficiently non-bloated.

--
M.O.

Marc Olschok <nobody@nowhere.invalid> wrote:
> Would news.gmane.io be suitable for you? I use it mainly for
> mailing list (groups starting with gmane.*), but it also carries
> articles from web-based blogs (groups starting with gwene.*).

Gmane is great for mailing lists and RSS feeds, but subjects that
have gone quiet or never appeared on Usenet often don't have active
mailing lists either. I went for a hunt through the groups list on
Gmane recently with this in mind, but only came back with one more
mailing list, also fairly inactive, and only vaguely interesting to
me.

If forums had complete RSS feeds, that might be good enough to use
in Gmane, but the ones I'm interested in either don't offer RSS at
all or only only show the beginning of posts, which makes it
useless.

I've now discovered forum-dl, which gets close to what I'm looking
for:
https://github.com/mikwielgus/forum-dl

It does seem to be aimed at one-time full downloads of forum
contents rather than regularly checking for new posts, and I don't
like Python. But at least it's an example of how to extract the
important data fields from all the forum platforms that I'm
interested in, and it supports output in MH format which is much
like the news spool format.

--
__ __
#_ < |\| |< _#