https://github.com/go-while/nntp-mongodb-storage

nntp-mongodb-storage is a Go module for storing Usenet articles in
MongoDB, featuring asynchronous insertion and deletion using worker
goroutines, as well as support for gzip and zlib compression. Developed
with the assistance of ChatGPT, an AI language model by OpenAI.

nntp-mongodb-storage is a MongoDB Article Storage

This repository contains a Go module for storing Usenet articles in MongoDB.

It provides functionality for inserting and deleting articles
asynchronously using worker goroutines, as well as support for
compression using gzip and zlib.
Getting Started

To use this module, you will need Go installed on your system. You can
install the module using go get:

go get github.com/go-while/nntp-mongodb-storage

Usage

See https://github.com/go-while/nntp-mongodb_test for integration.

go get github.com/go-while/nntp-mongodb_test

Load_MongoDB Function

The Load_MongoDB function is a part of the mongostorage package in the
provided code. It serves as a convenient initializer to set up the
MongoDB configuration and establish connections to the MongoDB database
and collections for article storage and deletion.

Function Signature

func Load_MongoDB(mongoURI, dbName, collectionName string, timeout
time.Duration, delWorker *MongoDeleteWorker, delQueue chan string,
insWorker *MongoInsertWorker, insQueue chan MongoArticle,
testAfterInsert bool)

Load_MongoDB Parameters

mongoURI (string): A string representing the MongoDB connection URI. It
should include the necessary authentication details and the address of
the MongoDB server.
mongoDatabaseName (string): A string representing the name of the
MongoDB database where the articles will be stored and retrieved from.
mongoCollection (string): A string representing the name of the
collection within the database where the articles will be stored.
mongoTimeout (int64): A duration in milliseconds that defines the
timeout for a connection attempt to the MongoDB server. If the
connection is not successful within this duration, it will time out.
delWorker (int): The number of worker goroutines to handle article
deletions. It determines the level of concurrency for deletion operations.
delQueue (int): The size of the delete queue. It specifies how many
delete requests can be buffered before the send operation blocks. If
delQueue is 0 or negative, a default value will be used.
insWorker (int): The number of worker goroutines to handle article
insertions. It determines the level of concurrency for insertion operations.
insQueue (int): The size of the insert queue. It specifies how many
article insertion requests can be buffered before the send operation
blocks. If insQueue is 0 or negative, a default value will be used.
getWorker (int): The number of worker goroutines to handle article
reads. It determines the level of concurrency for read operations.
getQueue (int): The size of the read queue. It specifies how many read
requests can be buffered before the send operation blocks. If getQueue
is 0 or negative, a default value will be used.
testAfterInsert (bool): A flag indicating whether to perform a test
after an article insertion. The specific test details are not provided
in the function, and the flag can be used for application-specific
testing purposes.

How Load_MongoDB works

MongoDB Connection: The Load_MongoDB function establishes a connection
to the MongoDB server specified in the mongoURI.

Database and Collections: It uses the dbName to select or create the
target database for article storage and retrieval. Within the selected
database, the function creates or retrieves the MongoDB collections for
storing articles.

Initialization: The function initializes the provided delWorker,
insWorker, and getWorker instances to handle article deletions,
insertions, and read operations, respectively.

Start Workers: The function starts the provided delWorker, insWorker,
and getWorker goroutines to process article deletions, insertions, and
reading operations concurrently.

Queue Setup: The function sets up the delQueue, insQueue, and getQueue,
which are channels used for communication between the main program and
the worker goroutines.

Article Deletions: Articles to be deleted will be enqueued in the
delQueue with their corresponding hashes, which will be processed by the
delete worker goroutines.

Article Insertions: Articles to be inserted will be enqueued in the
insQueue as MongoArticle instances, which will be processed by the
insert worker goroutines. The testAfterInsert flag controls whether a
test is performed after each article insertion. Unfortunately, the
specific details of this test are not provided in the code.

Read (Get) Operations: Read requests for articles will be enqueued in
the getQueue, which will be processed by the reader worker goroutines.
This allows concurrent reading of articles from the MongoDB collections.

You can adjust the number of worker goroutines and queue sizes based on
your application's requirements and the available resources.

Usage Load_MongoDB

The Load_MongoDB function is typically called at the start of the
program to set up the MongoDB configuration and prepare the worker
goroutines for article storage and deletion.

After calling this function, the main program can use the delQueue and
insQueue channels to enqueue articles for deletion or insertion,
respectively.
DefaultDelQueue

DefaultDelQueue is a channel used to handle the deletion of articles
from MongoDB.

It is typically used to pass MessageIDHashes of articles that need to be
deleted.

When an article needs to be deleted from MongoDB, its corresponding
MessageIDHash is sent into the DefaultDelQueue.

The deletion process is usually handled by a separate goroutine (worker)
that listens to the DefaultDelQueue and performs the actual deletion
operation in the background.

Developers can customize this queue if they need to implement their own
deletion logic or prioritize deletion operations differently.
DefaultDelWorker

DefaultDelWorker is a goroutine (worker) responsible for processing
deletion operations from the DefaultDelQueue.

It waits for MessageIDHashes to be sent into the DefaultDelQueue and
performs the deletion operation for each received MessageIDHash.

The number of DefaultDelWorker goroutines can be adjusted based on the
desired concurrency and performance requirements.

Having multiple workers handling deletion can improve the deletion
throughput, especially when dealing with a large number of articles.
DefaultInsQueue

DefaultInsQueue is a channel used to handle the insertion of articles
into MongoDB.

It is typically used to pass MongoArticle objects that represent the
articles to be inserted.

When a new article needs to be inserted into MongoDB, the MongoArticle
object is sent into the DefaultInsQueue.

The insertion process is usually handled by a separate goroutine
(worker) that listens to the DefaultInsQueue and performs the actual
insertion operation in the background.

Developers can customize this queue if they need to implement their own
insertion logic or prioritize insertion operations differently.
DefaultInsWorker

DefaultInsWorker is a goroutine (worker) responsible for processing
insertion operations from the DefaultInsQueue.

It waits for MongoArticle objects to be sent into the DefaultInsQueue
and performs the insertion operation for each received MongoArticle.

The number of DefaultInsWorker goroutines can be adjusted based on the
desired concurrency and performance requirements.

Having multiple workers handling insertion can improve the insertion
throughput, especially when dealing with a large number of articles.

Overall, the DefaultDelQueue, DefaultDelWorker, DefaultInsQueue, and
DefaultInsWorker are part of the infrastructure provided by the
mongostorage package to efficiently manage the insertion and deletion of
articles in a concurrent and asynchronous manner. They help in
maximizing performance and scalability when dealing with a large number
of articles in a MongoDB database. Developers can fine-tune the number
of workers and customize the queues if needed to meet their specific use
cases and requirements.
MongoArticle Struct

The MongoArticle struct is part of the mongostorage package and
represents an article to be stored in a MongoDB database. This struct
holds various fields that represent different attributes of the article,
such as its content, metadata, and compression-related information.
Fields

MessageIDHash: This field stores the SHA256 hash of the MessageID
field. The MessageID is a unique identifier for each article, and by
storing its hash, the storage process can use it as an index to
efficiently search for and identify articles in the MongoDB database.

MessageID: This field contains the unique identifier (Message-ID)
of the article. It is typically in the format of an email address
enclosed in angle brackets (e.g., <example@example.com>). The MessageID
is used to uniquely identify each article and prevent duplicates.

Head: This field holds the header of the article, which contains
metadata and information about the article, such as the author, subject,
date, and newsgroups. The header is usually a series of lines, and in
this case, it is stored as a byte slice.

https://github.com/go-while/nntp-mongodb-storage

nntp-mongodb-storage

MongoDB Storage Package for Usenet Articles

The mongostorage package offers a set of convenient functions for
interacting with a MongoDB database and managing Usenet articles.

It provides efficient and reliable methods for inserting, deleting,
and retrieving Usenet articles in a MongoDB collection.

With the mongostorage package, you can easily store, manage, and
query Usenet articles in a MongoDB database, making it a valuable tool
for any application that deals with Usenet data.

Key Functions: mongostorage.Func(...)

InsertOneArticle: Inserts a single article into the MongoDB collection.

InsertManyArticles: Performs a bulk insert of multiple articles
into the MongoDB collection.

IsDup: Checks if an error is a duplicate key error in MongoDB.

DeleteManyArticles: Deletes multiple articles from the MongoDB
collection based on a list of MessageIDHashes.

DeleteArticlesByMessageIDHash: Deletes an article from the MongoDB
collection by its MessageIDHash.

RetrieveArticleByMessageIDHash: Retrieves an article from the
MongoDB collection by its MessageIDHash.

RetrieveArticlesByMessageIDHashes: Retrieves articles from the
MongoDB collection based on a list of MessageIDHashes.

RetrieveHeadByMessageIDHash: Retrieves the "Head" data of an
article based on its MessageIDHash.

RetrieveBodyByMessageIDHash: Retrieves the "Body" data of an
article based on its MessageIDHash.

CheckIfArticleExistsByMessageIDHash: Checks if an article exists in
the MongoDB collection based on its MessageIDHash.

Using External Context

The functions in the mongostorage package accept a ctx
context.Context and mongo options, allowing you to provide your own
MongoDB context from the main program.

In Go, the context.Context is used to manage the lifecycle of
operations and carry deadlines, cancellation signals, and other
request-scoped values across API boundaries.

It ensures the graceful termination of operations and handling of
long-running tasks.

To use these functions with an external context, you can create
your MongoDB client and collection in the main program and pass the
context.Context to the functions.

This allows the MongoDB operations to be context-aware and respect
the context's timeout and cancellation signals.

Extending Context Timeout

If you need to extend the timeout of the context for specific
operations, you can use the ExtendContextTimeout function provided by
the package.

Here's an example of how to use it:

// Extending the context timeout for a specific operation
newCtx, newCancel := mongostorage.ExtendContextTimeout(ctx, cancel, 20)
// defer newCancel() <- defer is only needed if you dont call
ExtendContextTimeout again. pass `cancel` to ExtendContextTimeout to
cancel there.
// Now use the newCtx for the MongoDB operation

Here's an example of how you can use the mongostorage functions
with an external context:

package main

import (
"context"
"log"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"yourproject/mongostorage"
)

func main() {
// Create a MongoDB client with a timeout for connecting to the server
client, err := mongo.Connect(context.Background(),
options.Client().ApplyURI("mongodb://localhost:27017").SetConnectTimeout(5*time.Second))
if err != nil {
log.Fatalf("Error connecting to MongoDB: %v", err)
}
defer client.Disconnect(context.Background())

// Get a reference to the MongoDB collection
collection :=
client.Database("yourdbname").Collection("yourcollectionname")

// Create a context with a timeout and pass it to the functions in the
mongostorage package
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()

// Example of inserting a single article
article := &mongostorage.MongoArticle{
MessageIDHash: strPtr("hash1"),
MessageID: strPtr("msgid1"),
Newsgroups: []string{"group1", "group2"},
Head: []byte("This is the head of article 1"),
Headsize: 100,
Body: []byte("This is the body of article 1"),
Bodysize: 200,
Enc: 0, // not compressed
}
err = mongostorage.InsertOneArticle(ctx, collection, article)
if err != nil {
log.Printf("Error inserting article: %v", err)
}

// Example of inserting multiple articles in bulk
articles := []*mongostorage.MongoArticle{
{
MessageIDHash: strPtr("hash1"),
MessageID: strPtr("msgid1"),
Newsgroups: []string{"group1", "group2"},
Head: []byte("This is the head of article 1"),
Headsize: 100,
Body: []byte("This is the body of article 1"),
Bodysize: 200,
Enc: 0, // not compressed
},
{
MessageIDHash: strPtr("hash1"), // will generate a duplicate error
MessageID: strPtr("msgid1"),
Newsgroups: []string{"group1", "group2"},
Head: []byte("This is the head of article 1"),
Headsize: 100,
Body: []byte("This is the body of article 1"),
Bodysize: 200,
Enc: 1, // indicator: compressed with GZIP (sender has to
apply de/compression)
},
{
MessageIDHash: strPtr("hash2"),
MessageID: strPtr("msgid2"),
Newsgroups: []string{"group3", "group4"},
Head: []byte("This is the head of article 2"),
Headsize: 150,
Body: []byte("This is the body of article 2"),
Bodysize: 300,
Enc: 2, // indicator: compressed with ZLIB (sender has to
apply de/compression)
},
// Add more articles as needed
}

err = mongostorage.InsertManyArticles(ctx, collection, articles)
if err != nil {
log.Printf("Error inserting articles: %v", err)
} else {
log.Println("Articles inserted successfully.")
}

// Check if the error is a duplicate key error using the IsDup function.
if IsDup(err) {
log.Println("Duplicate key error: The document already exists.")
// Handle the duplicate key error here, if needed.
} else {
log.Println("Other error occurred:", err)
// Handle other write errors, if needed.
}

// Example of retrieving an article
messageIDHash := "your-article-message-id-hash"
article, err := mongostorage.RetrieveArticleByMessageIDHash(ctx,
collection, messageIDHash)
if err != nil {
log.Printf("Error retrieving article: %v", err)
} else if article != nil {
log.Println("Article found:", article)
} else {
log.Println("Article not found.")
}

// Example of retrieving multiple articles based on a list of
MessageIDHashes
msgIDHashes := []*string{"hash1", "hash2", "hash3"}
articles, err := mongostorage.RetrieveArticleByMessageIDHashes(ctx,
collection, msgIDHashes)
if err != nil {
log.Printf("Error retrieving articles: %v", err)
} else {
for _, article := range articles {
if article != nil {
log.Println("Article found:", article)
} else {
log.Println("Article not found.")
}
}
}

// Example of deleting article(s).
msgidhashes := []string{"hash1", "hash2", "hash3"}
success := mongostorage.DeleteManyArticles(ctx, collection, msgidhashes)
if success {
log.Println("Articles deleted successfully.")
}

}

The MessageIDHash field in the MongoArticle struct serves as an
identifier for each article in the MongoDB collection.

The name "hash" in MessageIDHash does not necessarily imply that it
must be a cryptographic hash; it can be any string, and it acts as a
unique identifier for the document in the collection.

The field MessageIDHash is connected to the MongoDB's special _id
field, which uniquely identifies each document in a collection.

MongoDB requires each document to have a unique value for its _id
field.

By setting the MessageIDHash as the value of the _id field, we
ensure that each article is uniquely identified in the collection.

Additionally, the MongoArticle struct also contains a separate
field named MessageID, which can store the original, non-hashed version
of the article's identifier.

This allows developers to store both the original identifier and
the hashed identifier in the same document, providing flexibility and
convenience when working with the articles.

Getting Started

To use this module, you will need Go installed on your system. You can
install the module using go get: