After switching to ovsqlite, I get the following error for one group anytime I
run expireover, whether targeting directly or from news.daily, the only output
is "can't expire fr.soc.politique":

$ echo fr.soc.politique | expireover -f - -Z tmp/lowmark
expireover: can't expire fr.soc.politique
Article lines processed 4119153
Articles dropped 0
Overview index dropped 0

Is there a way to get more output?

Hi Jesse,

> After switching to ovsqlite, I get the following error for one group anytime I
> run expireover, whether targeting directly or from news.daily, the only output
> is "can't expire fr.soc.politique":
>
> $ echo fr.soc.politique | expireover -f - -Z tmp/lowmark
> expireover: can't expire fr.soc.politique
> Article lines processed 4119153
> Articles dropped 0
> Overview index dropped 0
>
> Is there a way to get more output?

There's unfortunately no more output to get, and I do not know why
you're facing that error only with one newsgroup.

Is there a matching rule in expire.ctl?
Nothing weird when you dump its overview?

ovsqlite-util -g -n fr.soc.politique

--
Julien ÉLIE

« Tout est dans tout, et réciproquement. » (Pierre Dac)

On Dec 10, 2023 at 3:24:02 AM CST, "Julien ÉLIE"
<iulius@nom-de-mon-site.com.invalid> wrote:

> Hi Jesse,
>
>> After switching to ovsqlite, I get the following error for one group anytime I
>> run expireover, whether targeting directly or from news.daily, the only output
>> is "can't expire fr.soc.politique":
>>
>> $ echo fr.soc.politique | expireover -f - -Z tmp/lowmark
>> expireover: can't expire fr.soc.politique
>> Article lines processed 4119153
>> Articles dropped 0
>> Overview index dropped 0
>>
>> Is there a way to get more output?
>
> There's unfortunately no more output to get, and I do not know why
> you're facing that error only with one newsgroup.
>
> Is there a matching rule in expire.ctl?
> Nothing weird when you dump its overview?
>
> ovsqlite-util -g -n fr.soc.politique

The only matching rule is the default one I have not to expire anything.

*:A:never:never:never

I don't need to expire this group, but found it odd that the error was
produced from the daily expireover, but only for this one group.

I can dump the contents with ovsqlite-util and, while I didn't examine all 4+
million lines, everything looks normal.

Hi Jesse,

>>> $ echo fr.soc.politique | expireover -f - -Z tmp/lowmark
>>> expireover: can't expire fr.soc.politique
>
> I don't need to expire this group, but found it odd that the error was
> produced from the daily expireover, but only for this one group.
>
> I can dump the contents with ovsqlite-util and, while I didn't examine all 4+
> million lines, everything looks normal.

That's indeed very odd, either a bug or a corruption of some data.

This error appears when the expire_one() function in
storage/ovsqlite/ovsqlite.c returns false.

If you wish and have a bit of time to investigate, is it possible to
rebuild INN with additional logs at each "return false"?
For instance, changing:

if (!write_request())
return false;

to:

if (!write_request()) {
syswarn("first write_request failed");
return false;
}

and so on...
There are 20 occurrences.

In the for loop, you may want to add the processed article number:

if (!unpack_now(response, &expires, sizeof expires)) {
syswarn("unpack_now expires failed for %lu", artnum);
return false;
}

Once the culprit is found, additional investigations will have to be
done, with again additional more precise logs, so it will take more time.
Tell me if you want to.
Also, for the changes of the 20 occurrences, do you want to do that
yourself or should I send you the modified ovsqlite.c file to your
e-mail address?

--
Julien ÉLIE

« Mon grand-père buvait un litre de whisky par jour et il est mort à 103
ans. À l'incinération, on a cru que le feu ne s'arrêterait jamais. »
(Sloppy White)

On Dec 11, 2023 at 2:50:49 PM CST, "Julien ÉLIE"
<iulius@nom-de-mon-site.com.invalid> wrote:

> Hi Jesse,
>
>>>> $ echo fr.soc.politique | expireover -f - -Z tmp/lowmark
>>>> expireover: can't expire fr.soc.politique
>>
>> I don't need to expire this group, but found it odd that the error was
>> produced from the daily expireover, but only for this one group.
>>
>> I can dump the contents with ovsqlite-util and, while I didn't examine all 4+
>> million lines, everything looks normal.
>
> That's indeed very odd, either a bug or a corruption of some data.
>
> This error appears when the expire_one() function in
> storage/ovsqlite/ovsqlite.c returns false.
>
> If you wish and have a bit of time to investigate, is it possible to
> rebuild INN with additional logs at each "return false"?
> For instance, changing:
>
> if (!write_request())
> return false;
>
> to:
>
> if (!write_request()) {
> syswarn("first write_request failed");
> return false;
> }
>
> and so on...
> There are 20 occurrences.
>
> In the for loop, you may want to add the processed article number:
>
> if (!unpack_now(response, &expires, sizeof expires)) {
> syswarn("unpack_now expires failed for %lu", artnum);
> return false;
> }
>
>
> Once the culprit is found, additional investigations will have to be
> done, with again additional more precise logs, so it will take more time.
> Tell me if you want to.
> Also, for the changes of the 20 occurrences, do you want to do that
> yourself or should I send you the modified ovsqlite.c file to your
> e-mail address?

I'm always open to troubleshooting, if you can send the modified file so I
don't accidentally muck it up, I would appreciate it. I will recompile and
provide new output. If it outputs the offending article number I can provide
the contents/output from sm, etc.

Hi all,

>> expireover: can't expire fr.soc.politique
>
> I'm always open to troubleshooting, if you can send the modified file so I
> don't accidentally muck it up, I would appreciate it.

After investigation (thanks again Jesse!), the problem comes from an
article (spam) with an insanely long Subject header field:

<rvcdut$26a$10@dont-email.me>

http://al.howardknight.net/?STYPE=msgid&MSGI=%3Crvcdut%2426a%2410%40dont-email.me%3E

The *headers* of that article have a size of 112 kilobytes, and both
ovdb and ovsqlite consider overview data of a single article as
corrupted if it exceeds 100 kb.
As discussed with Bo Lindbergh, a check is currently missing: also only
allowing overview data of that size at addition time. Otherwise,
overview data of any size is added, and then no longer considered valid
when accessing it. Only spam or buggy articles are likely to have such
an insane size for headers, so we won't really miss articles with that
check.

--
Julien ÉLIE

« Affirmanti incumbit probatio. »