This is the situation which I face right now.

Suppose I run two milters to filter out unwanted mail.

One has a large memory footprint, but it executes quickly.

The other has a small memory footprint, but can take a long time to
execute - because it does things like lookups in DNSBLs which can
easily take ten seconds.

Both milters allocate resources, and so both need to hook the 'abort'
and 'close' callbacks so they can clean up after themselves.

The large, fast milter may know that it has nothing left to do for any
particular message, but since Sendmail doesn't call its close callback
until after the second milter has finished its processing (even if the
first milter returns 'SMFIS_ACCEPT' from the connect callback) it must
stick around doing nothing but waste resources until the second milter
has terminated.

There's room in memory for half a dozen copies of the large milter, a
hundred or more of the small one. If the average execution times for
the two milters differ by a factor of twenty, then running five copies
of the large milter to a hundred of the small one seems about right.
But there's no point in running more copies of the smaller, slower
milter than of the larger, faster one because as things are the slow
ones just hold things up for the fast ones.

Ideally, I should like to be able to tell Sendmail to call the close
callback for the first milter when the first milter knows it's done
all it needs to do, but let the second milter continue processing.
The first milter can then see the next message in the queue and get on
with things.

My search skills don't seem to be up to finding how to do this, or if
it's even possible. Thanks in advance for any thoughts. Suggestions
of the "buy more memory" variety will be sent to /dev/null.

Ged.

G.W. Haywood wrote:

> The large, fast milter may know that it has nothing left to do for any
> particular message, but since Sendmail doesn't call its close callback
> until after the second milter has finished its processing (even if the
> first milter returns 'SMFIS_ACCEPT' from the connect callback) it must
> stick around doing nothing but waste resources until the second milter
> has terminated.

Why doesn't the "large" milter invoke its cleanup function as soon
as it "know[s] that it has nothing left to do"? The cleanup function
can set a flag in the milter context that it was called and did its
work (release all the memory), so further calls can just check that.

--
Note: please read the netiquette before posting. I will almost never
reply to top-postings which include a full copy of the previous
article(s) at the end because it's annoying, shows that the poster
is too lazy to trim his article, and it's wasting the time of all readers.

On Thu, 3 Jun 2021, Claus Assmann wrote:

> G.W. Haywood wrote:
>
>> The large, fast milter may know that it has nothing left to do for any
>> particular message, but since Sendmail doesn't call its close callback
>> until after the second milter has finished its processing (even if the
>> first milter returns 'SMFIS_ACCEPT' from the connect callback) it must
>> stick around doing nothing but waste resources until the second milter
>> has terminated.
>
> Why doesn't the "large" milter invoke its cleanup function as soon
> as it "know[s] that it has nothing left to do"? The cleanup function
> can set a flag in the milter context that it was called and did its
> work (release all the memory), so further calls can just check that.

Thanks, but it's not quite as simple as that.

Yes it could clean up as it goes along, but there's a *lot* of state
information. Some of it is per message, some is per connection, some
is per recipient, some per sender, some is persistent... In a milter
with ten or more callbacks, each having not only its 'personal' set of
state information but also with some overlaps between them, it quickly
gets ugly, difficult to manage - and so, error-prone. New code would
carry the risks of any code which would otherwise not be needed. I've
spent over five years working on the large milter. It now does what I
want, and it's more or less ready for others to try using it. They're
queueing. Doing what you suggest would mean another significant chunk
of work coding and testing before I'd feel comfortable releasing it (I
don't have the energy to maintain two versions of it, so releasing one
with 'faster' resource management and one without isn't an option).

The 'close' and 'abort' callbacks are there for exactly this purpose.
They exist anyway and have been exercised for years; it would be much
easier, cleaner and more efficient if they could be used as I suggest.

Is there no way, then, short of hacking the Sendmail source? As you
know I'm not above trying my hand at that. I'd have thought it would
be a good _FFR_ to have in Sendmail anyway. I'm probably not the best
qualified around here to take it on, although I routinely run a couple
of other patches (which would very likely make your toes curl).

Ged.

G.W. Haywood wrote:
> On Thu, 3 Jun 2021, Claus Assmann wrote:

> > Why doesn't the "large" milter invoke its cleanup function as soon
> > as it "know[s] that it has nothing left to do"? The cleanup function
> > can set a flag in the milter context that it was called and did its
> > work (release all the memory), so further calls can just check that.

> Thanks, but it's not quite as simple as that.

Maybe someone else understands the problem because I don't.

! Ideally, I should like to be able to tell Sendmail to call the close
! callback for the first milter when the first milter knows it's done

Why does it make a difference between sendmail "call[ing] the close
callback" and your own milter doing that?

! The large, fast milter may know that it has nothing left to do for any
! particular message, but since Sendmail doesn't call its close callback
! until after the second milter has finished its processing (even if the
! first milter returns 'SMFIS_ACCEPT' from the connect callback)

If your "milter returns 'SMFIS_ACCEPT' from the connect callback"
then why would it even allocate resources which it doesn't need?
If your milter returns SMFIS_ACCEPT from a transaction callback
then sendmail has no way to know that your milter doesn't even want
the subsequent transactions, i.e., you would have to come up with
another return code.

Anway, you got the source, so you can hack it to do what you
consider necessary for your special situation.

--
Note: please read the netiquette before posting. I will almost never
reply to top-postings which include a full copy of the previous
article(s) at the end because it's annoying, shows that the poster
is too lazy to trim his article, and it's wasting the time of all readers.

On Sat, 5 Jun 2021, Claus Assmann wrote:

> Maybe someone else understands the problem because I don't.

I guess I haven't explained it very well. Let me try again.

I have two milters. One is big (huge) but fast, the other is small
but it can be very slow. Often, the small one holds up the big one so
that the big one consumes a lot of resources while doing *nothing* but
wait for the MTA to release it from duty. That can be a nuisance, it
can hold up the flow of mail because there's no milter child process
available for Sendmail to connect to when a new connection comes in.
All the existing child processes are busy, some of them doing nothing.

> ! Ideally, I should like to be able to tell Sendmail to call the close
> ! callback for the first milter when the first milter knows it's done
>
> Why does it make a difference between sendmail "call[ing] the close
> callback" and your own milter doing that?

Until I read your question above, I had the impression that callbacks
are for the MTA to call. It never occurred to me to call them myself
and I'd have expected doing that to break the communications between
Sendmail and the milter. Does Sendmail not care if I call the close
callback myself? If it's allowed by the milter protocol, then the
problem might be solved. :) Granted I haven't read that it isn't, I
guess I just ass-u-me-d it isn't. I'll experiment with the idea.

> If your "milter returns 'SMFIS_ACCEPT' from the connect callback"
> then why would it even allocate resources which it doesn't need?

As I said, some of the state is persistent and it allocates quite a
variety of resources, some of which it allocates at 'connect' in the
expectation of using it both there and in later callbacks. When it
deallocates, it has to choose what it frees (garbage collection) and
when it frees it. On top of all this it's written in Perl so before
it even allocates any storage the thing starts off pretty big anyway
and there's not a lot I can do about that - short of rewriting parts
of it in C, which is on the cards when some of my outlandish theory
is either justified or proved not to be useful. It generally looks
like I'm on the right track, at least other people seem to see much
more spam and malicious stuff in their INBOX than I do.

> If your milter returns SMFIS_ACCEPT from a transaction callback
> then sendmail has no way to know that your milter doesn't even want
> the subsequent transactions, i.e., you would have to come up with
> another return code.

Hmmm, that doesn't sound so good for my situation and it's not what
I've taken home from the docs. The milter docs state as follows:

[quote source=file:///.../sendmail-8.17.0.0/libmilter/docs/api.html]

SMFIS_ACCEPT

For a connection-oriented routine, accept this connection without
further filter processing; ***CALL XXFI_CLOSE***. <=== my ephasis.

For a message- or recipient-oriented routine, accept this message
without further filtering.

[/quote]

There it says "call xxfi_close", which at first I took to mean that
the MTA would call xxfi_close immediately. Unfortunately it doesn't
call the callback until after completing all other milter processing
which is why I'm in this, er, bind. My naive thought processes ran
along the lines of

"If the milter replies SMFIS_ACCEPT, then that must tell Sendmail that
the only subsequent transaction will be close (or perhaps abort)".

Is that wrong? If so, why?

> Anway, you got the source, so you can hack it to do what you
> consider necessary for your special situation.

Quite so. Thanks again for your help, and for your dedication in the
face of all the stupid questions.

Ged.

G.W. Haywood wrote:
> On Sat, 5 Jun 2021, Claus Assmann wrote:

> > Why does it make a difference between sendmail "call[ing] the close
> > callback" and your own milter doing that?

> Until I read your question above, I had the impression that callbacks
> are for the MTA to call. It never occurred to me to call them myself

In C they are functions which are called by libmilter,
using a documented API and required return values.

> and I'd have expected doing that to break the communications between
> Sendmail and the milter. Does Sendmail not care if I call the close
> callback myself? If it's allowed by the milter protocol, then the

In C the callbacks do not communicate with the MTA,
obviously that's only handled by libmilter.

I don't know anything about whatever "libmilter" perl uses,
so you would have to check its docs.

--
Note: please read the netiquette before posting. I will almost never
reply to top-postings which include a full copy of the previous
article(s) at the end because it's annoying, shows that the poster
is too lazy to trim his article, and it's wasting the time of all readers.

On Sun, 6 Jun 2021, Claus Assmann wrote:

> G.W. Haywood wrote:
>> On Sat, 5 Jun 2021, Claus Assmann wrote:
>
>>> Why does it make a difference between sendmail "call[ing] the close
>>> callback" and your own milter doing that?
>
>> Until I read your question above, I had the impression that callbacks
>> are for the MTA to call. It never occurred to me to call them myself
>
> In C they are functions which are called by libmilter,
> using a documented API and required return values.

Sorry, I wasn't clear. I was treating libmilter as part of Sendmail
itself, because libmilter is normally compiled together with Sendmail
(at least that's how I've always done it for milters written in C).

>> and I'd have expected doing that to break the communications between
>> Sendmail and the milter. Does Sendmail not care if I call the close
>> callback myself? If it's allowed by the milter protocol, then the
>
> In C the callbacks do not communicate with the MTA,
> obviously that's only handled by libmilter.

Yes, I understand that of course. But the question stands, or maybe
phrased in a different way: Does Sendmail not care if close callbacks
are called *other* than because Sendmail has for example seen the end
of the message plus the client's 'QUIT' command and told the milters
(via libmilter) about that?

> I don't know anything about whatever "libmilter" perl uses,
> so you would have to check its docs.

I'm the maintainer of the Perl equivalent of libmilter which I use. I
took it over in a sadly unmaintained state a few years ago, fixed the
many reported bugs and overhauled the docs. The interface is called
Sendmail::PMilter. (For anyone else reading that's not to be confused
with Sendmail::Milter, which is also unmaintained and seems abandoned
by its author. I tried to take over maintainership of that too, but
the author was most unhelpful.)

Sendmail::PMilter does for Perl milters more or less what libmilter
does for C milters. It closely mimics the libmilter communications.
But now you have me asking if it makes a difference whether libmilter
is built or not. That also hadn't occurred to me. Thanks! It looks
like I have a bit more work to do.

PS: Sorry I have to keep changing the Eszett to 'ss'. My news client
(Alpine) chokes on that character and refuses to send the message.