Hi,

hopefully I have stripped the problematic code to something
representative but simple enough ...

I (using SBCL and Slime on macOS) have the following:

(defparameter *call-for-action*
(uiop:launch-program "nc -l 11111" :input :stream :output :stream))

to listen on some example port. Some third party application writes
lines to this stream, upon which I want to act according to the content
of the line.

Since only occasionally a new line arrives on the stream and I do not
want to get stuck if nothing happens (this can be taken as a sign that
the third party application is done with its business), I am listening
to the port. If nothing is there, I listen again after 10 s. If
something is there (on first or second try), I read it, and the action
returns a non-nil value.

(defun handle-one-action ()
(let ((stream (uiop:process-info-output *call-for-action*)))
(when (or (listen stream)
(progn (sleep 10) (listen stream)))
(read-line stream nil)
(some-action-returning-non-nil))))

Now I am looping over this:

(loop (unless (handle-one-action) (return)))

The loop works fine and acts "immediately" upon line after line, until
the third party application is done (there are other ways to confirm
this). Then the loop gets stuck (I waited for more than 10 seconds ...
(-:).

While stepping through the debugger and evaluating forms, I was able to
find out that

(listen stream)

returns T even after the app is done. However, neither a read-line nor a
read-char were able to read anything and thus blocked. When I aborted
the debugger and tried

(loop (unless (handle-one-action) (return)))

again, it returned as expected after 10 seconds. The debugger revealed
that now before the read-line evaluating

(listen stream)

returned NIL.

So I understand why my code hangs, but not why "listen" detects something
when there seems to be nothing.

Any hints on how I can ensure a robust exit from the loop? Pointers much
appreciated!

Axel
--
-X- | in memoriam John Conway
--X | 1937-2020
XXX | A glider from his "Game of Life"

* Axel Reichert <m21r60sj3q.fsf@axel-reichert.de> :
Wrote on Tue, 07 Sep 2021 13:47:21 +0200:
> hopefully I have stripped the problematic code to something
> representative but simple enough ...
>
> I (using SBCL and Slime on macOS) have the following:

[No sbcl/uiop here - so consume the following with a pinch of salt]

> (defparameter *call-for-action*
> (uiop:launch-program "nc -l 11111" :input :stream :output :stream))
>
> to listen on some example port. Some third party application writes
> lines to this stream, upon which I want to act according to the content
> of the line.

[snip]

> (defun handle-one-action ()
> (let ((stream (uiop:process-info-output *call-for-action*)))
> (when (or (listen stream)
> (progn (sleep 10) (listen stream)))
> (read-line stream nil)
> (some-action-returning-non-nil))))
>
> Now I am looping over this:
> (loop (unless (handle-one-action) (return)))
> The loop works fine and acts "immediately" upon line after line, until
> the third party application is done (there are other ways to confirm
> this). Then the loop gets stuck (I waited for more than 10 seconds ...
> (-:).
>
> While stepping through the debugger and evaluating forms, I was able to
> find out that
>
> (listen stream)
>
> returns T even after the app is done.

When the connection to netcat from the remote host is dropped, netcat
exits. The netcat process is dead. (uiop:process-alive-p
*call-for-action*) would return NIL.

But the streams which the Lisp has created may still be open - they may
have unread data and can still be read. So LISTEN returns T

PEEK-CHAR may return T and READ-CHAR may return a value. But once the
characters are read off the stream you should get an EOF condition on
the stream you are reading and then LISTEN should yield NIL.

READ-LINE should encouter EOF in case there isn't a newline on the
stream.

I've checked now and this is how CCL behaves. LISTEN returns T until
there is unread data - when it hits EOF it returns NIL

[Note - you can't inspect the ccl streams from another thread unless you
call CCL:RUN-PROGRAM with :SHARING :EXTERNAL stream arguments]

> However, neither a read-line nor a
> read-char were able to read anything and thus blocked. When I aborted
> the debugger and tried
>
> (loop (unless (handle-one-action) (return)))
>
> again, it returned as expected after 10 seconds. The debugger revealed
> that now before the read-line evaluating
>
> (listen stream)
>
> returned NIL.

That means there is no more data now and the stream buffer has been
emptied.

> So I understand why my code hangs, but not why "listen" detects something
> when there seems to be nothing.

I don't think this matches my experience.

> Any hints on how I can ensure a robust exit from the loop? Pointers much
> appreciated!

I would ditch this whole approach (using sleep 10, etc.) - and try to
use iolib streams but that i suspect that won't be more any more robust,
cross platform.

Hi Axel,

I do not follow what you are trying to do. Your *call-for-action*
process is a netcat server already waiting for connections and handling
clients as netcat do. So I don't know what a listen to its stream from
sbcl will do.

Here is an complete (no external netcat involved) usocket example of a
timing out server that might be what you want:
--8<---------------cut here---------------start------------->8---
(asdf:load-system :usocket)

(defun myserver (&optional (port 11111) (timeout 10))
(let ((connections (list (usocket:socket-listen usocket:*wildcard-host* port
:reuse-address t))))
(unwind-protect
(loop for sockets = (usocket:wait-for-input connections :ready-only t
:timeout timeout)
while sockets do
(dolist (ready sockets)
(cond ((usocket:stream-server-usocket-p ready) ;; this is a new connection to me
(let ((client (usocket:socket-accept ready)))
(push client connections)))
(t ;; there is message from some client
(setf connections (remove ready connections))
(handle-client ready)))))
(dolist (c connections) (usocket:socket-close c)))))

(defun handle-client (socket)
(let ((stream (usocket:socket-stream socket)))
(unwind-protect
(when (listen stream)
(format t "~&message: ~a~%" (read-line stream)))
(usocket:socket-close socket))))
--8<---------------cut here---------------end--------------->8---

Best regards,
--
Manuel Giraud

* Manuel Giraud <87zgsn8nwp.fsf@elite.giraud> :
Wrote on Wed, 08 Sep 2021 10:36:06 +0200:

> I do not follow what you are trying to do. Your *call-for-action*
> process is a netcat server already waiting for connections and handling
> clients as netcat do. So I don't know what a listen to its stream from
> sbcl will do.

It's common-lisp's LISTEN on lisp streams

http://www.lispworks.com/documentation/lw71/CLHS//Body/f_listen.htm

Madhu <enometh@meer.net> writes:

> When the connection to netcat from the remote host is dropped, netcat
> exits. The netcat process is dead. (uiop:process-alive-p
> *call-for-action*) would return NIL.

This is not the case. It returns T.

> PEEK-CHAR may return T

Yes, in a sense. In fact it returned #\Nul, which screwed up things
further downstream. After a read-char of #\Nul, (listen ...) returned
NIL. Now I grab that NUL char and listen again.

So there is progress: It does not hang any more, but I am still
struggling with the correct return values in my loop. But not today.

Thanks for the reminder on the existence of peek-char!

> I would ditch this whole approach (using sleep 10, etc.) - and try to
> use iolib streams

Perhaps. Manuel's post pointed in the same direction.

Thanks for now, I will conduct some trials.

Axel
--
-X- | in memoriam John Conway
--X | 1937-2020
XXX | A glider from his "Game of Life"

Manuel Giraud <manuel@ledu-giraud.fr> writes:

> I do not follow what you are trying to do. Your *call-for-action*
> process is a netcat server already waiting for connections and handling
> clients as netcat do.

It seems I was losing the forest for the trees. You mean I should
instead "directly" (in Lisp) listen to the third party application? True
enough ...

> Here is an complete (no external netcat involved) usocket example of a
> timing out server that might be what you want:

Sounds good, many thanks, I will give it a try if my "new hope" (see
reply to Madhu) fails.

Best regards

Axel
--
-X- | in memoriam John Conway
--X | 1937-2020
XXX | A glider from his "Game of Life"

Madhu <enometh@meer.net> writes:

> * Manuel Giraud <87zgsn8nwp.fsf@elite.giraud> :
> Wrote on Wed, 08 Sep 2021 10:36:06 +0200:
>
>> I do not follow what you are trying to do. Your *call-for-action*
>> process is a netcat server already waiting for connections and handling
>> clients as netcat do. So I don't know what a listen to its stream from
>> sbcl will do.
>
> It's common-lisp's LISTEN on lisp streams

Yes, I know. But my question was more what would result from a call to
CL listen on a stream that is already handled by netcat... but I may be
missing something here.
--
Manuel Giraud