On 2023-12-31, Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
> On Sun, 31 Dec 2023 17:20:16 -0000 (UTC), Kaz Kylheku wrote:
>
>> On 2023-12-31, Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
>>
>>> On Sun, 31 Dec 2023 07:27:56 -0000 (UTC), Kaz Kylheku wrote:
>>>
>>>> Cygnal is probably the best way to port a POSIX program to Windows.
>>>
>>> There is a better way now, and that is WSL2.
>>
>> I don't think so. With Cygnal, you deliver an .exe accompanied by a DLL
>> or two, and that's it.
>
> WSL2 at least lets you run a proper Linux distro, with proper package
> management.

That's nice, but not what you want when you want to ship a Windows
application: .exe plus a few .dll files, which run as ordinary Windows
applications without a special additional environment.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca
NOTE: If you use Google Groups, I don't see you, unless you're whitelisted.

On Mon, 1 Jan 2024 03:20:05 -0000 (UTC), Kaz Kylheku wrote:

> On 2023-12-31, Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
>
>> WSL2 at least lets you run a proper Linux distro, with proper package
>> management.
>
> That's nice, but not what you want when you want to ship a Windows
> application ...

Which is why WSL2 is likely to become a mandatory part of future Windows
installs. I’m not saying that was Microsoft’s conscious plan in
introducing it; but it is simply going to be the path of least resistance
going forward.

On 2024-01-01, Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
> On Mon, 1 Jan 2024 03:20:05 -0000 (UTC), Kaz Kylheku wrote:
>
>> On 2023-12-31, Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
>>
>>> WSL2 at least lets you run a proper Linux distro, with proper package
>>> management.
>>
>> That's nice, but not what you want when you want to ship a Windows
>> application ...
>
> Which is why WSL2 is likely to become a mandatory part of future Windows
> installs.

Might work in the future, and definitely won't work on older Windows,
like 7.

How easily do WSL2 programs work with the native Windows environment?

Suppose that the current directory on the F: drive is \data,
and the WSL2 program is given the path F:file.txt. Will it understand
that the path means F:\data\file.txt?

The Cygnal program will do that.

WSL's FAQ says this:

You can also access your local machine’s file system from within the
Linux Bash shell – you’ll find your local drives mounted under the
/mnt folder. For example, your C: drive is mounted under /mnt/c:

That's not native; it's like /cygdrive/c in Cygwin. I fixed that;
I got rid of /cygdrive/c and most of the path munging.

WSL2 does not look like a way, in any shape or form, of porting a
POSIX program to be a *native* Windows application.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca
NOTE: If you use Google Groups, I don't see you, unless you're whitelisted.

On 2024-01-01, Kaz Kylheku <433-929-6894@kylheku.com> wrote:
> On 2024-01-01, Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
>> On Mon, 1 Jan 2024 03:20:05 -0000 (UTC), Kaz Kylheku wrote:
>>
>>> On 2023-12-31, Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
>>>
>>>> WSL2 at least lets you run a proper Linux distro, with proper package
>>>> management.
>>>
>>> That's nice, but not what you want when you want to ship a Windows
>>> application ...
>>
>> Which is why WSL2 is likely to become a mandatory part of future Windows
>> installs.
>
> Might work in the future, and definitely won't work on older Windows,
> like 7.
>
> How easily do WSL2 programs work with the native Windows environment?
>
> Suppose that the current directory on the F: drive is \data,
> and the WSL2 program is given the path F:file.txt. Will it understand
> that the path means F:\data\file.txt?
>
> The Cygnal program will do that.
>
> WSL's FAQ says this:
>
> You can also access your local machine’s file system from within the
> Linux Bash shell – you’ll find your local drives mounted under the
> /mnt folder. For example, your C: drive is mounted under /mnt/c:
>
> That's not native; it's like /cygdrive/c in Cygwin. I fixed that;
> I got rid of /cygdrive/c and most of the path munging.

In a Cygnal program you can use Win32 APIs. Can you use Win32 APIs
in a WSL2 program?

In the same C program, you can do this

#include <termios.h>

and use tcgetattr/tcsetattr to fiddle with TTY settings that
really work with Windows consoles, and you can do this:

#include <windows.h>

and use MessageBox, CreateWindow and whatnot.

Your mostly POSIX program can have some Windows-specific personality
bits to it.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca
NOTE: If you use Google Groups, I don't see you, unless you're whitelisted.

On Mon, 1 Jan 2024 08:07:50 -0000 (UTC), Kaz Kylheku wrote:

> On 2024-01-01, Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
>
>> Which is why WSL2 is likely to become a mandatory part of future
>> Windows installs.
>
> Might work in the future, and definitely won't work on older Windows,
> like 7.

Which nobody uses any more--nobody that Microsoft cares about, anyway.

> How easily do WSL2 programs work with the native Windows environment?

They don’t need to, won’t need to.

Windows programs, on the other hand, will be working, unbeknownst to them,
through an emulation layer not entirely unlike WINE, onto the Linux
kernel.

> Suppose that the current directory on the F: drive is \data,
> and the WSL2 program is given the path F:file.txt. Will it understand
> that the path means F:\data\file.txt?

No, it means what you said, which is “F:file.txt”. Just tried this:

ldo@theon:~> touch F:file.txt
ldo@theon:~> ls -l F:file.txt
-rw-r--r-- 1 ldo users 0 Jan 2 10:32 F:file.txt

On 2024-01-01, Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
> On Mon, 1 Jan 2024 08:07:50 -0000 (UTC), Kaz Kylheku wrote:
>
>> On 2024-01-01, Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
>>
>>> Which is why WSL2 is likely to become a mandatory part of future
>>> Windows installs.
>>
>> Might work in the future, and definitely won't work on older Windows,
>> like 7.
>
> Which nobody uses any more--nobody that Microsoft cares about, anyway.
>
>> How easily do WSL2 programs work with the native Windows environment?
>
> They don’t need to, won’t need to.

It's nice that you can drop that requirement. I have that requirement.

I use the Cygnal run-time to support a programming language in the Lisp
family, and that programming language has access to a POSIX view of the
world, but also supports Windows programming via FFI.

This is achieved via simple mechanism: a run-time support C library DLL
that has the POSIX stuff earnestly implemented on top of Win32.

(It's analogous to Visual Studio's redistributable run-time library
or the newer Universal CRT. Those do not have POSIX support in them,
(or not any that is worth a damn) and also don't have a Unix-like build
environment that supports a configure shell script and all that.)

> Windows programs, on the other hand, will be working, unbeknownst to them,
> through an emulation layer not entirely unlike WINE, onto the Linux
> kernel.

That's just building conjectures upon guesses.

>> Suppose that the current directory on the F: drive is \data,
>> and the WSL2 program is given the path F:file.txt. Will it understand
>> that the path means F:\data\file.txt?
>
> No, it means what you said, which is “F:file.txt”. Just tried this:
>
> ldo@theon:~> touch F:file.txt
> ldo@theon:~> ls -l F:file.txt
> -rw-r--r-- 1 ldo users 0 Jan 2 10:32 F:file.txt

Well, that is not native Windows behavior, is it! In Windows, F: is a
drive letter name. If it's not followed by a slash or backslash, the
rest of it is a relative path resolved relative to the current working
directory for that drive. Each drive has its own current working
directory! There is also a current drive, the "currently logged drive".

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca
NOTE: If you use Google Groups, I don't see you, unless you're whitelisted.

On Mon, 1 Jan 2024 22:51:59 -0000 (UTC), Kaz Kylheku wrote:

> On 2024-01-01, Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
>
>> On Mon, 1 Jan 2024 08:07:50 -0000 (UTC), Kaz Kylheku wrote:
>>
>>> How easily do WSL2 programs work with the native Windows environment?
>>
>> They don’t need to, won’t need to.
>
> It's nice that you can drop that requirement. I have that requirement.

You have the requirement to run Windows software, you only care that it
works, not that it is running on a “true” Windows kernel rather than on
Linux via emulation.

>>> Suppose that the current directory on the F: drive is \data,
>>> and the WSL2 program is given the path F:file.txt. Will it understand
>>> that the path means F:\data\file.txt?
>>
>> No, it means what you said, which is “F:file.txt”. Just tried this:
>>
>> ldo@theon:~> touch F:file.txt
>> ldo@theon:~> ls -l F:file.txt
>> -rw-r--r-- 1 ldo users 0 Jan 2 10:32 F:file.txt
>
> Well, that is not native Windows behavior ...

Of course not. It’s Linux behaviour. That’s how programs running under
Linux expect things to work. Or at least native Linux behaviour: I’m sure
an emulation layer like WINE can make things look more Windows-like, if
you want. But then, it will no longer be a “WSL2 program”, will it? It
will just be a “Windows program”.

Julieta Shem <jshem@yaxenu.org> writes:

[...]

> In C I would have used select. But how do I select in Racket? I found
> Racket's sync/timeout, a procedure based on Racket's ``events''. The
> reference dissertates briefly about events at
>
> https://docs.racket-lang.org/reference/sync.html
>
> and they cite
>
> John H. Reppy, Concurrent Programming in ML, Cambridge University
> Press, 1999. https://doi.org/10.1017/CBO9780511574962
>
> as a reference. It's unclear to me that this is what I'm looking for,
> but it might very well be.
>
> #lang racket/base
> (define (read-line-timeout [timeout 60] [port (current-input-port)])
> (define port-or-false
> (sync/timeout timeout port))
> (and port-or-false (read-line port-or-false)))
>
> (define (interact-with-timeout timeout)
> (define ln (read-line-timeout timeout))
> (cond [(not ln)
> (say "> God, you're slow.")]
> [(eof-object? ln)
> (say "> Okay, see you later.")]
> [else
> (say (format "< ~a" ln))
> (say "> But why do you say that?")
> (interact-with-timeout timeout)]))
>
> (define (say s) (displayln s) (flush-output))
>
> (module+ main
> (say "> Hello. Say anything and press RET.")
> (interact-with-timeout 3))

As an exercise for in CL, I tried to write such procedure in CL using
select(2) or poll(2). I failed: did not find documentation to use the
SB-UNIX package. I see all procedures I want are there, but I have no
instruction as how to use data construtors to prepare arguments for a
procedure such as UNIX-FAST-SELECT. I found

https://koji-kojiro.github.io/sb-docs/build/html/index.html

which seems helpful, but with no examples. Given that I could write
such program in C using select(2) or poll(2), would there be anything
you could say here that would give me a direction? Thank you! Any
interesting publication that I so far failed to have found? (For
instance, I have Peter Seibel's Practical Common Lisp, but it doesn't
seem to help me with this exercise.)

On 2024-01-05, Julieta Shem <jshem@yaxenu.org> wrote:
> which seems helpful, but with no examples. Given that I could write
> such program in C using select(2) or poll(2), would there be anything
> you could say here that would give me a direction? Thank you! Any
> interesting publication that I so far failed to have found? (For
> instance, I have Peter Seibel's Practical Common Lisp, but it doesn't
> seem to help me with this exercise.)

One thing to watch out for, in any language, is if you're reading
from high level buffered streams but trying to use select or poll.
(This goes for C stdio also.)

You don't want to be accidentally blocking in select or poll
while the stream contains unread, buffered data (already transferred
from the operating system descriptor to user space).

That can happen if the stream reads more data than your program
requested. E.g. you're looking for one byte, and poll. The stream
initially has nothing so this is right. Poll unblocks when data
becomes available, so you read the one byte from the stream. But suppose
that the stream actually reads 500 bytes from the OS, giving you 1 and
buffering 499. After that, you have to read the available 499 bytes
from the stream before calling poll (unless poll is aware of streams,
rather than just a thin wrapper for the POSIX function.)

You have to put the underlying descriptor into non-blocking mode
(even if you're only reading from that one). Then whenever it polls
positive, read all available data from the stream until it hits an
error from the file descriptor with EWOULDBLOCK.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca
NOTE: If you use Google Groups, I don't see you, unless you're whitelisted.

Kaz Kylheku <433-929-6894@kylheku.com> writes:

> On 2024-01-05, Julieta Shem <jshem@yaxenu.org> wrote:
>> which seems helpful, but with no examples. Given that I could write
>> such program in C using select(2) or poll(2), would there be anything
>> you could say here that would give me a direction? Thank you! Any
>> interesting publication that I so far failed to have found? (For
>> instance, I have Peter Seibel's Practical Common Lisp, but it doesn't
>> seem to help me with this exercise.)
>
> One thing to watch out for, in any language, is if you're reading
> from high level buffered streams but trying to use select or poll.
> (This goes for C stdio also.)
>
> You don't want to be accidentally blocking in select or poll
> while the stream contains unread, buffered data (already transferred
> from the operating system descriptor to user space).
>
> That can happen if the stream reads more data than your program
> requested. E.g. you're looking for one byte, and poll. The stream
> initially has nothing so this is right. Poll unblocks when data
> becomes available, so you read the one byte from the stream. But suppose
> that the stream actually reads 500 bytes from the OS, giving you 1 and
> buffering 499. After that, you have to read the available 499 bytes
> from the stream before calling poll (unless poll is aware of streams,
> rather than just a thin wrapper for the POSIX function.)
>
> You have to put the underlying descriptor into non-blocking mode
> (even if you're only reading from that one). Then whenever it polls
> positive, read all available data from the stream until it hits an
> error from the file descriptor with EWOULDBLOCK.

I need to learn to put file descriptors in non-blocking mode. Thanks
for reminding me. Surely an abstraction to handle this is desired. The
problem seems to boil down to reading a partial line in the buffer
without having the rest of it --- say coming from the network still.
The procedure for reading such lines should somehow hold the partial
line until it completes.

I managed to discover UNIX-SIMPLE-POLL in SB-UNIX. Here's a procedure
that suffers from the problem you warned me above.

--8<---------------cut here---------------start------------->8---
(defun read-line-timeout (s)
"reads a line from stdin or times out in S seconds."
(let ((ready (sb-unix:unix-simple-poll 0 :input (* 1000 s))))
(cond (ready (format t "I read: ~a~%" (read-line)))
(t (format t "God, you're slow.~%")))))

;; main =
(read-line-timeout 3)
--8<---------------cut here---------------end--------------->8---

* Julieta Shem <87frzcwgxi.fsf @yaxenu.org> :
Wrote on Fri, 05 Jan 2024 01:19:21 -0300:
> I need to learn to put file descriptors in non-blocking mode. Thanks
> for reminding me. Surely an abstraction to handle this is desired. The
> problem seems to boil down to reading a partial line in the buffer
> without having the rest of it --- say coming from the network still.
> The procedure for reading such lines should somehow hold the partial
> line until it completes.
>
> I managed to discover UNIX-SIMPLE-POLL in SB-UNIX. Here's a procedure
> that suffers from the problem you warned me above.
>
> (defun read-line-timeout (s)
> "reads a line from stdin or times out in S seconds."
> (let ((ready (sb-unix:unix-simple-poll 0 :input (* 1000 s))))
> (cond (ready (format t "I read: ~a~%" (read-line)))
> (t (format t "God, you're slow.~%")))))
>
> ;; main =
> (read-line-timeout 3)

I think a better way to approach this is to (somehow) set a
stream-timout on the stream and have read-line (on your special stream)
return nil (or a supplied on-timeout value) if there is no data on the
stream. your lisp code in your thread would still deal with a blocking
read.

Madhu <enometh@meer.net> writes:

> * Julieta Shem <87frzcwgxi.fsf @yaxenu.org> :
> Wrote on Fri, 05 Jan 2024 01:19:21 -0300:
>> I need to learn to put file descriptors in non-blocking mode. Thanks
>> for reminding me. Surely an abstraction to handle this is desired. The
>> problem seems to boil down to reading a partial line in the buffer
>> without having the rest of it --- say coming from the network still.
>> The procedure for reading such lines should somehow hold the partial
>> line until it completes.
>>
>> I managed to discover UNIX-SIMPLE-POLL in SB-UNIX. Here's a procedure
>> that suffers from the problem you warned me above.
>>
>> (defun read-line-timeout (s)
>> "reads a line from stdin or times out in S seconds."
>> (let ((ready (sb-unix:unix-simple-poll 0 :input (* 1000 s))))
>> (cond (ready (format t "I read: ~a~%" (read-line)))
>> (t (format t "God, you're slow.~%")))))
>>
>> ;; main =
>> (read-line-timeout 3)
>
> I think a better way to approach this is to (somehow) set a
> stream-timout on the stream and have read-line (on your special stream)
> return nil (or a supplied on-timeout value) if there is no data on the
> stream. your lisp code in your thread would still deal with a blocking
> read.

Sounds interesting. It's a stream like stdin, stdout, stderr, but it
has a timeout associate with it. Reading or writing to it times out if
nothing usual happens before. Is that the idea?

It's a bit scary that this isn't done by any library given all the years
of SBCL. What's up with that? I'm not talking about CL as a standard
--- I'm talking about the implementations. In practice we need all of
these system-specific things.

I was expecting replies in the form --- hold it, young Skywalker: here's
how we've doing this for decades. I'm a total beginner. The procedure
above is my first procedure in CL. It makes me feel that the most
useful thing I can do in CL is learn about the SBCL's FFI. That's no
complaint. I'm just trying to understand the Lisp world.

On Fri, 05 Jan 2024 17:34:20 +0530, Madhu wrote:

> I think a better way to approach this is to (somehow) set a
> stream-timout on the stream and have read-line (on your special stream)
> return nil (or a supplied on-timeout value) if there is no data on the
> stream. your lisp code in your thread would still deal with a blocking
> read.

Remember the main point of select/poll, in that it lets you non-
deterministically wait on a number of file descriptors at once, and
respond immediately to the first one that has any I/O to do.

Lawrence D'Oliveiro <ldo@nz.invalid> writes:

> On Fri, 05 Jan 2024 17:34:20 +0530, Madhu wrote:
>
>> I think a better way to approach this is to (somehow) set a
>> stream-timout on the stream and have read-line (on your special stream)
>> return nil (or a supplied on-timeout value) if there is no data on the
>> stream. your lisp code in your thread would still deal with a blocking
>> read.
>
> Remember the main point of select/poll, in that it lets you non-
> deterministically wait on a number of file descriptors at once, and
> respond immediately to the first one that has any I/O to do.

I think Madhu's mind is on handling just a single file descriptor but
with a timeout (and with all the conform of a stream).

Julieta Shem <jshem@yaxenu.org> writes:

> Kaz Kylheku <433-929-6894@kylheku.com> writes:
>
>> On 2024-01-05, Julieta Shem <jshem@yaxenu.org> wrote:
>>> which seems helpful, but with no examples. Given that I could write
>>> such program in C using select(2) or poll(2), would there be anything
>>> you could say here that would give me a direction? Thank you! Any
>>> interesting publication that I so far failed to have found? (For
>>> instance, I have Peter Seibel's Practical Common Lisp, but it doesn't
>>> seem to help me with this exercise.)
>>
>> One thing to watch out for, in any language, is if you're reading
>> from high level buffered streams but trying to use select or poll.
>> (This goes for C stdio also.)
>>
>> You don't want to be accidentally blocking in select or poll
>> while the stream contains unread, buffered data (already transferred
>> from the operating system descriptor to user space).
>>
>> That can happen if the stream reads more data than your program
>> requested. E.g. you're looking for one byte, and poll. The stream
>> initially has nothing so this is right. Poll unblocks when data
>> becomes available, so you read the one byte from the stream. But suppose
>> that the stream actually reads 500 bytes from the OS, giving you 1 and
>> buffering 499. After that, you have to read the available 499 bytes
>> from the stream before calling poll (unless poll is aware of streams,
>> rather than just a thin wrapper for the POSIX function.)
>>
>> You have to put the underlying descriptor into non-blocking mode
>> (even if you're only reading from that one). Then whenever it polls
>> positive, read all available data from the stream until it hits an
>> error from the file descriptor with EWOULDBLOCK.
>
> I need to learn to put file descriptors in non-blocking mode. Thanks
> for reminding me.

It turns out SB-POSIX provides fcntl().

--8<---------------cut here---------------start------------->8---
(require 'sb-posix)
(defun ndelay-on (fd)
(sb-posix:fcntl fd
sb-posix:f-setfl
(logior sb-posix:o-nonblock
(sb-posix:fcntl fd sb-posix:f-getfl 0))))
--8<---------------cut here---------------end--------------->8---

Reading the source of SB-UNIX, I managed to learn how to invoke
UNIX-READ. So I was able to write the following procedure. It puts the
FD into non-blocking mode, then it allocates a buffer using SB-ALIEN,
whose memory I have no idea who is freeing. I had to learn how to close
the c-string. Small buffer of size 10 on purpose.

--8<---------------cut here---------------start------------->8---
(defun ndelay-read-from-fd (fd)
(ndelay-on fd)
(let ((buf (make-alien (array char 10))))
(multiple-value-bind (nbytes-or-nil errno)
;; spare a byte for closing the c-string
(sb-unix:unix-read fd (alien-sap buf) (1- 10))
(cond ((null nbytes-or-nil) (values nil errno))
(t
;; close the c-string
(setf (deref (deref buf 0) nbytes-or-nil) 0)
(values nbytes-or-nil buf))))))
--8<---------------cut here---------------end--------------->8---

As a sample program, I wrote ndelay-read-line. The ``read-line'' is
surely illusory because I'm not parsing lines at all. Maybe Madhu is
going to wrap this code with a stream ``somehow''. :-)

--8<---------------cut here---------------start------------->8---
(defun ndelay-read-line ()
(defun strerror (n) (if (= n 11) "EGAIN" "EDONTKNOW"))
(multiple-value-bind (nbytes-or-nil alien-or-errno) (ndelay-read-from-fd 0)
(cond ((and (null nbytes-or-nil))
(format t "errno = ~a~%" (strerror alien-or-errno)))
((= 0 nbytes-or-nil)
(format t "eof (read returned 0 bytes)~%"))
(t (format t "~a..."
(cast alien-or-errno c-string))
(ndelay-read-line)))))

;; main =
(ndelay-read-line)
--8<---------------cut here---------------end--------------->8---

We can call this a second exercise. I suppose I have enough for the
third exercise of polling the FD with a timeout and be sure we will not
block if the data is not a full line. It will surely take more
abstraction. (Eventually I need to learn what is being allocated and
who is to free it and when.)

Someone pointed me out to CFFI at

https://cffi.common-lisp.dev/

with the promise of being more expressive than SB-ALIEN.

* Julieta Shem <87jzonvg81.fsf @yaxenu.org> :
Wrote on Fri, 05 Jan 2024 14:32:14 -0300:

> Madhu <enometh@meer.net> writes:
>> I think a better way to approach this is to (somehow) set a
>> stream-timout on the stream and have read-line (on your special stream)
>> return nil (or a supplied on-timeout value) if there is no data on the
>> stream. your lisp code in your thread would still deal with a blocking
>> read.
>
> Sounds interesting. It's a stream like stdin, stdout, stderr, but it
> has a timeout associate with it. Reading or writing to it times out if
> nothing usual happens before. Is that the idea?
>
> It's a bit scary that this isn't done by any library given all the years
> of SBCL. What's up with that? I'm not talking about CL as a standard
> --- I'm talking about the implementations. In practice we need all of
> these system-specific things.
>
> I was expecting replies in the form --- hold it, young Skywalker: here's
> how we've doing this for decades. I'm a total beginner. The procedure
> above is my first procedure in CL. It makes me feel that the most
> useful thing I can do in CL is learn about the SBCL's FFI. That's no
> complaint. I'm just trying to understand the Lisp world.

I searched for a published example but I didn't find it -- I have an
example in an implementation of a "https stream" a subclass of fd-stream
(probably derived from something "emarsden" posted somewhere, but i
can't find it) which specializes the fd-stream class and adds a
connection timeout. IIRC This integrates with the SERVE-EVENT
abstraction which is a co-ooperative multiprocessing layer about select,
and the implementation is tied to SERVE-EVENT, if the fd is not usable
(data does not arrive on it in the given time), it raises a timeout
error.

On second thoughts I don't think it will solve your problem. I just
thought this was the way to approach the problem: call READ and handle a
timeout condition (from the implementation)

I think the CMUCL code base still has the code for a netnews client for
hemlock, though it doesn't use this strategy. You can see the code from
1980, (which has not yet been deleted by some open source who has been
hired to delete it) here
https://gitlab.common-lisp.net/cmucl/cmucl/-/blob/master/src/hemlock/netnews.lisp
(or wihout js)
https://gitlab.common-lisp.net/cmucl/cmucl/-/raw/master/src/hemlock/netnews.lisp

I checked trivial-nntp (probably in quicklisp) but it just wraps usocket
and doesnt deal with timeouts. iolib also doesnt seem to expose any
setsock SO_RCVTIMEO, though it can probably used, i havent figured it
out, i'll have to look again later.

Julieta Shem <jshem@yaxenu.org> writes:

> Funny you think that Debian and Red Hat ever had it. What they seem to
> have done --- with all due respect --- is to compress files with
> metadata. Pretty useful, but I think that /management/ should be
> applied to GNU Guix, with due credit to Nix, the pioneer.

If you are into GNU Guix (I am currently being tempted to switch), then
you are aware of Guile (methinks you mentioned it in another thread),
which brings me back to your original question: Since Guile is
advertised with its POSIX capabilities, I expect you could find
a better/more familiar technique there.

But that is just guessing and gut feeling, not based on any research.

Best regards

Axel

Axel Reichert <mail@axel-reichert.de> writes:

> Julieta Shem <jshem@yaxenu.org> writes:
>
>> Funny you think that Debian and Red Hat ever had it. What they seem to
>> have done --- with all due respect --- is to compress files with
>> metadata. Pretty useful, but I think that /management/ should be
>> applied to GNU Guix, with due credit to Nix, the pioneer.
>
> If you are into GNU Guix (I am currently being tempted to switch), then
> you are aware of Guile (methinks you mentioned it in another thread),

I did.

> which brings me back to your original question: Since Guile is
> advertised with its POSIX capabilities, I expect you could find a
> better/more familiar technique there.
>
> But that is just guessing and gut feeling, not based on any research.

Relative to my desires that's good intuition. I've been looking for a
Lisp as a medium of expression and Racket does appear to be the most
sophisticated one. However, it's still too early for me to be really
taken by any language --- it takes so many years. People here called my
attention to Common Lisp and even though I've only effectively spent a
few hours on it I'm impressed by how much fun it has been. Somehow
Racket has not done this for me, despite a much more considerable
investment I've already made into it --- relative to Common Lisp. But,
as you hypothesized, seeing how much fun I was having just considering
Common Lisp I did think of Guile precisely because of the GNU Guix
connection. (I asked Kaz Kylheku what he thought of Guile, but I think
he did not follow up or I did not see it somehow.)

Right now --- the way I feel --- is kinda of a no-brainer: Common Lisp
is fun. Why is it more fun than the all the beauty of Racket? I don't
know. I could elaborate, but a beginner's perspective doesn't sound
very fruitful really.

I half-jokingly remarked to a friend yesterday that reading Common Lisp
documentation I felt like a computing historian. The SBCL manual,
section 9.9, gives an example of a C function in K&R-style C --- pasted
below. (I was just messing around.) Nevertheless, I do enjoy the
history of computing quite a lot and it does feel good to actually use
tools older than the current ones that actually feels /quite/ superior
--- whatever that means. Even if it were inferior, it would still be
fun to use, but since it feels quite the opposite then it's surreally
interesting. (I saw a quote by Rich Hickey yesterday saying something
like --- patterns mean you've ran out of language --- or something like
that. I thought it was quite spot on.)

--8<---------------cut here---------------start------------->8---
struct c_struct *c_function (i, s, r, a)
int i;
char *s;
struct c_struct *r;
int a[10];
{ int j;
struct c_struct *r2;
printf("i = %d\n", i);
printf("s = %s\n", s);
printf("r->x = %d\n", r->x);
printf("r->s = %s\n", r->s);
for (j = 0; j < 10; j++) printf("a[%d] = %d.\n", j, a[j]);
r2 = (struct c_struct *) malloc (sizeof(struct c_struct));
r2->x = i + 5;
r2->s = "a C string";
return(r2);
};
--8<---------------cut here---------------end--------------->8---

Thanks for the Guile suggestion.

Madhu <enometh@meer.net> writes:

> * Julieta Shem <87jzonvg81.fsf @yaxenu.org> :
> Wrote on Fri, 05 Jan 2024 14:32:14 -0300:
>
>> Madhu <enometh@meer.net> writes:
>>> I think a better way to approach this is to (somehow) set a
>>> stream-timout on the stream and have read-line (on your special stream)
>>> return nil (or a supplied on-timeout value) if there is no data on the
>>> stream. your lisp code in your thread would still deal with a blocking
>>> read.
>>
>> Sounds interesting. It's a stream like stdin, stdout, stderr, but it
>> has a timeout associate with it. Reading or writing to it times out if
>> nothing usual happens before. Is that the idea?
>>
>> It's a bit scary that this isn't done by any library given all the years
>> of SBCL. What's up with that? I'm not talking about CL as a standard
>> --- I'm talking about the implementations. In practice we need all of
>> these system-specific things.
>>
>> I was expecting replies in the form --- hold it, young Skywalker: here's
>> how we've doing this for decades. I'm a total beginner. The procedure
>> above is my first procedure in CL. It makes me feel that the most
>> useful thing I can do in CL is learn about the SBCL's FFI. That's no
>> complaint. I'm just trying to understand the Lisp world.
>
> I searched for a published example but I didn't find it -- I have an
> example in an implementation of a "https stream" a subclass of fd-stream
> (probably derived from something "emarsden" posted somewhere, but i
> can't find it) which specializes the fd-stream class and adds a
> connection timeout. IIRC This integrates with the SERVE-EVENT
> abstraction which is a co-ooperative multiprocessing layer about select,
> and the implementation is tied to SERVE-EVENT, if the fd is not usable
> (data does not arrive on it in the given time), it raises a timeout
> error.
>
> On second thoughts I don't think it will solve your problem.

Let's talk about the problem, which I think we did not do too much yet.
The exercise is not an NNTP client, but a server. I think the server
should not be totally naive when it comes to waiting for peers to talk.
(Even though I believe a denial-of-service might be a too-difficult
problem to solve. I don't even know what ``perfect'' protection against
such attacks would be.) I would imagine that TCP already solves the
timeout problem, but still --- I feel like it should be an easy exercise
for a program to handle that itself. So, my exercise was to read a file
descriptor with a timeout --- give me your command or die.

> I just thought this was the way to approach the problem: call READ and
> handle a timeout condition (from the implementation)
>
> I think the CMUCL code base still has the code for a netnews client for
> hemlock, though it doesn't use this strategy. You can see the code from
> 1980, (which has not yet been deleted by some open source who has been
> hired to delete it) here
> https://gitlab.common-lisp.net/cmucl/cmucl/-/blob/master/src/hemlock/netnews.lisp
> (or wihout js)
> https://gitlab.common-lisp.net/cmucl/cmucl/-/raw/master/src/hemlock/netnews.lisp

Thanks. That's fun reading.

> I checked trivial-nntp (probably in quicklisp) but it just wraps
> usocket and doesnt deal with timeouts. iolib also doesnt seem to
> expose any setsock SO_RCVTIMEO, though it can probably used, i havent
> figured it out, i'll have to look again later.

Thanks for looking into it.

The strategy that I would apply to the exercise is to use a TCP server.
I suppose usocket implies sockets but my exercise expects to handle just
standard input, so I guess I could not really use a socket library for
anything.

Julieta Shem <jshem@yaxenu.org> writes:

> taken by any language --- it takes so many years. People here called my
> attention to Common Lisp and even though I've only effectively spent a
> few hours on it I'm impressed by how much fun it has been. Somehow
> Racket has not done this for me, despite a much more considerable
> investment I've already made into it --- relative to Common Lisp.

https://ashwinram.org/1986/01/28/a-short-ballad-dedicated-to-the-growth-of-programs/

On Sat, 06 Jan 2024 18:43:24 -0300, Julieta Shem wrote:

> So, my exercise was to read a file
> descriptor with a timeout --- give me your command or die.

Typically you will have multiple concurrent client connections going on.
So you want an event loop that monitors all connections. Each one will
have a last-active timer. The poll(2) call will have a timeout that
corresponds to the earliest of these timer expirations. So if you don’t
get any activity in that time (or even if you do), you start disconnecting
idle clients.

I have a worked example of this, in Python using asyncio here
<https://gitlab.com/ldo/chunk_protocol_example>.

Lawrence D'Oliveiro <ldo@nz.invalid> writes:

> On Sat, 06 Jan 2024 18:43:24 -0300, Julieta Shem wrote:
>
>> So, my exercise was to read a file
>> descriptor with a timeout --- give me your command or die.
>
> Typically you will have multiple concurrent client connections going on.
> So you want an event loop that monitors all connections. Each one will
> have a last-active timer. The poll(2) call will have a timeout that
> corresponds to the earliest of these timer expirations. So if you don’t
> get any activity in that time (or even if you do), you start disconnecting
> idle clients.

The exercise is much easier than that --- it's an NNTP server, so the
service doesn't need to handle clients in a centralized manner. The TCP
server will spawn the nntpd, which will only handle a single client.

* Julieta Shem <87ttnpsysl.fsf @yaxenu.org> :
Wrote on Sat, 06 Jan 2024 22:43:54 -0300:
> Lawrence D'Oliveiro <ldo@nz.invalid> writes:
>
>> On Sat, 06 Jan 2024 18:43:24 -0300, Julieta Shem wrote:
>>
>>> So, my exercise was to read a file
>>> descriptor with a timeout --- give me your command or die.
>>
>> Typically you will have multiple concurrent client connections going
>> on. So you want an event loop that monitors all connections. Each
>> one will have a last-active timer. The poll(2) call will have a
>> timeout that corresponds to the earliest of these timer
>> expirations. So if you don’t get any activity in that time (or even
>> if you do), you start disconnecting idle clients.
>
> The exercise is much easier than that --- it's an NNTP server, so the
> service doesn't need to handle clients in a centralized manner. The
> TCP server will spawn the nntpd, which will only handle a single
> client.

TCPD is a nice abstraction but in my mind there is a conflict, bringing
in lisp, also brings an unbearable urge to manage the file descriptors
from lisp in a long running stateful lisp process, Personally I never
got over this particular dissonance ("If I use lisp I _have_ to take
advantage of this feature long-running-process -- or bust"), and prefer
short small executables programs for processes. [OTOH I remember cl-http
on buggy linux in 1996 which would result in a number of TCP connections
which were never cleaned up for 2 days]

Likewise "If I use Lisp I _have_ to take advantage of the stream
abstraction and all the goodies of its stream design". In lisp stdin
(when it is also *terminal-io*) is special because it is an interactive
stream. You can call PEEK on it to see if there is data to be read and
and handle some aspects of interactivity primitively through that. CL
defines INTERACTIVE-STREAM-P and extensions like gray streams purport to
support it. but when you get stdin from a fdstream or TCPstream, it
isn't really special.

Madhu <enometh@meer.net> writes:

> * Julieta Shem <87ttnpsysl.fsf @yaxenu.org> :
> Wrote on Sat, 06 Jan 2024 22:43:54 -0300:
>> Lawrence D'Oliveiro <ldo@nz.invalid> writes:
>>
>>> On Sat, 06 Jan 2024 18:43:24 -0300, Julieta Shem wrote:
>>>
>>>> So, my exercise was to read a file
>>>> descriptor with a timeout --- give me your command or die.
>>>
>>> Typically you will have multiple concurrent client connections going
>>> on. So you want an event loop that monitors all connections. Each
>>> one will have a last-active timer. The poll(2) call will have a
>>> timeout that corresponds to the earliest of these timer
>>> expirations. So if you don’t get any activity in that time (or even
>>> if you do), you start disconnecting idle clients.
>>
>> The exercise is much easier than that --- it's an NNTP server, so the
>> service doesn't need to handle clients in a centralized manner. The
>> TCP server will spawn the nntpd, which will only handle a single
>> client.
>
> TCPD is a nice abstraction but in my mind there is a conflict, bringing
> in lisp, also brings an unbearable urge to manage the file descriptors
> from lisp in a long running stateful lisp process, Personally I never
> got over this particular dissonance ("If I use lisp I _have_ to take
> advantage of this feature long-running-process -- or bust"), and prefer
> short small executables programs for processes. [OTOH I remember cl-http
> on buggy linux in 1996 which would result in a number of TCP connections
> which were never cleaned up for 2 days]
>
> Likewise "If I use Lisp I _have_ to take advantage of the stream
> abstraction and all the goodies of its stream design". In lisp stdin
> (when it is also *terminal-io*) is special because it is an interactive
> stream. You can call PEEK on it to see if there is data to be read and
> and handle some aspects of interactivity primitively through that. CL
> defines INTERACTIVE-STREAM-P and extensions like gray streams purport to
> support it. but when you get stdin from a fdstream or TCPstream, it
> isn't really special.

I don't understand. The program is being written to handle a TCP
connection. Terminals are not going to be connected to it. How would
you handle that?

I am thankful I can write the program with my keyboard attached to it
and then later naturally switch it all to a TCP connection. Isn't that
great?

* Julieta Shem <87edetd6vz.fsf @yaxenu.org> :
Wrote on Sun, 07 Jan 2024 02:55:44 -0300:
> Madhu <enometh@meer.net> writes:
>> TCPD is a nice abstraction but in my mind there is a conflict, bringing
>> in lisp, also brings an unbearable urge to manage the file descriptors
>> from lisp in a long running stateful lisp process, Personally I never
[...]

>> Likewise "If I use Lisp I _have_ to take advantage of the stream
>> abstraction and all the goodies of its stream design". In lisp stdin
>> (when it is also *terminal-io*) is special because it is an interactive
>> stream. You can call PEEK on it to see if there is data to be read and
>> and handle some aspects of interactivity primitively through that. CL
>> defines INTERACTIVE-STREAM-P and extensions like gray streams purport to
>> support it. but when you get stdin from a fdstream or TCPstream, it
>> isn't really special.
>
> I don't understand. The program is being written to handle a TCP
> connection. Terminals are not going to be connected to it. How would
> you handle that?

I wouldn't. I just meant that using lisp for a program that reads
standard input and writes to standard output would not be able to use
the (admittedly limited) interactive features of interactive-stream-p
and peek, which were designed to handle interactivity.

Once you have to set non-blocking on the FD you've already breached the
abstraction barrier (that limits you dealing solely with stdin
stdout). When you want to do timeouts on the input, that is a another
breach. You're no longer programming within the niceties where you
program just reads stdin and handles stdout and leave TCPD to handle all
the network and timeout issues. So now there is a sort of square/round
peg/hole.

> I am thankful I can write the program with my keyboard attached to it
> and then later naturally switch it all to a TCP connection. Isn't that
> great?