Hi,

I want to transfert some data between applications through a memory buffer.
The buffer transfert between applications is under control.
My problem is with the buffer content.
I though I'll use a Stream writing/reading in/from the memory buffer.
How can I achieve this ? I've found no example doing this.
Note : I use Ada 2012.

Nicolas

Le 16/02/2024 à 10:41, DrPi a écrit :
> Hi,
>
> I want to transfert some data between applications through a memory buffer.
> The buffer transfert between applications is under control.
> My problem is with the buffer content.
> I though I'll use a Stream writing/reading in/from the memory buffer.
> How can I achieve this ? I've found no example doing this.
> Note : I use Ada 2012.
>
I don't know if this is what you want, but at least it is an example of
using streams...
Package Storage_Streams, from Adalog's components page:
https://adalog.fr/en/components.html#Storage_Stream

--
J-P. Rosen
Adalog
2 rue du Docteur Lombard, 92441 Issy-les-Moulineaux CEDEX
https://www.adalog.fr https://www.adacontrol.fr

On 2024-02-16 10:41, DrPi wrote:

> I want to transfert some data between applications through a memory buffer.
> The buffer transfert between applications is under control.
> My problem is with the buffer content.
> I though I'll use a Stream writing/reading in/from the memory buffer.
> How can I achieve this ? I've found no example doing this.

It of course depends on the target operating system. You need to create
a shared region or memory mapped file etc. You also need system-wide
events to signal the stream ends empty or full.

Simple Components has an implementation interprocess streams for usual
suspects:

http://www.dmitry-kazakov.de/ada/components.htm#12.7

> Note : I use Ada 2012.

No problem, it is kept Ada 95 compatible.

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Hi,

AWS comes with a memory stream implementation.

https://github.com/AdaCore/aws/blob/master/include/memory_streams.ads

You may want to have a look here.

Have a nice day,

--
  Pascal Obry /  Magny Les Hameaux (78)

  The best way to travel is by means of imagination

  http://photos.obry.net

  gpg --keyserver keys.gnupg.net --recv-key F949BD3B

DrPi <314@drpi.fr> writes:

> Hi,
>
> I want to transfert some data between applications through a memory buffer.
> The buffer transfert between applications is under control.
> My problem is with the buffer content.
> I though I'll use a Stream writing/reading in/from the memory
> buffer. How can I achieve this ? I've found no example doing this.
> Note : I use Ada 2012.
>
> Nicolas

A spec and body for an implementation I've had since 2008:

https://github.com/simonjwright/coldframe/blob/alire/src/common/coldframe-memory_streams.ads
https://github.com/simonjwright/coldframe/blob/alire/src/common/coldframe-memory_streams.adb

On Fri, 16 Feb 2024 10:41:12 +0100, DrPi wrote:

> I though I'll use a Stream writing/reading in/from the memory buffer.

Wouldn’t be simplest to let the OS manage the buffering for you?

<https://manpages.debian.org/7/pipe.en.html>

On 2024-02-16 22:54, Lawrence D'Oliveiro wrote:
> On Fri, 16 Feb 2024 10:41:12 +0100, DrPi wrote:
>
>> I though I'll use a Stream writing/reading in/from the memory buffer.
>
> Wouldn’t be simplest to let the OS manage the buffering for you?
>
> <https://manpages.debian.org/7/pipe.en.html>

That would make applications OS-dependent.

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Le 16/02/2024 à 10:41, DrPi a écrit :
> I want to transfert some data between applications through a memory buffer.
> The buffer transfert between applications is under control.
> My problem is with the buffer content.
> I though I'll use a Stream writing/reading in/from the memory buffer.
> How can I achieve this ? I've found no example doing this.
> Note : I use Ada 2012.

Thanks all for your answers.

Concerning the OS and the buffer transfert mechanism, as I said, this is
under control. I use Windows and the WM_COPYDATA message.

My usage is a bit special. The writing process writes a bunch of data in
a memory buffer then requests this buffer to be transferred to another
process by way of WM_COPYDATA. The receiving process reads the data from
the "new" memory buffer. I say "new" since the address is different from
the one used in the writing process (of course it can not be the same).

The library Jean-Pierre pointed me to perfectly matches this usage.
Light and easy to use. Thanks.
One enhancement I see is to manage the buffer size to avoid buffer
overflow (or did I missed something ?).

Thanks again to everybody.

Nicolas

Le 17/02/2024 à 14:36, DrPi a écrit :
> The library Jean-Pierre pointed me to perfectly matches this usage.
Light and easy to use. Thanks.
:-)

> One enhancement I see is to manage the buffer size to avoid buffer
overflow (or did I missed something ?).

I don't see what you mean here... On the memory side, we are
reading/writing bytes from memory, there is no notion of overflow. And
the number of bytes processed by Read/Write is given by the size of
Item, so no overflow either...

--
J-P. Rosen
Adalog
2 rue du Docteur Lombard, 92441 Issy-les-Moulineaux CEDEX
https://www.adalog.fr https://www.adacontrol.fr

On 2024-02-17 14:36, DrPi wrote:

> Concerning the OS and the buffer transfert mechanism, as I said, this is
> under control. I use Windows and the WM_COPYDATA message.
>
> My usage is a bit special. The writing process writes a bunch of data in
> a memory buffer then requests this buffer to be transferred to another
> process by way of WM_COPYDATA. The receiving process reads the data from
> the "new" memory buffer. I say "new" since the address is different from
> the one used in the writing process (of course it can not be the same).

You ask Windows to copy a chunk of memory from one process space into
another, so yes it is physically different memory. Different or same
address tells nothing because under Windows System.Address is virtual
and can point anywhere.

As you may guess it is a quite heavy overhead, not only because of
copying data between process spaces, but also because of sending and
dispatching Windows messages.

Note, that if you implement stream Read/Write as individual Windows
messages it will become extremely slow. GNAT optimizes streaming of some
built-in objects, e.g. String. But as a general case you should expect
that streaming of any non-scalar object would cause multiple calls to
Read/Write and thus multiple individual Windows messages.

An efficient way to exchange data under Windows is a file mapping. See
CreateFileMapping and MapViewOfFile.

https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-createfilemappinga

https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-mapviewoffile

Then use CreateEvent with a name to signal states of the stream buffer
system-wide. Named Windows events are shared between processes.

https://learn.microsoft.com/en-us/windows/win32/api/synchapi/nf-synchapi-createeventa

[ This is how interprocess stream is implemented for Windows in Simple
Components ]

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Le 17/02/2024 à 15:26, J-P. Rosen a écrit :
> Le 17/02/2024 à 14:36, DrPi a écrit :
> > The library Jean-Pierre pointed me to perfectly matches this usage.
> Light and easy to use. Thanks.
> :-)
>
> > One enhancement I see is to manage the buffer size to avoid buffer
> overflow (or did I missed something ?).
>
> I don't see what you mean here... On the memory side, we are
> reading/writing bytes from memory, there is no notion of overflow. And
> the number of bytes processed by Read/Write is given by the size of
> Item, so no overflow either...
>
A memory buffer IS limited in size. It is either a peripheral buffer or
a memory buffer you create yourself (my case). In either case, its size
is limited. When writing in the stream, you have to care to not overflow
the buffer.

On 2024-02-17 15:26, J-P. Rosen wrote:

> On the memory side, we are
> reading/writing bytes from memory, there is no notion of overflow.

In the Simple Components there is a pipe stream.

type Pipe_Stream
( Size : Stream_Element_Count
) is new Root_Stream_Type with private;

When a task writes the stream full (Size elements), it gets blocked
until another task reads something out.

Another implementation

type Storage_Stream
( Block_Size : Stream_Element_Count
) is new Root_Stream_Type with private;

rather allocates a new block of memory. The allocated blocks get reused
when their contents is read out.

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Le 17/02/2024 à 15:28, Dmitry A. Kazakov a écrit :
> On 2024-02-17 14:36, DrPi wrote:
>
>> Concerning the OS and the buffer transfert mechanism, as I said, this
>> is under control. I use Windows and the WM_COPYDATA message.
>>
>> My usage is a bit special. The writing process writes a bunch of data
>> in a memory buffer then requests this buffer to be transferred to
>> another process by way of WM_COPYDATA. The receiving process reads the
>> data from the "new" memory buffer. I say "new" since the address is
>> different from the one used in the writing process (of course it can
>> not be the same).
>
> You ask Windows to copy a chunk of memory from one process space into
> another, so yes it is physically different memory. Different or same
> address tells nothing because under Windows System.Address is virtual
> and can point anywhere.
>
> As you may guess it is a quite heavy overhead, not only because of
> copying data between process spaces, but also because of sending and
> dispatching Windows messages.
>
> Note, that if you implement stream Read/Write as individual Windows
> messages it will become extremely slow. GNAT optimizes streaming of some
> built-in objects, e.g. String. But as a general case you should expect
> that streaming of any non-scalar object would cause multiple calls to
> Read/Write and thus multiple individual Windows messages.
>
> An efficient way to exchange data under Windows is a file mapping. See
> CreateFileMapping and MapViewOfFile.
>
>
> https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-createfilemappinga
>
>
> https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-mapviewoffile
>
> Then use CreateEvent with a name to signal states of the stream buffer
> system-wide. Named Windows events are shared between processes.
>
>
> https://learn.microsoft.com/en-us/windows/win32/api/synchapi/nf-synchapi-createeventa
>
> [ This is how interprocess stream is implemented for Windows in Simple
> Components ]
>
In my use case, there is no performance problem.
The purpose is to make an editor simple instance. When you launch the
editor the first time, everything is done as usual. Next time you launch
the editor (for example by double clicking on a file in file explorer)
the init code of the editor detects an instance of the editor is already
running, transfers the command line arguments to the first instance and
exit.
The buffer transfert occurs once when starting a new instance of the editor.

However, I keep your solution in mind. I might need it one day.
Thanks.

"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:

> Note, that if you implement stream Read/Write as individual Windows
> messages it will become extremely slow. GNAT optimizes streaming of
> some built-in objects, e.g. String. But as a general case you should
> expect that streaming of any non-scalar object would cause multiple
> calls to Read/Write and thus multiple individual Windows messages.

Our motivation for the memory stream was the equivalent of this for
UDP messages; GNAT.Sockets behaves (behaved?) exactly like this, so we
buffered the result of 'Output & wrote the constructed buffer to the
socket; on the other side, we read the UDP message, stuffed its contents
into a memory stream, then let the client 'Input.

I can't remember at this distance in time, but I think I would have
liked to construct a memory stream on the received UDP packet rather
than copying the content; the compiler wouldn't let me. Perhaps worth
another try.

Le 17/02/2024 à 15:42, DrPi a écrit :
> Le 17/02/2024 à 15:26, J-P. Rosen a écrit :
>> Le 17/02/2024 à 14:36, DrPi a écrit :
>> > The library Jean-Pierre pointed me to perfectly matches this
usage. Light and easy to use. Thanks.
>> 🙂
>>
>> > One enhancement I see is to manage the buffer size to avoid
buffer overflow (or did I missed something ?).
>>
>> I don't see what you mean here... On the memory side, we are
reading/writing bytes from memory, there is no notion of overflow. And
the number of bytes processed by Read/Write is given by the size of
Item, so no overflow either...
>>
> A memory buffer IS limited in size. It is either a peripheral buffer
or a memory buffer you create yourself (my case). In either case, its
size is limited. When writing in the stream, you have to care to not
overflow the buffer.

The purpose of this stream is to access raw memory, so there is no
notion of "buffer size". It is up to you to match your (user) buffer
with the memory buffer. Of course, you can add a layer with all the
checks you want...

[PS] I tried to respond to your email, but it bounced...
--
J-P. Rosen
Adalog
2 rue du Docteur Lombard, 92441 Issy-les-Moulineaux CEDEX
https://www.adalog.fr https://www.adacontrol.fr

On 2024-02-17 19:09, Simon Wright wrote:

> I can't remember at this distance in time, but I think I would have
> liked to construct a memory stream on the received UDP packet rather
> than copying the content; the compiler wouldn't let me. Perhaps worth
> another try.

UDP is a kind of thing... Basically, there is no use of UDP except for
broadcasting, e.g. in LAN discovery.

In all other cases it is either TCP or multicast. Since UDP does not
guarantee either delivery or ordering. It would be a huge overhead to
implement reliable buffered streams on top of UDP, with sequence
numbers, acknowledgements, re-sending, reordering etc.

As for taking apart a UDP packet, it is straightforward. You simply
declare a stream element array of the packet size and map it on the
packet using:

pragma Import (Ada, A);
for A'Address use UDP_Packet'Address;

And somewhere

pragma Assert (Stream_Element'Size = 8);

just in case...

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

On Sat, 17 Feb 2024 10:19:22 +0100, Dmitry A. Kazakov wrote:

> On 2024-02-16 22:54, Lawrence D'Oliveiro wrote:
>
>> On Fri, 16 Feb 2024 10:41:12 +0100, DrPi wrote:
>>
>>> I though I'll use a Stream writing/reading in/from the memory buffer.
>>
>> Wouldn’t be simplest to let the OS manage the buffering for you?
>>
>> <https://manpages.debian.org/7/pipe.en.html>
>
> That would make applications OS-dependent.

That’s a standard POSIX function. I think even M****s*ft W**d*ws has
something resembling it.

On Sat, 17 Feb 2024 14:36:46 +0100, DrPi wrote:

> My usage is a bit special. The writing process writes a bunch of data in
> a memory buffer then requests this buffer to be transferred to another
> process by way of WM_COPYDATA.

I thought Windows had pipes.

On Sat, 17 Feb 2024 15:42:13 +0100, DrPi wrote:

> A memory buffer IS limited in size. It is either a peripheral buffer or
> a memory buffer you create yourself (my case). In either case, its size
> is limited. When writing in the stream, you have to care to not overflow
> the buffer.

With pipes, the OS takes care of this for you. Once its kernel buffer is
full, further writes are automatically blocked until a reader has drained
something from the buffer.

It’s called “flow control”.

On 2024-02-17 23:00, Lawrence D'Oliveiro wrote:
> On Sat, 17 Feb 2024 10:19:22 +0100, Dmitry A. Kazakov wrote:
>
>> On 2024-02-16 22:54, Lawrence D'Oliveiro wrote:
>>
>>> On Fri, 16 Feb 2024 10:41:12 +0100, DrPi wrote:
>>>
>>>> I though I'll use a Stream writing/reading in/from the memory buffer.
>>>
>>> Wouldn’t be simplest to let the OS manage the buffering for you?
>>>
>>> <https://manpages.debian.org/7/pipe.en.html>
>>
>> That would make applications OS-dependent.
>
> That’s a standard POSIX function. I think even M****s*ft W**d*ws has
> something resembling it.

Yes, Windows has a POSIX layer which nobody ever uses.

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

On 2024-02-18 01:00, Lawrence D'Oliveiro wrote:
> On Sat, 17 Feb 2024 14:36:46 +0100, DrPi wrote:
>
>> My usage is a bit special. The writing process writes a bunch of data in
>> a memory buffer then requests this buffer to be transferred to another
>> process by way of WM_COPYDATA.
>
> I thought Windows had pipes.

Yes it has, but very rarely used though much better designed than UNIX
pipes. See

https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-createnamedpipea

In general Windows has much richer and better API regarding interprocess
communication than Linux. After all Windows NT was sort of descendant of
VMS, which was light years ahead of UNIX Sys V. In recent times Linux
improved, e.g. they added futex stuff etc. BSD is far worse than Linux
in respect of API.

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:

> UDP is a kind of thing... Basically, there is no use of UDP except for
> broadcasting, e.g. in LAN discovery.

Worked for us, sending radar measurements p-2-p at 200 Hz

> As for taking apart a UDP packet, it is straightforward. You simply
> declare a stream element array of the packet size and map it on the
> packet using:
>
> pragma Import (Ada, A);
> for A'Address use UDP_Packet'Address;
>
> And somewhere
>
> pragma Assert (Stream_Element'Size = 8);
>
> just in case...

OK if the participants all have the same endianness. We used XDR (and
the translation cost is nil if the host is big-endian, as PowerPCs are;
all the critical machines were PowerPC).

On 2024-02-18 01:00, Lawrence D'Oliveiro wrote:
> On Sat, 17 Feb 2024 14:36:46 +0100, DrPi wrote:
>
>> My usage is a bit special. The writing process writes a bunch of data in
>> a memory buffer then requests this buffer to be transferred to another
>> process by way of WM_COPYDATA.
>
> I thought Windows had pipes.

It does,
We use it for out IPC in both Linux and Windows.
Works very well.
We use named pipes - where each process knows its name through
via env-var
At start they create a named pipe with that name

We use anonymous pipes for client communication

--
/Björn

On 2024-02-18 11:06, Simon Wright wrote:

> OK if the participants all have the same endianness. We used XDR (and
> the translation cost is nil if the host is big-endian, as PowerPCs are;
> all the critical machines were PowerPC).

I always override stream attributes and use portable formats. E.g. some
chained code for integers. Sign + exponent + normalized mantissa for
floats, again chained. That is all. There is no need in XDR, JSON, ASN.1
or other data representation mess. They are just worthless overhead.

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

On Sun, 18 Feb 2024 11:06:16 +0100, Dmitry A. Kazakov wrote:

> On 2024-02-18 01:00, Lawrence D'Oliveiro wrote:
>> On Sat, 17 Feb 2024 14:36:46 +0100, DrPi wrote:
>>
>>> My usage is a bit special. The writing process writes a bunch of data in
>>> a memory buffer then requests this buffer to be transferred to another
>>> process by way of WM_COPYDATA.
>>
>> I thought Windows had pipes.
>
> Yes it has, but very rarely used though much better designed than UNIX
> pipes.

So why don’t programmers use it?

> In general Windows has much richer and better API regarding interprocess
> communication than Linux.

So why is it that Windows programs tend to avoid running multiple processes?