Barry, thanks for your reply. 

On the theory that it is not yet possible to pass data from a non-Python language to Python with multiprocessing.shared_memory, I bypassed the problem by attaching 4 bytes to my FIFO pipe message from NASM to Python:

byte_val = v[10:14]

where v is the message read from the FIFO.  Then:

breakup = int.from_bytes(byte_val, "big")
print("this is breakup " + str(breakup))

Python prints:  this is breakup 32894

Note that I had to switch from little endian to big endian.  Python is little endian by default, but in this case it's big endian. 

However, if anyone on this list knows how to pass data from a non-Python language to Python in multiprocessing.shared_memory please let me (and the list) know. 

Thanks. 

Feb 1, 2022, 14:20 by barry@barrys-emacs.org:

>
>
>> On 1 Feb 2022, at 20:26, Jen Kris via Python-list <python-list@python.org> wrote:
>>
>> I am using multiprocesssing.shared_memory to pass data between NASM and Python. The shared memory is created in NASM before Python is called. Python connects to the shm: shm_00 = shared_memory.SharedMemory(name='shm_object_00',create=False).
>>
>> I have used shared memory at other points in this project to pass text data from Python back to NASM with no problems. But now this time I need to pass a 32-bit integer (specifically 32,894) from NASM to Python.
>>
>> First I convert the integer to bytes in a C program linked into NASM:
>>
>> unsigned char bytes[4]
>> unsigned long int_to_convert = 32894;
>>
>> bytes[0] = (int_to_convert >> 24) & 0xFF;
>> bytes[1] = (int_to_convert >> 16) & 0xFF;
>> bytes[2] = (int_to_convert >> 8) & 0xFF;
>> bytes[3] = int_to_convert & 0xFF;
>> memcpy(outbuf, bytes, 4);
>>
>> where outbuf is a pointer to the shared memory. On return from C to NASM, I verify that the first four bytes of the shared memory contain what I want, and they are 0, 0, -128, 126 which is binary 00000000 00000000 10000000 01111110, and that's correct (32,894).
>>
>> Next I send a message to Python through a FIFO to read the data from shared memory. Python uses the following code to read the first four bytes of the shared memory:
>>
>> byte_val = shm_00.buf[:4]
>> print(shm_00.buf[0])
>> print(shm_00.buf[1])
>> print(shm_00.buf[2])
>> print(shm_00.buf[3])
>>
>> But the bytes show as 40 39 96 96, which is exactly what the first four bytes of this shared memory contained before I called C to overwrite them with the bytes 0, 0, -128, 126. So Python does not see the updated bytes, and naturally int.from_bytes(byte_val, "little") does not return the result I want.
>>
>> I know that Python refers to shm00.buf, using the buffer protocol. Is that the reason that Python can't see the data that has been updated by another language?
>>
>> So my question is, how can I alter the data in shared memory in a non-Python language to pass back to Python?
>>
>
> Maybe you need to use a memory barrier to force the data to be seen by another cpu?
> Maybe use shm lock operation to sync both sides?
> Googling I see people talking about using stdatomic.h for this.
>
> But I am far from clear what you would need to do.
>
> Barry
>
>>
>> Thanks,
>>
>> Jen
>>
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>>

I applaud trying to find the right solution but wonder if a more trivial solution is even being considered. It ignores big and little endians and just converts your data into another form and back.

If all you want to do is send an integer that fit in 32 bits or 64 bits, why not convert it to a character string in a form that both machines will see the same way and when read back, convert it back to an integer?

As long as both side see the same string, this can be done in reasonable time and portably.

Or am I missing something? Is "1234" not necessarily seen in the same order, or "1.234e3" or whatever?

Obviously, if the mechanism is heavily used and multiple sides keep reading and even writing the same memory location, this is not ideal. But having different incompatible processors looking at the same memory is also not.

-----Original Message-----
From: Dennis Lee Bieber <wlfraed@ix.netcom.com>
To: python-list@python.org
Sent: Wed, Feb 2, 2022 12:30 am
Subject: Re: Data unchanged when passing data to Python in multiprocessing shared memory

On Wed, 2 Feb 2022 00:40:22 +0100 (CET), Jen Kris <jenkris@tutanota.com>

declaimed the following:

>

> breakup = int.from_bytes(byte_val, "big")

>print("this is breakup " + str(breakup))

>

>Python prints:  this is breakup 32894

>

>Note that I had to switch from little endian to big endian.  Python is little endian by default, but in this case it's big endian. 

>

    Look at the struct module. I'm pretty certain it has flags for big or

little end, or system native (that, or run your integers through the

various "network byte order" functions that I think C and Python both

support.

https://www.gta.ufrj.br/ensino/eel878/sockets/htonsman.html

>However, if anyone on this list knows how to pass data from a non-Python language to Python in multiprocessing.shared_memory please let me (and the list) know. 

    <pondering>MMU cache lines not writing through to RAM? Can't find

anything on Google to force a cache flush</pondering> Can you test on a

different OS? (Windows vs Linux)

--

    Wulfraed                Dennis Lee Bieber        AF6VN

    wlfraed@ix.netcom.com    http://wlfraed.microdiversity.freeddns.org/

--

https://mail.python.org/mailman/listinfo/python-list

It's not clear to me from the struct module whether it can actually auto-detect endianness.  I think it must be specified, just as I had to do with int.from_bytes().  In my case endianness was dictated by how the four bytes were populated, starting with the zero bytes on the left. 

Feb 1, 2022, 21:30 by wlfraed@ix.netcom.com:

> On Wed, 2 Feb 2022 00:40:22 +0100 (CET), Jen Kris <jenkris@tutanota.com>
> declaimed the following:
>
>>
>> breakup = int.from_bytes(byte_val, "big")
>>
> >print("this is breakup " + str(breakup))
>
>>
>>
> >Python prints:  this is breakup 32894
>
>>
>>
> >Note that I had to switch from little endian to big endian.  Python is little endian by default, but in this case it's big endian. 
>
>>
>>
> Look at the struct module. I'm pretty certain it has flags for big or
> little end, or system native (that, or run your integers through the
> various "network byte order" functions that I think C and Python both
> support.
>
> https://www.gta.ufrj.br/ensino/eel878/sockets/htonsman.html
>
>
> >However, if anyone on this list knows how to pass data from a non-Python language to Python in multiprocessing.shared_memory please let me (and the list) know. 
>
> <pondering>MMU cache lines not writing through to RAM? Can't find
> anything on Google to force a cache flush</pondering> Can you test on a
> different OS? (Windows vs Linux)
>
>
>
> --
> Wulfraed Dennis Lee Bieber AF6VN
> wlfraed@ix.netcom.com http://wlfraed.microdiversity.freeddns.org/
> --
> https://mail.python.org/mailman/listinfo/python-list
>

An ASCII string will not work.  If you convert 32894 to an ascii string you will have five bytes, but you need four.  In my original post I showed the C program I used to convert any 32-bit number to 4 bytes. 

Feb 2, 2022, 10:16 by python-list@python.org:

> I applaud trying to find the right solution but wonder if a more trivial solution is even being considered. It ignores big and little endians and just converts your data into another form and back.
>
> If all you want to do is send an integer that fit in 32 bits or 64 bits, why not convert it to a character string in a form that both machines will see the same way and when read back, convert it back to an integer?
>
> As long as both side see the same string, this can be done in reasonable time and portably.
>
> Or am I missing something? Is "1234" not necessarily seen in the same order, or "1.234e3" or whatever?
>
> Obviously, if the mechanism is heavily used and multiple sides keep reading and even writing the same memory location, this is not ideal. But having different incompatible processors looking at the same memory is also not.
>
> -----Original Message-----
> From: Dennis Lee Bieber <wlfraed@ix.netcom.com>
> To: python-list@python.org
> Sent: Wed, Feb 2, 2022 12:30 am
> Subject: Re: Data unchanged when passing data to Python in multiprocessing shared memory
>
>
> On Wed, 2 Feb 2022 00:40:22 +0100 (CET), Jen Kris <jenkris@tutanota.com>
>
> declaimed the following:
>
>
>
>>
>>
>> breakup = int.from_bytes(byte_val, "big")
>>
>
> >print("this is breakup " + str(breakup))
>
>>
>>
>
> >Python prints:  this is breakup 32894
>
>>
>>
>
> >Note that I had to switch from little endian to big endian.  Python is little endian by default, but in this case it's big endian. 
>
>>
>>
>
>     Look at the struct module. I'm pretty certain it has flags for big or
>
> little end, or system native (that, or run your integers through the
>
> various "network byte order" functions that I think C and Python both
>
> support.
>
>
>
> https://www.gta.ufrj.br/ensino/eel878/sockets/htonsman.html
>
>
>
>
>
> >However, if anyone on this list knows how to pass data from a non-Python language to Python in multiprocessing.shared_memory please let me (and the list) know. 
>
>
>
>     <pondering>MMU cache lines not writing through to RAM? Can't find
>
> anything on Google to force a cache flush</pondering> Can you test on a
>
> different OS? (Windows vs Linux)
>
>
>
>
>
>
>
> --
>
>     Wulfraed                Dennis Lee Bieber        AF6VN
>
>     wlfraed@ix.netcom.com    http://wlfraed.microdiversity.freeddns.org/
>
> --
>
> https://mail.python.org/mailman/listinfo/python-list
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

Jen,

I would not be shocked at incompatibilities in the system described making it hard to exchange anything, including text, but am not clear if there is a limitation of four bytes in what can be shared. For me, a character string can use any number of contiguous bytes in memory that some kind of pointer or table lookup provides access to.

Clearly you could squeeze larger number in by not writing decimals but using hexadecimal notation that adds a to f as valid entries, and of course you can make your own customized base 32 or if you use more symbols (such as upper and lower case as different) you could add a few symbols and stretch it out to base 60 or 64.

Like I said, I may have missed something. If you KNOW that system A will interact with systems B and C and so on, and each machine knows what kind it is, and especially if it knows which kind of machine left it something in shared memory, there has to be a way to coordinate between them. If character strings in ASCII or UTF8 or EBCDIC or some kind of raw are allowed, and used with just a few characters that can spell out numbers, I would think much is possible. If needed, perhaps a fixed size could be set aside and the string use a null terminator if shorter.

Of course if this big-endian issue also scrambles bytes used in strings, forget it.

Or, maybe shared memory is not the easy way to go, even it it might be faster.

-----Original Message-----
From: Jen Kris <jenkris@tutanota.com>
To: Avi Gross <avigross@verizon.net>
Cc: python-list@python.org <python-list@python.org>
Sent: Wed, Feb 2, 2022 1:27 pm
Subject: Re: Data unchanged when passing data to Python in multiprocessing shared memory

An ASCII string will not work.  If you convert 32894 to an ascii string you will have five bytes, but you need four.  In my original post I showed the C program I used to convert any 32-bit number to 4 bytes. 

Feb 2, 2022, 10:16 by python-list@python.org:

I applaud trying to find the right solution but wonder if a more trivial solution is even being considered. It ignores big and little endians and just converts your data into another form and back.

If all you want to do is send an integer that fit in 32 bits or 64 bits, why not convert it to a character string in a form that both machines will see the same way and when read back, convert it back to an integer?

As long as both side see the same string, this can be done in reasonable time and portably.

Or am I missing something? Is "1234" not necessarily seen in the same order, or "1.234e3" or whatever?

Obviously, if the mechanism is heavily used and multiple sides keep reading and even writing the same memory location, this is not ideal. But having different incompatible processors looking at the same memory is also not.

-----Original Message-----

From: Dennis Lee Bieber <wlfraed@ix.netcom.com>

To: python-list@python.org

Sent: Wed, Feb 2, 2022 12:30 am

Subject: Re: Data unchanged when passing data to Python in multiprocessing shared memory

On Wed, 2 Feb 2022 00:40:22 +0100 (CET), Jen Kris <jenkris@tutanota.com>

declaimed the following:

breakup = int.from_bytes(byte_val, "big")

>print("this is breakup " + str(breakup))

>Python prints:  this is breakup 32894

>Note that I had to switch from little endian to big endian.  Python is little endian by default, but in this case it's big endian. 

    Look at the struct module. I'm pretty certain it has flags for big or

little end, or system native (that, or run your integers through the

various "network byte order" functions that I think C and Python both

support.

https://www.gta.ufrj.br/ensino/eel878/sockets/htonsman.html

>However, if anyone on this list knows how to pass data from a non-Python language to Python in multiprocessing.shared_memory please let me (and the list) know. 

    <pondering>MMU cache lines not writing through to RAM? Can't find

anything on Google to force a cache flush</pondering> Can you test on a

different OS? (Windows vs Linux)

--

    Wulfraed                Dennis Lee Bieber        AF6VN

    wlfraed@ix.netcom.com    http://wlfraed.microdiversity.freeddns.org/

--

https://mail.python.org/mailman/listinfo/python-list

--

https://mail.python.org/mailman/listinfo/python-list

On Thu, 3 Feb 2022 at 13:32, Avi Gross via Python-list
<python-list@python.org> wrote:
>
> Jen,
>
> I would not be shocked at incompatibilities in the system described making it hard to exchange anything, including text, but am not clear if there is a limitation of four bytes in what can be shared. For me, a character string can use any number of contiguous bytes in memory that some kind of pointer or table lookup provides access to.
>
> Clearly you could squeeze larger number in by not writing decimals but using hexadecimal notation that adds a to f as valid entries, and of course you can make your own customized base 32 or if you use more symbols (such as upper and lower case as different) you could add a few symbols and stretch it out to base 60 or 64.
>
> Like I said, I may have missed something. If you KNOW that system A will interact with systems B and C and so on, and each machine knows what kind it is, and especially if it knows which kind of machine left it something in shared memory, there has to be a way to coordinate between them. If character strings in ASCII or UTF8 or EBCDIC or some kind of raw are allowed, and used with just a few characters that can spell out numbers, I would think much is possible. If needed, perhaps a fixed size could be set aside and the string use a null terminator if shorter.
>
> Of course if this big-endian issue also scrambles bytes used in strings, forget it.
>
> Or, maybe shared memory is not the easy way to go, even it it might be faster.
>

Structs are fine, and are highly unlikely to be the OP's problem.
Switching to a different system won't help.

Unfortunately I can't offer any serious help about the actual problem,
as I don't know what's going on with the shared memory issue.

ChrisA