Rocksolid Light

Welcome to RetroBBS

mail  files  register  newsreader  groups  login

Message-ID:  

Delta: We never make the same mistake three times. -- David Letterman

devel / comp.lang.python / Re: Embedding Python crash on PyTuple_New

SubjectAuthor
o Re: Embedding Python crash on PyTuple_NewMRAB

1
Re: Embedding Python crash on PyTuple_New

<mailman.15.1637688685.31677.python-list@python.org>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=20590&group=comp.lang.python#20590

  copy link   Newsgroups: comp.lang.python
Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: python@mrabarnett.plus.com (MRAB)
Newsgroups: comp.lang.python
Subject: Re: Embedding Python crash on PyTuple_New
Date: Tue, 23 Nov 2021 17:31:22 +0000
Lines: 165
Message-ID: <mailman.15.1637688685.31677.python-list@python.org>
References: <d0b57569-ef0d-3ab4-9918-9cd498f1e658@sphaero.org>
<4eb693c6-fd5f-7c54-246d-0b081a0488d5@mrabarnett.plus.com>
<c827faf9-0871-fde3-83a0-fa19576b3c62@sphaero.org>
<d6c7a17a-b24c-9a45-a94e-7bbc3d8fb1c1@mrabarnett.plus.com>
<f7711e63-a9a7-eb7a-b2b1-0c7ebd9829af@mrabarnett.plus.com>
<8788bcb1-5e02-b241-3533-43c40d363906@sphaero.org>
<1d070d88-8b19-c1ea-c700-4a47876fe871@mrabarnett.plus.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Trace: news.uni-berlin.de RcVZC/2m2/zMqmjFmKXQuAZVPElntGMZgXaeYL8YMw6Q==
Return-Path: <python@mrabarnett.plus.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=plus.com header.i=@plus.com header.b=bJbHH/MX;
dkim-adsp=none (unprotected policy); dkim-atps=neutral
X-Spam-Status: OK 0.003
X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'looks': 0.02; 'message:':
0.03; 'url-ip:140.82/16': 0.03; 'debug': 0.07; 'string': 0.07;
'that?': 0.07; 'allocation': 0.09; 'compiler': 0.09; 'const':
0.09; 'from:addr:python': 0.09; 'int': 0.09;
'received:192.168.1.64': 0.09; 'skip:_ 20': 0.09; 'skip:z 20':
0.09; 'skip:\xc2 20': 0.09; 'string,': 0.09; 'subject:Python':
0.12; 'url:github': 0.14; 'memory': 0.15; 'that.': 0.15; 'url-
ip:140/8': 0.15; '"after': 0.16; '"c"': 0.16; '1240': 0.16;
'1262': 0.16; '1977': 0.16; '1987': 0.16; '8bit%:55': 0.16;
'=\xc2\xa0': 0.16; '>>>>': 0.16; '>>>>>': 0.16; '>>>>>>': 0.16;
'char': 0.16; 'compiled': 0.16; 'cpython': 0.16;
'from:addr:mrabarnett.plus.com': 0.16; 'from:name:mrab': 0.16;
'iterate': 0.16; 'members?': 0.16; 'message-
id:@mrabarnett.plus.com': 0.16; 'passing.': 0.16;
'received:plus.net': 0.16; 'static': 0.16; 'tuple': 0.16; 'void':
0.16; 'wrote:': 0.16; 'python': 0.16; 'says': 0.17; 'to:addr
:python-list': 0.20; "i've": 0.22; 'returns': 0.22; 'code': 0.23;
'skip:p 30': 0.23; "i'd": 0.24; 'skip:- 10': 0.25; 'stuff': 0.25;
'follows:': 0.26; 'object': 0.26; 'creating': 0.27; 'else': 0.27;
'bit': 0.27; 'function': 0.27; '>>>': 0.28; 'expect': 0.28;
'sense': 0.28; 'this?': 0.29; 'header:User-Agent:1': 0.30;
'code,': 0.31; 'program': 0.31; "doesn't": 0.32; 'assume': 0.32;
'required,': 0.32; "wouldn't": 0.32; 'received:192.168.1': 0.32;
'but': 0.32; "i'm": 0.33; "i'll": 0.33; 'there': 0.33; '0);':
0.33; 'same': 0.34; 'appreciated.': 0.34; 'header:In-Reply-To:1':
0.34; 'running': 0.34; 'trying': 0.35; 'handling': 0.35; 'runs':
0.35; 'yes,': 0.35; 'following': 0.35; 'fix': 0.36; 'errors':
0.36; 'change': 0.36; 'currently': 0.37; 'really': 0.37; 'using':
0.37; "it's": 0.37; 'skip:_ 30': 0.37; 'received:192.168': 0.37;
'could': 0.38; 'setting': 0.39; 'valid': 0.39; 'use': 0.39;
'8bit%:71': 0.40; 'something': 0.40; '8bit%:32': 0.60; 'url-
ip:104.21/16': 0.61; 'here.': 0.61; "there's": 0.61; 'me.': 0.62;
'to:': 0.62; 'skip:\xc2 10': 0.62; 'format': 0.62; 'point.': 0.62;
'received:212': 0.62; 'showing': 0.62; 'skip:z 10': 0.62; 'here':
0.62; 'skip:o 20': 0.63; 'destroy': 0.63; '8bit%:9': 0.64; 'skip:r
20': 0.64; 'full': 0.64; 'representing': 0.64; 'skip:v 30': 0.64;
'parts': 0.65; 'required': 0.65; 'skip:t 30': 0.67; 'further':
0.69; '8bit%:63': 0.69; 'times': 0.69; '2016': 0.70; 'skip:f 30':
0.71; 'attention': 0.71; '8bit%:31': 0.73; '8bit%:100': 0.76;
'limits': 0.76; '8bit%:67': 0.81; 'reference.': 0.81; 'returned':
0.81; '1);': 0.84; 'break;': 0.84; 'pointer': 0.84; 'sure.': 0.84;
'surprised': 0.84; "that'll": 0.84; 'type.': 0.84; 'val': 0.84;
'weird': 0.84; '\xc2\xa0\xc2\xa0\xc2\xa0\xc2\xa0\xc2\xa0': 0.84;
'caused': 0.86; 'stable': 0.93
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=plus.com; s=042019;
t=1637688683; bh=viaMHx5JF9BX6DzBT7ju/OwlswNroLI7Zq8Pcr83Yq8=;
h=Date:Subject:To:References:From:In-Reply-To;
b=bJbHH/MXz6CyAtnilFkcBje+RWDFrUq5XS3ojKzMWXULT7Cz7POW6SMMFfjx0y+R3
3eNboQi0zurk00atC/YeNimfXzsBU3HNxR3kqFTDOBzOs/QbDQAyv0+YDSn5Y3HG8P
NA+Pe4wVxTGP6a7zU+z6II62rHZVPjtNSt4GEiqQSVaA3C81+QPZfqmlhtdMv/jR4D
nn48v9oSv4Fm4DoXsZWhQVTpvBbg8s+xI/qyBj3tZ22FH4aJ1g+mnhYoN7v10Kb8h0
Zu7IyNVafbJ4XgUauzSzQ7/AzJu5Pm0HA/EAgcR6Pd9o11YPkXnc3kw2jVC4jQdpK1
mYbKciYLoTf3Q==
X-Clacks-Overhead: "GNU Terry Pratchett"
X-CM-Score: 0.00
X-CNFS-Analysis: v=2.4 cv=Ev+xEQQA c=1 sm=1 tr=0 ts=619d256b
a=0nF1XD0wxitMEM03M9B4ZQ==:117 a=0nF1XD0wxitMEM03M9B4ZQ==:17
a=IkcTkHD0fZMA:10 a=NEAV23lmAAAA:8 a=I-tyrQj3AAAA:8 a=mDteEoc956f1kUHLONMA:9
a=QEXdDO2ut3YA:10 a=Dxv1P9M6eLVtfm6HFeJx:22
X-AUTH: mrabarnett@:2500
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
Thunderbird/91.3.2
Content-Language: en-GB
In-Reply-To: <8788bcb1-5e02-b241-3533-43c40d363906@sphaero.org>
X-CMAE-Envelope: MS4xfGBnN7QDXnyOXi9NwfOSQyTs0Yt455sOSmXdX97KQsGknDeSCFbdug9w39T84yMfCao0Zpv1o/Q3iARNGN909yXNRhaoqzCqBZX/z3AVefh69OQzcOBF
pUG825jJ+N8uFMODKmIJTL2SmWzBps1KF6aQq84/NiDL7aZXSJChq9s492ijzgkcQLWu/gQEuu3jy+3abTrv+gVEg6XY9YoXT1s=
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.37
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <1d070d88-8b19-c1ea-c700-4a47876fe871@mrabarnett.plus.com>
X-Mailman-Original-References: <d0b57569-ef0d-3ab4-9918-9cd498f1e658@sphaero.org>
<4eb693c6-fd5f-7c54-246d-0b081a0488d5@mrabarnett.plus.com>
<c827faf9-0871-fde3-83a0-fa19576b3c62@sphaero.org>
<d6c7a17a-b24c-9a45-a94e-7bbc3d8fb1c1@mrabarnett.plus.com>
<f7711e63-a9a7-eb7a-b2b1-0c7ebd9829af@mrabarnett.plus.com>
<8788bcb1-5e02-b241-3533-43c40d363906@sphaero.org>
 by: MRAB - Tue, 23 Nov 2021 17:31 UTC

On 2021-11-23 16:04, Arnaud Loonstra wrote:
> On 23-11-2021 16:37, MRAB wrote:
>> On 2021-11-23 15:17, MRAB wrote:
>>> On 2021-11-23 14:44, Arnaud Loonstra wrote:
>>>> On 23-11-2021 15:34, MRAB wrote:
>>>>> On 2021-11-23 12:07, Arnaud Loonstra wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I've got Python embedded successfully in a program up until now as I'm
>>>>>> now running into weird GC related segfaults. I'm currently trying to
>>>>>> debug this but my understanding of CPython limits me here.
>>>>>>
>>>>>> I'm creating a Tuple in C but it crashes on creating it after a while.
>>>>>> It doesn't make sense which makes me wonder something else must be
>>>>>> happening? Could be it just crashes here because the GC is cleaning up
>>>>>> stuff completely unrelated to the allocation of the new tuple? How
>>>>>> can I
>>>>>> troubleshoot this?
>>>>>>
>>>>>> I've got CPython compiled with  --with-valgrind --without-pymalloc
>>>>>> --with-pydebug
>>>>>>
>>>>>> In C I'm creating a tuple with the following method:
>>>>>>
>>>>>> static PyObject *
>>>>>> s_py_zosc_tuple(pythonactor_t *self, zosc_t *oscmsg)
>>>>>> {
>>>>>>       assert(self);
>>>>>>       assert(oscmsg);
>>>>>>       char *format = zosc_format(oscmsg);
>>>>>>
>>>>>>       PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
>>>>>>
>>>>>> It segfaults here (frame 16) after 320 times (consistently)
>>>>>>
>>>>>>
>>>>>> 1   __GI_raise             raise.c          49   0x7ffff72c4e71
>>>>>> 2   __GI_abort             abort.c          79   0x7ffff72ae536
>>>>>> 3   fatal_error            pylifecycle.c    2183 0x7ffff7d84b4f
>>>>>> 4   Py_FatalError          pylifecycle.c    2193 0x7ffff7d878b2
>>>>>> 5   _PyObject_AssertFailed object.c         2200 0x7ffff7c93cf2
>>>>>> 6   visit_decref           gcmodule.c       378  0x7ffff7dadfd5
>>>>>> 7   tupletraverse          tupleobject.c    623  0x7ffff7ca3e81
>>>>>> 8   subtract_refs          gcmodule.c       406  0x7ffff7dad340
>>>>>> 9   collect                gcmodule.c       1054 0x7ffff7dae838
>>>>>> 10  collect_with_callback  gcmodule.c       1240 0x7ffff7daf17b
>>>>>> 11  collect_generations    gcmodule.c       1262 0x7ffff7daf3f6
>>>>>> 12  _PyObject_GC_Alloc     gcmodule.c       1977 0x7ffff7daf4f2
>>>>>> 13  _PyObject_GC_Malloc    gcmodule.c       1987 0x7ffff7dafebc
>>>>>> 14  _PyObject_GC_NewVar    gcmodule.c       2016 0x7ffff7daffa5
>>>>>> 15  PyTuple_New            tupleobject.c    118  0x7ffff7ca4da7
>>>>>> 16  s_py_zosc_tuple        pythonactor.c    366  0x55555568cc82
>>>>>> 17  pythonactor_socket     pythonactor.c    664  0x55555568dac7
>>>>>> 18  pythonactor_handle_msg pythonactor.c    862  0x55555568e472
>>>>>> 19  pythonactor_handler    pythonactor.c    828  0x55555568e2e2
>>>>>> 20  sphactor_actor_run     sphactor_actor.c 855  0x5555558cb268
>>>>>> ... <More>
>>>>>>
>>>>>> Any pointer really appreciated.
>>>>>>
>>>>> You're creating a tuple that'll have the same number of members as
>>>>> the length of a string? That looks strange to me.
>>>>>
>>>>> How are you setting the tuple's members?
>>>>
>>>> It's from a serialisation format called OSC. The string describes the
>>>> type of bytes, every character is a type.
>>>>
>>>> I'm creating the tuple as follows:
>>>>
>>>> PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
>>>>
>>>> Then I iterate the OSC message using the format string, (just showing
>>>> handling an int (i))
>>>>
>>>>       char type = '0';
>>>>       Py_ssize_t pos = 0;
>>>>       const void *data =  zosc_first(oscmsg, &type);
>>>>       while(data)
>>>>       {
>>>>           switch (type)
>>>>           {
>>>>           case('i'):
>>>>           {
>>>>               int32_t val = 9;
>>>>               int rc = zosc_pop_int32(oscmsg, &val);
>>>>               assert(rc == 0);
>>>>               PyObject *o = PyLong_FromLong((long)val);
>>>>               assert( o );
>>>>               rc = PyTuple_SetItem(rettuple, pos, o);
>>>>               assert(rc == 0);
>>>>               break;
>>>>           }
>>>>
>>>> Full code is here:
>>>>
>>>> https://github.com/hku-ect/gazebosc/blob/822452dfa27201db274d37ce09e835d98fe500b2/Actors/pythonactor.c#L360
>>>>
>>>>
>>> Looking at that code, you have:
>>>
>>>       PyObject *o = Py_BuildValue("s#", str, 1);
>>>
>>> what I'd check is the type of the 1 that you're passing. Wouldn't the
>>> compiler assume that it's an int?
>>>
>>> The format string tells the function to expect a Py_ssize_t, but how
>>> would the compiler know that?
>>>
>> Looking at https://www.mankier.com/3/zosc, it says for 'T' and 'F' "(no
>> value required)", but you're doing:
>>
>>     int rc = zosc_pop_bool(oscmsg, &bl);
>>
>> If no value is required, is there a bool there to be popped?
>
> The value is only required to set a user provided boolean to the value
> in the message. There's no boolean value encoded in the message, only
> the T and F in the format string.
>
> With regards to the Py_BuildValue("s#", str, 1);, that's a valid point.
> I'll fix that. However in the segfaults I'm testing that code is not
> touched.

You can use "C" as a format string for Py_BuildValue to convert a C int
representing a character to a Python string.

> I'm now testing different parts of the code to see if it runs stable.
> I've found it runs stable if I do not process the returned tuple.
>
> PyObject *pReturn = PyObject_CallMethod(self->pyinstance,
> "handleSocket", "sOsss",
> oscaddress,
> py_osctuple,
> ev->type, ev->name, strdup(ev->uuid));
> Py_XINCREF(pReturn);
>
Why the Py_XINCREF? PyObject_CallMethod returns a new reference. The
Py_DECREF that you do later won't destroy the object because of that
additional Py_XINCREF, so that's a memory leak.

> https://github.com/hku-ect/gazebosc/blob/505b30c46bf3f78d188c3f575c80e294d3db7e5d/Actors/pythonactor.c#L673
>
> and a bit further in the code I convert the Python tuple to an OSC message:
>
> zosc_t *retosc = s_py_zosc(pAddress, pData);
>
> https://github.com/hku-ect/gazebosc/blob/505b30c46bf3f78d188c3f575c80e294d3db7e5d/Actors/pythonactor.c#L732
>
> If I change that line to:
>
> zosc_t *retosc = zosc_create("/temp", "ii", 32, 64);
>
> It runs stable.
>
> I would turn my attention to s_py_zosc function but I'm not sure. Since
> the errors are GC related it could caused anywhere?
>
Basically, yes, but I won't be surprised if it was due to too few
INCREFs or too many DECREFs somewhere.

> https://github.com/hku-ect/gazebosc/blob/505b30c46bf3f78d188c3f575c80e294d3db7e5d/Actors/pythonactor.c#L286
>
Incidentally, in s_py_zosc_tuple, you're not doing "assert(rc == 0);"
after "after zosc_pop_float" or "zosc_pop_double".


Click here to read the complete article

devel / comp.lang.python / Re: Embedding Python crash on PyTuple_New

1
server_pubkey.txt

rocksolid light 0.9.81
clearnet tor