Rocksolid Light

Welcome to RetroBBS

mail  files  register  newsreader  groups  login

Message-ID:  

The clothes have no emperor. -- C. A. R. Hoare, commenting on ADA.

devel / comp.lang.python / Re: The GIL and PyEval_RestoreThread

SubjectAuthor
o Re: The GIL and PyEval_RestoreThreadPeter Ebden

1
Re: The GIL and PyEval_RestoreThread

<mailman.351.1695806100.23016.python-list@python.org>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=28469&group=comp.lang.python#28469

  copy link   Newsgroups: comp.lang.python
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!fu-berlin.de!uni-berlin.de!not-for-mail
From: peter@thoughtmachine.net (Peter Ebden)
Newsgroups: comp.lang.python
Subject: Re: The GIL and PyEval_RestoreThread
Date: Wed, 27 Sep 2023 10:14:45 +0100
Lines: 106
Message-ID: <mailman.351.1695806100.23016.python-list@python.org>
References: <CAJANk5Ezbnadnp8Kg-yafCB3SOfrrgHwbsPBqEkzDrArWrOyEQ@mail.gmail.com>
<620973cc-7e7f-7f30-3f34-56cb05c5084b@mrabarnett.plus.com>
<CAJANk5FqamBC+5spH2w=xMc=ncBqXs-8dyPSH9gkXSr2bH75Aw@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Trace: news.uni-berlin.de kLCi43bbADdX9sR09FibIAnXAM8jpnDnOZPJSTozn8qw==
Cancel-Lock: sha1:2cu9IQE/ELMgdD48JlpjsFN69qI= sha256:spr4sxNJ9fJboQSYNxTcQ/SnPR/7w7Jw9Jh6YIOh8iM=
Return-Path: <pebers@thoughtmachine.net>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=thoughtmachine.net header.i=@thoughtmachine.net
header.b=sT2d+gXT; dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.013
X-Spam-Evidence: '*H*': 0.97; '*S*': 0.00; 'looks': 0.02; '(which':
0.04; 'parallel': 0.05; 'thread': 0.05; 'variable': 0.05; '2023':
0.07; 'else.': 0.07; 'that?': 0.07; '1:1': 0.09; '27,': 0.09;
'acquired': 0.09; 'cc:addr:python-list': 0.09; 'occasionally':
0.09; 'originally': 0.09; 'regex': 0.09; 'threads': 0.09; '&gt;':
0.14; 'cc:no real name:2**0': 0.14; 'url:mailman': 0.15;
'&quot;if': 0.16; "(i'm": 0.16; 'call,': 0.16; 'case).': 0.16;
'discarding': 0.16; 'gil': 0.16; 'subject:GIL': 0.16; 'super':
0.16; 'though?': 0.16; 'usable': 0.16; 'void': 0.16; 'wales.':
0.16; 'wrote:': 0.16; 'python': 0.16; 'says': 0.17;
'cc:addr:python.org': 0.20; 'all,': 0.20; 'machine': 0.22;
'written': 0.22; "i've": 0.22; 'maybe': 0.22; 'code': 0.23;
'skip:p 30': 0.23; "i'd": 0.24; 'url-ip:188.166.95.178/32': 0.25;
'url-ip:188.166.95/24': 0.25; 'url:listinfo': 0.25; 'cc:2**0':
0.25; 'url-ip:188.166/16': 0.25; "wasn't": 0.26; "isn't": 0.27;
'local': 0.27; 'expect': 0.28; 'output': 0.28; 'email
addr:python.org&gt;': 0.28; 'wrong': 0.28; 'it,': 0.29; 'seem':
0.31; 'saved': 0.31; 'url-ip:188/8': 0.31; 'think': 0.32;
'consent': 0.32; 'guess': 0.32; 'python-list': 0.32; 'sep': 0.32;
'specified': 0.32; 'square,': 0.32; 'message-id:@mail.gmail.com':
0.32; 'but': 0.32; "i'm": 0.33; 'someone': 0.34; 'release': 0.34;
'able': 0.34; 'same': 0.34; "didn't": 0.34; 'header:In-Reply-
To:1': 0.34; 'received:google.com': 0.34; 'running': 0.34; '"if':
0.35; 'particularly': 0.35; 'track': 0.35; 'recipient': 0.36;
'possibly': 0.36; 'source': 0.36; "it's": 0.37; 'read': 0.38;
'two': 0.39; 'quite': 0.39; 'otherwise': 0.39; 'use': 0.39;
'england': 0.39; 'wed,': 0.39; 'explain': 0.40; 'happen': 0.40;
'url:maps': 0.40; 'both': 0.40; 'want': 0.40; 'try': 0.40;
'should': 0.40; 'here.': 0.61; 'url-ip:142.250.179/24': 0.61;
'url:amp': 0.62; 'me.': 0.62; 'follow': 0.62; 'email': 0.63;
'share': 0.63; 'pass': 0.64; 'url:gmail': 0.64; 'skip:r 20': 0.64;
'lock': 0.64; 'company': 0.64; 'well': 0.65; '&amp;': 0.65;
'skip:t 20': 0.66; 'skip:1 20': 0.67; 'url:q': 0.67; 'back': 0.67;
'number:': 0.67; 'url:5': 0.67; 'time,': 0.67; 'only.': 0.69;
'result,': 0.69; 'subject:The': 0.70; 'claim': 0.71;
'interesting': 0.71; 'street': 0.71; 'content': 0.72; 'little':
0.73; 'produces': 0.76; 'reply': 0.77; 'states': 0.80; 'acquire':
0.84; 'email name:&lt;python-list': 0.84; 'happen;': 0.84;
'locking': 0.84; 'released.': 0.84; 'restored': 0.84; 'restoring':
0.84; 'url:new': 0.84; 'url:street': 0.84; 'url:source': 0.89;
'sleeping': 0.91; 'locked': 0.93
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=thoughtmachine.net; s=google; t=1695806096; x=1696410896; darn=python.org;
h=cc:to:subject:message-id:date:from:in-reply-to:references
:mime-version:from:to:cc:subject:date:message-id:reply-to;
bh=6vg5x82MO3KGuDWWcduXsyfkoHlLh8JRNS2Ky7+ucGI=;
b=sT2d+gXTFUVljku7/NKUM39V05VFFFIlavd3cEKb6nOdFfLJSBlm5M+WoiKtSmFF9k
siJj4jf3wuVNmSWzzzZY7Bn+SVpMuL18VD/2peWhQ42MxZqyXcs7o1ku2yxcuPK6N7o7
XvjCYJXLnigQCWSfDpcX2nvm2/jq2H+Jin+WuFxr5cKR3kJt0pMC9dOwbPxDcqwaFnj/
PC7/bvTVHwJ4Wdzzull3rq7R/UWUPf1wlFu3Qa82Xm+CR347xohLQ4+9V6sMGrJuVfXn
ua8xljOnic4FH85pVE4vNS+FLBuAAJOD5PaosJfQWCshSoayNgEWXQo8IWxPAH6WqGQE
Zf3w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20230601; t=1695806096; x=1696410896;
h=cc:to:subject:message-id:date:from:in-reply-to:references
:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id
:reply-to;
bh=6vg5x82MO3KGuDWWcduXsyfkoHlLh8JRNS2Ky7+ucGI=;
b=S1/TUN4laK343Yl6xjX0iIXM1EHDuSvOATsRbenAvL7rqXD7BHsHOtxovVffUs0Ymh
1VUNLVexLkhCGYbmSKyY2w8V4W8RlDoJKVQuxol851SP8npr9I8SNE7loSu/wElDB2lV
qXEW5b9tZMS/Y1MuuKX0CrtP3ypCoh/t6vhSyxISFeALu5WNtNrhmyHwhOSdqx8PoEX8
XtYpXAmFc5wQasNcioeesEBPfZ0Y8Bepwu1rYxRiuYoKL7h6qEdztf1CfGDVKd9AwxyR
LKUKzaUYvsfftN31pvBL8GkE9JaX+WP/XGu65cmobPv99QAZ2UpaVYJBJUZk+SZ7HzNk
n5dA==
X-Gm-Message-State: AOJu0Yw1Wul8MP4b/ycAoTxz4oA9NbLdbsgRHMzBlfVb0pLE80LRii0n
5jYIEUy+kW+QR43iQckN/1U1IF8wAChkEWQjzLqoJuGrf5JYrqda7grv1OpsaHalixN7S+DYbIz
ky+IODpdGq+MqGMkIn5QIN82QT4ZvClqB08XFAZM=
X-Google-Smtp-Source: AGHT+IE5zIIXOpBZwN3r5MNjtnGKKoSm2QTRVFxqiAHXCu2GAyBGNmlZ+JVz7BIUKbGlfve/rYSu1P/y26eU9tXY/Kc=
X-Received: by 2002:a4a:918b:0:b0:576:9e0b:ca09 with SMTP id
d11-20020a4a918b000000b005769e0bca09mr1455388ooh.0.1695806096557; Wed, 27 Sep
2023 02:14:56 -0700 (PDT)
In-Reply-To: <620973cc-7e7f-7f30-3f34-56cb05c5084b@mrabarnett.plus.com>
X-Content-Filtered-By: Mailman/MimeDel 2.1.39
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <CAJANk5FqamBC+5spH2w=xMc=ncBqXs-8dyPSH9gkXSr2bH75Aw@mail.gmail.com>
X-Mailman-Original-References: <CAJANk5Ezbnadnp8Kg-yafCB3SOfrrgHwbsPBqEkzDrArWrOyEQ@mail.gmail.com>
<620973cc-7e7f-7f30-3f34-56cb05c5084b@mrabarnett.plus.com>
 by: Peter Ebden - Wed, 27 Sep 2023 09:14 UTC

The thread variable I'm passing in is the one I originally got from calling
Py_NewInterpreter. I'd assumed that I didn't need to particularly track the
one I get back from SaveThread since it should always be the one I restored
previously (which does seem to be the case).

> It looks like you're resuming the same thread twice. As it's already
resumed the second time, no wonder it's not blocking!

That isn't how I read the docs though? It says "If the lock has been
created, the current thread must not have acquired it, otherwise deadlock
ensues." That suggests to me that it should try to acquire the GIL again
and wait until it can (although possibly also that it's not an expected use
and Python thread states are expected to be more 1:1 with C threads).

On Wed, Sep 27, 2023 at 3:53 AM MRAB via Python-list <python-list@python.org>
wrote:

> On 2023-09-26 14:20, Peter Ebden via Python-list wrote:
> > Hi all,
> >
> > I've been working on embedding Python and have an interesting case around
> > locking with PyEval_RestoreThread which wasn't quite doing what I expect,
> > hoping someone can explain what I should expect here.
> >
> > I have a little example (I'm running this in parallel from two different
> > threads; I have some more C code for that but I don't think it's super
> > interesting):
> >
> > void run_python(PyThreadState* thread) {
> > LOG("Restoring thread %p...", thread);
> > PyEval_RestoreThread(thread);
> > LOG("Restored thread %p", thread);
> > PyRun_SimpleString("import time; print('sleeping'); time.sleep(3.0)");
> > LOG("Saving thread...");
> > PyThreadState* saved_thread = PyEval_SaveThread();
> > LOG("Saved thread %p", saved_thread);
> > }
> >
> > This produces output like
> > 11:46:48.110058893: Restoring thread 0xabc480...
> > 11:46:48.110121656: Restored thread 0xabc480
> > 11:46:48.110166060: Restoring thread 0xabc480...
> > sleeping
> > 11:46:48.110464194: Restored thread 0xabc480
> > sleeping
> > 11:46:51.111307541: Saving thread...
> > 11:46:51.111361075: Saved thread 0xabc480
> > 11:46:51.113116633: Saving thread...
> > 11:46:51.113177605: Saved thread 0xabc480
> >
> > The thing that surprises me is that both threads seem to be able to pass
> > PyEval_RestoreThread before either reaches the corresponding
> > PyEval_SaveThread call, which I wasn't expecting to happen; I assumed
> that
> > since RestoreThread acquires the GIL, that thread state would remain
> locked
> > until it's released.
> >
> > I understand that the system occasionally switches threads, which I guess
> > might well happen with that time.sleep() call, but I wasn't expecting the
> > same thread to become usable somewhere else. Maybe I am just confusing
> > things by approaching the same Python thread from multiple OS threads
> > concurrently and should be managing my own locking around that?
> >
> Storing the result of PyEval_SaveThread in a local variable looks wrong
> to me.
>
> In the source for the regex module, I release the GIL with
> PyEval_SaveThread and save its result. Then, when I want to claim the
> GIL, I pass that saved value to PyEval_RestoreThread.
>
> You seem to be releasing the GIL and discarding the result, so which
> thread are you resuming when you call PyEval_RestoreThread?
>
> It looks like you're resuming the same thread twice. As it's already
> resumed the second time, no wonder it's not blocking!
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

--
Thought Machine Group Limited, a company registered in England & Wales.
Registered number: 11114277. 
Registered Office: 5 New Street Square,
London EC4A 3TW
<https://maps.google.com/?q=5+New+Street+Square,+London+EC4A+3TW&entry=gmail&source=g>.

The content of this email is confidential and intended for the recipient
specified in message only. It is strictly forbidden to share any part of
this message with any third party, without a written consent of the sender.
If you received this message by mistake, please reply to this message and
follow with its deletion, so that we can ensure such a mistake does not
occur in the future.

devel / comp.lang.python / Re: The GIL and PyEval_RestoreThread

1
server_pubkey.txt

rocksolid light 0.9.81
clearnet tor