Message-ID:

It's easy to get on the internet and forget you have a life -- Topic on #LinuxGER

devel / comp.lang.python / Re: Using a background thread with asyncio/futures with flask

Re: Using a background thread with asyncio/futures with flask

<mailman.19.1711200353.3468.python-list@python.org>

https://www.rocksolidbbs.com/devel/article-flat.php?id=29728&group=comp.lang.python#29728

copy link Newsgroups: comp.lang.python

Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: frank@chagford.com (Frank Millman)
Newsgroups: comp.lang.python
Subject: Re: Using a background thread with asyncio/futures with flask
Date: Sat, 23 Mar 2024 15:25:28 +0200
Lines: 118
Message-ID: <mailman.19.1711200353.3468.python-list@python.org>
References: <8a7231df-c30c-4fa6-9daa-546b3453b547@thomasnyberg.com>
<09efe843-195e-48c0-a9cb-6fe05e3a6921@solute.de>
<dcdef19d-cca1-4ce3-af8d-ae79abf79ec8@thomasnyberg.com>
<1322c7e8-2153-400d-afc5-7f2993031c08@chagford.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Trace: news.uni-berlin.de CTBQcfEXdE/m44XZkGly8AAkciLr1VM4xJ+uovIgO11A==
Cancel-Lock: sha1:LMkA/Dl3GXVv1NQw/WJaz/BwCbs= sha256:bvuIStMNy/waLtB784d/dAKL8p7btd5Kxe2Db/gMFgk=
Return-Path: <frank@chagford.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=chagford.com header.i=@chagford.com header.b=jyhL40kV;
dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.008
X-Spam-Evidence: '*H*': 0.98; '*S*': 0.00; 'def': 0.04; 'loop': 0.07;
'queue': 0.07; 'import': 0.15; '(after': 0.16; '(when': 0.16;
'exhibits': 0.16; 'from:addr:chagford.com': 0.16;
'from:addr:frank': 0.16; 'from:name:frank millman': 0.16;
'message-id:@chagford.com': 0.16; 'received:196.35': 0.16;
'received:196.35.198': 0.16; 'received:197.90': 0.16;
'received:197.90.32': 0.16; 'received:197.90.32.26': 0.16;
'received:synaq.com': 0.16; 'subject:thread': 0.16; 'thread.':
0.16; 'threading': 0.16; 'wrote:': 0.16; 'pm,': 0.19; 'to:addr
:python-list': 0.20; 'version': 0.23; 'install': 0.23;
'installed': 0.23; 'run': 0.23; 'flask': 0.26; 'library': 0.26;
'printed': 0.28; 'header:User-Agent:1': 0.30; 'passes': 0.32;
'python-list': 0.32; 'skip:= 50': 0.32; 'subject:Using': 0.32;
'but': 0.32; 'same': 0.34; 'header:In-Reply-To:1': 0.34;
'following': 0.35; 'using': 0.37; 'put': 0.38; 'reference': 0.60;
'skip:o 10': 0.61; 'skip:\xc2 10': 0.62; 'skip:i 20': 0.62;
'here': 0.62; 'skip:o 20': 0.63; 'skip:t 40': 0.64; 'skip:t 30':
0.67; 'await': 0.69; 'received:196': 0.69; 'skip:a 40': 0.70;
'skip:f 30': 0.71; 'future': 0.72; '8bit%:100': 0.76; 'true:':
0.84
ARC-Seal: i=1; cv=none; a=rsa-sha256; d=synaq.com; s=securemail; t=1711200350;
b=fT59NkUhmMcABKEOf2SJcZ/havxP+ZX6WzWlyRIPfhwy5TDo3NUNurilCCs0xIVzxq8iGWKhL7
dYIQ/wotiUC4aObTzPcreaaw3dOGxAutgsVzu/LkxxsBxLFpX9uzHspiT7XHkKfyCdzYEmJD1W
VMVih0ZUef/CQViEImz8UaGV5sZ6Av63q/4WFKysfrc+DOQs009673WPJ8wRo5OQ/cJlK3ohEh
mVJlK/FX7V3j4yBifwtlneSNKZX3zUEaxR3AbckItnwM1lKdqelgCHLGFsC7zrUVw9gl1OmVGQ
0IVWcNXNKECYZQsWaX3kCW2mjDyvXvcpvZMjdgF2fpAc1Q==;
ARC-Authentication-Results: i=1; synaq.com;
iprev=fail smtp.remote-ip=197.90.32.26;
auth=pass (PLAIN) smtp.auth=frank@chagford.com;
arc=none
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed; d=synaq.com; s=securemail;
t=1711200350;
bh=CvzCFWrSXCuaD3LdeDzsdUyQhmOkoTLXPDGJMWN3asM=;
h=Content-Transfer-Encoding:Content-Type:In-Reply-To:From:References:To:
Subject:MIME-Version:Date:Message-ID:DKIM-Signature;
b=HLArU9jlH4A3tjCfazvsJzqnm84cuHO/FHUytOwYawEKHsQ1o46AhqZtun6ugzJZqF2aPV3DuX
vXceD5Z0EhEXEEnHudH71ilTmyACcA9/22avyqfERDbBuh6vgPeQe6SpBLvNEbK6YV6hX13QO1
AvePa2Et334pwJnZbxdyZAqvFukuVzqGNhRPnwI3kQ5a8KfVT9930qt1TlT8hqMONY4awvOqrn
tT6c4EMwkFs8RuJLGGyVCOCUl86dDszt5vlxiV0BlgBqApQ0xmew5l8rToccltjFHllejGVQ3W
BXpw/2mMiDBvT4ZZuLzHn3L06OnE1tlA2KGXOSrjSXd43A==;
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=chagford.com; s=securemail; h=In-Reply-To:From:To:Date:Message-ID;
bh=OKUrUV1m7f3Dq7UEcNqgmpO7pZIsv1R+oPQmL4Qzl6c=; b=jyhL40kVcHNdcoYSDerB0Tmwa3
tus937lkD2vHF7HBY+v4pgo2TfAzCBcInUmm1EVknw00zU0sO+l7bvHztkKBIokbdAfSaqYA9+ZJi
aTlNdc2NVjucXmTQl5URl9j5RR8QyFOXW/DyOEo0eK6KnvEUlZrZ5gNloDW0lf5UiURhOyKiezo6d
AW1ZE/IbjQVEy7JdbeNdY3jdRui56J/AjAlpeFxsUa22EUFuxWUj3eM0ItlAyOvALbRXVzyO8KiQK
jzybn5fzbPQS3VmEScvWra/TpU5fj+/7H5agztZPKibyy+lxUMoXHa94uYcs0uyohfhQRrrZ1NAQv
NYLQmhHg==;
Authentication-Results: synaq.com; iprev=fail smtp.remote-ip=197.90.32.26;
auth=pass (PLAIN) smtp.auth=frank@chagford.com;
arc=none
User-Agent: Mozilla Thunderbird
Content-Language: en-US
In-Reply-To: <dcdef19d-cca1-4ce3-af8d-ae79abf79ec8@thomasnyberg.com>
X-Red-Router: yes
X-SYNAQ-Pinpoint-Information: Please contact SYNAQ for more information
X-SYNAQ-Pinpoint-ID: 1ro1NX-0000I0-6V
X-SYNAQ-Pinpoint: No virus infections found
X-Pinpoint-From: frank@chagford.com
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <1322c7e8-2153-400d-afc5-7f2993031c08@chagford.com>
X-Mailman-Original-References: <8a7231df-c30c-4fa6-9daa-546b3453b547@thomasnyberg.com>
<09efe843-195e-48c0-a9cb-6fe05e3a6921@solute.de>
<dcdef19d-cca1-4ce3-af8d-ae79abf79ec8@thomasnyberg.com>

by: Frank Millman - Sat, 23 Mar 2024 13:25 UTC

On 2024-03-22 12:08 PM, Thomas Nyberg via Python-list wrote:
> Hi,
>
> Yeah so flask does support async (when installed with `pip3 install
> flask[async]), but you are making a good point that flask in this case
> is a distraction. Here's an example using just the standard library that
> exhibits the same issue:
>
> `app.py`
> ```
> import asyncio
> import threading
> import time
> from queue import Queue
>
>
> in_queue = Queue()
> out_queue = Queue()
>
>
> def worker():
>     print("worker started running")
>     while True:
>         future = in_queue.get()
>         print(f"worker got future: {future}")
>         time.sleep(5)
>         print("worker sleeped")
>         out_queue.put(future)
>
>
> def finalizer():
>     print("finalizer started running")
>     while True:
>         future = out_queue.get()
>         print(f"finalizer got future: {future}")
>         future.set_result("completed")
>         print("finalizer set result")
>
>
> threading.Thread(target=worker).start()
> threading.Thread(target=finalizer).start()
>
>
> async def main():
>     future = asyncio.get_event_loop().create_future()
>     in_queue.put(future)
>     print(f"main put future: {future}")
>     result = await future
>     print(result)
>
>
> if __name__ == "__main__":
>     loop = asyncio.get_event_loop()
>     loop.run_until_complete(main())
> ```
>
> If I run that I see the following printed out (after which is just hangs):
>
> ```

Combining Dieter's and Mark's ideas, here is a version that works.

It is not pretty! call_soon_threadsafe() is a loop function, but the
loop is not accessible from a different thread. Therefore I include a
reference to the loop in the message passed to in_queue, which in turn
passes it to out_queue.

Frank

=======================================================

import asyncio
import threading
import time
from queue import Queue

in_queue = Queue()
out_queue = Queue()

def worker():
print("worker started running")
while True:
loop, future = in_queue.get()
print(f"worker got future: {future}")
time.sleep(5)
print("worker sleeped")
out_queue.put((loop, future))

def finalizer():
print("finalizer started running")
while True:
loop, future = out_queue.get()
print(f"finalizer got future: {future}")
loop.call_soon_threadsafe(future.set_result, "completed")
print("finalizer set result")

threading.Thread(target=worker, daemon=True).start()
threading.Thread(target=finalizer, daemon=True).start()

async def main():
loop = asyncio.get_event_loop()
future = loop.create_future()
in_queue.put((loop, future))
print(f"main put future: {future}")
result = await future
print(result)

if __name__ == "__main__":
# loop = asyncio.get_event_loop()
# loop.run_until_complete(main())
asyncio.run(main())

Subject	Author
Re: Using a background thread with asyncio/futures with flask	Frank Millman