Message-ID:

The universe seems neither benign nor hostile, merely indifferent. -- Sagan

devel / comp.lang.python / Re: Using a background thread with asyncio/futures with flask

Re: Using a background thread with asyncio/futures with flask

<mailman.11.1711106969.3468.python-list@python.org>

https://www.rocksolidbbs.com/devel/article-flat.php?id=29719&group=comp.lang.python#29719

copy link Newsgroups: comp.lang.python

Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: frank@chagford.com (Frank Millman)
Newsgroups: comp.lang.python
Subject: Re: Using a background thread with asyncio/futures with flask
Date: Fri, 22 Mar 2024 13:28:56 +0200
Lines: 70
Message-ID: <mailman.11.1711106969.3468.python-list@python.org>
References: <8a7231df-c30c-4fa6-9daa-546b3453b547@thomasnyberg.com>
<7af73918-8209-4393-8aa2-140940567456@chagford.com>
<e7140924-fcfd-4816-b905-8e950ab2dfdd@chagford.com>
<c114bcae-8c8c-4b97-bade-7ba4ae35dcf1@chagford.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Trace: news.uni-berlin.de JynxyTCZtVh2rogruGrkXgTNzMAvaRKS9phBrCwKtxcA==
Cancel-Lock: sha1:8PdhXRUW6Mo8fHj4Hw+2W1hWAPA= sha256:nRQdPYwiAJyHsejKoj6RJOnAVPsAScFb8AWDvBkL4o4=
Return-Path: <frank@chagford.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=chagford.com header.i=@chagford.com header.b=ZCPca52s;
dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.009
X-Spam-Evidence: '*H*': 0.98; '*S*': 0.00; 'def': 0.04; 'loop': 0.07;
'blocking': 0.09; 'instances': 0.09; 'threads': 0.09; 'import':
0.15; 'app,': 0.16; 'expert.': 0.16; 'from:addr:chagford.com':
0.16; 'from:addr:frank': 0.16; 'from:name:frank millman': 0.16;
'functions.': 0.16; 'message-id:@chagford.com': 0.16; 'millman':
0.16; 'received:196.35': 0.16; 'received:196.35.198': 0.16;
'received:197.90': 0.16; 'received:197.90.32': 0.16;
'received:197.90.32.26': 0.16; 'received:synaq.com': 0.16;
'subject:thread': 0.16; 'tasks.': 0.16; 'wrote:': 0.16; 'problem':
0.16; 'pm,': 0.19; 'to:addr:python-list': 0.20; 'version': 0.23;
'task': 0.26; 'bit': 0.27; 'background': 0.28; 'header:User-
Agent:1': 0.30; 'think': 0.32; 'python-list': 0.32;
'subject:Using': 0.32; 'but': 0.32; 'work.': 0.34; 'header:In-
Reply-To:1': 0.34; 'change': 0.36; 'way': 0.38; 'put': 0.38;
'use': 0.39; 'still': 0.40; 'case.': 0.40; 'something': 0.40;
'skip:o 10': 0.61; 'skip:\xc2 10': 0.62; 'skip:i 20': 0.62;
'point.': 0.62; 'here': 0.62; 'skip:o 20': 0.63; 'your': 0.64;
'similar': 0.65; 'back': 0.67; 'await': 0.69; 'received:196':
0.69; 'skip:a 40': 0.70; '8bit%:6': 0.71; 'skip:f 30': 0.71;
'future': 0.72; '8bit%:100': 0.76; 'true:': 0.84
ARC-Seal: i=1; cv=none; a=rsa-sha256; d=synaq.com; s=securemail; t=1711106968;
b=v4vRlsNHOsJhR58LUucdH8t17QiqBqp7Oz9fQmy7NbGbVh40yBylArs0OwoRgSjIh3o5SaHryJ
W8HOaH4yUV/2C7SGLHTm+PkuUGW99hGa691z+5CArSyAsw7JOZc53N2N+HBZMkDtoaRdUBhdpA
LoPgkPmOYuncB6lNw69B9J+ay6YbV6qbGNRcIn1ANIJeJE59kZEv0lxD7qjihAdiq9ZZV59sp1
cES/bBsweFTldz0gLkzrT10ISbdAabxj+yxgI5EdV5AanZWD+djjMK7FcOAi+67T+0/T+GWvQo
9H2+0lX/+jAilZcYVxegrQSqm1KeQqBd67MZwIssjh8EEw==;
ARC-Authentication-Results: i=1; synaq.com;
iprev=fail smtp.remote-ip=197.90.32.26;
auth=pass (PLAIN) smtp.auth=frank@chagford.com;
arc=none
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed; d=synaq.com; s=securemail;
t=1711106968;
bh=m+7nCo63W+xfSD7aJdpeuC0jwI6Z+ol0qW8vIzWoNKU=;
h=Content-Transfer-Encoding:Content-Type:In-Reply-To:From:References:To:
Subject:MIME-Version:Date:Message-ID:DKIM-Signature;
b=tCIOBOWRG5eY7+0HEi6zniimCfFcLp3JVmn3p+pvuYZtBjFfKmpBRvFX088bpatM5nry5Gifq2
RVofo2H3SZFrj/kJ8liPj6q/dsRp+rQSjsZ9TFqMWIXixXP6ZqkAn4ghZwlUyfu+YRPjS1qmER
ruW6LaWIvs75UKqABL1pi0O4oNVd2ysa6z9HTXuB+vZu7qGsJ369t5IjtYI4BGmZVkt/8Q2p2R
/GJWX+fZp1A8Kcus/IEP3DSzZAqvt0bXNEWLIx2LlOY1VAWiS/YQeSxU+Zrv/17QEaFgPrGz41
xMOaUNfTSiCehijrMnj6gSi8jl6e2zHYwOQigA+tuELrlA==;
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=chagford.com; s=securemail; h=In-Reply-To:From:To:Date:Message-ID;
bh=z9seah50KxSY2qAJZyzucf2rGBEOzDaX6BBN3RhXiJc=; b=ZCPca52s0LgfMpUY2c7KrvSyp2
pkRp5v9ab7cGIiypOlQoEYZDYSjgHvkHklgj+tNnz3XUYtHnbEZdzKGorQhOh5adiJo2pLPxHL3hO
eEE+qksUxF2yQBE+SzppaFr2qFHhEAdNVA4tXK0tmYvGVyjBPyZ6K4B0zuOTqzvz9Kg9Oob7jshuR
7qloUWxp4LbhXsgOK6YsI3FmhohH0CYlDM5+hBQaCuKA9x9Sl07YXmDdrmEfpcj3UEDyqpOMGr9wj
ceWSZwfQoZ1r3sqKld6CU4pkKb2BtyXoGgXycaeteVswg4YDZHr6zOC5HI6+kAB+STb90yB0gYHaU
9angv/Nw==;
Authentication-Results: synaq.com; iprev=fail smtp.remote-ip=197.90.32.26;
auth=pass (PLAIN) smtp.auth=frank@chagford.com;
arc=none
User-Agent: Mozilla Thunderbird
Content-Language: en-US
In-Reply-To: <e7140924-fcfd-4816-b905-8e950ab2dfdd@chagford.com>
X-Red-Router: yes
X-SYNAQ-Pinpoint-Information: Please contact SYNAQ for more information
X-SYNAQ-Pinpoint-ID: 1rnd54-0000V7-Kt
X-SYNAQ-Pinpoint: No virus infections found
X-Pinpoint-From: frank@chagford.com
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <c114bcae-8c8c-4b97-bade-7ba4ae35dcf1@chagford.com>
X-Mailman-Original-References: <8a7231df-c30c-4fa6-9daa-546b3453b547@thomasnyberg.com>
<7af73918-8209-4393-8aa2-140940567456@chagford.com>
<e7140924-fcfd-4816-b905-8e950ab2dfdd@chagford.com>

by: Frank Millman - Fri, 22 Mar 2024 11:28 UTC

On 2024-03-22 1:23 PM, Frank Millman via Python-list wrote:
> On 2024-03-22 12:09 PM, Frank Millman via Python-list wrote:
>>
>> I am no expert. However, I do have something similar in my app, and it
>> works.
>>
>> I do not use 'await future', I use 'asyncio.wait_for(future)'.
>>
>
> I tested it and it did not work.
>
> I am not sure, but I think the problem is that you have a mixture of
> blocking and non-blocking functions.
>
> Here is a version that works. However, it is a bit different, so I don't
> know if it fits your use case.
>
> I have replaced the threads with background asyncio tasks.
>
> I have replaced instances of queue.Queue with asyncio.Queue.
>
> Frank
>
> ===============================================
>
> import asyncio
>
> in_queue = asyncio.Queue()
> out_queue = asyncio.Queue()
>
> async def worker():
>     print("worker started running")
>     while True:
>         future = await in_queue.get()
>         print(f"worker got future: {future}")
>         await asyncio.sleep(5)
>         print("worker sleeped")
>         await out_queue.put(future)
>
> async def finalizer():
>     print("finalizer started running")
>     while True:
>         future = await out_queue.get()
>         print(f"finalizer got future: {future}")
>         future.set_result("completed")
>         print("finalizer set result")
>
> async def main():
>     asyncio.create_task(worker()) # start a background task
>     asyncio.create_task(finalizer()) # ditto
>     future = asyncio.get_event_loop().create_future()
>     await in_queue.put(future)
>     print(f"main put future: {future}")
>     result = await asyncio.wait_for(future, timeout=None)
>     print(result)
>
> if __name__ == "__main__":
>     # loop = asyncio.get_event_loop()
>     # loop.run_until_complete(main())
>
>     # this is the preferred way to start an asyncio app
>     asyncio.run(main())
>
>

One more point.

If I change 'await asyncio.wait_for(future, timeout=None)' back to your
original 'await future', it still works.

Subject	Author
Re: Using a background thread with asyncio/futures with flask	Frank Millman