RetroBBS - comp.lang.python - Re: Recommendations in terms of threading, multi-threading and/or asynchronous processes/programming? - Sent Mail

Re: Recommendations in terms of threading, multi-threading and/or asynchronous processes/programming? - Sent Mail - Mozilla Thunderbird

<mailman.1428.1673032786.20444.python-list@python.org>

copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=25440&group=comp.lang.python#25440

copy link Newsgroups: comp.lang.python

Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: rosuav@gmail.com (Chris Angelico)
Newsgroups: comp.lang.python
Subject: Re: Recommendations in terms of threading, multi-threading and/or
asynchronous processes/programming? - Sent Mail - Mozilla Thunderbird
Date: Sat, 7 Jan 2023 06:19:33 +1100
Lines: 45
Message-ID: <mailman.1428.1673032786.20444.python-list@python.org>
References: <599c7ea1-3dec-551c-e565-e0583f4eab9b@gmail.com>
<CAPTjJmq+JK5rLEH7caecerijgtarQs6yGDiy979cobKt-9YX7Q@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
X-Trace: news.uni-berlin.de SSXrlWsLfUXmON56WGemyA4piVcqy5kTMUmkhU7dTZrQ==
Return-Path: <rosuav@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=gmail.com header.i=@gmail.com header.b=NxPRQ3oe;
dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.060
X-Spam-Evidence: '*H*': 0.88; '*S*': 0.00; '(for': 0.05; 'thread':
0.05; 'usage': 0.05; '2023': 0.07; 'cpu': 0.07; 'loop': 0.07;
'asynchronous': 0.09; 'fewer': 0.09; 'tasks,': 0.09; 'test,':
0.09; 'threads': 0.09; 'memory': 0.15; '(especially': 0.16;
'chrisa': 0.16; 'easiest': 0.16; 'from:addr:rosuav': 0.16;
'from:name:chris angelico': 0.16; 'interface,': 0.16;
'received:209.85.218': 0.16; 'subject:programming': 0.16;
'variables,': 0.16; 'well-suited': 0.16; 'whichever': 0.16;
'wrote:': 0.16; 'python': 0.16; 'to:addr:python-list': 0.20;
'basically': 0.22; 'ran': 0.22; 'sat,': 0.22; 'run': 0.23;
'code,': 0.31; 'carefully': 0.32; 'message-id:@mail.gmail.com':
0.32; 'but': 0.32; 'able': 0.34; 'same': 0.34; 'header:In-Reply-
To:1': 0.34; 'received:google.com': 0.34; 'running': 0.34;
'trying': 0.35; 'runs': 0.35; 'from:addr:gmail.com': 0.35;
'also,': 0.36; 'count': 0.36; 'possibly': 0.36; 'processes': 0.36;
"it's": 0.37; 'received:209.85': 0.37; 'way': 0.38; 'put': 0.38;
'received:209': 0.39; 'single': 0.39; 'use': 0.39; '(with': 0.39;
'seeing': 0.39; 'define': 0.40; 'forms': 0.40; 'serious': 0.40;
'want': 0.40; 'should': 0.40; 'event': 0.40; 'higher': 0.60;
'in,': 0.60; 'limited': 0.62; 'subject': 0.63; 'great': 0.63;
'skip:m 20': 0.63; 'between': 0.63; 'share': 0.63; 'needs.': 0.64;
'your': 0.64; 'process.': 0.65; 'look': 0.65; 'entire': 0.67;
'generally': 0.67; 'right': 0.68; 'await': 0.69; 'collect': 0.69;
'states,': 0.69; 'terms': 0.70; '500': 0.70; 'above,': 0.70;
'global': 0.73; "you'll": 0.73; 'easy': 0.74; 'process,': 0.75;
'limits': 0.76; 'background,': 0.84; 'delays': 0.84;
'management.': 0.84; 'subject: \n ': 0.84; 'subject:Mail': 0.84;
'channels': 0.91; 'goals,': 0.91
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
h=to:subject:message-id:date:from:in-reply-to:references:mime-version
:from:to:cc:subject:date:message-id:reply-to;
bh=he2b1WNS6Ku2XvmCbaGj83bpIYOun0b6sm4qFjtGHPg=;
b=NxPRQ3oei4fxO/eXeeggGD5D+NDzd9tVzzBAeGVD5HBl5lQ3brSoSdVqh+GcftuOzz
VkNk3Y69mhTXR9i1V+kk9bfHX0JXk9AtMiR0rUAVLhklicqIwf8dYL0nidrIE/bqo/Vy
LOkh0/42HTp/5pMUypsmH+dFI7rFgq/XP8ksQWd9p+ZzwAGeac0EMxFm/gqdXACLGWuP
L8w+hDyY3V3mzSNsbddI7+4zJa5zG8OakiPt8sZNdDClS3Qr3BhGWT1TILCfFycAUMLZ
bsuLh9Oq9h4zt8WIfDko7aVe5SzTfTbo0Jclcl7qqUHPUNurMrOn6hWkn95XPJF68Mi/
Ji2w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20210112;
h=to:subject:message-id:date:from:in-reply-to:references:mime-version
:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;
bh=he2b1WNS6Ku2XvmCbaGj83bpIYOun0b6sm4qFjtGHPg=;
b=r4Ux355mjVMjEalkY1B36tj5mZthWyfU7HuQ6rTR4cgjilmOJXIEOyIulSR8G2z4c7
4ScTDjnV8TYKRFDKPujr0jnQkbsaJQdFaEI8H1vY1N6/2m3r+mmt2aRfxWMr44j8rsS5
5Gy4KEDHRLDqpKkY0dEGzqct+hB8dBAyATtF1JkIk06YWGak7vX6fhBM4uC05hg97ml8
t5vRn+iFqBTBTbehA9cPU36XoBWNLtMv8ZX8/V2F85dnLyg++oRs7np4m7YvnVxisPUm
oH+pBeIzsVWa5bv8bOC7GfyBNdkoLN4JpoWE0nHLD0vZ4bStkIerK8mr0bTS6RJTcnyR
jUig==
X-Gm-Message-State: AFqh2kpkpSGrgSKRe61wiOqnWWJH7XqBd1AJeE1bBQaqamb4OIQjua+W
oaOfoaycS58RCT3umzlqeO3AC4mFZjYiPuhZt8V6JoPG
X-Google-Smtp-Source: AMrXdXsHzBDUmvE2v4vTLjdF5w0en6tvrkzxWjM73cH2YIpG/OyahbZddBB4LHLNS4QY3Q2mxY5Ex/E2FDp1D7nViN8=
X-Received: by 2002:a17:906:1ec1:b0:7c1:b65:ad79 with SMTP id
m1-20020a1709061ec100b007c10b65ad79mr3834447ejj.402.1673032784443; Fri, 06
Jan 2023 11:19:44 -0800 (PST)
In-Reply-To: <599c7ea1-3dec-551c-e565-e0583f4eab9b@gmail.com>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <CAPTjJmq+JK5rLEH7caecerijgtarQs6yGDiy979cobKt-9YX7Q@mail.gmail.com>
X-Mailman-Original-References: <599c7ea1-3dec-551c-e565-e0583f4eab9b@gmail.com>

by: Chris Angelico - Fri, 6 Jan 2023 19:19 UTC

On Sat, 7 Jan 2023 at 04:54, jacob kruger <jacob.kruger.work@gmail.com> wrote:
>
> I am just trying to make up my mind with regards to what I should look
> into working with/making use of in terms of what have put in subject line?
>
>
> As in, if want to be able to trigger multiple/various threads/processes
> to run in the background, possibly monitoring their states, either via
> interface, or via global variables, but, possibly while processing other
> forms of user interaction via the normal/main process, what would be
> recommended?
>

Any. All. Whatever suits your purpose.

They all have different goals, different tradeoffs. Threads are great
for I/O bound operations; they're easy to work with (especially in
Python), behave pretty much like just having multiple things running
concurrently, and generally are the easiest to use. But you'll run
into limits as your thread count climbs (with a simple test, I started
seeing delays at about 10,000 threads, with more serious problems at
100,000), so it's not well-suited for huge scaling. Also, only one
thread at a time can run Python code, which limits them to I/O-bound
tasks like networking.

Multiple processes take a lot more management. You have to carefully
define your communication channels (for instance, a
multiprocessing.Queue() to collect results), but they can do CPU-bound
tasks in parallel. So multiprocessing is a good way to saturate all of
your CPU cores. Big downsides include it being much harder to share
information between the processes, and much MUCH higher resource usage
than threads (with the same test as the above, I ran into limitations
at just over 500 processes - way fewer than the 10,000 threads!).

Asynchronous I/O runs a single thread in a single process. So like
multithreading, it's only good for I/O bound tasks like networking.
It's harder to work with, though, since you have to be very careful to
include proper await points, and you can stall out the entire event
loop with one mistake (common culprits being synchronous disk I/O, and
gethostbyname). But the upside is that you get near-infinite tasks,
basically just limited by available memory (or other resources).

Use whichever one is right for your needs.

ChrisA

Subject	Author
Re: Recommendations in terms of threading, multi-threading and/or	Chris Angelico

<Overfiend> penis jokes are okay in mixed company. VMS is NOT!!!

devel / comp.lang.python / Re: Recommendations in terms of threading, multi-threading and/or asynchronous processes/programming? - Sent Mail - Mozilla Thunderbird

devel / comp.lang.python / Re: Recommendations in terms of threading, multi-threading and/or asynchronous processes/programming? - Sent Mail - Mozilla Thunderbird