Rocksolid Light

Welcome to RetroBBS

mail  files  register  newsreader  groups  login

Message-ID:  

One can't proceed from the informal to the formal by formal means.

devel / comp.lang.python / Re: Puzzling behaviour of Py_IncRef

SubjectAuthor
o Re: Puzzling behaviour of Py_IncRefTony Flury

1
Re: Puzzling behaviour of Py_IncRef

<mailman.263.1643184219.3079.python-list@python.org>

  copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=21227&group=comp.lang.python#21227

  copy link   Newsgroups: comp.lang.python
Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: tony.flury@btinternet.com (Tony Flury)
Newsgroups: comp.lang.python
Subject: Re: Puzzling behaviour of Py_IncRef
Date: Wed, 26 Jan 2022 08:03:36 +0000
Lines: 175
Message-ID: <mailman.263.1643184219.3079.python-list@python.org>
References: <112a6b03-9a98-2d12-3492-67e4327ad32f@btinternet.com>
<1DCDD357-E281-4024-95F8-0C9F066F1439@barrys-emacs.org>
<8201bfa3-8626-a509-bbde-801566e2bc33@btinternet.com>
<f5510b80-6ebb-fc45-3a1a-82d8bba37968@mrabarnett.plus.com>
<3d73447c-c793-fb9e-c690-5b6ab2951e31@btinternet.com>
Reply-To: tony.flury@btinternet.com
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Trace: news.uni-berlin.de Rjtsg3uA5CF636nXaiSa9Qek5eV23+miJ4M6p/+OB3Ig==
Return-Path: <tony.flury@btinternet.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=btinternet.com header.i=@btinternet.com header.b=fCbnOXon;
dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.023
X-Spam-Evidence: '*H*': 0.95; '*S*': 0.00; 'argument': 0.04; 'def':
0.04; '2022': 0.05; 'despite': 0.05; 'angelico': 0.09;
'effectively': 0.09; 'example:': 0.09; 'itself,': 0.09;
'obviously': 0.09; 'skip:\xc2 20': 0.09; 'memory': 0.15;
'problem.': 0.15; '(so': 0.16; '*second*': 0.16; '2022,': 0.16;
'>>>>': 0.16; '>>>>>': 0.16; '>>>>>>': 0.16; '>>>>>>>': 0.16;
'>>>>>>>>': 0.16; 'anyway.': 0.16; 'assuming': 0.16; 'barry':
0.16; 'behaviour': 0.16; 'chrisa': 0.16; 'complete.': 0.16;
'count.': 0.16; 'discarding': 0.16; 'drops': 0.16; 'greg': 0.16;
'incremented': 0.16; 'indeed': 0.16; 'nodes': 0.16; 'odd': 0.16;
'python\xe2\x80\x99s': 0.16; 'wrote:': 0.16; 'python': 0.16;
'code.': 0.17; "can't": 0.17; 'calls': 0.19; 'to:addr:python-
list': 0.20; 'first,': 0.22; 'fri,': 0.22; 'returns': 0.22;
'code': 0.23; '(and': 0.25; 'anything': 0.25; 'actual': 0.25;
'saying': 0.25; 'seems': 0.26; 'behavior': 0.26; 'do,': 0.26;
'library': 0.26; 'object': 0.26; "wasn't": 0.26; "isn't": 0.27;
'else': 0.27; 'coming': 0.27; 'function': 0.27; '>>>': 0.28;
'chris': 0.28; 'expect': 0.28; 'purpose': 0.28; 'suggest': 0.28;
'it,': 0.29; 'header:User-Agent:1': 0.30; 'code,': 0.31; 'am,':
0.31; 'deep': 0.31; 'do.': 0.32; 'execution': 0.32; 'happening':
0.32; 'point,': 0.32; 'python-list': 0.32; 'returning': 0.32;
'received:192.168.1': 0.32; 'but': 0.32; 'there': 0.33; 'same':
0.34; 'header:In-Reply-To:1': 0.34; 'trying': 0.35; 'meaning':
0.35; 'yes,': 0.35; 'cases': 0.36; 'count': 0.36; 'change': 0.36;
'those': 0.36; 'really': 0.37; 'using': 0.37; "it's": 0.37;
'received:192.168': 0.37; 'way': 0.38; 'could': 0.38; 'means':
0.38; 'two': 0.39; 'adding': 0.39; 'setting': 0.39; 'break': 0.39;
'seeing': 0.39; 'still': 0.40; 'double': 0.40; 'received:213':
0.40; 'reports': 0.40; 'something': 0.40; 'should': 0.40;
'reference': 0.60; 'tell': 0.60; 'method': 0.61; "there's": 0.61;
'skip:\xc2 10': 0.62; 'here': 0.62; 'simply': 0.63; 'email': 0.63;
'once': 0.63; 'leading': 0.63; "you'd": 0.64; 'clear': 0.64;
'imagine': 0.64; 'your': 0.64; 'named': 0.65; 'back': 0.67;
'outside': 0.67; 'accept': 0.67; 'right': 0.68; 'result,': 0.69;
'within': 0.69; '2nd': 0.70; 'interesting': 0.71; '8bit%:100':
0.76; 'reply-to:no real name:2**0': 0.78; 'header:Reply-To:1':
0.79; 'anthony': 0.84; 'discarded': 0.84; 'from.': 0.84;
'garbage': 0.84; 'increments': 0.84; 'macro': 0.84; 'maintains':
0.84; 'parent.': 0.84; 'referenced': 0.84; 'scope': 0.84; 'self)':
0.84; 'statement.': 0.84; 'tony': 0.84;
'\xc2\xa0\xc2\xa0\xc2\xa0\xc2\xa0': 0.84;
'\xc2\xa0\xc2\xa0\xc2\xa0\xc2\xa0\xc2\xa0': 0.84; 'caused': 0.86;
'deliberately': 0.91; 'return.': 0.91; 'immediate': 0.97
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=btinternet.com;
s=btmx201904; t=1643184217;
bh=a7vZAJT3TzeRlWpS9PG6QO0QM0ZmuScYS5c9jLpVKAM=;
h=Message-ID:Date:MIME-Version:Reply-To:Subject:To:References:From:In-Reply-To;
b=fCbnOXonc5C13erPqDcwkhFFgFq8rvnGkgkU20mWsOisaBQQNVzeMxEIt9cvvG6318HU756Lqqrnij0EZW7+xglKng0VpGW+UZRV/eNX70NnrtLt7NyFYmnmsxu3Jax+/yv6ElZAsqXPHARl5GbElwlNr8jEgJ3t7rHO0dzDdZOw3zjg//FSgf95PSoOzWNYtl7LEyf8Pm+PV6L0SrJQR5/2mfTDv+Hs9XsY7au2QtNGYQVNnmh2ITVz6P5qGaz++aZ+QQ7X4BEsfNKXLmzyiasE0WhF6eLZuSCWZfX3DXQ+qzz8zRICQFOXrIa2MGyTIi5C8ibCauWmWBtEoFSd0A==
Authentication-Results: btinternet.com;
auth=pass (PLAIN) smtp.auth=tony.flury@btinternet.com;
bimi=skipped
X-SNCR-Rigid: 6139452E12AE25E5
X-Originating-IP: [81.153.144.133]
X-OWM-Source-IP: 81.153.144.133 (GB)
X-OWM-Env-Sender: tony.flury@btinternet.com
X-VadeSecure-score: verdict=clean score=0/300, class=clean
X-RazorGate-Vade: gggruggvucftvghtrhhoucdtuddrgedvvddrfedtgdduudeiucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecuueftkffvkffujffvgffngfevqffopdfqfgfvnecuuegrihhlohhuthemuceftddunecunecujfgurhepkfffgggfrhfuvfhfhfgjtgfgsehtkeertddtfeejnecuhfhrohhmpefvohhnhicuhfhluhhrhicuoehtohhnhidrfhhluhhrhiessghtihhnthgvrhhnvghtrdgtohhmqeenucggtffrrghtthgvrhhnpeegkeeuleeivdeuudfggffglefftedvleekkeekgffhieelueehudejueettddtkeenucfkphepkedurdduheefrddugeegrddufeefnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehhvghloheplgduledvrdduieekrddurddugeejngdpihhnvghtpeekuddrudehfedrudeggedrudeffedpmhgrihhlfhhrohhmpehtohhnhidrfhhluhhrhiessghtihhnthgvrhhnvghtrdgtohhmpdhnsggprhgtphhtthhopedupdhrtghpthhtohepphihthhhohhnqdhlihhsthesphihthhhohhnrdhorhhg
X-RazorGate-Vade-Verdict: clean 0
X-RazorGate-Vade-Classification: clean
X-SNCR-hdrdom: btinternet.com
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
Thunderbird/91.5.0
Content-Language: en-US
In-Reply-To: <f5510b80-6ebb-fc45-3a1a-82d8bba37968@mrabarnett.plus.com>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <3d73447c-c793-fb9e-c690-5b6ab2951e31@btinternet.com>
X-Mailman-Original-References: <112a6b03-9a98-2d12-3492-67e4327ad32f@btinternet.com>
<1DCDD357-E281-4024-95F8-0C9F066F1439@barrys-emacs.org>
<8201bfa3-8626-a509-bbde-801566e2bc33@btinternet.com>
<f5510b80-6ebb-fc45-3a1a-82d8bba37968@mrabarnett.plus.com>
 by: Tony Flury - Wed, 26 Jan 2022 08:03 UTC

On 26/01/2022 01:29, MRAB wrote:
> On 2022-01-25 23:50, Tony Flury via Python-list wrote:
>>
>> On 25/01/2022 22:28, Barry wrote:
>>>
>>>> On 25 Jan 2022, at 14:50, Tony Flury via
>>>> Python-list<python-list@python.org>  wrote:
>>>>
>>>> 
>>>>> On 20/01/2022 23:12, Chris Angelico wrote:
>>>>>> On Fri, 21 Jan 2022 at 10:10, Greg
>>>>>> Ewing<greg.ewing@canterbury.ac.nz>  wrote:
>>>>>> On 20/01/22 12:09 am, Chris Angelico wrote:
>>>>>>> At this point, the refcount has indeed been increased.
>>>>>>>
>>>>>>>>            return self;
>>>>>>>>       }
>>>>>>> And then you say "my return value is this object".
>>>>>>>
>>>>>>> So you're incrementing the refcount, then returning it without
>>>>>>> incrementing the refcount. Your code is actually equivalent to
>>>>>>> "return
>>>>>>> self".
>>>>>> Chris, you're not making any sense. This is C code, so there's no
>>>>>> way that "return x" can change the reference count of x.
>>>>> Yeah, I wasn't clear there. It was equivalent to *the Python code*
>>>>> "return self". My apologies.
>>>>>
>>>>>>   > The normal thing to do is to add a reference to whatever you're
>>>>>>   > returning. For instance, Py_RETURN_NONE will incref None and
>>>>>> then
>>>>>>   > return it.
>>>>>>   >
>>>>>>
>>>>>> The OP understands that this is not a normal thing to do. He's
>>>>>> trying to deliberately leak a reference for the purpose of
>>>>>> diagnosing
>>>>>> a problem.
>>>>>>
>>>>>> It would be interesting to see what the actual refcount is after
>>>>>> calling this function.
>>>> After calling this without a double increment in the function the
>>>> ref count is still only 1 - which means that the 'return self'
>>>> effectively does a double decrement. My original message includes
>>>> the Python code which calls this 'leaky' function and you can see
>>>> that despite the 'leaky POC' doing an increment ref count drops
>>>> back to one after the return.
>>>>
>>>> You are right this is not a normal thing to do, I am trying to
>>>> understand the behaviour so my library does the correct thing in
>>>> all cases - for example - imagine you have two nodes in a tree :
>>>>
>>>> A --- > B
>>>>
>>>> And your Python code has a named reference to A, and B also
>>>> maintains a reference to A as it's parent.
>>>>
>>>> In this case I would expect A to have a reference count of 2
>>>> (counted as 3 through sys.getrefcount() - one for the named
>>>> reference in the Python code - and one for the link from B back to
>>>> A; I would also expect B to have a reference count here of 1 (just
>>>> the reference from A - assuming nothing else referenced B).
>>>>
>>>> My original code was incrementing the ref counts of A and B and
>>>> then returning A. within the Python test code A had a refcount of 1
>>>> (and not the expected 2), but the refcount from B was correct as
>>>> far as I could tell.
>>>>
>>>>
>>>>> Yes, and that's why I was saying it would need a *second* incref.
>>>>>
>>>>> ChrisA
>>>> Thank you to all of you for trying to help - I accept that the only
>>>> way to make the code work is to do a 2nd increment.
>>>>
>>>> I don't understand why doing a 'return self' would result in a
>>>> double decrement - that seems utterly bizzare behaviour - it
>>>> obviously works, but why.
>>> The return self in C will not change the ref count.
>>>
>>> I would suggest setting a break point in your code and stepping out
>>> of the function and seeing that python’s code does to the ref count.
>>>
>>> Barry
>>
>> Barry,
>>
>> something odd is going on because the Python code isn't doing anything
>> that would cause the reference count to go from 3 inside the C function
>> to 1 once the method call is complete.
>>
>> As far as I know the only things that impact the reference counts are :
>>
>>    * Increments due to assigning a new name or adding it to a container.
>>    * Increment due to passing the object to a function (since that binds
>>      a new name)
>>    * Decrements due to deletion of a name
>>    * Decrement due to going out of scope
>>    * Decrement due to being removed from a container.
>>
>> None of those things are happening in the python code.
>>
>> As posted in the original message - immediately before the call to the C
>> function/method sys.getrefcount reports the count to be 2 (meaning it is
>> actually a 1).
>>
>> Inside the C function the ref count is incremented and the Py_REFCNT
>> macro reports the count as 3 inside the C function as expected (1 for
>> the name in the Python code, 1 for the argument as passed to the C
>> function, and 1 for the increment), so outside the function one would
>> expect the ref count to now be 2 (since the reference caused by calling
>> the function is then reversed).
>>
>> However - Immediately outside the C function and back in the Python code
>> sys.getrefcount reports the count to be 2 again - meaning it is now
>> really 1. So that means that the refcount has been decremented twice
>> in-between the return of the C function and the execution of the
>> immediate next python statement. I understand one of those decrements -
>> the parameter's ref count is incremented on the way in so the same
>> object is decremented on the way out (so that calls don't leak
>> references) but I don't understand where the second decrement is coming
>> from.
>>
>> Again there is nothing in the Python code that would cause that
>> decrement - the decrement behavior is in the Python runtime.
>>
> The function returns a result, an object.
>
> The calling code is discarding the result, so it's being DECREFed.
>
> For example:
>
>     def foo():
>         return Node()
>
> returns a new node, so its refcount is 1.
>
> Calling 'foo' as statement:
>
>     foo()
>
> discards the result; the result is DECREFed back to 0 and garbage
> collected.
>
> If you wanted your C function to return None, you'd have:
>
>     Py_INCREF(Py_NONE);
>     return Py_None;
>
> or, more succinctly:
>
>     Py_RETURN_NONE;
>
> But you're returning the object itself, and you're INCREFing it first,
> which is what you need to do anyway.
>
> The 'extra' DECREF is coming from the result (i.e. self) being discarded.
>
> If it wasn't DECREFed, a function could create a new object (refcount
> == 1) and return it, and if the function was being called as a
> statement, it would be discarded with the refcount still == 1, leading
> to a memory leak.

So according to that I should increment twice if and only if the calling
code is using the result - which you can't tell in the C code - which is
very odd behaviour.

There is clearly something very deep here that I am simply not
understanding.

--
Anthony Flury
email : anthony.flury@btinternet.com

devel / comp.lang.python / Re: Puzzling behaviour of Py_IncRef

1
server_pubkey.txt

rocksolid light 0.9.81
clearnet tor