It is __del__ calling twice for some instances?

[Max Yuzhakov]

Hello!

It is correct behaviour for python to call __del__ on some
identity of a class object more than once?

In brief I shall describe a situation. Sorry for my english.

For debugin purposes I'm put in my module global counters
for counting __init__ and __del__ calls.

This is a sample code for clearness:
-------------------------------------------------------
init_cnt = 0
del_cnt = 0

class foo:
def __init__(self):
global init_cnt
init_cnt += 1

def __del__(self):
global del_cnt
del_cnt += 1

def stat():
print "init_cnt = %d" % init_cnt
print "del_cnt = %d" % del_cnt
print "difference = %d" % init_cnt-del_cnt
-------------------------------------------------------
And the result of a stat() call in some moment of time
looks like so:

init_cnt = 6233
del_cnt = 6234
difference = -1

It is __del__ called twice for some instance?

Thanks in advance!
--
GMT More Then ...

[Max Yuzhakov]

Max Yuzhakov writes:

MY print "difference = %d" % init_cnt-del_cnt

Little correction.

print "difference = %d" % (init_cnt-del_cnt)
--
GMT More Then ...

[Fredrik Lundh]

Max Yuzhakov wrote:

This is a sample code for clearness:

that code snippet doesn't create any foo() instances, what I can see...

</F>

[Duncan Booth]

Max Yuzhakov wrote:

It is correct behaviour for python to call __del__ on some
identity of a class object more than once?

Not with the code which you gave as an example, but in the general case
yes, the only guarantee that Python gives about the __del__ method on an
instance is that it will be called zero, one or more than one times during
the run of the program. In practice there are various situations where
__del__ will not be called, but it is only called multiple times if you
resurrect the object during a call to __del__.

The output from your stat function could, of course, also be generated by
creating and destroying lots of foo objects in another thread. If a foo was
both created and destroyed between the first two print statements, and
another one was created and destroyed in the middle of the evaluation of
the last print statement then you could see the output you described
without any multiple __del__ calls in the same object.

You should post a working code sample which generates your output if you
want a more useful answer.

[Max Yuzhakov]

Duncan Booth ÐÉÛÅÔ:

DB Not with the code which you gave as an example, but in the general case
DB yes, the only guarantee that Python gives about the __del__ method on an
DB instance is that it will be called zero, one or more than one times during
DB the run of the program. In practice there are various situations where
DB __del__ will not be called, but it is only called multiple times if you
DB resurrect the object during a call to __del__.

No, only presented code in __del__ do
increment of global variable del_cnt.

DB The output from your stat function could, of course, also be generated by
DB creating and destroying lots of foo objects in another thread. If a foo was
DB both created and destroyed between the first two print statements, and
DB another one was created and destroyed in the middle of the evaluation of
DB the last print statement then you could see the output you described
DB without any multiple __del__ calls in the same object.

No, it's a single-threaded code.

DB You should post a working code sample which generates your output if you
DB want a more useful answer.

I'm sorry for mess with a code. I have forgotten to write, that is
only fragments for demonstration of essence of a question.

Real code in this module has 880 lines and I shall not
abuse your time. I shall try to solve a problem by debuger.

And thank for Your answer!
--
GMT More Then ...

[Max Yuzhakov]

Duncan Booth wrote:

DB You should post a working code sample which generates your output if you
DB want a more useful answer.

Hello!

Today I have found a compact variant of a code which shows my question:
-----------------------------------------------------------------------
#!/usr/local/bin/python -d
# -*- coding: koi8-u -*-

class foo:
def __init__(self, other):
self.other = other

global ini_cnt
ini_cnt +=1

def __del__(self):
global del_cnt
del_cnt +=1

def stat():
print "-"*20
print "ini_cnt = %d" % ini_cnt
print "del_cnt = %d" % del_cnt
print "difference = %d" % (ini_cnt-del_cnt)

ini_cnt = 0
del_cnt = 0
loop_cnt = 75

a = foo(None)

for i in xrange(loop_cnt ):
a = foo(a)

stat()
a = None
stat()
-----------------------------------------------------------------------

And result is:
--------------------
ini_cnt = 76
del_cnt = 0
difference = 76
--------------------
ini_cnt = 76
del_cnt = 77
difference = -1

Why for some instance __del__ called twice?
Such behaviour of __del__ seems to me unpredictable.

Thanks for Your attention!
--
GMT More Then ...

[Duncan Booth]

Max Yuzhakov wrote:

Why for some instance __del__ called twice?
Such behaviour of __del__ seems to me unpredictable.

Here's a slightly modified version of your code. The 51st object destroyed
gets its __del__ method called twice. It doesn't matter how long your loop
is, every 50th object gets special treatment (or perhaps it is the 49th).

The good news is that there is a way to stop it happening: just add an
explicit "del self.other" at the end of __del__. The bad news is that if
your list is too long that will cause a flood of error messages and won't
call the destructors at all past the first 1000.

As to why it happens, there is a mechanism in Python to stop unlimited
stack being used when objects are freed: when the stack gets too deep then
instead of being released, the Py_DECREF call puts the object into a
trashcan list and the objects aren't released until the stack has unwound.
It looks like there must be a bug round the trashcan mechanism somewhere.

BTW, the behaviour is completely different if you use a new style class,
but still somewhat bizarre: for new style classes only the first 25 objects
get freed when you clear a, the remainder are only released by the garbage
collector.

#!/usr/local/bin/python -d
# -*- coding: koi8-u -*-

class foo:
def __init__(self, other):
self.other = other
self._deleted = False

global ini_cnt
ini_cnt +=1

def __del__(self):
if self._deleted:
print "aargh!"
self._deleted = True
global del_cnt
del_cnt +=1
print "del",del_cnt," at",id(self)

def stat():
print "-"*20
print "ini_cnt = %d" % ini_cnt
print "del_cnt = %d" % del_cnt
print "difference = %d" % (ini_cnt-del_cnt)

ini_cnt = 0
del_cnt = 0
loop_cnt = 54

a = foo(None)

for i in xrange(loop_cnt ):
a = foo(a)

stat()
a = None
stat()

[Duncan Booth]

Duncan Booth wrote:

As to why it happens, there is a mechanism in Python to stop unlimited
stack being used when objects are freed: when the stack gets too deep
then instead of being released, the Py_DECREF call puts the object
into a trashcan list and the objects aren't released until the stack
has unwound. It looks like there must be a bug round the trashcan
mechanism somewhere.

I figured out what is going on in the code to deallocate an old-style class
instance:

The reference count is temporarily incremented.

If the class has a __del__ method then a descriptor is created for the
method and called. When the call returns, the descriptor is released.

Then the object itself is released using special code to avoid a recursive
call to the deallocator.

However, if the trashcan mechanism is invoked by the attempt to release the
descriptor, it actually queues the descriptor in the trashcan. Since the
descriptor contains a reference to the object it has effectively
resurrected it. This means the special code to avoid the recursive call
simply decrements the reference count but does not release anything (the
object has been resurrected by the descriptor). When the descriptor is
later released the __del__ method is triggered a second time.

[Patrick Maupin]

Duncan Booth wrote:

Duncan Booth wrote:

As to why it happens, there is a mechanism in Python to stop unlimited
stack being used when objects are freed: when the stack gets too deep
then instead of being released, the Py_DECREF call puts the object
into a trashcan list and the objects aren't released until the stack
has unwound. It looks like there must be a bug round the trashcan
mechanism somewhere.

I figured out what is going on in the code to deallocate an old-style class
instance:

The reference count is temporarily incremented.

If the class has a __del__ method then a descriptor is created for the
method and called. When the call returns, the descriptor is released.

Then the object itself is released using special code to avoid a recursive
call to the deallocator.

However, if the trashcan mechanism is invoked by the attempt to release the
descriptor, it actually queues the descriptor in the trashcan. Since the
descriptor contains a reference to the object it has effectively
resurrected it. This means the special code to avoid the recursive call
simply decrements the reference count but does not release anything (the
object has been resurrected by the descriptor). When the descriptor is
later released the __del__ method is triggered a second time.

This looks like some good code to add to the python unit tests.

>From your description, it appears the problem is that the object is

placed in the trashcan after calling __del__ once. Perhaps the choice
to place it in the trashcan could be made instead of calling __del__
the first time, rather than after calling __del__?

Regards,
Pat

[Max Yuzhakov]

Duncan Booth wrote:

DB I figured out what is going on in the code to deallocate an old-style class
DB instance:
DB>
DB The reference count is temporarily incremented.
DB>
DB If the class has a __del__ method then a descriptor is created for the
DB method and called. When the call returns, the descriptor is released.
DB>
DB Then the object itself is released using special code to avoid a recursive
DB call to the deallocator.
DB>
DB However, if the trashcan mechanism is invoked by the attempt to release the
DB descriptor, it actually queues the descriptor in the trashcan. Since the
DB descriptor contains a reference to the object it has effectively
DB resurrected it. This means the special code to avoid the recursive call
DB simply decrements the reference count but does not release anything (the
DB object has been resurrected by the descriptor). When the descriptor is
DB later released the __del__ method is triggered a second time.

Thank You for so detailed explanation!
--
GMT More Then ...