473,320 Members | 1,978 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

atexit + threads = bug?

Consider the following program (underscores are used to force
indentation):
------------------------------------------------
import atexit, threading, time

def atExitFunc():
____print 'atExitFunc called.'

atexit.register(atExitFunc)

class T(threading.Thread):
____def run(self):
________assert not self.isDaemon()
________print 'T before sleep.'
________time.sleep(1.0)
________print 'T after sleep.'

T().start()
print 'Main thread finished.'
------------------------------------------------

I would expect the program to print 'atExitFunc called.' after 'T after
sleep.', but instead, it prints (on Windows XP with Python 2.3.5 or
2.4.2):
------------------------------------------------
T before sleep.
Main thread finished.
atExitFunc called.
T after sleep.
------------------------------------------------

atExitFunc is called when the main thread terminates, rather than when
the process exits. The atexit documentation contains several warnings,
but nothing about this. Is this a bug?

Jan 12 '06 #1
7 6035

David> atExitFunc is called when the main thread terminates, rather than
David> when the process exits. The atexit documentation contains
David> several warnings, but nothing about this. Is this a bug?

This might be a bug, but I can't see how it can be in atexit. Atexit just
registers its own sys.exitfunc function, then when it's called, calls all
the individual exit functions that have been registered with it. It has no
control over when sys.exitfunc is invoked. sys.exitfunc is called as the
first action of Py_Finalize. It appears that Py_Finalize is called when the
main thread exits.

Skip
Jan 12 '06 #2
[David Rushby]
Consider the following program (underscores are used to force
indentation):
------------------------------------------------
import atexit, threading, time

def atExitFunc():
____print 'atExitFunc called.'

atexit.register(atExitFunc)

class T(threading.Thread):
____def run(self):
________assert not self.isDaemon()
________print 'T before sleep.'
________time.sleep(1.0)
________print 'T after sleep.'

T().start()
print 'Main thread finished.'
------------------------------------------------

I would expect the program to print 'atExitFunc called.' after 'T after
sleep.',
Why? I expect very little ;-)
but instead, it prints (on Windows XP with Python 2.3.5 or
2.4.2):
------------------------------------------------
T before sleep.
Main thread finished.
atExitFunc called.
T after sleep.
------------------------------------------------
That's not what I saw just now on WinXP Pro SP2. With 2.3.5 and 2.4.2
I saw this order instead:

Main thread finished
atExitFunc called.
T before sleep.
T after sleep.

The relative order of "Main thread finished." and "T before sleep" is
purely due to timing accidents; it's even possible for "T after
sleep." to appear before "Main thread finished.", although it's not
possible for "T after sleep." to appear before "T before sleep.". In
fact, there are only two orderings you can count on here:

T before sleep < T after sleep
Main thread finished < atExitFunc called

If you need more than that, you need to add synchronization code.
atExitFunc is called when the main thread terminates, rather than when
the process exits.
Is there a difference between "main thread terminates" and "the
process exits" on Windows? Not in C. It so happens that Python's
threading module _also_ registers an atexit callback, which does a
join() on all the threads you created and didn't mark as daemon
threads. Because threading.py's atexit callback was registered first,
it gets called last when Python is shutting down, and it doesn't
return until it joins all the non-daemon threads still sitting around.
Your atexit callback runs first because it was registered last. That
in turn makes it _likely_ that you'll see (as we both saw) "at
exitFunc called." before seeing "T after sleep.", but doesn't
guarantee that.

Don't by fooled by _printing_ "Main thread finished", BTW: that's
just a sequence of characters ;-). The main thread still does a lot
of work after that point, to tear down the interpreter in a sane
order. Part of that work is threading.py waiting for your threads to
finish.
The atexit documentation contains several warnings,
but nothing about this. Is this a bug?


It doesn't look like a bug to me, and I doubt Python wants to make
stronger promises than it does now about the exact order of assorted
exit gimmicks.

You can reliably get "atExitFunc called." printed last by delaying
your import of the threading module until after you register your
atExitFunc callback. If you register that first, it's called last,
and threading.py's wait-for-threads-to-end callback gets called first
then. That callback won't return before your worker thread finishes.

There's no promise that will continue to work forever, though. This
is fuzzy stuff vaguely covered by the atexit doc's "In particular,
other core Python modules are free to use atexit without the
programmer's knowledge." threading.py happens to be such a module
today, but maybe it won't be tomorrow.
Jan 12 '06 #3
>> I would expect...
The relative order of "Main thread finished." and "T before
sleep" is purely due to timing accidents...
Sure, I realize that the interactions between threads have no
guaranteed order except what the programmer imposes upon them. I
should have qualified my statement of expectation more carefully.
In fact, there are only two orderings you can count on here:
T before sleep < T after sleep
Main thread finished < atExitFunc called


I understand your explanation and can live with the consequences, but
the atexit docs sure don't prepare the reader for this.

They say, "Functions thus registered are automatically executed upon
normal interpreter termination." It seems like sophistry to argue that
"normal interpreter termination" has occurred when there are still
threads other than the main thread running.

Suppose that today I promise to donate my body to science "upon my
death", and tomorrow, I'm diagnosed with a gradual but inexorable
illness that will kill me within ten years. I wouldn't expect to be
strapped down and dissected immediately after hearing the diagnosis, on
the basis that the mere prophecy of my death is tantamount to the death
itself.

Jan 12 '06 #4
[David Rushby]
...
I understand your explanation and can live with the consequences, but
the atexit docs sure don't prepare the reader for this.
In fact, they don't mention threading.py at all.
They say, "Functions thus registered are automatically executed upon
normal interpreter termination." It seems like sophistry to argue that
"normal interpreter termination" has occurred when there are still
threads other than the main thread running.
Well, since atexit callbacks are written in Python, it's absurd on the
face of it to imagine that they run after the interpreter has torn
itself down. Clearly Python is still running at that point, or they
wouldn't get run at all.

It's also strained to imagine that threads have nothing to do with
shutdown, since the threading docs say "the entire Python program
exits when only daemon threads are left". It's not magic that
prevents Python from exiting when non-daemon threads are still
running. You happened to use the same non-magical hack that
threading.py uses to fulfill that promise, and you're seeing
consequences of their interaction. In Python as well as in C, atexit
only works well when it's got exactly zero or one users <0.1 wink>.

You're welcome to suggest text you'd like better, but microscopic
examination of details most people will never care about makes for bad
docs in a different way. To get a full picture of how CPython's
shutdown works, you need to explain all of Py_Finalize() in English,
and you need to get agreement on which details are accidents and which
are guaranteed.

Now it's probably a fact that you couldn't care less about 99.9% of
those finalization details: you only care about the one that just bit
you. How are you going to beef up the docs in such a way that you
would have _found_ the bit you cared about, among the vast bulk of new
detail you don't care about?

You aren't, so you could settle for suggesting new words that just
cover the bit you care about. Give it a try!
Suppose that today I promise to donate my body to science "upon my
death", and tomorrow, I'm diagnosed with a gradual but inexorable
illness that will kill me within ten years. I wouldn't expect to be
strapped down and dissected immediately after hearing the diagnosis, on
the basis that the mere prophecy of my death is tantamount to the death
itself.


Next time, quit while you're ahead ;-)
Jan 12 '06 #5
[Tim Peters]
[David Rushby]
They say, "Functions thus registered are automatically executed upon
normal interpreter termination." It seems like sophistry to argue that
"normal interpreter termination" has occurred when there are still
threads other than the main thread running.
Well, since atexit callbacks are written in Python, it's absurd on the
face of it to imagine that they run after the interpreter has torn
itself down.


Of course.
It's also strained to imagine that threads have nothing to do
with shutdown...
I don't imagine that.
You're welcome to suggest text you'd like better...


What I'd like is for the behavior to become less surprising, so that
the text could describe reasonable behavior, instead of retrofitting
the text to more clearly explain (what I regard as) flawed behavior.

What would be unreasonable about adding a
join_nondaemonic_threads()
call before the call to
call_sys_exitfunc()
near the beginning of Py_Finalize?

Instead of _MainThread.__exitfunc having to rely on atexit.register to
ensure that it gets called, join_nondaemonic_threads would call
_MainThread.__exitfunc (or some functional equivalent). Both
join_nondaemonic_threads and call_sys_exitfunc would execute while the
interpreter "is still entirely intact", as the Py_Finalize comment
says.

The opening paragraph of the atexit docs could then read:
"The atexit module defines a single function to register cleanup
functions. Functions thus registered are automatically executed when
the main thread begins the process of tearing down the interpreter,
which occurs after all other non-daemonic threads have terminated and
the main thread has nothing but cleanup code left to execute."

This seems simple. Am I overlooking something?

Jan 12 '06 #6

David> What would be unreasonable about adding a
David> join_nondaemonic_threads()
David> call before the call to
David> call_sys_exitfunc()
David> near the beginning of Py_Finalize?

David> Instead of _MainThread.__exitfunc having to rely on
David> atexit.register to ensure that it gets called,
David> join_nondaemonic_threads would call _MainThread.__exitfunc (or
David> some functional equivalent). Both join_nondaemonic_threads and
David> call_sys_exitfunc would execute while the interpreter "is still
David> entirely intact", as the Py_Finalize comment says.

...

David> This seems simple. Am I overlooking something?

A patch? <0.5 wink>

Skip
Jan 13 '06 #7
[Skip]
[David]
This seems simple. Am I overlooking something?

A patch? <0.5 wink>


I'm willing to write a patch if it stands a good chance of being
accepted. So far, though, Tim has seemed resistant to the idea. Maybe
he has reasons that I'm ignorant of?

Jan 13 '06 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: Serve Laurijssen | last post by:
Does the C++ standard say whether destructors are called before or after "atexit" functions are called. Or is nothing said about that?
8
by: JKop | last post by:
Let's say that when your program ends (no matter how) that you want a certain block of code to be executed at the end. Here's the code: std::cout << "The program will now end.\n";...
2
by: Steve Lambert | last post by:
Hi, Is it possible for the functions registered with atexit() to have access to parameter to exit i.e EXIT_SUCCESS or EXIT_FAILURE? I'd like to register a set of routines to be executed on...
5
by: prouleau001 | last post by:
Hi all! Since that the decorator syntax is upon us, I think it would be good if atexit.register() was returning the function passed as argument. This simple change to the library would solve a...
1
by: Parapura Rajkumar | last post by:
hey all I was wondering if there a way to override atexit in my own module. The problem I was facing is that the VS2005 seems to ignore functions registered with atexit once it starts processing...
20
by: Aek | last post by:
We recently moved our large codebase over from VS7 to 8 and found that we now get access violations in atexit calls at shutdown when debugging the application in VS2005. This occurs in static...
16
by: Laurent Deniau | last post by:
I would like to know if the use of the pointer ref in the function cleanup() below is valid or if something in the norm prevents this kind of cross-reference during exit(). I haven't seen anything...
2
by: Christopher Pisz | last post by:
I am attempting to write a "Phoenix Singleton" using the book "Modern C++ Design" by Alexandrescu I do not understand his use of... #ifndef ATEXIT_FIXED std::atexit(Kill); #endif ....in the...
18
by: lak | last post by:
I am studying the Advanced programming in the unix environment. There they says that we can register upto 32 functions with atexit(). Why that is limited to 32 functions? can any one tell the...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.