473,707 Members | 2,273 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

thread/queue bug

And sorry I got ticked, frustrating week
And I could help more, being fairly experienced with
threading issues and race conditions and such, but
as I tried to indicate in the first place, you've
provided next to no useful (IMHO) information to
let anyone help you more than this


This is about 5% of the code.
Uses no locks.
I am mystified, I have written probably 100,000 lines
of Python and never seen a thread just lock up and quit
running. It happens on a Queue() statement so my suspicion
is a bug. ??

I have kludged around it by putting all the thread/queue stuff
in the main program and import the stuff I don't want to
distribute. But mysteries haunt your dreams, sooo...
#!/usr/local/bin/python

# pnetx.py

from threading import *
from time import *
from Queue import Queue
from socket import *
import sys
import os

# glob is a DUMMY CLASS

glob.listenq = Queue(1000)

def listener():
while 1:
msg,addrport = listenersocket. recvfrom(BUFSIZ E)
addr = addrport[0]
glob.listenq.pu t (( msg,addr),1)
if msg == 'DIE': break

def procmsgs():
while 1:
msg,addr = glob.listenq.ge t(1)
if msg == 'DIE': break
wds = msg.split(';')
if wds[0] == 'S': recvsucc(msg); continue
if wds[0] == 'E': recvevent(msg); continue
if wds[0] == 'ACK': recvack(msg[4:]); continue
if wds[0] == 'MONITOR':
if addr not in monitorlist:
print 'This line ALWAYS PRINTS'
queuelist.appen d( Queue(0) )
## The above fails if this code is imported
## It doesn't matter whether it is imported
## as .py or .pyc
## also mq = Queue(0); queuelist.appen d(mq) # fails
print 'This line ALWAYS PRINTS if i execute this source'
print 'but NEVER PRINTS if I import '
print 'this source from a 1 line program'
print 'Thats weird'
monitoron.appen d( 1 )
monitorlist.app end( addr )

queuelist = [Queue(0)]
listenersocket = socket(AF_INET, SOCK_DGRAM)
listenersocket. bind(ListenAddr )

procm = Thread(target=p rocmsgs)
procm.start()
listenr = Thread(target=l istener)
listenr.start()

## Then start a bunch of other threads and queuea.
Don't spend a lot of time on this, not worth it.
I just thought someone might have experienced
something similar.

Jul 18 '05 #1
9 2780
phil wrote:
Uses no locks.
It does use locks implicitly, though, since even just
importing threading will do that, and creating a Queue
does too.
I am mystified, I have written probably 100,000 lines
of Python and never seen a thread just lock up and quit
running. It happens on a Queue() statement so my suspicion
is a bug. ??
You have the source to Queue.py in your standard library
folder. Why not throw a few more print statements into
its __init__ and see what you learn?
I have kludged around it by putting all the thread/queue stuff
in the main program and import the stuff I don't want to
distribute. But mysteries haunt your dreams, sooo...
#!/usr/local/bin/python

[snip source]

I cobbled together a "working" version of your code
and ran it just fine, whether imported or run directly.
No lockups. On Windows XP.

Are you by any chance running on a new version of the
Linux kernel, where the threading model has changed?
(Or was it just RedHat 9.0?)

I don't know the details, but I know folks have had trouble
with this and Python... For example, I found this
reference to the issue:
http://groups.google.ca/groups?th=fe...line.com#link2

-Peter
Jul 18 '05 #2
On Fri, 10 Dec 2004 16:18:51 -0600, phil <ph***********@ anvilcom.com> wrote:
And sorry I got ticked, frustrating week
threading problems can do that ;-)

You are obviusly deeper into this than I can get from a cursory scan,
but I'll make some general debugging comments ;-)
And I could help more, being fairly experienced with
threading issues and race conditions and such, but
as I tried to indicate in the first place, you've
provided next to no useful (IMHO) information to
let anyone help you more than this


This is about 5% of the code.
Uses no locks.
I am mystified, I have written probably 100,000 lines
of Python and never seen a thread just lock up and quit
running. It happens on a Queue() statement so my suspicion
is a bug. ??

Or a once-in-a-blue-moon resource access deadlock of some kind?

I have kludged around it by putting all the thread/queue stuff
in the main program and import the stuff I don't want to
distribute. But mysteries haunt your dreams, sooo...
#!/usr/local/bin/python

# pnetx.py

from threading import *
from time import *
from Queue import Queue
from socket import * Do you really need to import * ? Though the above should be safe import-wise
you are setting yourself up for name-clash problems by putting so many in
the same space. E.g., if something happened to match a misspelling typo in
your program, you wouldn't get a name error exception. Etc., etc.

One of the first rules of debugging is to eliminate the unnecessary ;-)
At least there doesn't seem to be name clashes between the above (except
starting with '_' which shouldn't get imported with *
d = {}
for imp in 'threading time Queue socket'.split() : ... m = __import__(imp)
... for name in m.__dict__.keys ():
... d.setdefault(na me, []).append(imp)
... for k,v in d.items(): ... if len(v)!=1: print k,v
...
_sleep ['threading', 'Queue']
__file__ ['threading', 'Queue', 'socket']
__all__ ['threading', 'Queue', 'socket']
__builtins__ ['threading', 'Queue', 'socket']
__name__ ['threading', 'time', 'Queue', 'socket']
_time ['threading', 'Queue']
__doc__ ['threading', 'time', 'Queue', 'socket']

Just verifying an assumption ;-)
OTOH, do you really need (ignoring wid and name and the first content of dir()):
dir() ['__builtins__', '__doc__', '__name__'] from threading import *
from time import *
from Queue import *
from socket import *
wid = 0
for name in dir(): ... print repr(name),
... wid += len(repr(name)) +1
... if wid>60:
... print
... wid = 0
...
'AF_APPLETALK' 'AF_INET' 'AF_IPX' 'AF_UNSPEC' 'AI_ADDRCONFIG'
'AI_ALL' 'AI_CANONNAME' 'AI_DEFAULT' 'AI_MASK' 'AI_NUMERICHOST '
'AI_PASSIVE' 'AI_V4MAPPED' 'AI_V4MAPPED_CF G' 'BoundedSemapho re'
'CAPI' 'Condition' 'EAI_ADDRFAMILY ' 'EAI_AGAIN' 'EAI_BADFLAGS'
'EAI_BADHINTS' 'EAI_FAIL' 'EAI_FAMILY' 'EAI_MAX' 'EAI_MEMORY'
'EAI_NODATA' 'EAI_NONAME' 'EAI_PROTOCOL' 'EAI_SERVICE' 'EAI_SOCKTYPE'
'EAI_SYSTEM' 'Empty' 'Event' 'Full' 'INADDR_ALLHOST S_GROUP' 'INADDR_ANY'
'INADDR_BROADCA ST' 'INADDR_LOOPBAC K' 'INADDR_MAX_LOC AL_GROUP'
'INADDR_NONE' 'INADDR_UNSPEC_ GROUP' 'IPPORT_RESERVE D' 'IPPORT_USERRES ERVED'
'IPPROTO_GGP' 'IPPROTO_ICMP' 'IPPROTO_IDP' 'IPPROTO_IGMP' 'IPPROTO_IP'
'IPPROTO_MAX' 'IPPROTO_ND' 'IPPROTO_PUP' 'IPPROTO_RAW' 'IPPROTO_TCP'
'IPPROTO_UDP' 'IP_ADD_MEMBERS HIP' 'IP_DEFAULT_MUL TICAST_LOOP'
'IP_DEFAULT_MUL TICAST_TTL' 'IP_DROP_MEMBER SHIP' 'IP_MAX_MEMBERS HIPS'
'IP_MULTICAST_I F' 'IP_MULTICAST_L OOP' 'IP_MULTICAST_T TL' 'IP_OPTIONS'
'IP_TOS' 'IP_TTL' 'Lock' 'MSG_DONTROUTE' 'MSG_OOB' 'MSG_PEEK'
'NI_DGRAM' 'NI_MAXHOST' 'NI_MAXSERV' 'NI_NAMEREQD' 'NI_NOFQDN'
'NI_NUMERICHOST ' 'NI_NUMERICSERV ' 'Queue' 'RAND_add' 'RAND_egd'
'RAND_status' 'RLock' 'SOCK_DGRAM' 'SOCK_RAW' 'SOCK_RDM' 'SOCK_SEQPACKET '
'SOCK_STREAM' 'SOL_IP' 'SOL_SOCKET' 'SOL_TCP' 'SOL_UDP' 'SOMAXCONN'
'SO_ACCEPTCONN' 'SO_BROADCAST' 'SO_DEBUG' 'SO_DONTROUTE' 'SO_ERROR'
'SO_KEEPALIVE' 'SO_LINGER' 'SO_OOBINLINE' 'SO_RCVBUF' 'SO_RCVLOWAT'
'SO_RCVTIMEO' 'SO_REUSEADDR' 'SO_SNDBUF' 'SO_SNDLOWAT' 'SO_SNDTIMEO'
'SO_TYPE' 'SO_USELOOPBACK ' 'SSLType' 'SSL_ERROR_EOF' 'SSL_ERROR_INVA LID_ERROR_CODE'
'SSL_ERROR_SSL' 'SSL_ERROR_SYSC ALL' 'SSL_ERROR_WANT _CONNECT'
'SSL_ERROR_WANT _READ' 'SSL_ERROR_WANT _WRITE' 'SSL_ERROR_WANT _X509_LOOKUP'
'SSL_ERROR_ZERO _RETURN' 'Semaphore' 'SocketType' 'TCP_NODELAY'
'Thread' 'Timer' '__builtins__' '__doc__' '__name__' 'accept2dyear'
'activeCount' 'altzone' 'asctime' 'clock' 'ctime' 'currentThread'
'daylight' 'enumerate' 'error' 'errorTab' 'gaierror' 'getaddrinfo'
'getdefaulttime out' 'getfqdn' 'gethostbyaddr' 'gethostbyname'
'gethostbyname_ ex' 'gethostname' 'getnameinfo' 'getprotobyname '
'getservbyname' 'gmtime' 'has_ipv6' 'herror' 'htonl' 'htons'
'inet_aton' 'inet_ntoa' 'localtime' 'mktime' 'ntohl' 'ntohs'
'setdefaulttime out' 'setprofile' 'settrace' 'sleep' 'socket'
'ssl' 'sslerror' 'strftime' 'strptime' 'struct_time' 'time' 'timeout'
'timezone' 'tzname' 'wid'


(Hm, should have pre-tested wid+len(current ) to limit width ;-)
import sys
import os

# glob is a DUMMY CLASS where from? accidentally snipped?
glob.listenq = Queue(1000)

def listener():
while 1:
msg,addrport = listenersocket. recvfrom(BUFSIZ E)
addr = addrport[0] what is the usage level on glob.listenq? It is finite 1000, so
you could theoretically block on the put. What do you get
if you print glob.listenq.qs ize here?
glob.listenq.pu t (( msg,addr),1)
if msg == 'DIE': break

def procmsgs():
while 1:
msg,addr = glob.listenq.ge t(1)
if msg == 'DIE': break Might be interesting to get some output when 'DIE' is recognized. Above too.
wds = msg.split(';') assert wds[0] in ['S', 'E', 'ACK', 'MONITOR'] # check assumptions. E.g.,
# split on ';' means spaces around
# the ';' are not eliminated if wds[0] == 'S': recvsucc(msg); continue
if wds[0] == 'E': recvevent(msg); continue
if wds[0] == 'ACK': recvack(msg[4:]); continue
if wds[0] == 'MONITOR':
if addr not in monitorlist:
print 'This line ALWAYS PRINTS'
queuelist.appen d( Queue(0) ) Why are you apparently creating *new* queues in this thread that you are not using
and appending them to queuelist in the main thread? Are you testing to see how many
can be created, or for some Queue creation bug?

What do you get if you print 'queuelist length = %s'%len(queueli st) here?
How often are you getting 'MONITOR'?
## The above fails if this code is imported Imported from where? Do you mean that this code works if it
is embedded in some larger code, and if you put an import of this
at the top of that code it doesn't work? That would not only be
moving the source, but also the point of invocation. I.e., the
import *executes* what you import once, putting the result in
the module's global name space (other than code that explicitly
accesses other modules etc.) Do you need to wrap part of this
in a function that you can call *after* importing the definitions?

If your real app has multiple threads accessing a main thread queuelist
without a lock, you might want to look closely and see if that needs
to be a queue mechanism itself.
## It doesn't matter whether it is imported
## as .py or .pyc
## also mq = Queue(0); queuelist.appen d(mq) # fails
print 'This line ALWAYS PRINTS if i execute this source'
print 'but NEVER PRINTS if I import '
print 'this source from a 1 line program'
print 'Thats weird'
monitoron.appen d( 1 )
monitorlist.app end( addr )
Do you want all the following to be executed once at the point of importing this?
queuelist = [Queue(0)] What is this for? (oops, I hope I restored the above line correctly)

listenersock et = socket(AF_INET, SOCK_DGRAM)
listenersocket .bind(ListenAdd r)

procm = Thread(target=p rocmsgs)
procm.start( )
listenr = Thread(target=l istener)
listenr.start( )

## Then start a bunch of other threads and queuea.
Don't spend a lot of time on this, not worth it.
I just thought someone might have experienced
something similar.


Just some thoughts. HTH.
Regards,
Bengt Richter
Jul 18 '05 #3
phil wrote:
And sorry I got ticked, frustrating week
>And I could help more, being fairly experienced with
>threading issues and race conditions and such, but
>as I tried to indicate in the first place, you've
>provided next to no useful (IMHO) information to
>let anyone help you more than this
This is about 5% of the code.
Uses no locks.


A lock is acquired in Queue.__init__( ).
I am mystified, I have written probably 100,000 lines
of Python and never seen a thread just lock up and quit
running. It happens on a Queue() statement so my suspicion
is a bug. ??

I have kludged around it by putting all the thread/queue stuff
in the main program and import the stuff I don't want to
distribute. But mysteries haunt your dreams, sooo...


What I believe to be a minimal example:

<freeze.py>
import Queue
import threading
import time

q = Queue.Queue(4)

def proc():
while True:
q.get(1)
Queue.Queue()
print "YADDA"

threading.Threa d(target=proc). start()

while True:
print "yadda"
q.put(None)
time.sleep(1)
</freeze.py>

<freezemain.p y>
import freeze
</freezemain.py>

Invoking freezemain.py produces

yadda
yadda
yadda
yadda
yadda
yadda

i. e. consistently q.maxsize + 2. One item about to be put, one already
taken before Queue.Queue(). Deferring execution of the module-level code
until after the import

<nofreeze.py>
import Queue
import threading
import time

q = Queue.Queue(4)

def proc():
while True:
q.get(1)
Queue.Queue()
print "YADDA"

def start():
threading.Threa d(target=proc). start()

while True:
print "yadda"
q.put(None)
time.sleep(1)
</nofreeze.py>

<nofreezemain.p y>
import nofreeze
nofreeze.start( )
</nofreezemain.py >

and invoking nofreezemain.py produces
yadda
YADDA
yadda
YADDA
yadda
YADDA
yadda
YADDA

apparently ad infinitum. Conclusion? None so far, though you might welcome
the confirmation that this is not a "once in a blue moon" accident as Bengt
Richter surmised.

Import _is_ a sensitive phase...

As an experiment I moved

try:
import thread
except ImportError:
import dummy_thread as thread

from the Queue.Queue.__i nit__() method to the module body -- and now
freezemain.py seems to work, too. So that would be an easy remedy, but sure
there is a reason why that import statement is in such an unusual place?

Peter


Jul 18 '05 #4
[Peter Otten]
What I believe to be a minimal example:

<freeze.py>
import Queue
import threading
import time

q = Queue.Queue(4)

def proc():
while True:
q.get(1)
Queue.Queue()
print "YADDA"

threading.Threa d(target=proc). start()

while True:
print "yadda"
q.put(None)
time.sleep(1)
</freeze.py>

<freezemain.p y>
import freeze
</freezemain.py>
CPython has an internal, reentrant import lock. When a thread does an
import, it acquires this lock, and the lock remains held until that
import is complete. As a consequence, no *other* thread can do an
import (it blocks waiting to obtain the internal import lock) until
the original import completes. So until "import freeze" returns in
the main thread, no other thread can do an import.

Partly for that reason, it's generally a Horrible Idea to start a
thread as a side effect of importing a module. That's what freeze.py
does, and you get the expected deadlock as a result. The main thread
is hung waiting for "import freeze" to return, and the spawned thread
is hung at an import in Queue.__init__( ) waiting for the main thread
to release the import lock.
Invoking freezemain.py produces

yadda
yadda
yadda
yadda
yadda
yadda

i. e. consistently q.maxsize + 2. One item about to be put, one
already taken before Queue.Queue(). Deferring execution of the
module-level code until after the import

<nofreeze.py>
import Queue
import threading
import time

q = Queue.Queue(4)

def proc():
while True:
q.get(1)
Queue.Queue()
print "YADDA"

def start():
threading.Threa d(target=proc). start()

while True:
print "yadda"
q.put(None)
time.sleep(1)
</nofreeze.py>

<nofreezemain.p y>
import nofreeze
nofreeze.start( )
</nofreezemain.py >

and invoking nofreezemain.py produces
yadda
YADDA
yadda
YADDA
yadda
YADDA
yadda
YADDA

apparently ad infinitum.
Right. Note that this is the same reason threaded tests in Python's
standard regression suite define a 'test_main()' function, called by
the regrtest.py driver after import of the test's module completes.
It's generally suicidal to start a thread as a side effect of an
import.

....
Import _is_ a sensitive phase...
It's quite easy to avoid thread problems in imports: never start a
thread as a side effect of importing, and you'll never get a deadlock
due to importing.
As an experiment I moved

try:
import thread
except ImportError:
import dummy_thread as thread

from the Queue.Queue.__i nit__() method to the module body --
and now freezemain.py seems to work, too. So that would be an
easy remedy, but sure there is a reason why that import
statement is in such an unusual place?


I think you'd have to ask Brett (who did most of the work on
dummy_thread and dummy_threading ). It doesn't really matter, though:
it's a general truth that starting a thread as a side effect of
importing is a recipe for deadlock, and hacking specific methods and
functions to avoid imports just moves the problem around. It's not a
goal that anything in the standard Python library cater to bad thread
practice here (the bad thread practice being, again, starting a thread
as a side effect of importing).
Jul 18 '05 #5
Op 2004-12-12, Tim Peters schreef <ti********@gma il.com>:

I think you'd have to ask Brett (who did most of the work on
dummy_thread and dummy_threading ). It doesn't really matter, though:
it's a general truth that starting a thread as a side effect of
importing is a recipe for deadlock, and hacking specific methods and
functions to avoid imports just moves the problem around. It's not a
goal that anything in the standard Python library cater to bad thread
practice here (the bad thread practice being, again, starting a thread
as a side effect of importing).


I don't see why starting a thread as a side effect of importing is
bad thread practice. Sure python doesn't cater for it, but IMO
that seems to be python failing.

--
Antoon Pardon
Jul 18 '05 #6
[Antoon Pardon]
I don't see why starting a thread as a side effect of importing is
bad thread practice. Sure python doesn't cater for it, but IMO
that seems to be python failing.


Obviously, it's bad practice in Python because it can lead to
deadlocks in Python. It's nearly tautological. Import is an
executable statement in Python, not, e.g., as in many other languages,
a declaration directed at the system linker. With that power comes
opportunities to shoot yourself, although they're generally easy to
avoid. Come up with a practical design that doesn't have this
limitation, and then perhaps your characterizatio n of the current
design as "a failing" would be both credible and constructive.

Apart from that, ya, I do think it would *uisually* be poor practice
to start a thread as a side effect of importing anyway. It's too
mysterious, and IME has caused trouble even when it didn't lead to
deadlocks. The fundamental purpose of import in Python is to add
useful names to the importer's namespace, and users of a module
generally think of it as doing no more than that.

Note that the OP's example had a module that, upon the first attempt
to import it, ran an infinite loop (even if it hadn't deadlocked), and
it's clearly severe abuse of import's purpose.to write a module M such
that "import M" *never* returns. Indeed, that's the other half of how
deadlock occurs: not only that the imported module spawn a thread as
a side effect of importing, but also that the imported module refuse
to allow the import to complete.

The current design actually supports spawning all the threads you like
as a side effect of importing, provided you ensure also that the
import ompletes. The easiest way to avoid trouble remains not to
spawn threads as a side effect of importing to begin with, although a
programmer determined to demonstrate their bad taste <wink> can easily
enough make it work.
Jul 18 '05 #7
Op 2004-12-13, Tim Peters schreef <ti********@gma il.com>:
[Antoon Pardon]
I don't see why starting a thread as a side effect of importing is
bad thread practice. Sure python doesn't cater for it, but IMO
that seems to be python failing.
Obviously, it's bad practice in Python because it can lead to
deadlocks in Python.


By that argument any use of locks is bad practice because it
can lead to deadlock.
It's nearly tautological. Import is an
executable statement in Python, not, e.g., as in many other languages,
a declaration directed at the system linker. With that power comes
opportunities to shoot yourself, although they're generally easy to
avoid. Come up with a practical design that doesn't have this
limitation, and then perhaps your characterizatio n of the current
design as "a failing" would be both credible and constructive.
If a car model has cranky brakes, I think I can call that a failing
even without having the ability to come up with a pratical design
that doesn's has those limitations.

I judge a language by what it can and cannot do, not by my ability
to correct the things I perceive as failings. For all I know python
may have taken some design decisions that might have seen perfectly
logical but now prohibit a a practical design that doesn't have this
limitation. I don't see why something like that would make this
any less a failing then when a practical design was easy in the
current implemenation.
Apart from that, ya, I do think it would *uisually* be poor practice
to start a thread as a side effect of importing anyway. It's too
mysterious, and IME has caused trouble even when it didn't lead to
deadlocks. The fundamental purpose of import in Python is to add
useful names to the importer's namespace, and users of a module
generally think of it as doing no more than that.
Importing a module in general also does some kind of initialisation.
If starting a thread is a logical thing to do in this initialisation
fase I see nothing wrong with it.
Note that the OP's example had a module that, upon the first attempt
to import it, ran an infinite loop (even if it hadn't deadlocked), and
it's clearly severe abuse of import's purpose.to write a module M such
that "import M" *never* returns. Indeed, that's the other half of how
deadlock occurs: not only that the imported module spawn a thread as
a side effect of importing, but also that the imported module refuse
to allow the import to complete.
Well I'll agree here. An import that has as a side effect that the
import doesn't return is bad practice.
The current design actually supports spawning all the threads you like
as a side effect of importing, provided you ensure also that the
import ompletes.
Well in that case I don't have any problems with it. The perceived
failing was because of only knowing part of it based on what I had
read in this thread.
The easiest way to avoid trouble remains not to
spawn threads as a side effect of importing to begin with, although a
programmer determined to demonstrate their bad taste <wink> can easily
enough make it work.


Well then probably I have a bad taste. I have though of designs in which
it seemed very natural to have a thread started as part of the
initialisation in a module. Other limitations of python didn't make
it workable but in priciple I saw nothing wrong with doing it.

--
Antoon Pardon
Jul 18 '05 #8
Antoon Pardon wrote:
Op 2004-12-13, Tim Peters schreef <ti********@gma il.com>:
[Antoon Pardon]
I don't see why starting a thread as a side effect of importing is
bad thread practice. Sure python doesn't cater for it, but IMO
that seems to be python failing.
Obviously, it's bad practice in Python because it can lead to
deadlocks in Python.

By that argument any use of locks is bad practice because it
can lead to deadlock.

Not at all. You mentioned locks, not Tim. The reason it can lead to
deadlocks is because import, being an executable statement, must
terminate correctly in order to return control to the module that
executed the import statement.
It's nearly tautological. Import is an
executable statement in Python, not, e.g., as in many other languages,
a declaration directed at the system linker. With that power comes
opportuniti es to shoot yourself, although they're generally easy to
avoid. Come up with a practical design that doesn't have this
limitation, and then perhaps your characterizatio n of the current
design as "a failing" would be both credible and constructive.

If a car model has cranky brakes, I think I can call that a failing
even without having the ability to come up with a pratical design
that doesn's has those limitations.

But in fact your situation is more closely analogous to a customer who's
bought a car that can be stopped by pressing on the brake pedal now
complaining that sideways pressure on the brake pedal doesn;t affect the
car's motion.
I judge a language by what it can and cannot do, not by my ability
to correct the things I perceive as failings. For all I know python
may have taken some design decisions that might have seen perfectly
logical but now prohibit a a practical design that doesn't have this
limitation. I don't see why something like that would make this
any less a failing then when a practical design was easy in the
current implemenation.
All that Tim was suggesting is that it's MORE SENSIBLE to start a thread
as the result of a specific call to programmed functionality rather than
as the side effect of an import. The reason for this is due to the
semantics of the import statement. If you perceive that as a failing
then you'd be better rewarded by an attempt to modify your perceptions.

I've always found "don't argue with Tim about Python" to be a useful
rule of thumb. He's wrong much less often than I am. I suspect he's also
wrong much less often than you ;-)
Apart from that, ya, I do think it would *uisually* be poor practice
to start a thread as a side effect of importing anyway. It's too
mysterious, and IME has caused trouble even when it didn't lead to
deadlocks. The fundamental purpose of import in Python is to add
useful names to the importer's namespace, and users of a module
generally think of it as doing no more than that.

Importing a module in general also does some kind of initialisation.
If starting a thread is a logical thing to do in this initialisation
fase I see nothing wrong with it.

But the reason, I suspect, that it's being suggested this is bad design
is because it would be tricky (in any language) to ensure that the
satisfactory conclusion of the import didn't depend on successful thread
startup (and possibly termination).

Plus, of course, in Python the side-effect of the import (creating the
thread) can happen precisely once, because the module body is executed
precisely once no matter how many times it's imported. That's not really
the issue, however.
Note that the OP's example had a module that, upon the first attempt
to import it, ran an infinite loop (even if it hadn't deadlocked), and
it's clearly severe abuse of import's purpose.to write a module M such
that "import M" *never* returns. Indeed, that's the other half of how
deadlock occurs: not only that the imported module spawn a thread as
a side effect of importing, but also that the imported module refuse
to allow the import to complete.

Well I'll agree here. An import that has as a side effect that the
import doesn't return is bad practice.

But that's precisely the risk you run when starting up threads!
The current design actually supports spawning all the threads you like
as a side effect of importing, provided you ensure also that the
import ompletes.

Well in that case I don't have any problems with it. The perceived
failing was because of only knowing part of it based on what I had
read in this thread.

The easiest way to avoid trouble remains not to
spawn threads as a side effect of importing to begin with, although a
programmer determined to demonstrate their bad taste <wink> can easily
enough make it work.

Well then probably I have a bad taste. I have though of designs in which
it seemed very natural to have a thread started as part of the
initialisation in a module. Other limitations of python didn't make
it workable but in priciple I saw nothing wrong with doing it.

Bearing in mind that module initialisation is always one-off, relying on
import to trigger such complex behavior is probably a design that will
mislead users into false expectations.

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119
Jul 18 '05 #9
Op 2004-12-14, Steve Holden schreef <st***@holdenwe b.com>:
Antoon Pardon wrote:
Op 2004-12-13, Tim Peters schreef <ti********@gma il.com>:
[Antoon Pardon]

I don't see why starting a thread as a side effect of importing is
bad thread practice. Sure python doesn't cater for it, but IMO
that seems to be python failing.

Obviously, it's bad practice in Python because it can lead to
deadlocks in Python.

By that argument any use of locks is bad practice because it
can lead to deadlock.

Not at all. You mentioned locks, not Tim.


That is beside the point. The argument was that starting a thread
as a side effect of importing is bad practice because it can lead to
deadlock. This suggest that a general condition for something
to be bad practice is if that something can lead to deadlock.
Locks are in that case.
It's nearly tautological. Import is an
executable statement in Python, not, e.g., as in many other languages,
a declaration directed at the system linker. With that power comes
opportunitie s to shoot yourself, although they're generally easy to
avoid. Come up with a practical design that doesn't have this
limitation , and then perhaps your characterizatio n of the current
design as "a failing" would be both credible and constructive.

If a car model has cranky brakes, I think I can call that a failing
even without having the ability to come up with a pratical design
that doesn's has those limitations.

But in fact your situation is more closely analogous to a customer who's
bought a car that can be stopped by pressing on the brake pedal now
complaining that sideways pressure on the brake pedal doesn;t affect the
car's motion.


You will have to explain how you come by that analogy.
I judge a language by what it can and cannot do, not by my ability
to correct the things I perceive as failings. For all I know python
may have taken some design decisions that might have seen perfectly
logical but now prohibit a a practical design that doesn't have this
limitation. I don't see why something like that would make this
any less a failing then when a practical design was easy in the
current implemenation.

All that Tim was suggesting is that it's MORE SENSIBLE to start a thread
as the result of a specific call to programmed functionality rather than
as the side effect of an import. The reason for this is due to the
semantics of the import statement. If you perceive that as a failing
then you'd be better rewarded by an attempt to modify your perceptions.

I've always found "don't argue with Tim about Python" to be a useful
rule of thumb. He's wrong much less often than I am. I suspect he's also
wrong much less often than you ;-)


With all respect I find that lousy advise. I don't mind that it will turn
out I'll be wrong most of the time. But I'll probably will have gained
a better understanding by the responses to my argument than by merely
accepting the word of Tim.

[ ... ]
Note that the OP's example had a module that, upon the first attempt
to import it, ran an infinite loop (even if it hadn't deadlocked), and
it's clearly severe abuse of import's purpose.to write a module M such
that "import M" *never* returns. Indeed, that's the other half of how
deadlock occurs: not only that the imported module spawn a thread as
a side effect of importing, but also that the imported module refuse
to allow the import to complete.

Well I'll agree here. An import that has as a side effect that the
import doesn't return is bad practice.

But that's precisely the risk you run when starting up threads!


Now you are confusing me. Is it a problem of an import that doesn't
return of is it a case of a race condition where the import has to
return in a timely fashion?

--
Antoon Pardon
Jul 18 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
5275
by: Ajay | last post by:
hi! my application consists of a GUI with a number of functions. One of these runs a server in a separate thread. the thread is started and given the function start_server to execute the problem is when within start_server i try to display anything using tkMessageBox, nothing gets displayed and my application crashes. Any ideas what i am doing wrong and how to correct it thanks
44
2364
by: Charles Law | last post by:
Hi guys. I'm back on the threading gig again. It's the age-old question about waiting for something to happen without wasting time doing it. Take two threads: the main thread and a worker thread. The worker thread is reading the serial port, waiting for something to happen (a service request). When it does it raises an event. Of course, the event is executed on the worker thread. The idea is that when the event is raised, the handler...
0
1619
by: Dave Coate | last post by:
I am working on a generic way to launch multiple similar processes (threads) at once, but limit the number of threads running at any one time to a number I set. As I understand it the following line makes a Queue "thread safe", so I do not need to explicitly lock and unlock it when multiple threads are working with it. 'Thread safe queue Private IndexQueue As Queue = Queue.Synchronized(New Queue)
6
16714
by: les | last post by:
Here's a class which uses 2.0 generics to implement an inter-thread message queue in C#. Any number of threads can post and read from the queue simultaneously, and the message object can be any type. There's a test driver at the bottom which demonstrates usage. /*-----------------------------------------------------------------------------------------------------*/ using System; using System.Collections.Generic;
6
2864
by: seb | last post by:
Hi, I am using pygtk for the first times. I am wondering what would be the best "pattern" to interface pygtk with a thread. The thread is collecting informations (over the network for example) or is doing some long calculations.
5
3803
by: admin | last post by:
ok This is my main. Pretty much it goes through each category and starts up 4 worker threads that then ask for groups to gether from. My problem is that when the thread gets done it keeps the mysql connections open so I end up with quite a few at the end. Is there a different way that I should do this? class Program { static string categories = { "emulation" , "audio" , "console" , "anime" , "xxx" , "tv" , "pictures" , "video" };
5
1788
by: P.J.M. Beker | last post by:
Hi there, I'm currently writing a program in which I use the FileMonitor to monitor a folder in which I store downloaded images. I know that I can't add much coding in the filemonitor's event in risk of losing some new entries, so I've deceided to create an update thread. This thread is created when the program's start and should (for various reason) run not in sync with the Filemonitor. The Filemonitor event creates an entry in a...
29
9123
by: NvrBst | last post by:
I've read a bit online seeing that two writes are not safe, which I understand, but would 1 thread push()'ing and 1 thread pop()'ing be thread-safe? Basically my situation is the follows: --Thread 1-- 1. Reads TCPIP Buffer 2. Adds Buffer to Queue (q.size() to check queue isn't full, and q.push_back(...)) 3. Signals Reading Thread Event & Goes Back to Wait for More Messages on TCPIP
12
2078
by: Ronny | last post by:
Thanks Chris, Looks nice but I miss the dual way communication. In the main thread to deliver paramters and data to the worker thread- how can I do that? Regards Ronny Take a look at the background worker thread component. It has what you want built into it. http://msdn.microsoft.com/en-us/library/8xs8549b.aspx
0
8697
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9158
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
9060
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
7921
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6615
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
4454
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4712
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3151
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
3
2096
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.