473,320 Members | 1,694 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

thread/queue bug

And sorry I got ticked, frustrating week
And I could help more, being fairly experienced with
threading issues and race conditions and such, but
as I tried to indicate in the first place, you've
provided next to no useful (IMHO) information to
let anyone help you more than this


This is about 5% of the code.
Uses no locks.
I am mystified, I have written probably 100,000 lines
of Python and never seen a thread just lock up and quit
running. It happens on a Queue() statement so my suspicion
is a bug. ??

I have kludged around it by putting all the thread/queue stuff
in the main program and import the stuff I don't want to
distribute. But mysteries haunt your dreams, sooo...
#!/usr/local/bin/python

# pnetx.py

from threading import *
from time import *
from Queue import Queue
from socket import *
import sys
import os

# glob is a DUMMY CLASS

glob.listenq = Queue(1000)

def listener():
while 1:
msg,addrport = listenersocket.recvfrom(BUFSIZE)
addr = addrport[0]
glob.listenq.put (( msg,addr),1)
if msg == 'DIE': break

def procmsgs():
while 1:
msg,addr = glob.listenq.get(1)
if msg == 'DIE': break
wds = msg.split(';')
if wds[0] == 'S': recvsucc(msg); continue
if wds[0] == 'E': recvevent(msg); continue
if wds[0] == 'ACK': recvack(msg[4:]); continue
if wds[0] == 'MONITOR':
if addr not in monitorlist:
print 'This line ALWAYS PRINTS'
queuelist.append( Queue(0) )
## The above fails if this code is imported
## It doesn't matter whether it is imported
## as .py or .pyc
## also mq = Queue(0); queuelist.append(mq) # fails
print 'This line ALWAYS PRINTS if i execute this source'
print 'but NEVER PRINTS if I import '
print 'this source from a 1 line program'
print 'Thats weird'
monitoron.append( 1 )
monitorlist.append( addr )

queuelist = [Queue(0)]
listenersocket = socket(AF_INET,SOCK_DGRAM)
listenersocket.bind(ListenAddr)

procm = Thread(target=procmsgs)
procm.start()
listenr = Thread(target=listener)
listenr.start()

## Then start a bunch of other threads and queuea.
Don't spend a lot of time on this, not worth it.
I just thought someone might have experienced
something similar.

Jul 18 '05 #1
9 2746
phil wrote:
Uses no locks.
It does use locks implicitly, though, since even just
importing threading will do that, and creating a Queue
does too.
I am mystified, I have written probably 100,000 lines
of Python and never seen a thread just lock up and quit
running. It happens on a Queue() statement so my suspicion
is a bug. ??
You have the source to Queue.py in your standard library
folder. Why not throw a few more print statements into
its __init__ and see what you learn?
I have kludged around it by putting all the thread/queue stuff
in the main program and import the stuff I don't want to
distribute. But mysteries haunt your dreams, sooo...
#!/usr/local/bin/python

[snip source]

I cobbled together a "working" version of your code
and ran it just fine, whether imported or run directly.
No lockups. On Windows XP.

Are you by any chance running on a new version of the
Linux kernel, where the threading model has changed?
(Or was it just RedHat 9.0?)

I don't know the details, but I know folks have had trouble
with this and Python... For example, I found this
reference to the issue:
http://groups.google.ca/groups?th=fe...line.com#link2

-Peter
Jul 18 '05 #2
On Fri, 10 Dec 2004 16:18:51 -0600, phil <ph***********@anvilcom.com> wrote:
And sorry I got ticked, frustrating week
threading problems can do that ;-)

You are obviusly deeper into this than I can get from a cursory scan,
but I'll make some general debugging comments ;-)
And I could help more, being fairly experienced with
threading issues and race conditions and such, but
as I tried to indicate in the first place, you've
provided next to no useful (IMHO) information to
let anyone help you more than this


This is about 5% of the code.
Uses no locks.
I am mystified, I have written probably 100,000 lines
of Python and never seen a thread just lock up and quit
running. It happens on a Queue() statement so my suspicion
is a bug. ??

Or a once-in-a-blue-moon resource access deadlock of some kind?

I have kludged around it by putting all the thread/queue stuff
in the main program and import the stuff I don't want to
distribute. But mysteries haunt your dreams, sooo...
#!/usr/local/bin/python

# pnetx.py

from threading import *
from time import *
from Queue import Queue
from socket import * Do you really need to import * ? Though the above should be safe import-wise
you are setting yourself up for name-clash problems by putting so many in
the same space. E.g., if something happened to match a misspelling typo in
your program, you wouldn't get a name error exception. Etc., etc.

One of the first rules of debugging is to eliminate the unnecessary ;-)
At least there doesn't seem to be name clashes between the above (except
starting with '_' which shouldn't get imported with *
d = {}
for imp in 'threading time Queue socket'.split(): ... m = __import__(imp)
... for name in m.__dict__.keys():
... d.setdefault(name, []).append(imp)
... for k,v in d.items(): ... if len(v)!=1: print k,v
...
_sleep ['threading', 'Queue']
__file__ ['threading', 'Queue', 'socket']
__all__ ['threading', 'Queue', 'socket']
__builtins__ ['threading', 'Queue', 'socket']
__name__ ['threading', 'time', 'Queue', 'socket']
_time ['threading', 'Queue']
__doc__ ['threading', 'time', 'Queue', 'socket']

Just verifying an assumption ;-)
OTOH, do you really need (ignoring wid and name and the first content of dir()):
dir() ['__builtins__', '__doc__', '__name__'] from threading import *
from time import *
from Queue import *
from socket import *
wid = 0
for name in dir(): ... print repr(name),
... wid += len(repr(name))+1
... if wid>60:
... print
... wid = 0
...
'AF_APPLETALK' 'AF_INET' 'AF_IPX' 'AF_UNSPEC' 'AI_ADDRCONFIG'
'AI_ALL' 'AI_CANONNAME' 'AI_DEFAULT' 'AI_MASK' 'AI_NUMERICHOST'
'AI_PASSIVE' 'AI_V4MAPPED' 'AI_V4MAPPED_CFG' 'BoundedSemaphore'
'CAPI' 'Condition' 'EAI_ADDRFAMILY' 'EAI_AGAIN' 'EAI_BADFLAGS'
'EAI_BADHINTS' 'EAI_FAIL' 'EAI_FAMILY' 'EAI_MAX' 'EAI_MEMORY'
'EAI_NODATA' 'EAI_NONAME' 'EAI_PROTOCOL' 'EAI_SERVICE' 'EAI_SOCKTYPE'
'EAI_SYSTEM' 'Empty' 'Event' 'Full' 'INADDR_ALLHOSTS_GROUP' 'INADDR_ANY'
'INADDR_BROADCAST' 'INADDR_LOOPBACK' 'INADDR_MAX_LOCAL_GROUP'
'INADDR_NONE' 'INADDR_UNSPEC_GROUP' 'IPPORT_RESERVED' 'IPPORT_USERRESERVED'
'IPPROTO_GGP' 'IPPROTO_ICMP' 'IPPROTO_IDP' 'IPPROTO_IGMP' 'IPPROTO_IP'
'IPPROTO_MAX' 'IPPROTO_ND' 'IPPROTO_PUP' 'IPPROTO_RAW' 'IPPROTO_TCP'
'IPPROTO_UDP' 'IP_ADD_MEMBERSHIP' 'IP_DEFAULT_MULTICAST_LOOP'
'IP_DEFAULT_MULTICAST_TTL' 'IP_DROP_MEMBERSHIP' 'IP_MAX_MEMBERSHIPS'
'IP_MULTICAST_IF' 'IP_MULTICAST_LOOP' 'IP_MULTICAST_TTL' 'IP_OPTIONS'
'IP_TOS' 'IP_TTL' 'Lock' 'MSG_DONTROUTE' 'MSG_OOB' 'MSG_PEEK'
'NI_DGRAM' 'NI_MAXHOST' 'NI_MAXSERV' 'NI_NAMEREQD' 'NI_NOFQDN'
'NI_NUMERICHOST' 'NI_NUMERICSERV' 'Queue' 'RAND_add' 'RAND_egd'
'RAND_status' 'RLock' 'SOCK_DGRAM' 'SOCK_RAW' 'SOCK_RDM' 'SOCK_SEQPACKET'
'SOCK_STREAM' 'SOL_IP' 'SOL_SOCKET' 'SOL_TCP' 'SOL_UDP' 'SOMAXCONN'
'SO_ACCEPTCONN' 'SO_BROADCAST' 'SO_DEBUG' 'SO_DONTROUTE' 'SO_ERROR'
'SO_KEEPALIVE' 'SO_LINGER' 'SO_OOBINLINE' 'SO_RCVBUF' 'SO_RCVLOWAT'
'SO_RCVTIMEO' 'SO_REUSEADDR' 'SO_SNDBUF' 'SO_SNDLOWAT' 'SO_SNDTIMEO'
'SO_TYPE' 'SO_USELOOPBACK' 'SSLType' 'SSL_ERROR_EOF' 'SSL_ERROR_INVALID_ERROR_CODE'
'SSL_ERROR_SSL' 'SSL_ERROR_SYSCALL' 'SSL_ERROR_WANT_CONNECT'
'SSL_ERROR_WANT_READ' 'SSL_ERROR_WANT_WRITE' 'SSL_ERROR_WANT_X509_LOOKUP'
'SSL_ERROR_ZERO_RETURN' 'Semaphore' 'SocketType' 'TCP_NODELAY'
'Thread' 'Timer' '__builtins__' '__doc__' '__name__' 'accept2dyear'
'activeCount' 'altzone' 'asctime' 'clock' 'ctime' 'currentThread'
'daylight' 'enumerate' 'error' 'errorTab' 'gaierror' 'getaddrinfo'
'getdefaulttimeout' 'getfqdn' 'gethostbyaddr' 'gethostbyname'
'gethostbyname_ex' 'gethostname' 'getnameinfo' 'getprotobyname'
'getservbyname' 'gmtime' 'has_ipv6' 'herror' 'htonl' 'htons'
'inet_aton' 'inet_ntoa' 'localtime' 'mktime' 'ntohl' 'ntohs'
'setdefaulttimeout' 'setprofile' 'settrace' 'sleep' 'socket'
'ssl' 'sslerror' 'strftime' 'strptime' 'struct_time' 'time' 'timeout'
'timezone' 'tzname' 'wid'


(Hm, should have pre-tested wid+len(current) to limit width ;-)
import sys
import os

# glob is a DUMMY CLASS where from? accidentally snipped?
glob.listenq = Queue(1000)

def listener():
while 1:
msg,addrport = listenersocket.recvfrom(BUFSIZE)
addr = addrport[0] what is the usage level on glob.listenq? It is finite 1000, so
you could theoretically block on the put. What do you get
if you print glob.listenq.qsize here?
glob.listenq.put (( msg,addr),1)
if msg == 'DIE': break

def procmsgs():
while 1:
msg,addr = glob.listenq.get(1)
if msg == 'DIE': break Might be interesting to get some output when 'DIE' is recognized. Above too.
wds = msg.split(';') assert wds[0] in ['S', 'E', 'ACK', 'MONITOR'] # check assumptions. E.g.,
# split on ';' means spaces around
# the ';' are not eliminated if wds[0] == 'S': recvsucc(msg); continue
if wds[0] == 'E': recvevent(msg); continue
if wds[0] == 'ACK': recvack(msg[4:]); continue
if wds[0] == 'MONITOR':
if addr not in monitorlist:
print 'This line ALWAYS PRINTS'
queuelist.append( Queue(0) ) Why are you apparently creating *new* queues in this thread that you are not using
and appending them to queuelist in the main thread? Are you testing to see how many
can be created, or for some Queue creation bug?

What do you get if you print 'queuelist length = %s'%len(queuelist) here?
How often are you getting 'MONITOR'?
## The above fails if this code is imported Imported from where? Do you mean that this code works if it
is embedded in some larger code, and if you put an import of this
at the top of that code it doesn't work? That would not only be
moving the source, but also the point of invocation. I.e., the
import *executes* what you import once, putting the result in
the module's global name space (other than code that explicitly
accesses other modules etc.) Do you need to wrap part of this
in a function that you can call *after* importing the definitions?

If your real app has multiple threads accessing a main thread queuelist
without a lock, you might want to look closely and see if that needs
to be a queue mechanism itself.
## It doesn't matter whether it is imported
## as .py or .pyc
## also mq = Queue(0); queuelist.append(mq) # fails
print 'This line ALWAYS PRINTS if i execute this source'
print 'but NEVER PRINTS if I import '
print 'this source from a 1 line program'
print 'Thats weird'
monitoron.append( 1 )
monitorlist.append( addr )
Do you want all the following to be executed once at the point of importing this?
queuelist = [Queue(0)] What is this for? (oops, I hope I restored the above line correctly)

listenersocket = socket(AF_INET,SOCK_DGRAM)
listenersocket.bind(ListenAddr)

procm = Thread(target=procmsgs)
procm.start()
listenr = Thread(target=listener)
listenr.start()

## Then start a bunch of other threads and queuea.
Don't spend a lot of time on this, not worth it.
I just thought someone might have experienced
something similar.


Just some thoughts. HTH.
Regards,
Bengt Richter
Jul 18 '05 #3
phil wrote:
And sorry I got ticked, frustrating week
>And I could help more, being fairly experienced with
>threading issues and race conditions and such, but
>as I tried to indicate in the first place, you've
>provided next to no useful (IMHO) information to
>let anyone help you more than this
This is about 5% of the code.
Uses no locks.


A lock is acquired in Queue.__init__().
I am mystified, I have written probably 100,000 lines
of Python and never seen a thread just lock up and quit
running. It happens on a Queue() statement so my suspicion
is a bug. ??

I have kludged around it by putting all the thread/queue stuff
in the main program and import the stuff I don't want to
distribute. But mysteries haunt your dreams, sooo...


What I believe to be a minimal example:

<freeze.py>
import Queue
import threading
import time

q = Queue.Queue(4)

def proc():
while True:
q.get(1)
Queue.Queue()
print "YADDA"

threading.Thread(target=proc).start()

while True:
print "yadda"
q.put(None)
time.sleep(1)
</freeze.py>

<freezemain.py>
import freeze
</freezemain.py>

Invoking freezemain.py produces

yadda
yadda
yadda
yadda
yadda
yadda

i. e. consistently q.maxsize + 2. One item about to be put, one already
taken before Queue.Queue(). Deferring execution of the module-level code
until after the import

<nofreeze.py>
import Queue
import threading
import time

q = Queue.Queue(4)

def proc():
while True:
q.get(1)
Queue.Queue()
print "YADDA"

def start():
threading.Thread(target=proc).start()

while True:
print "yadda"
q.put(None)
time.sleep(1)
</nofreeze.py>

<nofreezemain.py>
import nofreeze
nofreeze.start()
</nofreezemain.py>

and invoking nofreezemain.py produces
yadda
YADDA
yadda
YADDA
yadda
YADDA
yadda
YADDA

apparently ad infinitum. Conclusion? None so far, though you might welcome
the confirmation that this is not a "once in a blue moon" accident as Bengt
Richter surmised.

Import _is_ a sensitive phase...

As an experiment I moved

try:
import thread
except ImportError:
import dummy_thread as thread

from the Queue.Queue.__init__() method to the module body -- and now
freezemain.py seems to work, too. So that would be an easy remedy, but sure
there is a reason why that import statement is in such an unusual place?

Peter


Jul 18 '05 #4
[Peter Otten]
What I believe to be a minimal example:

<freeze.py>
import Queue
import threading
import time

q = Queue.Queue(4)

def proc():
while True:
q.get(1)
Queue.Queue()
print "YADDA"

threading.Thread(target=proc).start()

while True:
print "yadda"
q.put(None)
time.sleep(1)
</freeze.py>

<freezemain.py>
import freeze
</freezemain.py>
CPython has an internal, reentrant import lock. When a thread does an
import, it acquires this lock, and the lock remains held until that
import is complete. As a consequence, no *other* thread can do an
import (it blocks waiting to obtain the internal import lock) until
the original import completes. So until "import freeze" returns in
the main thread, no other thread can do an import.

Partly for that reason, it's generally a Horrible Idea to start a
thread as a side effect of importing a module. That's what freeze.py
does, and you get the expected deadlock as a result. The main thread
is hung waiting for "import freeze" to return, and the spawned thread
is hung at an import in Queue.__init__() waiting for the main thread
to release the import lock.
Invoking freezemain.py produces

yadda
yadda
yadda
yadda
yadda
yadda

i. e. consistently q.maxsize + 2. One item about to be put, one
already taken before Queue.Queue(). Deferring execution of the
module-level code until after the import

<nofreeze.py>
import Queue
import threading
import time

q = Queue.Queue(4)

def proc():
while True:
q.get(1)
Queue.Queue()
print "YADDA"

def start():
threading.Thread(target=proc).start()

while True:
print "yadda"
q.put(None)
time.sleep(1)
</nofreeze.py>

<nofreezemain.py>
import nofreeze
nofreeze.start()
</nofreezemain.py>

and invoking nofreezemain.py produces
yadda
YADDA
yadda
YADDA
yadda
YADDA
yadda
YADDA

apparently ad infinitum.
Right. Note that this is the same reason threaded tests in Python's
standard regression suite define a 'test_main()' function, called by
the regrtest.py driver after import of the test's module completes.
It's generally suicidal to start a thread as a side effect of an
import.

....
Import _is_ a sensitive phase...
It's quite easy to avoid thread problems in imports: never start a
thread as a side effect of importing, and you'll never get a deadlock
due to importing.
As an experiment I moved

try:
import thread
except ImportError:
import dummy_thread as thread

from the Queue.Queue.__init__() method to the module body --
and now freezemain.py seems to work, too. So that would be an
easy remedy, but sure there is a reason why that import
statement is in such an unusual place?


I think you'd have to ask Brett (who did most of the work on
dummy_thread and dummy_threading). It doesn't really matter, though:
it's a general truth that starting a thread as a side effect of
importing is a recipe for deadlock, and hacking specific methods and
functions to avoid imports just moves the problem around. It's not a
goal that anything in the standard Python library cater to bad thread
practice here (the bad thread practice being, again, starting a thread
as a side effect of importing).
Jul 18 '05 #5
Op 2004-12-12, Tim Peters schreef <ti********@gmail.com>:

I think you'd have to ask Brett (who did most of the work on
dummy_thread and dummy_threading). It doesn't really matter, though:
it's a general truth that starting a thread as a side effect of
importing is a recipe for deadlock, and hacking specific methods and
functions to avoid imports just moves the problem around. It's not a
goal that anything in the standard Python library cater to bad thread
practice here (the bad thread practice being, again, starting a thread
as a side effect of importing).


I don't see why starting a thread as a side effect of importing is
bad thread practice. Sure python doesn't cater for it, but IMO
that seems to be python failing.

--
Antoon Pardon
Jul 18 '05 #6
[Antoon Pardon]
I don't see why starting a thread as a side effect of importing is
bad thread practice. Sure python doesn't cater for it, but IMO
that seems to be python failing.


Obviously, it's bad practice in Python because it can lead to
deadlocks in Python. It's nearly tautological. Import is an
executable statement in Python, not, e.g., as in many other languages,
a declaration directed at the system linker. With that power comes
opportunities to shoot yourself, although they're generally easy to
avoid. Come up with a practical design that doesn't have this
limitation, and then perhaps your characterization of the current
design as "a failing" would be both credible and constructive.

Apart from that, ya, I do think it would *uisually* be poor practice
to start a thread as a side effect of importing anyway. It's too
mysterious, and IME has caused trouble even when it didn't lead to
deadlocks. The fundamental purpose of import in Python is to add
useful names to the importer's namespace, and users of a module
generally think of it as doing no more than that.

Note that the OP's example had a module that, upon the first attempt
to import it, ran an infinite loop (even if it hadn't deadlocked), and
it's clearly severe abuse of import's purpose.to write a module M such
that "import M" *never* returns. Indeed, that's the other half of how
deadlock occurs: not only that the imported module spawn a thread as
a side effect of importing, but also that the imported module refuse
to allow the import to complete.

The current design actually supports spawning all the threads you like
as a side effect of importing, provided you ensure also that the
import ompletes. The easiest way to avoid trouble remains not to
spawn threads as a side effect of importing to begin with, although a
programmer determined to demonstrate their bad taste <wink> can easily
enough make it work.
Jul 18 '05 #7
Op 2004-12-13, Tim Peters schreef <ti********@gmail.com>:
[Antoon Pardon]
I don't see why starting a thread as a side effect of importing is
bad thread practice. Sure python doesn't cater for it, but IMO
that seems to be python failing.
Obviously, it's bad practice in Python because it can lead to
deadlocks in Python.


By that argument any use of locks is bad practice because it
can lead to deadlock.
It's nearly tautological. Import is an
executable statement in Python, not, e.g., as in many other languages,
a declaration directed at the system linker. With that power comes
opportunities to shoot yourself, although they're generally easy to
avoid. Come up with a practical design that doesn't have this
limitation, and then perhaps your characterization of the current
design as "a failing" would be both credible and constructive.
If a car model has cranky brakes, I think I can call that a failing
even without having the ability to come up with a pratical design
that doesn's has those limitations.

I judge a language by what it can and cannot do, not by my ability
to correct the things I perceive as failings. For all I know python
may have taken some design decisions that might have seen perfectly
logical but now prohibit a a practical design that doesn't have this
limitation. I don't see why something like that would make this
any less a failing then when a practical design was easy in the
current implemenation.
Apart from that, ya, I do think it would *uisually* be poor practice
to start a thread as a side effect of importing anyway. It's too
mysterious, and IME has caused trouble even when it didn't lead to
deadlocks. The fundamental purpose of import in Python is to add
useful names to the importer's namespace, and users of a module
generally think of it as doing no more than that.
Importing a module in general also does some kind of initialisation.
If starting a thread is a logical thing to do in this initialisation
fase I see nothing wrong with it.
Note that the OP's example had a module that, upon the first attempt
to import it, ran an infinite loop (even if it hadn't deadlocked), and
it's clearly severe abuse of import's purpose.to write a module M such
that "import M" *never* returns. Indeed, that's the other half of how
deadlock occurs: not only that the imported module spawn a thread as
a side effect of importing, but also that the imported module refuse
to allow the import to complete.
Well I'll agree here. An import that has as a side effect that the
import doesn't return is bad practice.
The current design actually supports spawning all the threads you like
as a side effect of importing, provided you ensure also that the
import ompletes.
Well in that case I don't have any problems with it. The perceived
failing was because of only knowing part of it based on what I had
read in this thread.
The easiest way to avoid trouble remains not to
spawn threads as a side effect of importing to begin with, although a
programmer determined to demonstrate their bad taste <wink> can easily
enough make it work.


Well then probably I have a bad taste. I have though of designs in which
it seemed very natural to have a thread started as part of the
initialisation in a module. Other limitations of python didn't make
it workable but in priciple I saw nothing wrong with doing it.

--
Antoon Pardon
Jul 18 '05 #8
Antoon Pardon wrote:
Op 2004-12-13, Tim Peters schreef <ti********@gmail.com>:
[Antoon Pardon]
I don't see why starting a thread as a side effect of importing is
bad thread practice. Sure python doesn't cater for it, but IMO
that seems to be python failing.
Obviously, it's bad practice in Python because it can lead to
deadlocks in Python.

By that argument any use of locks is bad practice because it
can lead to deadlock.

Not at all. You mentioned locks, not Tim. The reason it can lead to
deadlocks is because import, being an executable statement, must
terminate correctly in order to return control to the module that
executed the import statement.
It's nearly tautological. Import is an
executable statement in Python, not, e.g., as in many other languages,
a declaration directed at the system linker. With that power comes
opportunities to shoot yourself, although they're generally easy to
avoid. Come up with a practical design that doesn't have this
limitation, and then perhaps your characterization of the current
design as "a failing" would be both credible and constructive.

If a car model has cranky brakes, I think I can call that a failing
even without having the ability to come up with a pratical design
that doesn's has those limitations.

But in fact your situation is more closely analogous to a customer who's
bought a car that can be stopped by pressing on the brake pedal now
complaining that sideways pressure on the brake pedal doesn;t affect the
car's motion.
I judge a language by what it can and cannot do, not by my ability
to correct the things I perceive as failings. For all I know python
may have taken some design decisions that might have seen perfectly
logical but now prohibit a a practical design that doesn't have this
limitation. I don't see why something like that would make this
any less a failing then when a practical design was easy in the
current implemenation.
All that Tim was suggesting is that it's MORE SENSIBLE to start a thread
as the result of a specific call to programmed functionality rather than
as the side effect of an import. The reason for this is due to the
semantics of the import statement. If you perceive that as a failing
then you'd be better rewarded by an attempt to modify your perceptions.

I've always found "don't argue with Tim about Python" to be a useful
rule of thumb. He's wrong much less often than I am. I suspect he's also
wrong much less often than you ;-)
Apart from that, ya, I do think it would *uisually* be poor practice
to start a thread as a side effect of importing anyway. It's too
mysterious, and IME has caused trouble even when it didn't lead to
deadlocks. The fundamental purpose of import in Python is to add
useful names to the importer's namespace, and users of a module
generally think of it as doing no more than that.

Importing a module in general also does some kind of initialisation.
If starting a thread is a logical thing to do in this initialisation
fase I see nothing wrong with it.

But the reason, I suspect, that it's being suggested this is bad design
is because it would be tricky (in any language) to ensure that the
satisfactory conclusion of the import didn't depend on successful thread
startup (and possibly termination).

Plus, of course, in Python the side-effect of the import (creating the
thread) can happen precisely once, because the module body is executed
precisely once no matter how many times it's imported. That's not really
the issue, however.
Note that the OP's example had a module that, upon the first attempt
to import it, ran an infinite loop (even if it hadn't deadlocked), and
it's clearly severe abuse of import's purpose.to write a module M such
that "import M" *never* returns. Indeed, that's the other half of how
deadlock occurs: not only that the imported module spawn a thread as
a side effect of importing, but also that the imported module refuse
to allow the import to complete.

Well I'll agree here. An import that has as a side effect that the
import doesn't return is bad practice.

But that's precisely the risk you run when starting up threads!
The current design actually supports spawning all the threads you like
as a side effect of importing, provided you ensure also that the
import ompletes.

Well in that case I don't have any problems with it. The perceived
failing was because of only knowing part of it based on what I had
read in this thread.

The easiest way to avoid trouble remains not to
spawn threads as a side effect of importing to begin with, although a
programmer determined to demonstrate their bad taste <wink> can easily
enough make it work.

Well then probably I have a bad taste. I have though of designs in which
it seemed very natural to have a thread started as part of the
initialisation in a module. Other limitations of python didn't make
it workable but in priciple I saw nothing wrong with doing it.

Bearing in mind that module initialisation is always one-off, relying on
import to trigger such complex behavior is probably a design that will
mislead users into false expectations.

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119
Jul 18 '05 #9
Op 2004-12-14, Steve Holden schreef <st***@holdenweb.com>:
Antoon Pardon wrote:
Op 2004-12-13, Tim Peters schreef <ti********@gmail.com>:
[Antoon Pardon]

I don't see why starting a thread as a side effect of importing is
bad thread practice. Sure python doesn't cater for it, but IMO
that seems to be python failing.

Obviously, it's bad practice in Python because it can lead to
deadlocks in Python.

By that argument any use of locks is bad practice because it
can lead to deadlock.

Not at all. You mentioned locks, not Tim.


That is beside the point. The argument was that starting a thread
as a side effect of importing is bad practice because it can lead to
deadlock. This suggest that a general condition for something
to be bad practice is if that something can lead to deadlock.
Locks are in that case.
It's nearly tautological. Import is an
executable statement in Python, not, e.g., as in many other languages,
a declaration directed at the system linker. With that power comes
opportunities to shoot yourself, although they're generally easy to
avoid. Come up with a practical design that doesn't have this
limitation, and then perhaps your characterization of the current
design as "a failing" would be both credible and constructive.

If a car model has cranky brakes, I think I can call that a failing
even without having the ability to come up with a pratical design
that doesn's has those limitations.

But in fact your situation is more closely analogous to a customer who's
bought a car that can be stopped by pressing on the brake pedal now
complaining that sideways pressure on the brake pedal doesn;t affect the
car's motion.


You will have to explain how you come by that analogy.
I judge a language by what it can and cannot do, not by my ability
to correct the things I perceive as failings. For all I know python
may have taken some design decisions that might have seen perfectly
logical but now prohibit a a practical design that doesn't have this
limitation. I don't see why something like that would make this
any less a failing then when a practical design was easy in the
current implemenation.

All that Tim was suggesting is that it's MORE SENSIBLE to start a thread
as the result of a specific call to programmed functionality rather than
as the side effect of an import. The reason for this is due to the
semantics of the import statement. If you perceive that as a failing
then you'd be better rewarded by an attempt to modify your perceptions.

I've always found "don't argue with Tim about Python" to be a useful
rule of thumb. He's wrong much less often than I am. I suspect he's also
wrong much less often than you ;-)


With all respect I find that lousy advise. I don't mind that it will turn
out I'll be wrong most of the time. But I'll probably will have gained
a better understanding by the responses to my argument than by merely
accepting the word of Tim.

[ ... ]
Note that the OP's example had a module that, upon the first attempt
to import it, ran an infinite loop (even if it hadn't deadlocked), and
it's clearly severe abuse of import's purpose.to write a module M such
that "import M" *never* returns. Indeed, that's the other half of how
deadlock occurs: not only that the imported module spawn a thread as
a side effect of importing, but also that the imported module refuse
to allow the import to complete.

Well I'll agree here. An import that has as a side effect that the
import doesn't return is bad practice.

But that's precisely the risk you run when starting up threads!


Now you are confusing me. Is it a problem of an import that doesn't
return of is it a case of a race condition where the import has to
return in a timely fashion?

--
Antoon Pardon
Jul 18 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Ajay | last post by:
hi! my application consists of a GUI with a number of functions. One of these runs a server in a separate thread. the thread is started and given the function start_server to execute the...
44
by: Charles Law | last post by:
Hi guys. I'm back on the threading gig again. It's the age-old question about waiting for something to happen without wasting time doing it. Take two threads: the main thread and a worker...
0
by: Dave Coate | last post by:
I am working on a generic way to launch multiple similar processes (threads) at once, but limit the number of threads running at any one time to a number I set. As I understand it the following...
6
by: les | last post by:
Here's a class which uses 2.0 generics to implement an inter-thread message queue in C#. Any number of threads can post and read from the queue simultaneously, and the message object can be any...
6
by: seb | last post by:
Hi, I am using pygtk for the first times. I am wondering what would be the best "pattern" to interface pygtk with a thread. The thread is collecting informations (over the network for...
5
by: admin | last post by:
ok This is my main. Pretty much it goes through each category and starts up 4 worker threads that then ask for groups to gether from. My problem is that when the thread gets done it keeps the...
5
by: P.J.M. Beker | last post by:
Hi there, I'm currently writing a program in which I use the FileMonitor to monitor a folder in which I store downloaded images. I know that I can't add much coding in the filemonitor's event in...
29
by: NvrBst | last post by:
I've read a bit online seeing that two writes are not safe, which I understand, but would 1 thread push()'ing and 1 thread pop()'ing be thread-safe? Basically my situation is the follows: ...
12
by: Ronny | last post by:
Thanks Chris, Looks nice but I miss the dual way communication. In the main thread to deliver paramters and data to the worker thread- how can I do that? Regards Ronny Take a look at the...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.