473,322 Members | 1,259 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,322 software developers and data experts.

simple Thread question

Having trouble with my first forays into threads. Basically, the
threads don't seem to be working in parallel (or you might say are
blocking). I've boiled my problems to the following short code block
and ensuing output. Seems like the output should be all interleaved
and of course it's not. Running Python 2.2 from ActiveState on
Windows XP (also doesn't work on Windows 2000).

Thanks in advance!
adeger

#================================================= ===

import threading

class TestThr(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)

def run(self, name):
import time
for i in range(1,11):
print 'thread ', name, ' instance ', str(i)
time.sleep(1)

threads = []
for inst in ('a', 'b', 'c'):
thread = TestThr()
thread.run(inst)
threads.append(thread)

# output below
thread a instance 1
thread a instance 2
thread a instance 3
thread a instance 4
thread a instance 5
thread a instance 6
thread a instance 7
thread a instance 8
thread a instance 9
thread a instance 10
thread b instance 1
thread b instance 2
thread b instance 3
thread b instance 4
thread b instance 5
thread b instance 6
thread b instance 7
thread b instance 8
thread b instance 9
thread b instance 10
thread c instance 1
thread c instance 2
thread c instance 3
thread c instance 4
thread c instance 5
thread c instance 6
thread c instance 7
thread c instance 8
thread c instance 9
thread c instance 10
Jul 18 '05 #1
14 2140
On 3 Aug 2004, adeger wrote:
Having trouble with my first forays into threads. Basically, the
threads don't seem to be working in parallel (or you might say are
blocking). I've boiled my problems to the following short code block
and ensuing output. Seems like the output should be all interleaved
and of course it's not. Running Python 2.2 from ActiveState on
Windows XP (also doesn't work on Windows 2000).


The Python interpreter isn't too thread-friendly. Because it's not
re-entrant, it has to make use of a Global Interpreter Lock in order to
keep internal structures from getting mangled. This lock only allows one
thread to access the interpreter at a time, and switches threads every
hundred or so bytecodes. The likely cause of your problem is that your
loops don't reach this switching threshold -- try using xrange(100) or
higher.

The GIL is released during blocking I/O (or other) operations, and C
extensions can release the lock if they plan on doing lots of non-Python
processing. Because of the former property, another thing you can try is
inserting a time.sleep(.1) inside of each loop -- being a blocking I/O
operation, this should cause the GIL to be released, and your threads will
switch each time through the loop.

Aside from the performance loss on parallel-processing machines, there is
usually no reason to worry about the GIL in Python code: so long as you
make proper use of the thread synchronization routines, everything will
work as you intend it to.

Jul 18 '05 #2
Christopher T King wrote:
The Python interpreter isn't too thread-friendly. Because it's not
re-entrant, it has to make use of a Global Interpreter Lock in order to
keep internal structures from getting mangled. ....


Notwithstanding the rest of your answer, Christopher, I have to
say that in my opinion, the Python interpreter is *very*
thread-friendly. Obviously this just depends on differing
ideas of what it means to be "friendly" to threads, but I find
Python to be the most reliable and easiest to use environment
for multi-threaded applications that I've ever used. I know
what you meant by this, but if nothing else the criticism doesn't
serve to promote Python very well...

The world would probably be a much better place if most of its
multi-threaded applications were rewritten in Python.

(And it would be better still if some of them were then
re-designed to use Twisted, but that's another story. ;-)

-Peter
Jul 18 '05 #3
Christopher T King wrote:
On 3 Aug 2004, adeger wrote:

Having trouble with my first forays into threads. Basically, the
threads don't seem to be working in parallel (or you might say are
blocking). I've boiled my problems to the following short code block
and ensuing output. Seems like the output should be all interleaved
and of course it's not. Running Python 2.2 from ActiveState on
Windows XP (also doesn't work on Windows 2000).
The Python interpreter isn't too thread-friendly. Because it's not
re-entrant, it has to make use of a Global Interpreter Lock in order to
keep internal structures from getting mangled. This lock only allows one
thread to access the interpreter at a time, and switches threads every
hundred or so bytecodes. The likely cause of your problem is that your
loops don't reach this switching threshold -- try using xrange(100) or
higher.


Er... you're jumping the gun a bit - no need to scare the OP away from
threads with all these details about the GIL when the problem was simply
that the threads were never started.

[snip] Because of the former property, another thing you can try is
inserting a time.sleep(.1) inside of each loop


Uh, did you read his code before responding? (hint: he's already doing
that) :)

-Dave
Jul 18 '05 #4
On Tue, 3 Aug 2004, Peter Hansen wrote:
Obviously this just depends on differing ideas of what it means to be
"friendly" to threads, but I find Python to be the most reliable and
easiest to use environment for multi-threaded applications that I've
ever used.


Poor choice of words on my part... I didn't mean API wise (though I'm not
a huge fan of threading), but rather implementation-wise (having the GIL
and all).

Jul 18 '05 #5
JCM
You should override the run method, but call thread.start() to kick
the execution off in a separate thread. If you call thread.run(),
you're just running your code in the same thread.

adeger <ad****@netlibrary.com> wrote:
Having trouble with my first forays into threads. Basically, the
threads don't seem to be working in parallel (or you might say are
blocking). I've boiled my problems to the following short code block
and ensuing output. Seems like the output should be all interleaved
and of course it's not. Running Python 2.2 from ActiveState on
Windows XP (also doesn't work on Windows 2000). Thanks in advance!
adeger #================================================= === import threading class TestThr(threading.Thread):
def __init__(self):
threading.Thread.__init__(self) def run(self, name):
import time
for i in range(1,11):
print 'thread ', name, ' instance ', str(i)
time.sleep(1) threads = []
for inst in ('a', 'b', 'c'):
thread = TestThr()
thread.run(inst)
threads.append(thread) # output below
thread a instance 1
thread a instance 2
thread a instance 3
thread a instance 4
thread a instance 5
thread a instance 6
thread a instance 7
thread a instance 8
thread a instance 9
thread a instance 10
thread b instance 1
thread b instance 2
thread b instance 3
thread b instance 4
thread b instance 5
thread b instance 6
thread b instance 7
thread b instance 8
thread b instance 9
thread b instance 10
thread c instance 1
thread c instance 2
thread c instance 3
thread c instance 4
thread c instance 5
thread c instance 6
thread c instance 7
thread c instance 8
thread c instance 9
thread c instance 10

Jul 18 '05 #6
On Tue, 3 Aug 2004, Dave Brueck wrote:
Er... you're jumping the gun a bit - no need to scare the OP away from
threads with all these details about the GIL when the problem was simply
that the threads were never started.

[snip]
Because of the former property, another thing you can try is
inserting a time.sleep(.1) inside of each loop


Uh, did you read his code before responding? (hint: he's already doing
that) :)


Wow, I really screwed that up :P I really should stop posting to this list
in a hurry. My eyes glanced at the short loop, the thread-starting code
(didn't notice the .run() instead of .start()), and the output, and
concluded "GIL funkiness" (something I've been hit with before).

FWIW, replacing .run() with .start() (and dropping name from .run()) and
dropping the sleep() produces the same symptoms the OP was showing.

Jul 18 '05 #7
Christopher T King wrote:
On Tue, 3 Aug 2004, Peter Hansen wrote:

Obviously this just depends on differing ideas of what it means to be
"friendly" to threads, but I find Python to be the most reliable and
easiest to use environment for multi-threaded applications that I've
ever used.

Poor choice of words on my part... I didn't mean API wise (though I'm not
a huge fan of threading), but rather implementation-wise (having the GIL
and all).


I realize that's what you meant, and I stick with my opinion that
it is *exactly that* which makes Python a much better environment
for all the threaded applications I've written than the other
environments I've tried in the past.

-Peter
Jul 18 '05 #8
JCM <jo**@work.com> wrote in message news:<ce**********@fred.mathworks.com>...
You should override the run method, but call thread.start() to kick
the execution off in a separate thread. If you call thread.run(),
you're just running your code in the same thread.


Thanks JCM and everyone else for your help! This wasn't that clear to
me reading the docs (I really DID read them) and your answer saved me
a bunch of personal (and program execution) time!
Jul 18 '05 #9

"adeger" <ad****@netlibrary.com> wrote in message
news:b8**************************@posting.google.c om...
JCM <jo**@work.com> wrote in message

news:<ce**********@fred.mathworks.com>...
You should override the run method, but call thread.start() to kick
the execution off in a separate thread. If you call thread.run(),
you're just running your code in the same thread.


Thanks JCM and everyone else for your help! This wasn't that clear to
me reading the docs (I really DID read them) and your answer saved me
a bunch of personal (and program execution) time!


It's worth pointing out that while print bytecodes are atomic, print
statements are not.
So unless a) you use your own lock to control output via "print"
or b) you pass only 1 argument to print, you run the risk of creating output
like:

....
thread a instance 33
thread a instance 34
thread a instance 35
thread a instance 36
thread a instance 37
thread a instance 38
thread a thread b instance instance 39
1
thread thread a instance b instance 2
40
thread a thread b instance 3
thread b instance 4
thread b instance 5
thread b instance 6
thread b instance 7
....
To create the desired bad behavior, comment out the three statements that
refer to plock.
---
import threading,time,sys

plock = threading.Lock()

class TestThr(threading.Thread):
def __init__(self,name):
self.name = name
threading.Thread.__init__(self)

def run(self):
for i in range(1,100):
plock.acquire()
print 'thread', self.name, 'instance', str(i)
plock.release()
#time.sleep(.1)

def test():
for tname in ('a', 'b', 'c'):
thread = TestThr(tname)
thread.start()

---
David
Jul 18 '05 #10
ad****@netlibrary.com (adeger) wrote in message news:<b8*************************@posting.google.c om>...
def run(self, name):
import time
for i in range(1,11):
print 'thread ', name, ' instance ', str(i)
time.sleep(1)


You're importing the time module for every TestThr() object you
create, that's no good. You only need to import the module once, at
the beginning of the file like you've done with threading.
Jul 18 '05 #11
Beeyah wrote:
ad****@netlibrary.com (adeger) wrote in message news:<b8*************************@posting.google.c om>...
def run(self, name):
import time
for i in range(1,11):
print 'thread ', name, ' instance ', str(i)
time.sleep(1)


You're importing the time module for every TestThr() object you
create, that's no good. You only need to import the module once, at
the beginning of the file like you've done with threading.


While it's true that that is all that's needed, it is not
required, and doesn't really provide any performance
improvements. I often do the above sort of thing where I
use a given module in only one place, especially when using
threads where it feels subtly cleaner to defer loading of
some modules until the thread actually starts.

Modules are really ever imported only once and after that
the "import" statement amounts to retrieving a reference
by looking it up in the sys.modules dictionary. In the
case of a builtin module like "time", there isn't even
any significant overhead involved in the first import,
as there might be with a .py module (where the timestamp
encoded in the .pyc file is compared with the timestamp
of the .py file, and the .py file is recompiled if
necessary, and then a new module object is created and
the bytecode is executed in its namespace, etc).

-Peter
Jul 18 '05 #12
In article <l8********************@powergate.ca>,
Peter Hansen <pe***@engcorp.com> wrote:

While it's true that that is all that's needed, it is not required,
and doesn't really provide any performance improvements. I often do
the above sort of thing where I use a given module in only one place,
especially when using threads where it feels subtly cleaner to defer
loading of some modules until the thread actually starts.


Except that the import lock makes this a Bad Idea, IMO.
--
Aahz (aa**@pythoncraft.com) <*> http://www.pythoncraft.com/

"To me vi is Zen. To use vi is to practice zen. Every command is a
koan. Profound to the user, unintelligible to the uninitiated. You
discover truth everytime you use it." --*****@lion.austin.ibm.com
Jul 18 '05 #13
Aahz wrote:
In article <l8********************@powergate.ca>,
Peter Hansen <pe***@engcorp.com> wrote:
While it's true that that is all that's needed, it is not required,
and doesn't really provide any performance improvements. I often do
the above sort of thing where I use a given module in only one place,
especially when using threads where it feels subtly cleaner to defer
loading of some modules until the thread actually starts.


Except that the import lock makes this a Bad Idea, IMO.


Does the import lock apply to the first import, or even
to all later ones where it can just find it in sys.modules?

-Peter
Jul 18 '05 #14
In article <aq********************@powergate.ca>,
Peter Hansen <pe***@engcorp.com> wrote:
Aahz wrote:
In article <l8********************@powergate.ca>,
Peter Hansen <pe***@engcorp.com> wrote:

While it's true that that is all that's needed, it is not required,
and doesn't really provide any performance improvements. I often do
the above sort of thing where I use a given module in only one place,
especially when using threads where it feels subtly cleaner to defer
loading of some modules until the thread actually starts.


Except that the import lock makes this a Bad Idea, IMO.


Does the import lock apply to the first import, or even
to all later ones where it can just find it in sys.modules?


It applies to all -- after all, sys.modules gets updated before the
module code is run...
--
Aahz (aa**@pythoncraft.com) <*> http://www.pythoncraft.com/

"To me vi is Zen. To use vi is to practice zen. Every command is a
koan. Profound to the user, unintelligible to the uninitiated. You
discover truth everytime you use it." --*****@lion.austin.ibm.com
Jul 18 '05 #15

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

38
by: jrlen balane | last post by:
basically what the code does is transmit data to a hardware and then receive data that the hardware will transmit. import serial import string import time from struct import * ser =...
1
by: Jeff Yang | last post by:
These days,I have read the artile "Safe, Simple Multithreading in Windows Forms, Part 1" on MSDN,and I also runed the example of it....
3
by: Peter | last post by:
Hello Thanks for reviewing my question. I would like to know how can I programmatically select a node Thanks in Advanc Peter
3
by: Elliot Rodriguez | last post by:
Hi: I am writing a WinForm app that contains a DataGrid control and a StatusBar control. My goal is to update the status bar using events from a separate class, as well as some other simple...
1
by: Brian Henry | last post by:
Hello, I was tring to learn socket's (being i never used them before) and have a simple question. I want to create a listner that will get any data recieved and print it out. I've been able to...
73
by: Claudio Grondi | last post by:
In the process of learning about some deeper details of Python I am curious if it is possible to write a 'prefix' code assigning to a and b something special, so, that Python gets trapped in an...
3
by: Ryan Pedersen | last post by:
I have a small program that executes a single very long running background thread today. The scope of all variables in the method that is running in this thread today are all local to the...
176
by: nw | last post by:
Hi, I previously asked for suggestions on teaching testing in C++. Based on some of the replies I received I decided that best way to proceed would be to teach the students how they might write...
17
by: Chris M. Thomasson | last post by:
I use the following technique in all of my C++ projects; here is the example code with error checking omitted for brevity: _________________________________________________________________ /*...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.