473,549 Members | 2,329 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Thread vs. generator problem

As I understand it, generators are supposed to run til they hit a
yield statement:

import time
def f():
print 1
time.sleep(3)
for i in range(2,5):
yield i

for k in f():
print k

prints "1" immediately, sleeps for 3 seconds, then prints 2, 3, and 4
without pausing, as expected. When I try to do it in a separate thread:

import time, itertools
def remote_iterate( iterator, cachesize=5):
# run iterator in a separate thread and yield its values
q = Queue.Queue(cac hesize)
def f():
print 'thread started'
for x in iterator:
q.put(x)
threading.Threa d(target=f).sta rt()
while True:
yield q.get()

g = remote_iterate( itertools.count )
print 'zzz...'
time.sleep(3)
print 'hi'
for i in range(5):
print g.next()

I'd expect to see 'thread started' immediately, then 'zzz...', then a 3
second pause, then 'hi', then the numbers 0..4. Instead, the thread
doesn't start until the 3 second pause has ended.

When I move the yield statement out of remote_iterate' s body and
instead have return a generator made in a new internal function, it
does what I expect:

import time, itertools
def remote_iterate( iterator, cachesize=5):
# run iterator in a separate thread and yield its values
q = Queue.Queue(cac hesize)
def f():
print 'thread started'
for x in iterator:
q.put(x)
threading.Threa d(target=f).sta rt()
def g():
while True:
yield q.get()
return g()

Any idea what's up? Is there some race condition, where the yield
statement freezes the generator before the new thread has started? Or
am I just overlooking something obvious?

Thanks.
May 26 '06 #1
3 1523
Paul Rubin wrote:
As I understand it, generators are supposed to run til they hit a
yield statement:

import time
def f():
print 1
time.sleep(3)
for i in range(2,5):
yield i

for k in f():
print k

prints "1" immediately, sleeps for 3 seconds, then prints 2, 3, and 4
without pausing, as expected. When I try to do it in a separate thread:

import time, itertools
def remote_iterate( iterator, cachesize=5):
# run iterator in a separate thread and yield its values
q = Queue.Queue(cac hesize)
def f():
print 'thread started'
for x in iterator:
q.put(x)
threading.Threa d(target=f).sta rt()
while True:
yield q.get()

g = remote_iterate( itertools.count )
print 'zzz...'
time.sleep(3)
print 'hi'
for i in range(5):
print g.next()

I'd expect to see 'thread started' immediately, then 'zzz...', then a 3
second pause, then 'hi', then the numbers 0..4. Instead, the thread
doesn't start until the 3 second pause has ended.


My 10-second analysis is that *none* the body of a generator runs until a value
is requested from it.

In [3]: def f():
...: print 'Starting f'
...: for i in range(3):
...: yield i
...:
...:

In [4]: g = f()

In [5]: for i in g:
...: print i
...:
...:
Starting f
0
1
2

In your first example, you instantiate the generator and then iterate over it
immediately; in your second, you separate the two things. I don't think threads
have anything to do with it.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

May 26 '06 #2
[Paul Rubin]
...
When I try to do it in a separate thread:

import time, itertools
def remote_iterate( iterator, cachesize=5):
# run iterator in a separate thread and yield its values
q = Queue.Queue(cac hesize)
def f():
print 'thread started'
for x in iterator:
q.put(x)
threading.Threa d(target=f).sta rt()
while True:
yield q.get()

g = remote_iterate( itertools.count )
You didn't run this code, right? itertools.count () was intended. In
any case, as when calling any generator, nothing in the body of
remote_iterate( ) is executed until the generator-iterator's next()
method is invoked. Nothing here does that. So, in particular, the
following is the _only_ line that can execute next:
print 'zzz...'
And then this line:
time.sleep(3)
And then this:
print 'hi'
And then this:
for i in range(5):
And then the first time you execute this line is the first time any
code in the body of remote_iterate( ) runs:
print g.next()

I'd expect to see 'thread started' immediately, then 'zzz...', then a 3
second pause, then 'hi', then the numbers 0..4. Instead, the thread
doesn't start until the 3 second pause has ended.
That's all as it must be.
When I move the yield statement out of remote_iterate' s body and
instead have return a generator made in a new internal function, it
does what I expect:

import time, itertools
def remote_iterate( iterator, cachesize=5):
Note that remote_iterate( ) is no longer a generator, so its body is
executed as soon as it's called.
# run iterator in a separate thread and yield its values
q = Queue.Queue(cac hesize)
def f():
print 'thread started'
for x in iterator:
q.put(x)
threading.Threa d(target=f).sta rt()
And so the thread starts when remote_iterate( ) is called.
def g():
while True:
yield q.get()
return g()

Any idea what's up? Is there some race condition, where the yield
statement freezes the generator before the new thread has started?
No.
Or am I just overlooking something obvious?


No, but it's not notably subtle either ;-)
May 26 '06 #3
"Tim Peters" <ti********@gma il.com> writes:
g = remote_iterate( itertools.count )
You didn't run this code, right? itertools.count () was intended.


Sorry, I made a cut-and-paste error posting the message. My test case
did use itertools.count ().
In any case, as when calling any generator, nothing in the body of
remote_iterate( ) is executed until the generator-iterator's next()
method is invoked.


Ooof, I see what happened now. My first test case was misleading and
made me think that the generator immediately executed until it reached
a yield statement. In fact it was the generator function (i.e. the
thing that made the generator), a separate object, that printed the
messages.

Thanks!
May 27 '06 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
1857
by: Roman Suzi | last post by:
(this is a repost with an addition - probably noone noticed my message first time) Hi! Just to be sure, is email package of Python 2.3 thread-safe or not (to use, for example, in python-milter?) P.S. And where can I find information on particular piece of standard library if it is thread-safe or need locking? I recall 'random' module...
3
3352
by: Mike Brown | last post by:
I have questions about thread safety in the 'random' module. When using the random.Random class (be it Mersenne Twister or Wichmann-Hill based), is it sufficiently thread-safe (preserving entropy and guarding against attack) to just have each thread work with its own random.Random instance? Or do I need to wrap my calls to each instance's...
4
6632
by: Jonathan Burd | last post by:
Greetings everyone, Here is a random string generator I wrote for an application and I'm wondering about the thread-safety of this function. I was told using static and global variables cause potential problems for thread-safety. So far, I'm only confused. I need a proper explanation for the concept so I can understand how to write...
10
2179
by: Jon Slaughter | last post by:
Since a thread doesn't have a Stop feature and I'm not supose to use abort, I'm wondering how I stop a thread? My problem is that I simply want to excute a function in the background and possibly "restart" it. What've been doing is essentially: if (SerialRowThread != null)
0
7546
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7471
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
7740
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
7830
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
1
5387
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
3517
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
0
3496
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
1082
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
784
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.