473,320 Members | 1,936 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Nested generator caveat

I met the following surprising behaviour
>>def gen0():
.... for i in range(3):
.... def gen1():
.... yield i
.... yield i, gen1()
....
>>for i,g in gen0(): print i, g.next()
....
0 0
1 1
2 2
>>for i,g in list(gen0()): print i, g.next()
....
0 2
1 2
2 2
If this is not a bug, it is at least quite confusing.
The apparent reason is that the free variables in
nested generator definitions are not bound (to a value) at invocation
time but only at access time.
Almost surely, the same applies to all locally defined functions
with free variables.
This would mean that locally defined functions with free
variables are very risky in generators.
--
Dieter
Jul 4 '08 #1
3 1864
On Jul 3, 9:20*pm, "Dieter Maurer" <die...@handshake.dewrote:
The apparent reason is that the free variables in
nested generator definitions are not bound (to a value) at invocation
time but only at access time.
<YawnThat's what it is supposed to do. Welcome to a dynamic
language.
Raymond
Jul 4 '08 #2
Dieter Maurer <di****@handshake.dewrote:
I met the following surprising behaviour
[code moved until later...]
The apparent reason is that the free variables in nested generator
definitions are not bound (to a value) at invocation time but only at
access time.
No. This is about the difference between binding and assignment.
Unfortunately, Python doesn't have explicit syntax for doing the former.

Here's what's actually going on in your generator.
>>>def gen0():
... for i in range(3):
... def gen1():
... yield i
... yield i, gen1()
The function gen0 contains a yield statement; it's therefore a
generator. It contains an assignment to a variable i (in this case,
it's implicit in the `for' loop). So, on entry to the code, a fresh
location is allocated, and the variable i is bound to it.

The function gen1 contains a yield statement too, so it's also a
generator. It contains a free reference to a variable i, so it shares
the binding in the outer scope.

Here's the important part: the for loop works by assigning to the
location named by i each time through. It doesn't rebind i to a fresh
location. So each time you kick gen1, it produces the current value of
i at that time. So...
>>>for i,g in gen0(): print i, g.next()
0 0
1 1
2 2
Here, the for loop in gen0 is suspended each iteration while we do some
printing. So the variable i (in gen0) still matches the value yielded
by gen0.

But...
>>>for i,g in list(gen0()): print i, g.next()
0 2
1 2
2 2
Here, gen0 has finished all of its iterations before we start kicking
any of the returned generators. So the value of i in gen0 is 2 (the
last element of range(3)).
Almost surely, the same applies to all locally defined functions
with free variables.
This would mean that locally defined functions with free
variables are very risky in generators.
It means that you must be careful about the difference between binding
and assignment when dealing with closures of whatever kind.

Here's an example without involving nested generators.

def gen():
for i in xrange(3):
yield lambda: i
for f in gen(): print f()
for f in list(gen()): print f()

To fix the problem, you need to arrange for something to actually rebind
a variable around your inner generator on each iteration through. Since
Python doesn't have any mechanism for controlling variable binding other
than defining functions, that's what you'll have to do.

def genfix():
for i in xrange(3):
def bind(i):
def geninner():
yield i
return geninner()
yield i, bind(i)

shows the general pattern, but since a generator has the syntactic form
of a function anyway, we can fold the two together.

def genfix2():
for i in xrange(3):
def geninner(i):
yield i
yield i, geninner(i)

Yes, this is cumbersome. A `let' statement would help a lot. Or
macros. ;-)

-- [mdw]
Jul 6 '08 #3
Excellent explanation by Mark Wooding. I would only like to add that
the standard pythonic idiom in such cases seems to be the (ab)use of a
default argument to the function, because these get evaluated at the
definition time:
def gen0():
for i in range(3):
def gen1(i = i):
yield i
yield i, gen1()
Jul 7 '08 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: Oleg Leschov | last post by:
Could there be means of exiting nested loops in python? something similar to labelled loops in perl.. I consider it irrating to have to make a flag for sole purpose of checking it after loop if...
0
by: Francis Avila | last post by:
A few days ago (see the 'itertools.flatten()?' thread from October 28) I became obsessed with refactoring a recursive generator that yielded the leaves of nested iterables. When the dust settled,...
1
by: boris bass | last post by:
Below is linenumbered as well as original code -------------------------------------- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML><HEAD> <META http-equiv=Content-Type...
7
by: aurora | last post by:
I love generator and I use it a lot. Lately I've been writing some recursive generator to traverse tree structures. After taking closer look I have some concern on its performance. Let's take...
20
by: Jack | last post by:
Hi all, I need to do syntax parsing of simple naturual languages, for example, "weather of London" or "what is the time", simple things like these, with Unicode support in the syntax. In Java,...
16
by: koutoo | last post by:
I start my code with some constants then a while statement. But I have some For statements towards the end within the While statement where I start getting some errors. I'm hoping I won't have to...
4
by: Wolfgang Draxinger | last post by:
If you know languages like Python or D you know, that nested functions can be really handy. Though some compilers (looking at GCC) provide the extension of nested functions, I wonder, how one...
0
by: Maric Michaud | last post by:
Le Tuesday 12 August 2008 11:29:18 Cousson, Benoit, vous avez écrit : This is a language limitation. This is because nested scope is implemented for python function only since 2.3 allow late...
0
by: Cousson, Benoit | last post by:
This is a language limitation. That was my understanding as well, but I think it is a pity to have that limitation. Don't you think that the same improvement that was done for method nested scope...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.