A gotcha: Python pain point?

[Beorn]

Consider this example:

>>def funcs(x):

... for i in range(5):
... def g(): return x + i
... yield g

I would expect the value of x used in g to be that at the function
declaration time, as if you've pass g a (x=x) argument, especially
after reading this post: http://lua-users.org/wiki/LuaScopingDiscussion

But:

>>[ fun() for fun in list(funcs(1)) ]

[5, 5, 5, 5, 5]

Whereas:

>>[ fun() for fun in funcs(1) ]

[1, 2, 3, 4, 5]

This came up while discussing Python pain points at
http://intertwingly.net/blog/2007/06...ts#c1181602242

I can see how it works now, but I haven't found an easy-to-read
documentation on this.

I guess it's debatable if perhaps every i used in the loop shouldn't
be a different i. It had me fooled, anyways.

Rgds,
Bjorn

[James Stroud]

Beorn wrote:

Consider this example:

>>def funcs(x):

... for i in range(5):
... def g(): return x + i
... yield g

I would expect the value of x used in g to be that at the function
declaration time, as if you've pass g a (x=x) argument, especially
after reading this post: http://lua-users.org/wiki/LuaScopingDiscussion

But:

>>[ fun() for fun in list(funcs(1)) ]

[5, 5, 5, 5, 5]

Whereas:

>>[ fun() for fun in funcs(1) ]

[1, 2, 3, 4, 5]

This came up while discussing Python pain points at
http://intertwingly.net/blog/2007/06...ts#c1181602242

I can see how it works now, but I haven't found an easy-to-read
documentation on this.

I guess it's debatable if perhaps every i used in the loop shouldn't
be a different i. It had me fooled, anyways.

Rgds,
Bjorn

If this isn't classified as a bug, then someone has some serious
explaining to do. Why would it be desirable for a generator to behave
differently in two different contexts. Should I import this to see how
many principles this behavior violates?

James

[Terry Reedy]

"Beorn" <bj****@gmail.comwrote in message
news:11**********************@z28g2000prd.googlegr oups.com...
| Consider this example:
|
| >>def funcs(x):
| ... for i in range(5):
| ... def g(): return x + i
| ... yield g
|
| I would expect the value of x used in g to be that at the function
| declaration time, as if you've pass g a (x=x) argument,

Since x is constant during the funcs call, does not matter. Perhaps you
meant i, which does vary? If so, put i=i in the header -- or perhaps ii=i
to give two different names to two things which are made to have the same
value.

In any case, the simple rule is that default argument expressions in the
*header* are evaluated at definition time while the *body* (past the doc
string, if any) is executed after a call.

Some people expect defaults to be executed every call; others expect part
of the body to be executed once. Both get in trouble. For nested
functions, outer call time is inner definition time and this can confuses.
Defining (but delaying the call of) multiple identical inner functions, as
you do in the first example below, also confuses.

| especially
| after reading this post: http://lua-users.org/wiki/LuaScopingDiscussion

Lua is not Python.

| But:
|
| >>[ fun() for fun in list(funcs(1)) ]
| [5, 5, 5, 5, 5]
|
| Whereas:
|
| >>[ fun() for fun in funcs(1) ]
| [1, 2, 3, 4, 5]

As Calderone explained, the simple rule works as long as one keeps track of
what is called when.

Terry Jan Reedy

[Diez B. Roggisch]

James Stroud wrote:

Beorn wrote:

>Consider this example:

> >>def funcs(x):

... for i in range(5):
... def g(): return x + i
... yield g

I would expect the value of x used in g to be that at the function
declaration time, as if you've pass g a (x=x) argument, especially
after reading this post: http://lua-users.org/wiki/LuaScopingDiscussion

But:

> >>[ fun() for fun in list(funcs(1)) ]

[5, 5, 5, 5, 5]

Whereas:

> >>[ fun() for fun in funcs(1) ]

[1, 2, 3, 4, 5]

This came up while discussing Python pain points at
http://intertwingly.net/blog/2007/06...ts#c1181602242

I can see how it works now, but I haven't found an easy-to-read
documentation on this.

I guess it's debatable if perhaps every i used in the loop shouldn't
be a different i. It had me fooled, anyways.

Rgds,
Bjorn

If this isn't classified as a bug, then someone has some serious
explaining to do. Why would it be desirable for a generator to behave
differently in two different contexts. Should I import this to see how
many principles this behavior violates?

It's no bug. The generator behaves the same way in both cases. What is
different is the calling time of the function.

Certainly a surprising outcome. But no bug.

Diez

[Terry Reedy]

"James Stroud" <js*****@mbi.ucla.eduwrote in message
news:f4**********@zinnia.noc.ucla.edu...
| Beorn wrote:
| Consider this example:
| >
| >>def funcs(x):
| ... for i in range(5):
| ... def g(): return x + i
| ... yield g
| >
| >>[ fun() for fun in list(funcs(1)) ]
| [5, 5, 5, 5, 5]
| >
| Whereas:
| >
| >>[ fun() for fun in funcs(1) ]
| [1, 2, 3, 4, 5]

| If this isn't classified as a bug,

It is not, it is well-documented behavior. Still, suggestions for
improvement might be considered.

| Why would it be desirable for a generator to behave
| differently in two different contexts.

I have no idea.

Each call of the generator function funcs behaves the same. It returns a
generator that yields 5 identical copies of the inner function g. The
multiple copies are not needed and only serve to confuse the issue.
Changing funcs to return a generator that yields the *same* function (five
times) gives the same behavior.

def funcs(x):
def g(): return x + i
for i in range(5):
yield g

print [ fun() for fun in list(funcs(1)) ]
print [ fun() for fun in funcs(1) ]

>># when run

[5, 5, 5, 5, 5]
[1, 2, 3, 4, 5]

What matters is the value of g's nonlocal var i (funcs' local var i) when
the yielded function g is *called*.

The difference between returning versus yielding an inner closure such as g
is this. If g is returned, the outer function has terminated and the
enclosed variable(s), i in this case, is frozen at its final value. If g
is yielded, the enclosed i is *live* as long as the generator is, and its
values can change between calls, as in the second print statement.

| Should I import this to see how
| many principles this behavior violates?

???

If you meant 'report' (on SF), please do not.

Terry Jan Reedy

[Diez B. Roggisch]

Beorn wrote:

Consider this example:

>>def funcs(x):

... for i in range(5):
... def g(): return x + i
... yield g

I would expect the value of x used in g to be that at the function

You mean i here, don't you?

declaration time, as if you've pass g a (x=x) argument, especially
after reading this post: http://lua-users.org/wiki/LuaScopingDiscussion

But:

>>[ fun() for fun in list(funcs(1)) ]

[5, 5, 5, 5, 5]

Whereas:

>>[ fun() for fun in funcs(1) ]

[1, 2, 3, 4, 5]

This came up while discussing Python pain points at
http://intertwingly.net/blog/2007/06...ts#c1181602242

I can see how it works now, but I haven't found an easy-to-read
documentation on this.

This has been discussed here very often. Python closures do capture the
names, not the values. If you want a value at a certain point, you need to
bind in the generated function scope by passing it as parameter. Like this:

def funcs(x):
for i in xrange(5):
def g(i=i): return x + i
yield g

Diez

[Marc 'BlackJack' Rintsch]

In <ma***************************************@python. org>, Terry Reedy
wrote:

| Should I import this to see how
| many principles this behavior violates?

???

If you meant 'report' (on SF), please do not.

I think he meant ``import this`` at the Python interpreter.

Ciao,
Marc 'BlackJack' Rintsch

[Beorn]

On Jun 12, 5:00 pm, "Diez B. Roggisch" <[e-mail]:

Beorn wrote:

[...]

I can see how it works now, but I haven't found an easy-to-read
documentation on this.

This has been discussed here very often. Python closures do capture the
names, not the values. If you want a value at a certain point, you need to
bind in the generated function scope by passing it as parameter. Like this:

def funcs(x):
for i in xrange(5):
def g(i=i): return x + i
yield g

Well, I would hope to see this documented somewhere prominently;
looking at the PEPs didn't give me much of a clue. I'm not even sure
how to describe the problem, so formulating search queries becomes
hard(!).

How does other languages, like Ruby, work? Sam (see original link
about pain points) was very surprised that the for loop didn't
introduce a new scope (is that the right word?) for each iteration?

Rgds,
Bjorn