My Generator Paradox!

[vbgunz]

I am afraid that this is the first time in which I would probably need
something explained to me as if I were a little child. I am having a
hard time getting this through my thick skull. What in the world is
wrong with this!?

''' ################################################## ######### '''

def generatorFunction(sequence=['item1', 'item2', 'item3']):
for item in sequence:
yield item

yieldedValue = generatorFunction()

'''this seems to work perfectly.'''
print '-' * 32
print yieldedValue # <generator object at 0xb723014c>
print yieldedValue.next() # item1
print yieldedValue.next() # item2
print yieldedValue.next() # item3

'''this is where things don't make any sense!'''
print '-' * 32
print generatorFunction() # <generator object at 0xb723022c>
print generatorFunction().next() # item1
print generatorFunction().next() # item1
print generatorFunction().next() # item1

''' ################################################## ######### '''

the first set of calls assigned to yieldedValue work but the second set
without assignment don't. I asked for help on this at #python (I love
those people in there!) and was told the following...
generatorFunction() is a call (obvious) when calling the second set, I
am resetting the iteration and this explains why I only and always get
item1.

ok. *but* why in the world does the first set of calls work?
technically, isn't yieldedValue == generatorFunction() on a name basis?
I mean isn't the following technically the same?

generatorFunction()
yieldedValue = generatorFunction()

aren't they both the same? To me they should be but obviously this
creates the point of this paradox. I don't understand what is happening
here... Can someone care to explain why the assignment works but not
the direct call? In a sense shouldn't the assignment yield the same
results as the direct call and vice versa? I am confused :(

Thank you for any help on this!

[Felipe Almeida Lessa]

Em Qui, 2006-03-16 Ã*s 16:17 -0800, vbgunz escreveu:

print generatorFunction() # <generator object at 0xb723022c>
print generatorFunction().next() # item1
print generatorFunction().next() # item1
print generatorFunction().next() # item1

Each time you say "generatorFunction()", it gives you a new generator,
thus returning the first item again.

[Michal Kwiatkowski]

vbgunz wrote:

def generatorFunction(sequence=['item1', 'item2', 'item3']):
for item in sequence:
yield item

yieldedValue = generatorFunction()
You're creating an iterator here and binding it to name yieldedValue
(which is bogus, it should be named valueGenerator or sth like that).
'''this seems to work perfectly.'''
print '-' * 32
print yieldedValue # <generator object at 0xb723014c>
print yieldedValue.next() # item1
print yieldedValue.next() # item2
print yieldedValue.next() # item3
You're calling your iterator's next() method getting all of values, as
expected.
'''this is where things don't make any sense!'''
print '-' * 32
print generatorFunction() # <generator object at 0xb723022c> You're creating a new iterator here.
print generatorFunction().next() # item1 Another anonymous iterator gets created here. Instantly its next()
method is called, yielding first value.
print generatorFunction().next() # item1 And so on...
generatorFunction() is a call (obvious) when calling the second set, I
am resetting the iteration and this explains why I only and always get
item1.

ok. *but* why in the world does the first set of calls work?
technically, isn't yieldedValue == generatorFunction() on a name
basis? I mean isn't the following technically the same?

generatorFunction()
yieldedValue = generatorFunction()

Well, first statement creates new iterator which is garbage collected
right away (as it has no bindings). Second statement creates an
iterator and binds it to name yieldedValue. Then it can be used as
typical iterator. Calling yieldedValue.next() just calls method of the
same iterator you've created a moment ago. It's still the same object.
The difference is like the difference between following two lines:

list() # creating a new list
new_list = list() # creating a new list and binding its name

So, rewriting your example from generator to dictionary objects:

alist = [1, 2, 3, 4, 5]
print alist # [1, 2, 3, 4, 5]
print alist.pop() # 5
print alist.pop() # 4
print alist.pop() # 3

print [1, 2, 3, 4, 5] # [1, 2, 3, 4, 5]
print [1, 2, 3, 4, 5].pop() # 5
print [1, 2, 3, 4, 5].pop() # 5
print [1, 2, 3, 4, 5].pop() # 5

Remember that generator is an object and you'll be fine.

mk
--
. o . >> http://joker.linuxstuff.pl <<
. . o It's easier to get forgiveness for being wrong
o o o than forgiveness for being right.

[David Wahler]

vbgunz wrote:

I am afraid that this is the first time in which I would probably need
something explained to me as if I were a little child. I am having a
hard time getting this through my thick skull. What in the world is
wrong with this!?

''' ################################################## ######### '''

def generatorFunction(sequence=['item1', 'item2', 'item3']):
for item in sequence:
yield item

yieldedValue = generatorFunction()

'''this seems to work perfectly.'''
print '-' * 32
print yieldedValue # <generator object at 0xb723014c>
print yieldedValue.next() # item1
print yieldedValue.next() # item2
print yieldedValue.next() # item3

'''this is where things don't make any sense!'''
print '-' * 32
print generatorFunction() # <generator object at 0xb723022c>
print generatorFunction().next() # item1
print generatorFunction().next() # item1
print generatorFunction().next() # item1

''' ################################################## ######### '''

the first set of calls assigned to yieldedValue work but the second set
without assignment don't. I asked for help on this at #python (I love
those people in there!) and was told the following...
generatorFunction() is a call (obvious) when calling the second set, I
am resetting the iteration and this explains why I only and always get
item1.

ok. *but* why in the world does the first set of calls work?
technically, isn't yieldedValue == generatorFunction() on a name basis?
I mean isn't the following technically the same?

generatorFunction()
yieldedValue = generatorFunction()

aren't they both the same?

[snip]

In that short example, they happen to be different, but equivalent,
objects, but that will not always be the case. Consider this:

a = generatorFunction()
b = generatorFunction()
print a.next()

item1

Now a and b are both generators for generatorFunction, but calling
a.next() again will return 'item2', while calling b.next() will return
'item1'. The value returned by generatorFunction is an object, which
has an internal state that makes it distinct from other objects of the
same type. So once your yieldedValue has been altered by calling the
next() method, it is no longer equivalent to a fresh instance of the
generator.

-- David

[Schüle Daniel]

it's easy to explain

class X:
pass

x=X()
y=X()

x and y are different instances
one can put in x
x.item = 1
y doesn't even have an attribute item for example

similar with generators
they are *different* objects of same kind generator

def fib(): .... a,b = 1,1
.... while True:
.... a,b = b,a+b
.... yield a,b
.... f1 = fib()
f2 = fib()
f1 <generator object at 0x4042866c> f2 <generator object at 0x404db42c> # different addresses f1 is f2 False f1.next() (1, 2) f1.next() (2, 3) f1.next() (3, 5)

f2.next() (1, 2)

it's only natural that each objects starts it's own fibonaci serie

hth, Daniel

[mensanator]

vbgunz wrote:

I am afraid that this is the first time in which I would probably need
something explained to me as if I were a little child. I am having a
hard time getting this through my thick skull. What in the world is
wrong with this!?

''' ################################################## ######### '''

def generatorFunction(sequence=['item1', 'item2', 'item3']):
for item in sequence:
yield item

yieldedValue = generatorFunction()

'''this seems to work perfectly.'''
print '-' * 32
print yieldedValue # <generator object at 0xb723014c>
print yieldedValue.next() # item1
print yieldedValue.next() # item2
print yieldedValue.next() # item3

'''this is where things don't make any sense!'''
print '-' * 32
print generatorFunction() # <generator object at 0xb723022c>
print generatorFunction().next() # item1
print generatorFunction().next() # item1
print generatorFunction().next() # item1

''' ################################################## ######### '''

the first set of calls assigned to yieldedValue work but the second set
without assignment don't. I asked for help on this at #python (I love
those people in there!) and was told the following...
generatorFunction() is a call (obvious) when calling the second set, I
am resetting the iteration and this explains why I only and always get
item1.

ok. *but* why in the world does the first set of calls work?
technically, isn't yieldedValue == generatorFunction() on a name basis?
I mean isn't the following technically the same?

generatorFunction()
yieldedValue = generatorFunction()
No. Look at this

a = generatorFunction()
b = generatorFunction()
a==b False

Why aren'y they the same? Here's a clue:
generatorFunction() <generator object at 0x00B28418> generatorFunction() <generator object at 0x00AD24E0>

Note the addresses are different.

Try this
b.next() 'item1' b.next() 'item2' generatorFunction().next() 'item1'

Just like your example, the generator re-initailized and
printed item1. But that's not the same generator as b
b.next()

'item3'


aren't they both the same? To me they should be but obviously this
creates the point of this paradox. I don't understand what is happening
here... Can someone care to explain why the assignment works but not
the direct call? In a sense shouldn't the assignment yield the same
results as the direct call and vice versa? I am confused :(

Thank you for any help on this!

[Fredrik Lundh]

"vbgunz" wrote:

I am afraid that this is the first time in which I would probably need
something explained to me as if I were a little child. I am having a
hard time getting this through my thick skull. What in the world is
wrong with this!?

''' ################################################## ######### '''

def generatorFunction(sequence=['item1', 'item2', 'item3']):
for item in sequence:
yield item

yieldedValue = generatorFunction()

'''this seems to work perfectly.'''
print '-' * 32
print yieldedValue # <generator object at 0xb723014c>
print yieldedValue.next() # item1
print yieldedValue.next() # item2
print yieldedValue.next() # item3

'''this is where things don't make any sense!'''
print '-' * 32
print generatorFunction() # <generator object at 0xb723022c>
print generatorFunction().next() # item1
print generatorFunction().next() # item1
print generatorFunction().next() # item1

''' ################################################## ######### '''

does the following surprise you too ?

f = open("filename")
print f.readline() # prints first line
print f.readline() # prints second line
print f.readline() # prints third line

print open("filename").readline() # prints first line
print open("filename").readline() # prints first line
print open("filename").readline() # prints first line

</F>

[vbgunz]

I believe I understand now. the yield keyword is sort of like a cousin
to return. return will bring back an object I can work with and so does
yield *but* yield's object will most likely support the .next() method.

So, if I worked with a function that ends with the return keyword and
it returns a list, I can run list operations and list methods on it. if
a function ends with the yield keyword a generator should return.

So, calling the function by it's name will always reset and initialize
the generator. Whereas assigning to the functions yielded return grants
access to the real generator in which I can use the next() method.

Maybe I've explained it wrong *but* it does make sense to me now. I
just couldn't grasp it because I am still new to the keyword yield and
didn't know it sort of works like return.

I really wish to thank you fellas so much for your examples and
explanations! I think I got it! I thank you all again!

[Robert Kern]

vbgunz wrote:

I believe I understand now. the yield keyword is sort of like a cousin
to return. return will bring back an object I can work with and so does
yield *but* yield's object will most likely support the .next() method.

No, that's not really how it works. When a generator function is called, it
returns the generator object immediately. None of the code inside is executed.
Every time you call that generator function, you get a new generator object with
the initial state. The objects that are yielded inside the code don't show up yet.

The code inside the generator gets executed only when the generator object is
iterated over (or its .next() method is called). The objects that are yielded
are the results of calling the .next() method.

--
Robert Kern
ro*********@gmail.com

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Ron Adam]

Robert Kern wrote:

vbgunz wrote:

I believe I understand now. the yield keyword is sort of like a cousin
to return. return will bring back an object I can work with and so does
yield *but* yield's object will most likely support the .next() method.

No, that's not really how it works. When a generator function is called, it
returns the generator object immediately. None of the code inside is executed.
Every time you call that generator function, you get a new generator object with
the initial state. The objects that are yielded inside the code don't show up yet.

The code inside the generator gets executed only when the generator object is
iterated over (or its .next() method is called). The objects that are yielded
are the results of calling the .next() method.

Maybe this will clarify it further.

def gen(n): .... while 1:
.... print 'before yield'
.... yield n
.... print 'after yield'
.... g = gen('hello')
g.next() before yield
'hello' g.next() after yield
before yield
'hello' g.next()

after yield
before yield
'hello'

When the next() method is called the generator runs until it reaches a
yield. At which point it's rests until the next() method is called again.

Although there are times when I wish it could run (as a thread) until it
reaches a yield and then continue after the next() method is called
until it reaches the next yield.

Cheers,
Ron

[vbgunz]

OK. I hope my understanding of the yield keyword and generators in a
general sense are now better understood. When a generator function is
assigned to an identifier, no code is executed and a generator is
immediately returned. When the next() method is called on the new
generator, code from top to bottom executes within the generator until
it reaches it's first yield. Many yields can appear within one
generator. When this is the case a next method call will execute code
from yield to yield. Code that appears in a loop after a yield keyword
is executed on the next() method call.

I hope I got it right. I love you guys for your patience and examples.
It is greatly appreciated and means very much to me! Thank you fellas!

[Ron Adam]

vbgunz wrote:

OK. I hope my understanding of the yield keyword and generators in a
general sense are now better understood. When a generator function is
assigned to an identifier, no code is executed and a generator is
immediately returned. When the next() method is called on the new
generator, code from top to bottom executes within the generator until
it reaches it's first yield. Many yields can appear within one
generator. When this is the case a next method call will execute code
from yield to yield. Code that appears in a loop after a yield keyword
is executed on the next() method call.

I hope I got it right. I love you guys for your patience and examples.
It is greatly appreciated and means very much to me! Thank you fellas!

Yep, looks like you have it. ;-)

Only need to add what happens if a generator exits the end after the
yield(s).

If it were a function it would return a None object even if it didn't
have a return at the end. But a generator raises a StopIteration
Exception.

def gen(): .... yield 'hello'
.... print 'all done'
.... g = gen()
g.next() 'hello' g.next()

all done
Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration

This is the signal to indicate iteration is finished. You don't see it
when you are using generators as iterators because it's usually caught
by the object or statement using the generator.

Cheers,
Ron