Yield

[Mateuszk87]

Hi.

may someone explain "yield" function, please. how does it actually work
and when do you use it?

thanks in forward

mateusz

[Fredrik Lundh]

Mateuszk87 wrote:

may someone explain "yield" function, please. how does it actually work
and when do you use it?

it returns a value from a function without actually terminating the
function; when the function is resumed, it'll continue to execute after
the yield.

a function that contains a yield statement is called a "generator", and
is most often used in a for-in loop, or in other contexts that expect a
sequence. the loop is automatically terminated when the function
returns in a usual way:

>>def gen():

.... yield 1
.... yield 2
.... yield 3
....

>>for item in gen():

.... print item
....
1
2
3

>>sum(gen())

6

>>[str(i) for i in gen()]

['1', '2', '3']

you can also use the generator "by hand"; when you call a generator
function, it returns a special "generator object", and then immediately
suspends itself. to run the generator, call its "next" method:

>>g = gen()
g

<generator object at 0x00AE64E0>

>>g.next()

1

>>g.next()

2

>>g.next()

3

when the generator is exhausted, it raises a StopIterator exception:

>>g.next()

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration

reference information:

http://effbot.org/pyref/yield.htm

hope this helps!

</F>

[Fredrik Lundh]

Fredrik Lundh wrote:

reference information:

also see:

http://effbot.org/pyfaq/what-is-a-generator.htm

</F>

[Carsten Haese]

On Wed, 2006-11-15 at 09:13 -0800, Mateuszk87 wrote:

Hi.

may someone explain "yield" function, please. how does it actually work
and when do you use it?

[There is probably a much better explanation of this somewhere on the
net already, but I feel like writing this out myself.]

"yield" is a keyword. In simple terms, if "yield" appears inside a
function, the function becomes a generator that can return multiple
results.

Consider the following trivial example of a function that calculates the
elements of some sequence and prints each one.

def squares(n):
for i in range(n):
print i**2

Now suppose it turns out that in addition to--or instead of--printing
the elements of this sequence, the caller wants to do something
different with them. You could handle this by building a list, and
return the list to the caller to loop over and operate on as they
choose. But if the list is very big, that's not very memory-efficient if
the caller only needs to operate on one element at a time.

This is where yield comes in. If you write this:

def squares(n):
for i in range(n):
yield i**2

squares is now a generator function. If you call squares(100), it won't
calculate any sequence elements immediately. It will instead return an
iterator the caller can loop over:

for el in squares(100):
print el # or do anything else the caller decides

The beauty of this is that you achieve physical separation between
generating elements of a sequence and using them. The generator function
only cares about how to produce the elements, and the caller only cares
about how to use them.

Hope this helps,

Carsten.

[Mateuszk87]

thx for the quick answer. i ll have a look.

[John Henry]

Thank you. This is very clear. I can see that this is useful in lots
of situations.

Fredrik Lundh wrote:

Mateuszk87 wrote:

may someone explain "yield" function, please. how does it actually work
and when do you use it?

it returns a value from a function without actually terminating the
function; when the function is resumed, it'll continue to execute after
the yield.

a function that contains a yield statement is called a "generator", and
is most often used in a for-in loop, or in other contexts that expect a
sequence. the loop is automatically terminated when the function
returns in a usual way:

>>def gen():

... yield 1
... yield 2
... yield 3
...

>>for item in gen():

... print item
...
1
2
3

>>sum(gen())

6

>>[str(i) for i in gen()]

['1', '2', '3']

you can also use the generator "by hand"; when you call a generator
function, it returns a special "generator object", and then immediately
suspends itself. to run the generator, call its "next" method:

>>g = gen()
>>g

<generator object at 0x00AE64E0>

>>g.next()

1

>>g.next()

2

>>g.next()

3

when the generator is exhausted, it raises a StopIterator exception:

>>g.next()

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration

reference information:

http://effbot.org/pyref/yield.htm

hope this helps!

</F>

[dwelch91]

Mateuszk87 wrote:

Hi.

may someone explain "yield" function, please. how does it actually work
and when do you use it?

thanks in forward

mateusz

http://docs.python.org/ref/yield.html

[olive]

dwelch91 a écrit :

http://docs.python.org/ref/yield.html

This is a perfect example that demonstrate how the actual python is bad
and most of the time useless (at least for me).

We really need mor example !

I would like to thanks Fredrik for his contribution to improve that.

Olivier.

[John Machin]

On 16/11/2006 5:45 PM, olive wrote:

dwelch91 a écrit :

>http://docs.python.org/ref/yield.html

This is a perfect example that demonstrate how the actual python is bad
and most of the time useless (at least for me).

I think that you left the word "documentation" out after "python" :-)

>
We really need mor example !

I would like to thanks Fredrik for his contribution to improve that.

Call me crazy, but after an admittedly quick read, the version on the
wiki seems to be about word for word with on the docs.python.org version.

Could you please tell us how you think the wiki version is an improvement?

Cheers,
John

[Fredrik Lundh]

olive wrote:

>http://docs.python.org/ref/yield.html

This is a perfect example that demonstrate how the actual python is bad
and most of the time useless (at least for me).

there's a place for (relatively) formal reference documentation, but
it's hardly ever the right place to learn why things are there, or how
they are supposed to be used.

We really need mor example !

I would like to thanks Fredrik for his contribution to improve that.

thanks!

(for the curious, my current plan is to use

http://effbot.org/zone/idea-seealso.htm

files to link from the official documentation to tutorials, example
collections, and blog articles. I encourage content producers to start
looking into generating such files for their document collections.)

</F>

[Fredrik Lundh]

John Machin wrote:

>I would like to thanks Fredrik for his contribution to improve that.

Call me crazy, but after an admittedly quick read, the version on the
wiki seems to be about word for word with on the docs.python.org version.

maybe he was thinking about the article I posted, or the FAQ link I
posted in a followup:

http://effbot.org/pyfaq/what-is-a-generator.htm

which was based on my article and the glossary entry from the tutorial:

http://effbot.org/pytut/glossary.htm#generator

Could you please tell us how you think the wiki version is an improvement?

an "add comment" link? direct URL:s for all concepts in the language?
extensive hyperlinking? semantic retargetable markup?

</F>

[John Machin]

On 16/11/2006 7:00 PM, Fredrik Lundh wrote:

John Machin wrote:

>>I would like to thanks Fredrik for his contribution to improve that.

Call me crazy, but after an admittedly quick read, the version on the
wiki seems to be about word for word with on the docs.python.org version.

maybe he was thinking about the article I posted, or the FAQ link I
posted in a followup:

http://effbot.org/pyfaq/what-is-a-generator.htm

which was based on my article and the glossary entry from the tutorial:

http://effbot.org/pytut/glossary.htm#generator

Quite possibly.

>

>Could you please tell us how you think the wiki version is an
improvement?

an "add comment" link? direct URL:s for all concepts in the language?
extensive hyperlinking? semantic retargetable markup?

Yes, they're great. Maybe I was thinking about the text contents only :-)

Cheers,
John

[Danny Colligan]

Now that we're on the subject, what are the advantages of using
generators over, say, list comprehensions or for loops? It seems to me
that virtually all (I won't say everything) the examples I've seen can
be done just as easily without using generators. For example,
Fredrik's initial example in the post:

>>a = [1,2,3]
for i in a: print i

....
1
2
3

>>sum(a)

6

>>[str(i) for i in a]

['1', '2', '3']

>>>

Carsten mentioned that generators are more memory-efficient to use when
dealing with large numbers of objects. Is this the main advantage of
using generators? Also, in what other novel ways are generators used
that are clearly superior to alternatives?

Thanks in advance,

Danny

On Nov 16, 3:14 am, John Machin <[e-mail]:

On 16/11/2006 7:00 PM, Fredrik Lundh wrote:

John Machin wrote:

>I would like to thanks Fredrik for his contribution to improve that.

Call me crazy, but after an admittedly quick read, the version on the
wiki seems to be about word for word with on the docs.python.org version.

maybe he was thinking about the article I posted, or the FAQ link I
posted in a followup:

http://effbot.org/pyfaq/what-is-a-generator.htm

which was based on my article and the glossary entry from the tutorial:

http://effbot.org/pytut/glossary.htm#generatorQuite possibly.

Could you please tell us how you think the wiki version is an
improvement?

an "add comment" link? direct URL:s for all concepts in the language?
extensive hyperlinking? semantic retargetable markup?Yes, they're great. Maybe I was thinking about the text contents only :-)

Cheers,
John

[Richard Brodie]

"Danny Colligan" <da***********@gmail.comwrote in message
news:11**********************@i42g2000cwa.googlegr oups.com...

Now that we're on the subject, what are the advantages of using
generators over, say, list comprehensions or for loops? It seems to me
that virtually all (I won't say everything) the examples I've seen can
be done just as easily without using generators.

The more trivial the example, the harder it is to see the advantage.
Suppose you wanted to sum the first 10000 primes. A quick Google
fins you Wensheng Wang's recipe:
http://aspn.activestate.com/ASPN/Coo.../Recipe/366178
Just add print sum(primes(10000)), and you're done.

[Danny Colligan]

The more trivial the example, the harder it is to see the advantage.

I absoultely agree. Thanks for pointing me out to some real-world
code. However, the function you pointed me to is not a generator
(there is no yield statement... it just returns the entire list of
primes). A generator version would be:

>>def primes(n):

.... if n<2: yield []
.... s=range(3,n+1,2)
.... mroot = n ** 0.5
.... half=(n+1)/2-1
.... i=0
.... m=3
.... while m <= mroot:
.... if s[i]:
.... j=(m*m-3)/2
.... s[j]=0
.... while j<half:
.... s[j]=0
.... j+=m
.... i=i+1
.... m=2*i+3
.... yield 2
.... for x in s:
.... if x: yield x
....

>>x = primes(11)
x.next()

2

>>x.next()

3

>>x.next()

5

>>x.next()

7

>>x.next()

11

>>x.next()

Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration

>>>

Danny Colligan

On Nov 16, 10:49 am, "Richard Brodie" <[e-mail]:

"Danny Colligan" <[e-mail] in messagenews:11**********************@i42g2000cwa.g ooglegroups.com...

Now that we're on the subject, what are the advantages of using
generators over, say, list comprehensions or for loops? It seems to me
that virtually all (I won't say everything) the examples I've seen can
be done just as easily without using generators.The more trivial the example, the harder it is to see the advantage.

Suppose you wanted to sum the first 10000 primes. A quick Google
fins you Wensheng Wang's recipe:http://aspn.activestate.com/ASPN/Coo.../Recipe/366178
Just add print sum(primes(10000)), and you're done.

[Carsten Haese]

On Thu, 2006-11-16 at 07:32 -0800, Danny Colligan wrote:

Carsten mentioned that generators are more memory-efficient to use when
dealing with large numbers of objects. Is this the main advantage of
using generators? Also, in what other novel ways are generators used
that are clearly superior to alternatives?

The memory efficiency is definitely a major advantage. Generators allow
you to efficiently produce, manipulate, and consume sequences of
arbitrary length. The length of the sequence could even be potentially
infinite, which is impossible to handle when you're working with actual
lists.

The memory efficiency aside, it's more elegant to write something like
this:

def squares(n):
for i in range(n):
yield i**2

than something like this:

def squares(n):
result = []
for i in range(n):
result.append(i**2)
return result

-Carsten

[Carsten Haese]

On Thu, 2006-11-16 at 08:09 -0800, Danny Colligan wrote:

The more trivial the example, the harder it is to see the advantage.

I absoultely agree. Thanks for pointing me out to some real-world
code. However, the function you pointed me to is not a generator
(there is no yield statement... it just returns the entire list of
primes). A generator version would be:

>def primes(n):

... if n<2: yield []
... s=range(3,n+1,2)
... mroot = n ** 0.5
... half=(n+1)/2-1
... i=0
... m=3
... while m <= mroot:
... if s[i]:
... j=(m*m-3)/2
... s[j]=0
... while j<half:
... s[j]=0
... j+=m
... i=i+1
... m=2*i+3
... yield 2
... for x in s:
... if x: yield x

Not quite:

>>x = primes(1)
x.next()

[]

>>x.next()

2

>>x.next()

Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration

To handle n<2 correctly, you have to "return" instead of "yield []".

>>x = primes(1)
x.next()

Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration

-Carsten

[Richard Brodie]

"Danny Colligan" <da***********@gmail.comwrote in message
news:11*********************@b28g2000cwb.googlegro ups.com...

I absoultely agree. Thanks for pointing me out to some real-world
code. However, the function you pointed me to is not a generator
(there is no yield statement... it just returns the entire list of
primes).

Oops, should have looked at the code more closely. Another example
would be os.walk() in the standard library.

[Tim Chase]

>I absoultely agree. Thanks for pointing me out to some real-world

>code. However, the function you pointed me to is not a generator
(there is no yield statement... it just returns the entire list of
primes).

Oops, should have looked at the code more closely. Another example
would be os.walk() in the standard library.

I've used a similar scheme for dealing with incoming data files.
Sometimes, I get them Emailed/FTP'ed so they're sitting in a
folder, so I can do a multitude of them by just pointing
os.walk() at them. Other times, I get them on specially
formatted CDs. It's nice to have the clean main-loop logic of:

for file_name in source:
do_stuff(file_name)

and be able to dynamically set "source" to an generator that
either wraps os.walk() for a given directory or repeatedly (1)
reads the file on the CD to know where it should be dumped, (2)
copies files off the CD to their associated dumping ground, (3)
ejects the CD, (4) yields the name of the file to process in the
dumping-ground directory, and (5) prompts the user if there are
more CDs to be processed.

Without "yield" functionality, I'd either have to have two
duplicate loops with similar internal logic (the do_stuff() sort
of code), or I'd have to read in all the files off the CD and
*then* process them all as a list afterwards.

Just one of the places I've recently been thankful for generators.

-tkc

[Fredrik Lundh]

Danny Colligan wrote:

Carsten mentioned that generators are more memory-efficient to use when
dealing with large numbers of objects. Is this the main advantage of
using generators? Also, in what other novel ways are generators used
that are clearly superior to alternatives?

the main advantage is that it lets you decouple the generation of data
from the use of data; instead of inlining calculations, or using a pre-
defined operation on their result (e.g printing them or adding them to a
container), you can simply yield them, and let the user use whatever
mechanism she wants to process them. i.e. instead of

for item in range:
calculate result
print result

you'll write

def generate():
for item in range:
yield result

and can then use

for item in generate():
print item

or

list(process(s) for s in generate())

or

sys.stdout.writelines(generate())

or

sum(generate())

etc, without having to change the generator.

you can also do various tricks with "endless" generators, such as the
following pi digit generator, based on an algorithm by Python's grand-
father Lambert Meertens:

def pi():
k, a, b, a1, b1 = 2, 4, 1, 12, 4
while 1:
# Next approximation
p, q, k = k*k, 2*k+1, k+1
a, b, a1, b1 = a1, b1, p*a+q*a1, p*b+q*b1
# Yield common digits
d, d1 = a/b, a1/b1
while d == d1:
yield str(d)
a, a1 = 10*(a%b), 10*(a1%b1)
d, d1 = a/b, a1/b1

import sys
sys.stdout.writelines(pi())

</F>

[Mathias Panzenboeck]

Fredrik Lundh schrieb:

Danny Colligan wrote:

>Carsten mentioned that generators are more memory-efficient to use when
dealing with large numbers of objects. Is this the main advantage of
using generators? Also, in what other novel ways are generators used
that are clearly superior to alternatives?

the main advantage is that it lets you decouple the generation of data
from the use of data; instead of inlining calculations, or using a pre-
defined operation on their result (e.g printing them or adding them to a
container), you can simply yield them, and let the user use whatever
mechanism she wants to process them. i.e. instead of

for item in range:
calculate result
print result

you'll write

def generate():
for item in range:
yield result

and can then use

for item in generate():
print item

or

list(process(s) for s in generate())

or

sys.stdout.writelines(generate())

or

sum(generate())

etc, without having to change the generator.

you can also do various tricks with "endless" generators, such as the
following pi digit generator, based on an algorithm by Python's grand-
father Lambert Meertens:

def pi():
k, a, b, a1, b1 = 2, 4, 1, 12, 4
while 1:
# Next approximation
p, q, k = k*k, 2*k+1, k+1
a, b, a1, b1 = a1, b1, p*a+q*a1, p*b+q*b1
# Yield common digits
d, d1 = a/b, a1/b1
while d == d1:
yield str(d)
a, a1 = 10*(a%b), 10*(a1%b1)
d, d1 = a/b, a1/b1

import sys
sys.stdout.writelines(pi())

</F>

or fibonacci:

def fib():
a, b = 0, 1
while True:
yield a
a, b = b, a+b

now get the first 10 ones:

>>from itertools import *
list(islice(fib(),10))

[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
or primes:

def primes():
yield 1
yield 2

p = 3
psofar = []

smallerPrimes = lambda: takewhile(lambda x: x+x <= p,psofar)

while True:
if all(p % n != 0 for n in smallerPrimes()):
psofar.append(p)
yield p

p += 2

etc.

[Mikael Olofsson]

Mathias Panzenboeck wrote:

def primes():
yield 1
yield 2
[snip rest of code]

Hmm... 1 is not a prime. See for instance

http://en.wikipedia.org/wiki/Prime_number

The definition given there is "In mathematics </wiki/Mathematics>, a
*prime number* (or a *prime*) is a natural number </wiki/Natural_number>
that has exactly two (distinct) natural number divisors
</wiki/Divisor>." The important part of the statement is "exactly
two...divisors", which rules out the number 1.

/MiO

[Mikael Olofsson]

I wrote:

The definition given there is "In mathematics </wiki/Mathematics>, a
*prime number* (or a *prime*) is a natural number
</wiki/Natural_numberthat has exactly two (distinct) natural number
divisors </wiki/Divisor>." The important part of the statement is
"exactly two...divisors", which rules out the number 1.

Or should I say: Thunderbird made me write:... Those freakin
</wiki/Mathematicsand </wiki/Natural_numberwas not visible before I
posted the thing. &%#&%#

/MiO