473,237 Members | 1,385 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,237 software developers and data experts.

What is executed when in a generator

I thought that the following sequence

gl=0
def gen(x):
global gl
gl=x
yield x

s=gen(1)

suspends the generator just before the yield, so after the
assignment of s gl becomes 1.

Well, no. It is still zero. If I put

print "something"

before the yield, this doesn't get executed either. *EVERYTHING*
from the beginning until the yield gets executed only upon s.next().

Could you tell me please where can I read something in depth about the
semantics of generators? I feel a bit lost.
Thank you.
Jerzy Karczmarczuk
Oct 4 '05 #1
5 2218
Jerzy Karczmarczuk wrote:
Could you tell me please where can I read something in depth about the
semantics of generators? I feel a bit lost.


the behaviour is described in the language reference manual:

http://docs.python.org/ref/yield.html

"When a generator function is called, it returns an iterator known as a
generator iterator, or more commonly, a generator. The body of the
generator function is executed by calling the generator's next()
method repeatedly until it raises an exception."

the above page points to the design document, which contains the full
story:

http://www.python.org/peps/pep-0255.html

"When a generator function is called, the actual arguments are bound to
function-local formal argument names in the usual way, but no code in
the body of the function is executed. Instead a generator-iterator
object is returned; this conforms to the iterator protocol /.../

Each time the .next() method of a generator-iterator is invoked, the
code in the body of the generator-function is executed until a yield
or return statement (see below) is encountered, or until the end of
the body is reached."

</F>

Oct 4 '05 #2
On Tue, 04 Oct 2005 12:48:03 +0100, Jerzy Karczmarczuk
<ka*****@info.unicaen.fr> wrote:
before the yield, this doesn't get executed either. *EVERYTHING*
from the beginning until the yield gets executed only upon s.next().

Could you tell me please where can I read something in depth about the
semantics of generators? I feel a bit lost.
Thank you.


This is probably what you want:

http://www.python.org/doc/2.4.2/ref/yield.html

But you've basically got the idea anyway. I think the important thing is
to think of the generator as a factory. When called, it returns you
something you can iterate over by calling the next() method. Think of it
almost like instantiation. In fact, you could write something functionally
equivalent like this:

gl=0

class GeneratorLikeBehaviour:
def __init__(self, x):
self.state = 0
self.x = x

def next(self):
global gl
if self.state == 0:
g1 = x
self.state = 1
return x
else:
raise StopIteration

s=gen(1)
Unless I've made a typo, you should get exactly the same behaviour.

The difference under the bonnet is that calling the generator has less
overheads, as it is not a true function call - stack frames etc are not
having to be set up fully. Instead they are (presumably) set aside between
calls to s.next()

Hope this helps
Matt
--

| Matt Hammond
| R&D Engineer, BBC Research & Development, Tadworth, Surrey, UK.
| http://kamaelia.sf.net/
| http://www.bbc.co.uk/rd/
Oct 4 '05 #3
Jerzy Karczmarczuk wrote:
I thought that the following sequence

gl=0
def gen(x):
global gl
gl=x
yield x

s=gen(1)

suspends the generator just before the yield, so after the
assignment of s gl becomes 1.

Well, no. It is still zero. If I put

print "something"

before the yield, this doesn't get executed either. *EVERYTHING*
from the beginning until the yield gets executed only upon s.next().

Could you tell me please where can I read something in depth about the
semantics of generators? I feel a bit lost.
Thank you.


The first hing you need to realise is that s is a generator, and it
needs to be used in an iterative context to see the (sequence of)
yielded results:
s = gen(1)
s <generator object at 0x4e028c>

The easiest way to do this is to use the generator in a loop - though of
course in this case the sequence of yielded results is going to be of
length 1 ...
for o in s: ... print o
...
1
It's easier to see the power of this when the generator yields several
results, either from a loop or otherwise:
def shg(n): ... for i in range(n):
... if i % 2:
... yield i
... s = shg(6)
s <generator object at 0x4e040c> for i in s: ... print i
...
1
3
5
The reason for this is so you can create multiple (parameterised)
generators using different calls:
s1 = shg(3)
s2 = shg(7)
print [i for i in s1] [1] print [i for i in s2] [1, 3, 5]
Fredrik Lundh has already pointed you to the full description, so I'll
content myself with adding that you can, if you want to you can call the
generator's next() method to access the next in its sequence of results:
s1 = shg(3)
s2 = shg(7)
print [(i, s2.next()) for i in s1] [(1, 1)] print [i for i in s2]

[3, 5]

Hope this makes things a little clearer.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

Oct 4 '05 #4
Thank you all for some precisions about yield and generators.
But I would like to underline that I am *not* a complete newbie,
and I understand this stuff in general. I read the yield.html
material quite a time ago. But thank you for the PEP, I should
have scanned it more carefully.

My problem was the *context of suspension*. You see, when some text
speaks about the preservation of local state, etc., *a priori* the
following sequence

def gen()
a
b
c
yield x
d
e
f
yield y

*COULD BE* understood as follows. Upon the call s=gen(), a, b and c
*get* executed, change the global state, install some local, and then
the system makes the snapshot, and returns the generator with its
context. The call to next returns x to the caller, but the generator
works for some time, and executes d, e and f before the next suspension.

Now I know that this is not true. The CO- flag of the gen() procedure
inhibits the execution of the code altogether, and a, b, c are executed
upon the first s.next; d, e and f - upon the second next. Etc. OK,
that's it, I accept such behaviour, although the alternative could be
interesting as well.

====
Steve Holden wrote:
The first hing you need to realise is that s is a generator, and it
needs to be used in an iterative context to see the (sequence of)
yielded results:


This is not exact, this is a *particular* consumer view of generators.
I don't want them for iterators, I use them as emulators of *lazy
programming*, I implement some co-recursive algorithms with them.
So i use next() when I wish, and never 'for'.

Thank you once more.

Jerzy Karczmarczuk
Oct 4 '05 #5
[ I've got no idea of your skill level, but since you say you feel a bit
lost I'll take baby steps here. Apologies if it's too low or high :-)
I'm also taking baby steps because less people understand generators than
they might. After all, most new programmers often want something like
generators when they start programming, but life gets in their way.
]

Jerzy Karczmarczuk wrote:
Could you tell me please where can I read something in depth about the
semantics of generators? I feel a bit lost.


Watch what happens if you take your example (I'm not convinced it's the
best example for this, but it is your example :-) on the python shell:
gl=0
def gen(x): .... global gl
.... gl=x
.... yield x
.... s=gen(1)
Clearly this has done something. What you haven't understood by the looks
of things is what and why. Let's take this session a bit further. Let's
find out the value of s: s <generator object at 0x40397a8c>

OK, so the call to the generator function returned a generator. That
should make sense to you. This will hopefully make more sense if we make
a few more calls to 'gen' and look at their values.
t=gen(2)
t <generator object at 0x40397bec>

As you'd expect the next call *also* creates a generator object. You'll
note that it has a different location ( "at 0x....." ) meaning they're
different objects.

Do that a few more times and we see the same: u=gen(3)
u <generator object at 0x403979ec> v=gen(4)
v <generator object at 0x40397c0c>

What this is doing is creating a generator object which can then be
asked repeatedly to run, and yield values.

A more accurate term might be iterating over the generator object. However
the /idea/ is that you create the object wrapping a context of/for running
that can suspend its own control and return intemediate values. Clearly
the generator also needs a way of telling us its finished iterating. Let's
make that more concrete. You call the .next() method of the object, and
when it's done it raises a StopIteration exception.

The way we do this with your code is as follows:
s.next() 1

Yay! That's the "1" that you were expecting to see. In your function you
also updated a global "gl". This is probably not a wise idea, but hey,
let's look at what happened to that value:
gl 1

Again, this is what you should have expected - since it's what you were
after.

Again, looking at your function, "yield x" was the last statement, so you
should be wondering what happens next:
s.next() Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration

As you can see we get told that it's dropped off the end. If we try calling
again:
s.next() Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration

If we retry with the generator objects created above, we can see:
t=gen(2)
u=gen(3)
v=gen(4)
t.next() 2 gl 2 v.next() 4 gl 4
u.next() 3 gl 3

Hopefully that's a clearer explanation of what's actually going on with
your code. A more complex example is based on the example in the PEP:

def configurable_fib(base1=1, base2=1):
a, b = base1, base2
while 1:
yield a
a,b = b, a+b
def configurable_fib(base1=1, base2=1): .... a, b = base1, base2
.... while 1:
.... yield a
.... a,b = b, a+b normal_fib = configurable_fib(1,1)
normal_fib.next() 1 normal_fib.next() 1 normal_fib.next() 2 normal_fib.next() 3

We can then create another one and ask that for values:
normal_fib2 = configurable_fib(1,1)
normal_fib.next(), normal_fib2.next() (5, 1) normal_fib.next(), normal_fib2.next() (8, 1) normal_fib.next(), normal_fib2.next() (13, 2)

Or we can create a few, with unusual bases for the fibonacci sequence and
find their first few values. Let's take the bases of 1 to 10.
fibs = [ configurable_fib(x,x) for x in range(1,11) ]
print [ x.next() for x in fibs ] [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] print [ x.next() for x in fibs ] [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] print [ x.next() for x in fibs ] [2, 4, 6, 8, 10, 12, 14, 16, 18, 20] print [ x.next() for x in fibs ] [3, 6, 9, 12, 15, 18, 21, 24, 27, 30] print [ x.next() for x in fibs ]

[5, 10, 15, 20, 25, 30, 35, 40, 45, 50]

This shows that returning a generator *object* rather than just a value
when you call the generator function is a useful behaviour, even if
initially when learning generators it's slightly counter intuitive.

One way to really understand them though is to try building something
with a whole load of generators. For those purposes, I wrote a tutorial
for our project which is based very much around having lots of generators.
The tutorial is made up of a set of learning exercises:

* http://kamaelia.sourceforge.net/MiniAxon/

We've tested this tutorial on a couple of novices at work (they learn
python one week and get this tutorial the next), and they've found it
relatively simple. The first hadn't done any programming before, except
a small amount of VB - he was a pre-university trainee. The second was
a university vacation trainee who'd done 2 years, but had no experience
of the ideas in the tutorial or python before joining us.

It's specifically targeted at novices and might be of use when seeing the
possibilities of what you can actually use generators for.

{ Incidentally if anyone reading this, not just yourself, decides to give
it a go, I'd really appreciate hearing back from you what you thought
of it, easy/difficult, clear/unclear, what level of experience you have,
etc. People don't have to really, I'm posting this because I've noticed
a couple of people have tried this so far as a means to trying to
understand generators :-) }

Best Regards and hope the above is useful,
Michael.

Oct 9 '05 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Harry Slaughter | last post by:
so i'm doing yet another mysql driven php app. same kind the rest of you build every day :) yes, it gets tedious having to write new code for each new DB that one works with. so this time i decided...
1
by: Michele Simionato | last post by:
This came up in the Italian newsgroup; I take the occasion to bring the question here since it is something which puzzled me for a while. The question is when we are expected to use itertools and...
23
by: Mike Meyer | last post by:
Ok, we've added list comprehensions to the language, and seen that they were good. We've added generator expressions to the language, and seen that they were good as well. I'm left a bit...
12
by: Nathan Sokalski | last post by:
What is the difference between the Page_Init and Page_Load events? When I was debugging my code, they both seemed to get triggered on every postback. I am assuming that there is some difference,...
9
by: Karlo Lozovina | last post by:
Here is it: --- class Human: def __init__(self, eye_one, eye_two): self.eye_one = eye_one self.eye_two = eye_two class Population: def __init__(self):
15
by: Brady Love | last post by:
I am currently working an an app that will post and edit blogs on blogger. Right now I have it so I can recive a list of the blogs and the content of those blogs using Atomizer. This is my first...
18
by: John Salerno | last post by:
Ok, I wrote this all by myself, which explains why it doesn't work. It is meant to take a number and generate the next number that follows according to the Morris sequence. It works for a single...
16
by: SLIMSHIM | last post by:
Hi, I"m new to c# and .net. I wrote a small program to add rows to an access table. the program goes thru the motions but the data never gets there. here is my code. I am intentionaly not using...
27
by: Steven D'Aprano | last post by:
I thought that an iterator was any object that follows the iterator protocol, that is, it has a next() method and an __iter__() method. But I'm having problems writing a class that acts as an...
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.