473,898 Members | 2,941 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Why don't generators execute until first yield?

Hi!

First a bit of context.

Yesterday I spent a lot of time debugging the following method in a
rather slim database abstraction layer we've developed:

,----
| def selectColumn(se lf, table, column, where={}, order_by=[], group_by=[]):
| """Performs a SQL select query returning a single column
|
| The column is returned as a list. An exception is thrown if the
| result is not a single column."""
| query = build_select(ta ble, [column], where, order_by, group_by)
| result = DBResult(self.r awQuery(query))
| if result.colcount != 1:
| raise QueryError("Que ry must return exactly one column", query)
| for row in result.fetchAll RowsAsList():
| yield row[0]
`----

I'd just rewritten the method as a generator rather than returning a
list of results. The following test then failed:

,----
| def testSelectColum nMultipleColumn s(self):
| res = self.fdb.select Column('db3ut1' , ['c1', 'c2'],
| {'c1':(1, 2)}, order_by='c1')
| self.assertRais es(db3.QueryErr or, self.fdb.select Column,
| 'db3ut1', ['c1', 'c2'], {'c1':(1, 2)}, order_by='c1')
`----

I expected this to raise a QueryError due to the result.colcount != 1
constraint being violated (as was the case before), but that isn't the
case. The constraint it not violated until I get the first result from
the generator.

Now to the main point. When a generator function is run, it immediately
returns a generator, and it does not run any code inside the generator.
Not until generator.next( ) is called is any code inside the generator
executed, giving it traditional lazy evaluation semantics. Why don't
generators follow the usual eager evaluation semantics of Python and
immediately execute up until right before the first yield instead?
Giving generators special case semantics for no good reason is a really
bad idea, so I'm very curious if there is a good reason for it being
this way. With the current semantics it means that errors can pop up at
unexpected times rather than the code failing fast.

Martin
Jun 27 '08 #1
13 1835
On Wed, May 7, 2008 at 2:29 AM, Martin Sand Christensen <ms*@es.aau.dkw rote:
Now to the main point. When a generator function is run, it immediately
returns a generator, and it does not run any code inside the generator.
Not until generator.next( ) is called is any code inside the generator
executed, giving it traditional lazy evaluation semantics. Why don't
generators follow the usual eager evaluation semantics of Python and
immediately execute up until right before the first yield instead?
Giving generators special case semantics for no good reason is a really
bad idea, so I'm very curious if there is a good reason for it being
this way. With the current semantics it means that errors can pop up at
unexpected times rather than the code failing fast.
Isn't lazy evaluation sort of the whole point of replacing a list with
an iterator? Besides which, running up to the first yield when
instantiated would make the generator's first iteration inconsistent
with the remaining iterations. Consider this somewhat contrived
example:

def printing_iter(s tuff):
for item in stuff:
print item
yield item

Clearly, the idea here is to create a generator that wraps another
iterator and prints each item as it yields it. But using your
suggestion, this would instead print the first item at the time the
generator is created, rather than when the first item is actually
iterated over.

If you really want a generator that behaves the way you describe, I
suggest doing something like this:

def myGenerator(arg s):
immediate_setup _code()

def generator():
for item in actual_generato r_loop():
yield item
return generator()
Jun 27 '08 #2
Martin Sand Christensen <ms*@es.aau.dkw rote:
Now to the main point. When a generator function is run, it
immediately
returns a generator, and it does not run any code inside the
generator.
Not until generator.next( ) is called is any code inside the generator
executed, giving it traditional lazy evaluation semantics. Why don't
generators follow the usual eager evaluation semantics of Python and
immediately execute up until right before the first yield instead?
You mean you expect the semantics of generators to be that when you
create them, or every time you call next() they run until they hit yield
and then (except for the initial run) return the result that was yielded
the time before? It is easy enough to implement that, but could be a bit
confusing for the user.
>>def greedy(fn):
def greedygenerator (*args, **kw):
def delayed():
it = iter(fn(*args, **kw))
try:
res = it.next()
except StopIteration:
yield None
return
yield None
for value in it:
yield res
res = value
yield res
it = delayed()
it.next()
return it
return greedygenerator
>>@greedy
def mygen(n):
for i in range(n):
print i
yield i

>>x = mygen(3)
0
>>list(x)
1
2
[0, 1, 2]
>>x = mygen(0)
list(x)
[]
>>>
Now try:

for command in getCommandsFrom User():
print "the result of that command was", execute(command )

where getCommandsFrom User is a greedy generator that reads from stdin,
and see why generators don't work that way.

Jun 27 '08 #3
>>>>"Ian" == Ian Kelly <ia*********@gm ail.comwrites:
IanIsn't lazy evaluation sort of the whole point of replacing a list
Ianwith an iterator? Besides which, running up to the first yield when
Ianinstantiated would make the generator's first iteration
Ianinconsistent with the remaining iterations.

That wasn't my idea, although that may not have come across quite
clearly enough. I wanted the generator to immediately run until right
before the first yield so that the first call to next() would start with
the first yield.

My objection is that generators _by default_ have different semantics
than the rest of the language. Lazy evaluation as a concept is great for
all the benefits it can provide, but, as I've illustrated, strictly lazy
evaluation semantics can be somewhat surprising at times and lead to
problems that are hard to debug if you don't constantly bear the
difference in mind. In this respect, it seems to me that my suggestion
would be an improvement. I'm not any kind of expert on languages,
though, and I may very well be missing a part of the bigger picture that
makes it obvous why things should be as they are.

As for code to slightly change the semantics of generators, that doesn't
really address the issue as I see it: if you're going to apply such code
to your generators, you're probably doing it exactly because you're
aware of the difference in semantics, and you're not going to be
surprised by it. You may still want to change the semantics, but for
reasons that are irrelevant to my point.

Martin
Jun 27 '08 #4
>>>>"Duncan" == Duncan Booth <du**********@i nvalid.invalidw rites:
[...]
DuncanNow try:
Duncan>
Duncan for command in getCommandsFrom User():
Duncan print "the result of that command was", execute(command )
Duncan>
Duncanwhere getCommandsFrom User is a greedy generator that reads from stdin,
Duncanand see why generators don't work that way.

I don't see a problem unless the generator isn't defined where it's
going to be used. In other similar input bound use cases, such as the
generator iterating over a query result set in my original post, I see
even less of a problem. Maybe I'm simply daft and you need to spell it
out for me. :-)

Martin
Jun 27 '08 #5
Martin Sand Christensen <ms*@es.aau.dkw rote:
>>>>>"Duncan" == Duncan Booth <du**********@i nvalid.invalidw rites:
[...]
DuncanNow try:
Duncan>
Duncan for command in getCommandsFrom User():
Duncan print "the result of that command was",
execute(command ) Duncan>
Duncanwhere getCommandsFrom User is a greedy generator that reads
from stdin, Duncanand see why generators don't work that way.

I don't see a problem unless the generator isn't defined where it's
going to be used. In other similar input bound use cases, such as the
generator iterating over a query result set in my original post, I see
even less of a problem. Maybe I'm simply daft and you need to spell it
out for me. :-)
It does this:
>>@greedy
def getCommandsFrom User():
while True:
yield raw_input('Comm and?')

>>for cmd in getCommandsFrom User():
print "that was command", cmd
Command?hello
Command?goodbye
that was command hello
Command?wtf
that was command goodbye
Command?

Traceback (most recent call last):
File "<pyshell#5 6>", line 1, in <module>
for cmd in getCommandsFrom User():
File "<pyshell#4 2>", line 11, in delayed
for value in it:
File "<pyshell#5 3>", line 4, in getCommandsFrom User
yield raw_input('Comm and?')
KeyboardInterru pt
Jun 27 '08 #6
Duncan Booth wrote:
It does this:
>>>@greedy
def getCommandsFrom User():
while True:
yield raw_input('Comm and?')

>>>for cmd in getCommandsFrom User():
print "that was command", cmd
Command?hello
Command?goodbye
that was command hello
Command?wtf
that was command goodbye
Command?

Not here..
In [7]: def getCommandsFrom User():
while True:
yield raw_input('Comm and?')
...:
...:

In [10]: for cmd in getCommandsFrom User(): print "that was command", cmd
....:
Command?hi
that was command hi
Command?there
that was command there
Command?wuwuwuw
that was command wuwuwuw
Command?
Jun 27 '08 #7
Marco Mariani wrote:
Not here..
Oh, sorry, I obviously didn't see the @greedy decorator amongst all the
quoting levels.

Anyway, the idea doesn't make much sense to me :)
Jun 27 '08 #8
Marco Mariani <ma***@sferacar ta.comwrote:
Duncan Booth wrote:
>It does this:
>>>>@greedy
def getCommandsFrom User():
while True:
yield raw_input('Comm and?')

>>>>for cmd in getCommandsFrom User():
print "that was command", cmd
Command?hell o
Command?goodby e
that was command hello
Command?wtf
that was command goodbye
Command?


Not here..
In [7]: def getCommandsFrom User():
while True:
yield raw_input('Comm and?')
...:
...:

In [10]: for cmd in getCommandsFrom User(): print "that was command",
cmd
....:
Command?hi
that was command hi
Command?there
that was command there
Command?wuwuwuw
that was command wuwuwuw
Command?
Perhaps if you'd copied all of my code (including the decorator that was
the whole point of it)...

Jun 27 '08 #9
Duncan Booth wrote:
Perhaps if you'd copied all of my code (including the decorator that was
the whole point of it)...
Sure, I missed the point. Python's symbols become quoting levels and
mess up messages.

Anyway, I would loathe to start execution of a generator before starting
to iterate through it. Especially when generators are passed around.
The current behavior makes perfect sense.

Jun 27 '08 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

23
3758
by: Francis Avila | last post by:
Below is an implementation a 'flattening' recursive generator (take a nested iterator and remove all its nesting). Is this possibly general and useful enough to be included in itertools? (I know *I* wanted something like it...) Very basic examples: >>> rl = , '678', 9]] >>> list(flatten(rl)) >>> notstring = lambda obj: not isinstance(obj, type(''))
9
2693
by: Francis Avila | last post by:
A little annoyed one day that I couldn't use the statefulness of generators as "resumable functions", I came across Hettinger's PEP 288 (http://www.python.org/peps/pep-0288.html, still listed as open, even though it's at least a year old and Guido doesn't seem very hot on the idea). I'm not too sure of its ideas on raising exceptions in generators from outside (although it looks like it might be convenient in some cases), but being able...
3
1658
by: David Stockwell | last post by:
Hi, Section 9.10 of the tutorial discusses the yield keyword. When I tried using it I get the following SyntaxError. What does this error mean? Does it mean we can't use yield in our code? Is yield a form of a 'return' ?? class9.py:71: Warning: 'yield' will become a reserved keyword in the future File "class9.py", line 71
1
1954
by: fishboy | last post by:
Howdy, I'm in middle of a personal project. Eventually it will download multipart binary attachments and look for missing parts on other servers. And so far I've got it to walk a newsgroup and download and decode single part binaries. I thought I'd post the code and see what people think. I'd appreciate any feedback. It's my first program with generators and I'm worried I'm making this twice and hard as it needs to be.
0
1877
by: fishboy | last post by:
Howdy, Sorry if this is a double post. First try seemed to go into hyperspace. I'm working on a personal project. It's going to be a multipart binary attachment downloader that will search alternate servers for missing pieces. This is the working code so far. It will walk a newsgroup and download and decode all the single part attachments. Just change server,user,password,group at the bottom to something less
8
2266
by: Timothy Fitz | last post by:
It seems to me that in python, generators are not truly coroutines. I do not understand why. What I see is that generators are used almost exclusively for generation of lists just-in-time. Side effects are frowned upon. Coroutines, in contrast, are like split functions where side effects are often as important or more important than return values. I am currently writing a real time strategy game where I have visual effects and use...
24
3983
by: Lasse Vågsæther Karlsen | last post by:
I need to merge several sources of values into one stream of values. All of the sources are sorted already and I need to retrieve the values from them all in sorted order. In other words: s1 = s2 = s3 = for value in ???(s1, s2, s3):
6
4541
by: Talin | last post by:
I've been using generators to implement backtracking search for a while now. Unfortunately, my code is large and complex enough (doing unification on math expressions) that its hard to post a simple example. So I decided to look for a simpler problem that could be used to demonstrate the technique that I am talking about. I noticed that PEP 255 (Simple Generators) refers to an implementation of the "8 Queens" problem in the lib/test...
18
4580
by: jess.austin | last post by:
hi, This seems like a difficult question to answer through testing, so I'm hoping that someone will just know... Suppose I have the following generator, g: def f() i = 0 while True: yield i
0
9993
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
1
10949
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10484
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
9662
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
8036
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
7187
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5882
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
4706
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
4296
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.