Why don't generators execute until first yield?

[Martin Sand Christensen]

Hi!

First a bit of context.

Yesterday I spent a lot of time debugging the following method in a
rather slim database abstraction layer we've developed:

,----
| def selectColumn(se lf, table, column, where={}, order_by=[], group_by=[]):
| """Performs a SQL select query returning a single column
|
| The column is returned as a list. An exception is thrown if the
| result is not a single column."""
| query = build_select(ta ble, [column], where, order_by, group_by)
| result = DBResult(self.r awQuery(query))
| if result.colcount != 1:
| raise QueryError("Que ry must return exactly one column", query)
| for row in result.fetchAll RowsAsList():
| yield row[0]
`----

I'd just rewritten the method as a generator rather than returning a
list of results. The following test then failed:

,----
| def testSelectColum nMultipleColumn s(self):
| res = self.fdb.select Column('db3ut1' , ['c1', 'c2'],
| {'c1':(1, 2)}, order_by='c1')
| self.assertRais es(db3.QueryErr or, self.fdb.select Column,
| 'db3ut1', ['c1', 'c2'], {'c1':(1, 2)}, order_by='c1')
`----

I expected this to raise a QueryError due to the result.colcount != 1
constraint being violated (as was the case before), but that isn't the
case. The constraint it not violated until I get the first result from
the generator.

Now to the main point. When a generator function is run, it immediately
returns a generator, and it does not run any code inside the generator.
Not until generator.next( ) is called is any code inside the generator
executed, giving it traditional lazy evaluation semantics. Why don't
generators follow the usual eager evaluation semantics of Python and
immediately execute up until right before the first yield instead?
Giving generators special case semantics for no good reason is a really
bad idea, so I'm very curious if there is a good reason for it being
this way. With the current semantics it means that errors can pop up at
unexpected times rather than the code failing fast.

Martin

[castironpi]

On May 7, 7:37*am, Marco Mariani <ma...@sferacar ta.comwrote:

Duncan Booth wrote:

Perhaps if you'd copied all of my code (including the decorator that was
the whole point of it)...

Sure, I missed the point. Python's symbols become quoting levels and
mess up messages.

Anyway, I would loathe to start execution of a generator before starting
to iterate through it. Especially when generators are passed around.
The current behavior makes perfect sense.

Question:

>>def f( ):

... print 0
... while 1:
... yield 1
...

>>g= f( )
g.next( )

0
1

>>g.next( )

1

>>g.next( )

1

This might fit the bill:

>>def dropfirst( h ):

... h.next( )
... return h
...

>>g= dropfirst( f( ) )

0

>>g.next( )

1

>>g.next( )

1

>>g.next( )

1

However as dropfirst is dropping a value, both caller -and- cally have
to designate a/the exception. Hold generators are better "first-
dropped", and you hold 'next' inherently causes side effects. @greedy
(from earlier) frees the caller of a responsibility/obligation.

What can follow without a lead?

The definitions may lean harder on the 'generation' as prior to the
'next': generators inherently don't cause side effects.

Or hold, first-dropped is no exception:

>>special= object( )
def f( ):

... print 0
... yield special
... while 1:
... yield 1
...

>>g= f( )
g.next( )

0
<object object at 0x00980470>

>>g.next( )

1

>>g.next( )

1

>>g.next( )

1

[Diez B. Roggisch]

Now to the main point. When a generator function is run, it immediately

returns a generator, and it does not run any code inside the generator.
Not until generator.next( ) is called is any code inside the generator
executed, giving it traditional lazy evaluation semantics. Why don't
generators follow the usual eager evaluation semantics of Python and
immediately execute up until right before the first yield instead?
Giving generators special case semantics for no good reason is a really
bad idea, so I'm very curious if there is a good reason for it being
this way. With the current semantics it means that errors can pop up at
unexpected times rather than the code failing fast.

The semantics of a generator are very clear: on .next(), run until the next
yield is reached and then return the yielded value. Plus of course the
dealing with StopIteration-stuff.

Your scenario would introduce a special-case for the first run, making it
necessary to keep additional state around (possibly introducing GC-issues
on the way), just for the sake of it. And violate the lazyness a generator
is all about. Think of a situation like this:

def g():
while True:
yield time.time()

Obviously you want to yield the time at the moment of .next() being called.
Not something stored from ages ago. If anything that setups the generator
shall be done immediatly, it's easy enough:

def g():
first_result = time.time()
def _g():
yield first_result
while True:
yield time.time()
return _()

Diez

[Michael Torrie]

Martin Sand Christensen wrote:

Why don't
generators follow the usual eager evaluation semantics of Python and
immediately execute up until right before the first yield instead?

A great example of why this behavior would defeat some of the purpose of
generators can be found in this amazing PDF presentation:

http://www.dabeaz.com/generators/Generators.pdf

Giving generators special case semantics for no good reason is a really
bad idea, so I'm very curious if there is a good reason for it being
this way. With the current semantics it means that errors can pop up at
unexpected times rather than the code failing fast.

Most assuredly they do have good reason. Consider the cases in the PDF
I just mentioned. Building generators that work on the output of other
generators allows assembling entire pipelines of behavior. A very
powerful feature that would be impossible if the generators had the
semantics you describe.

If you want generators to behave as you suggest they should, then a
conventional for x in blah approach is likely the better way to go.

I use a generator anytime I want to be able to iterate across something
that has a potentially expensive cost, in terms of memory or cpu, to do
all at once.

[castironpi]

On May 7, 4:51*pm, Michael Torrie <torr...@gmail. comwrote:

Martin Sand Christensen wrote:

Why don't
generators follow the usual eager evaluation semantics of Python and
immediately execute up until right before the first yield instead?

A great example of why this behavior would defeat some of the purpose of
generators can be found in this amazing PDF presentation:

http://www.dabeaz.com/generators/Generators.pdf

Giving generators special case semantics for no good reason is a really
bad idea, so I'm very curious if there is a good reason for it being
this way. With the current semantics it means that errors can pop up at
unexpected times rather than the code failing fast.

Most assuredly they do have good reason. *Consider the cases in the PDF
I just mentioned. *Building generators that work on the output of other
generators allows assembling entire pipelines of behavior. *A very
powerful feature that would be impossible if the generators had the
semantics you describe.

If you want generators to behave as you suggest they should, then a
conventional for x in blah approach is likely the better way to go.

I use a generator anytime I want to be able to iterate across something
that has a potentially expensive cost, in terms of memory or cpu, to do
all at once.

The amount of concentration you can write in a program in a sitting
(fixed amount of time) is kind of limited. Sounds like @greedy was
the way to go. The recall implementation may have a short in the
future, but isn't functools kind of full? Has wraptools been
written? Is it any different?

Naming for @greedy also comes to question. My humble opinion muscles
glom on to @early vs. @late; @yieldprior; @dropfirst; @cooperative.
Thesaurus.com adds @ahead vs. @behind.