frange() question

[George Trojan]

A while ago I found somewhere the following implementation of frange():

def frange(limit1, limit2 = None, increment = 1.):
"""
Range function that accepts floats (and integers).
Usage:
frange(-2, 2, 0.1)
frange(10)
frange(10, increment = 0.5)
The returned value is an iterator. Use list(frange) for a list.
"""
if limit2 is None:
limit2, limit1 = limit1, 0.
else:
limit1 = float(limit1)
count = int(math.ceil(limit2 - limit1)/increment)
return (limit1 + n*increment for n in range(count))

I am puzzled by the parentheses in the last line. Somehow they make
frange to be a generator:

>print type(frange(1.0, increment=0.5))

<type 'generator'>
But I always thought that generators need a keyword "yield". What is
going on here?

George

[Tim Golden]

George Trojan wrote:

A while ago I found somewhere the following implementation of frange():

..
..
..

return (limit1 + n*increment for n in range(count))

I am puzzled by the parentheses in the last line. Somehow they make
frange to be a generator:

But I always thought that generators need a keyword "yield". What is
going on here?

That's what's known as a generator expression:

http://docs.python.org/ref/genexpr.html
http://www.python.org/dev/peps/pep-0289/

TJG

[Stargaming]

On Thu, 20 Sep 2007 09:08:17 -0400, George Trojan wrote:

A while ago I found somewhere the following implementation of frange():

def frange(limit1, limit2 = None, increment = 1.):
"""
Range function that accepts floats (and integers). Usage:
frange(-2, 2, 0.1)
frange(10)
frange(10, increment = 0.5)
The returned value is an iterator. Use list(frange) for a list.
"""
if limit2 is None:
limit2, limit1 = limit1, 0.
else:
limit1 = float(limit1)
count = int(math.ceil(limit2 - limit1)/increment) return (limit1 +
n*increment for n in range(count))

I am puzzled by the parentheses in the last line. Somehow they make
frange to be a generator:

>print type(frange(1.0, increment=0.5))

<type 'generator'>
But I always thought that generators need a keyword "yield". What is
going on here?

George

Consider the following:

def foo():
yield 1
def bar():
return foo()

Still, ``type(bar())`` would be a generator.

I don't want to tell you anything wrong because I don't know how
generators are implemented on the C level but it's more like changing
foo's (or frange's, in your example) return value.

HTH,
Stargaming

[John J. Lee]

George Trojan <ge***********@noaa.govwrites:

A while ago I found somewhere the following implementation of frange():

def frange(limit1, limit2 = None, increment = 1.):
"""
Range function that accepts floats (and integers).
Usage:
frange(-2, 2, 0.1)
frange(10)
frange(10, increment = 0.5)
The returned value is an iterator. Use list(frange) for a list.
"""
if limit2 is None:
limit2, limit1 = limit1, 0.
else:
limit1 = float(limit1)
count = int(math.ceil(limit2 - limit1)/increment)
return (limit1 + n*increment for n in range(count))

I am puzzled by the parentheses in the last line. Somehow they make
frange to be a generator:

>>print type(frange(1.0, increment=0.5))

<type 'generator'>

Functions are never generators, senso stricto. There are "generator
functions", which *return* (or yield) generators when you call them.
It's true sometimes people refer to generator functions as simply
"generators", but in examples like the above, it's useful to remember
that they are two different things.

In this case, frange isn't a generator function, because it doesn't
yield. Instead, it returns the result of evaluating a generator
expression (a generator). The generatator expression plays the same
role as a generator function -- calling a generator function gives you
a generator object; evaluating a generator expression gives you a
generator object. There's nothing to stop you returning that
generator object, which makes this function behave just like a regular
generator function.
John

[Carsten Haese]

On Thu, 2007-09-20 at 18:55 +0000, John J. Lee wrote:

Functions are never generators, senso stricto. There are "generator
functions", which *return* (or yield) generators when you call them.

Actually, a generator function is a function that returns a generator.
The generator, in turn, is an object that wraps the iterator protocol
around a resumable function. When the generator's next() method is
called, it resumes its wrapped function that runs until it yields or
finishes. When the wrapped function yields a value, it is suspended, and
the yielded value is returned as the result of next(). If the wrapped
function finished, next() raises StopIteration.

Generators are a special case of iterators. Any object that implements
the iterator protocol (which among other less important things mandates
a next() method that returns the next value or raises StopIteration) is
an iterator. Generators simply have their special way (by calling into a
resumable function) of implementing the iterator protocol, but any
stateful object that returns a value or raises StopIteration in next()
is an iterator.

It's true sometimes people refer to generator functions as simply
"generators", but in examples like the above, it's useful to remember
that they are two different things.

In this case, frange isn't a generator function, because it doesn't
yield. Instead, it returns the result of evaluating a generator
expression (a generator). The generatator expression plays the same
role as a generator function -- calling a generator function gives you
a generator object; evaluating a generator expression gives you a
generator object. There's nothing to stop you returning that
generator object, which makes this function behave just like a regular
generator function.

Generator expressions still "yield", but you don't see the yield in the
Python source code. Observe:

>>def gf1():

.... for i in (1,2,3):
.... yield i
....

>>def gf2():

.... return (i for i in (1,2,3))
....

>>g1 = gf1()
g2 = gf2()
g1

<generator object at 0xb7f66d8c>

>>g2

<generator object at 0xb7f66f4c>

>>dir(g1)

['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__',
'__init__', '__iter__', '__new__', '__reduce__', '__reduce_ex__',
'__repr__', '__setattr__', '__str__', 'close', 'gi_frame', 'gi_running',
'next', 'send', 'throw']

>>dir(g2)

['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__',
'__init__', '__iter__', '__new__', '__reduce__', '__reduce_ex__',
'__repr__', '__setattr__', '__str__', 'close', 'gi_frame', 'gi_running',
'next', 'send', 'throw']

>>import dis
dis.dis(g1.gi_frame.f_code.co_code)

0 SETUP_LOOP 19 (to 22)
3 LOAD_CONST 4 (4)
6 GET_ITER

> 7 FOR_ITER 11 (to 21)

10 STORE_FAST 0 (0)
13 LOAD_FAST 0 (0)
16 YIELD_VALUE
17 POP_TOP
18 JUMP_ABSOLUTE 7

> 21 POP_BLOCK
22 LOAD_CONST 0 (0)

25 RETURN_VALUE

>>dis.dis(g2.gi_frame.f_code.co_code)

0 SETUP_LOOP 18 (to 21)
3 LOAD_FAST 0 (0)

> 6 FOR_ITER 11 (to 20)

9 STORE_FAST 1 (1)
12 LOAD_FAST 1 (1)
15 YIELD_VALUE
16 POP_TOP
17 JUMP_ABSOLUTE 6

> 20 POP_BLOCK
21 LOAD_CONST 0 (0)

24 RETURN_VALUE

As you can see, except for a minor optimization in the byte code, there
is no discernible difference between the generator that's returned from
a generator expression and the generator that's returned from the
generator function. Note in particular that the byte code for g2
contains a YIELD_VALUE operation just like g1 does.

In fact, generator expressions are merely a short-hand notation for
simple generator functions of a particular form. The generator
expression

g = (expr(x) for x in some_iterable)

produces a generator that is almost indistinguishable from, and
operationally identical to, the generator produced by the following
code:

def __anonymous():
for x in some_iterable:
yield expr(x)
g = __anonymous()
del __anonymous

Hope this helps,

--
Carsten Haese
http://informixdb.sourceforge.net

[Michael J. Fromberger]

In article <fc**********@news.nems.noaa.gov>,
George Trojan <ge***********@noaa.govwrote:

A while ago I found somewhere the following implementation of frange():

def frange(limit1, limit2 = None, increment = 1.):
"""
Range function that accepts floats (and integers).
Usage:
frange(-2, 2, 0.1)
frange(10)
frange(10, increment = 0.5)
The returned value is an iterator. Use list(frange) for a list.
"""
if limit2 is None:
limit2, limit1 = limit1, 0.
else:
limit1 = float(limit1)
count = int(math.ceil(limit2 - limit1)/increment)
return (limit1 + n*increment for n in range(count))

I am puzzled by the parentheses in the last line. Somehow they make
frange to be a generator:

>print type(frange(1.0, increment=0.5))

<type 'generator'>
But I always thought that generators need a keyword "yield". What is
going on here?

Hi, George,

The expression returned is a "generator expression",

return (limit1 + n*increment for n in range(count))

Thus, although frange itself is not written as a generator, it does
return a generator as its result. The syntax is like that of list
comprehensions; see:

<http://docs.python.org/ref/genexpr.html>

Cheers,
-M

--
Michael J. Fromberger | Lecturer, Dept. of Computer Science
http://www.dartmouth.edu/~sting/ | Dartmouth College, Hanover, NH, USA

[John J. Lee]

Carsten Haese <ca*****@uniqsys.comwrites:

On Thu, 2007-09-20 at 18:55 +0000, John J. Lee wrote:

>Functions are never generators, senso stricto. There are "generator
functions", which *return* (or yield) generators when you call them.

Actually, a generator function is a function that returns a generator.

Read what I wrote again. What makes you begin your sentence with
"Actually", rather than "Putting it another way"?
[...snip more sensible stuff I must assume was directed at the OP...]
John