delay and force in Python

[Will Stuyvesant]

Here is a question for people who are more comfortable than
I am with new Python stuff like generators.

I am having fun implementing things from the Wizard book
(Abelson, Sussman, "Structure and Interpretation of Computer
Programs") in Python. In chapter 3.5 it is about streams as
delayed lists.

Streams are interesting because they are to lists like
xrange is to range. Could save a lot on memory and
computations.

Below is my straight translation from Scheme code in the
Wizard book to Python. It works, but now I want to rewrite
the delay and force functions using Python's new stuff,
generators or iterators or what-have-you. I have the
feeling that that is possible, can you do it?

The program below creates a stream with the numbers 1..995
and then filters the stream, keeping only the even numbers,
and then prints the second number in the stream (implemented
as the first number of the tail, just like in the 3.5
Section in the Wizard book).
.. # file: teststreams.py
..
.. def delay(exp): return lambda: exp
..
.. def force(o): return o()
..
.. def cons_stream(a,b ): return [a, delay(b)]
..
.. def stream_hd(s): return s[0]
..
.. # we use tl for cdr
.. def tl(x):
.. if len(x) == 2: return x[1]
.. else: return x[1:]
..
.. def stream_tl(s): return force(tl(s))
..
.. def stream_enumerat e_interval(low, high):
.. if low > high:
.. return None
.. else:
.. return cons_stream(
.. low,
.. stream_enumerat e_interval(low+ 1, high))
..
.. def stream_filter(p red, s):
.. if s is None:
.. return None
.. elif pred(stream_hd( s)):
.. return cons_stream(
.. stream_hd(s),
.. stream_filter(p red, stream_tl(s)))
.. else:
.. return stream_filter(p red, stream_tl(s))
..
.. def isEven(n): return n % 2 == 0
..
.. print stream_hd(strea m_tl(stream_fil ter(
.. isEven,
.. stream_enumerat e_interval(1,99 5))))
.. # 4

Something else: this crashes with a "maximum recursion reached"
.. print stream_enumerat e_interval(1,99 8)

while this does not crash
.. print stream_enumerat e_interval(1,90 0)
this means Python has a maximum of something like 900
recursions?

[floydophone]

Check this out:

def mygen():
for i in xrange(0, 10):
print "about to yield",i
yield i

g = mygen()
print "calling next()", g.next()
print "calling next()", g.next()
print "calling next()", g.next()

[Benji York]

Will Stuyvesant wrote:

Streams are interesting because they are to lists like
xrange is to range. Could save a lot on memory and
computations.
I think you're looking for generators.
Below is my straight translation from Scheme code in the
Wizard book to Python. Something else: this crashes with a "maximum recursion reached"
. print stream_enumerat e_interval(1,99 8)

Unlike Scheme, Python isn't designed for heavily recursive algorithms.
Here's a more Pythonic equivalent of your code:

def count(start, stop):
i = start
while i < stop:
yield i
i += 1

def even(gen):
for x in gen:
if x % 2 == 0:
yield x

numbers = even(count(1, 999))
first = numbers.next()
second = numbers.next()

print second
--
Benji York
be***@benjiyork .com

[Nick Coghlan]

Will Stuyvesant wrote:

The program below creates a stream with the numbers 1..995
and then filters the stream, keeping only the even numbers,
and then prints the second number in the stream (implemented
as the first number of the tail, just like in the 3.5
Section in the Wizard book).
How's this:

Py> from itertools import islice
Py> print islice((x for x in xrange(1, 996) if x % 2 == 0), 1, 2).next()
4

Breaking it into pieces:

Py> from itertools import islice
Py> stream = (x for x in xrange(1, 996) if x % 2 == 0)
Py> second_item = islice(stream, 1, 2).next()
Py> print second_item
4

And most of the stream hasn't been consumed yet:
Py> print stream.next()
6
Py> unconsumed = list(stream)
Py> len(unconsumed)
494

And this version has no problem with recursion limits:
Py> print islice((x for x in xrange(1, sys.maxint) if x % 2 == 0), 1, 2).next()
4

(xrange can't handle Python longs, unfortunately, so we *are* constrained by
sys.maxint. However, since my machine only has half a gig of RAM, the above is
still a damn sight quicker than the equivalent list comprehension would be!)
Something else: this crashes with a "maximum recursion reached"
. print stream_enumerat e_interval(1,99 8)

while this does not crash
. print stream_enumerat e_interval(1,90 0)
this means Python has a maximum of something like 900
recursions?

The CPython implementation is limited by the stack size allocated by the C
runtime library. The exact recursion limit is platform dependent, but something
around 1000 sounds fairly normal.

Cheers,
Nick.

--
Nick Coghlan | nc******@email. com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

[Dave Benjamin]

Will Stuyvesant wrote:

. def delay(exp): return lambda: exp

If you look at the definition of "delay" in SICP, you'll notice that
it's defined as "syntax sugar", in other words, a macro. Since Python
does not have macros, you'll have to just use "lambda", because by
defining "delay" as a function, you're forcing the expression "exp" to
evaluate immediately. In other words, theres a crucial difference between::

delay(sys.stdou t.write('evalua ted\n'))

and::

lambda: sys.stdout.writ e('evaluated\n' )

If you type in those two snippets, you'll see what I mean.

Coincidentally, I attempted a similar experiment just a couple of days
ago, and here's what I came up with::

# Stream.py - Stream class inspired by SICP

class Stream(object):
pass

class EndOfStream(Exc eption):
pass

class Item(Stream):
def __init__(self, current, nextf):
self.current = current
self.nextf = nextf

next = property(lambda self: self.nextf())

def fold(self, f, init):
return f(self.current, self.next.fold( f, init))

def __getitem__(sel f, n):
if n == 0:
return self.current
else:
return self.next[n - 1]

def __str__(self):
return '<Stream.Item: %s, ...>' % self.current
__repr__ = __str__

class End(Stream):
def fold(self, f, init):
return init

def __getitem__(sel f, n):
raise EndOfStream()

def _fail(self):
raise EndOfStream()

current = property(_fail)
next = property(_fail)

def __str__(self):
return '<Stream.End>'
__repr__ = __str__

Here's how it works::

Python 2.4 (#1, Dec 4 2004, 20:10:33)
[GCC 3.3.3 (cygwin special)] on cygwin
Type "help", "copyright" , "credits" or "license" for more information.

import Stream
s = Stream.Item(1, lambda: Stream.Item(2, lambda: Stream.End()))
s <Stream.Item: 1, ...> s.current 1 s.next <Stream.Item: 2, ...> s.next.current 2 s.next.next <Stream.End> s.next.next.nex t Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "Stream.py" , line 37, in _fail
raise EndOfStream()
Stream.EndOfStr eam def evens(n=0): ... return Stream.Item(n, lambda: evens(n + 2))
... s = evens()
s.current 0 s.next.current 2 s.next.next.cur rent

4

I don't know if this is useful or not, but it was an interesting
exercise nonetheless.

Dave

[Dave Benjamin]

Will Stuyvesant wrote:

Something else: this crashes with a "maximum recursion reached"
. print stream_enumerat e_interval(1,99 8)

while this does not crash
. print stream_enumerat e_interval(1,90 0)
this means Python has a maximum of something like 900
recursions?

See "sys.getrecursi onlimit()" and "sys.setrecursi onlimit()". The lack of
tail-call optimization means you probably won't get very far with this
approach (nor with mine). ;)

Dave

[Will Stuyvesant]

Yes you are right, if you just want to carry an expression
around then lambda does it; but delay was not intended as a
top-level function. Perhaps you think that my silly stream
implementation in the original post builds the whole list,
but it does not:

o = stream_enumerat e_interval(11,1 21)
print o [11, <function <lambda> at 0x00BA8670>] f = o[1]
r = f()
print r [12, <function <lambda> at 0x00BA86B0>]

instead it builds a list of lambda's, and that is so
desirable <wink>
Others suggested generators indeed, and I can imagine now a
usage pattern like
s = stream_generato r(initValue, next, stop, other_params)
stream_hd(strea m_tl(stream_fil ter(isEven,s)))

where the idea is to pass a generator to a function all the
time. What I don't like though is that I then, as I
understand it, have to code some for-loop inside that
function, I try to avoid for-loops. But a functional form
defined with a for-loop does not seem all that bad. Anyway
I have some reading-up to do, about things some posters use
and I forgot to keep up with since I have been coding Python
1.5.2 style for ages now: properties, itertools, ...printed
some stuff already, thanks!

[infidel]

It took me a while to figure out what the "translated " code was trying
to do. Here's a quick example that I think accomplishes the same
thing:

for i, n in enumerate(x for x in xrange(998) if x % 2 == 0):

.... if i == 1:
.... print n
....
2

[Bengt Richter]

On Wed, 19 Jan 2005 23:53:28 +1000, Nick Coghlan <nc******@iinet .net.au> wrote:
[...]

Something else: this crashes with a "maximum recursion reached"
. print stream_enumerat e_interval(1,99 8)

while this does not crash
. print stream_enumerat e_interval(1,90 0)
this means Python has a maximum of something like 900
recursions?

The CPython implementation is limited by the stack size allocated by the C
runtime library. The exact recursion limit is platform dependent, but something
around 1000 sounds fairly normal.

ISTM you forgot something ;-)
(I.e., that 1000 is only a default value (don't know if same on all platforms)).

import sys
help(sys.setrec ursionlimit)

Help on built-in function setrecursionlim it in module sys:

setrecursionlim it(...)
setrecursionlim it(n)

Set the maximum depth of the Python interpreter stack to n. This
limit prevents infinite recursion from causing an overflow of the C
stack and crashing Python. The highest possible limit is platform-
dependent.

Regards,
Bengt Richter

[infidel]

Of course I meant to put a break out of the loop after the print
statement. Duh on me.