delay and force in Python

[Will Stuyvesant]

Here is a question for people who are more comfortable than
I am with new Python stuff like generators.

I am having fun implementing things from the Wizard book
(Abelson, Sussman, "Structure and Interpretation of Computer
Programs") in Python. In chapter 3.5 it is about streams as
delayed lists.

Streams are interesting because they are to lists like
xrange is to range. Could save a lot on memory and
computations.

Below is my straight translation from Scheme code in the
Wizard book to Python. It works, but now I want to rewrite
the delay and force functions using Python's new stuff,
generators or iterators or what-have-you. I have the
feeling that that is possible, can you do it?

The program below creates a stream with the numbers 1..995
and then filters the stream, keeping only the even numbers,
and then prints the second number in the stream (implemented
as the first number of the tail, just like in the 3.5
Section in the Wizard book).
.. # file: teststreams.py
..
.. def delay(exp): return lambda: exp
..
.. def force(o): return o()
..
.. def cons_stream(a,b ): return [a, delay(b)]
..
.. def stream_hd(s): return s[0]
..
.. # we use tl for cdr
.. def tl(x):
.. if len(x) == 2: return x[1]
.. else: return x[1:]
..
.. def stream_tl(s): return force(tl(s))
..
.. def stream_enumerat e_interval(low, high):
.. if low > high:
.. return None
.. else:
.. return cons_stream(
.. low,
.. stream_enumerat e_interval(low+ 1, high))
..
.. def stream_filter(p red, s):
.. if s is None:
.. return None
.. elif pred(stream_hd( s)):
.. return cons_stream(
.. stream_hd(s),
.. stream_filter(p red, stream_tl(s)))
.. else:
.. return stream_filter(p red, stream_tl(s))
..
.. def isEven(n): return n % 2 == 0
..
.. print stream_hd(strea m_tl(stream_fil ter(
.. isEven,
.. stream_enumerat e_interval(1,99 5))))
.. # 4

Something else: this crashes with a "maximum recursion reached"
.. print stream_enumerat e_interval(1,99 8)

while this does not crash
.. print stream_enumerat e_interval(1,90 0)
this means Python has a maximum of something like 900
recursions?

[Will Stuyvesant]

Just for the record, an implementation without using generators,
somewhat like in Sect. 3.5 of the Wizard book, and without recursion
limit problems.
..
.. def stream_hd(s): # the head of the stream
.. return s[0]
..
.. def stream_tl(s): # the tail of the stream
.. return s[1]()
..
.. ##
.. # The low argument is required: the first element of the stream.
.. # You either use high= or stop_f= to define the end of the
.. # stream. Omit both for an infinite stream. stop_f is a function
.. # that takes low as argument and returns True or False.
.. # The next_f function takes one argument (an element of the stream,
.. # same type as the low argument has)
.. # The stop_f function takes one argument (same type as low)
.. # @param low object
.. # @param high object
.. # @param next_f function
.. # @param stop_f function
.. # @return list, a stream
.. def make_stream(low , high=None, next_f=None, stop_f=None):
.. if next_f is None: next_f = lambda x: x + 1
.. if high is not None: # using high
.. if low > high:
.. return None
.. else:
.. return [low, lambda: make_stream(
.. next_f(low), high=high, next_f=next_f)]
.. elif stop_f is not None: # using stop_f
.. if low > stop_f(low):
.. return None
.. else:
.. return [low, lambda: make_stream(
.. next_f(low), next_f=next_f, stop_f=stop_f)]
.. else: # infinite stream
.. return [low, lambda: make_stream(
.. next_f(low), next_f=next_f)]
..
.. ##
.. # iterative version of filter
.. # @param pred function, (stream-element) -> bool
.. # @param s list, a stream
.. # @return list, a stream
.. def stream_filter(p red, s):
.. if s is None: return []
.. while True:
.. elem = stream_hd(s)
.. if pred(elem):
.. return [elem, lambda: stream_filter(p red, stream_tl(s))]
.. else:
.. s = stream_tl(s)
.. if s is None: return []
..
..
.. if __name__ == '__main__':
..
.. def is100some(x): return x % 100 == 0
.. assert(stream_h d(stream_tl(str eam_tl(stream_f ilter(
.. is100some,
.. make_stream(1, 11111111111111) )))) == 300)
..
.. def add_33(x): return x + 33
.. assert(stream_h d(stream_tl(str eam_filter(
.. is100some,
.. # stream 1,34,67,100,...
.. make_stream(1,9 99999999, next_f = add_33)))) == 3400)
..
.. assert(stream_h d(stream_tl(str eam_filter(
.. is100some,
.. # infinite stream 1,2,...
.. make_stream(1)) )) == 200)
..
.. def mod_20000(x): return x % 20000 == 0
.. # this will need more evaluations than the recursion limit count
.. infinite_filter = stream_filter(m od_20000, make_stream(1))
.. assert( stream_hd(strea m_tl(infinite_f ilter)) == 40000 )
..

[Nick Coghlan]

Nick Coghlan wrote:

Will Stuyvesant wrote:

The program below creates a stream with the numbers 1..995
and then filters the stream, keeping only the even numbers,
and then prints the second number in the stream (implemented
as the first number of the tail, just like in the 3.5
Section in the Wizard book).

How's this:

Py> from itertools import islice
Py> print islice((x for x in xrange(1, 996) if x % 2 == 0), 1, 2).next()
4

Wouldn't it be nice if this could be spelt:

print (x for x in xrange(1, 996) if x % 2 == 0)[2]

Well, I just put a patch on SF to enable exactly that:
http://www.python.org/sf/1108272

Cheers,
Nick.

--
Nick Coghlan | nc******@email. com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

[Fredrik Lundh]

Nick Coghlan wrote:

How's this:

Py> from itertools import islice
Py> print islice((x for x in xrange(1, 996) if x % 2 == 0), 1, 2).next()
4

Wouldn't it be nice if this could be spelt:

print (x for x in xrange(1, 996) if x % 2 == 0)[2]

as I've always said, the sooner we can all use the itertools goodies without
even noticing, the better.

</F>

[Peter Otten]

Nick Coghlan wrote:

Py> print islice((x for x in xrange(1, 996) if x % 2 == 0), 1, 2).next()
4

Wouldn't it be nice if this could be spelt:

print (x for x in xrange(1, 996) if x % 2 == 0)[2]

Well, I just put a patch on SF to enable exactly that:
http://www.python.org/sf/1108272

I like it. Of course you always have to bear in mind that one giant leap for
a list could be _many_ small steps for an iterator.

Peter

[Nick Coghlan]

Peter Otten wrote:

Nick Coghlan wrote:

Py> print islice((x for x in xrange(1, 996) if x % 2 == 0), 1, 2).next()
4

Wouldn't it be nice if this could be spelt:

print (x for x in xrange(1, 996) if x % 2 == 0)[2]

Well, I just put a patch on SF to enable exactly that:
http://www.python.org/sf/1108272

I like it. Of course you always have to bear in mind that one giant leap for
a list could be _many_ small steps for an iterator.

Indeed. The main cases I am thinking of involve picking off the first few items
of an iterator (either to use them, or to throw them away before using the rest).

And if an app actually *needs* random access, there's a reason lists still exist ;)

Cheers,
Nick.

--
Nick Coghlan | nc******@email. com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

[John J. Lee]

Nick Coghlan <nc******@iinet .net.au> writes:
[...]

(xrange can't handle Python longs, unfortunately, so we *are*
constrained by sys.maxint. However, since my machine only has half a
gig of RAM, the above is still a damn sight quicker than the
equivalent list comprehension would be!)

[...]

Other way 'round: if you had even more RAM, the listcomp would be even
slower for this job!
John

[Bengt Richter]

On Tue, 25 Jan 2005 23:53:26 +1000, Nick Coghlan <nc******@iinet .net.au> wrote:

Peter Otten wrote:

Nick Coghlan wrote:

Py> print islice((x for x in xrange(1, 996) if x % 2 == 0), 1, 2).next()
4

Wouldn't it be nice if this could be spelt:

print (x for x in xrange(1, 996) if x % 2 == 0)[2]

Well, I just put a patch on SF to enable exactly that:
http://www.python.org/sf/1108272

I like it. Of course you always have to bear in mind that one giant leap for
a list could be _many_ small steps for an iterator.

Indeed. The main cases I am thinking of involve picking off the first few items
of an iterator (either to use them, or to throw them away before using the rest).

And if an app actually *needs* random access, there's a reason lists still exist ;)

You can bail out of a generator expression with a raise-StopIteration expression spelled iter([]).next() ;-)

def show(x): print x,; return x ... list(show(x) for x in xrange(20) if x<8 or iter([]).next()) 0 1 2 3 4 5 6 7
[0, 1, 2, 3, 4, 5, 6, 7]

Well, since
list(show(x) for x in xrange(20) if x<8) 0 1 2 3 4 5 6 7
[0, 1, 2, 3, 4, 5, 6, 7]

this change due to adding iter([]).next() might be more convincing:
list(show(x) for x in xrange(20) if x<8 or True) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]

list(show(x) for x in xrange(20) if x<8 or iter([]).next() or True) 0 1 2 3 4 5 6 7
[0, 1, 2, 3, 4, 5, 6, 7]
Applied to example,
print list(x for i,x in enumerate(x for x in xrange(1, 996) if x % 2 ==0) if i<3 or iter([]).next())[2] 6

or in case you just want one item as result,
print list(x for i,x in enumerate(x for x in xrange(1, 996) if x % 2 ==0) if i==2 or i==3 and iter([]).next())[0]

6

Regards,
Bengt Richter

[Nick Coghlan]

Bengt Richter wrote:

You can bail out of a generator expression with a raise-StopIteration expression spelled iter([]).next() ;-)

>>> list(show(x) for x in xrange(20) if x<8 or iter([]).next() or True)

0 1 2 3 4 5 6 7
[0, 1, 2, 3, 4, 5, 6, 7]

This is both neat and incredibly arcane at the same time :)

Cheers,
Nick.
--
Nick Coghlan | nc******@email. com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net