pre-PEP: Simple Thunks

[Brian Sabbey]

Here is a first draft of a PEP for thunks. Please let me know what you
think. If there is a positive response, I will create a real PEP.

I made a patch that implements thunks as described here. It is available
at:
http://staff.washington.edu/sabbey/py_do

Good background on thunks can be found in ref. [1].

Simple Thunks
-------------

Thunks are, as far as this PEP is concerned, anonymous functions that
blend into their environment. They can be used in ways similar to code
blocks in Ruby or Smalltalk. One specific use of thunks is as a way to
abstract acquire/release code. Another use is as a complement to
generators.

A Set of Examples
=============== ==

Thunk statements contain a new keyword, 'do', as in the example below. The
body of the thunk is the suite in the 'do' statement; it gets passed to
the function appearing next to 'do'. The thunk gets inserted as the first
argument to the function, reminiscent of the way 'self' is inserted as the
first argument to methods.

def f(thunk):
before()
thunk()
after()

do f():
stuff()

The above code has the same effect as:

before()
stuff()
after()

Other arguments to 'f' get placed after the thunk:

def f(thunk, a, b):
# a == 27, b == 28
before()
thunk()
after()

do f(27, 28):
stuff()

Thunks can also accept arguments:

def f(thunk):
thunk(6,7)

do x,y in f():
# x==6, y==7
stuff(x,y)

The return value can be captured

def f(thunk):
thunk()
return 8

do t=f():
# t not bound yet
stuff()

print t
==> 8

Thunks blend into their environment

def f(thunk):
thunk(6,7)

a = 20
do x,y in f():
a = 54
print a,x,y

==> 54,6,7

Thunks can return values. Since using 'return' would leave it unclear
whether it is the thunk or the surrounding function that is returning, a
different keyword should be used. By analogy with 'for' and 'while' loops,
the 'continue' keyword is used for this purpose:

def f(thunk):
before()
t = thunk()
# t == 11
after()

do f():
continue 11

Exceptions raised in the thunk pass through the thunk's caller's frame
before returning to the frame in which the thunk is defined:

def catch_everythin g(thunk):
try:
thunk()
except:
pass # SomeException gets caught here

try:
do catch_everythin g():
raise SomeException
except:
pass # SomeException doesn't get caught here because it was
already caught

Because thunks blend into their environment, a thunk cannot be used after
its surrounding 'do' statement has finished:

thunk_saver = None
def f(thunk):
global thunk_saver
thunk_saver = thunk

do f():
pass

thunk_saver() # exception, thunk has expired

'break' and 'return' should probably not be allowed in thunks. One could
use exceptions to simulate these, but it would be surprising to have
exceptions occur in what would otherwise be a non-exceptional situation.
One would have to use try/finally blocks in all code that calls thunks
just to deal with normal situations. For example, using code like

def f(thunk):
thunk()
prevent_core_me ltdown()

with code like

do f():
p = 1
return p

would have a different effect than using it with

do f():
return 1

This behavior is potentially a cause of bugs since these two examples
might seem identical at first glance.

The thunk evaluates in the same frame as the function in which it was
defined. This frame is accessible:

def f(thunk):
frame = thunk.tk_frame

do f():
pass

Motivation
==========

Thunks can be used to solve most of the problems addressed by PEP 310 [2]
and PEP 288 [3].

PEP 310 deals with the abstraction of acquire/release code. Such code is
needed when one needs to acquire a resource before its use and release it
after. This often requires boilerplate, it is easy to get wrong, and
there is no visual indication that the before and after parts of the code
are related. Thunks solve these problems by allowing the acquire/release
code to be written in a single, re-usable function.

def acquire_release (thunk):
f = acquire()
try:
thunk(f)
finally:
f.release()

do t in acquire_release ():
print t

More generally, thunks can be used whenever there is a repeated need for
the same code to appear before and after other code. For example,

do WaitCursor():
compute_for_a_l ong_time()

is more organized, easier to read and less bug-prone than the code

DoWaitCursor(1)
compute_for_a_l ong_time()
DoWaitCursor(-1)

PEP 288 tries to overcome some of the limitations of generators. One
limitation is that a 'yield' is not allowed in the 'try' block of a
'try'/'finally' statement.

def get_items():
f = acquire()
try:
for i in f:
yield i # syntax error
finally:
f.release()

for i in get_items():
print i

This code is not allowed because execution might never return after the
'yield' statement and therefore there is no way to ensure that the
'finally' block is executed. A prohibition on such yields lessens the
suitability of generators as a way to produce items from a resource that
needs to be closed. Of course, the generator could be wrapped in a class
that closes the resource, but this is a complication one would like to
avoid, and does not ensure that the resource will be released in a timely
manner. Thunks do not have this limitation because the thunk-accepting
function is in control-- execution cannot break out of the 'do' statement
without first passing through the thunk-accepting function.

def get_items(thunk ): # <-- "thunk-accepting function"
f = acquire()
try:
for i in f:
thunk(i) # A-OK
finally:
f.release()

do i in get_items():
print i

Even though thunks can be used in some ways that generators cannot, they
are not nearly a replacement for generators. Importantly, one has no
analogue of the 'next' method of generators when using thunks:

def f():
yield 89
yield 91

g = f()
g.next() # == 89
g.next() # == 91

[1] see the "Extended Function syntax" thread,
http://mail.python.org/pipermail/pyt...2003-February/
[2] http://www.python.org/peps/pep-0310.html
[3] http://www.python.org/peps/pep-0288.html

[Leif K-Brooks]

Brian Sabbey wrote:

Thunk statements contain a new keyword, 'do', as in the example below.
The body of the thunk is the suite in the 'do' statement; it gets passed
to the function appearing next to 'do'. The thunk gets inserted as the
first argument to the function, reminiscent of the way 'self' is
inserted as the first argument to methods.
It would probably make more sense to pass the thunk as the last
argument, not as the first. That would make it easier to create
functions with optional thunks, as in:

def print_nums(star t, end, thunk=None):
for num in xrange(start, end+1):
if thunk is not None:
num = thunk(num)
print num

print_nums(1, 3) # prints 1, 2, 3

do num print_nums(1, 3): # prints 2, 4, 6
continue num * 2
Because thunks blend into their environment, a thunk cannot be used
after its surrounding 'do' statement has finished

Why? Ordinary functions don't have that restriction:

def foo(): .... x = 1
.... def bar():
.... return x
.... return bar
.... foo()()

1

[peufeu]

I think your proposal is very interesting, I've been missing code blocks
in Python more and more as time goes by.
I'll answer to both the 'thunks" proposal and the "suite-based keywords"
proposal here.

I find the Ruby syntax rather dirty though, because it has a lot of
implicit stuff, treats code blocks as different from normal arguments,
allows passing only one code block, needs a proc keyword, has yield
execute an implicit block... all this, coming from a Python "explicit is
better than implicit" background (which makes a lot of sense) is simply
ugly.
I like your syntax but have a few comments.
I'll give you an unordered list of ideas, up to you to do what you like
with them.

Keep in mind that most of the problems come from the "space is
significant" thing, which is IMHO a very good idea, but prevents us from
putting code in expressions, like :

func( a,b, def callback( x ):
print x
)

or does it ? maybe this syntax could be made to work ?

*************** *************** **********
Comments on the thunks.

First of all I view code blocks as essential to a language. They are very
useful for a lot of common programming tasks (like defining callbacks in
an elegant way) :

button = create_button( "Save changes" ):
do
self.save()

However it seems your thunks can't take parameters, which to me is a big
drawback. In ruby a thunk can take parameters. Simple examples :

field = edit_field( "initial value", onsubmit={ |value| if
self.validate(v alue) then do something else alert( "the value is invalid"
) } )
[1,3,4].each { |x| puts x }

This has the advantage that the interface to the thunk (ie. its
parameters) are right there before your eyes instead of being buried in
the thunk invocation code inside the edit_field.

a more complex random example :

fields['password1'] = edit_field( "Enter Password" )
fields['password2'] = edit_field( "Enter it again", onsubmit = {|value,
other_values| if value != other_values['password_1'] then alert('the two
passwords must be the same !") }

So I think it's essential that thunks take parameters and return a value
(to use them effectively as callbacks).
What shall distinguish them from a simple alteration to def(): which
returns the function as a value, and an optional name ? really I don't
know, but it could be the way they handle closures and share local
variables with the defining scope. Or it could be that there is no need
for two kinds of function/blocks and so we can reuse the keyword def() :

If you wish to modify def(), you could do, without creating any keyword :

# standard form
f = def func( params ):
code

# unnamed code block taking params
func = def (params):
code

Note that the two above are equivalent with regard to the variable
"func", ie. func contains the defined function. Actually I find def
funcname() to be bloat, as funcname = def() has the same functionality,
but is a lot more universal.

# unnamed block taking no params
f = def:
code

*************** *************** *************** ******
Comments on the suite-based keywords.

Did you notice that this was basically a generalized HEREDOC syntax ?

I'd say that explicit is better than implicit, hence...

Your syntax is :

do f(a,b):
a block

passes block as the last parameter of f.
I don't like it because it hides stuff.

I'd write :

f(a,b,@>,@>):
"""a very
large multi-line
string"""
def (x):
print x

Here the @> is a symbol (use whatever you like) to indicate "placeholde r
for something which is on the next line".
Indentation indicates that the following lines are indeed argument for
the function. The : at the end of the function call is there for coherence
with the rest of the syntax.
Notice that, this way, there is no need for a "do" keyword, as the code
block created by an anonymous def() is no different that the other
parameter, in this case a multiline string.

This has many advantages.

It will make big statements more readable :

instead of :
f( a,b, [some very big expression made up of nested class constructors
like a form defintion ], c, d )

write :
f( a, b, @>, c, d ):
[the very big expression goes here]

So, independently of code blocks, this already improves the readability
of big statements.
You could also use named parameters (various proposals):

f( a,b, c=@>, d=@> ):
value of c
value of d

or :

f( a,b, @*> ):
value of c
value of d

f( a,b, @**> ):
c: value of c
d: value of d

Notice how this mimics f( a,b, * ) and f(a,b, ** ) for multiple
arguments, and multiple named arguments. Do you like it ? I do. Especially
the named version where you cant' get lost in the param block because you
see their names !

Now if you say that def returns the defined function as a value, you
don't need a do keyword.

So, for instance :

def withfile( fname, thunk, mode = "r" ):
f = open( fname, mode )
thunk(f)
f.close()

then :

withfile( "afile.txt" , @>, "w" ):
def (f):
f.write( something )

Now, you may say that the def on an extra line is ugly, then just write :

withfile( "afile.txt" , @>, "w" ): def (f):
f.write( something )

If you really like do you can make it a synonym for def and then :

withfile( "afile.txt" , @>, "w" ): do (f):
f.write( something )

The two ":" seem a bit weird but I think they're OK.

'break' and 'return' should probably not be allowed in thunks. One
I do think return should be allowed in a thunk. After all it's a function
block, so why ditch useful functionality ?
yield should also be allowed, after all why can a thunk not be a
generator ? This would be powerful.

def withfile( fname, thunk, mode = "r" ):
f = open( fname, mode )
r = thunk(f)
f.close()
return r

val = withfile( "afile.txt" , @>, "w" ):
def (f):
f.write( something )
return something

Well, it seems I have no more ideas for now.
What do you think about all this ?
The thunk evaluates in the same frame as the function in which it was
defined. This frame is accessible:

Hm ?
You mean like a function closure, or that local variables defined inside
the thunk will be visible to the caller ?
This might change a lot of things and it becomes like a continuation, are
you going to recode stackless ?

[Kay Schluehr]

Brian Sabbey wrote:

def get_items(thunk ): # <-- "thunk-accepting function"
f = acquire()
try:
for i in f:
thunk(i) # A-OK
finally:
f.release()

do i in get_items():
print i

Seems like You want to solve the addressed generator problem by
manipulating the syntax: "make generators look more function like",
because father compiler won't complain ;-) Sorry, but IMO this is
hackery and has nothing to do with good language design and I consider
this as extremely harmfull. Instead of making things explicit it does
it the other way round and tries to make Python code more obscure.
Moreover I can't notice the superiority of thunks in Your other
examples over more common techniques like decorators for pre- and
postconditions and the GOF command pattern. I thinks the place for such
ideas are Michael Hudsons famous bytecodehacks.

-1 for from me for thunks in Python.

Ciao,
Kay

[Bengt Richter]

On Fri, 15 Apr 2005 16:44:58 -0700, Brian Sabbey <sa****@u.washi ngton.edu> wrote:

Here is a first draft of a PEP for thunks. Please let me know what you
think. If there is a positive response, I will create a real PEP.

I made a patch that implements thunks as described here. It is available
at:
http://staff.washington.edu/sabbey/py_do

Good background on thunks can be found in ref. [1].
UIAM most of that pre-dates decorators. What is the relation of thunks
to decorators and/or how might they interact?

Simple Thunks
-------------

Thunks are, as far as this PEP is concerned, anonymous functions that
blend into their environment. They can be used in ways similar to code
blocks in Ruby or Smalltalk. One specific use of thunks is as a way to
abstract acquire/release code. Another use is as a complement to
generators.
"blend into their environment" is not very precise ;-)
If you are talking about the code executing in the local namespace
as if part of a suite instead of apparently defined in a separate function,
I think I would prefer a different syntax ;-)

A Set of Examples
============== ===

Thunk statements contain a new keyword, 'do', as in the example below. The
body of the thunk is the suite in the 'do' statement; it gets passed to
the function appearing next to 'do'. The thunk gets inserted as the first
argument to the function, reminiscent of the way 'self' is inserted as the
first argument to methods.

def f(thunk):
before()
thunk()
after()

do f():
stuff()

The above code has the same effect as:

before()
stuff()
after() Meaning "do" forces the body of f to be exec'd in do's local space? What if there
are assignments in f? I don't think you mean that would get executed in do's local space,
that's what the thunk call is presumably supposed to do...

But let's get on to better examples, because this is probably confusing some, and I think there
are better ways to spell most use cases than we're seeing here so far ;-)

I want to explore using the thunk-accepting function as a decorator, and defining an anonymous
callable suite for it to "decorate" instead of using the do x,y in deco: or do f(27, 28): format.

To define an anonymous callable suite (aka thunk), I suggest the syntax for
do x,y in deco:
suite
should be
@deco
(x, y): # like def foo(x, y): without the def and foo
suite

BTW, just dropping the def makes for a named thunk (aka callable suite), e.g.
foo(x, y):
suite
which you could call like
foo(10, 4)
with the local-where-suite-was-define effect of
x = 10
y = 4
suite

BTW, a callable local suite also makes case switching by calling through locals()[xsuitename]()
able to rebind local variables. Also, since a name is visible in an enclosing scope, it could
conceivably provide a mechanism for rebinding there. E.g.,

def outer():
xsuite(arg):
x = arg
def inner():
xsuite(5)
x = 2
print x # => 2
inner()
print x # => 5

But it would be tricky if outer returned inner as a closure.
Or if it returned xsuite, for that matter. Probably simplest to limit
callable suites to the scope where they're defined.

Other arguments to 'f' get placed after the thunk:

def f(thunk, a, b):
# a == 27, b == 28
before()
thunk()
after()

do f(27, 28):
stuff() I'm not sure how you intend this to work. Above you implied (ISTM ;-)
that the entire body of f would effectively be executed locally. But is that
true? What if after after() in f, there were a last statment hi='from last statement of f'
Would hi be bound at this point in the flow (i.e., after d f(27, 28): stuff() )?

I'm thinking you didn't really mean that. IOW, by magic at the time of calling thunk from the
ordinary function f, thunk would be discovered to be what I call an executable suite, whose
body is the suite of your do statement.

In that case, f iself should not be a callable suite, since its body is _not_ supposed to be called locally,
and other than the fact that before and after got called, it was not quite exact to say it was _equivalent_ to

before()
stuff() # the do suite
after()

In that case, my version would just not have a do, instead defining the do suite
as a temp executable suite, e.g., if instead
we make an asignment in the suite, to make it clear it's not just a calling thing, e.g.,

do f(27, 28):
x = stuff()

then my version with explict name callable suite would be

def f(thunk, a, b):
# a == 27, b == 28
before()
thunk()
after()

set_x():
x = stuff() # to make it plain it's not just a calling thing

f(set_x, 27, 28)
# x is now visible here as local binding

but a suitable decorator and an anonymous callable suite (thunk defined my way ;-) would make this

@f(27, 28)
(): x = stuff()


Thunks can also accept arguments:

def f(thunk):
thunk(6,7)

do x,y in f():
# x==6, y==7
stuff(x,y)
IMO
@f
(x, y): stuff(x, y) # like def foo(x, y): stuff(x, y)

is clearer, once you get used to the missing def foo format

The return value can be captured
This is just a fallout of f's being an ordinary function right?
IOW, f doesn't really know thunk is not an ordinary callable too?
I imagine that is for the CALL_FUNCTION byte code implementation to discover?
def f(thunk):
thunk()
return 8

do t=f():
# t not bound yet
stuff()

print t
==> 8 That can't be done very well with a decorator, but you could pass an
explicit named callable suite, e.g.,

thunk(): stuff()
t = f(thunk)
Thunks blend into their environment ISTM this needs earlier emphasis ;-)

def f(thunk):
thunk(6,7)

a = 20
do x,y in f():
a = 54
print a,x,y

==> 54,6,7
IMO that's more readable as

def f(thunk):
thunk(6, 7)
@f
(x, y): # think def foo(x, y): with "def foo" missing to make it a thunk
a = 54
print a,x,y

IMO we need some real use cases, or we'll never be able to decide what's really useful.
Thunks can return values. Since using 'return' would leave it unclear
whether it is the thunk or the surrounding function that is returning, a
different keyword should be used. By analogy with 'for' and 'while' loops,
the 'continue' keyword is used for this purpose: Gak ;-/
def f(thunk):
before()
t = thunk()
# t == 11
after()

do f():
continue 11
I wouldn't think return would be a problem if the compiler generated a
RETURN_CS_VALUE instead of RETURN_VALUE when it saw the end of
the callable suite (hence _CS_) (or thunk ;-)
Then it's up to f what to do with the result. It might pass it to after() sometimes.

Exceptions raised in the thunk pass through the thunk's caller's frame
before returning to the frame in which the thunk is defined: But it should be possible to have try/excepts within the thunk, IWT?
def catch_everythin g(thunk):
try:
thunk()
except:
pass # SomeException gets caught here

try:
do catch_everythin g():
raise SomeException
except:
pass # SomeException doesn't get caught here because it was
already caught

Because thunks blend into their environment, a thunk cannot be used after
its surrounding 'do' statement has finished:

thunk_saver = None
def f(thunk):
global thunk_saver
thunk_saver = thunk

do f():
pass

thunk_saver( ) # exception, thunk has expired Why? IWT the above line would be equivalent to executing the suite (pass) in its place.
What happens if you defined

def f(thunk):
def inner(it):
it()
inner(thunk)

do f():
x = 123

Of course, I'd spell it
@f
(): x = 123

Is there a rule against that (passing thunk on to inner)?

'break' and 'return' should probably not be allowed in thunks. One could
use exceptions to simulate these, but it would be surprising to have
exceptions occur in what would otherwise be a non-exceptional situation.
One would have to use try/finally blocks in all code that calls thunks
just to deal with normal situations. For example, using code like

def f(thunk):
thunk()
prevent_core_me ltdown()

with code like

do f():
p = 1
return p

would have a different effect than using it with

do f():
return 1

This behavior is potentially a cause of bugs since these two examples
might seem identical at first glance. I think less so with decorator and anonymous callable suite format

@f
(): return 1 # as in def foo(): return 1 -- mnemonically removing "def foo"

The thunk evaluates in the same frame as the function in which it was
defined. This frame is accessible:

def f(thunk):
frame = thunk.tk_frame # no connection with tkinter, right? Maybe thunk._frame would also say be careful ;-)
assert sys._getframe(1 ) is frame # ?? when does that fail, if it can?
do f():
pass

Motivation
==========

Thunks can be used to solve most of the problems addressed by PEP 310 [2]
and PEP 288 [3].

PEP 310 deals with the abstraction of acquire/release code. Such code is
needed when one needs to acquire a resource before its use and release it
after. This often requires boilerplate, it is easy to get wrong, and
there is no visual indication that the before and after parts of the code
are related. Thunks solve these problems by allowing the acquire/release
code to be written in a single, re-usable function.

def acquire_release (thunk):
f = acquire()
try:
thunk(f)
finally:
f.release()

do t in acquire_release ():
print t
That could be done as a callable suite and decorator
@acquire_releas e
(t): print t # like def foo(t): print t except that it's a thunk (or anonymous callable suite ;-)

BTW, since this callable suite definition is not named, there is no name binding
involved, so @acquire_releas e as a decorator doesn't have to return anything.
More generally, thunks can be used whenever there is a repeated need for
the same code to appear before and after other code. For example,

do WaitCursor():
compute_for_a_l ong_time()

is more organized, easier to read and less bug-prone than the code

DoWaitCursor(1 )
compute_for_a_ long_time()
DoWaitCursor (-1)
That would reduce to

@WaitCursor
(): compute_for_a_l ong_time()
PEP 288 tries to overcome some of the limitations of generators. One
limitation is that a 'yield' is not allowed in the 'try' block of a
'try'/'finally' statement.

def get_items():
f = acquire()
try:
for i in f:
yield i # syntax error
finally:
f.release()

for i in get_items():
print i

This code is not allowed because execution might never return after the
'yield' statement and therefore there is no way to ensure that the
'finally' block is executed. A prohibition on such yields lessens the
suitability of generators as a way to produce items from a resource that
needs to be closed. Of course, the generator could be wrapped in a class
that closes the resource, but this is a complication one would like to
avoid, and does not ensure that the resource will be released in a timely
manner. Thunks do not have this limitation because the thunk-accepting
function is in control-- execution cannot break out of the 'do' statement
without first passing through the thunk-accepting function.

def get_items(thunk ): # <-- "thunk-accepting function"
f = acquire()
try:
for i in f:
thunk(i) # A-OK
finally:
f.release()

do i in get_items():
print i
@get_items
(i): print i

But no yields in the thunk either, that would presumably not be A-OK ;-)
Even though thunks can be used in some ways that generators cannot, they
are not nearly a replacement for generators. Importantly, one has no
analogue of the 'next' method of generators when using thunks:

def f():
yield 89
yield 91

g = f()
g.next() # == 89
g.next() # == 91

[1] see the "Extended Function syntax" thread,
http://mail.python.org/pipermail/pyt...2003-February/
[2] http://www.python.org/peps/pep-0310.html
[3] http://www.python.org/peps/pep-0288.html

It's interesting, but I think I'd like to explore the decorator/callable suite version in some real
use cases. IMO the easy analogy with def foo(args): ... for specifying the thunk call parameters,
and the already established decorator call mechanism make this attractive. Also, if you allow named
thunks (I don't quite understand why they should "expire" if they can be bound to something. The
"thunk-accepting function" does not appear to "know" that the thunk reference will turn out to
be a real thunk as opposed to any other callable, so at that point it's just a reference,
and should be passable anywhere -- except maybe out of its defining scope, which would necessitate
generating a peculiar closure.

For a named callable suite, the decorator would have to return the callable, to preserve the binding,
or change it to something else useful. But without names, I could envisage a stack of decorators like
(not really a stack, since they are independent, and the first just uses the thunk to store a switch
class instance and a bound method to accumulate the cases ;-)

@make_switch
(switch, case): pass
@case(1,2,3,5)
(v): print 'small prime: %s'% v
@case(*'abc')
(v): print 'early alpha: %r'% v
@case()
(v): print 'default case value: %r'%v

and then being able to call
switch('b')
and see early alpha: 'b'

Not that this example demonstrates the local rebinding capability of thunks, which was the whole
purpose of this kind of switch definition ;-/
Just substitute some interesting suites that bind something, in the place of the prints ;-)

Regards,
Bengt Richter

[Ron_Adam]

On Fri, 15 Apr 2005 16:44:58 -0700, Brian Sabbey
<sa****@u.washi ngton.edu> wrote:

Simple Thunks
-------------

Thunks are, as far as this PEP is concerned, anonymous functions that
blend into their environment. They can be used in ways similar to code
blocks in Ruby or Smalltalk. One specific use of thunks is as a way to
abstract acquire/release code. Another use is as a complement to
generators.
I'm not familiar with Ruby or Smalltalk. Could you explain this
without referring to them?

A Set of Examples
============== ===

Thunk statements contain a new keyword, 'do', as in the example below. The
body of the thunk is the suite in the 'do' statement; it gets passed to
the function appearing next to 'do'. The thunk gets inserted as the first
argument to the function, reminiscent of the way 'self' is inserted as the
first argument to methods.

def f(thunk):
before()
thunk()
after()

do f():
stuff()

The above code has the same effect as:

before()
stuff()
after()
You can already do this, this way.

def f(thunk): .... before()
.... thunk()
.... after()
.... def before(): .... print 'before'
.... def after(): .... print 'after'
.... def stuff(): .... print 'stuff'
.... def morestuff(): .... print 'morestuff'
.... f(stuff) before
stuff
after f(morestuff) before
morestuff
after

This works with arguments also.

Other arguments to 'f' get placed after the thunk:

def f(thunk, a, b):
# a == 27, b == 28
before()
thunk()
after()

do f(27, 28):
stuff()

Can you explain what 'do' does better?

Why is the 'do' form better than just the straight function call?

f(stuff, 27, 28)

The main difference I see is the call to stuff is implied in the
thunk, something I dislike in decorators. In decorators, it works
that way do to the way the functions get evaluated. Why is it needed
here?

When I see 'do', it reminds me of 'do loops'. That is 'Do' involves
some sort of flow control. I gather you mean it as do items in a
list, but with the capability to substitute the named function. Is
this correct?

Cheers,
Ron

[Brian Sabbey]

Leif K-Brooks wrote:

Brian Sabbey wrote:

Thunk statements contain a new keyword, 'do', as in the example below. The
body of the thunk is the suite in the 'do' statement; it gets passed to the
function appearing next to 'do'. The thunk gets inserted as the first
argument to the function, reminiscent of the way 'self' is inserted as the
first argument to methods.

It would probably make more sense to pass the thunk as the last argument, not
as the first. That would make it easier to create functions with optional
thunks, as in:

def print_nums(star t, end, thunk=None):
for num in xrange(start, end+1):
if thunk is not None:
num = thunk(num)
print num

print_nums(1, 3) # prints 1, 2, 3

do num print_nums(1, 3): # prints 2, 4, 6
continue num * 2

That seems like a good idea to me.

I suppose it also makes sense to have the thunk last because it appears
after all the other arguments in the function call.

Because thunks blend into their environment, a thunk cannot be used after
its surrounding 'do' statement has finished

Why? Ordinary functions don't have that restriction:

def foo(): ... x = 1
... def bar():
... return x
... return bar
... foo()()

1

Thunks, as I implemented them, don't create a closure. I believe that
creating a closure will require a performance penalty. Since one use of
thunks is in loops, it seems that their performance may often be
important.

I believe that saving the thunk and calling it a later time is a somewhat
hackish way to use thunks. Explicitly defining a function (perhaps with a
suite-based keyword :) ) seems to me to be a more readable way to go.

But, yes, it is an arbitrary restriction. If it turns out that
performance isn't really affected by creating a closure, or that
performance doesn't matter as much as I think it does, then this
restriction could be lifted.

-Brian

[Brian Sabbey]

On Sat, 16 Apr 2005, Ron_Adam wrote:

Thunks are, as far as this PEP is concerned, anonymous functions that
blend into their environment. They can be used in ways similar to code
blocks in Ruby or Smalltalk. One specific use of thunks is as a way to
abstract acquire/release code. Another use is as a complement to
generators.
I'm not familiar with Ruby or Smalltalk. Could you explain this
without referring to them?

Hopefully my example below is more understandable. I realize now that I
should have provided more motivational examples.

def f(thunk):
before()
thunk()
after()

do f():
stuff()

The above code has the same effect as:

before()
stuff()
after()

You can already do this, this way.

def f(thunk): ... before()
... thunk()
... after()
... def before(): ... print 'before'
... def after(): ... print 'after'
... def stuff(): ... print 'stuff'
... def morestuff(): ... print 'morestuff'
... f(stuff) before
stuff
after f(morestuff) before
morestuff
after
This works with arguments also.

Yes, much of what thunks do can also be done by passing a function
argument. But thunks are different because they share the surrounding
function's namespace (which inner functions do not), and because they can
be defined in a more readable way.
Other arguments to 'f' get placed after the thunk:

def f(thunk, a, b):
# a == 27, b == 28
before()
thunk()
after()

do f(27, 28):
stuff()

Can you explain what 'do' does better?

Why is the 'do' form better than just the straight function call?

f(stuff, 27, 28)

The main difference I see is the call to stuff is implied in the
thunk, something I dislike in decorators. In decorators, it works
that way do to the way the functions get evaluated. Why is it needed
here?

You're right that, in this case, it would be better to just write
"f(stuff, 27, 28)". That example was just an attempt at describing the
syntax and semantics rather than to provide any sort of motivation. If
the thunk contained anything more than a call to 'stuff', though, it would
not be as easy as passing 'stuff' to 'f'. For example,

do f(27, 28):
print stuff()

would require one to define and pass a callback function to 'f'. To me,
'do' should be used in any situation in which a callback *could* be used,
but rarely is because doing so would be awkward. Probably the simplest
real-world example is opening and closing a file. Rarely will you see
code like this:

def with_file(callb ack, filename):
f = open(filename)
callback(f)
f.close()

def print_file(file ):
print file.read()

with_file(print _file, 'file.txt')

For obvious reasons, it usually appears like this:

f = open('file.txt' )
print f.read()
f.close()

Normally, though, one wants to do a lot more than just print the file.
There may be many lines between 'open' and 'close'. In this case, it is
easy to introduce a bug, such as returning before calling 'close', or
re-binding 'f' to a different file (the former bug is avoidable by using
'try'/'finally', but the latter is not). It would be nice to be able to
avoid these types of bugs by abstracting open/close. Thunks allow you to
make this abstraction in a way that is more concise and more readable than
the callback example given above:

do f in with_file('file .txt'):
print f.read()

Thunks are also more useful than callbacks in many cases since they allow
variables to be rebound:

t = "no file read yet"
do f in with_file('file .txt'):
t = f.read()

Using a callback to do the above example is, in my opinion, more
difficult:

def with_file(callb ack, filename):
f = open(filename)
t = callback(f)
f.close()
return t

def my_read(f):
return f.read()

t = with_file(my_re ad, 'file.txt')

When I see 'do', it reminds me of 'do loops'. That is 'Do' involves
some sort of flow control. I gather you mean it as do items in a
list, but with the capability to substitute the named function. Is
this correct?

I used 'do' because that's what ruby uses for something similar. It can
be used in a flow control-like way, or as an item-in-a-list way. For
example, you could replace 'if' with your own version (not that you would
want to):

def my_if(thunk, val):
if val:
thunk()

do my_if(a): # same as "if a:"
assert a

(replacing "else" wouldn't be possible without allowing multiple thunks.)

Or you could create your own 'for' (again, NTYWWT):

def my_for(thunk, vals):
for i in vals:
thunk(i)

do i in my_for(range(10 )):
print i
-Brian

[Brian Sabbey]

pe****@free.fr wrote:

Keep in mind that most of the problems come from the "space is
significant" thing, which is IMHO a very good idea, but prevents us from
putting code in expressions, like :

func( a,b, def callback( x ):
print x
)

or does it ? maybe this syntax could be made to work ?
Hmm. I'd like to think that suite-based keywords do make this example
work. One just has to let go of the idea that all arguments to a function
appear inside parentheses.
*************** *************** **********
Comments on the thunks.

First of all I view code blocks as essential to a language. They are
very useful for a lot of common programming tasks (like defining callbacks in
an elegant way) :

button = create_button( "Save changes" ):
do
self.save()

However it seems your thunks can't take parameters, which to me is a
big drawback. In ruby a thunk can take parameters. Simple examples :

field = edit_field( "initial value", onsubmit={ |value| if
self.validate(v alue) then do something else alert( "the value is invalid" ) }
)
[1,3,4].each { |x| puts x }
Thunks can take parameters:

do value in field = edit_field("ini tial value"):
if self.validate(v alue):
something
else:
alert("the value is invalid")

But callbacks are better defined with suite-based keywords:

field = edit_field("ini tial value"):
def onsubmit(value) :
if self.validate(v alue):
something
else:
alert("the value is invalid")
This has the advantage that the interface to the thunk (ie. its
parameters) are right there before your eyes instead of being buried in the
thunk invocation code inside the edit_field.
In the cases in which one is defining a callback, I agree that it would be
preferable to have the thunk parameters not buried next to 'do'.
However, I do not consider thunks a replacement for callbacks. They are a
replacement for code in which callbacks could be used, but normally aren't
because using them would be awkward. Your 'withfile' example below is a
good one. Most people wouldn't bother defining a 'withfile' function,
they would just call 'open' and 'close' individually.
So I think it's essential that thunks take parameters and return a
value (to use them effectively as callbacks).
What shall distinguish them from a simple alteration to def(): which
returns the function as a value, and an optional name ? really I don't know,
but it could be the way they handle closures and share local variables with
the defining scope. Or it could be that there is no need for two kinds of
function/blocks and so we can reuse the keyword def() :
Right, defining a function with 'def' is different than defining a thunk
because thunks share the namespace of the surrounding function, functions
do not:

x = 1
def f():
x = 2 # <- creates a new x
g(f)
print x # ==> 1

do g():
x = 2
print x # ==> 2 ( assuming 'g' calls the thunk at least once)

If you wish to modify def(), you could do, without creating any
keyword :

# standard form
f = def func( params ):
code

# unnamed code block taking params
func = def (params):
code

Note that the two above are equivalent with regard to the variable
"func", ie. func contains the defined function. Actually I find def
funcname() to be bloat, as funcname = def() has the same functionality, but
is a lot more universal.

# unnamed block taking no params
f = def:
code
I'm confused. These examples seem to do something different than what a
'do' statement would do. They create a function with a new namespace and
they do not call any "thunk-accepting" function.
*************** *************** *************** ******
Comments on the suite-based keywords.

Did you notice that this was basically a generalized HEREDOC syntax ?

I'd say that explicit is better than implicit, hence...

Your syntax is :

do f(a,b):
a block

passes block as the last parameter of f.
I don't like it because it hides stuff.
Yes, it hides stuff. It doesn't seem to me any worse than what is done
with 'self' when calling a method though. Once I got used to 'self'
appearing automatically as the first parameter to a method, it wasn't a
big deal for me.
I'd write :

f(a,b,@>,@>):
"""a very
large multi-line
string"""
def (x):
print x

Here the @> is a symbol (use whatever you like) to indicate
"placeholde r for something which is on the next line".
Indentation indicates that the following lines are indeed argument
for the function. The : at the end of the function call is there for
coherence with the rest of the syntax.
Notice that, this way, there is no need for a "do" keyword, as the
code block created by an anonymous def() is no different that the other
parameter, in this case a multiline string.

This has many advantages.

It will make big statements more readable :

instead of :
f( a,b, [some very big expression made up of nested class
constructors like a form defintion ], c, d )

write :
f( a, b, @>, c, d ):
[the very big expression goes here]

So, independently of code blocks, this already improves the
readability of big statements.
It might be useful in some situations to be able to define suite-based
arguments by their order rather than their keyword. But, to me, one of
Python's strengths is that it avoids special syntactic characters. So I
can't say I like "@>" very much. Perhaps there is another way of allowing
this.
You could also use named parameters (various proposals):

f( a,b, c=@>, d=@> ):
value of c
value of d

or :

f( a,b, @*> ):
value of c
value of d

f( a,b, @**> ):
c: value of c
d: value of d

Notice how this mimics f( a,b, * ) and f(a,b, ** ) for multiple
arguments, and multiple named arguments. Do you like it ? I do. Especially
the named version where you cant' get lost in the param block because you see
their names !
I like those ideas, but maybe just leave off the "@>".

f(a,b,c=,d=):
c = 1
d = 2

f(a,b,*)
1
2

f(a,b,**)
c = 1
d = 2

Another problem is that one may create bindings in the suite that should
not be keywords. Explicitly defining the keywords would be useful in this
case too. For example,

f(a,b,c=,d=):
c = [i**2 for i in [1,2]] # 'i' is temporary
d = 2

Here 'i' would not be passed as a keyword argument, because it 'c' and 'd'
are explicitly defined as the only keyword arguments.

Now if you say that def returns the defined function as a value, you
don't need a do keyword.

So, for instance :

def withfile( fname, thunk, mode = "r" ):
f = open( fname, mode )
thunk(f)
f.close()

then :

withfile( "afile.txt" , @>, "w" ):
def (f):
f.write( something )

Now, you may say that the def on an extra line is ugly, then just write :

withfile( "afile.txt" , @>, "w" ): def (f):
f.write( something )

If you really like do you can make it a synonym for def and then :

withfile( "afile.txt" , @>, "w" ): do (f):
f.write( something )

The two ":" seem a bit weird but I think they're OK.

Again, there is the problem of a new namespace being created when using
'def'. Also, it's annoying to have to use 'def' (even though one could
just put it on the same line).

'break' and 'return' should probably not be allowed in thunks. One

I do think return should be allowed in a thunk. After all it's a
function block, so why ditch useful functionality ?
yield should also be allowed, after all why can a thunk not be a
generator ? This would be powerful.

def withfile( fname, thunk, mode = "r" ):
f = open( fname, mode )
r = thunk(f)
f.close()
return r

val = withfile( "afile.txt" , @>, "w" ):
def (f):
f.write( something )
return something

Well, it seems I have no more ideas for now.
What do you think about all this ?

Well here you're using a type of suite-based keywords, so 'return' is ok.
Inside thunks I still think 'return' would be confusing. I don't think
one can replace thunks with suite-based keywords.

The thunk evaluates in the same frame as the function in which it was
defined. This frame is accessible:

Hm ?
You mean like a function closure, or that local variables defined
inside the thunk will be visible to the caller ?
This might change a lot of things and it becomes like a continuation,
are you going to recode stackless ?

I meant that a thunk is not like a function closure. Variables defined in
the thunk won't be visible to the caller of the thunk (except through
tk_frame), but they will be visible to the function surrounding the thunk.
I made an initial implementation, and it didn't require anything like
stackless.

-Brian

[Brian Sabbey]

Bengt Richter wrote:

Good background on thunks can be found in ref. [1].
UIAM most of that pre-dates decorators. What is the relation of thunks
to decorators and/or how might they interact?

Hmm, I think you answered this below better than I could ;).

def f(thunk):
before()
thunk()
after()

do f():
stuff()

The above code has the same effect as:

before()
stuff()
after()

Meaning "do" forces the body of f to be exec'd in do's local space? What if there
are assignments in f? I don't think you mean that would get executed in do's local space,
that's what the thunk call is presumably supposed to do...

Yes, I see now that there is an ambiguity in this example that I did not
resolve. I meant that the suite of the 'do' statement gets wrapped up as
an anonymous function. This function gets passed to 'f' and can be used
by 'f' in the same way as any other function. Bindings created in 'f' are
not visible from the 'do' statement's suite, and vice-versa. That would
be quite trickier than I intended.

Other arguments to 'f' get placed after the thunk:

def f(thunk, a, b):
# a == 27, b == 28
before()
thunk()
after()

do f(27, 28):
stuff()

I'm not sure how you intend this to work. Above you implied (ISTM ;-)
that the entire body of f would effectively be executed locally. But is that
true? What if after after() in f, there were a last statment hi='from last statement of f'
Would hi be bound at this point in the flow (i.e., after d f(27, 28): stuff() )?

I didn't mean to imply that. The body of 'f' isn't executed any
differently if it were called as one normally calls a function. All of
its bindings are separate from the bindings in the 'do' statement's scope.
I'm thinking you didn't really mean that. IOW, by magic at the time of calling thunk from the
ordinary function f, thunk would be discovered to be what I call an executable suite, whose
body is the suite of your do statement.
yes
In that case, f iself should not be a callable suite, since its body is _not_ supposed to be called locally,
and other than the fact that before and after got called, it was not quite exact to say it was _equivalent_ to

before()
stuff() # the do suite
after()
yes, I said "same effect as" instead of "equivalent " so that too much
wouldn't be read from that example. I see now that I should have been
more clear.
In that case, my version would just not have a do, instead defining the do suite
as a temp executable suite, e.g., if instead
we make an asignment in the suite, to make it clear it's not just a calling thing, e.g.,

do f(27, 28):
x = stuff()

then my version with explict name callable suite would be

def f(thunk, a, b):
# a == 27, b == 28
before()
thunk()
after()

set_x():
x = stuff() # to make it plain it's not just a calling thing

f(set_x, 27, 28)
# x is now visible here as local binding

but a suitable decorator and an anonymous callable suite (thunk defined my way ;-) would make this

@f(27, 28)
(): x = stuff()

Hmm, but this would require decorators to behave differently than they do
now. Currently, decorators do not automatically insert the decorated
function into the argument list. They require you to define 'f' as:

def f(a, b):
def inner(thunk):
before()
thunk()
after()
return inner

Having to write this type of code every time I want an thunk-accepting
function that takes other arguments would be pretty annoying to me.

Thunks can also accept arguments:

def f(thunk):
thunk(6,7)

do x,y in f():
# x==6, y==7
stuff(x,y)

IMO
@f
(x, y): stuff(x, y) # like def foo(x, y): stuff(x, y)

is clearer, once you get used to the missing def foo format

OK. I prefer a new keyword because it seems confusing to me to re-use
decorators for this purpose. But I see your point that decorators can be
used this way if one allows anonymous functions as you describe, and if
one allows decorators to handle the case in which the function being
decorated is anonymous.

The return value can be captured

This is just a fallout of f's being an ordinary function right?
IOW, f doesn't really know thunk is not an ordinary callable too?
I imagine that is for the CALL_FUNCTION byte code implementation to discover?

yes

def f(thunk):
thunk()
return 8

do t=f():
# t not bound yet
stuff()

print t
==> 8 That can't be done very well with a decorator, but you could pass an
explicit named callable suite, e.g.,

thunk(): stuff()
t = f(thunk)

But not having to do it that way was most of the purpose of thunks.

Thunks blend into their environment

ISTM this needs earlier emphasis ;-)

def f(thunk):
thunk(6,7)

a = 20
do x,y in f():
a = 54
print a,x,y

==> 54,6,7

IMO that's more readable as

def f(thunk):
thunk(6, 7)
@f
(x, y): # think def foo(x, y): with "def foo" missing to make it a thunk
a = 54
print a,x,y

IMO we need some real use cases, or we'll never be able to decide what's really useful.

Thunks can return values. Since using 'return' would leave it unclear
whether it is the thunk or the surrounding function that is returning, a
different keyword should be used. By analogy with 'for' and 'while' loops,
the 'continue' keyword is used for this purpose:

Gak ;-/

def f(thunk):
before()
t = thunk()
# t == 11
after()

do f():
continue 11

I wouldn't think return would be a problem if the compiler generated a
RETURN_CS_VALUE instead of RETURN_VALUE when it saw the end of
the callable suite (hence _CS_) (or thunk ;-)
Then it's up to f what to do with the result. It might pass it to after() sometimes.

It wouldn't be a problem to use 'return' instead of 'continue' if people
so desired, but I find 'return' more confusing because a 'return' in 'for'
suites returns from the function, not from the suite. That is, having
'return' mean different things in these two pieces of code would be
confusing:

for i in [1,2]:
return 10

do i in each([1,2]):
return 10

Exceptions raised in the thunk pass through the thunk's caller's frame
before returning to the frame in which the thunk is defined: But it should be possible to have try/excepts within the thunk, IWT?

yes, it is possible and they are allowed in the example implementation.

def catch_everythin g(thunk):
try:
thunk()
except:
pass # SomeException gets caught here

try:
do catch_everythin g():
raise SomeException
except:
pass # SomeException doesn't get caught here because it was
already caught

Because thunks blend into their environment, a thunk cannot be used after
its surrounding 'do' statement has finished:

thunk_saver = None
def f(thunk):
global thunk_saver
thunk_saver = thunk

do f():
pass

thunk_saver() # exception, thunk has expired

Why? IWT the above line would be equivalent to executing the suite (pass) in its place.

The restriction on saving thunks for later is for performance reasons and
because I believe it is hacky to use thunks in that way.
What happens if you defined

def f(thunk):
def inner(it):
it()
inner(thunk)

do f():
x = 123

Of course, I'd spell it
@f
(): x = 123

Is there a rule against that (passing thunk on to inner)?

No, that is fine, as long as execution has not yet left the 'do'
statement.

'break' and 'return' should probably not be allowed in thunks. One could
use exceptions to simulate these, but it would be surprising to have
exceptions occur in what would otherwise be a non-exceptional situation.
One would have to use try/finally blocks in all code that calls thunks
just to deal with normal situations. For example, using code like

def f(thunk):
thunk()
prevent_core_me ltdown()

with code like

do f():
p = 1
return p

would have a different effect than using it with

do f():
return 1

This behavior is potentially a cause of bugs since these two examples
might seem identical at first glance.

I think less so with decorator and anonymous callable suite format

@f
(): return 1 # as in def foo(): return 1 -- mnemonically removing "def foo"

In your syntax, 'return' would return from the thunk. With the 'do'
syntax, 'return' would return from the surrounding function. So the issue
does not arise with your syntax.

The thunk evaluates in the same frame as the function in which it was
defined. This frame is accessible:

def f(thunk):
frame = thunk.tk_frame

# no connection with tkinter, right? Maybe thunk._frame would also say be careful ;-)
assert sys._getframe(1 ) is frame # ?? when does that fail, if it can?

'tk' is supposed to remind one of 'thunk'. I believe that follows a
standard naming convention in python (e.g. see generators).

I don't see how that assert can fail.

I see what you're getting at with decorators and anonymous functions, but
there are a couple of things that, to me, make it worth coming up with a
whole new syntax. First, I don't like how one does not get the thunk
inserted automatically into the argument list of the decorator, as I
described above. Also, I don't like how the return value of the decorator
cannot be captured (and is already used for another purpose). The fact
that decorators require one more line of code is also a little bothersome.
Decorators weren't intended to be used this way, so it seems somewhat
hacky to do so. Why not a new syntax?

-Brian