A friendlier, sugarier lambda -- a proposal for Ruby-like blocks in python

[brenocon]

Hi all --

Compared to the Python I know and love, Ruby isn't quite the same.
However, it has at least one terrific feature: "blocks". Whereas in
Python a
"block" is just several lines of locally-scoped-together code, in Ruby
a
"block" defines a closure (anonymous function). To avoid confusion
let's call
them Ruby block-closures. I see them as just a syntax for defining
closures
and passing them into method calls. I think something analogous could
be added
to Python in a very simple manner that would make closures much more
readable
and usable, and nail some use cases nicely.

To define a new closure and pass it into a function call, there are two
current
methods: inline 'def' and 'lambda'. Consider the following Twisted-ish
code:

deferred = fetchPage('http ://python.org')
def _showResponse(r esponse)
print "fancy formatting: %s" % response.text
deferred.addCal lback(_showResp onse)

Lots of Twisted code has to be written backwards like this.
Theoretically, it
might be nice to use lambda right in the addCallback() call, like:

deferred.addCal lback(lambda r: print("fancy formatting %s"
%r.text) )

But this is awkward since the lambda is constrained to be one line; you
can't
come back later and add much to the callback's code. Furthermore, this
example
isn't even legal, because 'print' isn't a function, but a statement --
lambda
is further constrained to only contain an expression.

Many have complained about this crippled-ness of lambda, but it
actually makes
some sense. Since Python uses colons and indentation to define blocks
of code,
it would be awkward to close a multiline lambda. The best I could
think of
would look like

deferred.addCal lback(lambda r:
print("fancy formatting %s" % r.text)
)

^
|

That trailing paranthesis is WAY un-Pythonic. We don't close code
blocks like
that! And in general, declaring big multiline anonymous functions in
the
middle of a list of normal variable arguments is weird -- it just
doesn't fit.
It's perfectly legal to pass in 4 closures, interspersed with number
and string
arguments. Imagine defining all of those inline with 'lambda'
expressions!
And what about nesting? And then there's the term "lambda", while a
great
homage to Lisp and computation theory, just isn't the friendliest
programming
vocab term.

(from my limited understanding,) Ruby block-closures assume a specific
use
case: You want to pass exactly one multiline, defined-right-there
closure to a
method when calling it. Therefore, you should get to define the
closure
*immediately following* the method call. I suggest a Python version
with a
keyword 'using' (or 'with'?) that takes the block of code as a closure,
and
passes it to the method call as the *last argument*. The above example
becomes:

deferred.addCal lback() using response:
print "fancy formatting %s" % response.text

and in general, the following two code snippets are equivalent:
def _f(x,y):
[do stuff with x and y]
function_with_c allback(a,b,c, _f)

function_with_c allback(a,b,c) using x,y:
[do stuff with x and y]
next_statement( )

.... where function_with_c allback wants a 2-arg function as its last
argument.
It gets to call _f, or equivalently the defined-right-there
closure/ruby-block,
however it wants to -- wait for an I/O operation to finish, whatever.
I'm not
so hot about the fact that it looks like addCallback() should be
completed
before the 'using' keyword kicks in, but this is the simplest I could
think of.

This syntax does not let you define a new function and store it as a
local
variable. Python already has inline 'def' for that (that is, you can
do a
'def' in any block of code you want, and it stays local to that scope.)
It
does not, strictly speaking, let you do anything new -- as Guido has
stated,
you could ditch lambda and achieve the equivalent by declaring the
little
callback function as an inline 'def', like in the first deferred
example here.

This only optimizes for the case of defining a closure only for the
purpose of
passing it in as an argument to a method. However, this must be the
only use
for anonymous functions in Python, since we already have inline 'def'.
I've
always felt that passing in a lambda in lisp and python, or the
equivalent
anonymous function(x,y) {} in javascript, or anonymous classes in java,
was
always awkward. You have to think about defining a new anonymous
function
*within* your method call and such, and then you have to close
parantheses
after it -- it's just weird, I dunno.

This proposal could also handle some of the use cases mentioned in PEP
310 and
340. If you design your callback-taking functions to have only one
callback
and have it as the last argument, you can trivially write lock
acquition (omit
'using' for a no-arg block-closure):

def protect(lock, f):
lock.acquire()
f()
lock.release()

protect(myLock) :
[do stuff that needs myLock to be acquired]

Of course, the definition of protect() might have try/finally wrapped
around
the f() call. (Interestingly, this starts looking like a way to define
new
control-like structures. I haven't thought through the implications.)

ActiveRecord, Rails' object-relational mapper, does almost exactly this
for
database transactions, and I have found it quite nice:

# User is a sqlobject/sqlalchemy/django/whatever ORM class;
# User.transactio n is a class method executing its passed-in
closure within
# the user table's START TRANSACTION and STOP TRANSACTION.

user1, user2 = getTwoUsers()
User.transactio n() using user1, user2:
someRaceConditi onProneStuff(us er1, user2)
moreRaceConditi onProneStuff(us er1, user2)

There might be some sort of overlap with PEP 343 and the 'with'
statement, but
I'm not sure exactly. Sorry I'm late to the game and commenting on
last year's
PEP's, but I've only started reading them. Note that PEP's 343 and 340
are
very focused on resource management -- but I think that letting one
define code
blocks as closures could make resource handling routines be easily
written in
Python. Furthermore, tons more stuff -- like Deferreds and such --
could be
added. (Ruby uses block-closures to do really basic constructs such as
foreach
iteration. Python does a fine job with "for x in L" and list and
generator
comprehensions. .. enough so that map/lambda is obsolete! I'm just
trying to
see if there are use cases in Python for block-closures.)

I've been trying to search for similar proposals but have come up dry.
Anything like this out there? I hear Ruby blocks are heavily inspired
by
Smalltalk; anyone know more?

Is it feasible to assume the primary use of closures is as part of an
argument
list, and such argument lists will want only one argument that is a
closure?
Does doing so avoid any big annoyances of functional programming?

Is this completely, totally incompatible with the current state of
Python and
previous deliberations :) ? e.g. I haven't thought much about how this
would
interact with yield and generators.

But really, I'm just idly curious -- does anyone think this might be
useful?
Take care,
Brendan

[brenocon]

[repost -- fixed formatting]
Hi all --

Compared to the Python I know and love, Ruby isn't quite the same.
However, it has at least one terrific feature: "blocks". Whereas in
Python a "block" is just several lines of locally-scoped-together code,
in Ruby a "block" defines a closure (anonymous function). To avoid
confusion let's call them Ruby block-closures. I see them as just a
syntax for defining closures and passing them into method calls. I
think something analogous could be added to Python in a very simple
manner that would make closures much more readable and usable, and nail
some use cases nicely.

To define a new closure and pass it into a function call, there are two
current methods: inline 'def' and 'lambda'. Consider the following
Twisted-ish code:

deferred = fetchPage('http ://python.org')
def _showResponse(r esponse)
print "fancy formatting: %s" % response.text
deferred.addCal lback(_showResp onse)

Lots of Twisted code has to be written backwards like this.
Theoretically, it might be nice to use lambda right in the
addCallback() call, like:

deferred.addCal lback(lambda r: print("fancy formatting %s"
%r.text) )

But this is awkward since the lambda is constrained to be one line; you
can't come back later and add much to the callback's code.
Furthermore, this example isn't even legal, because 'print' isn't a
function, but a statement -- lambda is further constrained to only
contain an expression.

Many have complained about this crippled-ness of lambda, but it
actually makes some sense. Since Python uses colons and indentation to
define blocks of code, it would be awkward to close a multiline lambda.
The best I could think of would look like

deferred.addCal lback(lambda r:
print("fancy formatting %s" % r.text)
)

^
|

That trailing paranthesis is WAY un-Pythonic. We don't close code
blocks like that! And in general, declaring big multiline anonymous
functions in the middle of a list of normal variable arguments is weird
-- it just doesn't fit. It's perfectly legal to pass in 4 closures,
interspersed with number and string arguments. Imagine defining all of
those inline with 'lambda' expressions! And what about nesting? And
then there's the term "lambda", while a great homage to Lisp and
computation theory, just isn't the friendliest programming
vocab term.

(from my limited understanding,) Ruby block-closures assume a specific
use case: You want to pass exactly one multiline, defined-right-there
closure to a method when calling it. Therefore, you should get to
define the closure *immediately following* the method call. I suggest
a Python version with a keyword 'using' (or 'with'?) that takes the
block of code as a closure, and passes it to the method call as the
*last argument*. The above example becomes:

deferred.addCal lback() using response:
print "fancy formatting %s" % response.text

and in general, the following two code snippets are equivalent:

def _f(x,y):
[do stuff with x and y]
function_with_c allback(a,b,c, _f)

function_with_c allback(a,b,c) using x,y:
[do stuff with x and y]
next_statement( )

.... where function_with_c allback wants a 2-arg function as its last
argument. It gets to call _f, or equivalently the defined-right-there
closure/ruby-block, however it wants to -- wait for an I/O operation to
finish, whatever. I'm not so hot about the fact that it looks like
addCallback() should be completed before the 'using' keyword kicks in,
but this is the simplest I could think of.

This syntax does not let you define a new function and store it as a
local variable. Python already has inline 'def' for that (that is, you
can do a 'def' in any block of code you want, and it stays local to
that scope.) It does not, strictly speaking, let you do anything new
-- as Guido has stated, you could ditch lambda and achieve the
equivalent by declaring the
little callback function as an inline 'def', like in the first deferred
example here.

This only optimizes for the case of defining a closure only for the
purpose of passing it in as an argument to a method. However, this
must be the only use for anonymous functions in Python, since we
already have inline 'def'. I've always felt that passing in a lambda in
lisp and python, or the equivalent anonymous function(x,y) {} in
javascript, or anonymous classes in java, was always awkward. You have
to think about defining a new anonymous
function *within* your method call and such, and then you have to close
parantheses after it -- it's just weird, I dunno.

This proposal could also handle some of the use cases mentioned in PEP
310 and 340. If you design your callback-taking functions to have only
one callback and have it as the last argument, you can trivially write
lock acquition (omit 'using' for a no-arg block-closure):

def protect(lock, f):
lock.acquire()
f()
lock.release()

protect(myLock) :
[do stuff that needs myLock to be acquired]

Of course, the definition of protect() might have try/finally wrapped
around the f() call. (Interestingly, this starts looking like a way to
define new control-like structures. I haven't thought through the
implications.)

ActiveRecord, Rails' object-relational mapper, does almost exactly this
for database transactions, and I have found it quite nice:

# User is a sqlobject/sqlalchemy/django/whatever ORM class;
# User.transactio n is a class method executing its passed-in
#closure within
# the user table's START TRANSACTION and STOP TRANSACTION.

user1, user2 = getTwoUsers()
User.transactio n() using user1, user2:
someRaceConditi onProneStuff(us er1, user2)
moreRaceConditi onProneStuff(us er1, user2)

There might be some sort of overlap with PEP 343 and the 'with'
statement, but I'm not sure exactly. Sorry I'm late to the game and
commenting on last year's PEP's, but I've only started reading them.
Note that PEP's 343 and 340 are very focused on resource management --
but I think that letting one define code blocks as closures could make
resource handling routines be easily written in Python. Furthermore,
tons more stuff -- like Deferreds and such -- could be added. (Ruby
uses block-closures to do really basic constructs such as foreach
iteration. Python does a fine job with "for x in L" and list and
generator comprehensions. .. enough so that map/lambda is obsolete! I'm
just trying to see if there are use cases in Python for
block-closures.)

I've been trying to search for similar proposals but have come up dry.
Anything like this out here? I hear Ruby blocks are heavily inspired
by Smalltalk; anyone know more?

Is it feasible to assume the primary use of closures is as part of an
argument list, and such argument lists will want only one argument that
is a closure? Does doing so avoid any big annoyances of functional
programming?

Is this completely, totally incompatible with the current state of
Python and previous deliberations :) ? e.g. I haven't thought much
about how this would interact with yield and generators.

But really, I'm just idly curious -- does anyone think this might be
useful?

Take care,
Brendan

[Kay Schluehr]

br******@gmail. com wrote:

But [embedding a definition, ks] is awkward since the lambda is constrained to be one
line; you
can't come back later and add much to the callback's code.
Furthermore, this example isn't even legal, because 'print' isn't a
function, but a statement -- lambda is further constrained to only
contain an expression.

Many have complained about this crippled-ness of lambda, but it
actually makes some sense. Since Python uses colons and indentation to
define blocks of code, it would be awkward to close a multiline lambda.
The best I could think of would look like

deferred.addCal lback(lambda r:
print("fancy formatting %s" % r.text)
)

^
|

That trailing paranthesis is WAY un-Pythonic.

Maybe it is not common place because some users are underinformed and
hypnotized by whitespace semantics but you can add a trailing backslash
to achieve line coninuation within a single expression:

>>deferred.addC allback(lambda r: puts("fancy formatting %s" \

.... ))

This works syntactically.

[...]

There might be some sort of overlap with PEP 343 and the 'with'
statement, but I'm not sure exactly. Sorry I'm late to the game and
commenting on last year's PEP's, but I've only started reading them.
Note that PEP's 343 and 340 are very focused on resource management --
but I think that letting one define code blocks as closures could make
resource handling routines be easily written in Python.

The with statement is already implemented in Python 2.5.

http://docs.python.org/whatsnew/pep-343.html

The main difference between the with statement and Ruby blocks is that
the with-statement does not support loops. Yielding a value of a
function decorated with a contextmanager and passing it to the BLOCK of
the with statement is essentially a one-shot. Therefore you can't use
the with statement to define iterators. It is not a lightweight visitor
pattern replacement as it is in Ruby. Hence the with- and the
for-statement are orthogonal to each other in Python.

[Paul Rubin]

br******@gmail. com writes:

deferred = fetchPage('http ://python.org')
def _showResponse(r esponse)
print "fancy formatting: %s" % response.text
deferred.addCal lback(_showResp onse)

Lots of Twisted code has to be written backwards like this.

But that's just ugly. The fetchPage function should take the callback
as an argument. In an asynchronous system it would even be buggy.
What happens if the page fetch completes before you add the callback?

[Kay Schluehr]

Paul Rubin wrote:

br******@gmail. com writes:

deferred = fetchPage('http ://python.org')
def _showResponse(r esponse)
print "fancy formatting: %s" % response.text
deferred.addCal lback(_showResp onse)

Lots of Twisted code has to be written backwards like this.

But that's just ugly. The fetchPage function should take the callback
as an argument. In an asynchronous system it would even be buggy.
What happens if the page fetch completes before you add the callback?

As long as the fetchPage object does not start fetching right after
being constructed but only after applying an additional activate()
method the code is not buggy. I'm not sure about this design either. Is
_showResponse really that characteristic for fetchPage that it can't be
an object method? Spreading tiny function definitions all over the code
may be finally not such a good idea compared with a few generic methods
that get just called? OO might run out of fashion these days but Python
is not Java and Pythons OO is pretty lightweight.

[Marc 'BlackJack' Rintsch]

In <11************ **********@m7g2 000cwm.googlegr oups.com>, Kay Schluehr
wrote:

br******@gmail. com wrote:

>Many have complained about this crippled-ness of lambda, but it
actually makes some sense. Since Python uses colons and indentation to
define blocks of code, it would be awkward to close a multiline lambda.
The best I could think of would look like

deferred.addCal lback(lambda r:
print("fancy formatting %s" % r.text)
)

^
|

That trailing paranthesis is WAY un-Pythonic.

Maybe it is not common place because some users are underinformed and
hypnotized by whitespace semantics but you can add a trailing backslash
to achieve line coninuation within a single expression:

>>>deferred.add Callback(lambda r: puts("fancy formatting %s" \

... ))

This works syntactically.

It works just fine without the trailing backslash. A "logical line" does
not end until all opened parenthesis and brackets are matched by their
closing counterpart.

Ciao,
Marc 'BlackJack' Rintsch

[Duncan Booth]

Paul Rubin <http://ph****@NOSPAM.i nvalidwrote:

br******@gmail. com writes:

> deferred = fetchPage('http ://python.org')
def _showResponse(r esponse)
print "fancy formatting: %s" % response.text
deferred.addCal lback(_showResp onse)

Lots of Twisted code has to be written backwards like this.

But that's just ugly. The fetchPage function should take the callback
as an argument. In an asynchronous system it would even be buggy.
What happens if the page fetch completes before you add the callback?

Or it should be trivial to give deferred a decorator method and write:

@deferred.callb ack
def _showResponse(r esponse):
...

[James Stroud]

br******@gmail. com wrote:

Hi all --

Compared to the Python I know and love, Ruby isn't quite the same.
However, it has at least one terrific feature: "blocks". Whereas in
Python a
"block" is just several lines of locally-scoped-together code, in Ruby
a
"block" defines a closure (anonymous function). To avoid confusion
let's call
them Ruby block-closures. I see them as just a syntax for defining
closures
and passing them into method calls. I think something analogous could
be added
to Python in a very simple manner that would make closures much more
readable
and usable, and nail some use cases nicely.

To define a new closure and pass it into a function call, there are two
current
methods: inline 'def' and 'lambda'. Consider the following Twisted-ish
code:

deferred = fetchPage('http ://python.org')
def _showResponse(r esponse)
print "fancy formatting: %s" % response.text
deferred.addCal lback(_showResp onse)

Lots of Twisted code has to be written backwards like this.
Theoretically, it
might be nice to use lambda right in the addCallback() call, like:

deferred.addCal lback(lambda r: print("fancy formatting %s"
%r.text) )

But this is awkward since the lambda is constrained to be one line; you
can't
come back later and add much to the callback's code. Furthermore, this
example
isn't even legal, because 'print' isn't a function, but a statement --
lambda
is further constrained to only contain an expression.

Many have complained about this crippled-ness of lambda, but it
actually makes
some sense. Since Python uses colons and indentation to define blocks
of code,
it would be awkward to close a multiline lambda. The best I could
think of
would look like

deferred.addCal lback(lambda r:
print("fancy formatting %s" % r.text)
)

^
|

That trailing paranthesis is WAY un-Pythonic. We don't close code
blocks like
that! And in general, declaring big multiline anonymous functions in
the
middle of a list of normal variable arguments is weird -- it just
doesn't fit.
It's perfectly legal to pass in 4 closures, interspersed with number
and string
arguments. Imagine defining all of those inline with 'lambda'
expressions!
And what about nesting? And then there's the term "lambda", while a
great
homage to Lisp and computation theory, just isn't the friendliest
programming
vocab term.

(from my limited understanding,) Ruby block-closures assume a specific
use
case: You want to pass exactly one multiline, defined-right-there
closure to a
method when calling it. Therefore, you should get to define the
closure
*immediately following* the method call. I suggest a Python version
with a
keyword 'using' (or 'with'?) that takes the block of code as a closure,
and
passes it to the method call as the *last argument*. The above example
becomes:

deferred.addCal lback() using response:
print "fancy formatting %s" % response.text

and in general, the following two code snippets are equivalent:
def _f(x,y):
[do stuff with x and y]
function_with_c allback(a,b,c, _f)

function_with_c allback(a,b,c) using x,y:
[do stuff with x and y]
next_statement( )

... where function_with_c allback wants a 2-arg function as its last
argument.
It gets to call _f, or equivalently the defined-right-there
closure/ruby-block,
however it wants to -- wait for an I/O operation to finish, whatever.
I'm not
so hot about the fact that it looks like addCallback() should be
completed
before the 'using' keyword kicks in, but this is the simplest I could
think of.

This syntax does not let you define a new function and store it as a
local
variable. Python already has inline 'def' for that (that is, you can
do a
'def' in any block of code you want, and it stays local to that scope.)
It
does not, strictly speaking, let you do anything new -- as Guido has
stated,
you could ditch lambda and achieve the equivalent by declaring the
little
callback function as an inline 'def', like in the first deferred
example here.

This only optimizes for the case of defining a closure only for the
purpose of
passing it in as an argument to a method. However, this must be the
only use
for anonymous functions in Python, since we already have inline 'def'.
I've
always felt that passing in a lambda in lisp and python, or the
equivalent
anonymous function(x,y) {} in javascript, or anonymous classes in java,
was
always awkward. You have to think about defining a new anonymous
function
*within* your method call and such, and then you have to close
parantheses
after it -- it's just weird, I dunno.

This proposal could also handle some of the use cases mentioned in PEP
310 and
340. If you design your callback-taking functions to have only one
callback
and have it as the last argument, you can trivially write lock
acquition (omit
'using' for a no-arg block-closure):

def protect(lock, f):
lock.acquire()
f()
lock.release()

protect(myLock) :
[do stuff that needs myLock to be acquired]

Of course, the definition of protect() might have try/finally wrapped
around
the f() call. (Interestingly, this starts looking like a way to define
new
control-like structures. I haven't thought through the implications.)

ActiveRecord, Rails' object-relational mapper, does almost exactly this
for
database transactions, and I have found it quite nice:

# User is a sqlobject/sqlalchemy/django/whatever ORM class;
# User.transactio n is a class method executing its passed-in
closure within
# the user table's START TRANSACTION and STOP TRANSACTION.

user1, user2 = getTwoUsers()
User.transactio n() using user1, user2:
someRaceConditi onProneStuff(us er1, user2)
moreRaceConditi onProneStuff(us er1, user2)

There might be some sort of overlap with PEP 343 and the 'with'
statement, but
I'm not sure exactly. Sorry I'm late to the game and commenting on
last year's
PEP's, but I've only started reading them. Note that PEP's 343 and 340
are
very focused on resource management -- but I think that letting one
define code
blocks as closures could make resource handling routines be easily
written in
Python. Furthermore, tons more stuff -- like Deferreds and such --
could be
added. (Ruby uses block-closures to do really basic constructs such as
foreach
iteration. Python does a fine job with "for x in L" and list and
generator
comprehensions. .. enough so that map/lambda is obsolete! I'm just
trying to
see if there are use cases in Python for block-closures.)

I've been trying to search for similar proposals but have come up dry.
Anything like this out there? I hear Ruby blocks are heavily inspired
by
Smalltalk; anyone know more?

Is it feasible to assume the primary use of closures is as part of an
argument
list, and such argument lists will want only one argument that is a
closure?
Does doing so avoid any big annoyances of functional programming?

Is this completely, totally incompatible with the current state of
Python and
previous deliberations :) ? e.g. I haven't thought much about how this
would
interact with yield and generators.

But really, I'm just idly curious -- does anyone think this might be
useful?
Take care,
Brendan

http://mail.python.org/pipermail/pyt...il/215805.html

[Bjoern Schliessmann]

Paul Rubin wrote:

But that's just ugly. The fetchPage function should take the
callback as an argument. In an asynchronous system it would even
be buggy.

It won't, in my understanding/experience, since the local context
must first terminate before the reactor takes control again and
handles other events. Twisted isn't concurrent like using threads
here, and adding callbacks immediately after getting a Deferred
instance returned is common practice.

What happens if the page fetch completes before you add the
callback?

If the Deferred is fired, all already added {call,err}backs will be
executed. If you add new ones at this point, they'll be executed
immediately. IIRC.

http://twistedmatrix.com/projects/co...wto/async.html
http://twistedmatrix.com/projects/co...wto/defer.html

Regards,
Björn

--
BOFH excuse #12:

dry joints on cable plug

[Paul Boddie]

Kay Schluehr wrote:

>
Spreading tiny function definitions all over the code
may be finally not such a good idea compared with a few generic methods
that get just called? OO might run out of fashion these days but Python
is not Java and Pythons OO is pretty lightweight.

I think you've successfully identified a recent trend in Python
development: the abandonment of fairly transparent object-oriented
techniques in favour of more opaque but supposedly more convenient
hybrid techniques. Unlike Java, Python's first class functions and
methods are already highly useful for callback-based systems - such
systems in Java suffer from interface proliferation and needless
one-off class definitions - but it seems to me that the subsequent
language evolution in favour of anonymous blocks brings fewer benefits
and can, as you say, diminish readability.

Of course, there are other similar areas in recent Python development.
Nested functions appear to eliminate the need for one-off classes:

def f(x): # class f:
def g(y): # def __init__(self, x):
return op(x, y) # self.x = x
return g # def __call__(self, y):
g = f(a) # return op(self.x, y)
# g = f(a)

Generators could be replaced by classes whose objects would act as
state machines. Here, the readability permitted by the enhancement is
significantly greater than that permitted by the language prior to the
introduction of the enhancement, although there may be cases where
state machines are more instructive representations in understanding a
system.

Generally, though, such enthusiasm for reducing the number of
keypresses involved in implementing something often seems to deny how
concise Python code already is in many situations.

Paul