A friendlier, sugarier lambda -- a proposal for Ruby-like blocks in python

[brenocon]

Hi all --

Compared to the Python I know and love, Ruby isn't quite the same.
However, it has at least one terrific feature: "blocks". Whereas in
Python a
"block" is just several lines of locally-scoped-together code, in Ruby
a
"block" defines a closure (anonymous function). To avoid confusion
let's call
them Ruby block-closures. I see them as just a syntax for defining
closures
and passing them into method calls. I think something analogous could
be added
to Python in a very simple manner that would make closures much more
readable
and usable, and nail some use cases nicely.

To define a new closure and pass it into a function call, there are two
current
methods: inline 'def' and 'lambda'. Consider the following Twisted-ish
code:

deferred = fetchPage('http ://python.org')
def _showResponse(r esponse)
print "fancy formatting: %s" % response.text
deferred.addCal lback(_showResp onse)

Lots of Twisted code has to be written backwards like this.
Theoretically, it
might be nice to use lambda right in the addCallback() call, like:

deferred.addCal lback(lambda r: print("fancy formatting %s"
%r.text) )

But this is awkward since the lambda is constrained to be one line; you
can't
come back later and add much to the callback's code. Furthermore, this
example
isn't even legal, because 'print' isn't a function, but a statement --
lambda
is further constrained to only contain an expression.

Many have complained about this crippled-ness of lambda, but it
actually makes
some sense. Since Python uses colons and indentation to define blocks
of code,
it would be awkward to close a multiline lambda. The best I could
think of
would look like

deferred.addCal lback(lambda r:
print("fancy formatting %s" % r.text)
)

^
|

That trailing paranthesis is WAY un-Pythonic. We don't close code
blocks like
that! And in general, declaring big multiline anonymous functions in
the
middle of a list of normal variable arguments is weird -- it just
doesn't fit.
It's perfectly legal to pass in 4 closures, interspersed with number
and string
arguments. Imagine defining all of those inline with 'lambda'
expressions!
And what about nesting? And then there's the term "lambda", while a
great
homage to Lisp and computation theory, just isn't the friendliest
programming
vocab term.

(from my limited understanding,) Ruby block-closures assume a specific
use
case: You want to pass exactly one multiline, defined-right-there
closure to a
method when calling it. Therefore, you should get to define the
closure
*immediately following* the method call. I suggest a Python version
with a
keyword 'using' (or 'with'?) that takes the block of code as a closure,
and
passes it to the method call as the *last argument*. The above example
becomes:

deferred.addCal lback() using response:
print "fancy formatting %s" % response.text

and in general, the following two code snippets are equivalent:
def _f(x,y):
[do stuff with x and y]
function_with_c allback(a,b,c, _f)

function_with_c allback(a,b,c) using x,y:
[do stuff with x and y]
next_statement( )

.... where function_with_c allback wants a 2-arg function as its last
argument.
It gets to call _f, or equivalently the defined-right-there
closure/ruby-block,
however it wants to -- wait for an I/O operation to finish, whatever.
I'm not
so hot about the fact that it looks like addCallback() should be
completed
before the 'using' keyword kicks in, but this is the simplest I could
think of.

This syntax does not let you define a new function and store it as a
local
variable. Python already has inline 'def' for that (that is, you can
do a
'def' in any block of code you want, and it stays local to that scope.)
It
does not, strictly speaking, let you do anything new -- as Guido has
stated,
you could ditch lambda and achieve the equivalent by declaring the
little
callback function as an inline 'def', like in the first deferred
example here.

This only optimizes for the case of defining a closure only for the
purpose of
passing it in as an argument to a method. However, this must be the
only use
for anonymous functions in Python, since we already have inline 'def'.
I've
always felt that passing in a lambda in lisp and python, or the
equivalent
anonymous function(x,y) {} in javascript, or anonymous classes in java,
was
always awkward. You have to think about defining a new anonymous
function
*within* your method call and such, and then you have to close
parantheses
after it -- it's just weird, I dunno.

This proposal could also handle some of the use cases mentioned in PEP
310 and
340. If you design your callback-taking functions to have only one
callback
and have it as the last argument, you can trivially write lock
acquition (omit
'using' for a no-arg block-closure):

def protect(lock, f):
lock.acquire()
f()
lock.release()

protect(myLock) :
[do stuff that needs myLock to be acquired]

Of course, the definition of protect() might have try/finally wrapped
around
the f() call. (Interestingly, this starts looking like a way to define
new
control-like structures. I haven't thought through the implications.)

ActiveRecord, Rails' object-relational mapper, does almost exactly this
for
database transactions, and I have found it quite nice:

# User is a sqlobject/sqlalchemy/django/whatever ORM class;
# User.transactio n is a class method executing its passed-in
closure within
# the user table's START TRANSACTION and STOP TRANSACTION.

user1, user2 = getTwoUsers()
User.transactio n() using user1, user2:
someRaceConditi onProneStuff(us er1, user2)
moreRaceConditi onProneStuff(us er1, user2)

There might be some sort of overlap with PEP 343 and the 'with'
statement, but
I'm not sure exactly. Sorry I'm late to the game and commenting on
last year's
PEP's, but I've only started reading them. Note that PEP's 343 and 340
are
very focused on resource management -- but I think that letting one
define code
blocks as closures could make resource handling routines be easily
written in
Python. Furthermore, tons more stuff -- like Deferreds and such --
could be
added. (Ruby uses block-closures to do really basic constructs such as
foreach
iteration. Python does a fine job with "for x in L" and list and
generator
comprehensions. .. enough so that map/lambda is obsolete! I'm just
trying to
see if there are use cases in Python for block-closures.)

I've been trying to search for similar proposals but have come up dry.
Anything like this out there? I hear Ruby blocks are heavily inspired
by
Smalltalk; anyone know more?

Is it feasible to assume the primary use of closures is as part of an
argument
list, and such argument lists will want only one argument that is a
closure?
Does doing so avoid any big annoyances of functional programming?

Is this completely, totally incompatible with the current state of
Python and
previous deliberations :) ? e.g. I haven't thought much about how this
would
interact with yield and generators.

But really, I'm just idly curious -- does anyone think this might be
useful?
Take care,
Brendan

[brenocon]

Kay Schluehr wrote:

The with statement is already implemented in Python 2.5.

http://docs.python.org/whatsnew/pep-343.html

The main difference between the with statement and Ruby blocks is that
the with-statement does not support loops. Yielding a value of a
function decorated with a contextmanager and passing it to the BLOCK of
the with statement is essentially a one-shot. Therefore you can't use
the with statement to define iterators. It is not a lightweight visitor
pattern replacement as it is in Ruby. Hence the with- and the
for-statement are orthogonal to each other in Python.

Thanks or the What's-New link, it clarified things for me. So there
are several ways to do things with code blocks now in python..
* for/while define loops around their blocks
* if defines contional control into its block
* with defines startup/cleanup context surrounding its block

Twisted addCallback() is a different pattern than either of these. The
code is deferred to execute at some later time. If there are many more
patterns of things you could want to do with a block, it might be nice
to have a blocks-are-closures mechanism.

[Alexey Borzenkov]

br******@gmail. com wrote:

Compared to the Python I know and love, Ruby isn't quite the same.
However, it has at least one terrific feature: "blocks".

Well, I particularly like how Boo (http://boo.codehaus.org) has done
it:

func(a, b, c) def(p1, p2, p3):
stmts

I was so attached to these "nameless" def-forms that I was even shocked
when I found that this doesn't work in python:

f = def(a, b):
return a*b

Another good feature of Boo, btw.

[bearophileHUGS]

Alexey Borzenkov:

I was so attached to these "nameless" def-forms that I was even shocked
when I found that this doesn't work in python:
f = def(a, b):
return a*b
Another good feature of Boo, btw.

I think Boo has some good things worth consideration (and maybe worth
to copy) and some bad things that I like less than Python ones.
Turning def and class into functions reduces the need of lambdas
(compared to a lambda the required return is the only thing in the way)
and I like it, but maybe it reduces code readabilty a bit for people
that have just started to program:

mul2 = def(a, b):
return a * b

Instead of:

def mul2(a, b):
return a * b

The lightweight Io language uses an even simpler approach, some
examples:
http://www.iolanguage.com/about/samplecode/
An example:

factorial := method(n, if(n == 1, 1, n * factorial(n - 1)))

The "method" function is used to define functions...

Bye,
bearophile

[Alexey Borzenkov]

be************@ lycos.com wrote:

but maybe it reduces code readabilty a bit for people
that have just started to program:

mul2 = def(a, b):
return a * b

Instead of:

def mul2(a, b):
return a * b

For such simple cases, yes. What about:

button.click += def(obj):
# do stuff

You obviously can't:

def button.click(ob j):
# do stuff

:-) And if you make intermediate function and then assign it somewhere,
it "pollutes namespace": it's still left there, unneeded.

[Kay Schluehr]

Paul Boddie wrote:

Kay Schluehr wrote:

Spreading tiny function definitions all over the code
may be finally not such a good idea compared with a few generic methods
that get just called? OO might run out of fashion these days but Python
is not Java and Pythons OO is pretty lightweight.

I think you've successfully identified a recent trend in Python
development: the abandonment of fairly transparent object-oriented
techniques in favour of more opaque but supposedly more convenient
hybrid techniques. Unlike Java, Python's first class functions and
methods are already highly useful for callback-based systems - such
systems in Java suffer from interface proliferation and needless
one-off class definitions - but it seems to me that the subsequent
language evolution in favour of anonymous blocks brings fewer benefits
and can, as you say, diminish readability.

I was actually more concerned about extensibility than readability.
Talking about classes being lightweight I mentioned classes to be
"first-class" objects and intended constructions like this:

class fetchPage_event (fetchPage('htt p://python.org')):
def _showResponse(s elf, response)
print "fancy formatting: %s" % response.text

where all the registration and framework magics happens when the
fetchPage_event class is created. Adding "callbacks" just happen when
you add a method to the class. Here you have your "block" with no
additional syntax. You can pass the fetchPage_event class around and
add and overwrite methods in subclasses in the same fashion. A subclass
can be created within any statement on the fly etc. Here you have an OO
solution that is both short and has a clean design.

[Fulvio]

*************** ********
Your mail has been scanned by InterScan MSS.
*************** ********
Hello,

Sorry, I very new in programming and the use of the classes still unknown.
I'm really interested to apply the right one as far as the results aren't as
expected.

Here's the example:

from poplib import POP3
ProtocolError = 'Error in Protocol' # handler for IMAP4 and POP3 errors

try:
_pop = POP3(args[0])
_pop.user(args[1])
except:
raise ProtocolError
a.append( 'STAT = POP Server %s reply\n %s\n' %(args[0], _pop.pass_(args[2])))
(numMsgs, totalSize) = _pop.stat()
for cnt in range(1, numMsgs +1):
#Need a cleanup for the envelope header 'Unix-From'

try:
msgs =_pop.top(cnt,0 )[1]
except:
raise ProtocolError

# Here I'd like to have some part removed and I known that some class in email
# will do the neat job without missing any of the needed detail

hdrs = [k for itms in ('from', 'to', 'cc',
'date', 'subject', 'reply-to', 'message-')
for k in msgs if k.startswith(it ms.capitalize() )]

# This isn't the right risult. Some time missing one of the wanted item.

_pop.quit()

I've put my opinions as comments, in order to copy and try. The args contains
(pop address, user, and password) respectively.
The list comprehension fails for some comparison, might be more accurate a
regex, I just fill too complicated for a simple scan and I know that there
are already good classes that will do the right way. I just miss the learning
how to use them.

This a part of a small program, which will attempt to organize all the emails,
locals and remote in order to do spam removal and/or backup the good ones.
I'd like somebody to join, not for a speedy result but for a joy of exchanging
technics and tricks.

F

[Kay Schluehr]

br******@gmail. com wrote:

Kay Schluehr wrote:

The with statement is already implemented in Python 2.5.

http://docs.python.org/whatsnew/pep-343.html

The main difference between the with statement and Ruby blocks is that
the with-statement does not support loops. Yielding a value of a
function decorated with a contextmanager and passing it to the BLOCK of
the with statement is essentially a one-shot. Therefore you can't use
the with statement to define iterators. It is not a lightweight visitor
pattern replacement as it is in Ruby. Hence the with- and the
for-statement are orthogonal to each other in Python.

Thanks or the What's-New link, it clarified things for me. So there
are several ways to do things with code blocks now in python..
* for/while define loops around their blocks
* if defines contional control into its block
* with defines startup/cleanup context surrounding its block

Twisted addCallback() is a different pattern than either of these. The
code is deferred to execute at some later time. If there are many more
patterns of things you could want to do with a block, it might be nice
to have a blocks-are-closures mechanism.

That's true. As I mentioned in my response to Paul Boddie I do think it
is a poor solution even in Ruby(!) but I agree that given the situation
we just have with poorly designed application frameworks ( no illusion:
they won't ever go away and I do not pretend to be smarter and doing it
better in any case ) blocks are a quite nice feature for providing ad
hoc solutions. I consider them as somewhat aligned with a
worse-is-better philosophy.

----

I recently created an extended lambda as an example for my EasyExtend
language extension framework for Python. These kind of showcases are,
at least for me, important for getting programming practice within the
system and explore and extend the framework. The two things I cared
about when altering the semantics of lambda were:
1) Enabling multiple expressions separated by ';'
2) Enabling so called "simple statements"

Python does not only distinguish between expressions and statements but
also between simple and compound statements within its grammar
description. A compound statement is typically multiline and contains a
block. A simple statement is something like print, exec, raise or
assert containing no block. So I actually extended lambda to the limit
of an anonymous closure containing no block.

One might take this into consideration and alter the premises. But this
would be up to you, breno. At the moment I do not recommend using EE
because it is under heavy reconstruction but for the not so distant
future ( end of november is my personal deadline for the next release )
I recommend taking a look on it by anyone who aims to walk the RoR path
of a customized domain specific language for Python. For those who are
dilligent there are will be quite a lot of examples. For some it might
be even fun.

[sjdevnull]

Alexey Borzenkov wrote:

be************@ lycos.com wrote:

but maybe it reduces code readabilty a bit for people
that have just started to program:

mul2 = def(a, b):
return a * b

Instead of:

def mul2(a, b):
return a * b

For such simple cases, yes. What about:

button.click += def(obj):
# do stuff

You obviously can't:

def button.click(ob j):
# do stuff

:-) And if you make intermediate function and then assign it somewhere,
it "pollutes namespace": it's still left there, unneeded.

If you're really uptight about it you can
def temp_function(. ..):
....
button.click = temp_function
del(temp_functi on)

But for something like the example I'd probably just use a local name
wherever the definition is needed; there's no namespace pollution in
any meaningful sense then.

[Steven D'Aprano]

On Sat, 14 Oct 2006 09:00:53 +0000, James Stroud wrote:

Compared to the Python I know and love, Ruby isn't quite the same.
However, it has at least one terrific feature: "blocks".

[snip 220-odd quoted lines]

http://mail.python.org/pipermail/pyt...il/215805.html

Hi James,

You seem to have mislaid your snipper. It is very useful for when you're
replying to a very large post, and all you want to do is add a single line
at the end. It reduces frustration in readers who find themselves
scrolling, and scrolling, and scrolling, and scrolling through quoted text
they've already read, wondering why you've quoted all this stuff if you
aren't actually commenting on it, and prevents readers from unfairly
dismissing you as just another "metoobie".

--
Steven.

[Bruno Desthuilliers]

Paul Boddie wrote:

Kay Schluehr wrote:

>Spreading tiny function definitions all over the code
may be finally not such a good idea compared with a few generic methods
that get just called? OO might run out of fashion these days but Python
is not Java and Pythons OO is pretty lightweight.

I think you've successfully identified a recent trend in Python
development: the abandonment of fairly transparent object-oriented
techniques in favour of more opaque but supposedly more convenient
hybrid techniques.

Just for the record : Ruby's code-blocks (closures, really) come from
Smalltalk, which is still the OneTrueObjectLa nguage(tm).
--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'o****@xiludom. gro'.split('@')])"