code blocks in Python

[Hung Jung Lu]

Hi,

I know people have talked about it before, but I am still really
confused from reading the old messages.

When I talk about code blocks, I am not talking about Lisp/Ruby/Perl,
so I am not making comparisons.

If you have a file, in Python you could do:

code = compile(code_st ring, 'FileName.py', 'exec')

and obtain a code object, which later on you can execute. That's the
kind of code block I have in mind. That is, no parameters, no
specification of scopes, no nothing else. Just a plain block of code
lines compiled into a code object.

It's not too much work to write a code string like:

my_codeblock = '''
x = x + 1
y = y + 2
'''

and then use compile() to turn it into a code object. But it seems to
me that Python could perfectly allow this syntax directly into the
language, something like:

codeblock my_codeblock:
x = x + 1
y = y + 2

so that users won't need to explicitly call the compile() function.
The advantage is that the codeblock is compiled at compile time, not
at run time, even if the codeblock is defined inside something else (a
class, a function, etc.)

Is there something obvious that I am missing here? Why isn't codeblock
part of the language? It seems to be an important building element for
meta-programming. Not only that, it seems to be much more fundamental
than functions. So, why isn't it part of Python? Any intrinsic
difficulties?

regards,

Hung Jung

[John Roth]

"Hung Jung Lu" <hu********@yah oo.com> wrote in message
news:8e******** *************** **@posting.goog le.com...

Hi,

I know people have talked about it before, but I am still really
confused from reading the old messages.

When I talk about code blocks, I am not talking about Lisp/Ruby/Perl,
so I am not making comparisons.

If you have a file, in Python you could do:

code = compile(code_st ring, 'FileName.py', 'exec')

and obtain a code object, which later on you can execute. That's the
kind of code block I have in mind. That is, no parameters, no
specification of scopes, no nothing else. Just a plain block of code
lines compiled into a code object.

It's not too much work to write a code string like:

my_codeblock = '''
x = x + 1
y = y + 2
'''

and then use compile() to turn it into a code object. But it seems to
me that Python could perfectly allow this syntax directly into the
language, something like:

codeblock my_codeblock:
x = x + 1
y = y + 2

so that users won't need to explicitly call the compile() function.
The advantage is that the codeblock is compiled at compile time, not
at run time, even if the codeblock is defined inside something else (a
class, a function, etc.)

Is there something obvious that I am missing here? Why isn't codeblock
part of the language? It seems to be an important building element for
meta-programming. Not only that, it seems to be much more fundamental
than functions. So, why isn't it part of Python? Any intrinsic
difficulties?
Just some random observations. First, I'm not at all clear on
what you want it to do. Some examples would be helpful. There
are entirely too many different ways I could interpret your reference
to metaprogramming for me to understand the usage you have in
mind.

If you want it as a replacement for the badly crippled lambda statement,
then I think there's quite a bit of interest. What's holding that up is
syntax, and an appropriate syntax isn't entirely obvious.

If you want it to do something else, like act like it was actually
compiled where it will be invoked, with identifier resolution
and all, then that's a completely different subject.

John Roth


regards,

Hung Jung

[Hung Jung Lu]

"John Roth" <ne********@jhr othjr.com> wrote in message news:<vr******* *****@news.supe rnews.com>...

Just some random observations. First, I'm not at all clear on
what you want it to do. Some examples would be helpful. There
are entirely too many different ways I could interpret your reference
to metaprogramming for me to understand the usage you have in
mind.
codeblock A:
x = 1
y = 2

codeblock B:
z = x + y
print z

exec A
exec B

This will be equivalent to writing the program:

x = 1
y = 2
z = x + y
print z
If you want it as a replacement for the badly crippled lambda statement,
then I think there's quite a bit of interest. What's holding that up is
syntax, and an appropriate syntax isn't entirely obvious.

Lambda is a totally different thing. Lambda is an anonymous
*function*. I emphasize *function* because functions take parameters
and return values. Code blocks are, well, blocks of code. No
parameters, no nothing else. Simple and plain blocks of code. Once you
have codeblocks, you wire them anyway you want. Think of them as
macros in C++, only that in Python you can wire them dynamically
during runtime. Python already does that. It's just that you need the
compile() function, your code string is not compiled until the
compile() function is executed, and the resulting object type is
"Code" (which is a fine name, even if the new keyword were
"codeblock" .) Have you tried the compile() function?

regards,

Hung Jung

[Bengt Richter]

On 22 Nov 2003 19:13:19 -0800, hu********@yaho o.com (Hung Jung Lu) wrote:

"John Roth" <ne********@jhr othjr.com> wrote in message news:<vr******* *****@news.supe rnews.com>...

Just some random observations. First, I'm not at all clear on
what you want it to do. Some examples would be helpful. There
are entirely too many different ways I could interpret your reference
to metaprogramming for me to understand the usage you have in
mind.

codeblock A:
x = 1
y = 2

codeblock B:
z = x + y
print z

exec A
exec B

This will be equivalent to writing the program:

x = 1
y = 2
z = x + y
print z

If you want it as a replacement for the badly crippled lambda statement,
then I think there's quite a bit of interest. What's holding that up is
syntax, and an appropriate syntax isn't entirely obvious.

Lambda is a totally different thing. Lambda is an anonymous
*function*. I emphasize *function* because functions take parameters
and return values. Code blocks are, well, blocks of code. No
parameters, no nothing else. Simple and plain blocks of code. Once you
have codeblocks, you wire them anyway you want. Think of them as
macros in C++, only that in Python you can wire them dynamically
during runtime. Python already does that. It's just that you need the
compile() function, your code string is not compiled until the
compile() function is executed, and the resulting object type is
"Code" (which is a fine name, even if the new keyword were
"codeblock". ) Have you tried the compile() function?

I note that:

compile('a=123' ,'','single') <code object ? at 0090D5A0, file "", line 1> import dis
dis.dis(compile ('a=123','','si ngle')) 1 0 LOAD_CONST 0 (123)
3 STORE_NAME 0 (a)
6 LOAD_CONST 1 (None)
9 RETURN_VALUE def foo(): a=123 ... dis.dis(foo)

1 0 LOAD_CONST 1 (123)
3 STORE_FAST 0 (a)
6 LOAD_CONST 0 (None)
9 RETURN_VALUE

So which code will you want when the compiler compiles an in-context block?
Will you want it to be a first-class object? Should you be able to
pass the code-block name to an arbitrary function or bind it globally, like you can
with compile's output? Are you willing to forego the optimized local access? Or want to
exec it later from anywhere? What should happen? You could call for magic creation of special
closures so that an exported code block would still refer back to the local name space
(which BTW could be interesting for generators to yield, to give external access to their
internal state). And what should assigning a function local codeblock to a global mean?
Maybe creating a one-shot generator where the codeblock could be used to preset state for
a single ordinary function call that re-uses previous state. Or should it just die like a
weak reference?

IOW, there are some things to think about, because the semantics could get interesting ;-)
It would be nice to have efficient local blocks, so you could write exec(locals()[switcharg])
or perhaps some more efficient builtin switch/case thing using them. The efficiency of
a large jump table (dict or tuple/list based) vs a chain of elifs is a good motivation for some uses,
but I wonder how common/important they are.

Regards,
Bengt Richter

[Hung Jung Lu]

bo**@oz.net (Bengt Richter) wrote in message news:<bp******* ***@216.39.172. 122>...

>>> compile('a=123' ,'','single') <code object ? at 0090D5A0, file "", line 1> >>> import dis
>>> dis.dis(compile ('a=123','','si ngle')) 1 0 LOAD_CONST 0 (123)
3 STORE_NAME 0 (a)
6 LOAD_CONST 1 (None)
9 RETURN_VALUE >>> def foo(): a=123 ... >>> dis.dis(foo)

1 0 LOAD_CONST 1 (123)
3 STORE_FAST 0 (a)
6 LOAD_CONST 0 (None)
9 RETURN_VALUE

So which code will you want when the compiler compiles an in-context block?

Will you want it to be a first-class object?
Yes, absolutely.
with compile's output? Are you willing to forego the optimized local access?
I know there is a performance issue, since in normal, local, non-named
codeblocks, local variables are retrieved by indices instead of names.
There is a question as how to hook up named codeblocks efficiently in
the scope of the non-named codeblocks. But one is gotta start
somewhere. Optimization comes later.
You could call for magic creation of special
closures so that an exported code block would still refer back to the local name space
(which BTW could be interesting for generators to yield, to give external access to their
internal state).
Again, something simple to start with. Closures comes later, if at all
necessary. I am not sure whether closure-like codeblocks are
fundamental enough, since they could be emulated by other means. E.g.

class generic: pass

codeblock A:
print c.x
c.x = 3

c1 = generic()
c2 = generic()
c1.x = 4
c1.code = A
c2.x = 5
c2.code = A

c = c1
exec c.code

c = c2
exec c.code
And what should assigning a function local codeblock to a global mean?
Maybe creating a one-shot generator where the codeblock could be used to preset state for
a single ordinary function call that re-uses previous state. Or should it just die like a
weak reference?
I don't know exactly what you mean by "function local codeblock". If
you refer to the function's code, currently accessible as f.func_code,
I've tried that, but it seems to bind things to the locals dictionary
of the function itself. Which is no good. E.g.:

def f():
x = 1

def g(x):
exec f.func_code
print x

g(2)
x = 3
exec f.func_code in globals()
print x

will print 2 and 3. I tried using exec...in... with various
namespaces, but the same thing. That is, the func_code remembers its
own local scope and refuses access/modify other local scopes. This is
very different from:

code = compile('x=1\n' , '<string>', 'exec')

def g(x):
exec code
print x

g(2)
x = 3
exec code
print x

which will print 1 and 1. That is, smart enough to take x from the
respective local and global dictionaries.

----------

If by "function local codeblock" you mean codeblocks define inside the
scope of a function, I really don't think they should receive any
special treatment for optimization, since a codeblock should be an
object that can be passed around, and should be independent of where
they are defined. It's like if you have the number 3, which is an
object, you can pass it around to anywhere, regardless where it was
first defined. No complications from nested-scopes, closures, etc.
Functions get complicated exactly because they carry all these extra
baggages.

----------

Well, all these are the kinds of things to think about. But named
codeblocks have well-defined meaning on their own right, independent
of implementation details. Theoretically, codeblocks don't know
whether the names inside them are local or global. It's a late-binding
object.
but I wonder how common/important they are.

I can't say in the case you have described. However, in general
codeblocks are extremely useful, right now they are not used as
widely, I think simply because the codestrings are not compiled and
checked at compile time: you need one extra step invoking the
compile() function. Named codeblocks will solve a barrage of nuisances
in today's programming world. They helps in (a) code re-use, (b) code
modularization. By (a) I mean that the same codeblock could be used in
a lot of places. By (b) I mean that you can isolate the effects of
codeblocks, and do code replacements a la AOP (Aspect-Oriented
Programming.)

It's just weird that Python has the compile() function, but has no
named codeblocks. I really can't understand why.

regards,

Hung Jung

[Hung Jung Lu]

hu********@yaho o.com (Hung Jung Lu) wrote in message >

codeblock A:
x = 1
y = 2

codeblock B:
z = x + y
print z

exec A
exec B

This will be equivalent to writing the program:

x = 1
y = 2
z = x + y
print z

It just occurred to me a better syntax:

def A:
x = 1
y = 2

def B:
z = x + y
print z

No new keyword needed, beside, it uses the same keyword as function
definition, so the purpose of the code block is clear: it's something
that you will invoke.

A proper name for this type of meta-programming is "Linear
Meta-programming". Just like in algebra you have linear algebra and
non-linear algebra, in meta-programming you have all sorts of
non-linear ways of taking code pieces and turn them into other code
pieces, these include e.g.: (a) compiling: turning a character string
into a binary code string, (b) encryption, (c) macro/token
substitution, (d) tweaking with the internals like meta-classes, etc.
Linear meta-programming is much more modest and well-organized. It's
like sequencing DNA bands. Each statement is like a DNA nucleotide,
each codeblock is like a DNA band. You just want to sequence the
codeblocks properly, and at times replace one block with some other
block, or insert additional blocks.

I still have yet to hear any difficulties/complaints.

I refuse to believe that Python developers have NEVER considered this
extension to the language. So, what's the problem?

Hung Jung

[Alexander Schmolck]

hu********@yaho o.com (Hung Jung Lu) writes:

I still have yet to hear any difficulties/complaints.

OK: 1. the semantics of your proposed extension remain somewhat nebulous
2. the extension appears pretty useless (AFAICT, given 1.)
3. you still haven't provided an example that would demonstrate otherwise
(i.e. something interesting that could be more easily written with
python + your extension than with python now).

'as

[Jeff Epler]

On Sun, Nov 23, 2003 at 04:34:39PM -0800, Hung Jung Lu wrote:

I still have yet to hear any difficulties/complaints.
Suppose I write this:

def C:
x = x + 1

def a():
x = 2
exec C
print x

def b():
exec C

how will the implementation of "code blocks" deal with the fact that in
a, x is a fastlocal with a particular index, but in b there's no local
x? (In cpython, "bare exec" has a significant negative impact of the
performance of a function, because all variable accesses happen though
LOAD_NAME instead of LOAD_GLOBAL / LOAD_FAST. Any new block-oriented
feature should be designed to avoid this problem)
I refuse to believe that Python developers have NEVER considered this
extension to the language. So, what's the problem?

Extensions along these lines have often been discussed. Consider reading
the archives. "Macros", "code blocks", "with blocks", and "closures" are
all terms you might consider using if you are searching for past messages.

Jeff

[Hung Jung Lu]

Jeff Epler <je****@unpytho nic.net> wrote in message news:<ma******* *************** *************** @python.org>...

def C:
x = x + 1

def a():
x = 2
exec C
print x

def b():
exec C

how will the implementation of "code blocks" deal with the fact that in
a, x is a fastlocal with a particular index, but in b there's no local
x? (In cpython, "bare exec" has a significant negative impact of the
performance of a function, because all variable accesses happen though
LOAD_NAME instead of LOAD_GLOBAL / LOAD_FAST. Any new block-oriented
feature should be designed to avoid this problem)

I agree that performance could be made better. But that's an issue for
later. If codeblock becomes a fundamental building block of a language
(say, Python 3.0), I am sure there are tricks beyond the current
global/local/builtin namespaces mechanisms
(LOAD_NAME/LOAD_FAST/LOAD_GLOBAL). But performance is an issue for
later. I would like to remind people that the compile() function
exists and works in Python. For whatever discussion on this subject,
the first thing to do is to replace the codeblock definition with the
compile function() and analyze from there on. In this case,
C=compile('x=x+ 1\n', '<string>', 'exec').

regards,

Hung Jung

[Skip Montanaro]

Jeff> how will the implementation of "code blocks" deal with the fact
Jeff> that in a, x is a fastlocal with a particular index, but in b
Jeff> there's no local x?

hjl> I agree that performance could be made better. But that's an issue
hjl> for later....

You ignored Jeff's primary questions. What are the semantics of a named
code block such as you propose? What if it's executed in a context which
doesn't contain the variables it uses? What expressive power does it give
to Python that Python doesn't currently have?

Skip