macro FAQ

[Andrew Dalke]

Here's a proposed Q&A for the FAQ based on a couple recent
threads. Appropriate comments appreciated

X.Y: Why doesn't Python have macros like in Lisp or Scheme?

Before answering that, a clarification on what 'macro' means.
A Lisp macro is a way of modifying code when that code is first
defined. It can rearrange the structure of the code, and add and
remove parts of it. Unlike C's #define macro language, Lisp
macros understand the structure of the expression and the
context (or "closure") in which it is found.

Here's a simple example of what a macro might do for Python.
A complaint about Python is that constructors for simple data
types require typing the parameter names three times, as in

class Country:
def __init__(self, name, capitol):
self.name = name
self.capitol = capitol

Suppose you could create a new way to create functions, which
looked at the argument list and automatically set the corresponding
member. This would let you replace the above with something
more like

class Country:
def __init__(self, name, capitol) [arg_assign]:
pass

where the 'arg_assign' would use the function definition
and code to make a new function used in place of the
existing one and with the 'self.name = name' and
'self.capitol = capitol' code automatically added to
the front of the code.

This is a very powerful technique and especially nice for
high-performance code where the macro can optimize
code before it is actually evaluated. Macros work well in
Lisp or Scheme, where the code is the data, all written
pretty much as a parse tree.

The first problem with implementing a macro system in Python
is coming up with a readable syntax; that is, how it's "spelled."
The example shown above, using the [] after the function
definition, was taken from a proposed syntax for adding
function modifiers like 'staticmethod' and 'classmethod' to
Python.

The second problem is that Python's code blocks do not
store the original parse tree, so there's nothing to manipulate.
If you use 'dir()' or the inspect module you'll see some fields
like the list of argument names and the default value to use
for the fields. Nowhere will you find the original parse tree.
(The C implementation of Python does give access to the
Python byte code, but this is not part of the language
specification and Java implementation doesn't support it.)

These problems can be resolved, and Dylan shows that
an infix-based/Algol-like language can support macros
similar to those used in Lisp.

The deeper question is, should Python include macros?

People with a Lisp background might have a problem
understanding this viewpoint. Macros add flexibility, and
more flexibilty lets programmers be more productive; or
so the argument goes.

Those against including macros in Python point out that
Python has a different focus. It has a broad target audience.
Beginners can pick it up pretty easily yet advanced
developers also enjoy working in the language. New features,
like new builtin functions, additional core types, expanding
the standard libraries, list comprensions, metaclasses and
deriving from object, and macros, all make it harder to
learn how to use Python well and shifts the balance more
towards advanced programmers.

More importantly, the detractors -- including those
with plenty of experience using macros in Lisp -- argue that
macros cause dialects to form. Macros can modify other code to
make it fit the problem better, while functions only use other
code but make no modifications. This makes them very
powerful but means that understanding a section of code
requires also knowing about any macros which might use the
code. In an extreme case which wouldn't be used in real
projects, every * could be replaced with a +.

(As an aside, some proponents argue that macros and
functions are essentially the same. Alex Martelli made
an interesting observation about one difference: it's often
worthwhile to turn a set of code into a function even if it
is only called from one place, because it helps simplify
the code into smaller chucks which are easier to understand.
Macros, on the other hand, should almost never be used
unless they are used many times.)

With only one or a small group of people working together
on a project there is little problem. Macros help in developing
idioms specific to the problem and group. When groups
share code, they also share idioms, and anyone who has had
to deal with UK vs. US English knows the effect idioms have
in mututal understanding.

Those against macros say their benefits do not outweigh
the likelihood that the Python community will be broken up
into distinct subsets, where people from one group find it
hard to use code from another.

The proponents for macros say this has not been a problem
in Lisp. Macros are misused, but so are functions and classes.
The same community feedback which encourages people to
use proper coding styles also encourages people to use macros
correctly. Those against reassert that not only can it be a
problem but it's happened to them personally.

At this point the discussion usually breaks down, with the
Lispers pulling out the old canard about Python (and every
other language) being a half-complete, poor reimplementation
of Lisp, while Pythonistas trot out the expression "Lisp is like
a ball of mud. You can add any amount of mud to it and it
still looks like a ball of mud."

Those interested in actual language improvements then ask for
concrete examples of how macros might be used for Python.
So far, the examples given are ones which could equally well
be done in Python using metaclasses, properties, and other
existing mechanisms. For example, the arg_assign example
given above could, I think, be done with a modified__new__.

The problem at this point seems to be that the two languages
have different enough object models and accepted best
practices that it's hard for an intuition based on Lisp to
know immediately how to apply that intuition to Python.

If you want to encourage Python to have macros, you must
therefore propose a syntax for it and give solid examples of
what the result would look like when applied to real Python
code. Simply stating that macros are needed is not good
enough nor are examples showing how macros are useful
for Lisp code. (Besides, believe it or not but some of us
don't know enough Lisp to follow your examples ;)

In the meanwhile, if you want to experiment with manipulating
code trees, you can use decompyle to turn byte code back
into Python source and the compiler module to turn source into
an AST and thence into byte code.

Andrew Dalke
da***@dalkescientific.com

[Michele Simionato]

"Andrew Dalke" <ad****@mindspring.com> wrote in message news:<Ds****************@newsread4.news.pas.earthl ink.net>...

Here's a proposed Q&A for the FAQ based on a couple recent
threads. Appropriate comments appreciated

X.Y: Why doesn't Python have macros like in Lisp or Scheme?

This is all very interesting and I do agree that there shouuld be a
FAQ about macros, nevertheless I don't see your message as suitable
for a FAQ. It is ways too verbose. I mean, if one asks for a macro,
s/he already knows what a macro is. The FAQ should be much
shorter. What you wrote is more useful and interesting for
Pythonistas than from Lispers.
I think your document should be made available on the Web somewhere,
but not in the FAQ. Of course the FAQ could and should give a pointer
to it. Maybe lispers would be helped from a page describing how
some of the things made in macros in Lisp can be made in Python.
This page should be referred in the FAQ too. It is clear, however,
that many things involving macros cannot just be done in Python
(I mean, unless implementing a custom macro interpreter in Python,
which actually I wrote few months ago as a proof of concept (google
on c.l.py for simionato macros, first hit) and which I DO NOT use ;).

Michele Simionato, Ph. D.
Mi**************@libero.it
http://www.phyast.pitt.edu/~micheles
--- Currently looking for a job ---

[Michele Simionato]

"Andrew Dalke" <ad****@mindspring.com> wrote in message news:<Ds****************@newsread4.news.pas.earthl ink.net>...

Here's a proposed Q&A for the FAQ based on a couple recent
threads. Appropriate comments appreciated

X.Y: Why doesn't Python have macros like in Lisp or Scheme?

This is all very interesting and I do agree that there shouuld be a
FAQ about macros, nevertheless I don't see your message as suitable
for a FAQ. It is ways too verbose. I mean, if one asks for a macro,
s/he already knows what a macro is. The FAQ should be much
shorter. What you wrote is more useful and interesting for
Pythonistas than from Lispers.
I think your document should be made available on the Web somewhere,
but not in the FAQ. Of course the FAQ could and should give a pointer
to it. Maybe lispers would be helped from a page describing how
some of the things made in macros in Lisp can be made in Python.
This page should be referred in the FAQ too. It is clear, however,
that many things involving macros cannot just be done in Python
(I mean, unless implementing a custom macro interpreter in Python,
which actually I wrote few months ago as a proof of concept (google
on c.l.py for simionato macros, first hit) and which I DO NOT use ;).

Michele Simionato, Ph. D.
Mi**************@libero.it
http://www.phyast.pitt.edu/~micheles
--- Currently looking for a job ---

[John J. Lee]

mi**@pitt.edu (Michele Simionato) writes:

"Andrew Dalke" <ad****@mindspring.com> wrote in message news:<Ds****************@newsread4.news.pas.earthl ink.net>...

Here's a proposed Q&A for the FAQ based on a couple recent
threads. Appropriate comments appreciated

X.Y: Why doesn't Python have macros like in Lisp or Scheme?

[...] I think your document should be made available on the Web somewhere,
but not in the FAQ. Of course the FAQ could and should give a pointer
to it.

[...]

+1, FWIW
John

[John J. Lee]

mi**@pitt.edu (Michele Simionato) writes:

"Andrew Dalke" <ad****@mindspring.com> wrote in message news:<Ds****************@newsread4.news.pas.earthl ink.net>...

Here's a proposed Q&A for the FAQ based on a couple recent
threads. Appropriate comments appreciated

X.Y: Why doesn't Python have macros like in Lisp or Scheme?

[...] I think your document should be made available on the Web somewhere,
but not in the FAQ. Of course the FAQ could and should give a pointer
to it.

[...]

+1, FWIW
John

[John J. Lee]

"Andrew Dalke" <ad****@mindspring.com> writes:

Here's a proposed Q&A for the FAQ based on a couple recent
threads. Appropriate comments appreciated [...] Those against including macros in Python point out that
Python has a different focus. It has a broad target audience.
Beginners can pick it up pretty easily yet advanced
developers also enjoy working in the language. New features,
like new builtin functions, additional core types, expanding
the standard libraries, list comprensions, metaclasses and
deriving from object, and macros, all make it harder to
learn how to use Python well and shifts the balance more
towards advanced programmers.

More importantly, the detractors -- including those
with plenty of experience using macros in Lisp -- argue that
macros cause dialects to form. Macros can modify other code to
make it fit the problem better, while functions only use other
code but make no modifications. This makes them very
powerful but means that understanding a section of code
requires also knowing about any macros which might use the
code. In an extreme case which wouldn't be used in real
projects, every * could be replaced with a +.

[...]

About the first paragraph here: I don't recall anyone arguing that,
but maybe I forget. Anyway, my problem with it is that it reinforces
the "Python is a scripting language for amateurs, not suitable for
large scale / systems programming" attitude. I think Python
demonstrates that there isn't much tension in reality between the
'advanced programmers' and the beginners -- both seem to want a simple
language. And as you say, the argument about dialects is more
important anyway.

Other than that, excellent stuff.
John

[John J. Lee]

"Andrew Dalke" <ad****@mindspring.com> writes:

Here's a proposed Q&A for the FAQ based on a couple recent
threads. Appropriate comments appreciated [...] Those against including macros in Python point out that
Python has a different focus. It has a broad target audience.
Beginners can pick it up pretty easily yet advanced
developers also enjoy working in the language. New features,
like new builtin functions, additional core types, expanding
the standard libraries, list comprensions, metaclasses and
deriving from object, and macros, all make it harder to
learn how to use Python well and shifts the balance more
towards advanced programmers.

More importantly, the detractors -- including those
with plenty of experience using macros in Lisp -- argue that
macros cause dialects to form. Macros can modify other code to
make it fit the problem better, while functions only use other
code but make no modifications. This makes them very
powerful but means that understanding a section of code
requires also knowing about any macros which might use the
code. In an extreme case which wouldn't be used in real
projects, every * could be replaced with a +.

[...]

About the first paragraph here: I don't recall anyone arguing that,
but maybe I forget. Anyway, my problem with it is that it reinforces
the "Python is a scripting language for amateurs, not suitable for
large scale / systems programming" attitude. I think Python
demonstrates that there isn't much tension in reality between the
'advanced programmers' and the beginners -- both seem to want a simple
language. And as you say, the argument about dialects is more
important anyway.

Other than that, excellent stuff.
John

[Doug Tolton]

Jacek Generowicz <ja**************@cern.ch> wrote in message news:<ty*************@lxplus014.cern.ch>...

Macros can modify other code to make it fit the problem better,

What do you mean by "other" code? Macros modify the code that is
passed to them as an argument, transforming it before the compiler
gets to see it. I get the impression that you believe that macros can
somehow modify code from other parts of the program. They can't.

Ultimately, macros just save you a lot of typing of source code. (And
thereby save you a lot of bugs.) If you can't type it as soure code,
then a macro can't do it.

while functions only use other code but make no modifications.

This only strengthens the above suspicion, but I'm not really sure
what you mean, here.

This makes them very powerful but means that understanding a section
of code requires also knowing about any macros which might use the
code.

What do you mean by "macros using code" ?

The macros are part of the code, just like functions are. To
understand the code, you must understand what the macros and functions
do.

In an extreme case which wouldn't be used in real projects, every *
could be replaced with a +.

This almost completely convinces me that you are very confused about
what macros can achieve.

Either, you are suggesting that one might write a macro which replaces
every * operator with a + operator, and then pass _the entire source
code_ of a project to it (possible, but patently absurd); or you are
suggesting that it is possible to write a macro "over here" which,
somehow, surreptitiously modifies existing source code "over there".

a) Macros cannot do the latter.

b) You can achieve something very similar in Python, by re-binding
attributes of __builtins__.

Think of the point of macros in another way. The point is _not_ to
take existing source code, and to change it. The point is to take
something that is not (necessarily) valid source code, and turn it
into valid source code, before the compiler gets to see it. Why?
Because this allows you to create a shorthand notation, and use macros
to expand it to real code. Think in terms of encoding design patterns.

I think Jacek has made an extremely good point here, and I want to
follow up on it to determine if everyone has a clear understanding of
what macros are and are not. Following I have outlined a series of
questions an answers to attempt to illuminate the issue. Please
suggest any clarifications or distinctions that need to be made. Also
note that the examples presented are purposefully trivial and could in
many instances be accomplished other ways, however I am presenting
these examples in order to provide a meaningful context to proceed
from.

Question: What are Macros?

Answer: Macros are an advanced templating system that allow dynamic
evaluation of code during run time. Essentially Macros allow you to
pass in some variables that are then expanded into valid Python syntax
which is then executed.

Question: Can't I accomplish this same functionality via string
parsing and eval?

Answer: Eval will accept any valid Python expression and execute it.
This is however regarded by many as a security hole. Macros allow
this type of expressiveness while limiting many of the security risks.
With a Macro you can more specifically control what types of actions
are permissible.

Question: I have some existing modules, if I import a macro from
someone else, could the macro make my existing code perform in
unexpected ways?

Answer: No. A macro doesn't modify existing code, in order to invoke
a macro it must be called just like a function call. So a Macro
couldn't alter the functionality of existing code any more than a
function could. The only way to accomplish such a feat would be to
manually open each file and parse through them and re-evaluate them.
This type of functionality is not what macros are about any more than
a function call would be.

Question: How would Macros work in practice?

Answer: Here is a trivial example macro to start with:

def HelloFunc(message):
print "%s\n" % (message)

defmacro uselessmacro(message):
macroexpansion = "HelloFunc(%s)" % (message)

defmacro callfunc(funcname, message):
macroexpansion = """
calllog = open("/var/log/functions.log", "a")
calllog.write("%(funcname)s was called.\n")
calllog.flush()
calllog.close()
%(funcname)s(%(message)s)
""" % locals()

if __name__ == "__main__":
uselessmacro("Hello World")
callfunc("HelloFunc", "Hello World")

# Note, Jacek's method of using indentation is an interesting
alternative to calling them as a
# function.

# Second note, this textarea box does not have a compiler, so please
forgive any syntax errors :)

uselessmacro simply passes it's arguments through to HelloFunc
undisturbed
callfunc logs all calls to a function before evaluating the function

You could clearly accomplish this functionality using other means,
however I have used such a trivial example in order to illustrate
several points:

1) Macros once they are defined would most likely be invoked the same
way any other function call would be or represented as a block of
code.

2) Macros taken in isolation would not allow any strange changes to
the language to be made (eg changing all *'s to +'s, or changing the
behavior of existing classes - metaclasses, builtins and magic methods
are required for this)

3) Understanding a macro's behavior is typically easy for someone
familiar with Python's syntax and constructs. In otherwords a Macro
allows you to encapsulate commonly used idioms into a cleaner syntax,
but would not allow you to alter the behavior of the language itself.

My examples are barely touching the surface of what is possible with
macros, but I wanted to start with some simple examples in order to
give everyone some solid examples of what Macros are, and what Macros
are not.

[Terry Reedy]

"Jacek Generowicz" <ja**************@cern.ch> wrote in message
news:ty*************@lxplus014.cern.ch...

This has turned out rather long, but I hope that it demystifies these scary exotic macros for at least one or two people on the list.

Definitely. The sort of concrete example I asked for. Thank you.

Terry

[Michele Simionato]

Jacek Generowicz <ja**************@cern.ch> wrote in message news:<ty*************@lxplus014.cern.ch>...

==== Example 1: a debugging aid ================================

One litte macro I have for debugging purposes (let's call it "show"),
allows me to print out the source code of a number of expressions,
along with their values. At this point, note that Python's horribly
irregular syntax <0.5 wink> already starts causing problems: should
the macro look like a block of code, or should it look like a function
call; in Lisp both look identical.

The "block" version of the show macro invocation:

show:
sin(x)
a
a + b + c

The "function" version of the show macro invocation:

show(sin(x), a, a + b + c)

In both cases, the action of the macro should be to replace itself
with its expansion, _before_ the compiler gets to see the source code.
The expansion should look like this:

print "sin(x) =>", sin(x)
print "a =>", a
print "a + b + c =>", a + b + c

Note the key points:

1) The macro receives some data, and transforms it into valid Python
source code

2) This happens before compile time

3) Nothing "outside" the macro call gets affected by the macro's
action.

Can this be implemented using metaclasses, or any other existing
Python mechanism ?
Yes.

http://groups.google.it/groups?hl=it....lang.python.*

At the end it uses metaclasses+exec+a simple argument parser based on
tokenize. EXPERIMENTAL code, which I would not trust except as proof
of concept.
=== Example 2: Alexander Schmolck's updating classes ===============

I think this also can be done with metaclasses (somebody already
posted a recipe in some other thread, Diederich?)

Michele Simionato, Ph. D.
Mi**************@libero.it
http://www.phyast.pitt.edu/~micheles
--- Currently looking for a job ---

[Jacek Generowicz]

dt*****@yahoo.com (Doug Tolton) writes:

Also note that the examples presented are purposefully trivial and
could in many instances be accomplished other ways,
I hoped that the examples that I presented satisfied the requirements
that

a) They be simple enough to understand

b) not be (at least easily) implementable using existing Python
features

c) Not be completely useless

Did I fail ? :-)
Essentially Macros allow you to pass in some variables that are then
expanded into valid Python syntax which is then executed.
Well, not just variables, data in general, some of which might be
variables, some of which might be variable names (identifiers) and
some of which might be (peudo-) code.
Question: Can't I accomplish this same functionality via string
parsing and eval?

Answer: Eval will accept any valid Python expression and execute it.
This is however regarded by many as a security hole. Macros allow
this type of expressiveness while limiting many of the security risks.
With a Macro you can more specifically control what types of actions
are permissible.
I don't think that this is relevant (or true, for that matter). Lisp
macros do not expand arbitrary data that a user passes in, they appear
as literals in your source code. The security issues are the same as
those surrounding "os.system('rm -rf /')" appearing literally in your
source code.

What is relevant, is the pain involved in doing it via strings. The
(pseudo) code that you pass to a Lisp macro, not only looks exactly
like Lisp code, but it is automatically tokenized and parsed for you,
and very easily manipulatable.
# Note, Jacek's method of using indentation is an interesting
alternative to calling them as a
# function.
It's probably worth pointing out, again, how important Lisp's
uniformity of syntax is, for its macro system. Generally speaking, a
macro accepts as arguments some mixture of data, identifiers and
expressions (let's forget about the existence of Python statements for
now, they complicate the situation even more). How do you pass in a
mixture of data and code?

Data is usually presented in a tuple:

foo(datum1, datum2, datum3)

Expressions are usually presented on separate lines:

def foo(...):
expression1
expression2
expression3

Let's try to think of what a Python with-open-file macro call would
look like.

In CL, with-open-file takes

1) a list containing an identifier (a symbol), a pathname (let's call
it a string) with some optional keywords.

2) arbitrary code

It then binds the identifer to a stream associated with the file, and
expands the source code inside an exception handling environment.

So, the expansion of a (simplistic) Python with_open_file might look
like this:

outfile = file("foo/bar", "w")
try:
print >> outfile, foo(...)
for i in range(10):
print >> outfile, bar(...)
except:
perform the necessary closing and cleaning up

But what should the corresponding call to with_open_file look like?

We have to pass in the identifier "outfile" (this is where it is very
handy to have symbols as a first-class data type, as in Lisp), and the
file name "foo/bar" ... but then we also have to pass in the chunk of
code

print >> outfile, foo(...)
for i in range(10):
print >> outfile, bar(i)

In Lisp it would look something like this:

(with-open-file (outfile "foo/bar" :direction :output)
(format outfile "~a" (foo ...))
(dotimes (i 10)
(format outfile "~&~a" (bar i))))

Now, I refuse to contemplate passing the source code in a string, in
the Python version.

Maybe we have to accept the distinction between data-like data, and
code-like data, having the macro accept both a tuple of arguments, and
blocks of code, so the call might look thus:

with_open_file("outfile", "foo/bar", "w"):
code:
print >> outfile, foo(...)
for i in range(10):
print >> outfile, bar(...)

(If you need to pass in more than one code block, each one would go in
a seperate "code:" block.)

What could the definition look like?

defmacro with_open_file(stream, filename, mode)(block1)
expand(stream) = file(filename, mode)
try:
expand(block1)
except:
perform the necessary closing and clearing up

(expand would be a function which gets replaced by its argument, at
macro expansion time.)

Hmm, just about bearable, I guess, but I doubt that it generalizes
very well.

Note that we didn't transform the source code argument, just pasted it
in directly. If we wanted to manipulate it first, we'd be in for
serious suffering.
You could clearly accomplish this functionality using other means,

I think it is best to steer clear of such examples, as experience
shows that some people will not be able to resist the temptation to
conclude (and suggest) that there is no point to macros, and the
discussion gets derailed. We've seen enough of "you could do this
quite easily without macros, but ...", and not enough of "This would
be soooo painful without macros ..." (I won't say "impossible", as
there is always Turing completeness.)

What was wrong with my original examples? I'd like to know, so that I
can replace them with better ones.

[Jacek Generowicz]

mi**@pitt.edu (Michele Simionato) writes:

Can this be implemented using metaclasses, or any other existing
Python mechanism ?
Yes.

http://groups.google.it/groups?hl=it....lang.python.*

Unfortunately I can't easily access this at the moment. I'll try to
have a look later.
At the end it uses metaclasses+exec+a simple argument parser based on
tokenize. EXPERIMENTAL code, which I would not trust except as proof
of concept.

If it uses exec, then it is faking up macros.

Lisp macros are little else besides offering language support for the
use of exec for this kind of thing.

If people start resorting to this sort of trick themselves, and
re-inventing it over and over again, then you might as well design it
properly, once and for all, and call it a macro system.

I submit that Python already has macros ... they just need a bit of
streamlining :-)

[Jacek Generowicz]

te****@cs.bgu.ac.il (Miki Tebeka) writes:

My personal favorites are the anaphoric macros (aif ...)

Though, you do realize that these are buggy, in terms of their
interaction with the package system :-)

[John J. Lee]

Jacek Generowicz <ja**************@cern.ch> writes:

"Andrew Dalke" <ad****@mindspring.com> writes:

Here's a proposed Q&A for the FAQ based on a couple recent threads.
Appropriate comments appreciated
Such an FAQ item is probably a good idea.

However, I think that we should distil out much of the opinion
(particularly the opinions about opinions of others :-) and try to
base it on fact:

I disagree. People reading a FAQ do want to know the opinion of
language users, and more importantly, of Guido, because that's what
explains why the language is as it is, and determines what will happen
in the future. That's fact too, just fact about people's opinions.

- what are macros (C-like and Lisp-like),

- what are the technical difficulties with introducing them into
Python
Arguably, these are both secondary issues given the fact that both
Guido (I presume?) and most Python users don't want macros. Obviously
some background should be in there explaining what macros are and how
they might apply to Python, but Andrew did just that.
[...snip useful technical criticism of Andrew's FAQ...] Could you please give me a reference to someone "with plenty of
experience using macros in Lisp" arguing this ?
Maybe he was referring to Laura Creighton? He posted a quote and a
link to another of her posts.
[...snip more technical points...] But then, functions which provide an abstraction are also more
difficult to write and understand than ones which merely "move code
out of the way", and you'd have to think twice whether the abstraction
is really useful, before deciding to pay the price for writing it, and
making readers understand it.
I'm amazed that you say this. I think most programmers would disagree
with you here (certainly I do). Sure, abstract functions require some
work to understand the concept involved, but the important way in
which one function is easier or harder to understand than another has
to do with how much detail you need to know, not how hard the concepts
are. Needing to know details causes bugs. So does needing to know
concepts (though less often), but you can't get rid of the concepts,
whereas you *can* hide the details. And that's what abstraction is:
hiding details.

(and there's no need to answer here that you think macros are a good
way of hiding detail: we've gathered that ;-)

[...] People in the music software "group" will find it hard to use code
from people writing software for bioinformatics ... with or without
macros. This has nothing to do with macros.
But people writing code using one bioinformatics library will not find
it hard to read code using another. That might not be the case if
macros were built-in.

OK, I promised some examples of macros. Note that I have not shown a
single line of Lisp code in these threads, because I agree that it is
likely to be meaningless to most readers of this group. I'll try
continue without having to resort to Lisp examples.

==== Example 1: a debugging aid ================================

One litte macro I have for debugging purposes (let's call it "show"),
allows me to print out the source code of a number of expressions,
along with their values. At this point, note that Python's horribly [...] The "function" version of the show macro invocation:

show(sin(x), a, a + b + c) [...is equivalent to...] print "sin(x) =>", sin(x)
print "a =>", a
print "a + b + c =>", a + b + c [...] Can this be implemented using metaclasses, or any other existing
Python mechanism ?
Yes, using Lisp ;-P -- you can do this in Python with an emacs macro.

=== Example 2: Alexander Schmolck's updating classes ===============

Alexander recently expressed the desire to have all existing instances
of a class be updated, when he changes the source of his class, and
re-evaluates the definition.

[...]

There was a thread recently about how the problem behind this might be
solved in Python. I started it (a search for "DOM" or "4DOM" or
"hierarchy" and my email address should turn it up).

No, wait a minute -- instances, not classes?? Sounds like an abuse of
the language to me. What *is* the underlying problem to be solved
here?
John

[John J. Lee]

Jacek Generowicz <ja**************@cern.ch> writes:

dt*****@yahoo.com (Doug Tolton) writes: [...] I hoped that the examples that I presented satisfied the requirements
that

a) They be simple enough to understand

b) not be (at least easily) implementable using existing Python
features

c) Not be completely useless

Did I fail ? :-)

Yes. :-)
John

[John J. Lee]

Jacek Generowicz <ja**************@cern.ch> writes:

mi**@pitt.edu (Michele Simionato) writes:

Can this be implemented using metaclasses, or any other existing
Python mechanism ?

Yes. [...] At the end it uses metaclasses+exec+a simple argument parser based on
tokenize. EXPERIMENTAL code, which I would not trust except as proof
of concept.

If it uses exec, then it is faking up macros.

Lisp macros are little else besides offering language support for the
use of exec for this kind of thing.

If people start resorting to this sort of trick themselves, and
re-inventing it over and over again, then you might as well design it
properly, once and for all, and call it a macro system.

[...]

Well, as has been repeatedly argued, that's just the point: people
*don't* do that over and over again. Who uses these 'faked-up
macros'? A few people who like messing around with metaclasses for
fun (note Michele's warning about 'EXPERIMENTAL' and 'proof of
concept'). If it's made easier, relatively sane people might do it
too, and then we'll all suffer ;-)

I'm sure you're capable of understanding this position, but the
argument has started going round in circles.
John

[Jacek Generowicz]

jj*@pobox.com (John J. Lee) writes:

I'm sure you're capable of understanding this position, but the
argument has started going round in circles.

As I've said repeatedly:

- I do _not_ necessarily advocate the inclusion of macros in Python.

- I merely wish to dispel the myth that macros make code less
readable[*]: they _ease_ understanding of the code.

[*] And they don't fragment languages either.

[John J. Lee]

Jacek Generowicz <ja**************@cern.ch> writes:

jj*@pobox.com (John J. Lee) writes: [...] I agree. However, my feeling was that the original contained too much
uneducated opinion. I was trying to suggest that we first establish
the facts, and then include opinions based on those facts, rather than
ones based on FUD. Particularly, we should banish the myth that macros
fragment languages, or make code less unedrstandable. (Quite the
reverse is true.)
It would be nice to make a definitive ruling :-), but it seems clear
most people here disagree with that. If Andrew's FAQ answer is going
to be linked to, no reason why other people can't supply an
alternative answer and have that linked to from the Python FAQ too.
[...] I agree that abstractions usually hide details. I disagree with the
claim that hiding details is the essence of abstraction. To me, the
essence of abstraction is noticing a similarity between certain
"things", understandig how they differ, and encoding the whole set of
them in one implementation, with an obvious degree of freedom which
selects a specific "thing" from the whole family.
Or, abstracting a bit: hiding details <wink>.

If I have a function which is 400 lines long, and I choose to make it
clearer by cutting and pasting bits of its body into smaller
functions, and calling those functions from the original, I haven't
abstracted anything; I have merely split it into more digestible
chunks.

We're probably in violent agreement.
John

[Doug Tolton]

Jacek Generowicz <ja**************@cern.ch> wrote in message news:<ty*************@lxplus085.cern.ch>...

dt*****@yahoo.com (Doug Tolton) writes:

Also note that the examples presented are purposefully trivial and
could in many instances be accomplished other ways,
I hoped that the examples that I presented satisfied the requirements
that

a) They be simple enough to understand

b) not be (at least easily) implementable using existing Python
features

c) Not be completely useless

Did I fail ? :-)

No I don't think you failed, I just wanted to nail down that specific
point. The reason I purposefully used trivial examples was to
illustrate the point the Macros don't somehow magically alter existing
source code, rather they operate on source code that is passed into
them. I actually liked your examples, I was just hoping to come at
them from a slightly different example.

Essentially Macros allow you to pass in some variables that are then
expanded into valid Python syntax which is then executed.

Well, not just variables, data in general, some of which might be
variables, some of which might be variable names (identifiers) and
some of which might be (peudo-) code.

Question: Can't I accomplish this same functionality via string
parsing and eval?

Answer: Eval will accept any valid Python expression and execute it.
This is however regarded by many as a security hole. Macros allow
this type of expressiveness while limiting many of the security risks.
With a Macro you can more specifically control what types of actions
are permissible.

I don't think that this is relevant (or true, for that matter). Lisp
macros do not expand arbitrary data that a user passes in, they appear
as literals in your source code. The security issues are the same as
those surrounding "os.system('rm -rf /')" appearing literally in your
source code.

What is relevant, is the pain involved in doing it via strings. The
(pseudo) code that you pass to a Lisp macro, not only looks exactly
like Lisp code, but it is automatically tokenized and parsed for you,
and very easily manipulatable.

Again, I wanted to focus the discussion for a moment to help alleviate
people's fear of Macro's, as they seem to be based on
mis-understanding what they are about. Please forgive my over
simplification.

# Note, Jacek's method of using indentation is an interesting
alternative to calling them as a
# function.

It's probably worth pointing out, again, how important Lisp's
uniformity of syntax is, for its macro system. Generally speaking, a
macro accepts as arguments some mixture of data, identifiers and
expressions (let's forget about the existence of Python statements for
now, they complicate the situation even more). How do you pass in a
mixture of data and code?

Data is usually presented in a tuple:

foo(datum1, datum2, datum3)

Expressions are usually presented on separate lines:

def foo(...):
expression1
expression2
expression3

Let's try to think of what a Python with-open-file macro call would
look like.

In CL, with-open-file takes

1) a list containing an identifier (a symbol), a pathname (let's call
it a string) with some optional keywords.

2) arbitrary code

It then binds the identifer to a stream associated with the file, and
expands the source code inside an exception handling environment.

So, the expansion of a (simplistic) Python with_open_file might look
like this:

outfile = file("foo/bar", "w")
try:
print >> outfile, foo(...)
for i in range(10):
print >> outfile, bar(...)
except:
perform the necessary closing and cleaning up

But what should the corresponding call to with_open_file look like?

We have to pass in the identifier "outfile" (this is where it is very
handy to have symbols as a first-class data type, as in Lisp), and the
file name "foo/bar" ... but then we also have to pass in the chunk of
code

print >> outfile, foo(...)
for i in range(10):
print >> outfile, bar(i)

In Lisp it would look something like this:

(with-open-file (outfile "foo/bar" :direction :output)
(format outfile "~a" (foo ...))
(dotimes (i 10)
(format outfile "~&~a" (bar i))))

Now, I refuse to contemplate passing the source code in a string, in
the Python version.

Maybe we have to accept the distinction between data-like data, and
code-like data, having the macro accept both a tuple of arguments, and
blocks of code, so the call might look thus:

with_open_file("outfile", "foo/bar", "w"):
code:
print >> outfile, foo(...)
for i in range(10):
print >> outfile, bar(...)

(If you need to pass in more than one code block, each one would go in
a seperate "code:" block.)

What could the definition look like?

defmacro with_open_file(stream, filename, mode)(block1)
expand(stream) = file(filename, mode)
try:
expand(block1)
except:
perform the necessary closing and clearing up

(expand would be a function which gets replaced by its argument, at
macro expansion time.)

Hmm, just about bearable, I guess, but I doubt that it generalizes
very well.

Note that we didn't transform the source code argument, just pasted it
in directly. If we wanted to manipulate it first, we'd be in for
serious suffering.

You could clearly accomplish this functionality using other means,

I think it is best to steer clear of such examples, as experience
shows that some people will not be able to resist the temptation to
conclude (and suggest) that there is no point to macros, and the
discussion gets derailed. We've seen enough of "you could do this
quite easily without macros, but ...", and not enough of "This would
be soooo painful without macros ..." (I won't say "impossible", as
there is always Turing completeness.)

What was wrong with my original examples? I'd like to know, so that I
can replace them with better ones.

I agree some people have been derailed by that concept, but I'd hoped
that making it painfully clear that this wasn't the only way to do it,
nor was it necessarily a representative sample of Macros, that maybe
some people would see conceptually what Macros are in an extremely
simple way. The Hello World of Macros.

Again, I thought your examples were fine, I just wanted to simplify
them to help the discussion along. If you hate them that much, we can
forget I ever posted them. :-p

Doug Tolton

[Christos TZOTZIOY Georgiou]

On 24 Aug 2003 16:34:08 +0200, rumours say that Jacek Generowicz
<ja**************@cern.ch> might have written:

[See at the end for an interactive session transcript -- a simple copy -
paste]

No, wait a minute -- instances, not classes?? Sounds like an abuse of
the language to me. What *is* the underlying problem to be solved
here?

It started in the Ruby thread: in Ruby classes are "open".

Say you're developing a class (foo), make a gazillion instances of it,
and play around with them as you are continuing to develop your
program. Suddenly you realize "Darn, I really need a fubar method!"

In Ruby you do something like:

class foo
def fubar ...

end
end

All your existing gazillion instances of foo have just acquired a
fubar method, and you can happily continue your exploration with all
the state you built up still intact.

The first thing you might try to do in Python is to add the fubar
method to the class source code, and re-evaluate it ... only to find
that your existing instances know nothing about the new method ... so
you have to go through the hassle of re-creating a gazillion of them,
before you can continue with what you were doing.

##### code starts here #####

class A(object): pass
a= A()
a.test() Traceback (most recent call last):
File "<pyshell#21>", line 1, in ?
a.test()
AttributeError: 'A' object has no attribute 'test' def test(self): print id(self)

A.test = test
a.test()

13111376

##### code ends here #####
--
TZOTZIOY, I speak England very best,
Microsoft Security Alert: the Matrix began as open source.

[Jacek Generowicz]

Christos "TZOTZIOY" Georgiou <tz**@sil-tec.gr> writes:

On 24 Aug 2003 16:34:08 +0200, rumours say that Jacek Generowicz
<ja**************@cern.ch> might have written:

The first thing you might try to do in Python is to add the fubar
method to the class source code, and re-evaluate it ... only to find
that your existing instances know nothing about the new method ... so
you have to go through the hassle of re-creating a gazillion of them,
before you can continue with what you were doing.

##### code starts here #####

class A(object): pass
a= A()
a.test() Traceback (most recent call last):
File "<pyshell#21>", line 1, in ?
a.test()
AttributeError: 'A' object has no attribute 'test' def test(self): print id(self)

A.test = test
a.test()

13111376

##### code ends here #####

No, you are missing the point. The point is that you want your source
code to be in sync with your program state, and you want your source
code to look natural. By "natural" I mean that you want your source
code to look like this:

class A(object):
def test(self):
print id(self)

rather than like this:

class A(object):
pass

def test(self):
print id(self)

A.test = test

If you still don't understand the motivation, please read the original
thread; there's no point in repeating the arguments all over again in
this thread, which is about something completely different.

[John J. Lee]

Jacek Generowicz <ja**************@cern.ch> writes:

jj*@pobox.com (John J. Lee) writes:

I'm sure you're capable of understanding this position, but the
argument has started going round in circles.
As I've said repeatedly:

- I do _not_ necessarily advocate the inclusion of macros in Python.

But you do advocate them in other languages, right? And that was a
significant part of the argument.

- I merely wish to dispel the myth that macros make code less
readable[*]: they _ease_ understanding of the code.
[*] And they don't fragment languages either.

John

[Jacek Generowicz]

jj*@pobox.com (John J. Lee) writes:

Jacek Generowicz <ja**************@cern.ch> writes:

- I do _not_ necessarily advocate the inclusion of macros in Python.

But you do advocate them in other languages, right? And that was a
significant part of the argument.

You've lost me.

What I advocate is that, in the process of constructing an FAQ or
article about macros in Python, one should try to exclude material
which sounds like this:

Q: What is static typing, and why isn't it included in Python?

A: Static typing is the requirement that the programmer write his
code while standing up. Some people believe that this causes
infertility.

Or, to make a closer analogy:

Q: What are functions, and why aren't they included in Assembler ?

A: Functions are a very powerful means of creating
abstarctions. Some people believe that functions make a program
more difficult to understand. Functions also cause languages to
split, as evidenced by the existence of many different languages
which support functions. If any programmer on your project writes
a function, random parts of the code _you_ wrote will stop working;
shortly thereafter frogs will rain from the sky for forty days.

Put another way, I advocate avoiding FUD.

To this end, I have tried to provide information which would help
readers of this list, who are unfamiliar with Lisp-style macros, to
understand what Lisp-style macros are.

If I am involved in an argument, then it is an argument about what
Lisp-style macros _are_ and what they can or can't do ... completely
independently of any language in which they may or may not be
available, now or in the future.

I offer no opinion on whether Lisp-style macros are a good idea. I
would like people to form their own opinions on the matter, but I
would like those opinions to be based on fact rather than on FUD.

Now, what is it that _you_ are arguing ?

[Hannu Kankaanpää]

Jacek Generowicz <ja**************@cern.ch> wrote in message news:<ty*************@lxplus014.cern.ch>...

==== Example 1: a debugging aid ================================
=== Example 2: Alexander Schmolck's updating classes ===============

Here's another example of a real life situation
where I thought it'd be good to have macros in Python.
I was writing a GUI system and I had to define lots of
code twice, both for the x-axis and y-axis. Such
as
if y > self.y - bs and y < self.y + bs + self.height:
return (abs(x - self.x) < bs,
abs(x - (self.x + self.width)) < bs)
else:
return False, False
and then
if x > self.x - bs and x < self.x + bs + self.width:
return (abs(y - self.y) < bs,
abs(y - (self.y + self.height)) < bs)
else:
return False, False
Obviously this was quite unsatisfactory. I ended up
putting the axis code in a separate class so I could
use them interchangeably. I.e. If I passed
func(self.y, self.x)
and then
func(self.x, self.y)

I would get the same effect on both axises. But this
would've been an excellent place for macros IMO (unless
there's a more clever solution as a whole). Using macros
that combine both function call and "code block" syntax,
I could've written a simple function like this:

defBoth getSize(self):
return self.size

And it would've been expanded to

def getWidth(self):
return self.width

def getHeight(self):
return self.height

The macro would've had to change all "size" to either
"width" or "height", and also change "pos" to either "x"
or "y" and so on.

This way, I could've got no duplicated code but
also a more intuitive interface than I currently have
(to get width, one needs to type obj.x.getSize() instead
of obj.getWidth()). And it's obvious this kind of "defBoth"
wouldn't be added as a language level construct -- Thus
macros are the only good solution.

[Jacek Generowicz]

ha******@yahoo.com.au (Hannu Kankaanpää) writes:

if y > self.y - bs and y < self.y + bs + self.height:
return (abs(x - self.x) < bs,
abs(x - (self.x + self.width)) < bs)
else:
return False, False
and then
if x > self.x - bs and x < self.x + bs + self.width:
return (abs(y - self.y) < bs,
abs(y - (self.y + self.height)) < bs)
else:
return False, False

Obviously this was quite unsatisfactory. I ended up
putting the axis code in a separate class
Do you mean "function" rather than "class" ?
so I could
use them interchangeably. I.e. If I passed
func(self.y, self.x)
and then
func(self.x, self.y)

I would get the same effect on both axises. But this
would've been an excellent place for macros IMO
I don't see what you gain by using a macro, wrt to using a function in
_this_ case. As no Python macro system exists, I cannot form an
opinion of what the two approaches would look like in Python, but in
Lisp the calls to the macro would look just like the calls to the
function, I think. The only difference would be in the definition of
the macro/function, and the macro would be no simpler, so you wouldn't
really gain anything.
Using macros
that combine both function call and "code block" syntax,
I could've written a simple function like this:

defBoth getSize(self):
return self.size

And it would've been expanded to

def getWidth(self):
return self.width

def getHeight(self):
return self.height

The macro would've had to change all "size" to either
"width" or "height", and also change "pos" to either "x"
or "y" and so on.
So you would not only replace whole symbols, but even fragments of
symbols (getSize -> getHeight), and thus macically/implicitly create
new symbols. Many people consider this bad practice. The objection
being that readers of the code come across a symbol in the source, go
off and search for its definition, and tricks like this mean that they
end up wasting their time.

The defstruct macro does this sort of thing; (defstruct foo ... )
would "magically" define a constructor called "make-foo" (and other
stuff). IIRC, the experience with defstruct led the designers of CLOS
to explicitly avoid doing this in CLOS' defclass macro (but I'm far
from the world's leading authority on Lisp history).

Incidentally, in this case, having a string based code representation
would make your job much easier than the structured representation
which Lisp uses. In a string you'd merely do a search and replace; in
an s-expression you would have to recursively search for all symbols
in all sub-expressions, and then do the search and replace within the
name of each symbol you find.
This way, I could've got no duplicated code but
also a more intuitive interface than I currently have
(to get width, one needs to type obj.x.getSize() instead
of obj.getWidth()). And it's obvious this kind of "defBoth"
wouldn't be added as a language level construct -- Thus
macros are the only good solution.

Cue metaclass solution ... :-)

(I suspect that even with metaclasses, you wouldn't be able to avoid
eval ... and that amounts to "informally" writing a macro)

[Hannu Kankaanpää]

Jacek Generowicz <ja**************@cern.ch> wrote in message news:<ty*************@pcepsft001.cern.ch>...

ha******@yahoo.com.au (Hannu Kankaanpää) writes:

Obviously this was quite unsatisfactory. I ended up
putting the axis code in a separate class
Do you mean "function" rather than "class" ?

Actually, I did mean class. Normally I'd have

class Widget:
def __init__(self):
self.x = 0
self.y = 0
self.width = 0
self.height = 0

def getWidth(self):
return self.width # normally they wouldn't be this empty!

def getHeight(self):
return self.height

But by wrapping it inside a new class, I could get rid of
the duplication (partly, at least):

class Widget:
def __init__(self):
self.x = Widget.Axis(0, 0)
self.y = Widget.Axis(0, 0)

class Axis:
def __init__(self, pos, size)
self.pos = pos
self.size = size

def getSize(self): # this is only once here now
return self.size

While this tiny example doesn't show decrease in code
size, it shows that I have a common definition for all
"Axis"-specific code inside the appropriate Axis class.
Rest of the Widget methods would be in the widget class.
Thus self.x.getSize() instead of self.getWidth().

so I could
use them interchangeably. I.e. If I passed
func(self.y, self.x)
and then
func(self.x, self.y)

I would get the same effect on both axises. But this
would've been an excellent place for macros IMO

I don't see what you gain by using a macro, wrt to using a function in
_this_ case.

Ok, it was a bad example. I hope the code above shows
a bit more clearly what I wanted. Anyway, without the code
in the axis-class, I would've had to often say

self.x = func(self.x, self.y, self.width, self.height)
self.y = func(self.y, self.x, self.height, self.width)

Instead of

func(self.x, self.y)
func(self.y, self.x)

Which could modify the axis-specific stuff within the
func()tion. (self.x is no longer a non-modifiable number,
but a modifiable class)
So you would not only replace whole symbols, but even fragments of
symbols (getSize -> getHeight), and thus macically/implicitly create
new symbols. Many people consider this bad practice.
Well, I don't, really. Like any macro that could do something
weird, it just needs to be properly understood by anyone who
wishes to read the code.
Incidentally, in this case, having a string based code representation
would make your job much easier than the structured representation
which Lisp uses. In a string you'd merely do a search and replace; in
an s-expression you would have to recursively search for all symbols
in all sub-expressions, and then do the search and replace within the
name of each symbol you find.

Well, such a recursive search isn't a problem - With help from
a macro ;)

This way, I could've got no duplicated code but
also a more intuitive interface than I currently have
(to get width, one needs to type obj.x.getSize() instead
of obj.getWidth()). And it's obvious this kind of "defBoth"
wouldn't be added as a language level construct -- Thus
macros are the only good solution.

Cue metaclass solution ... :-)

How could metaclasses help? I'm quite inexperienced with them.
Anyway, if I take the eval route, I might as well do

defBoth('''getSize():
return size''')

,retreive appropriate locals() from the stack and modify it
to include the new functions. I'd rather not, though :-)