Macro expansion: intercept statement interpretation

[Caleb Hattingh]

Hi

Here is a script I want to be able to write (explanation appears after):

*** start of script ***

import MyCustomMacroLib # This does the magic I would like help for.

# This is not python, but the module imported above
# will use this block internally and prevent it
# getting to the interpreter.
defmacro MyMacro1:
form = """for <#1> := <#2> to <#3> do begin
<#4multiline>
end"""
subsituteWith = """for <#1> in range(<#2>,<#3>+1):
<#4multiline>"""

for i := 0 to 3 do begin # This is not python syntax, but the module
print i # imported above makes it so that this
print 'hello' # is first checked against defined macros
end # and substitutions are made if syntax
# matches
""" Should give:
0
hello
1
hello
2
hello
3
hello
"""

*** end of script ***

I want the module "MyCustomMacroLib" to set up something where I can have
write code that checks each of the statements of this script, and does
macro expansion before passing the result to the python interpreter.

When the (undefined in python) "defmacro" statement is encountered, the
module sets this up in memory as some kind of "macro" object, and as the
remainder of the lines is parsed, a syntax check is made to see if a macro
fits, and if so the expansion is performed and the result is passed to the
interpreter/compiler.

This particular (useless) example sets up a macro where loops can be
written in objectpascal form. I actually have no particular need for this
kind of functionality (I think?), but it looks cool!. I got interested
after reading a couple of Paul Grahams essays about lisp. I also suspect
there actually may be some real problem domains where extending the
language syntax may be beneficial - I just cannot think of any right now :)

I can figure out the minor practical details of things like handling
indention, code objects and the like myself, but I don't know where to
start to try and intercept statements before they get to the interpreter.
Some of the docs I read today appear to indicate that there may also be
differences in the way interactive interpreter and command-line execution
deal with statements. I am lost in the python internals!

Note that I could easily write a script that takes another script as an
argument, preprocesses it, and evaluates the resulting list of python
statments. I do however, prefer a drop-in module as shown above. Plus, I
would also like to do this dynamically in the interpreter.

Any ideas, or is this impossible? (gratuitous guru-bait)

Thanks
Caleb

[Benjamin Niemann]

You could try to override __import__ to first parse the source for macro
definitions and do the expansions. Usage would then be:

import DoMagicWithImportModule
import ModuleWithMacros

(Your code wouldn't work, because python throws a SyntaxError long
before it even tries to execute 'import MyCustomMacroLib')
The code at http://docs.python.org/lib/examples-imp.html could be
extended to first read the contents of the module (after find_module)
and passes the expanded source (e.g. as a StringIO) to load_module.

This has of course the problem (that many implementations of macro
expansion share) that line numbers in exception dumps have not much
relation to lines of the unexpanded code...

[Benjamin Niemann]

Benjamin Niemann wrote:

You could try to override __import__ to first parse the source for macro
definitions and do the expansions. Usage would then be:

import DoMagicWithImportModule
import ModuleWithMacros

(Your code wouldn't work, because python throws a SyntaxError long
before it even tries to execute 'import MyCustomMacroLib')
The code at http://docs.python.org/lib/examples-imp.html could be
extended to first read the contents of the module (after find_module)
and passes the expanded source (e.g. as a StringIO) to load_module.

This has of course the problem (that many implementations of macro
expansion share) that line numbers in exception dumps have not much
relation to lines of the unexpanded code...

If this actually works, a nice application could be rapid prototyping of
syntax extensions to python ('import this module and you can test and
see why my decorator syntax is better than yours' ;)

[Caleb Hattingh]

On Wed, 25 Aug 2004 00:07:22 +0200, Benjamin Niemann <pi**@odahoda.de>
wrote:

...
If this actually works, a nice application could be rapid prototyping of
syntax extensions to python ('import this module and you can test and
see why my decorator syntax is better than yours' ;)

Well spotted! I didn't think of that. The ongoing decorator thread could
have had some real-world testing being done. Also, the kind of macro
expansion I want to play around could avoid the need for decorators to be
added to the language in the first place...