Limits of Metaprogramming

[Wilson]

Hi all,

I have an interesting problem that I'm hoping can be solved with
metaprogramming , but I don't know how far Python supports code
generation (and I don't know if I'm taking the correct approach
either... hence why I'm asking on this group):

I'd like to write a program that writes/manipulates a statemachine. My
idea was that I would define states in a .py file along with their
transitions and other metadata. I could then write a suite of programs
that would manipulate these states in interesting ways, such as
generate diagrams and documentation, test suites, do verifications and
also edit them. These programs would import the .py file and use
introspection upon the module's classes (states would be modelled as
classes). For example, a documentation tool could use graphviz to draw
the statediagram with the transition table and docstring...

My problem is that I don't know if it's possible to edit these states
and then write them back to .py. Firstly, if my editing tool was to
create a new state, I would want to create the class (using type) and
attach it to the imported state module somehow. I can't find
information on whether this is possible. Secondly, I'd like to
manipulate the class and then use the inspect module to get the source
and write it back to file. Now I'm pretty sure the inspect module will
only write the source code as found in the original .py file. Is there
a way to generate the code for a class or other python objects in a
generic way? Or will I have to bite the bullet and write to custom
file and generate code from that (like glade with XML)? Or is there
another solution with templates that I'm over looking?

Having the classes in memory, editing their attributes, adding methods
and then serialising them to code has an elegance to it in my mind and
would hopefully simplify things. The result would be directly
executable by the interpreter itself and editable by python
developers. But I fear that I may need an intermediary notation/format
for storage. Any advice/hints would be very useful.

Thanks,
Paul

[Jeff]

You could write a class composed of states and then use the pickle
module to serialize it to disk.

[Wilson]

On 4 Aug, 12:55, Jeff <jeffo...@gmail .comwrote:

You could write a class composed of states and then use the pickle
module to serialize it to disk.

Thanks Jeff.

I guess this is my intermediary format!

[Tomasz Rola]

On Mon, 4 Aug 2008, Wilson wrote:

Hi all,

Howdy,

I am not sure if my remarks will be of any use for you, but here it goes.

I have an interesting problem that I'm hoping can be solved with
metaprogramming , but I don't know how far Python supports code
generation (and I don't know if I'm taking the correct approach
either... hence why I'm asking on this group):

I'd like to write a program that writes/manipulates a statemachine. My
idea was that I would define states in a .py file along with their
transitions and other metadata. I could then write a suite of programs
that would manipulate these states in interesting ways, such as
generate diagrams and documentation, test suites, do verifications and
also edit them. These programs would import the .py file and use
introspection upon the module's classes (states would be modelled as
classes). For example, a documentation tool could use graphviz to draw
the statediagram with the transition table and docstring...

Maybe you have actually two problems.

One problem is processing source code into something useful, but
not executable. So it does not necessarily need metaprogramming to be
solved, just some "code comprehension" (like get a python grammar,
lex/yacc and read a lot of docs how to use them).

My problem is that I don't know if it's possible to edit these states
and then write them back to .py. Firstly, if my editing tool was to
create a new state, I would want to create the class (using type) and
attach it to the imported state module somehow. I can't find
information on whether this is possible. Secondly, I'd like to
manipulate the class and then use the inspect module to get the source
and write it back to file. Now I'm pretty sure the inspect module will
only write the source code as found in the original .py file. Is there
a way to generate the code for a class or other python objects in a
generic way? Or will I have to bite the bullet and write to custom
file and generate code from that (like glade with XML)? Or is there
another solution with templates that I'm over looking?

Well, "everything " is doable in a Turing-complete language :-). However,
this one problem - code generation and using it on the fly - may be better
solved with some other language, perhaps?

Not that I want to dissuade you (or anyone else) from using Python, but
last time I checked the state of things (was a good couple of years ago, I
wanted to solve some math-related problem with metaprogramming in Python)
I decided I would do better if I learned Scheme. At least in Scheme,
program structure is so simple, it is just a list (or a sequence of them),
so manipulating code is IMHO much easier.

I mean, even if in theory every language in Turing-complete class is equal
to each other (one can do the same kind of stuff in one as in the other),
in practice they are not equal at all. So doing everything in one ultimate
language of choice is not very wise from my point of view.

It does not mean you should immediately go to

http://en.wikipedia.org/wiki/Scheme_...mming_language)

but you may do so, and then make an informed choice by yourself. To be
better informed, you may also google for more material - introductions,
opinions, comparisons and some first-hand experience.

On the other hand, writing "metafiles" and interpreters for them - ugh...
:-). I have read somewhere, that some programmers resist using Scheme
(or some other Lisp) for solving their problem and end up with writing
their own Lisp-like (language|file format) and interpreter(s) for it. So
you better watch yourself. If this effort is going to be something more
permament than "done and go", then you will probably find yourself in the
situation of reinventing the wheel, as your project goes more and more
complex. If this is going to happen, you might use existing wheels and
save some time for thinking on something better.

Having the classes in memory, editing their attributes, adding methods
and then serialising them to code has an elegance to it in my mind and
would hopefully simplify things. The result would be directly
executable by the interpreter itself and editable by python
developers. But I fear that I may need an intermediary notation/format
for storage. Any advice/hints would be very useful.

Yes, there is something elegant in this way of thinking, at least for me.

As I (indirectly) said above, I am not up to date with recent Python
abilities (I still use Python for a lot of things, I just do not overuse
it for things, that it seems to be not so good at). I hope someone knows
better and steps in to prove me wrong - it is always good to learn
something new. Also, if you have just one problem, then learning a new
language because of it is a bit too much (or maybe not, if you enjoy it
:-) ). But if you plan doing similar or more difficult things in the
future, then knowing Scheme may be good for you. Even if you choose to not
use it, you will be aware of what can be done with it so this knowledge
can provide some ready to use templates.

And yes, there are also some other languages in Lisp family, but I think
Scheme is best choice if you don't know any of them already. It is quite
small, it is well defined (I mean, it has some real specification,
instead of being "specified by its implementation" ) and there is a lot of
info about it on the net that could be easily understood. And, last but
not least, it has metaprogramming included with nice bunch of additional
stuff. For a start, PLT's DrScheme looks nice (IMHO - yes, there are
other nice looking Scheme implementations but this one is probably best
fitted for a beginner):

http://www.plt-scheme.org/

Regards,
Tomasz Rola

--
** A C programmer asked whether computer had Buddha's nature. **
** As the answer, master did "rm -rif" on the programmer's home **
** directory. And then the C programmer became enlightened... **
** **
** Tomasz Rola mailto:to****** ***@bigfoot.com **

[Wilson]

On 4 Aug, 14:47, Tomasz Rola <[e-mail] m.plwrote:

On Mon, 4 Aug 2008, Wilson wrote:

Hi all,

Howdy,

I am not sure if my remarks will be of any use for you, but here it goes.

I have an interesting problem that I'm hoping can be solved with
metaprogramming , but I don't know how far Python supports code
generation (and I don't know if I'm taking the correct approach
either... hence why I'm asking on this group):

I'd like to write a program that writes/manipulates a statemachine. My
idea was that I would define states in a .py file along with their
transitions and other metadata. I could then write a suite of programs
that would manipulate these states in interesting ways, such as
generate diagrams and documentation, test suites, do verifications and
also edit them. These programs would import the .py file and use
introspection upon the module's classes (states would be modelled as
classes). For example, a documentation tool could use graphviz to draw
the statediagram with the transition table and docstring...

Maybe you have actually two problems.

One problem is processing source code into something useful, but
not executable. So it does not necessarily need metaprogramming to be
solved, just some "code comprehension" (like get a python grammar,
lex/yacc and read a lot of docs how to use them).

I do want my resultant code to be executable; I want to be able to use
"import" to get programmatical access to the classes and structures it
contains, such that they can be manipulated, added to, exercised and
drawn (using graphviz) via introspection.. . After any manipulation,
i'd like to be able to flatten them back into a .py file (as source
such that they are readable as Python code).

My problem is that I don't know if it's possible to edit these states
and then write them back to .py. Firstly, if my editing tool was to
create a new state, I would want to create the class (using type) and
attach it to the imported state module somehow. I can't find
information on whether this is possible. Secondly, I'd like to
manipulate the class and then use the inspect module to get the source
and write it back to file. Now I'm pretty sure the inspect module will
only write the source code as found in the original .py file. Is there
a way to generate the code for a class or other python objects in a
generic way? Or will I have to bite the bullet and write to custom
file and generate code from that (like glade with XML)? Or is there
another solution with templates that I'm over looking?

Well, "everything " is doable in a Turing-complete language :-). However,
this one problem - code generation and using it on the fly - may be better
solved with some other language, perhaps?

Not that I want to dissuade you (or anyone else) from using Python, but
last time I checked the state of things (was a good couple of years ago, I
wanted to solve some math-related problem with metaprogramming in Python)
I decided I would do better if I learned Scheme. At least in Scheme,
program structure is so simple, it is just a list (or a sequence of them),
so manipulating code is IMHO much easier.

I mean, even if in theory every language in Turing-complete class is equal
to each other (one can do the same kind of stuff in one as in the other),
in practice they are not equal at all. So doing everything in one ultimate
language of choice is not very wise from my point of view.

It does not mean you should immediately go to

http://en.wikipedia.org/wiki/Scheme_...mming_language)

but you may do so, and then make an informed choice by yourself. To be
better informed, you may also google for more material - introductions,
opinions, comparisons and some first-hand experience.

On the other hand, writing "metafiles" and interpreters for them - ugh...
:-). I have read somewhere, that some programmers resist using Scheme
(or some other Lisp) for solving their problem and end up with writing
their own Lisp-like (language|file format) and interpreter(s) for it. So
you better watch yourself. If this effort is going to be something more
permament than "done and go", then you will probably find yourself in the
situation of reinventing the wheel, as your project goes more and more
complex. If this is going to happen, you might use existing wheels and
save some time for thinking on something better.

I appreciate these comments, and I know the quote that you are
refering to although I can't find it now... Something like:
" Every sufficiently large application has a poor/incomplete
implementation of LISP embedded within it ". I've looked at LISP
before and do appreciate its elegance, but Python has a beauty of its
own in its pragmatism, standard libraries and community. So I'll
choose to stick with it.

Having the classes in memory, editing their attributes, adding methods
and then serialising them to code has an elegance to it in my mind and
would hopefully simplify things. The result would be directly
executable by the interpreter itself and editable by python
developers. But I fear that I may need an intermediary notation/format
for storage. Any advice/hints would be very useful.

Yes, there is something elegant in this way of thinking, at least for me.

As I (indirectly) said above, I am not up to date with recent Python
abilities (I still use Python for a lot of things, I just do not overuse
it for things, that it seems to be not so good at). I hope someone knows
better and steps in to prove me wrong - it is always good to learn
something new. Also, if you have just one problem, then learning a new
language because of it is a bit too much (or maybe not, if you enjoy it
:-) ). But if you plan doing similar or more difficult things in the
future, then knowing Scheme may be good for you. Even if you choose to not
use it, you will be aware of what can be done with it so this knowledge
can provide some ready to use templates.

I've been through quite a few of the SICP lectures and again, do
appreciate its elegance and they have changed the way I program.

And yes, there are also some other languages in Lisp family, but I think
Scheme is best choice if you don't know any of them already. It is quite
small, it is well defined (I mean, it has some real specification,
instead of being "specified by its implementation" ) and there is a lot of
info about it on the net that could be easily understood. And, last but
not least, it has metaprogramming included with nice bunch of additional
stuff. For a start, PLT's DrScheme looks nice (IMHO - yes, there are
other nice looking Scheme implementations but this one is probably best
fitted for a beginner):

http://www.plt-scheme.org/

Regards,
Tomasz Rola

Thanks for your comments. But I believe my solution may lie in using a
template language such as cheetah. Too much is already invested in
Python!

Best Regards,
Paul

[John Nagle]

Wilson wrote:

Hi all,

I have an interesting problem that I'm hoping can be solved with
metaprogramming , but I don't know how far Python supports code
generation (and I don't know if I'm taking the correct approach
either... hence why I'm asking on this group):

I'd like to write a program that writes/manipulates a statemachine. My
idea was that I would define states in a .py file along with their
transitions and other metadata.

You're probably better off coming up with a representation for a
state machine (which might be a pickled collection of Python objects)
then providing a "compile" operation which cranks out a .py file
with code to implement the state machine.

John Nagle

[Tomasz Rola]

On Mon, 4 Aug 2008, Wilson wrote:

" Every sufficiently large application has a poor/incomplete
implementation of LISP embedded within it ".

Yep, this is either exact or very close copy of what I have read.

I've looked at LISP
before and do appreciate its elegance, but Python has a beauty of its
own in its pragmatism, standard libraries and community. So I'll
choose to stick with it.

[...]

I've been through quite a few of the SICP lectures and again, do
appreciate its elegance and they have changed the way I program.

[...]

Thanks for your comments. But I believe my solution may lie in using a
template language such as cheetah. Too much is already invested in
Python!

Best Regards,
Paul

Ok, so you do know something about Lisp - that is good :-). Of course,
since you have already existing Python code, converting is not for you.
Just to make sure you stay informed, there is CLPython:

http://common-lisp.net/project/clpython/

I did not try this thing, but I am very pleased it exists. I have some
Python code myself and knowing there is a way to reuse it if I choose some
other way makes me feel better. Or rather, knowing I can easily merge the
two (or more) ways.

Regards,
Tomasz Rola

--
** A C programmer asked whether computer had Buddha's nature. **
** As the answer, master did "rm -rif" on the programmer's home **
** directory. And then the C programmer became enlightened... **
** **
** Tomasz Rola mailto:to****** ***@bigfoot.com **

[Wilson]

On 4 Aug, 16:50, John Nagle <na...@animats. comwrote:

Wilson wrote:

Hi all,

I have an interesting problem that I'm hoping can be solved with
metaprogramming , but I don't know how far Python supports code
generation (and I don't know if I'm taking the correct approach
either... hence why I'm asking on this group):

I'd like to write a program that writes/manipulates a statemachine. My
idea was that I would define states in a .py file along with their
transitions and other metadata.

* * You're probably better off coming up with a representation for a
state machine (which might be a pickled collection of Python objects)
then providing a "compile" operation which cranks out a .py file
with code to implement the state machine.

* * * * * * * * * * * * * * * * John Nagle

I agree. I guess my 'representation ' is the big question!

Thanks!

[castironpi]

On Aug 4, 4:48*am, Wilson <PaulAlexWil... @gmail.comwrote :

Hi all,

My problem is that I don't know if it's possible to edit these states
and then write them back to .py. Firstly, if my editing tool was to
create a new state, I would want to create the class (using type) and
attach it to the imported state module somehow. I can't find

Thanks,
Paul

Yes it's possible, using type, but you can't attach it to the module
in any persistent way. The next time you load the module, it's back
to the way it was.

I think this may be the hang-up in your design: if you think about
generating lines of code directly from the user's actions, you might
stay on the right track. At that point, you can place them in the .py
file. Then you can reimport it, or just execute the new lines
directly.

If this is your class,

class Auto54701:
def next( self, input ):
if( something ):
return value1
return value2

Here's how to work with it at the interactive prompt:

>>s= """class Auto54701:

... def next( self, input ):
... if( something ):
... return value1
... return value2
... """

>>s

'class Auto54701:\n def next( self, input ):\n if( something ):
\n ret
urn value1\n return value2\n'

>>exec( s )
Auto54701

<class __main__.Auto54 701 at 0x00A80C60>

>>>

And by the way, you can't use pickle to store types. There needs to
be some fleshed-out code to back them up.

Two more alternatives, which I can discuss more later, one where you
implement code as data, but it ends up being a lot like Python
anyway. It only works if your classes' methods are going to be
simple, probably sequential if-s and assignments at most.

class Auto54701:
def next( self, input ):
if( something1 ):
return value1
elif( something2 ):
return value2
elif( something3 ):
return value3
elif( something4 ):
return value4
return value5

can be stored in data as

[ ( something1, value1 ), ( something2, value2 ), ... ]

, and just checked in a loop, that you only have in code once, rather
than many times.

class GenericAuto:
def next( self, input ):
for something, value in conditionpairs:
if( something ):
return value
return valuen

Two, if all your methods will have uniform signatures and closures,
you can store class methods as only their co_code objects:

>>C.g.im_func.f unc_code.co_cod e

'd\x00\x00S'

And fabricate them dynamically into full live types as needed.

[Wilson]

On Aug 4, 6:49*pm, castironpi <castiro...@gma il.comwrote:

On Aug 4, 4:48*am, Wilson <PaulAlexWil... @gmail.comwrote :

Hi all,

My problem is that I don't know if it's possible to edit these states
and then write them back to .py. Firstly, if my editing tool was to
create a new state, I would want to create the class (using type) and
attach it to the imported state module somehow. I can't find

Thanks,
Paul

Yes it's possible, using type, but you can't attach it to the module
in any persistent way. *The next time you load the module, it's back
to the way it was.

I think this may be the hang-up in your design: if you think about
generating lines of code directly from the user's actions, you might
stay on the right track. *At that point, you can place them in the .py
file. *Then you can reimport it, or just execute the new lines
directly.

If this is your class,

class Auto54701:
* def next( self, input ):
* * *if( something ):
* * * *return value1
* * *return value2

Here's how to work with it at the interactive prompt:

>s= """class Auto54701:

... * def next( self, input ):
... * * *if( something ):
... * * * *return value1
... * * *return value2
... """>>s

'class Auto54701:\n *def next( self, input ):\n * * if( something ):
\n * * * ret
urn value1\n * * return value2\n'>>exec ( s )

>Auto54701

<class __main__.Auto54 701 at 0x00A80C60>

And by the way, you can't use pickle to store types. *There needs to
be some fleshed-out code to back them up.

Two more alternatives, which I can discuss more later, one where you
implement code as data, but it ends up being a lot like Python
anyway. *It only works if your classes' methods are going to be
simple, probably sequential if-s and assignments at most.

class Auto54701:
* def next( self, input ):
* * *if( something1 ):
* * * *return value1
* * *elif( something2 ):
* * * *return value2
* * *elif( something3 ):
* * * *return value3
* * *elif( something4 ):
* * * *return value4
* * return value5

can be stored in data as

* [ ( something1, value1 ), ( something2, value2 ), ... ]

, and just checked in a loop, that you only have in code once, rather
than many times.

class GenericAuto:
* def next( self, input ):
* * for something, value in conditionpairs:
* * * if( something ):
* * * * return value
* * return valuen

Two, if all your methods will have uniform signatures and closures,
you can store class methods as only their co_code objects:

>C.g.im_func.fu nc_code.co_code

'd\x00\x00S'

And fabricate them dynamically into full live types as needed.

Thanks for your comments and advice. This second option intrigues me;
could you elaborate further, I don't follow you...

Thanks Paul