By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
440,092 Members | 1,546 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 440,092 IT Pros & Developers. It's quick & easy.

Running autogenerated code in another python instance

P: n/a
Hi all,

I've got an application that I'm writing that autogenerates python code
which I then execute with exec(). I know that this is not the best way to
run things, and I'm not 100% sure as to what I really should do. I've had a
look through Programming Python and the Python Cookbook, which have given me
ideas, but nothing has gelled yet, so I thought I'd put the question to the
community. But first, let me be a little more detailed in what I want to
do:

I have a python module (called pyvisi, but you don't need to know that)
which attempts to simplify the writing of scripts for high performance
computing visualisation applications. What it does is provides a layer
between the user and the actual renderer backend that is actually going to
process the code. Effectively all my app does is to autogenerate the code
the user would have to write were they savvy with the python interface to
vtk (the visualisation toolkit). This reduces the effort on the part of the
user quite a lot. What I have currently is my python module generating the
equivalent vtk-python code and then executing this in an exec(). This is
not nice (and potentially very slow), especially if one has to share around
data, so what I want to do is have a separate python process or thread which
just sits there accepting the autogenerated text strings as if the user were
writing them directly at the python prompt (or equivalent), and returning
any error messages generated. Also, I want to be able to share data between
the two processes/threads so that one doesn't have to turn numerical data
into a string which is then turned back into numerical data inside the
exec() call (ugly, I know, but it works).

Maybe a picture will help as well (time going down the page):

Main Proc(1)
|
|------------> Renderer(2)
| |
| <-- Data(3) --> |
| |
| more commands |
| ---------------> |
| |
| even more cmds |
| ---------------> |
| |
| render finished |
| shut down Rndrr |
| <--------------- |
|
|
main proc continues or finishes
(1) the main process where the python code destined for the backend is
generated
(2) the secondary process which accepts and runs the code it receives
(3) data to visualised; shared between the two processes

Ok, hopefully you get what I want to do now... So, what is the best way to
do this? Threads share memory, so this is handy to share the data around,
however, how does one send arbitrary commands to be processed by a thread?
One way to do this would be to use pipes, but this means that I can't share
the data around as easily. I've also seen the Pyro project as a
possibility, but I would like to keep this as "core python" as possible.

Any help or advice would be really (really!) appreciated.

TIA

Paul
--
Paul Cochrane
Earth Systems Science Computational Centre
University of Queensland, Brisbane, Queensland 4072, Australia
E: cochrane at esscc dot uq dot edu dot au
Nov 2 '05 #1
Share this Question
Share on Google+
8 Replies


P: n/a
On Wed, 2 Nov 2005 06:08:22 +0000 (UTC), Paul Cochrane <co******@shake56.esscc.uq.edu.au> wrote:
Hi all,

I've got an application that I'm writing that autogenerates python code
which I then execute with exec(). I know that this is not the best way to
run things, and I'm not 100% sure as to what I really should do. I've had a
look through Programming Python and the Python Cookbook, which have given me
ideas, but nothing has gelled yet, so I thought I'd put the question to the
community. But first, let me be a little more detailed in what I want to
do:
[...]
Any help or advice would be really (really!) appreciated.

It's a little hard to tell without knowing more about your
user input (command language?) syntax that is translated to
or feeds the process that "autogenerates python code".

E.g., is it a limited python subset that you are accepting as input,
or a command language that you might implement using the cmd module, or ?
There are lots of easy things you could do without generating and exec-ing
python code per se. How complex is a user session state? What modifies it?
What actions are possible? History? Undo? Is the data a static passive
resource to view, or partly generated or made accessible by prior actions
in a session? Single user or collaborative? Shared access to everything or
only to static data? Etc., etc.

Some examples of user input and corresponding generated python might help,
with some idea of the kind of visualization aspects being controlled ;-)

Regards,
Bengt Richter
Nov 2 '05 #2

P: n/a
Hi!

I did not understand anything with your chatterer message.
But, perhaps, the code below will help you.

@-salutations

Michel Claveau

def runcode(srcode):
import sys,traceback
sret=True
try:
ccod=compile(srcode, 'Paul-code', 'exec')
flagcompile=True
except:
print "compilo-error"
flagcompile=False
if flagcompile==True:
try:
exec(ccod,globals(),globals())
sret=True
except:
tb=sys.exc_info()[2] #for traceback
print str(sys.exc_info()[0])
print traceback.format_exc()
sret=False
return sret
s="""global c
a=1000
b=123
c=a+b"""

if runcode(s):
print c

Nov 2 '05 #3

P: n/a
With good spaces :
def runcode(srcode):
import sys,traceback
sret=True
try:
ccod=compile(srcode, 'Paul-code', 'exec')
flagcompile=True
except:
print "compilo-error"
flagcompile=False
if flagcompile==True:
try:
exec(ccod,globals(),globals())
sret=True
except:
tb=sys.exc_info()[2] #for traceback
print str(sys.exc_info()[0])
print traceback.format_exc()
sret=False
return sret
s="""global c
a=1000
b=123
c=a+b"""
if runcode(s):
print c

s="""global c
a=1000
b=123
c=aaa+b"""
if runcode(s):
print c
Nov 2 '05 #4

P: n/a
On Wed, 02 Nov 2005 06:33:28 +0000, Bengt Richter wrote:
On Wed, 2 Nov 2005 06:08:22 +0000 (UTC), Paul Cochrane <co******@shake56.esscc.uq.edu.au> wrote:
Hi all,

I've got an application that I'm writing that autogenerates python code
which I then execute with exec(). I know that this is not the best way to
run things, and I'm not 100% sure as to what I really should do. I've had a
look through Programming Python and the Python Cookbook, which have given me
ideas, but nothing has gelled yet, so I thought I'd put the question to the
community. But first, let me be a little more detailed in what I want to
do:

Bengt,

Thanks for your reply!
It's a little hard to tell without knowing more about your
user input (command language?) syntax that is translated to
or feeds the process that "autogenerates python code". Ok, I'll try and clarify things as much as I can.
E.g., is it a limited python subset that you are accepting as input,
or a command language that you might implement using the cmd module, or ? It's basically just a command language I guess. Perhaps it's best to
explain using an example. Here's the pyvisi code that generates a very
simple line plot using the vtk renderer module:

"""
Example of plotting lines with pyvisi
"""

# set up some data to plot
from Numeric import *

x = arange(10, typecode=Float)
y = x**2

# example code for how a user would write a script in pyvisi
from pyvisi import * # base level visualisation stuff
# import the objects to render the scene using the specific renderer
#from pyvisi.renderers.gnuplot import * # gnuplot
from pyvisi.renderers.vtk import * # vtk
#from pyvisi.renderers.plplot import * # plplot

# define the scene object
# a Scene is a container for all of the kinds of things you want to put
# into your plot for instance, images, meshes, arrow/vector/quiver plots,
# contour plots, spheres etc.
scene = Scene()

# create a LinePlot object
plot = LinePlot(scene)

# add some helpful info to the plot
plot.title = 'Example 2D line plot'
plot.xlabel = 'x'
plot.ylabel = 'x^2'

plot.linestyle = 'lines'

# assign some data to the plot
plot.setData(x, y)

# render the scene to screen
scene.render(pause=True, interactive=True)

# save the scene out to file
## png
plot.setData(x, y) # have to do this now because we've already
# render()ed the scene, will be removed in the
# future
scene.save(fname="simpleLinePlot.png", format=PngImage())

This code then gets translated by the pyvisi module into the vtk-python
code:

import vtk
from Numeric import *
# LinePlot.__init__()
_plot = vtk.vtkXYPlotActor()
_renderer = vtk.vtkRenderer()
_renderWindow = vtk.vtkRenderWindow()
_renderWindow.AddRenderer(_renderer)
_renderWindow.SetSize(640,480)
_renderer.SetBackground(1,1,1)
# Renderer._initRendererModule
# LinePlot.setData()
_x = array([0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0])
_xData = vtk.vtkDataArray.CreateDataArray(vtk.VTK_FLOAT)
_xData.SetNumberOfTuples(len(_x))
_y0 = array([0.0, 1.0, 4.0, 9.0, 16.0, 25.0, 36.0, 49.0, 64.0, 81.0])
_y0Data = vtk.vtkDataArray.CreateDataArray(vtk.VTK_FLOAT)
_y0Data.SetNumberOfTuples(len(_y0))
for i in range(len(_x)):
_xData.SetTuple1(i,_x[i])
for i in range(len(_x)):
_y0Data.SetTuple1(i,_y0[i])
_fieldData0 = vtk.vtkFieldData()
_fieldData0.AllocateArrays(2)
_fieldData0.AddArray(_xData)
_fieldData0.AddArray(_y0Data)
_dataObject0 = vtk.vtkDataObject()
_dataObject0.SetFieldData(_fieldData0)
_plot.AddDataObjectInput(_dataObject0)
_plot.SetXValuesToValue()
_plot.SetDataObjectXComponent(0,0)
_plot.SetDataObjectYComponent(0,1)
_plot.GetXAxisActor2D().GetProperty().SetColor(0, 0, 0)
_plot.GetYAxisActor2D().GetProperty().SetColor(0, 0, 0)
_renderer.SetBackground(1.0, 1.0, 1.0)
_lut = vtk.vtkLookupTable()
_lut.Build()
_colours = []
_colours.append(_lut.GetColor(0))

_plot.SetPlotColor(0, _colours[0][0], _colours[0][1], _colours[0][2])
_plot.SetPosition(0.1, 0.1)
_plot.SetWidth(0.8)
_plot.SetHeight(0.8)
# Scene.render()
_lut = vtk.vtkLookupTable()
_refLut = vtk.vtkLookupTable()
_lut.Build()
_refLut.Build()
for _i in range(256):
_lut.SetTableValue(_i, _refLut.GetTableValue(255-_i))
_iRenderer = vtk.vtkRenderWindowInteractor()
_iRenderer.SetRenderWindow(_renderWindow)
# LinePlot.render()
_renderer.AddActor2D(_plot)
_plot.SetTitle('Example 2D line plot')
_plot.SetXTitle('x')
_plot.SetYTitle('x^2')
_renderWindow.Render()
_iRenderer.Start()
Which is pretty ugly for the user to have to get to grips with, so I'm
trying to simplify the interface to the back end. I'm writing other
backends so that one only needs to change one line of code to then use
gnuplot or plplot as the renderer to generate the plot. Of course some
renderers can do things others can't, but that's another issue...

There are lots of easy things you could do without generating and exec-ing
python code per se. I'd love to know of other options. I like the idea of generating the code
one would have to write for a particular renderer so that if the user wanted
to, they could use the autogenerated code to form the basis of a specific
visualisation script they could then hack themselves. I also like it
because I can see how the pyvisi code maps directly to the back end.
How complex is a user session state? Not 100% sure. My intention is that this is just a simple matter of
translating the commands specified at the high level into the lower level
stuff in the background. It can handle plotting several different graphs in
a loop as the data used to generate the graph changes, and it can handle
interactive display of the graphics in vtk (a vtk feature really, via
OpenGL), but nothing too tricky there. exec() is handy for running
commands generated on the fly, however, it doesn't keep the state of the
interpreter and so any variables generated on a previous call are lost.
Also, exec ignores globals etc (at least AFAIK). What would be great
would be to run the generated commands within the same scope and state,
then I wouldn't need to worry about this separate thread/process idea. I
just don't know how to do that.
What modifies it?
What actions are possible? History? Undo? There's no history or undo, just processing of the script.
Is the data a static passive
resource to view, or partly generated or made accessible by prior actions
in a session? The data can be generated by prior actions in a given python script, but
basically pyvisi is set up to grab the data it's passed, and then work out
how best to render it. So in some sense the data is static and passive,
however can change at some later date, but one has to recall the setData()
method and then the render() method of the scene object to process any
changes that have occured in the data.
Single user or collaborative? Definitely single user.
Shared access to everything or
only to static data? Etc., etc. Preferably only shared access to static data, but I do some variable name
mangling to try and keep autogenerated variables separate to anything the
user has created themselves.
Some examples of user input and corresponding generated python might help,
with some idea of the kind of visualization aspects being controlled ;-)

With my initial post I didn't want to put in examples as it was already
quite long and I didn't want to put people off replying. btw: I really
appreciate your feedback; I usually work best bouncing ideas off people to
find a clear way to do something.

One of the main ideas of the module is to distill the common visualisation
tasks down to a simple set of commands, and then let the interface work out
how to actually implement that. One wants to be able to put titles, and
labels and axes on things, one wants to plot 3D surfaces of data, arrows of
vector fields, ellipses of tensor data, isosurfaces of 3D scalar data, and
be able to display this to the screen and save to file. Other visualisation
aspects to be controlled are such things as camera angles, number of
contours in a plot, different kinds of colourmaps, etc.

Just to try and be of some help, here's another example of plotting some
data and its output. This time it's plotting of a random vector field with
arrows in three dimensions. First the pyvisi code:

"""
Example of plotting a 3D vector field with pyvisi
"""

# set up some data to plot
from Numeric import *

dim = 10

# initialise the positions of the vectors
x = zeros((dim,dim), typecode=Float)
y = zeros((dim,dim), typecode=Float)
z = zeros((dim,dim), typecode=Float)

# initialise the vector displacements
# (I may need to rethink how this works in the interface)
dx = zeros((dim,dim), typecode=Float)
dy = zeros((dim,dim), typecode=Float)
dz = zeros((dim,dim), typecode=Float)

# set the positions randomly, and set the displacements to some smaller
# random number but of mean zero instead of distributed between 0 and 1
import random
random.seed()
for i in range(dim):
for j in range(dim):
x[i,j] = random.random()
y[i,j] = random.random()
z[i,j] = random.random()
dx[i,j] = (random.random()-0.5)/5.0
dy[i,j] = (random.random()-0.5)/5.0
dz[i,j] = (random.random()-0.5)/5.0

# example code for how a user would write a script in pyvisi
from pyvisi import * # base level visualisation stuff
# import the objects to render the scene using the specific renderer
#from pyvisi.renderers.gnuplot import * # gnuplot
from pyvisi.renderers.vtk import * # vtk
#from pyvisi.renderers.povray import * # povray

# define the scene object
# a Scene is a container for all of the kinds of things you want to put
# into your plot for instance, images, meshes, arrow/vector/quiver
# plots,
# contour plots, spheres etc.
scene = Scene()

# create a ArrowPlot3D object
plot = ArrowPlot3D(scene)

# add some helpful info to the plot
plot.title = 'Example 3D arrow/quiver/vector field plot'
plot.xlabel = 'x'
plot.ylabel = 'y'
plot.zlabel = 'z'

# assign some data to the plot
plot.setData(x, y, z, dx, dy, dz)

# render the scene to screen
scene.render(pause=True, interactive=True)

# save the scene out to file
plot.setData(x, y, z, dx, dy, dz) # have to do this because we've already
# render()ed the scene. This requirement
# will be removed in the future
scene.save(fname="arrowPlot3D.png", format=PngImage())
And now the generated vtk-python code:

import vtk
from Numeric import *
_renderer = vtk.vtkRenderer()
_renderWindow = vtk.vtkRenderWindow()
_renderWindow.AddRenderer(_renderer)
_renderWindow.SetSize(640,480)
_renderer.SetBackground(1,1,1)
# Renderer._initRendererModule
# ArrowPlot3D.__init__()
# ArrowPlot3D.setData()
_x = array([0.877228328994, 0.412795474983, 0.163560730946, 0.89780023349,
0.129244456665, 0.40180234598, 0.542135110974, 0.46894547296,
<snip>
_y = array([0.750790237727, 0.355648864325, 0.659517997105, 0.0381646378866,
<snip>
_z = array([0.26125343592, 0.941964065844, 0.510850248162, 0.131409524918,
<snip>
_dx = array([0.0474228215121, -0.0940162717243, -0.0926294230912,
<snip>
_dy = array([0.0853229942053, 0.0358464500795, -0.00461198953457,
<snip>
_dz = array([-0.0559212949124, 0.0827739088553, 0.0849306644324,
<snip>
_points = vtk.vtkPoints()
_points.SetNumberOfPoints(100)
_points.InsertPoint(0, 0.877228, 0.750790, 0.261253)
<snip>
_vectors = vtk.vtkFloatArray()
_vectors.SetNumberOfComponents(3)
_vectors.SetNumberOfTuples(100)
_vectors.SetName("vectors")
_vectors.InsertTuple3(0, 0.047423, 0.085323, -0.055921)
<snip>
_grid = vtk.vtkUnstructuredGrid()
_grid.SetPoints(_points)
_grid.GetPointData().AddArray(_vectors)
_grid.GetPointData().SetActiveVectors("vectors")
# Scene.render()
_lut = vtk.vtkLookupTable()
_refLut = vtk.vtkLookupTable()
_lut.Build()
_refLut.Build()
for _i in range(256):
_lut.SetTableValue(_i, _refLut.GetTableValue(255-_i))
_iRenderer = vtk.vtkRenderWindowInteractor()
_iRenderer.SetRenderWindow(_renderWindow)
# ArrowPlot3D.render()
_arrow = vtk.vtkArrowSource()
_maxNorm = _grid.GetPointData().GetVectors().GetMaxNorm()
_glyph = vtk.vtkGlyph3D()
_glyph.ScalingOn()
_glyph.SetScaleModeToScaleByVector()
_glyph.SetColorModeToColorByVector()
_glyph.SetScaleFactor(0.1/_maxNorm)
_glyph.SetInput(_grid)
_glyph.SetSource(_arrow.GetOutput())
_glyph.ClampingOff()
_stripper = vtk.vtkStripper()
_stripper.SetInput(_glyph.GetOutput())
_lut = vtk.vtkLookupTable()
_lut.Build()
_refLut = vtk.vtkLookupTable()
_refLut.Build()
for i in range(256):
_lut.SetTableValue(i, _refLut.GetTableValue(255-i))
_mapper = vtk.vtkPolyDataMapper()
_mapper.SetInput(_stripper.GetOutput())
_mapper.SetScalarRange(0, _maxNorm)
_actor = vtk.vtkActor()
_actor.SetMapper(_mapper)
_renderer.AddActor(_actor)
_font_size = 14
_textProp = vtk.vtkTextProperty()
_textProp.SetFontSize(_font_size)
_textProp.SetFontFamilyToArial()
_textProp.BoldOff()
_textProp.ItalicOff()
_textProp.ShadowOff()
_textProp.SetColor(0,0,0)
_titleMapper = vtk.vtkTextMapper()
_titleMapper.SetInput("Example 3D arrow/quiver/vector field plot")
_titleProp = _titleMapper.GetTextProperty()
_titleProp.ShallowCopy(_textProp)
_titleProp.SetJustificationToCentered()
_titleProp.SetVerticalJustificationToTop()
_titleProp.SetFontSize(18)
_titleActor = vtk.vtkTextActor()
_titleActor.SetMapper(_titleMapper)
_titleActor.GetPositionCoordinate().SetCoordinateS ystemToNormalizedDisplay()
_titleActor.GetPositionCoordinate().SetValue(0.5, 0.95)
_renderer.AddActor(_titleActor)
_axes = vtk.vtkCubeAxesActor2D()
_axes.SetCamera(_renderer.GetActiveCamera())
_axes.SetFlyModeToOuterEdges()
_axes.SetBounds(min(_x)-_maxNorm, max(_x)+_maxNorm, min(_y)-_maxNorm,
max(_y)+_maxNorm, min(_z)-_maxNorm, max(_z)+_maxNorm)
_axes.SetXLabel("x")
_axes.SetYLabel("y")
_axes.SetZLabel("z")
_axesProp = _axes.GetProperty()
_axesProp.SetColor(0,0,0)
_axesTitleProp = _axes.GetAxisTitleTextProperty()
_axesTitleProp.ShallowCopy(_textProp)
_axesLabelProp = _axes.GetAxisLabelTextProperty()
_axesLabelProp.ShallowCopy(_textProp)
_axesLabelProp.SetFontSize(8)
_renderer.AddActor(_axes)
_renderer.ResetCamera()
_renderer.SetBackground(1,1,1)
_renderWindow.Render()
_iRenderer.Start()

As you can see from the snipping, there's good reason for me to want to
share data objects around... I know it's ugly, but it got it working
quickly (I'm a scientist, not a proper software developer...) Hopefully you
can also see that the pyvisi script is a _lot_ simpler than what is required
to generate the visualisation.

Ok. I think that's enough for one post. Again, thanks heaps for your
reply, I do appreciate it, and hopefully now you can understand my question
a bit more clearly. I look forward to hearing from you soon.

Regards,

Paul

--
Paul Cochrane
Earth Systems Science Computational Centre
University of Queensland Brisbane Queensland 4072 Australia
E: cochrane at esscc dot uq dot edu dot au
Nov 3 '05 #5

P: n/a
On Thu, 3 Nov 2005, Paul Cochrane wrote:
On Wed, 02 Nov 2005 06:33:28 +0000, Bengt Richter wrote:
On Wed, 2 Nov 2005 06:08:22 +0000 (UTC), Paul Cochrane <co******@shake56.esscc.uq.edu.au> wrote:
I've got an application that I'm writing that autogenerates python
code which I then execute with exec(). I know that this is not the
best way to run things, and I'm not 100% sure as to what I really
should do. I've had a look through Programming Python and the Python
Cookbook, which have given me ideas, but nothing has gelled yet, so I
thought I'd put the question to the community. But first, let me be a
little more detailed in what I want to do:

Paul, this is a rather interesting problem. There are two aspects to it,
which i believe are probably separable: getting instructions from the
client to the server, and getting data back from the server to the client.
The former is more complex, i think, and what's attracted the attention so
far.

The first thing i'd say is that, while eval/exec is definitely a code
smell, that doesn't mean it's never the right solution. If you need to be
able to express complex things, python code might well be the best way to
do it, and the best way to evaluate python code is eval/exec.
It's a little hard to tell without knowing more about your user input
(command language?) syntax that is translated to or feeds the process
that "autogenerates python code".
It's basically just a command language I guess.


Hang on - the stuff that the user writes is what you're calling "pyvisi
code", is that right? That doesn't look like 'just a command language',
that looks like python, using a library you've written. Or is there
another language, the "just a command language", on top of that?

And what you call "vtk-python code" - this is python again, but using the
renderer's native library, right?

And you generate the vtk-python from the pyvisi-python by executing the
pyvisi-python, there being (pluggable renderer-specific) logic in the guts
of your pyvisi classes to emit the vtk-python code, right? You're not
parsing anything?
There are lots of easy things you could do without generating and exec-ing
python code per se.


I'd love to know of other options. I like the idea of generating the
code one would have to write for a particular renderer so that if the
user wanted to, they could use the autogenerated code to form the basis
of a specific visualisation script they could then hack themselves.


If you want vtk-python code as an intermediate, i think you're stuck with
eval/exec [1].
One of the main ideas of the module is to distill the common visualisation
tasks down to a simple set of commands, and then let the interface work out
how to actually implement that.


Okay. There's a classic design pattern called Interpreter that applies
here. This is one of the more complex patterns, and one that's rather
poorly explained in the Gang of Four book, so it's not well-known.

Basically, the idea is that you provide classes which make it possible for
a program to build structures encoding a series of instructions -
essentially, you define a language whose concrete syntax is objects, not
text - then you write code which takes such structures and carries out the
instructions encoded in them - an interpreter, in other words.

For example, here's a very simple example for doing basic arithmetic:

# the language

class expression(object):
pass

class constant(expression):
def __init__(self, value):
self.value = value

class unary(expression):
def __init__(self, op, arg):
self.op = op
self.arg = arg

class binary(expression):
def __init__(self, op, arg_l, arg_r):
self.op = op
self.arg_l = arg_l
self.arg_r = arg_r

# the interpreter

UNARY_OPS = {
"-": lambda x: -x,
"|": lambda x: abs(x) # apologies for abnormal syntax
}

BINARY_OPS = {
"+": lambda l, r: l + r,
"-": lambda l, r: l - r,
"*": lambda l, r: l * r,
"/": lambda l, r: l / r,
}

def evaluate(expr):
if isinstance(expr, constant):
return expr.value
elif isinstance(expr, unary):
op = UNARY_OPS[expr.op]
arg = evaluate(expr.arg)
return op(arg)
elif isinstance(expr, binary):
op = BINARY_OPS[expr.op]
arg_l = evaluate(expr.arg_l)
arg_r = evaluate(expr.arg_r)
return op(arg_l, arg_r)
else:
raise Exception, "unknown expression type: " + str(type(expr))

# a quick demo

expr = binary("-",
binary("*",
constant(2.0),
constant(3.0)),
unary("-",
binary("/",
constant(4.0),
constant(5.0))))

print evaluate(expr)

This is by no means a useful or well-designed bit of code, and there are
several things that could have been done differently (bare vs wrapped
constants, operations defined by a symbol vs expression subtypes for each
operation, etc), but i hope it gets the idea across - representing a
language using an object graph, which lets you write programs that can
speak that language.

Your code is already doing something a bit like this - you build scene
graphs, then call render on them to get them to do something. Instead of
that, you'd pass the whole scene to a renderer object, which would do the
rendering (directly, rather than by generating code). The point is that
the renderer could be in another process, provided you have a way to move
the scene graph from one process to another - the pickle module, for
example, or a custom serialisation format if you feel like reinventing the
wheel.

An approach like this has a natural solution to your second problem, too -
the evaluator function can return objects, which again can just be pickled
and sent over the network.

tom

[1] Okay, so there is a way to do this without ever actually creating
python code. You're not going to like this.

You need to apply the interpreter pattern to python itself. Well, a
simplified subset of it. Looking at your generated vtk-python code, you
basically do the following things:

- call methods with variables and literals as arguments
- throwing away the result
- or keeping it in a variable
- do for loops over ranges of integers

To make things a bit simpler, i'm going to add:

- getting attributes
- subscripting arrays
- return a value

You also need to do some arithmetic, i think; i leave that as an exercise
for the reader.

So we need a language like:

class statement(object):
pass

class invoke(statement):
def __init__(self, var, target, method, args):
self.var = var # name of variable for result; None to throw away
self.target = target
self.method = method
self.args = args

class get(statement):
def __init__(self, var, target, field):
self.var = var
self.target = target
self.field = field

class subscript(statement):
def __init__(self, var, target, index):
self.var = var
self.target = target
self.index = index

class forloop(statement):
def __init__(self, var, limit, body):
self.var = var
self.limit = limit
self.body = body # tuple of statements

def return_(statement):
def __init__(self, value):
self.value = value

With which we can write a script like:

vtk_script = [
invoke("_plot", "vtk", "vtkXYPlotActor", ()),
invoke("_renderer", "vtk", "vtkRenderer", ()),
invoke("_renderWindow", "vtk", "vtkRenderWindow", ()),
invoke(None, "_renderWindow", "AddRenderer", ("_renderer")),
invoke(None, "_renderWindow", "SetSize", (640, 480)),
invoke(None, "_renderer", "SetBackground", (1, 1, 1)),
invoke("_x", None, "array", ([0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0],)),
get("vtk_vtkDataArray", "vtk", "vtkDataArray"),
get("vtk_VTK_FLOAT", "vtk", "VTK_FLOAT"),
invoke("_xData", "vtk_vtkDataArray", "CreateDataArray", ("vtk_VTK_FLOAT",)),
invoke("_x_len", None, "len", ("_x",)),
invoke(None, "_xData", "SetNumberOfTuples", ("_x_len",)),
invoke("_y0", None, "array", ([0.0, 1.0, 4.0, 9.0, 16.0, 25.0, 36.0, 49.0, 64.0, 81.0],)),
invoke("_y0Data", "vtk_vtkDataArray", "CreateDataArray", ("vtk_VTK_FLOAT",)),
invoke("_y0_len", None, "len", ("_y0",)),
invoke(None, "_y0Data", "SetNumberOfTuples", ("_y0_len",)),
forloop("i", "_x_len", (
subscript("_x_i", "_x", "i"),
invoke(None, "_xData", "SetTuple1", ("i", "_x_i")),)),
# etc
]

Or rather, your pyvisi classes can generate structures like this, exactly
as they currently generate code.

Code to convert this into python source is trivial, so i'll gloss over
that. The interpreter looks like this:

def isiterable(x):
return hasattr(x, "__iter__")

def execute(script, vars=None):
if (vars == None):
vars = {}
# initialise variables with 'vtk' and anything else you need
def decode_arg(arg):
if (isinstance(arg, str)):
return vars[arg]
elif (isiterable(arg)):
return map(decode_arg, arg)
else:
return arg
for stmt in script:
if (isinstance(stmt, invoke)):
target = vars[stmt.target]
method = getattr(target, stmt.method)
args = decode_arg(stmt.args)
result = method(*args)
if (stmt.var != None):
vars[stmt.var] = result
elif (isinstance(stmt, get)):
target = vars[stmt.target]
vars[stmt.var] = getattr(target, stmt.field)
elif (isinstance(stmt, subscript)):
target = vars[stmt.target]
vars[stmt.var] = getattr(target, decode_arg(stmt.index))
elif (isinstance(stmt, forloop)):
var = stmt.var
limit = decode_arg(stmt.limit)
body = stmt.body
for i in range(limit):
vars[var] = i
execute(body, vars)
elif (isinstance(stmt, return_)):
return vars[stmt.value]
# nb won't work from inside a for loop!
# you can use an exception to handle returns properly

Note that i haven't tested this, i've just written it off the top of my
head, so it probably won't work, but maybe you get the idea.

To be honest, i'd go with exec.

--
Fitter, Happier, More Productive.
Nov 3 '05 #6

P: n/a
On Thu, 03 Nov 2005 14:23:53 +1000, Paul Cochrane <co******@esscc.uq.edu.au> wrote:
On Wed, 02 Nov 2005 06:33:28 +0000, Bengt Richter wrote:
On Wed, 2 Nov 2005 06:08:22 +0000 (UTC), Paul Cochrane <co******@shake56.esscc.uq.edu.au> wrote:
Hi all,

I've got an application that I'm writing that autogenerates python code
which I then execute with exec(). I know that this is not the best way to
run things, and I'm not 100% sure as to what I really should do. I've had a
look through Programming Python and the Python Cookbook, which have given me
ideas, but nothing has gelled yet, so I thought I'd put the question to the
community. But first, let me be a little more detailed in what I want to
do:


Bengt,

Thanks for your reply!
It's a little hard to tell without knowing more about your
user input (command language?) syntax that is translated to
or feeds the process that "autogenerates python code".

Ok, I'll try and clarify things as much as I can.

[...snip great reply...]

I owe you another reply, but I started and I couldn't spend the time to
do it justice. But in the meanwhile, I would suggest thinking about how
the MVC (model-view-controller) concept might help you factor things.
ISTM you are already part way there. See

http://en.wikipedia.org/wiki/MVC

for some infos. To that I'd add that if you want a top level interactive visualization tool,
you might want to look at it like a kind of IDE, where you might want a few workspace/project kind
of top level commands to help manage what you would otherwise do by way of manually creating
directory subtrees for various things etc. Anyway, if you want to use template code and edit
it and then compile and run it, you could select templates and make working copies automatically
and invoke some favorite editor on it from the top level command interpreter. Having these
pieces consistently arranged in projects/workspaces and shared template spaces etc. would
make it a single command to create a fresh space and not worry about colliding with something
you did just to show someone a little demo, etc. This is not the central topic, but good useability
is nice ;-)

That way if you just wanted to re-run something, you'd just select it and skip calling the
editor. Or if a step was to generate data, you could either create the data source program
by several steps or possibly just go on to define or invoke a visualization step with
a particular renderer and/or output, knowing that the data source was already set up in a
standard way. You could also consider borrowing unix piping/redirection concepts
for some command syntax, for composition of standard interface actions (not to mention
invoking the real thing in a subprocess when appropriate). Just free-associating here ;-)
Anyway, gotta go for now, sorry.

Regards,
Bengt Richter
Nov 7 '05 #7

P: n/a
On Thu, 03 Nov 2005 19:56:48 +0000, Tom Anderson wrote:
On Thu, 3 Nov 2005, Paul Cochrane wrote:
On Wed, 02 Nov 2005 06:33:28 +0000, Bengt Richter wrote:
On Wed, 2 Nov 2005 06:08:22 +0000 (UTC), Paul Cochrane <co******@shake56.esscc.uq.edu.au> wrote:

I've got an application that I'm writing that autogenerates python
code which I then execute with exec(). I know that this is not the
best way to run things, and I'm not 100% sure as to what I really
should do. I've had a look through Programming Python and the Python
Cookbook, which have given me ideas, but nothing has gelled yet, so I
thought I'd put the question to the community. But first, let me be a
little more detailed in what I want to do:
Paul, this is a rather interesting problem. There are two aspects to it,
which i believe are probably separable: getting instructions from the
client to the server, and getting data back from the server to the client.
The former is more complex, i think, and what's attracted the attention so
far. Tom, thanks heaps for your reply!
The first thing i'd say is that, while eval/exec is definitely a code
smell, that doesn't mean it's never the right solution. If you need to be
able to express complex things, python code might well be the best way to
do it, and the best way to evaluate python code is eval/exec. After looking at the problem a bit more I've come up with a simpler
solution to the problem I initially posted; and it involves exec. I've
realised that what I've been doing wrong is to compile the code first
before I exec it. If I just exec the generated code (within a predefined
namespace) then pyvisi does all of the things I want it to do without
needing separate processes for generating and rendering the code. This
has also solved my data passing problem. I now just need to pass a
reference to the data into the namespace where I'm running the
generated code and it all works really nicely. Basically the problem had
been staring me in the face for ages and I just hadn't seen it. (duh!)
It's a little hard to tell without knowing more about your user input
(command language?) syntax that is translated to or feeds the process
that "autogenerates python code".


It's basically just a command language I guess.


Hang on - the stuff that the user writes is what you're calling "pyvisi
code", is that right? That doesn't look like 'just a command language',
that looks like python, using a library you've written. Or is there
another language, the "just a command language", on top of that?

You're right, it isn't a command language (I looked up the cmd module in
Python in a Nutshell and then realised that a large part of my reply was
in the "I guess" part). It is just python code that is being run.
And what you call "vtk-python code" - this is python again, but using
the renderer's native library, right? That's correct.
And you generate the vtk-python from the pyvisi-python by executing the
pyvisi-python, there being (pluggable renderer-specific) logic in the
guts of your pyvisi classes to emit the vtk-python code, right? You're
not parsing anything? No parsing going on. Just a translation of high-level ideas into the low
level of the renderer underneath.
There are lots of easy things you could do without generating and
exec-ing python code per se.
I'd love to know of other options. I like the idea of generating the
code one would have to write for a particular renderer so that if the
user wanted to, they could use the autogenerated code to form the basis
of a specific visualisation script they could then hack themselves.


If you want vtk-python code as an intermediate, i think you're stuck
with eval/exec [1].

So do I. And as I said above, it seems to have simplified my problem
significantly. I've also managed to speed the code up as well!

Thanks heaps for the rest of your comments and suggestions. They've been
really helpful for me to see alternative and probably cleaner ways of
doing what I want to do.
To be honest, i'd go with exec.

:-)

Again, thanks for your help, I really do appreciate it.

Paul
Nov 7 '05 #8

P: n/a
On Mon, 07 Nov 2005 00:01:29 +0000, Bengt Richter wrote:
On Thu, 03 Nov 2005 14:23:53 +1000, Paul Cochrane <co******@esscc.uq.edu.au> wrote:
On Wed, 02 Nov 2005 06:33:28 +0000, Bengt Richter wrote:
On Wed, 2 Nov 2005 06:08:22 +0000 (UTC), Paul Cochrane <co******@shake56.esscc.uq.edu.au> wrote:

Hi all,

I've got an application that I'm writing that autogenerates python code
which I then execute with exec(). I know that this is not the best way to
run things, and I'm not 100% sure as to what I really should do. I've had a
look through Programming Python and the Python Cookbook, which have given me
ideas, but nothing has gelled yet, so I thought I'd put the question to the
community. But first, let me be a little more detailed in what I want to
do:

Bengt,

Thanks for your reply!
It's a little hard to tell without knowing more about your
user input (command language?) syntax that is translated to
or feeds the process that "autogenerates python code".

Ok, I'll try and clarify things as much as I can.

[...snip great reply...]

Thanks :-)
I owe you another reply, but I started and I couldn't spend the time to
do it justice. But in the meanwhile, I would suggest thinking about how
the MVC (model-view-controller) concept might help you factor things.
ISTM you are already part way there. See
http://en.wikipedia.org/wiki/MVC Thanks for the tip.
for some infos. To that I'd add that if you want a top level interactive visualization tool,
you might want to look at it like a kind of IDE, where you might want a few workspace/project kind
of top level commands to help manage what you would otherwise do by way of manually creating
directory subtrees for various things etc. Anyway, if you want to use template code and edit
it and then compile and run it, you could select templates and make working copies automatically
and invoke some favorite editor on it from the top level command interpreter. Having these
pieces consistently arranged in projects/workspaces and shared template spaces etc. would
make it a single command to create a fresh space and not worry about colliding with something
you did just to show someone a little demo, etc. This is not the central topic, but good useability
is nice ;-) I think this is probably a bit more complex than I really want. All I
want to do is to take the pain out of scripting visualisation of large and
complex data sets. My code will eventually run inside a batch job, or
behind a web-based GRID interface thing (still being developed by other
people in my group). The interactive part of vtk is just a
handy side effect; allowing users to rotate objects within a window and
see what's on the other side etc is really nice, but not necessarily the
main aim. Nevertheless, I've found out what I've been doing wrong all
along: I've been compiling the code objects before exec-ing them. By just
running exec on the generated code I can run the code generation and
rendering within the one process (since not compiling beforehand allows me
to maintain state), and I can pass the data around as the objects
themselves and not this convoluted convert to string, convert back to
number process I had going. It's also now a lot faster (which is always a
bonus)!
That way if you just wanted to re-run something, you'd just select it and skip calling the
editor. Or if a step was to generate data, you could either create the data source program
by several steps or possibly just go on to define or invoke a visualization step with
a particular renderer and/or output, knowing that the data source was already set up in a
standard way. You could also consider borrowing unix piping/redirection concepts
for some command syntax, for composition of standard interface actions (not to mention
invoking the real thing in a subprocess when appropriate). Just free-associating here ;-) These are great ideas, but probably a bit more than I can handle atm :-)
Anyway, gotta go for now, sorry.

No worries! I really appreciate your feedback.

Paul
Nov 7 '05 #9

This discussion thread is closed

Replies have been disabled for this discussion.