Looking for advice: supporting multiple embedded interpreters

[Paul Miller]

Some background first - we have some software that embeds a Python
interpreter into a host application. Scripts are loaded dynamically and
used. But we want to support the ability to edit scripts while the app is
running. This means we have to "unload" the script and cause a reload.
Normal module reloading tricks don't work because if you try to reimport a
script that imports another script, and if the nested script is changed, it
might not be reloaded itself. Also, we have 2 parts of the software, each
using Python for its own set of scripts. We don't want scripts loaded into
each parts to "see" scripts loaded into the other part. Ideally, we would
like multiple "instances" of a Python interpreter to manage this.

So, for Python 2.2, I came up with a system that works. When we need a new
self-contained Python interpreter, I use Py_NewInterpreter(), swap it in
using PyThreadState_Swap, load my built in modules, and swap it back out.
When I need to run some code in that interpreter, I swap it back in, load
the module I need, call methods in it, and swap it back out. When I'm done
with the interpreter, I swap it back in and call Py_EndInterpreter.

When I want to force a reload of all the script code in a given
interpreter, I just delete the interpreter, create a new one, and load the
scripts into that one. This has worked flawlessly. And each "part" of my
application can use a different interpreter without modules and globals in
each one interfering with the other.

Now, with Python 2.3, this code doesn't seem to work anymore. Someone told
me it is likely because of the extensive rewrite of GUSI or whatnot. It is
important to note that I'm NOT really doing any threading. I just was
self-contained interpreters for the above reasons.

What I am wondering is if there a reliable method in 2.3 that does what I
need?

It has recently come to my attention that Lutz Paelike is in exactly the
same situation I am in, so I don't think this is a fringe concept.

[Josiah Carlson]

> Some background first - we have some software that embeds a Python

interpreter into a host application. Scripts are loaded dynamically and
used. But we want to support the ability to edit scripts while the app is
running. This means we have to "unload" the script and cause a reload.
Normal module reloading tricks don't work because if you try to reimport a
script that imports another script, and if the nested script is changed, it
might not be reloaded itself. Also, we have 2 parts of the software, each
using Python for its own set of scripts. We don't want scripts loaded into
each parts to "see" scripts loaded into the other part. Ideally, we would
like multiple "instances" of a Python interpreter to manage this.

So, for Python 2.2, I came up with a system that works. When we need a new
self-contained Python interpreter, I use Py_NewInterpreter(), swap it in
using PyThreadState_Swap, load my built in modules, and swap it back out.
When I need to run some code in that interpreter, I swap it back in, load
the module I need, call methods in it, and swap it back out. When I'm done
with the interpreter, I swap it back in and call Py_EndInterpreter.

When I want to force a reload of all the script code in a given
interpreter, I just delete the interpreter, create a new one, and load the
scripts into that one. This has worked flawlessly. And each "part" of my
application can use a different interpreter without modules and globals in
each one interfering with the other.

Now, with Python 2.3, this code doesn't seem to work anymore. Someone told
me it is likely because of the extensive rewrite of GUSI or whatnot. It is
important to note that I'm NOT really doing any threading. I just was
self-contained interpreters for the above reasons.

What I am wondering is if there a reliable method in 2.3 that does what I
need?

It has recently come to my attention that Lutz Paelike is in exactly the
same situation I am in, so I don't think this is a fringe concept.

My advice: use the 'reload' builtin to reload your python modules.

- Josiah

[David Bolen]

Paul Miller <pa**@fxtech.com> writes:

What I am wondering is if there a reliable method in 2.3 that does
what I need?

It has recently come to my attention that Lutz Paelike is in exactly
the same situation I am in, so I don't think this is a fringe concept.

I can't address why it doesn't work in 2.3, but just a question - have
you thought of not using independent interpreter states, but just
tracking the contents of sys.modules and clearing out any new modules
at reload time? That would force even nested imports to be reloaded.
You can find an example of doing this in the unittestgui.py module,
for example, that is part of PyUnit, as it uses this approach to
ensure that all modules under test are reloaded at the start of
execution of a test suite.

I expect that you might be able to get even fancier by installing a
custom import hook, but just flushing sys.modules is pretty simple and
should work in any Python release.

Of course, it won't deal with stray references you may still have
around to the old module or module objects, but since the new
interpreter approach definitely can't be exhibiting that behavior
anyway, I expect you aren't using any older objects after such a
reload.

-- David

[Paul Miller]

I can't address why it doesn't work in 2.3, but just a question - have
you thought of not using independent interpreter states, but just
tracking the contents of sys.modules and clearing out any new modules
at reload time? That would force even nested imports to be reloaded.

Hmm - although not as "clean" as having multiple interpreters (since
globals would still be visible between modules), it WOULD get around the
more pervasive problem, which was the reloading. I'll take a look at this.

THANKS!

[Paul Miller]

I can't address why it doesn't work in 2.3, but just a question - have
you thought of not using independent interpreter states, but just
tracking the contents of sys.modules and clearing out any new modules
at reload time? That would force even nested imports to be reloaded.
You can find an example of doing this in the unittestgui.py module,
for example, that is part of PyUnit, as it uses this approach to
ensure that all modules under test are reloaded at the start of
execution of a test suite.

At first I thought this would work, but then I remembered that I have other
instances of interpreters and pointers to modules, objects, and functions
pointing into them. One of my issues is I do not want to affect other
loaded modules, and if I blow away the global module table, then I would
mess up all the other instances of scripts that I want to leave alone.