Some questions embedding Pythong in C (callback for variable assignment in the dictionary?)

[jordi]

Hi,

I'm starting to use Python embedded in a C program. I'm using Python
to execute several scripts using as a variables information retrieved
for several multithread "agents" written in C. The method is:

1- Create a thread for each agent (I use a pool thread, every agent
run in a different thread)
2- Precompile a different script for each agent.
3- Make the agent retrieve its data
4- Convert this data to Python variables
5- Execute a Python script.

The steps 3 to 5 are executed again and again (there is a timer that
launch the execution). As it's a multithreaded program my code is
something like this:

- In each thread initialization:

PyEval_AcquireLock();
interp = Py_NewInterpreter();
PyThreadState_Swap(interp);

globals = PyDict_New();
PyDict_SetItemString(globals, "__builtins__", PyEval_GetBuiltins());
pCode = Py_CompileString(ScriptPython, "testPython", Py_file_input
);

(This "ScriptPython" is different depending on the type of agent)

- Then, every time the C "agent" retrieves its data:

PyEval_AcquireThread(interp);
/* Here I have to modify the PyDictionary, globals or locals */
result = PyEval_EvalCode((PyCodeObject *)pCode, globals,
0/*locals*/);
PyEval_ReleaseThread(interp);

I have some questions:

1.- Is there any way to make the embedded Python interpret make a
callback to the C program asking for variables when they are not
defined in the Dictionary ? That is, implementing an "variable
on-demand" method. I have a problem because an "agent" can have
hundreds or thousands variables and maybe a script only uses a small
number of them. Inserting all the variables every time can be very
time consuming. I think Lua language can do that using "fallbacks"
or "tag methods" but I don't know if Python has something like that.

2.- What is the best way to use Python embedded in a C program?.
Instead of creating a new interpreter (Py_NewInterpreter();) I tried
using only a Python interpreter and creating a different state for
each thread:

interp = PyThreadState_New(PyMainThreadState->interp);

This solution was much worse than creating different Interpreters. Is
it normal?

3.- Are there any optimization tricks to use in embedded Python to
improve performance?

4.- Have you used any other script language for embedding in a
multithreaded program? I'm looking for something portable, fast and
free for commercial use. I have tested Python and Lua and Lua seems
more suitable (and much faster) for simple embedding in a
multithreaded program.

Best regards,

Jordi