Embedded Python interpreter and sockets

[wahn]

Hi,

Here is a problem I came across and need some help with. I developed a
little Python script with some classes which runs standalone and
communicates with a database via sockets. So far everything works fine.
I also successfully embedded the Python interpreter into a plugin for a
commercial 3D package (in this case Houdini - http://www.sidefx.com/ -
using the HDK). I can use the interpreter to run Python commands and
get values back and forth between Python and C++. Anyway, I can also
successfully import SOME modules, but unfortunately not ALL. One
example would be the socket module which works fine in the standalone
version of my Python script but NOT from the embedded interpreter:

Traceback (most recent call last):
File "<string>", line 40, in ?
File "/swdevl/jwalter/Houdini/ROPS/HoudiniToMayaBake/damn.py", line
10, in ?
import socket
File
"/job/SYSTEMS/3PS/python/2.4.1/linux_intel//lib/python2.4/socket.py",
line 45, in ?
import _socket
ImportError:
/job/SYSTEMS/3PS/python/2.4.1/linux_intel/lib/python2.4/lib-dynload/_socket.so:
undefined symbol: PyObject_GenericGetAttr

Other modules which get imported before just work fine (e.g. re and
string) ...

The Python library gets linked into the plugin DSO as a static library
.... Any ideas what's going wrong and why?

Cheers,

Jan

[chris.atlee]

Hi Jan,

I believe the problem lies with how Houdini uses dlopen() to open your
plugin. It uses RTLD_LOCAL to load your plugin, which means that all
your plugin's symbols (including the python symbols) are private to
that library. Subsequent dlopen() calls, including those made by the
python library, won't be able to see any of those symbols. This
problem isn't unique to Python, or Houdini, it's just a matter of how
dlopen() is being used.

One thing that may work (I just tried here, and it looks like it
does...but it's late on a Friday night, so I may have overlooked
something) is creating a wrapper plugin. Say you have your plugin
called myplugin.so which links aginst libpython. Create a new plugin
called myplugin_wrapper.so which exports the same required symbols.
myplugin_wrapper will need to dlopen("myplugin.so", RTLD_GLOBAL), and
then call the real methods inside myplugin.so. Since myplugin.so is
the one linked against libpython, and it is being loaded with
RTLD_GLOBAL, then the python symbols should be available to other
shared libraries loaded at a future time.

Let me know if this works out! If you need any help, head on over to
http://www.sidefx.com/forum.

Cheers,
Chris

[chris.atlee]

Hi Jan,

I believe the problem lies with how Houdini uses dlopen() to open your
plugin. It uses RTLD_LOCAL to load your plugin, which means that all
your plugin's symbols (including the python symbols) are private to
that library. Subsequent dlopen() calls, including those made by the
python library, won't be able to see any of those symbols. This
problem isn't unique to Python, or Houdini, it's just a matter of how
dlopen() is being used.

One thing that may work (I just tried here, and it looks like it
does...but it's late on a Friday night, so I may have overlooked
something) is creating a wrapper plugin. Say you have your plugin
called myplugin.so which links aginst libpython. Create a new plugin
called myplugin_wrapper.so which exports the same required symbols.
myplugin_wrapper will need to dlopen("myplugin.so", RTLD_GLOBAL), and
then call the real methods inside myplugin.so. Since myplugin.so is
the one linked against libpython, and it is being loaded with
RTLD_GLOBAL, then the python symbols should be available to other
shared libraries loaded at a future time.

Let me know if this works out! If you need any help, head on over to
http://www.sidefx.com/forum.

Cheers,
Chris

[wahn]

Hi Chris,

Thanks for your help. I'll try that ...

Cheers,

Jan

[wahn]

Hey Chris,

I fixed the problem in another way (don't ask me why that works). One
detail I didn't talk about is that I use the Boost.Python library. So I
just made sure that I load the socket module before I import my own
Python script (using that socket module):

....
object
main_module((handle<>(borrowed(PyImport_AddModule( "__main__")))));
object
socket_module((handle<>(borrowed(PyImport_AddModul e("socket")))));
object main_namespace = main_module.attr("__dict__");
handle<>
result((allow_null(PyRun_String("...",
Py_file_input,
main_namespace.ptr(),
main_namespace.ptr()))));
....

Maybe someone can shed some light on what's going on here but for the
moment I'm happy that it works at all !!!

Cheers,

Jan

[wahn]

Hi,

actually that didn't solve the problem. As soon as you do something
with the socket module it fails. Well, the solution I came up with is
simply link the ../_socket.so into my Houdini plugin DSO which is ugly
but solves the problem for the moment ...

Happy hacking,

Jan