Embedding Python in Windows tutorial?

[Roose]

With some googling I have found these resources:

http://docs.python.org/ext/win-dlls.html
http://www.python.org/doc/faq/windows.html

I have a large Win32/MFC/C/C++ application that has an embedded scripting
language (a very limited one). I would like to rip it out and replace it
with Python. I am thinking that this would be relatively simple since the
scripting language is a very small interface between the UI and the
engine -- there would only be a handful functions I would need to wrap.

However in googling I haven't been able to come up with a completely
straightforward way to do it. I see all these caveats mentioned casually,
and not explained, which kind of scares me.

Examples:

1. I heard Python23.dll is built with MS VC++ 6. I am using .NET 2003 (VC++
7.1). Is this a problem? I wouldn't think so since I am not statically
linking... but someone said the C runtime is different between the two?
When would that be a problem?

2. Any sample VS.NET projects / boilerplate DLL code that loads
Python23.dll into the? Sure this is probably pretty simple -- add an
include paths, add a DLL dependency... but I'm sure there is at least one
gotcha and if I can avoid repeating other's mistakes, I'd prefer that.

3. How does C++ change the story? I assume at the least I have to extern
"C" any functions that are called by Python, right? What else?

4. The program in question has no installer -- it is an internal tool, and
people simply download the binaries .exe/.dll's/etc. from a source control
system. Does that present any problems? Is all they need is Python23.dll
in their system dir, which is installed if you install Python? But I am
going to need to write some Python wrappers for functions... does this mean
I need to use distutils and all that? One thing I didn't find clear is that
distutils will "install" a module on your own machine -- but what do you
need to do to get it onto other's machines? (e.g. non-engineers who do not
have compilers)

It seems like what I am doing is pretty much the "canonical" embedding
task -- scripting an engine of C/C++ code and also needing to access this
scripting through the GUI (i.e. so it is BOTH extending and embedding), so I
would like to know if there are some more complete Windows-specific
resources out there.

Also, any general advice on the process would be interesting. Out of all
the people who have done some embedding -- was it worth it? Were there a
lot of headaches? One of my concerns now is that it will decrease the
debuggability of the application. In a pure C/C++ app in VS.NET, it is
pretty simple to debug. But now I will have a layer of Python in between,
which could make it a big pain. In general this is kind of an "extra"
project at work that I don't want to spend a whole lot of time on. (It will
help "sell" python to a large group of very capable engineers, if that
motivates anyone to help me. : ) )

Thanks in advance for any help.

Roose

[Daniel Dittmar]

Roose wrote:

1. I heard Python23.dll is built with MS VC++ 6. I am using .NET 2003 (VC++
7.1). Is this a problem? I wouldn't think so since I am not statically
linking... but someone said the C runtime is different between the two?
When would that be a problem?
- extensions compiled with .Net 2003 seem to run fine with Python 2.3
- I've heard that Python 2.4 will use .Net 2003 anyway
2. Any sample VS.NET projects / boilerplate DLL code that loads
Python23.dll into the? Sure this is probably pretty simple -- add an
include paths, add a DLL dependency... but I'm sure there is at least one
gotcha and if I can avoid repeating other's mistakes, I'd prefer that.
If you use the normal Python API, you link with the import lib
python23.lib, which will load python23.dll. If you put python23.dll into
the same directory as your application, everything should be fine.
4. The program in question has no installer -- it is an internal tool, and
people simply download the binaries .exe/.dll's/etc. from a source control
system. Does that present any problems? Is all they need is Python23.dll
in their system dir, which is installed if you install Python? But I am
I suggest that you distribute everything Python together with you
application. I found that switching to a new Python version is a hassle
when you have to tell everyone to do the upgrade.

- distribute python23.dll in the same directory as your application
- distribute the Python standard lib as a .zip archive
-- use the script compileall.py to generate the .pyc files
for the Python lib
-- use the -d and -f options to set the source directory
in the .pyc files. It is irritating if you get a Python traceback
and the filenames look plausible, but don't exist. Something like
"PYTHONROOT/os.py" makes it clear that the python files comes from
the distribution.
-- zip all the .pyc files into an archive (.py are optional)
-- add that .zip file to your distribution
-- add <python>/DLLs/zlib.pyd to your distribution (needed to import
modules from .zip files)
-- in your application: add that .zip file to the module search path,
either by changing the environment variable PYTHONPATH or by
manipulation sys.path through the Python API
-- you can start simple by packing only the .py files. This will work,
but it will be a bit slower as Python has to compile the files
for every run of your application

If you distribute Python together with your application, you should
occasionally test that you have everything included:
in the registry, rename
HKEY_LOCAL_MACHINE/Software/Python/PythonCore/2.3. That way, your
embedded Python won't be able to fall back on your installed Python.
going to need to write some Python wrappers for functions... does this mean
I need to use distutils and all that? One thing I didn't find clear is that
distutils will "install" a module on your own machine -- but what do you
need to do to get it onto other's machines? (e.g. non-engineers who do not
have compilers)
Your wrapper functions are part of your application. You make them known
to the Python runtime during your application, so there is no need to
install them via distutils.
One of my concerns now is that it will decrease the
debuggability of the application. In a pure C/C++ app in VS.NET, it is
pretty simple to debug. But now I will have a layer of Python in between,
which could make it a big pain. In general this is kind of an "extra"

You certainly shouldn't try to debug through the Python code. Set
breakpoints on all your extension functions. Once you reached Python,
use 'go' to go right through it.

You should also have the ability to report all exceptions in scripts
complete with backtraces. This will help you now with debugging and
later your customers with their own scripts.

Daniel

[Roose]

Thanks a lot for the answers -- they were quite helpful. Anyone else have
any links?

I still haven't gotten around to doing this, I might put it off since it
seems like more work than I would think... as I said I thought this was the
"canonical" thing to do and it would be pretty much a cut and paste job from
examples.

Does anyone else work with Python and C/C++ regularly? I thought there
would be more experts here... or are people not using Windows?

Not to be rude, but it is a little disheartening that everyone jumps all
over that "warehouse" thread rather than something a little more mundane but
utilitarian... : )

Roose

"Daniel Dittmar" <da************@sap.corp> wrote in message
news:cl**********@news.sap-ag.de...

Roose wrote:

1. I heard Python23.dll is built with MS VC++ 6. I am using .NET 2003 (VC++ 7.1). Is this a problem? I wouldn't think so since I am not statically
linking... but someone said the C runtime is different between the two?
When would that be a problem?

- extensions compiled with .Net 2003 seem to run fine with Python 2.3
- I've heard that Python 2.4 will use .Net 2003 anyway

2. Any sample VS.NET projects / boilerplate DLL code that loads
Python23.dll into the? Sure this is probably pretty simple -- add an
include paths, add a DLL dependency... but I'm sure there is at least one gotcha and if I can avoid repeating other's mistakes, I'd prefer that.

If you use the normal Python API, you link with the import lib
python23.lib, which will load python23.dll. If you put python23.dll into
the same directory as your application, everything should be fine.

4. The program in question has no installer -- it is an internal tool, and people simply download the binaries .exe/.dll's/etc. from a source control system. Does that present any problems? Is all they need is Python23.dll in their system dir, which is installed if you install Python? But I am

I suggest that you distribute everything Python together with you
application. I found that switching to a new Python version is a hassle
when you have to tell everyone to do the upgrade.

- distribute python23.dll in the same directory as your application
- distribute the Python standard lib as a .zip archive
-- use the script compileall.py to generate the .pyc files
for the Python lib
-- use the -d and -f options to set the source directory
in the .pyc files. It is irritating if you get a Python traceback
and the filenames look plausible, but don't exist. Something like
"PYTHONROOT/os.py" makes it clear that the python files comes from
the distribution.
-- zip all the .pyc files into an archive (.py are optional)
-- add that .zip file to your distribution
-- add <python>/DLLs/zlib.pyd to your distribution (needed to import
modules from .zip files)
-- in your application: add that .zip file to the module search path,
either by changing the environment variable PYTHONPATH or by
manipulation sys.path through the Python API
-- you can start simple by packing only the .py files. This will work,
but it will be a bit slower as Python has to compile the files
for every run of your application

If you distribute Python together with your application, you should
occasionally test that you have everything included:
in the registry, rename
HKEY_LOCAL_MACHINE/Software/Python/PythonCore/2.3. That way, your
embedded Python won't be able to fall back on your installed Python.

going to need to write some Python wrappers for functions... does this mean I need to use distutils and all that? One thing I didn't find clear is that distutils will "install" a module on your own machine -- but what do you
need to do to get it onto other's machines? (e.g. non-engineers who do not have compilers)

Your wrapper functions are part of your application. You make them known
to the Python runtime during your application, so there is no need to
install them via distutils.

> One of my concerns now is that it will decrease the
debuggability of the application. In a pure C/C++ app in VS.NET, it is
pretty simple to debug. But now I will have a layer of Python in between, which could make it a big pain. In general this is kind of an "extra"

You certainly shouldn't try to debug through the Python code. Set
breakpoints on all your extension functions. Once you reached Python,
use 'go' to go right through it.

You should also have the ability to report all exceptions in scripts
complete with backtraces. This will help you now with debugging and
later your customers with their own scripts.

Daniel