Python, COM Servers, and Multi-Threading

[Carl Waldbieser]

I have been considering using Python and the Reportlab library for
generating PDF reports for the back-end of a web based application. The
application runs most of its background tasks on a dedicated server that is
Windows based (Win2K or Win2k3). The program that launches the tasks
requires a COM-based interface, so I wrote a Python COM server using Mark
Hammond's PythonCom libraries and import and run the reporlab modules from
there.

I had been reading up on Python and it's handling of the multiple threads,
specifically the Global Interpreter Lock (GIL). I got to wondering if a
multi-processor machine machine would be able to take advantage of its
extra processing power using this setup. I am guessing that the GIL is
global with respect to each instance of a running Python interpreter, so if
say 4 interpreters were running, a 4 processor machine would be able to
take advantage of this. However, I am not quite sure how launching my
reports via COM behaves-- if I launched 4 reports this way, would that be
like launching 4 seperate instances of the Python interpreter, or would it
be just a single instance, and therefore run into the limitations of the
GIL? If so, can anybody offer suggestions as to a design that would be
better able to take advantage of a multi-processor machine?

Thanks,
Carl Waldbieser

[Steve Holden]

Carl Waldbieser wrote:

I have been considering using Python and the Reportlab library for
generating PDF reports for the back-end of a web based application. The
application runs most of its background tasks on a dedicated server that is
Windows based (Win2K or Win2k3). The program that launches the tasks
requires a COM-based interface, so I wrote a Python COM server using Mark
Hammond's PythonCom libraries and import and run the reporlab modules from
there.
Have you selected a specific threading model for your COM server? This
will affect how hardware resource are allocated as well as how sharing
takes place between different uses.
I had been reading up on Python and it's handling of the multiple threads,
specifically the Global Interpreter Lock (GIL). I got to wondering if a
multi-processor machine machine would be able to take advantage of its
extra processing power using this setup. I am guessing that the GIL is
global with respect to each instance of a running Python interpreter, so if
say 4 interpreters were running, a 4 processor machine would be able to
take advantage of this. However, I am not quite sure how launching my
reports via COM behaves-- if I launched 4 reports this way, would that be
like launching 4 seperate instances of the Python interpreter, or would it
be just a single instance, and therefore run into the limitations of the
GIL? If so, can anybody offer suggestions as to a design that would be
better able to take advantage of a multi-processor machine?

I am certainly no expert on COM threading models, but I would suspect
that if you specify you want your COM server to be in-process then you
would get a per-process interpreter.

You also need to be aware that if you trigger this stuff with IIS then
the web server itself performs thread pooling and (I believe) allocates
only one process per "application", so this may defeat your desire to
use all four processors concurrently.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/