473,394 Members | 1,219 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,394 software developers and data experts.

Multithreaded COM server problem...

I'm writing a multithreaded COM server to manage a pool of hardware resources.
All objects are designed to be thread-safe, and I've set sys.coinit_flags to
COINIT_MULTITHREADED before importing pythoncom.

The problem is that all requests to the server seem to be serialized by COM. To
demonstrate the problem, I'm including a simple test server that exposes one
interface object, PyComThreads.Application, with a single method sleep(delay).
I'm also including 2 test programs -- test20.py uses the server to delay 20
seconds, and test1.py delays only one second. The server prints status messages
to the trace collector debugging tool when creating the Application object, and
at the beginning and end of the specified delay.
When I run the 20-second test program, then a couple seconds later run the
1-second test program, I had expected to see something like this:

Object 8087872 in thread 8160416 created
Object 8087872 in thread 8160416 delays 20 seconds
Object 8086008 in thread 8156272 created
Object 8086008 in thread 8156272 delays 1 seconds
Object 8086008 delay ends
Object 8087872 delay ends
Instead, I see:

Object 8087872 in thread 8160416 created
Object 8087872 in thread 8160416 delays 20 seconds
Object 8087872 delay ends
Object 8086008 in thread 8160416 created
Object 8086008 in thread 8160416 delays 1 seconds
Object 8086008 delay ends
Apparently the requests from both client applications are being serialized by
the COM interface.

I need each request (or at least each interface object) to run in its own
thread, or in a pool of threads, and haven't been able to figure out how to
accomplish this. Any suggestions would be appreciated.
Regards,
John

----------
PyComThreads.py:

import sys
import threading
from time import sleep
import win32traceutil

sys.coinit_flags = 0 # 0 == pythoncom.COINIT_MULTITHREADED # !!!!!
import pythoncom

class Application:
""" Test version of a Multi-threaded local server """

_reg_progid_ = 'PyComThreads.Application'
_reg_verprogid_ = 'PyComThreads.Application.100'
_reg_clsctx_ = pythoncom.CLSCTX_LOCAL_SERVER
_reg_clsid_ = '{56BEC27D-EDC4-43A0-AEB7-77E4A1381C0F}'

_public_methods_ = ['sleep']
_public_attrs_ = []
_readonly_attrs_ = []

def __init__(self):
print 'Object %s in thread %s created' % \
(id(self), id(threading.currentThread()))

def sleep(self, delay):
print 'Object %s in thread %s delays %s seconds' % \
(id(self), id(threading.currentThread()), delay)
sleep(delay)
print 'Object %s delay ends' % id(self)

# COM server registration, etc.
if __name__ == '__main__':
if hasattr(sys, 'argv'):
# If *no* command-line arguments, we were not invoked as a server.
# Assume the user wants us to self-register.
if len(sys.argv) == 1:
sys.argv.append('--register')

import win32com.server.register
win32com.server.register.UseCommandLine(Applicatio n, debug=0)

----------
test20.py:

from win32com.client import Dispatch
app=Dispatch('PyComThreads.Application')
app.sleep(20)

----------
test1.py:

from win32com.client import Dispatch
app=Dispatch('PyComThreads.Application')
app.sleep(1)

Jul 18 '05 #1
9 16220
Hi !
* Sorry for my bad english : french is very more easy, and nice ;-) *

I have try another way. I use a procedure for manage thread :

GlobalTacheID=[None]*16

def tache(ordre, numtache=1, nomtache=None, fonction=None):
global GlobalTacheID
import threading
if ordre=="Lance":
GlobalTacheID[numtache] =
threading.Thread(target=fonction,name=nomtache)
GlobalTacheID[numtache].start()
print(nomtache+' num.'+str(numtache))
elif ordre=="Liste":
return(string.join(threading.enumerate(),'\n'))
elif ordre=="Etat":
return(str(GlobalTacheID[numtache].isAlive()))

And, in my public_method, i call this procedure, with the parameters
"ordre=Lance" for run a thread, "ordre=Etat" for see his state (fonction=
the name of the method 'threaded').

And it's OK. I can run a thread from an Excel-COM-client. I can also run an
thread on a method caontained in a string gived by Excel.

Python is very smart-dynamic-agile language.

Bonne soirée
--
Michel Claveau
mél : http://cerbermail.com/?6J1TthIa8B
site : http://mclaveau.com
Jul 18 '05 #2
John Lull wrote:
I'm writing a multithreaded COM server to manage a pool of hardware resources.
All objects are designed to be thread-safe, and I've set sys.coinit_flags to
COINIT_MULTITHREADED before importing pythoncom.


Note that this flag is effeectively ignored for the server. Objects are
created in the same "apartment" as their creator. This, the code
implementing your object need not specify this, but the code *creating*
the object must. This is what determines the appartment.

COM threading rules are complex, but "Python Programming on Win32" (1/2
by me :) covers these rules in words I would have trouble finding again :)

Mark.

Jul 18 '05 #3
Mark Hammond <mh******@skippinet.com.au> wrote (with possible deletions):
John Lull wrote:
I'm writing a multithreaded COM server to manage a pool of hardware resources.
All objects are designed to be thread-safe, and I've set sys.coinit_flags to
COINIT_MULTITHREADED before importing pythoncom.


Note that this flag is effeectively ignored for the server. Objects are
created in the same "apartment" as their creator. This, the code
implementing your object need not specify this, but the code *creating*
the object must. This is what determines the appartment.

COM threading rules are complex, but "Python Programming on Win32" (1/2
by me :) covers these rules in words I would have trouble finding again :)


It covers threading rules for in-process servers pretty thoroughly.

Unfortunately, this one has to be a local server since it's providing shared
access to a pool of hardware devices from multiple distributed clients. I've
carefully reviewed chapters 5 & 12, and appendix D, and wasn't able to find
anything addressing threading models in the local server in detail. If I've
missed something, I'd be grateful for any additional hints.

Thanks.
Regards,
John

Jul 18 '05 #4
John Lull wrote:
Mark Hammond <mh******@skippinet.com.au> wrote (with possible deletions):

John Lull wrote:

I'm writing a multithreaded COM server to manage a pool of hardware resources.
All objects are designed to be thread-safe, and I've set sys.coinit_flags to
COINIT_MULTITHREADED before importing pythoncom.


Note that this flag is effeectively ignored for the server. Objects are
created in the same "apartment" as their creator. This, the code
implementing your object need not specify this, but the code *creating*
the object must. This is what determines the appartment.

COM threading rules are complex, but "Python Programming on Win32" (1/2
by me :) covers these rules in words I would have trouble finding again :)

It covers threading rules for in-process servers pretty thoroughly.

Unfortunately, this one has to be a local server since it's providing shared
access to a pool of hardware devices from multiple distributed clients. I've
carefully reviewed chapters 5 & 12, and appendix D, and wasn't able to find
anything addressing threading models in the local server in detail. If I've
missed something, I'd be grateful for any additional hints.


The problem is that your client code is not running a message loop. If
you change the loop of your client test code to something like:

for i in range(delay)*10:
time.sleep(0.1)
pythoncom.PumpWaitingMessages()

It works as you expect. A better choice would probably be
win32event.MsgWaitForMultipleObjects, but that depends on what your app
really does.

Mark.

Jul 18 '05 #5
John Lull wrote:
Mark Hammond <mh******@skippinet.com.au> wrote (with possible
deletions):

John Lull wrote:
...
Unfortunately, this one has to be a local server since it's providing shared
access to a pool of hardware devices from multiple distributed clients. I've
carefully reviewed chapters 5 & 12, and appendix D, and wasn't able to find
anything addressing threading models in the local server in detail. If I've
missed something, I'd be grateful for any additional hints.


The problem is that your client code is not running a message loop. If
you change the loop of your client test code to something like:

for i in range(delay)*10:
time.sleep(0.1)
pythoncom.PumpWaitingMessages()

It works as you expect. A better choice would probably be
win32event.MsgWaitForMultipleObjects, but that depends on what your app
really does.

Mark.

I presume you meant my server code.


Nope - I meant the client. The server is already running such a loop
thank to localserver.py, which is hosting the object.

The client code's main (and only) thread is blocked in a system call,
but it appears COM wants it to pump messages so the marshalling magic
happens. I can only speculate why COM needs this to happen in this
trivial case, but COM's rules do state this requirement.
This still leaves all calls to the
server running in a single thread, however. If I insert a call to
PumpWaitingMessages() in a short operation, and it happens to start a
long operation, the result of that short operation will be delayed
until the long operation completes. This will make my server unusable.
Make the change I suggested, and it works as you hope.
At first glance this seems to do what I want -- requests to the server
seem to run from a thread pool. However, I also get intermittent (but
frequest) startup errors, with a Microsoft Visual C++ Runtime Library
error dialog (during pythoncom.PumpMessages but before my server
module gets imported) reporting:
Runtime Error!
Program: c:\Apps\Python2.2\pythonw.exe
abnormal program termination


That sucks, and is almost certainly a thread-state error. If you have a
debug build of Python, it will offer to break into the debugger at this
point.

Mark.

Jul 18 '05 #6
Mark Hammond <mh******@skippinet.com.au> wrote (with possible
deletions):
John Lull wrote:
Mark Hammond <mh******@skippinet.com.au> wrote (with possible
deletions):

John Lull wrote:
...
Unfortunately, this one has to be a local server since it's providing shared
access to a pool of hardware devices from multiple distributed clients. I've
carefully reviewed chapters 5 & 12, and appendix D, and wasn't able to find
anything addressing threading models in the local server in detail. If I've
missed something, I'd be grateful for any additional hints.

The problem is that your client code is not running a message loop. If
you change the loop of your client test code to something like:

for i in range(delay)*10:
time.sleep(0.1)
pythoncom.PumpWaitingMessages()

It works as you expect. A better choice would probably be
win32event.MsgWaitForMultipleObjects, but that depends on what your app
really does.

Mark.

I presume you meant my server code.


Nope - I meant the client. The server is already running such a loop
thank to localserver.py, which is hosting the object.

The client code's main (and only) thread is blocked in a system call,
but it appears COM wants it to pump messages so the marshalling magic
happens. I can only speculate why COM needs this to happen in this
trivial case, but COM's rules do state this requirement.


Now I'm *really* confused.

Perhaps I need to clarify a bit. The sleep() method in my sample
server is a perhaps-too-simplified substitute for what the real server
is doing. It provides a variety of high-level operations on a piece of
hardware. Some of the operations take as long as several dozen
seconds, others take a millisecond or so. I need the client to block
waiting for completion of each operation, regardless of how long the
operation takes. I cannot break one of the long operations up into a
series of calls from the client -- it must be implemented as a single
call. My example would, perhaps, have been clearer if I'd named the
method someLongRunningMethod() instead of sleep().

I've tried doing roughly what you suggested inside my test client,
calling PumpWaitingMessages() both before and after each COM
operation. That still leaves me with the same basic problem -- inside
the server, all of my COM objects are created on the server's main
thread, instead of on a separate thread for each client. That leaves
all COM operations serialized through that single thread. My test
client at the moment looks like this:

import sys
from pythoncom import PumpWaitingMessages
from win32com.client import Dispatch

PumpWaitingMessages()
app=Dispatch('PyComThreads.Application')
PumpWaitingMessages()
app.someLongRunningMethod(20)
PumpWaitingMessages()

If this is not essentially what you meant, could you please let me
know? The server is exactly like I had posted originally, except for
renaming its sleep() method to someLongRunningMethod().

That sucks, and is almost certainly a thread-state error.


That was my thought.
Thanks.

Regards,
John
Jul 18 '05 #7
John Lull wrote:
Now I'm *really* confused.

Perhaps I need to clarify a bit. The sleep() method in my sample
server is a perhaps-too-simplified substitute for what the real server
is doing. It provides a variety of high-level operations on a piece of
hardware. Some of the operations take as long as several dozen
seconds, others take a millisecond or so. I need the client to block
waiting for completion of each operation, regardless of how long the
operation takes. I cannot break one of the long operations up into a
series of calls from the client -- it must be implemented as a single
call. My example would, perhaps, have been clearer if I'd named the
method someLongRunningMethod() instead of sleep().


Is there any way you can do it asynchronously? The main thread spawns a
second thread to do the work. The main thread then spins around a
MsgWaitForMultipleObjects, with an object set by the second thread. The
main thread will then be able to run a message pump, but also detect
when the second thread is done.

Apart from that, I am afraid I am out of ideas. I believe however that
you are hitting pure COM issues, and nothing related to Python. Looking
for the answer beyond the Python world may be fruitful.

Mark.

Jul 18 '05 #8
Mark Hammond <mh******@skippinet.com.au> wrote (with possible
deletions):
Is there any way you can do it asynchronously? The main thread spawns a
second thread to do the work. The main thread then spins around a
MsgWaitForMultipleObjects, with an object set by the second thread. The
main thread will then be able to run a message pump, but also detect
when the second thread is done.
Yes, except there doesn't seem to be any way to return a value from
the first call until after the second call completes.
Apart from that, I am afraid I am out of ideas. I believe however that
you are hitting pure COM issues, and nothing related to Python. Looking
for the answer beyond the Python world may be fruitful.


The three reasonable possibilities I see at the moment are:

1. Dig into exactly how apartment-threaded servers are supposed to be
written in C or C++ & see how much of that is transferrable to Python.
I'm not confident this would give me a solution, though.

2. Activate the server in 2 steps -- have the main thread create (on
client request) "interlock" object that, when created, fires up a new
thread for the client who created it. The new thread then calls
factory.RegisterClassFactories() for the real worker object, then
starts a message pump. The client then requests COM to create the
worker object. The worker object creation method then calls
factory.RevokeClassFactories() to revoke that thread's registration.
The interlock and worker classes have to coordinate so that only one
worker thread is registered as class factory at any time, and only one
client is able to create the worker object at any time. The main
thread also has to register as class factory only for the interlock
object, not for the worker object. This requires changes to one py2exe
module and to all applications that use my server, and will complicate
shutdown, since I have to shut down the new thread only when all
object's it's pumping messages for are gone. There's also the
substantial possibility that the same bug I ran into when setting
sys.coinit_flags=0 in localserver.py will rear its head here.

3. Dig into why setting sys.coinit_flags=0 in localserver.py doesn't
work. This is probably the right thing to do, both because I know it
will yield a solution, and because it would solve the same issue for
anyone else with similar needs. Unfortunately it means I have to dig
into Python threading internals, pythoncom internals, and the whole
COM mechanism rather more heavily than I'd hoped to.
Regards,
John

Jul 18 '05 #9
John Lull wrote:
1. Dig into exactly how apartment-threaded servers are supposed to be
written in C or C++ & see how much of that is transferrable to Python.
I'm not confident this would give me a solution, though. .... 3. Dig into why setting sys.coinit_flags=0 in localserver.py doesn't
work. This is probably the right thing to do, both because I know it
will yield a solution, and because it would solve the same issue for
anyone else with similar needs. Unfortunately it means I have to dig
into Python threading internals, pythoncom internals, and the whole
COM mechanism rather more heavily than I'd hoped to.


I think these are pretty-much the same option. The win32com extensions
make no attempt to insulate you from these threading issues - it will
blindly do what you ask. In that regard, Python already acts very much
like a C/C++ application - under the covers, we have a C++ pointer to a
COM object, and when Python code tries to make a call, from whatever
thread, we just make the call. The same thing is true in reverse - we
hand out a C++ pointer, and when an incoming call is made on that, we
blindly call Python on that thread.

The Python GIL is in the picture, but I see no evidence it has any
bearing in this case. Certainly, in the simple example we started with,
no threads were blocked waiting for this lock.

Re sys.soinit_flags - I am fairly confident that this will cause
CoInitEx to be called with the specified flag as pythoncom is imported.
However, note that any exceptions raised by this function are masked
when called implicitly during that import. Try explicitly calling
pythoncom.CoInitializeEx(0) near the top of localserver.py - any
exceptions you see would also have happened for the implicit call made
at import time.

I'd welcome any evidence that a C++ app would behave differently, and am
keen to help you resolve (or at least explain) this for everyone's
future benefit.

Mark.

Jul 18 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Elbert Lev | last post by:
I started with Python two weeks ago and already saved some time and efforts while writing 2 programs: 1. database extraction and backup tool, which runs once a month and creates a snapshot of...
2
by: pradyumna | last post by:
In Project settins - C/C++ - Code Generation, what is the difference between the option "Multithreaded" and "Multithreaded DLL". I understand that on selecting multithreaded option, single and...
6
by: Dan Kelley | last post by:
We have a multithreaded app that responds to events, and writes these events to a text file. This text file is used by an external system for further processing. We want to be able to write...
1
by: daniel.bron | last post by:
I'm maintaining a C++ application written by a developer who has now left. The app is a multithreaded client/server app for financial data. My compiler is MS visual C++ 6.0. I'm a C++ neophyte....
7
by: Sidd | last post by:
Hi, I tried finding and example of multithreaded client-serve program in python. Can any one please tell me how to write a multithreaded client-server programn in python such that 1.It can handle...
7
by: Pavils Jurjans | last post by:
Hello, I wanted to get some light in the subject. As I develop ASP.NET applications, it's necessary to understand how exactly the server- communication happens. It seems like initially...
2
by: Alan Kemp | last post by:
Hi, I have a problem that is half python, half design. I have a multithreaded network server working, each client request spawns a new thread which deals with that client for as long as it is...
3
by: akmkat | last post by:
Hi all, I am in a great problem. I am trying to implement a multithreaded server using java, first look at the code... /*------- Main Server (server.java)--------------*/ import java.io.* ;...
4
by: cj | last post by:
I have a multithreaded transaction processing app that now needs to write to a sql db. I assume I will have to make the connection in the main program and then each thread/transaction would create...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.