471,330 Members | 1,783 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,330 software developers and data experts.

Async XMLRPC and job processing

I would like to set up a server that takes XMLRPC requests and
processes them asynchronously. The XMLRPC server part is trivial in
python. The job processing part is the part that I am having trouble
with. I have been looking at how to use threadpool, but I can't see
how to get that working. I would like to have the XMLRPC part of
things do something like:

def method1(a,b,c):
jobid=workRequest(long_method1,[a,b,c])
return(jobid)

def method2(a,b,c):
jobid=workRequest(long_method2,[a,b,c])
return(jobid)

def long_method1(a,b,c)
do lots of heavy computation, etc.
store results in files in a given directory, etc
return result

..... for any number of methods

Again, pretty straightforward. However, I run into problems with the
threadpool and xmlrpc server both waiting. In particular, if I do
something like:

server = SimpleXMLRPCServer.SimpleXMLRPCServer(.....)
server.serve_forever()

Where can tell the threadpool that I have set up to wait
indefinitely? Both are blocking.

Thanks,
Sean

Oct 17 '07 #1
1 2913
Sean Davis wrote:
I would like to set up a server that takes XMLRPC requests and
processes them asynchronously. The XMLRPC server part is trivial in
python. The job processing part is the part that I am having trouble
with. I have been looking at how to use threadpool, but I can't see
how to get that working. I would like to have the XMLRPC part of
things do something like:

def method1(a,b,c):
jobid=workRequest(long_method1,[a,b,c])
return(jobid)

def method2(a,b,c):
jobid=workRequest(long_method2,[a,b,c])
return(jobid)

def long_method1(a,b,c)
do lots of heavy computation, etc.
store results in files in a given directory, etc
return result

.... for any number of methods

Again, pretty straightforward. However, I run into problems with the
threadpool and xmlrpc server both waiting. In particular, if I do
something like:

server = SimpleXMLRPCServer.SimpleXMLRPCServer(.....)
server.serve_forever()

Where can tell the threadpool that I have set up to wait
indefinitely? Both are blocking.

Thanks,
Sean
This site shows how to make it multi-threaded:

http://aspn.activestate.com/ASPN/Coo.../Recipe/425043

But now you have to contend with the integrity of your data, provided
your going be storing data through these processes. You may want to look
into the Queue module to create a sort of message queue so your data can
be properly synchronized.

Hope this helps.

Adonis
Oct 17 '07 #2

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

reply views Thread by Passynkov, Vadim | last post: by
reply views Thread by David Hirschfield | last post: by
6 posts views Thread by half.italian | last post: by
reply views Thread by Benjamin Grieshaber | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.