473,728 Members | 1,514 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Job queue using xmlrpc and threading

I'm trying to implement an application that will listen for requests,
run them when they are present but also be able to add new requests
even while it's running. I've tried to do this using the thread and
xmlrpc modules - the idea is that an XML-RPC exposed object tell the
queue thread object to add a job. If there are no jobs running, it
creates a file, adds the new job to the end and then another
consumption thread starts working through the jobs in the file. New
jobs coming in are just added to the end of the file by the queue
thread.

Unfortunately, I can't get it to work. The problem is that the
consumption thread seems to read the job queue before it gets written,
even though I've used a lock. I've also had the application get to the
stage where it ignores ctrl-c, which is a little worrying - I fear it
doesn't bode well for future stability... I don't have a lot of
experience with multi-threaded applications, so I may well have chosen
a poor approach.

I've posted the code below. It's in three parts, the job queue, the
manager that listens for new requests and an application to add jobs
to the queue. Sorry for the long listings...

Any guidance gratefully received,

Peter

===
testqueue.py:

import thread
import time
import shutil
import os

class JobQueue:

def __init__(self, filename):
self.queuefile = filename
self.jobthread = 0
# lock for the jobfile queue file
self.jfqlock = thread.allocate _lock()

def addJob(self, jobfileuri, email):
self.jfqlock.ac quire()
if not self.jobthread:
print "starting jobfile consumption thread"
if os.access(self. queuefile, os.R_OK):
print "cleaning stale jobfile queue file"
try:
os.remove(self. queuefile)
except:
print "problem removing jobfile queue file"
raise
self.jobthread = thread.start_ne w_thread(self.m ain, ())
else:
print "using existing jobfile consumption thread in file",
self.queuefile
fh = open(self.queue file, "a")
# choose "::::" as a delimiter
print >fh, jobfileuri + "::::" + email
self.jfqlock.re lease()
return 1

def main(self):
while 1:
self.jfqlock.ac quire()
rfh = open(self.queue file, "r")
# breakpoint()
finput = rfh.readline()
print "found:", finput
if not finput:
print "finished jobfiles. Closing thread"
rfh.close()
self.jobthread = 0
self.jfqlock.re lease()
return
else:
print "found jobfile in queue: attempting to run"
# most of this is to shift up the lines in the file
tmpname = self.queuefile + ".tmp"
wfh = open(tmpname, "w")
line = rfh.readline()
while line:
wfh.write(line)
line = rfh.readline()
wfh.close()
rfh.close()
shutil.move(tmp name, self.queuefile)
self.jfqlock.re lease()
# lop off the trailing line break
print
print "***run Starting***"
try:
self.runJob(fin put[:-1])
print "***run finished***"
except:
print "***run failed***"
print

def runJob(self, job):
time.sleep(2.0)
print "running job", job
if not report:
print "some problem with run. Cannot mail out report"
return
===
queuemanager.py

from testqueue import JobQueue
from SimpleXMLRPCSer ver import *
class QM:
def __init__(self, filename):
self.jq = JobQueue("queue .txt")

def addJob(self, jobname):
self.jq.addJob( jobname, "tester@testdom ain")

if __name__=="__ma in__":
qm = QM("jobqueue.tx t")
rpcserver = SimpleXMLRPCSer ver(("localhost ", 8000))
rpcserver.regis ter_instance(qm )
rpcserver.serve _forever()

===
addjob.py:

import xmlrpclib
import sys

server = "localhost"
port = 8000

serveradd = "http://%s:%s" % (server, port)
manager = xmlrpclib.Serve rProxy(serverad d)

jobname = sys.argv[1]

manager.addJob( jobname)
Sep 22 '08 #1
1 2559
Try using the Queue module - http://docs.python.org/lib/module-Queue.html.
Here is a tutorial with it - http://www.artfulcode.net/articles/m...eading-python/.
Sep 22 '08 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

9
2783
by: phil | last post by:
And sorry I got ticked, frustrating week >And I could help more, being fairly experienced with >threading issues and race conditions and such, but >as I tried to indicate in the first place, you've >provided next to no useful (IMHO) information to >let anyone help you more than this This is about 5% of the code. Uses no locks.
6
4994
by: James Radke | last post by:
Hello, I have a multithreaded windows NT service application (vb.net 2003) that I am working on (my first one), which reads a message queue and creates multiple threads to perform the processing for long running reports. When the processing is complete it uses crystal reports to load a template file, populate it, and then export it to a PDF. It works fine so far....
9
1550
by: mateom | last post by:
Hello, I'm using Queue to send images from one thread to another, and some of the images are not appearing in the consumer thread....maybe 1 in 3 arrive. I've tried passing the image data in both string form and as a PIL Image object, with the same result. It does work, however, if I use zlib to compress the image string before passing it to Queue and then decompress it in the consumer thread. So, my question: Does Queue have some...
13
2547
by: Jonathan Amsterdam | last post by:
I think there's a slight design flaw in the Queue class that makes it hard to avoid nested monitor deadlock. The problem is that the mutex used by the Queue is not easy to change. You can then easily get yourself into the following situation (nested monitor deadlock): Say we have a class that contains a Queue and some other things. The class's internals are protected by a mutex M. Initially the Queue is empty. The only access to the...
2
2126
by: shipcreak | last post by:
I have an interesting problem with a sort of producer-consumer system for error logging. Consider the following code: <code> SyncLock _eventList.SyncRoot Dim item As ExceptionLogEntry ' If there are items, get the top one If _eventList.Count > 0 Then
0
1449
by: Michael Bayer | last post by:
Hi - i was just going through this thread: http://mail.python.org/ pipermail/python-list/2006-April/336948.html , where it is suggested that the Lock instance used by Queue.Queue should be publically configurable. I have identified another situation where a Queue can be deadlocked, one which is also alleviated by configuring the type of Lock used by the Queue (or just changing it to an RLock). The scenario arises when the Queue is...
2
2568
by: Thomas Ploch | last post by:
Hello folks, I am having troubles with implementing a timed queue. I am using the 'Queue' module to manage several queues. But I want a timed access, i.e. only 2 fetches per second max. I am horribly stuck on even how I actually could write it. Has somebody done that before? And when yes, how is the best way to implement it? Thanks, Thomas
3
1606
by: seb | last post by:
Hi, I am writing to a file some basic information using the logging module. It is working but in the log file some line are printed several time. I had put some print debugging messages in the logging function (so they appear on the consile) and they are called once only. Obviously there is some understantding of the logging module that I am missing. My simple logging program (see below) is called by several processes. In this way I...
0
1069
by: Eugene Anthony | last post by:
First I leave the server running. Then I open up the client and click the button to send a message to the queue. So when I check the queue its empty. Possible because all the message in the queue has been read and deleted. But I am interested to have my code reading one message from the queue for every 'n' seconds. How is it done?. Client.aspx.cs -------------
0
8897
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9416
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
8123
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6711
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
6012
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4524
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4787
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3232
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
3
2159
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.