473,320 Members | 2,189 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Logging thread with Queue and multiple threads to log messages

I am trying to put up a queue (through a logging thread) so that all
worker threads can ask it to log messages. However, the problem I am
facing is that, well, the logging thread itself is running forever.
It does not know when it should exit. Any suggestion? None of the
worker threads knows for sure when the logging thread should exit
since they don't know what other worker threads are doing. I also try
to set a time limit for the logging thread, but it does not work.
# mylogging.py - used by everyone else that needs to log messages
import threading
import Queue
import time

# This is the queue object that's used to hold log events.
# LoggingThread knows about it, and the log() function below
# knows about it. No one else is allowed to see it.
_log_queue = Queue.Queue()

class LoggingThread(threading.Thread):
def __init__(self,logfile,duration):
threading.Thread.__init__(self)
self.logfile = logfile
self.duration = duration

def run(self):
f=open(self.logfile, 'w')
#it's probably not the right way to set an end time here
#since the logging thread should wait until all worker threads
are done
end = time.time() + self.duration + 10
while(end time.time()):
message = _log_queue.get()
f.write(message + '\n') # (or whatever)

f.close()

# Kick off the logging thread.
LoggingThread("logs/myapp.log",20).start()

def log(msg):
_log_queue.put(msg)


class Worker(threading.Thread):
def __init__(self,no,duration):
threading.Thread.__init__(self)
self.no = no
self.duration = duration

def run(self):
end = time.time() + self.duration

while(end time.time()):
log('Thread Object (%d):(%d), Time:%s in seconds %d'%
(self.no,self.duration,time.ctime(),time.time()))
time.sleep(10)

print 'Done with worker (%d) at %s'%(self.no,time.ctime())

def main():
children = []
args = parseArgs()

for i in range(args.threads):
log('i=%d'%(i))
children.append(Worker(i,args.duration))
children[i].start()
time.sleep(0.1)
Nov 9 '08 #1
3 5721
On Nov 9, 8:28 pm, "scriptlear...@gmail.com" <scriptlear...@gmail.com>
wrote:
I am trying to put up a queue (through aloggingthread) so that all
worker threads can ask it to log messages. However, the problem I am
facing is that, well, theloggingthread itself is running forever.
It does not know when it should exit. Any suggestion? None of the
worker threads knows for sure when theloggingthread should exit
since they don't know what other worker threads are doing. I also try
to set a time limit for theloggingthread, but it does not work.

# mylogging.py - used by everyone else that needs to log messages
import threading
import Queue
import time

# This is the queue object that's used to hold log events.
# LoggingThread knows about it, and the log() function below
# knows about it. No one else is allowed to see it.
_log_queue = Queue.Queue()

class LoggingThread(threading.Thread):
def __init__(self,logfile,duration):
threading.Thread.__init__(self)
self.logfile = logfile
self.duration = duration

def run(self):
f=open(self.logfile, 'w')
#it's probably not the right way to set an end time here
#since theloggingthread should wait until all worker threads
are done
end = time.time() + self.duration + 10
while(end time.time()):
message = _log_queue.get()
f.write(message + '\n') # (or whatever)

f.close()

# Kick off theloggingthread.
LoggingThread("logs/myapp.log",20).start()

def log(msg):
_log_queue.put(msg)

class Worker(threading.Thread):
def __init__(self,no,duration):
threading.Thread.__init__(self)
self.no = no
self.duration = duration

def run(self):
end = time.time() + self.duration

while(end time.time()):
log('Thread Object (%d):(%d), Time:%s in seconds %d'%
(self.no,self.duration,time.ctime(),time.time()))
time.sleep(10)

print 'Done with worker (%d) at %s'%(self.no,time.ctime())

def main():
children = []
args = parseArgs()

for i in range(args.threads):
log('i=%d'%(i))
children.append(Worker(i,args.duration))
children[i].start()
time.sleep(0.1)
Take a look at this example test script to see how to use logging in a
multi-threaded environment:

http://dpaste.com/hold/89734/

Regards,

Vinay Sajip
Nov 10 '08 #2
Hi

On Nov 9, 8:28*pm, "scriptlear...@gmail.com" <scriptlear...@gmail.com>
wrote:
I am trying to put up a queue (through a logging thread) so that all
worker threads can ask it to log messages.
There is no need to do anything like this, the logging module is
thread safe and you can happily just create loggers in a thread and
use them, you can even use loggers that where created outside of the
thread. We use logging from threads all the time and it works
flawlessly.

As mentioned by Vinay in your other thread the problem you have is
that your main thread exists before the worker threads, which makes
atexit run the exithandlers and logging registers the
logging.shutdown() function which does flush all logging buffers and
closes all handles (i.e. closes the logfile). So as said before by
simply calling .join() on all the worker threads inside the main
thread your problem will be solved. You might get away with making
your threads daemonic, but I can't guarentee you won't run into race
conditions in that case. If you want to be really evil you could get
into muddling with atexit...

Regards
Floris
Nov 10 '08 #3
Thank you guys, indeed, calling join() for each thread actually solved
the problem.
I misunderstood it and call join() right after start() so it didn't
work the way I wanted.

for i in range(args.threads):
children[i].join()
Nov 11 '08 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: Julia | last post by:
Hi, So my server is complete,I have all the code and libraries which I need but now I want to add exception management and logging capabilities I would like you to suggest a good book about...
13
by: Paul | last post by:
Hi, How do I wait until a thread is finished his job then continue to the original thread? public void main(string args) { Thread t = new Thread(new ThreadStart(DoWork)); t.Start();
3
by: Oscar Thornell | last post by:
Hi, I am thinking about doing all my logging asynchronously using a delegate. The main resaon for this would be performance and responsiveness of the application (an ASP.NET app). //Exampel...
10
by: Marty | last post by:
Hi, Does anybody is experiencing a lot of RAM consumption when using many threads ? If yes, how can we reduce that level of used memory? Thanks tou! Marty
6
by: les | last post by:
Here's a class which uses 2.0 generics to implement an inter-thread message queue in C#. Any number of threads can post and read from the queue simultaneously, and the message object can be any...
6
by: Dave | last post by:
I have a service that has 6 different threads. Each thread has a timer on it that elapses at about the same time (if not the same time). When the timer elapses I am trying to log a message by...
5
by: admin | last post by:
ok This is my main. Pretty much it goes through each category and starts up 4 worker threads that then ask for groups to gether from. My problem is that when the thread gets done it keeps the...
3
by: Ross Boylan | last post by:
I would like my different threads to log without stepping on each other. Past advice on this list (that I've found) mostly says to send the messages to a Queue. That would work, but bypasses...
2
by: ZHENG Zhong | last post by:
Hi, I implemented a small logging library with the API like this: logger& log = log_manager::instance().get_logger("my_logger"); log.stream(DEBUG) << "this is a debug message" << std::endl;...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.