473,543 Members | 1,892 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Logging thread with Queue and multiple threads to log messages

I am trying to put up a queue (through a logging thread) so that all
worker threads can ask it to log messages. However, the problem I am
facing is that, well, the logging thread itself is running forever.
It does not know when it should exit. Any suggestion? None of the
worker threads knows for sure when the logging thread should exit
since they don't know what other worker threads are doing. I also try
to set a time limit for the logging thread, but it does not work.
# mylogging.py - used by everyone else that needs to log messages
import threading
import Queue
import time

# This is the queue object that's used to hold log events.
# LoggingThread knows about it, and the log() function below
# knows about it. No one else is allowed to see it.
_log_queue = Queue.Queue()

class LoggingThread(t hreading.Thread ):
def __init__(self,l ogfile,duration ):
threading.Threa d.__init__(self )
self.logfile = logfile
self.duration = duration

def run(self):
f=open(self.log file, 'w')
#it's probably not the right way to set an end time here
#since the logging thread should wait until all worker threads
are done
end = time.time() + self.duration + 10
while(end time.time()):
message = _log_queue.get( )
f.write(message + '\n') # (or whatever)

f.close()

# Kick off the logging thread.
LoggingThread(" logs/myapp.log",20). start()

def log(msg):
_log_queue.put( msg)


class Worker(threadin g.Thread):
def __init__(self,n o,duration):
threading.Threa d.__init__(self )
self.no = no
self.duration = duration

def run(self):
end = time.time() + self.duration

while(end time.time()):
log('Thread Object (%d):(%d), Time:%s in seconds %d'%
(self.no,self.d uration,time.ct ime(),time.time ()))
time.sleep(10)

print 'Done with worker (%d) at %s'%(self.no,ti me.ctime())

def main():
children = []
args = parseArgs()

for i in range(args.thre ads):
log('i=%d'%(i))
children.append (Worker(i,args. duration))
children[i].start()
time.sleep(0.1)
Nov 9 '08 #1
3 5732
On Nov 9, 8:28 pm, "scriptlear...@ gmail.com" <scriptlear...@ gmail.com>
wrote:
I am trying to put up a queue (through aloggingthread) so that all
worker threads can ask it to log messages. However, the problem I am
facing is that, well, theloggingthrea d itself is running forever.
It does not know when it should exit. Any suggestion? None of the
worker threads knows for sure when theloggingthrea d should exit
since they don't know what other worker threads are doing. I also try
to set a time limit for theloggingthrea d, but it does not work.

# mylogging.py - used by everyone else that needs to log messages
import threading
import Queue
import time

# This is the queue object that's used to hold log events.
# LoggingThread knows about it, and the log() function below
# knows about it. No one else is allowed to see it.
_log_queue = Queue.Queue()

class LoggingThread(t hreading.Thread ):
def __init__(self,l ogfile,duration ):
threading.Threa d.__init__(self )
self.logfile = logfile
self.duration = duration

def run(self):
f=open(self.log file, 'w')
#it's probably not the right way to set an end time here
#since theloggingthrea d should wait until all worker threads
are done
end = time.time() + self.duration + 10
while(end time.time()):
message = _log_queue.get( )
f.write(message + '\n') # (or whatever)

f.close()

# Kick off theloggingthrea d.
LoggingThread(" logs/myapp.log",20). start()

def log(msg):
_log_queue.put( msg)

class Worker(threadin g.Thread):
def __init__(self,n o,duration):
threading.Threa d.__init__(self )
self.no = no
self.duration = duration

def run(self):
end = time.time() + self.duration

while(end time.time()):
log('Thread Object (%d):(%d), Time:%s in seconds %d'%
(self.no,self.d uration,time.ct ime(),time.time ()))
time.sleep(10)

print 'Done with worker (%d) at %s'%(self.no,ti me.ctime())

def main():
children = []
args = parseArgs()

for i in range(args.thre ads):
log('i=%d'%(i))
children.append (Worker(i,args. duration))
children[i].start()
time.sleep(0.1)
Take a look at this example test script to see how to use logging in a
multi-threaded environment:

http://dpaste.com/hold/89734/

Regards,

Vinay Sajip
Nov 10 '08 #2
Hi

On Nov 9, 8:28*pm, "scriptlear...@ gmail.com" <scriptlear...@ gmail.com>
wrote:
I am trying to put up a queue (through a logging thread) so that all
worker threads can ask it to log messages.
There is no need to do anything like this, the logging module is
thread safe and you can happily just create loggers in a thread and
use them, you can even use loggers that where created outside of the
thread. We use logging from threads all the time and it works
flawlessly.

As mentioned by Vinay in your other thread the problem you have is
that your main thread exists before the worker threads, which makes
atexit run the exithandlers and logging registers the
logging.shutdow n() function which does flush all logging buffers and
closes all handles (i.e. closes the logfile). So as said before by
simply calling .join() on all the worker threads inside the main
thread your problem will be solved. You might get away with making
your threads daemonic, but I can't guarentee you won't run into race
conditions in that case. If you want to be really evil you could get
into muddling with atexit...

Regards
Floris
Nov 10 '08 #3
Thank you guys, indeed, calling join() for each thread actually solved
the problem.
I misunderstood it and call join() right after start() so it didn't
work the way I wanted.

for i in range(args.thre ads):
children[i].join()
Nov 11 '08 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
1819
by: Julia | last post by:
Hi, So my server is complete,I have all the code and libraries which I need but now I want to add exception management and logging capabilities I would like you to suggest a good book about logging and exceptions handling for multiple threads server Some question I have in mind are 1.Where is the best place in the component hierarchy...
13
2047
by: Paul | last post by:
Hi, How do I wait until a thread is finished his job then continue to the original thread? public void main(string args) { Thread t = new Thread(new ThreadStart(DoWork)); t.Start();
3
4499
by: Oscar Thornell | last post by:
Hi, I am thinking about doing all my logging asynchronously using a delegate. The main resaon for this would be performance and responsiveness of the application (an ASP.NET app). //Exampel delegate void LogDelegate(string message); LogDelegate logger = new LogDelegate(Log.Debug); logger.DynamicInvoke("Some message..");
10
1454
by: Marty | last post by:
Hi, Does anybody is experiencing a lot of RAM consumption when using many threads ? If yes, how can we reduce that level of used memory? Thanks tou! Marty
6
16696
by: les | last post by:
Here's a class which uses 2.0 generics to implement an inter-thread message queue in C#. Any number of threads can post and read from the queue simultaneously, and the message object can be any type. There's a test driver at the bottom which demonstrates usage. ...
6
4207
by: Dave | last post by:
I have a service that has 6 different threads. Each thread has a timer on it that elapses at about the same time (if not the same time). When the timer elapses I am trying to log a message by writing to a text file. The problem is that not every thread is able to log its message. I thought it was because each thread is trying to use the...
5
3781
by: admin | last post by:
ok This is my main. Pretty much it goes through each category and starts up 4 worker threads that then ask for groups to gether from. My problem is that when the thread gets done it keeps the mysql connections open so I end up with quite a few at the end. Is there a different way that I should do this? class Program { static string...
3
4282
by: Ross Boylan | last post by:
I would like my different threads to log without stepping on each other. Past advice on this list (that I've found) mostly says to send the messages to a Queue. That would work, but bypasses the logging module's facilities. The logging module itself is "thread-safe", but I think that just means that individual output is protected. If I...
2
4036
by: ZHENG Zhong | last post by:
Hi, I implemented a small logging library with the API like this: logger& log = log_manager::instance().get_logger("my_logger"); log.stream(DEBUG) << "this is a debug message" << std::endl; log.stream(INFO) << "this is an info message" << std::endl;
0
7411
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7354
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
7594
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
7746
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
0
5888
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
0
4898
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3394
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
0
3394
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
0
643
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.