Hi all,
I have a small python project i am working on. Basically i always have
two threads. A "Read" thread that sits in a loop reading a line at a
time from some input (Usually stdin) and then generating events to be
processed and a "Proc" thread that processes incoming events from a
queue. There will be additional threads as well that asynchronously
insert events into the queue to be processed, but they are not a part
of this so i have omitted them.
What i want to know is: "What is the standard/best way of implementing
such a pattern that works in the presence of errors and particularly
with the KeyboardInterrupt exception?"
Some sample code is shown below. This code works as is, except in the
case where the "Proc" thread wants to initiate the exit of the
application.
For example:
* running the code below and pressing Ctrl + C works fine as the Read
thread is initiating the shutdown.
* running the code below and entering:
pquit
some other data
<Ctrl + D>
will cause oytput:
Processing: pquit
Proc: Initiating quit
and then it HANGS waiting for the Read thread to exit.
Some questions i have that are:
* KeyboardInterrupt exception seems to only be recieved by the main
thread. Is this ALWAYS the case across all UNIX + windows platforms
(not so worried about others)?
* Can i somehow get the Proc thread to force the Read thread to
generate a KeyboardInterrupt or somehow exit its blocking "for line in
fin:" call?
Thanks,
Brendon
--- SNIP ---
# Two or more threads
#
# proc : Is a processing thread that basically reads events from a
event queue and processes them
# read : Is a thread reading in a loop from stdin and generating
events for "proc"
# * : Other additional threads that may asynchronously add events to
the queue to be processed
import Queue
import threading
import sys
def Read(queue, fin, fout):
ret = (1, 'Normal Exit')
try:
for line in fin:
queue.put((0, line))
#raise Exception("Blah")
#raise "Blah"
except KeyboardInterrupt: ret = (1, 'KeyboardInterrupt')
except Exception, e: ret = (1, 'ERROR: ' + str(e))
except: ret = (1, 'UNKNOWN-ERROR')
# Notify Proc thread that we are exiting.
queue.put(ret)
print >>fout, 'Read: Initiating quit'
def Proc(queue, fout, ignore):
quit = False
while not quit:
(evt_type, evt_data) = queue.get()
if evt_type == 0:
print >>fout, 'Processing: ' + str(evt_data)
if evt_data.startswith('pquit'):
print >>fout, 'Proc: Initiating quit'
quit = True
elif evt_type == 1:
print >>fout, 'Quit: ' + str(evt_data)
quit = True
class MyThread(threading.Thread):
def __init__(self, func, queue, file1, file2, *args, **kwds):
threading.Thread.__init__(self, *args, **kwds)
self.func = func
self.queue = queue
self.file1 = file1
self.file2 = file2
self.start()
def run(self):
return self.func(self.queue, self.file1, self.file2)
if __name__ == '__main__':
queue = Queue.Queue()
# Read thread is the main thread and seems to get the
KeyboardInterrupt exception.
t = MyThread(Proc, queue, sys.stderr, None)
Read(queue, sys.stdin, sys.stderr)
# Read thread is NOT the main thread and never seems to get the
KeyboardInterrupt exception.
# This doesnt work for that reason.
#t = MyThread(Read, queue, sys.stdin, sys.stderr)
#Proc(queue, sys.stderr, None)
# @@@Brendon How do we notify the Read thread that they should
exit?
# If the Read thread initiated the quit then all is fine.
# If the Proc thread initiated the quit then i need to get the
Read
# thread to exit too somehow. But it is currently blocking in a
read
# on an input file.
print >>sys.stderr, 'Joining thread.'
t.join()
7  4737 
En Wed, 18 Jun 2008 21:39:41 -0300, Brendon Costa <br*****@christian.net>
escribió:
I have a small python project i am working on. Basically i always have
two threads. A "Read" thread that sits in a loop reading a line at a
time from some input (Usually stdin) and then generating events to be
processed and a "Proc" thread that processes incoming events from a
queue. There will be additional threads as well that asynchronously
insert events into the queue to be processed, but they are not a part
of this so i have omitted them.
What i want to know is: "What is the standard/best way of implementing
such a pattern that works in the presence of errors and particularly
with the KeyboardInterrupt exception?"
I don't know the "standard" way, but perhaps you can get some ideas from
this recent thread: http://groups.google.com/group/comp....36f1bdd1d2d83/
Some sample code is shown below. This code works as is, except in the
case where the "Proc" thread wants to initiate the exit of the
application.
You might try using the PyThreadState_SetAsyncExc function (from the
Python C API) to inject a KeyboardInterrupt exception into the Read thread
- but I don't know if it will work at all, the execution might be blocked
waiting for an I/O call to complete, and never return to Python code...
--
Gabriel Genellina
I don't know the "standard" way, but perhaps you can get some ideas from
this recent thread:http://groups.google.com/group/comp....thread/thread/...
I had a quick read through that thread. I think i will need some more
time to think about what they are saying in there though. They seem to
talking about killing a python thread kind of similar to the C
functions TerminateThread() in windows or pthread_cancel() on UNIX
which are not suitable for my purpose.
You might try using the PyThreadState_SetAsyncExc function (from the
Python C API) to inject a KeyboardInterrupt exception into the Read thread
- but I don't know if it will work at all, the execution might be blocked
waiting for an I/O call to complete, and never return to Python code...
Unfortunately that is the problem. It is blocking in a IO system call
and i want to force it to exit that with an error, hopefully causing a
Python exception. I looked at what you mentioned and it is described a
bit here too: http://sebulba.wikispaces.com/recipe+thread2
I really need a python mechanism for interrupting blocking system
calls. have dealt with this sort of thing before in C/C++ on windows
and UNIX. For UNIX it is really a matter of sending a signal to the
process (SIGINTR), the main thread which is the only one in Python to
accept signals (others as i understand are masked) will get the signal
and and return with an EINTR breaking out of the blocking read
hopefully a python exception of Interrupted IO or KeyboardInterrupt.
Note that this only works for the one thread , but that is all i need.
For windows, it is possible to do a similar thing using:
GenerateConsoleCtrlEvent(CTRL_BREAK_EVENT, 0) with which behaves a bit
like a UNIX signal and i assume is what causes the KeyboardInterrupt
in the first place.
The problem i have is how do I achieve this in python?
I tested this a bit more. My windows example was incorrect. It should
have used CTRL_C_EVENT. But even then, the problem is that the process
will also close the console window from which it was called because of
the 0. Also this requires that the process actually have a console and
is not a GUI application.
Is there some other method rather than a blocking "for line in fin:"
that i can use for reading from a file like device that plays nicely
with other threads asynchronously waking it up?
Sorry for all the posts. I am giving updates as i look further into
the problem.
I tested this a bit more. My windows example was incorrect. It should
have used CTRL_C_EVENT. But even then, the problem is that the process
will also close the console window from which it was called because of
the 0. Also this requires that the process actually have a console and
is not a GUI application.
Is there some other method rather than a blocking "for line in fin:"
that i can use for reading from a file like device that plays nicely
with other threads asynchronously waking it up?
Sorry for all the posts. I am giving updates as i look further into
the problem.
On Jun 19, 7:54*am, Dennis Lee Bieber <wlfr...@ix.netcom.comwrote:
On Wed, 18 Jun 2008 21:33:42 -0700 (PDT), Brendon Costa
<bren...@christian.netdeclaimed the following in comp.lang.python:
Unfortunately that is the problem. It is blocking in a IO system call
and i want to force it to exit that with an error, hopefully causing a
Python exception. I looked at what you mentioned and it is described a
bit here too:http://sebulba.wikispaces.com/recipe+thread2
* * * * There is no "safe" means of forcing a Python thread to exit... Heck,
I believe the MSDN is full of warnings that killing a thread at the
Windows OS level is fraught with danger.
* * * * Unfortunately, Windows doesn't allow for use of select() on anything
other than sockets -- otherwise you could code a wait that included a
"die" object and "write" to that object to break the blocking wait.
* * * * If your blocking call has no time-out variant, you may bestuck...
If only the main thread can receive KeyboardInterrupt, is there any
reason why you couldn't move the functionality of the Read thread into
the main thread? It looks like it's not doing any work, just waiting
for the Proc thread to finish.
You could start the Proc thread, do the current Read thread
functionality until the interrupt occurs, put the apporpriate message
in the queue, and then wait for the Proc thread to finish.
If only the main thread can receive KeyboardInterrupt, is there any
reason why you couldn't move the functionality of the Read thread into
the main thread? It looks like it's not doing any work, just waiting
for the Proc thread to finish.
You could start the Proc thread, do the current Read thread
functionality until the interrupt occurs, put the apporpriate message
in the queue, and then wait for the Proc thread to finish.
It is already doing that. You will notice that the Proc() function is
called by a threading.Thread instance so Proc() is running in a
thread, but the Read() function is being called by the main thread
right after this. It DOES work with the Ctrl + C, but i can find no
way at all of closing down the script from within the Proc() thread.
The relevant bit of code is:
t = MyThread(Proc, queue, sys.stderr, None)
Read(queue, sys.stdin, sys.stderr)
In the end, the problem is that i am trying to multiplex IO and other
operations. In UNIX i would use select with the input file descriptor
and an anonymous pipe for the extra commands to achieve this without
any threads, but this script needs to run primarily on a windows box
and i would like to use it on UNIX too. I thought i could use threads
to achieve the IO Multiplexing in python, but it seems not or at least
not simply.
How do people usually manage IO multiplexing (not just using sockets)
cross platform in python?
I only need to multiplex two sources really:
* Input file or stdin
* A input event queue
This will have messages injected from various locations: timers,
the processing thread itself, and possibly from a single GUI thread at
a later point in time.
Though i can foresee that at some point in the future i may also need
to multiplex those two above and some sockets (For a server with a few
clients).
I was thinking of looking at some asynchronous IO libraries for python
on Windows + UNIX, any suggestions (Must include more than just
sockets)?
On Jun 19, 11:09*pm, Brendon Costa <bren...@christian.netwrote:
If only the main thread can receive KeyboardInterrupt, is there any
reason why you couldn't move the functionality of the Read thread into
the main thread? It looks like it's not doing any work, just waiting
for the Proc thread to finish.
You could start the Proc thread, do the current Read thread
functionality until the interrupt occurs, put the apporpriate message
in the queue, and then wait for the Proc thread to finish.
It is already doing that. You will notice that the Proc() function is
called by a threading.Thread instance so Proc() is running in a
thread, but the Read() function is being called by the main thread
right after this. It DOES work with the Ctrl + C, but i can find no
way at all of closing down the script from within the Proc() thread.
The relevant bit of code is:
t = MyThread(Proc, queue, sys.stderr, None)
Read(queue, sys.stdin, sys.stderr)
In the end, the problem is that i am trying to multiplex IO and other
operations. In UNIX i would use select with the input file descriptor
and an anonymous pipe for the extra commands to achieve this without
any threads, but this script needs to run primarily on a windows box
and i would like to use it on UNIX too. I thought i could use threads
to achieve the IO Multiplexing in python, but it seems not or at least
not simply.
How do people usually manage IO multiplexing (not just using sockets)
cross platform in python?
I only need to multiplex two sources really:
* Input file or stdin
* A input event queue
* *This will have messages injected from various locations: timers,
the processing thread itself, and possibly from a single GUI thread at
a later point in time.
Though i can foresee that at some point in the future i may also need
to multiplex those two above and some sockets (For a server with a few
clients).
I was thinking of looking at some asynchronous IO libraries for python
on Windows + UNIX, any suggestions (Must include more than just
sockets)?
They either use an event-driven library.. or they use a timeout of
around 1 second. 1 second will definitely waste power on laptops (and
desktops), but it *works*.
python-safethread has this fixed - any lowlevel trickery needed is
done for you - but it's not ported to windows yet. This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Vegard Bakke |
last post by:
From whet I can see, Python documentation is lacking a very important
piece of information. Very few functions have documented what
exceptions they raise, and under what cicumstances. Is this a...
|
by: Anthony Baxter |
last post by:
On behalf of the Python development team and the Python community, I'm
happy to announce the release of Python 2.3.1 (final).
Python 2.3.1 is a pure bug fix release of Python 2.3, released in...
|
by: Follower |
last post by:
Hi,
I've run into an issue which seems to have been discussed previously
on `python-dev` but only in context of Zope3:
"Fun with 2.3 shutdown" -- Tim Peters...
|
by: Drake |
last post by:
I have a general question of Python style, or perhaps just good
programming practice.
My group is developing a medium-sized library of general-purpose
Python functions, some of which do I/O....
|
by: taylorcarr |
last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
|
by: Charles Arthur |
last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
|
by: aa123db |
last post by:
Variable and constants
Use var or let for variables and const fror constants.
Var foo ='bar';
Let foo ='bar';const baz ='bar';
Functions
function $name$ ($parameters$) {
}
...
|
by: ryjfgjl |
last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: BarryA |
last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
|
by: Sonnysonu |
last post by:
This is the data of csv file
1 2 3
1 2 3
1 2 3
1 2 3
2 3
2 3
3
the lengths should be different i have to store the data by column-wise with in the specific length.
suppose the i have to...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers,...
| |
