Peter Hansen <pe***@engcorp. com> writes:
al****@gmail.co m wrote: I googled as suggested, and the answer isn't crystal clear. My
impression is that the problem is that a python thread must acquire the
GIL in order to execute, and the strategy for deciding which thread
should get the GIL when multiple threads are waiting for it is not
based on priority. Is that correct?
That's basically correct. I don't actually know what
the strategy is, though I suspect it's either not
formally documented or explicitly not defined, though
for a given platform there may be some non-arbitrary
pattern...
(...)
I expect the Python interpreter has little to say over thread
prioritization and choice of execution, although it does impose some
granularity on the rate of switching. The GIL itself is implemented
on the lower layer lock implementation, which is taken from the native
threading implementation for the platform.
Therefore, when multiple Python threads are waiting for the GIL, which
one is going to get released will depend on when the underlying OS
satisfies the lock request from the threads, which should be based on
the OS thread scheduling system and have nothing to do with Python
per-se.
I do believe you are correct in that the Python GIL prevents thread
pre-emption by the OS (because all other Python threads are waiting on
the GIL and not in a running state), but the actual act of switching
threads at a switching point (sys.setcheckin terval()) would be an OS
only decision, and subject to whatever standard platform thread
scheduling rules were in place.
So if you were to use a platform specific method to control thread
priority, that method should be honored by the Python threads (subject
to the granularity of the system check interval for context switches).
For example, here's a Windows approach that fiddles with the thread
priority:
- - - - - - - - - - - - - - - - - - - - - - - - -
import threading
import ctypes
import time
w32 = ctypes.windll.k ernel32
THREAD_SET_INFO RMATION = 0x20
THREAD_PRIORITY _ABOVE_NORMAL = 1
class DummyThread(thr eading.Thread):
def __init__(self, begin, name, iterations):
threading.Threa d.__init__(self )
self.begin = begin
self.tid = None
self.iterations = iterations
self.setName(na me)
def setPriority(sel f, priority):
if not self.isAlive():
print 'Unable to set priority of stopped thread'
handle = w32.OpenThread( THREAD_SET_INFO RMATION, False, self.tid)
result = w32.SetThreadPr iority(handle, priority)
w32.CloseHandle (handle)
if not result:
print 'Failed to set priority of thread', w32.GetLastErro r()
def run(self):
self.tid = w32.GetCurrentT hreadId()
name = self.getName()
self.begin.wait ()
while self.iterations :
print name, 'running'
start = time.time()
while time.time() - start < 1:
pass
self.iterations -= 1
if __name__ == "__main__":
start = threading.Event ()
normal = DummyThread(sta rt, 'normal', 10)
high = DummyThread(sta rt, 'high', 10)
normal.start()
high.start()
# XXX - This line adjusts priority - XXX
high.setPriorit y(THREAD_PRIORI TY_ABOVE_NORMAL )
# Trigger thread execution
start.set()
- - - - - - - - - - - - - - - - - - - - - - - - -
And the results of running this with and without the setPriority call:
Without: With:
normal running high running
high running high running
normal running high running
high running high running
normal running normal running
high running high running
normal running high running
high running high running
normal running high running
high running normal running
normal running high running
high running high running
normal running normal running
high running normal running
normal running normal running
high running normal running
normal running normal running
high running normal running
normal running normal running
high running normal running
I'm not entirely positive why the normal thread gets occasionally
executed before the high thread is done. It might be that the
interpreter is actually releasing the GIL in the code I've written for
the thread's run() (maybe during the I/O) which opens up an
opportunity, or it may be that Windows is boosting the other thread
occasionally to avoid starvation. So I expect the normal thread is
getting occasional bursts of bytecode execution (the syscheckinterva l).
But clearly the OS level prioritization is largely driving things.
-- David