GIL on SMP (need quick answer for the boss :) - thanks!

[max khesin]

I am working on a windoze service (deamon) that needs to spawn up to
NumCPU heavyweight processes on SMP and wait for results, which have to
be logged. So I have to do something like (total pseudocode, thread
pool not implemented, but demonstrates the problem):

Logger log
def ThreadFoo(job):
log.write(os.open(job).read())

while(1):
jobs = GetJobs()
threads = []
for job in jobs:
threads.append(SpawnThread(ThreadFoo, job))
waitFor(threads)
I have 3 questions:

1) (this is the killer) - is GIL going to get me into trouble? (i guess
not, since all the python threads can run on the same CPU, not
preventing the processes from being scheduled to different CPUs by the OS)
2) any problems having multiple threads write to the log (the standard
Python logger). In general, is there an easy way to determine if a
particular Python lib is thread-safe, at least for the standard libs?
3) Has anyone already implemented a thread pool lib that I can use (GPL
not ok for this :()

thanks,

max

[Martin v. Löwis]

max khesin wrote:

I have 3 questions:

1) (this is the killer) - is GIL going to get me into trouble?
Define "trouble". You will not get in conflict with law enforcement
because of that Python code :-)

If you are asking "does it do what you think it does?", then "yes,
it does precisely what I think it does."

If you are asking "does it do what I think it does?", then "I
don't know, because I don't know what you are thinking".
2) any problems having multiple threads write to the log (the standard
Python logger).
It should work fine.
In general, is there an easy way to determine if a
particular Python lib is thread-safe, at least for the standard libs?

No. It starts with the lack of a clear definition of "thread-safe".
In most cases, if a library has what many people would consider a
serious problem, this problem has been either documented or fixed
by now.

OTOH, many libraries implemented in pure Python will act strangely
if two threads try to manipulate the same data structures. In many
cases, this is not considered a serious problem because one would
normally not attempt to use that data structure from two threads.

Regards,
Martin