473,320 Members | 1,862 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Thread imbalance

I have a program running several threads. One of them must be done
every (Specified time, usually 1 second). The whole point to having a
thread is do this. However, I've noticed the following. When there is
another part of the program that is active, this thread slips into
disuse, ei, it's only ran about once every 4-5 seconds, or perhaps it
waits for a lul in the computing process. How can I ensure that this
does not happen? This thread uses little processing power, so it could
be set to a high priority, if there is a way to do this. Thanks!

Feb 2 '06 #1
17 1530
Tuvas wrote:
I have a program running several threads. One of them must be done
every (Specified time, usually 1 second). The whole point to having a
thread is do this. However, I've noticed the following. When there is
another part of the program that is active, this thread slips into
disuse, ei, it's only ran about once every 4-5 seconds, or perhaps it
waits for a lul in the computing process. How can I ensure that this
does not happen? This thread uses little processing power, so it could
be set to a high priority, if there is a way to do this. Thanks!

I think that might be difficult using python threads simply based on how
python controls the global interpreter lock.

One suggestion I might have is to have python release the global
interpreter lock more frequently, you can read about the global
interpreter lock here:

http://docs.python.org/api/threads.html

You might also be able to use some timer/condition construct in
combination with this, something like

Thread A:
if watchdogTimer():
conditionVar.aquire()
conditionVar.notify(threadB)
conditionVar.release()

ThreadB:
while(1):
conditionVar.aquire()
conditionVar.wait()
functionToDoSomething()

This is pseudo python of course, if you need to know about these objects
I would suggest consulting the python manual.

-carl

--

Carl J. Van Arsdall
cv*********@mvista.com
Build and Release
MontaVista Software

Feb 2 '06 #2
Tuvas wrote:
waits for a lul in the computing process. How can I ensure that this
does not happen? This thread uses little processing power, so it could
be set to a high priority, if there is a way to do this. Thanks!


Python is bad for concurrently executing/computing threads, but it
shouldn't be that bad - do you have lots of compute-intensive threads?

If you are running on a unix-like platform, see documentation for
signal() and SIGALRM - maybe it will help your task.
Feb 4 '06 #3
Ivan Voras wrote:
Tuvas wrote:
waits for a lul in the computing process. How can I ensure that this
does not happen? This thread uses little processing power, so it could
be set to a high priority, if there is a way to do this. Thanks!


Python is bad for concurrently executing/computing threads, but it
shouldn't be that bad - do you have lots of compute-intensive threads?


Just in case anyone coming along in the future reads this statement, for
the record I'd like to say this is obviously a matter of opinion or
interpretation, since in my view Python is *excellent* for threads
(which are generally considered "concurrently executing", so I think
that adjective is redundant, too).

Ivan, what makes you say that Python is bad for threads? Did the
qualifcation "concurrently executing/computing" have some significance
that I missed?

-Peter

Feb 5 '06 #4
Peter Hansen wrote:
Ivan, what makes you say that Python is bad for threads? Did the
qualifcation "concurrently executing/computing" have some significance
that I missed?


Because of the GIL (Giant interpreter lock). It can be a matter of
opinion, but by "good threading implementation" I mean that all threads
in the application should run "natively" on the underlying (p)threads
library at all times, without implicit serialization. For example, Java
and perl do this, possibly also lua and C#. Python and Ruby have a giant
interpreter lock which prevents two threads of pure python code (not
"code written in C" :)) ) in one interperter process executing at the
same time.

Much can be said about "at the same time" part, but at least in one case
it is unambiguous: in multiprocessor machines. Someone writing a
multithreaded server application could get in trouble if he doesn't know
this (of course, it depends on the exact circumstances and the purpose
of the application; for example system calls which block, such as read()
and write() can execute concurrently, while two "while True: pass"
threads cannot). This is not new information, it's available in many
forms in this newsgroup's archives.

I think it would also make implicit problems in the situation like that
of the OP, where there's a "watchdog" thread monitoring some job or
jobs, and the job(s) intefere(s) with the GIL by contenting for it,
possibly when there are lots of threads or frequent spawning of
processes or threads.

I don't have the intention to badmouth Python, but to emphasize that one
should be familira with all the details of the tool one's using :)
Feb 5 '06 #5
Ivan Voras wrote:
Peter Hansen wrote:
Ivan, what makes you say that Python is bad for threads? Did the
qualifcation "concurrently executing/computing" have some significance
that I missed?


Because of the GIL (Giant interpreter lock).
...
Much can be said about "at the same time" part, but at least in one case
it is unambiguous: in multiprocessor machines.


Okay, I thought that might be what you were talking about.

I agree that in multiprocessor situations the GIL can be a problem.

I'm equally certain that the GIL is not involved in the OP's problem,
unless he's running custom extensions that he's produced without any
awareness of the GIL.

I'm pretty sure that, for the OP's situation, Python threading is
perfectly fine and the problems he's facing are not inherent, but
related to the way he is doing things. I don't see enough information
in his post to help further though.

-Peter

Feb 5 '06 #6
In article <ds**********@bagan.srce.hr>, Ivan Voras <iv****@fer.hr> wrote:
Peter Hansen wrote:

Ivan, what makes you say that Python is bad for threads? Did the
qualifcation "concurrently executing/computing" have some significance
that I missed?


Because of the GIL (Giant interpreter lock). It can be a matter of
opinion, but by "good threading implementation" I mean that all threads
in the application should run "natively" on the underlying (p)threads
library at all times, without implicit serialization. For example, Java
and perl do this, possibly also lua and C#. Python and Ruby have a giant
interpreter lock which prevents two threads of pure python code (not
"code written in C" :)) ) in one interperter process executing at the
same time.


When did Perl gain threads? If you read Bruce Eckel, you also know that
the Java threading system has been buggy for something like a decade.
--
Aahz (aa**@pythoncraft.com) <*> http://www.pythoncraft.com/

"19. A language that doesn't affect the way you think about programming,
is not worth knowing." --Alan Perlis
Feb 5 '06 #7
Ivan Voras wrote:
opinion, but by "good threading implementation" I mean that all threads
in the application should run "natively" on the underlying (p)threads
library at all times, without implicit serialization. For example, Java
and perl do this, possibly also lua and C#.


Lua has coroutines rather than threads. It can cooperate with
threading implemented by a host application or library.

See the coroutines chapter in
http://www.lua.org/pil/index.html

Neil
Feb 5 '06 #8
Neil Hodgson wrote:
Lua has coroutines rather than threads. It can cooperate with
threading implemented by a host application or library.


I mentioned it because, as far as I know the Lua's intepreter doesn't do
implicit locking on its own, and if I want to run several threads of
pure Lua code, it's possible if I take care of data sharing and
synchronization myself.
Feb 6 '06 #9
Aahz wrote:

When did Perl gain threads?
At least in 2001:
http://mail.python.org/pipermail/pyt...st/017079.html
http://www.xav.com/perl/lib/Pod/perlthrtut.html
If you read Bruce Eckel, you also know that
the Java threading system has been buggy for something like a decade.


I'm sure precisely this is the reason why all Java Enterprise systems
(J2EE) are threaded network servers :)
Feb 6 '06 #10
Dennis Lee Bieber wrote:
FYI: That's GLOBAL Interpreter Lock


Yes, sorry about the confusion.
Feb 6 '06 #11
The stuff that it runs aren't heavily processor intensive, but rather
consistant. It's looking to read incoming data. For some reason when it
does this, it won't execute other threads until it's done. Hmmm.
Perhaps I'll just have to work on a custom read function that doesn't
depend so much on processing power.

Feb 6 '06 #12
Ivan Voras:
I mentioned it because, as far as I know the Lua's intepreter doesn't do
implicit locking on its own, and if I want to run several threads of
pure Lua code, it's possible if I take care of data sharing and
synchronization myself.


Lua's interpreter will perform synchronization if you create
multiple threads that attach to a shared Lua state. You have to provide
some functions for Lua to call to perform the locking.

If you create multiple Lua states they are completely separate
worlds and do not need to be synchronized.

Neil
Feb 6 '06 #13
Tuvas wrote:
The stuff that it runs aren't heavily processor intensive, but rather
consistant. It's looking to read incoming data. For some reason when it
does this, it won't execute other threads until it's done. Hmmm.
Perhaps I'll just have to work on a custom read function that doesn't
depend so much on processing power.


That is odd. Python should release its global lock while waiting
for I/O, and thus other threads should run. What read function
are you using? Can you provide a minimal example of your problem?
--
--Bryan
Feb 6 '06 #14
The read function used actually is from a library in C, for use over a
CAN interface. The same library appears to work perfectly find over C.
I wrote an extention module for it. The function t_config is the
threaded function that will call a function called config once per
second. Note the time.time() timer

def t_config():
while (TRUE):
if(can_send_config):
config()
ts=time.time()
time.sleep(1)
if time.time()-ts>1.5:
print "Long time detected
else:
time.sleep(.1)

When nothing is happening, the thread runs pretty consistantly at 1
second. However, when it is doing IO through this C function or
whatever, the length of time increases, for the time.sleep() function.
Kind of strange, isn't it? I can explain in more detail if it's needed,
but I don't know how much it'll help...

Feb 6 '06 #15
Peter Hansen wrote:
Ivan Voras wrote:
Python is bad for concurrently executing/computing threads, but it
shouldn't be that bad - do you have lots of compute-intensive threads?


Just in case anyone coming along in the future reads this statement, for
the record I'd like to say this is obviously a matter of opinion or
interpretation, since in my view Python is *excellent* for threads


Ivan already gave his answer: the GIL. The GIL is no real problem
on single processors. As commodity machines go to
multi-poly-hyper-cell-core, or whatever they call it, the GIL is
becoming much less attractive.

There are other issues of note:

As the OP's problem points out, Python threads don't offer any
priority control.

Timeouts on the threading module's Condition objects are
implemented by active polling. Timeouts in threading.Event and
Queue.Queue are implemented via threading.Condition so they
inherit the polling implementation.

A thread can wait on just one
Lock/RLock/Condition/Semaphore/Event/Timer at a time. There is no
analog of Win32's WaitMultipleObjects(). As an example, suppose
we wanted to implement timeouts without polling by waiting for
either a Semaphore or a timer. With Python's standard thread and
threading libraries, there's no good way to do so.
--
--Bryan
Feb 6 '06 #16
Tuvas wrote:
The read function used actually is from a library in C, for use over a
CAN interface. The same library appears to work perfectly find over C. [...] When nothing is happening, the thread runs pretty consistantly at 1
second. However, when it is doing IO through this C function or
whatever, the length of time increases, for the time.sleep() function.
Kind of strange, isn't it? I can explain in more detail if it's needed,
but I don't know how much it'll help...


That's enough detail that I'll take a most-likely-guess:

When your C function runs, by default you hold the aforementioned
Global Interpreter Lock (GIL). Unless you explicitly release it,
you will hold it until you return, and nothing else will run.
Python's own I/O operations always release the GIL right before
calling any potentially-blocking operation, and reacquire it
right after.

See:

http://docs.python.org/api/threads.html
--
--Bryan
Feb 6 '06 #17
Adding that bit of code seems to have fixed the problem, thanks alot!

Feb 7 '06 #18

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

14
by: adeger | last post by:
Having trouble with my first forays into threads. Basically, the threads don't seem to be working in parallel (or you might say are blocking). I've boiled my problems to the following short code...
4
by: Gilles Leblanc | last post by:
Hi I have started a small project with PyOpenGL. I am wondering what are the options for a GUI. So far I checked PyUI but it has some problems with 3d rendering outside the Windows platform. I...
7
by: Ivan | last post by:
Hi I have following problem: I'm creating two threads who are performing some tasks. When one thread finished I would like to restart her again (e.g. new job). Following example demonstrates...
4
by: Matthew Groch | last post by:
Hi all, I've got a server that handles a relatively high number of concurrent transactions (on the magnitude of 1000's per second). Client applications establish socket connections with the...
5
by: Razzie | last post by:
Hi all, A question from someone on a website got me thinking about this, and I wondered if anyone could explain this. A System.Threading.Timer object is garbage collected if it has no...
16
by: droopytoon | last post by:
Hi, I start a new thread (previous one was "thread timing") because I have isolated my problem. It has nothing to do with calling unmanaged C++ code (I removed it in a test application). I...
9
by: mareal | last post by:
I have noticed how the thread I created just stops running. I have added several exceptions to the thread System.Threading.SynchronizationLockException System.Threading.ThreadAbortException...
13
by: Bob Day | last post by:
Using vs2003, vb.net I start a thread, giving it a name before start. Code snippet: 'give each thread a unique name (for later identification) Trunk_Thread.Name = "Trunk_0_Thread" ' allow...
7
by: Charles Law | last post by:
My first thought was to call WorkerThread.Suspend but the help cautions against this (for good reason) because the caller has no control over where the thread actually stops, and it might have...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.