473,695 Members | 2,568 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Problem in threading

I have written a code to figure out the difference in excecution time
of a func before and after using threading...

Expand|Select|Wrap|Line Numbers
  1. #!/usr/bin/env python
  2.  
  3. import threading
  4. import time
  5. loops = [5000,5000]
  6.  
  7. def loop(self, nsec):
  8. for i in range(1,nsec):
  9. t=i*5000
  10. s=t/10*15555
  11.  
  12. def main():
  13. threads = []
  14. nloops = [0,1]
  15. for i in nloops:
  16. print '\nSpawning Thread',i,' value: ',loops[i]
  17. t = threading.Thread(target=loop,args=(i, loops[i]))
  18. threads.append(t)
  19.  
  20. for i in nloops: # start threads
  21. threads[i].start()
  22.  
  23. for i in nloops: # wait for all
  24. threads[i].join
  25.  
  26. if __name__ == '__main__':
  27. start = time.clock()
  28. loop(1,5000)
  29. print 'Time Taken: ', time.clock()-start
  30. start = time.clock()
  31. main()
  32. print 'Time Taken: ', time.clock()-start
  33.  
Some times this code executes and some times it hangs to death :o(
Am I am doing something wrong?? Also the difference of time is not much...
How do we best optimize our task by using threads... please help...

Thanks and Regards,
Gurpreet Singh
Jul 18 '05 #1
12 1739
Gurpreet Sachdeva <gu************ ***@gmail.com> wrote:
for i in nloops: # wait for all
threads[i].join


Missing () after 'join'.
Alex
Jul 18 '05 #2
Gurpreet Sachdeva wrote:
Also the difference of time is not much...
How do we best optimize our task by using threads... please help...


For most tasks splitting the processing into separate threads will result
in an increase in the total time to complete the task. The only times when
it may result in a decrease in the running time are when the time the task
takes does not entirely depend on the time taken by the CPU, or when
multiple CPU's are involved.

Unfortunately the latter case isn't handled well by Python, so don't expect
multiple CPU's to help speed up a multi-threaded Python program by very
much. That leaves the former case: if your task has to stop and wait for
something else to happen (e.g. data to be read from a network, or to be
read from a disc file), then splitting it into multiple threads may allow
the waits to be overlapped with useful processing which could result in an
overall decrease in processing time.

A good use of threads is to improve responsiveness of a system. For example
if you ensure that GUI processing happens on a separate thread from some
CPU intensive computation then you can ensure that the GUI remains
responsive while the computation is running. It won't make the computation
complete any faster (in fact it will probably be slower), but the user will
remain happier. Similarly network applications are usually multi-threaded
so that all requests get a fair chance to complete instead of having to
wait for the slowest to complete before others can run.

If you do have multiple processors available and want to speed up a Python
program then you probably have to look at multiple processes rather than
multiple threads. Alternatively you could move parts of the processing out
of the Python environment by rewriting inner loops in C and releasing the
interpreter lock, but that isn't usually the best option.
Jul 18 '05 #3
Duncan Booth <du**********@i nvalid.invalid> writes:
That leaves the former case: if your task has to stop and wait for
something else to happen (e.g. data to be read from a network, or to
be read from a disc file), then splitting it into multiple threads
may allow the waits to be overlapped with useful processing which
could result in an overall decrease in processing time.


If you're on a Unix-like system, you'll use less aspirin if you do
this kind of thing with async I/O and the select module. If you're on
Windows, the restrictions on what you can select on make it much less
useful. Python's threading models is pretty primitive. You get the C
model (which is error-prone), the Java model (in 2.4, and also
error-prone), or Queues. Queues aren't error-prone, but result in the
same kind of behavior as you get with select: start I/O, do computing
while I/O is going on, block until I/O is complete.

<mike
--
Mike Meyer <mw*@mired.or g> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
Jul 18 '05 #4
Mike Meyer wrote:
Python's threading models is pretty primitive. You get the C
model (which is error-prone), the Java model (in 2.4, and also
error-prone), or Queues.


Can you please expand on your words above? I have no idea
what you are talking about with the "Java model" and your
implication that it is something new in Python 2.4. As
far as I know, nothing significant with respect to threads
has changed in Python recently.

-Peter
Jul 18 '05 #5
I wrote:
Also the difference of time is not much...
How do we best optimize our task by using threads... please help...

Duncan Booth Wrote:The only times when it may result in a decrease
in the running time... are when the time the task...when
multiple CPU's are involved. I fotgot to mention that I am running that script on a 4 node Open SSI
cluster with 8 processors to do the job...
Unfortunately the latter case isn't handled well by Python Very Strange!
A good use of threads is to improve responsiveness of a system. For example
if you ensure that GUI processing happens on a separate thread from some
CPU intensive computation then you can ensure that the GUI remains
responsive while the computation is running. It won't make the computation
complete any faster (in fact it will probably be slower), but the user will
remain happier. Similarly network applications are usually multi-threaded
so that all requests get a fair chance to complete instead of having to
wait for the slowest to complete before others can run.
So That means blindly using threads on any process won't help!
probably have to look at multiple processes rather than
multiple threads.
How do I do that??? Does that mean forking the processes or does that
mean to change the entire architecture to do small tasks in different
processes???
Alternativel y you could move parts of the processing out
of the Python environment by rewriting inner loops in C


Any reference/documentation for that???

Thanks and Regards
Gurpreet
Jul 18 '05 #6

"Gurpreet Sachdeva" <gu************ ***@gmail.com> wrote in message
news:ma******** *************** *************** @python.org...

So That means blindly using threads on any process won't help!


It depends on what "help" means to you. Both Windows and Unix (and it's
variances) are considered "thread-weak" OSes. So, using thread will come
with some cost. The long gone IBM OS/2 is a classic example of a
"thread-strong" OS.

Still, a good use of thread would be, for example, a name-pipe server. See
for instance:

http://www-106.ibm.com/developerwork...,l=252,p=pipes

Here you will see a performance improvement when threads are being used.

The example you listed isn't a good use of thread for performance
improvement sake.

Jul 18 '05 #7
>>> So That means blindly using threads on any process won't help!
It depends on what "help" means to you.
Help means to improve processing speed in achieving a particular
task... *Help* here also means that I have a processor farm, how do I
best use them to get maximum processing speed out of them...
The example you listed isn't a good use of thread for performance


The example was a dummy one to become friendly with Threads and to
understand the working/power of them... The example which will be
finally used with threads take 5 Days to complete... I want to convert
that in few hours!

Long Journey ahead...

Thanks,
Garry
Jul 18 '05 #8
Gurpreet Sachdeva wrote:
So That means blindly using threads on any process won't help!
It depends on what "help" means to you.

Help means to improve processing speed in achieving a particular
task... *Help* here also means that I have a processor farm, how do I
best use them to get maximum processing speed out of them...

And that's the crux of your problem.

Firstly, you've discovered that attempts to partition a task using
threads won't work well with a compute-intensive algorithm, since
multiple threads will simply contend against each other for CPU in a
single process.

This is not assisted by Python's use of a global interpreter lock (GIL)
to ensure thread-safety.

A thread can release the GIL, but typically it will do this only when
it's involved in some blocking operation. This means that threading can
be useful to speed up network operations, for example, but even then you
might get a better speedup using explicitly asynchronous techniques
based on non-blocking sockets.
The example you listed isn't a good use of thread for performance

The example was a dummy one to become friendly with Threads and to
understand the working/power of them... The example which will be
finally used with threads take 5 Days to complete... I want to convert
that in few hours!

In that case you definitely need to be looking at multiprocess
algorithms if you are sticking with Python. Since each process has its
own copy of the interpreter, they each also have their own GIL, and so
the operating system will be able to schedule the processes in parallel
on separate CPUs.
Long Journey ahead...

Indeed, but an interesting one, no doubt. Good luck.

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119
Jul 18 '05 #9
"It's me" <it***@yahoo.co m> writes:
It depends on what "help" means to you. Both Windows and Unix (and it's
variances) are considered "thread-weak" OSes. So, using thread will come
with some cost. The long gone IBM OS/2 is a classic example of a
"thread-strong" OS.

(...)

Interesting - can you clarify what you perceive as the differences
between a thread-weak and thread-strong OS? If given the choice, I
would probably refer to Windows (at least NT based systems, let's
ignore 9x) as thread-strong, and yes, often think of Windows as
preferring thread based solutions, while Unix would often prefer
process based.

Windows is far more efficient at handling large numbers of threads
than it is processes, with much less overhead and there is lots of
flexibility in terms of managing threads and their resources. Threads
are first class OS objects at the kernel and scheduler level (waitable
and manageable).

I can't think of anything offhand specific that OS/2 did with respect
to threads that isn't as well supported by current Win32 systems.

-- David
Jul 18 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
2448
by: D. Shifflett | last post by:
Hi all, I am having trouble with a program that ran fine on Python 2.0 (#0, Mar 1 2001, 01:47:55) on linux2 but will not work on Python 2.3.2 (#1, Oct 8 2003, 17:33:47) on linux2
2
2977
by: Egor Bolonev | last post by:
hi all my program terminates with error i dont know why it tells 'TypeError: run() takes exactly 1 argument (10 given)' =program==================== import os, os.path, threading, sys def get_all_files(path): """return all files of folder path, scan with subfolders
2
1416
by: hecklar | last post by:
This is my first time posting here, so i apologize if i'm posting in the wrong subgroup or whatever, but here goes... I’m having a problem with threading and events (permissions?) in a VB.net Windows application (background service). I’m trying to write an application that processes files, launching a new thread for each file that is dropped into a certain folder. Now, the application works like a charm on my Win2000 machine, but...
2
1332
by: peleme | last post by:
Hi, Here is a code example to visualize my problem. ---------------------------------------------------------------------------------- import thread import threading from time import sleep def a():
0
285
by: fiefie.niles | last post by:
I am having problem with thread. I have a Session class with public string variable (called Message) that I set from my Main program. In the session class it checks for the value of Message while inside it's "read loop" waiting for data from the client. I find that many times while inside the "read loop" it missed many of the value that was assigned to the public Message variable. For example, the main program send number 1 thru 100, but...
9
3267
by: esakal | last post by:
Hello, I'm programming an application based on CAB infrastructure in the client side (c# .net 2005) Since my application must be sequencally, i wrote all the code in the UI thread. my problem occurs when i try to show a progress bar. The screen freezes. I know i'm not the first one to ask about it. but i'm looking
12
2290
by: Justin | last post by:
I can attach my code if anyone wants to see it however I'll try to ask my question with some mark up code first. I'm having a problem terminating my process while using DoEvents. For example: Button.text = start If button.text = start then button.text = stop
5
6928
by: CCLeasing | last post by:
For an application I'm creating I want to create a 'fake' progress bar. By fake I mean a progress bar that looks like it's doing something but actually isn't. I know philosophically this isn't sound. But my little app is a 'fake' app and is designed to look like another - hence this seeming crazy situation of needing to fake a progess bar. PROBLEM.
2
1893
by: =?Utf-8?B?TWljaGFlbCBXLg==?= | last post by:
Dear ng, i have developed a winforms application with vs2005. An progress from with a animation and a timer is shown while the application is working. The trouble is, that often a ThreadAbordException occurs and i can not identify whats the reason is.
0
1088
by: Bieniu | last post by:
I have DataList control on my own contro land it is bind to SqlDataSource control. My problem is that DataList is getting needed data twice what is for me very strange. I have some code in OnSelected event of SqlDataSource, and i was suprised when it was fired twice for every page loading. His is code of datalist: <tr> <td style="width: 5%;"> </td> <td colspan="3">
0
8647
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, well explore What is ONU, What Is Router, ONU & Routers main usage, and What is the difference between ONU and Router. Lets take a closer look ! Part I. Meaning of...
0
9132
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
8864
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8838
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7682
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development projectplanning, coding, testing, and deploymentwithout human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6506
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupr who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5842
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4351
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
3024
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.