473,840 Members | 1,670 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

multi threading in multi processor (computer)

Hello,

Is anyone has experiance in running python code to run multi thread
parallel in multi processor. Is it possible ?

Can python manage which cpu shoud do every thread?

Sincerely Yours,
Pujo

Jul 18 '05
37 4910
Killing the GIL is proposing a silver bullet where there is no werewolf-ly,


About the only reason for killing the GIL is /us/. We, purists,
pythonistas, language nuts, or what not, who for some reason or
other simply hate the idea of the GIL. I'd view it as an artistic
desire, unurgent, something to plan for the future canvas upon
which our painting is painted...

C//

Jul 18 '05 #11
Jack Diederich <ja**@performan cedrivers.com> writes:
From reading this
thread every couple months on c.l.py for the last few years it is my
opinion that the number of people who think threading is the only solution
to their problem greatly outnumber the number of people who actually have
such a problem (like, nearly all of them).


Here here. I find that threading typically introduces worse problems
than it purports to solve.

<mike
--
Mike Meyer <mw*@mired.or g> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
Jul 18 '05 #12
Here here. I find that threading typically introduces worse problems
than it purports to solve.


I recently worked on a software effort, arguably one of the most
important software efforts in existence, in which individuals
responsible for critical performance of the application threw
arbitrarily large numbers of threads at a problem, on a multi
processor machine, on a problem that was intrinsically IO-bound.

The ease with which one can get into problems with threads (and
these days, also with network comms) leads to many problems if
the engineers aren't acquainted sufficiently with the theory.

Don't get me started on the big clusterfucks I've seen evolve
from CORBA...

C//

Jul 18 '05 #13
Mike Meyer wrote:
Jack Diederich <ja**@performan cedrivers.com> writes:

From reading this
thread every couple months on c.l.py for the last few years it is my
opinion that the number of people who think threading is the only solution
to their problem greatly outnumber the number of people who actually have
such a problem (like, nearly all of them).

Here here. I find that threading typically introduces worse problems
than it purports to solve.


In my experience, threads should mainly be used if you need asynchronous access
to a synchronous operation. You spawn the thread to make the call, it blocks on
the relevant API, then notifies the main thread when it's done.

Since any sane code will release the GIL before making the blocking call, this
scales to multiple CPU's just fine.

Another justification for threads is when you have a multi-CPU machine, and a
processor intensive operation you'd like to farm off to a separate CPU. In that
case, you can treat the long-running operation like any other synchronous call,
and farm off a thread that releases the GIL before start the time-consuming
operation.

The only time the GIL "gets in the way" is if the long-running operation you
want to farm off is itself implemented in Python.

However, consider this: threads run on a CPU, so if you want to run multiple
threads concurrently, you either need multiple CPU's or a time-slicing scheduler
that fakes it.

Here's the trick: PYTHON THREADS DO NOT RUN DIRECTLY ON THE CPU. Instead, they
run on a Python Virtual Machine (or the JVM/CLR Runtime/whatever), which then
runs on the CPU. So, if you want to run multiple Python threads concurrently,
you need multiple PVM's or a timeslicing scheduler. The GIL represents the latter.

Now, Python *could* try to provide the ability to have multiple virtual machines
in a single process in order to more effectively exploit multiple CPU's. I have
no idea if Java or the CLR work that way - my guess it that they do (or
something that looks the same from a programmer's POV). But then, they have
Sun/Microsoft directly financing the development teams.

A much simpler suggestion is that if you want a new PVM, just create a new OS
process to run another copy of the Python interpreter. The effectiveness of your
multi-CPU utilisation will then be governed by your OS's ability to correctly
schedule multiple processes rather than by the PVM's ability to fake multiple
processes using threads (Hint: the former is likely to be much better than the
latter).

Additionally, schemes for inter-process communication are often far more
scaleable than those for inter-thread communication, since the former generally
can't rely on shared memory (although good versions may utilise it for
optimisation purposes). This means they can usually be applied to clustered
computing rather effectively.

I would *far* prefer to see effort expended on making the idiom mentioned in the
last couple of paragraphs simple and easy to use, rather than on a misguided
effort to "Kill the GIL".

Cheers,
Nick.

P.S. If the GIL *really* bothers you, check out Stackless Python. As I
understand it, it does its best to avoid the C stack (and hence threads) altogether.

--
Nick Coghlan | nc******@email. com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net
Jul 18 '05 #14
In article <86************ @guru.mired.org >, Mike Meyer <mw*@mired.or g> wrote:

Here here. I find that threading typically introduces worse problems
than it purports to solve.


Depends what you're trying to do with threads. Threads are definitely a
good technique for managing long-running work in a GUI application.
Threads are also good for handling blocking I/O. Threads can in theory
be useful for computational processing, but Python provides almost no
support for that.
--
Aahz (aa**@pythoncra ft.com) <*> http://www.pythoncraft.com/

"The joy of coding Python should be in seeing short, concise, readable
classes that express a lot of action in a small amount of clear code --
not in reams of trivial code that bores the reader to death." --GvR
Jul 18 '05 #15
aa**@pythoncraf t.com (Aahz) writes:
In article <86************ @guru.mired.org >, Mike Meyer <mw*@mired.or g> wrote:

Here here. I find that threading typically introduces worse problems
than it purports to solve.

Threads are also good for handling blocking I/O.


Actually, this is one of the cases I was talking about. I find it
saner to convert to non-blocking I/O and use select() for
synchronization . That solves the problem, without introducing any of
the headaches related to shared access and locking that come with
threads.

<mike
--
Mike Meyer <mw*@mired.or g> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
Jul 18 '05 #16
Actually, this is one of the cases I was talking about. I find it
saner to convert to non-blocking I/O and use select() for
synchronizatio n. That solves the problem, without introducing any of
the headaches related to shared access and locking that come with
threads.


Threads aren't always the right entity for dealing with asynchronicity,
one might say.

C//

Jul 18 '05 #17
Mike Meyer <mw*@mired.or g> writes:
Threads are also good for handling blocking I/O.


Actually, this is one of the cases I was talking about. I find it
saner to convert to non-blocking I/O and use select() for
synchronization . That solves the problem, without introducing any of
the headaches related to shared access and locking that come with
threads.


It's just a different style with its own tricks and traps. Threading
for blocking i/o is a well-accepted idiom and if Python supports
threads at all, people will want to use them that way.
Jul 18 '05 #18
In article <86************ @guru.mired.org >, Mike Meyer <mw*@mired.or g> wrote:
aa**@pythoncra ft.com (Aahz) writes:
In article <86************ @guru.mired.org >, Mike Meyer <mw*@mired.or g> wrote:

Here here. I find that threading typically introduces worse problems
than it purports to solve.


Threads are also good for handling blocking I/O.


Actually, this is one of the cases I was talking about. I find
it saner to convert to non-blocking I/O and use select() for
synchronizatio n. That solves the problem, without introducing any of
the headaches related to shared access and locking that come with
threads.


It may be saner, but Windows doesn't support select() for file I/O, and
Python's threading mechanisms make this very easy. If one's careful
with application design, there should be no locking problems. (Have you
actually written any threaded applications in Python?)
--
Aahz (aa**@pythoncra ft.com) <*> http://www.pythoncraft.com/

"The joy of coding Python should be in seeing short, concise, readable
classes that express a lot of action in a small amount of clear code --
not in reams of trivial code that bores the reader to death." --GvR
Jul 18 '05 #19
On Monday 14 February 2005 00:53, Aahz wrote:
In article <86************ @guru.mired.org >, Mike Meyer <mw*@mired.or g>

wrote:
aa**@pythoncra ft.com (Aahz) writes:
In article <86************ @guru.mired.org >, Mike Meyer <mw*@mired.or g> wrote:Here here. I find that threading typically introduces worse problems
than it purports to solve.

Threads are also good for handling blocking I/O.


Actually, this is one of the cases I was talking about. I find
it saner to convert to non-blocking I/O and use select() for
synchronizatio n. That solves the problem, without introducing any of
the headaches related to shared access and locking that come with
threads.


It may be saner, but Windows doesn't support select() for file I/O, and
Python's threading mechanisms make this very easy. If one's careful
with application design, there should be no locking problems. (Have you
actually written any threaded applications in Python?)


Hehe.. the first thing a google search on "python non-blocking io threading"
returns "Threading is Evil".

Personally I need a solution which touches this discussion. I need to run
multiple processes, which I communicate with via stdin/out, simultaneously,
and my plan was to do this with threads. Any favorite document pointers,
common traps, or something else which could be good to know?
Cheers,

Frans

Jul 18 '05 #20

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

77
5397
by: Jon Skeet [C# MVP] | last post by:
Please excuse the cross-post - I'm pretty sure I've had interest in the article on all the groups this is posted to. I've finally managed to finish my article on multi-threading - at least for the moment. I'd be *very* grateful if people with any interest in multi-threading would read it (even just bits of it - it's somewhat long to go through the whole thing!) to check for accuracy, effectiveness of examples, etc. Feel free to mail...
2
2125
by: JohnFol | last post by:
I know Windows / SQL etc can utilise multiple processors. In the good old days of coding, you simply wrote the .EXE and Windows would run it on a single processor (or a given processor for multi-processor machines) I know also that threads exist, but am having a few problems understanding how they are split across processors so here is an easy example. I have a VB.Net written application that is processor intensive. It has not been...
1
2096
by: Carl Waldbieser | last post by:
I have been considering using Python and the Reportlab library for generating PDF reports for the back-end of a web based application. The application runs most of its background tasks on a dedicated server that is Windows based (Win2K or Win2k3). The program that launches the tasks requires a COM-based interface, so I wrote a Python COM server using Mark Hammond's PythonCom libraries and import and run the reporlab modules from there. ...
2
4061
by: john | last post by:
For example, if you have a dual-processor computer, and your process runs for an hour using 100% of both CPUs, would Process.TotalProcessorTime return a TimeSpan representing one hour or two? thanks
20
5976
by: dotyet | last post by:
Hi Everyone, It would be a real big help if anyone can shed light on whether DB2 UDB 8.2 on Solaris will make optimum use of a Sun T2000 server. The server has 1 CPU with 8 cores and each core has 4 threads (32 virtual CPUs). We are exploring this option againts an equivalent performance pSeries box. But before we delve deeper, we want some basic info on that. Any suggestions would be helpful.
2
1674
by: PH | last post by:
Hi guys; I got a single processor computer, running an application that launches 2 threads. Each of these threads listens for incoming connections in a specific port, so there is a Loop . Until inside each of them.
5
1950
by: Coaster | last post by:
I am designing a process which will spawn a good number of threads and some of them will execute a c++ process which is quite memory intensive (although not multithreaded). This will run on a 2 cpu (both dual core) server. What do I need to do if anything in order to spread the workload across the cpu's / cores in order to tune it properly? I googled it and found this post but its somewhat dated and didn't know if it applied to 2.0
5
3528
by: George Maicovschi | last post by:
As multi-threading is not built in PHP I've been using a hack letting the Apache server handle the multi-threading issues, but I'm really curious of other approaches to this issue. If anyone has any ideas I'd be more than glad to hear them up and discuss them.
1
1714
by: =?Utf-8?B?QU1lcmNlcg==?= | last post by:
Sorry this is so long winded, but here goes. Following the model of http://msdn2.microsoft.com/en-us/library/system.runtime.remoting.channels.ipc.ipcchannel.aspx I made a remote object using the IpcChannel Class (vs 2005, vb, fw 2.0). Everyting works fine. The object is registered with WellKnownObjectMode.Singleton The remote object appears at the bottom of this posting. The code is deliberately obtuse to expose an issue about when...
0
9860
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
10603
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10660
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
9440
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
7838
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
7023
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5685
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
4498
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
3
3138
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.