473,840 Members | 1,670 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

multi threading in multi processor (computer)

Hello,

Is anyone has experiance in running python code to run multi thread
parallel in multi processor. Is it possible ?

Can python manage which cpu shoud do every thread?

Sincerely Yours,
Pujo

Jul 18 '05
37 4911
In article <ma************ *************** ************@py thon.org>,
Frans Englich <fr***********@ telia.com> wrote:

Personally I need a solution which touches this discussion. I need to run
multiple processes, which I communicate with via stdin/out, simultaneously,
and my plan was to do this with threads. Any favorite document pointers,
common traps, or something else which could be good to know?


Threads and forks tend to be problematic. This is one case I'd recommend
against threads.
--
Aahz (aa**@pythoncra ft.com) <*> http://www.pythoncraft.com/

"The joy of coding Python should be in seeing short, concise, readable
classes that express a lot of action in a small amount of clear code --
not in reams of trivial code that bores the reader to death." --GvR
Jul 18 '05 #21
Mike Meyer wrote:
aa**@pythoncraf t.com (Aahz) writes:
Threads are also good for handling blocking I/O.


Actually, this is one of the cases I was talking about. I find it
saner to convert to non-blocking I/O and use select() for
synchronization . That solves the problem, without introducing any of
the headaches related to shared access and locking that come with
threads.


Use a communicating sequential processes model for the threading and you don't
have many data synchronisation problems because you have barely any shared
access - no application data is ever shared between threads, they only send
messages to each other via message queues. Most threads simply block on their
incoming message queue permanently. Those doing blocking I/O set an appropriate
timeout on the I/O call so they can check for messages occasionally.

Conveniently, you end up with an architecture that supports switching to
multiple processes, or even multiple machines just by changing the transport
mechanism used by the message system.

(We did exactly this for a GUI application - detached the GUI so it talked to a
server via CORBA instead of via direct DLL calls. This meant the server could be
ported to a different platform without having to port the far more platform
specific GUI. This would have been much harder if we weren't already using a CSP
model for communication between different parts of the system)

Cheers,
Nick.

--
Nick Coghlan | nc******@email. com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net
Jul 18 '05 #22
Mike Meyer wrote:
aa**@pythoncraf t.com (Aahz) writes:

In article <86************ @guru.mired.org >, Mike Meyer <mw*@mired.or g> wrote:
Here here. I find that threading typically introduces worse problems
than it purports to solve.


Threads are also good for handling blocking I/O.

Actually, this is one of the cases I was talking about. I find it
saner to convert to non-blocking I/O and use select() for
synchronization . That solves the problem, without introducing any of
the headaches related to shared access and locking that come with
threads.


This whole tangent to the original thread intrigues me - I've found that if
you're going to use threads in any language, Python is the one to use because
the GIL reduces so many of the problems common to multithreaded programming (I'm
not a huge fan of the GIL, but its presence effectively prevents a pure Python
multithreaded app from corrupting the interpreter, which is especially handy for
those just learning Python or programming).

I've done a lot of networking applications using select/poll (usually for
performance reasons) and found that going that route *can* in some cases
simplify things but it requires looking at the problem differently, often from
perspectives that seem unnatural to me - it's not just an implementation detail
but one you have to consider during design.

One nice thing about using threads is that components of your application that
are logically separate can remain separate in the code as well - the
implementations don't have to be tied together at some common dispatch loop, and
a failure to be completely non-blocking in one component doesn't necessarily
spell disaster for the entire app (I've had apps in production where one thread
would die or get hung but I was relieved to find out that the main service
remained available).

Another related benefit is that a lot of application state is implicitly and
automatically managed by your local variables when the task is running in a
separate thread, whereas other approaches often end up forcing you to think in
terms of a state machine when you don't really care* and as a by-product you
have to [semi-]manually track the state and state transitions - for some
problems this is fine, for others it's downright tedious.

Anyway, if someone doesn't know about alternatives to threads, then that's a
shame as other approaches have their advantages (often including a certain
elegance that is just darn *cool*), but I wouldn't shy away from threads too
much either - especially in Python.

-Dave

* Simple case in point: a non-blocking logging facility. In Python you can just
start up a thread that pops strings off a Queue object and writes them to an
open file. A non-threaded version is more complicated to implement, debug, and
maintain.
Jul 18 '05 #23
Quoth Dave Brueck <da**@pythonapo crypha.com>:
....
| Another related benefit is that a lot of application state is implicitly and
| automatically managed by your local variables when the task is running in a
| separate thread, whereas other approaches often end up forcing you to think in
| terms of a state machine when you don't really care* and as a by-product you
| have to [semi-]manually track the state and state transitions - for some
| problems this is fine, for others it's downright tedious.

I don't know if the current Stackless implementation has regained any
of this ground, but at least of historical interest here, the old one's
ability to interrupt, store and resume a computation could be used to

As you may know, it used to be, in Stackless Python, that you could have
both. Your function would suspend itself, the select loop would resume it,
for something like serialized threads. (The newer version of Stackless
lost this continuation feature, but for all I know there may be new
features that regain some of that ground.)

I put that together with real OS threads once, where the I/O loop was a
message queue instead of select. A message queueing multi-threaded
architecture can end up just as much a state transition game.

I like threads when they're used in this way, as application components
that manage some device-like thing like a socket or a graphic user interface
window, interacting through messages. Even then, though, there tend to
be a lot of undefined behaviors in events like termination of the main
thread, receipt of signals, etc.

Donn Cave, do**@drizzle.co m
Jul 18 '05 #24
Irmen de Jong wrote:
the GIL must die.

I couldn't resist:
http://www.razorvine.net/img/GIL.jpg


Neither could I:

http://ecritters.biz/diegil.png

(In case it's not entirely obvious, the stick figure just slices the GIL
into two pieces with his sword, causing its blood to splatter on the wall.)
Jul 18 '05 #25
In article <ma************ *************** ************@py thon.org>,
Dave Brueck <da**@pythonapo crypha.com> wrote:
Donn Cave wrote:

[... re stackless inside-out event loop ]
I put that together with real OS threads once, where the I/O loop was a
message queue instead of select. A message queueing multi-threaded
architecture can end up just as much a state transition game.


Definitely, but for many cases it does not - having each thread represent a
distinct "worker" that pops some item of work off one queue, processes it,
and
puts it on another queue can really simplify things. Often this maps to
real-world objects quite well, additional steps can be inserted or removed
easily (and dynamically), and each worker can be developed, tested, and
debugged
independently.


Well, one of the things that makes the world interesting is
how many different universes we seem to be coming from, but
in mine, when I have divided an application into several
thread components, about the second time I need to send a
message from one thread to another, the sender needs something
back in return, as in T2 = from_thread_B(T 1). At this point,
our conventional procedural model breaks up along a state fault,
so to speak, like

...
to_thread_B(T1)
return

def continue_from_T 1(T1, T2):
...

So, yeah, now I have a model where each thread pops, processes
and pushes messages, but only because my program spent the night
in Procrustes' inn, not because it was a natural way to write
the computation. In a procedural language, anyway - there are
interesting alternatives, in particular a functional language
called O'Haskell that models threads in a "reactive object"
construct, an odd but elegant mix of state machine and pure
functional programming, but it's kind of a research project
and I know of nothing along these lines that's really supported
today.

Donn Cave, do**@u.washingt on.edu
Jul 18 '05 #26
aa**@pythoncraf t.com (Aahz) writes:
(Have you
actually written any threaded applications in Python?)


Yes. Have you ever asked a polite question?

<mike
--
Mike Meyer <mw*@mired.or g> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
Jul 18 '05 #27
In article <86************ @guru.mired.org >, Mike Meyer <mw*@mired.or g> wrote:
aa**@pythoncra ft.com (Aahz) writes:

(Have you
actually written any threaded applications in Python?)


Yes. Have you ever asked a polite question?


Yes. I just get a bit irritated with some of the standard lines that
people use.
--
Aahz (aa**@pythoncra ft.com) <*> http://www.pythoncraft.com/

"The joy of coding Python should be in seeing short, concise, readable
classes that express a lot of action in a small amount of clear code --
not in reams of trivial code that bores the reader to death." --GvR
Jul 18 '05 #28
aa**@pythoncraf t.com (Aahz) writes:
[phr] The day is coming when even cheap computers have multiple cpu's.
See hyperthreading and the coming multi-core P4's, and the finally
announced Cell processor.

Conclusion: the GIL must die.
It's not clear to what extent these processors will perform well with
shared memory space. One of the things I remember most about Bruce
Eckel's discussions of Java and threading is just how broken Java's
threading model is in certain respects when it comes to CPU caches
failing to maintain cache coherency.


Um??? I'm not experienced with multiprocessors but I thought that
maintaining cache coherency was a requirement. What's the deal? If
coherency isn't maintained, is it really multiprocessing ?
It's always going to be true that getting fully scaled performance
will require more CPUs with non-shared memory -- that's going to
mean IPC with multiple processes instead of threads.


But unless you use shared memory, the context switch overhead from
IPC becomes a bad bottleneck.

See http://poshmodule.sourceforge.net/posh/html/node1.html
for an interesting scheme of working around the GIL by spreading
naturally multi-threaded applications into multiple processes
(using shared memory). It would simplify things a lot if you could
just use threads.
Jul 18 '05 #29
Leif K-Brooks wrote:
Irmen de Jong wrote:
the GIL must die.

I couldn't resist:
http://www.razorvine.net/img/GIL.jpg

Neither could I:

http://ecritters.biz/diegil.png

(In case it's not entirely obvious, the stick figure just slices the GIL
into two pieces with his sword, causing its blood to splatter on the wall.)


Naah.
What about:
http://www.razorvine.net/img/killGIL.jpg

--Irmen
Jul 18 '05 #30

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

77
5397
by: Jon Skeet [C# MVP] | last post by:
Please excuse the cross-post - I'm pretty sure I've had interest in the article on all the groups this is posted to. I've finally managed to finish my article on multi-threading - at least for the moment. I'd be *very* grateful if people with any interest in multi-threading would read it (even just bits of it - it's somewhat long to go through the whole thing!) to check for accuracy, effectiveness of examples, etc. Feel free to mail...
2
2125
by: JohnFol | last post by:
I know Windows / SQL etc can utilise multiple processors. In the good old days of coding, you simply wrote the .EXE and Windows would run it on a single processor (or a given processor for multi-processor machines) I know also that threads exist, but am having a few problems understanding how they are split across processors so here is an easy example. I have a VB.Net written application that is processor intensive. It has not been...
1
2096
by: Carl Waldbieser | last post by:
I have been considering using Python and the Reportlab library for generating PDF reports for the back-end of a web based application. The application runs most of its background tasks on a dedicated server that is Windows based (Win2K or Win2k3). The program that launches the tasks requires a COM-based interface, so I wrote a Python COM server using Mark Hammond's PythonCom libraries and import and run the reporlab modules from there. ...
2
4062
by: john | last post by:
For example, if you have a dual-processor computer, and your process runs for an hour using 100% of both CPUs, would Process.TotalProcessorTime return a TimeSpan representing one hour or two? thanks
20
5976
by: dotyet | last post by:
Hi Everyone, It would be a real big help if anyone can shed light on whether DB2 UDB 8.2 on Solaris will make optimum use of a Sun T2000 server. The server has 1 CPU with 8 cores and each core has 4 threads (32 virtual CPUs). We are exploring this option againts an equivalent performance pSeries box. But before we delve deeper, we want some basic info on that. Any suggestions would be helpful.
2
1674
by: PH | last post by:
Hi guys; I got a single processor computer, running an application that launches 2 threads. Each of these threads listens for incoming connections in a specific port, so there is a Loop . Until inside each of them.
5
1950
by: Coaster | last post by:
I am designing a process which will spawn a good number of threads and some of them will execute a c++ process which is quite memory intensive (although not multithreaded). This will run on a 2 cpu (both dual core) server. What do I need to do if anything in order to spread the workload across the cpu's / cores in order to tune it properly? I googled it and found this post but its somewhat dated and didn't know if it applied to 2.0
5
3528
by: George Maicovschi | last post by:
As multi-threading is not built in PHP I've been using a hack letting the Apache server handle the multi-threading issues, but I'm really curious of other approaches to this issue. If anyone has any ideas I'd be more than glad to hear them up and discuss them.
1
1714
by: =?Utf-8?B?QU1lcmNlcg==?= | last post by:
Sorry this is so long winded, but here goes. Following the model of http://msdn2.microsoft.com/en-us/library/system.runtime.remoting.channels.ipc.ipcchannel.aspx I made a remote object using the IpcChannel Class (vs 2005, vb, fw 2.0). Everyting works fine. The object is registered with WellKnownObjectMode.Singleton The remote object appears at the bottom of this posting. The code is deliberately obtuse to expose an issue about when...
0
9861
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
10605
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10665
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10301
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
9444
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
7839
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
7023
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5685
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
4498
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.