Hi guys,
I'm sorry, I'm not sure this is the correct group to be asking this
kind of question...
I'm going to develop a software package that includes a web server
(and PHP code) , a database, and some Python code of course. I am
being strongly suggested to make it to work on a dual- or multi-core
computer, but I'm confused on how to take advantage of the multiple
CPUs.
>From what I read, I think that simply by making the package run in
several separate processes (web server, database server, Python
interpreter, etc.), and/or using multiple threads (which I will
anyway) the package should be able to use multiple CPUs.
Is my understanding correct? So actually, all I have to do is just
write my multi-threaded Python code as usual? And how is it decided
which process and which threads go to CPU 1, CPU 2, etc.? Perhaps the
BIOS?
Any advice greatly appreciated.
Andy 8 1898
Andy wrote:
I'm going to develop a software package that includes a web server
(and PHP code) , a database, and some Python code of course. I am
being strongly suggested to make it to work on a dual- or
multi-core computer,
No problem. CPython will work on any dual core CPU.
but I'm confused on how to take advantage of the multiple CPUs.
First: Use a web server that can make use of multiple cores.
Second: Use a data base that can make use of multiple cores.
Third, for using multiple cores from CPython: this is an FAQ, please
look at the mailing list archives. Whether you should optimize your
python application to use all cores strongly depends on what your
Python application actually does.
From what I read, I think that simply by making the package run in
several separate processes (web server, database server, Python
interpreter, etc.), and/or using multiple threads (which I will
anyway) the package should be able to use multiple CPUs.
CPython has the GIL (global interpreter lock). Please search for it
in the archives, it's been discussed exhaustingly.
And how is it decided which process and which threads go to CPU 1,
CPU 2, etc.? Perhaps the BIOS?
No, the kernel (i. e., Linux). The BIOS is completely out of this.
Regards,
Björn
--
BOFH excuse #388:
Bad user karma.
Andy wrote:
Hi guys,
I'm sorry, I'm not sure this is the correct group to be asking this
kind of question...
I'm going to develop a software package that includes a web server
(and PHP code) , a database, and some Python code of course. I am
being strongly suggested to make it to work on a dual- or multi-core
computer, but I'm confused on how to take advantage of the multiple
CPUs.
>>From what I read, I think that simply by making the package run in
several separate processes (web server, database server, Python
interpreter, etc.), and/or using multiple threads (which I will
anyway) the package should be able to use multiple CPUs.
Is my understanding correct? So actually, all I have to do is just
write my multi-threaded Python code as usual? And how is it decided
which process and which threads go to CPU 1, CPU 2, etc.? Perhaps the
BIOS?
Any advice greatly appreciated.
Andy
The Python interpreter is not multi-cpu aware, so using Python threads
won't work on multiple CPUs. If your tasks are CPU-bound, then fork
multiple processes. Most web servers (Apache) can handle this
automatically for you.
Andy wrote:
I'm going to develop a software package that includes a web server
(and PHP code) , a database, and some Python code of course. I am
being strongly suggested to make it to work on a dual- or multi-core
computer, but I'm confused on how to take advantage of the multiple
CPUs.
From what I read, I think that simply by making the package run in
several separate processes (web server, database server, Python
interpreter, etc.), and/or using multiple threads (which I will
anyway) the package should be able to use multiple CPUs.
Right. There is a theoretical possibility that Python's GIL
could limit thread parallelism, but in this kind of application
it should not matter in the least. Either you're I/O bound, or
there are plenty of tasks the operating system could schedule.
If you have the option to run the Python interpreter within a
web-server process, that's usually a performance winner.
Is my understanding correct? So actually, all I have to do is just
write my multi-threaded Python code as usual? And how is it decided
which process and which threads go to CPU 1, CPU 2, etc.? Perhaps the
BIOS?
The O.S. actually. A lot of really smart people have put a whole
lot of work into making the O.S. do that well.
If you usually write your Python apps multi-threaded, as I do,
that's fine. Multi-core efficiency has only a little to do with
it. Web service are, for the most part, intrinsically
parallelizable: run multiple web servers, and multiple Python
interpreters serving multiple connections.
The only hard part is shared data. Scalability is all about
the database.
--
--Bryan
Andy <fu******@gmail .comwrites:
From what I read, I think that simply by making the package run in
several separate processes (web server, database server, Python
interpreter, etc.), and/or using multiple threads (which I will
anyway) the package should be able to use multiple CPUs.
Python threading doesn't support multiple CPU's because of the GIL.
One approach to using your multiple cores is to embed a Python
interpreter into a web server that runs in multiple processes,
e.g. mod_python under Apache.
Another is write your application as a separate process (e.g. as an
FastCGI), then run multiple instances of it, connected to a concurrent
web server. For what I'm currently doing, we're using lighthttpd as
the http server and flup to connect to a set of Python FCGI's.
For that matter, if your machine is just dual core, maybe it's ok to
just run a single Python process, figuring that will run on one core
and your database/httpd will run on the other one. However, you
should figure the dual core situation won't last. Dual socket
motherboards are fairly cheap these days, so we have a number of
4-core machines (two AMD dual core cpu's); Intel is already shipping
quad core cpu's, so that puts 8 cores in a dual socket board (you can
already buy Macintoshes configured that way). Higher end server
motherboards have 4 sockets, so you have to expect that 16-core
servers will be common pretty soon.
So if you're working on a cpu-intensive application it's worth your
while figuring out how to parallelize it.
Note that the most careful concurrency stuff probably is in the
database. Serious ones already know how to use multiple CPU's.
samwyse wrote:
The Python interpreter is not multi-cpu aware, so using Python
threads won't work on multiple CPUs.
Are you sure about this?
Regards,
Björn
--
BOFH excuse #12:
dry joints on cable plug
Paul Rubin wrote:
Python threading doesn't support multiple CPU's because of the
GIL.
:s/support/take full advantage of/
Regards,
Björn
--
BOFH excuse #46:
waste water tank overflowed onto computer
Thanks guys for the suggestions.
Andy
Andy wrote:
Thanks guys for the suggestions.
Andy
It might be that you have to set the CPU affinity for your python process so
that it works at all. I had that problem on a dual core machine with
hyperthread enabled. Using taskset
( http://www.linuxcommand.org/man_pages/taskset1.html) helped solving the
problem:
taskset -c <CPU#python ...
Best regards,
wr
--
pinkrose.dhis.o rg, This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: cwahlmeier |
last post by:
Greetings,
I am running DB2 Workgroup Edition on an Intel box with two physical
CPUs. It is running Windows with hyperthreading. Thus, the operating
system thinks it has 4 cpus. I am about to upgrade to another dual CPU
box, but this one runs the new dual core chip. So, after
hyperthreading, it will look like there are 8 CPUs.
Anybody know how DB2 Workgroup Edition will react? Furthermore, will I
be in violation of my license?
|
by: webwarrior |
last post by:
Hi,
Is there a reason why we have to pay more for licensing for a different
kind of processor?
Why are we not charged for the Hyperthreading on some processors also.
If Oracle is really conserned about the low end business market (small
and medium), then they should drop their attitude on Dual Core
processors.
|
by: bruce_brodinsky |
last post by:
Don't know whether to post this on a hardware or software board, so
here goes:
I wrote a c# chess program which searches for checkmate. Now, it's
single-threaded. But I was thinking. I just got a Dell XPS dual-core
machine. If I modify the chess program to use threading, will it take
advantage of the dual-core machine? If so, is there any special code
(besides .Net threading) that I have to implement? Thanks in advance -
Bruce
|
by: malv |
last post by:
Multiprocessing has been pushed into the field as the chip
manufacturers can no longer deliver as they were used to for many
years.
The general public has been conditioned to believe that 1 + 1 = 2 but
this is in fact not that simple.
Although software manufacturers can with some effort adapt packages to
exploit these dual core architectures, the same is not true for other
development work were no large distribution of the application is...
|
by: robert |
last post by:
There is a strange freeze/crash only on dual core machines:
I have a python app (Python 2.3.5 /Pythonwin build 203 / Windows)
running with no stability problems on normal machines (Or a crash is so
rare, that absolutely nobody obverses it, though the overall majority of
users uses single core machines). Threads, network & pythonwin/win32ui
all in use.
Yet, from 3 users, _all_ using a Dual Processor System (XEON, amd x2
3800+)...
| |
by: Simon Wittber |
last post by:
I've just bought a new notebook, which has a dual core CPU.
I write cross platform games in Python, and I'd really like to be able
to use this second core (on my machine, and on user's machines) for any
new games I might write.
I know threads won't help (in CPython at least) so I'm investigating
other types of concurrency which I might be able to use. I really like
the PyLinda approach, however I need to be able to pass around all the...
|
by: robert |
last post by:
I'd like to use multiple CPU cores for selected time consuming Python computations (incl. numpy/scipy) in a frictionless manner.
Interprocess communication is tedious and out of question, so I thought about simply using a more Python interpreter instances (Py_NewInterpreter) with extra GIL in the same process.
I expect to be able to directly push around Python Object-Trees between the 2 (or more) interpreters by doing some careful locking.
...
|
by: Woody Ling |
last post by:
I am starting to config a 64 bits DB2 in IBM 595 AIX box with 2 dual
core CPU and I would like to assigned one 'processor' for one db
partition. Should I config it as a 4 nodes or 2 nodes instances? How
about other setting such as IO cleaner, Default degree etc?
|
by: Paul Sijben |
last post by:
I am running a multi-threaded python application in a dual core intel
running Ubuntu.
I am using python 2.5.1 that I compiled myself. At random points I am
getting segmentation faults (sometimes indicating a duplicate free).
Below is the backtrace of the latest segfault.
I am thinking this might be an issue related to the dual core CPU so I
am now running the app with affinity to one CPU to test this hypothesis.
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
| |
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |