473,394 Members | 1,517 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,394 software developers and data experts.

Development for dual core machine

Hi guys,

I'm sorry, I'm not sure this is the correct group to be asking this
kind of question...

I'm going to develop a software package that includes a web server
(and PHP code) , a database, and some Python code of course. I am
being strongly suggested to make it to work on a dual- or multi-core
computer, but I'm confused on how to take advantage of the multiple
CPUs.
>From what I read, I think that simply by making the package run in
several separate processes (web server, database server, Python
interpreter, etc.), and/or using multiple threads (which I will
anyway) the package should be able to use multiple CPUs.

Is my understanding correct? So actually, all I have to do is just
write my multi-threaded Python code as usual? And how is it decided
which process and which threads go to CPU 1, CPU 2, etc.? Perhaps the
BIOS?

Any advice greatly appreciated.
Andy

Aug 19 '07 #1
8 1880
Andy wrote:
I'm going to develop a software package that includes a web server
(and PHP code) , a database, and some Python code of course. I am
being strongly suggested to make it to work on a dual- or
multi-core computer,
No problem. CPython will work on any dual core CPU.
but I'm confused on how to take advantage of the multiple CPUs.
First: Use a web server that can make use of multiple cores.

Second: Use a data base that can make use of multiple cores.

Third, for using multiple cores from CPython: this is an FAQ, please
look at the mailing list archives. Whether you should optimize your
python application to use all cores strongly depends on what your
Python application actually does.
From what I read, I think that simply by making the package run in
several separate processes (web server, database server, Python
interpreter, etc.), and/or using multiple threads (which I will
anyway) the package should be able to use multiple CPUs.
CPython has the GIL (global interpreter lock). Please search for it
in the archives, it's been discussed exhaustingly.
And how is it decided which process and which threads go to CPU 1,
CPU 2, etc.? Perhaps the BIOS?
No, the kernel (i. e., Linux). The BIOS is completely out of this.

Regards,
Björn

--
BOFH excuse #388:

Bad user karma.

Aug 19 '07 #2
Andy wrote:
Hi guys,

I'm sorry, I'm not sure this is the correct group to be asking this
kind of question...

I'm going to develop a software package that includes a web server
(and PHP code) , a database, and some Python code of course. I am
being strongly suggested to make it to work on a dual- or multi-core
computer, but I'm confused on how to take advantage of the multiple
CPUs.
>>From what I read, I think that simply by making the package run in
several separate processes (web server, database server, Python
interpreter, etc.), and/or using multiple threads (which I will
anyway) the package should be able to use multiple CPUs.

Is my understanding correct? So actually, all I have to do is just
write my multi-threaded Python code as usual? And how is it decided
which process and which threads go to CPU 1, CPU 2, etc.? Perhaps the
BIOS?

Any advice greatly appreciated.
Andy
The Python interpreter is not multi-cpu aware, so using Python threads
won't work on multiple CPUs. If your tasks are CPU-bound, then fork
multiple processes. Most web servers (Apache) can handle this
automatically for you.
Aug 19 '07 #3
Andy wrote:
I'm going to develop a software package that includes a web server
(and PHP code) , a database, and some Python code of course. I am
being strongly suggested to make it to work on a dual- or multi-core
computer, but I'm confused on how to take advantage of the multiple
CPUs.

From what I read, I think that simply by making the package run in
several separate processes (web server, database server, Python
interpreter, etc.), and/or using multiple threads (which I will
anyway) the package should be able to use multiple CPUs.
Right. There is a theoretical possibility that Python's GIL
could limit thread parallelism, but in this kind of application
it should not matter in the least. Either you're I/O bound, or
there are plenty of tasks the operating system could schedule.

If you have the option to run the Python interpreter within a
web-server process, that's usually a performance winner.
Is my understanding correct? So actually, all I have to do is just
write my multi-threaded Python code as usual? And how is it decided
which process and which threads go to CPU 1, CPU 2, etc.? Perhaps the
BIOS?
The O.S. actually. A lot of really smart people have put a whole
lot of work into making the O.S. do that well.

If you usually write your Python apps multi-threaded, as I do,
that's fine. Multi-core efficiency has only a little to do with
it. Web service are, for the most part, intrinsically
parallelizable: run multiple web servers, and multiple Python
interpreters serving multiple connections.

The only hard part is shared data. Scalability is all about
the database.
--
--Bryan
Aug 19 '07 #4
Andy <fu******@gmail.comwrites:
From what I read, I think that simply by making the package run in
several separate processes (web server, database server, Python
interpreter, etc.), and/or using multiple threads (which I will
anyway) the package should be able to use multiple CPUs.
Python threading doesn't support multiple CPU's because of the GIL.

One approach to using your multiple cores is to embed a Python
interpreter into a web server that runs in multiple processes,
e.g. mod_python under Apache.

Another is write your application as a separate process (e.g. as an
FastCGI), then run multiple instances of it, connected to a concurrent
web server. For what I'm currently doing, we're using lighthttpd as
the http server and flup to connect to a set of Python FCGI's.

For that matter, if your machine is just dual core, maybe it's ok to
just run a single Python process, figuring that will run on one core
and your database/httpd will run on the other one. However, you
should figure the dual core situation won't last. Dual socket
motherboards are fairly cheap these days, so we have a number of
4-core machines (two AMD dual core cpu's); Intel is already shipping
quad core cpu's, so that puts 8 cores in a dual socket board (you can
already buy Macintoshes configured that way). Higher end server
motherboards have 4 sockets, so you have to expect that 16-core
servers will be common pretty soon.

So if you're working on a cpu-intensive application it's worth your
while figuring out how to parallelize it.

Note that the most careful concurrency stuff probably is in the
database. Serious ones already know how to use multiple CPU's.
Aug 19 '07 #5
samwyse wrote:
The Python interpreter is not multi-cpu aware, so using Python
threads won't work on multiple CPUs.
Are you sure about this?

Regards,
Björn

--
BOFH excuse #12:

dry joints on cable plug

Aug 19 '07 #6
Paul Rubin wrote:
Python threading doesn't support multiple CPU's because of the
GIL.
:s/support/take full advantage of/

Regards,
Björn

--
BOFH excuse #46:

waste water tank overflowed onto computer

Aug 19 '07 #7
Thanks guys for the suggestions.

Andy

Aug 20 '07 #8
Andy wrote:
Thanks guys for the suggestions.

Andy
It might be that you have to set the CPU affinity for your python process so
that it works at all. I had that problem on a dual core machine with
hyperthread enabled. Using taskset
(http://www.linuxcommand.org/man_pages/taskset1.html) helped solving the
problem:

taskset -c <CPU#python ...

Best regards,
wr
--
pinkrose.dhis.org,
Aug 20 '07 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
by: cwahlmeier | last post by:
Greetings, I am running DB2 Workgroup Edition on an Intel box with two physical CPUs. It is running Windows with hyperthreading. Thus, the operating system thinks it has 4 cpus. I am about to...
2
by: webwarrior | last post by:
Hi, Is there a reason why we have to pay more for licensing for a different kind of processor? Why are we not charged for the Hyperthreading on some processors also. If Oracle is really...
2
by: bruce_brodinsky | last post by:
Don't know whether to post this on a hardware or software board, so here goes: I wrote a c# chess program which searches for checkmate. Now, it's single-threaded. But I was thinking. I just got...
6
by: malv | last post by:
Multiprocessing has been pushed into the field as the chip manufacturers can no longer deliver as they were used to for many years. The general public has been conditioned to believe that 1 + 1 =...
2
by: robert | last post by:
There is a strange freeze/crash only on dual core machines: I have a python app (Python 2.3.5 /Pythonwin build 203 / Windows) running with no stability problems on normal machines (Or a crash is...
23
by: Simon Wittber | last post by:
I've just bought a new notebook, which has a dual core CPU. I write cross platform games in Python, and I'd really like to be able to use this second core (on my machine, and on user's machines)...
62
by: robert | last post by:
I'd like to use multiple CPU cores for selected time consuming Python computations (incl. numpy/scipy) in a frictionless manner. Interprocess communication is tedious and out of question, so I...
15
by: Woody Ling | last post by:
I am starting to config a 64 bits DB2 in IBM 595 AIX box with 2 dual core CPU and I would like to assigned one 'processor' for one db partition. Should I config it as a 4 nodes or 2 nodes...
3
by: Paul Sijben | last post by:
I am running a multi-threaded python application in a dual core intel running Ubuntu. I am using python 2.5.1 that I compiled myself. At random points I am getting segmentation faults (sometimes...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.