473,242 Members | 1,816 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,242 software developers and data experts.

Cross platform way of finding number of processors on a machine?


Is there a way to find the number of processors on a machine (on linux/
windows/macos/cygwin) using python code (using the same code/cross
platform code)?

Oct 5 '07 #1
11 4880
On 10/4/07, John <we**********@yahoo.comwrote:
>
Is there a way to find the number of processors on a machine (on linux/
windows/macos/cygwin) using python code (using the same code/cross
platform code)?
There's no single call that will give you the same info on every
platform, but you can obviously write this yourself and switch based
on os.uname()[0] in most cases.

For Darwin, you can just use the subprocess module to call 'sysctl
hw.logicalcpu'. I'm not sure if there's a more direct way in python
to use sysctl. (hw.logicalcpu_max is what the hardware maximally
supports, but someone may have started their machine with OF blocking
some of the processors and you should probably respect that decision)

For Linux you can read /proc/cpuinfo and parse that information. Be
somewhat careful with this, however, if your processors support HT,
they will show as 2, and that may or may not be what you want. You
can deterministically parse this information out if you know which
processor families are truly multi-core, and which are HT.

For Win32, the cheap and dirty way is to read the NUMBER_OF_PROCESSORS
environment variable. 99% of the time, this will be correct, so it
might be sufficient for your purposes. There is a system call
available on windows that will net you the true number of virtual
cpus, but I don't know what it is.

I don't have a cygwin install laying around, but my guess is there's a
sysctl option there too.

One note: *all* of these methods will tell you the virtual number of
CPUs in a machine, not the physical number. There's almost no reason
why you care about the distinction between these two numbers, but if
you do, you'll have to go to great lengths to probe the actual
hardware on each platform. (And pre-WinXP, Windows doesn't actually
know the difference - all processors are presumed to be physical).

Also, Darwin and Linux will easily allow you to get the speed of the
processors, but on x86 these numbers are not the maximums due to C1E
and EIST (x86 processors from all vendors are capable of changing
speeds depending on load).

--
Nick
Oct 5 '07 #2
Nicholas Bastin wrote:
On 10/4/07, John <we**********@yahoo.comwrote:
>Is there a way to find the number of processors on a machine (on linux/
windows/macos/cygwin) using python code (using the same code/cross
platform code)?

There's no single call that will give you the same info on every
platform, but you can obviously write this yourself and switch based
on os.uname()[0] in most cases.
Second that point about not getting one cross-platform
answer. Under Windows, WMI is often the way to go for
these things:

http://msdn2.microsoft.com/en-us/library/aa394373.aspx

but, as Nicholas noted:

"""
Windows Server 2003, Windows XP, and Windows 2000:
This property is not available.
"""

since it's presumably not available in anything earlier either,
that leaves you with Vista or the early-adopter editions of the
next Windows Server product.

TJG
Oct 5 '07 #3
On 5 Okt., 09:58, Tim Golden <m...@timgolden.me.ukwrote:
Nicholas Bastin wrote:
On 10/4/07, John <weekender...@yahoo.comwrote:
Is there a way to find the number of processors on a machine (on linux/
windows/macos/cygwin) using python code (using the same code/cross
platform code)?
There's no single call that will give you the same info on every
platform, but you can obviously write this yourself and switch based
on os.uname()[0] in most cases.

Second that point about not getting one cross-platform
answer. Under Windows, WMI is often the way to go for
these things:

http://msdn2.microsoft.com/en-us/library/aa394373.aspx

but, as Nicholas noted:

"""
Windows Server 2003, Windows XP, and Windows 2000:
This property is not available.
"""

since it's presumably not available in anything earlier either,
that leaves you with Vista or the early-adopter editions of the
next Windows Server product.

TJG
Remarkable. I've a two years old Windows XP dual core notebook and
when I'm asking Python I get the correct answer:
>>import os
os.environ['NUMBER_OF_PROCESSORS']
2

However this feature might not be guaranteed by Microsoft for
arbitrary computers using their OS?

Kay

Oct 5 '07 #4
[Tim Golden]
> """
Windows Server 2003, Windows XP, and Windows 2000:
This property is not available.
"""

since it's presumably not available in anything earlier either,
that leaves you with Vista or the early-adopter editions of the
next Windows Server product.
Kay Schluehr wrote:
Remarkable. I've a two years old Windows XP dual core notebook and
when I'm asking Python I get the correct answer:
>>>import os
os.environ['NUMBER_OF_PROCESSORS']
2

However this feature might not be guaranteed by Microsoft for
arbitrary computers using their OS?
Ahem. I was referring (and not very clearly, now I look
back at my post) to a particular property within the
WMI class, not to the general concept of counting
processors.

TJG
Oct 5 '07 #5
Nicholas Bastin <ni*********@gmail.comwrote:
On 10/4/07, John <we**********@yahoo.comwrote:

Is there a way to find the number of processors on a machine (on linux/
windows/macos/cygwin) using python code (using the same code/cross
platform code)?

There's no single call that will give you the same info on every
platform, but you can obviously write this yourself and switch based
on os.uname()[0] in most cases.

For Darwin, you can just use the subprocess module to call 'sysctl
hw.logicalcpu'. I'm not sure if there's a more direct way in python
to use sysctl. (hw.logicalcpu_max is what the hardware maximally
supports, but someone may have started their machine with OF blocking
some of the processors and you should probably respect that decision)

For Linux you can read /proc/cpuinfo and parse that information. Be
somewhat careful with this, however, if your processors support HT,
they will show as 2, and that may or may not be what you want. You
can deterministically parse this information out if you know which
processor families are truly multi-core, and which are HT.
On any unix/posix system (OSX and linux should both qualify) you can use
>>import os
os.sysconf('SC_NPROCESSORS_ONLN')
2
>>>
(From my Core 2 Duo laptop running linux)

--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick
Oct 5 '07 #6
John <we**********@yahoo.comwrote:
Is there a way to find the number of processors on a machine (on linux/
windows/macos/cygwin) using python code (using the same code/cross
platform code)?
From processing <http://cheeseshop.python.org/pypi/processing/0.34:

def cpuCount():
'''
Returns the number of CPUs in the system
'''
if sys.platform == 'win32':
try:
num = int(os.environ['NUMBER_OF_PROCESSORS'])
except (ValueError, KeyError):
pass
elif sys.platform == 'darwin':
try:
num = int(os.popen('sysctl -n hw.ncpu').read())
except ValueError:
pass
else:
try:
num = os.sysconf('SC_NPROCESSORS_ONLN')
except (ValueError, OSError, AttributeError):
pass

if num >= 1:
return num
else:
raise NotImplementedError

--
Lawrence, oluyede.org - neropercaso.it
"It is difficult to get a man to understand
something when his salary depends on not
understanding it" - Upton Sinclair
Oct 6 '07 #7
On 6 Okt, 12:45, ra...@dot.com (Lawrence Oluyede) wrote:
>
From processing <http://cheeseshop.python.org/pypi/processing/0.34:
[...]
num = os.sysconf('SC_NPROCESSORS_ONLN')
It's interesting what new (or obscure) standard library functions (and
system functions) can be discovered through these kinds of
discussions. However, this one seems to report the number of "virtual
CPUs" in situations like mine where the CPU supports hyperthreading,
as Nicholas Bastin says. One way to get the real number of CPUs (on
Linux-based systems) is to parse /proc/cpuinfo, and to count the
number of distinct "physical id" values.
>From the experiments I've done with pprocess [1], pretending that a
hyperthreading "virtual CPU" is as good as a real CPU (or CPU core)
tends to be counter-productive: the performance benefits in moving
from the utilisation of one to two "virtual CPUs" are virtually non-
existent.

Paul

[1] http://www.python.org/pypi/pprocess

Oct 6 '07 #8
Paul Boddie <pa**@boddie.org.ukwrote:
From the experiments I've done with pprocess [1], pretending that a
hyperthreading "virtual CPU" is as good as a real CPU (or CPU core)
tends to be counter-productive: the performance benefits in moving
from the utilisation of one to two "virtual CPUs" are virtually non-
existent.
We came the same conclusion at work. One of my colleagues has a machine
with HyperThreading and he noticed no real performance increase with the
processing library altough on my machine (which is a dual Core 2 Duo)
the performance boost was definitely there :-)

The good side of this is hyperthreading with parallel processes is like
using a mono-CPU machine so we don't have to take into account it like
if it was an entirely different kind of architecture.

--
Lawrence, oluyede.org - neropercaso.it
"It is difficult to get a man to understand
something when his salary depends on not
understanding it" - Upton Sinclair
Oct 6 '07 #9
Lawrence Oluyede <ra***@dot.comwrote:
John <we**********@yahoo.comwrote:
Is there a way to find the number of processors on a machine (on linux/
windows/macos/cygwin) using python code (using the same code/cross
platform code)?

From processing <http://cheeseshop.python.org/pypi/processing/0.34:

def cpuCount():
'''
Returns the number of CPUs in the system
'''
if sys.platform == 'win32':
try:
num = int(os.environ['NUMBER_OF_PROCESSORS'])
except (ValueError, KeyError):
pass
elif sys.platform == 'darwin':
try:
num = int(os.popen('sysctl -n hw.ncpu').read())
except ValueError:
pass
else:
try:
num = os.sysconf('SC_NPROCESSORS_ONLN')
except (ValueError, OSError, AttributeError):
pass

if num >= 1:
return num
else:
raise NotImplementedError

There is a bug in that code...

NotImplementedError will never be raised because num won't have been
set. It will raise "UnboundLocalError: local variable 'num'
referenced before assignment" instead

--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick
Oct 6 '07 #10
On 10/6/07, Lawrence Oluyede <ra***@dot.comwrote:
John <we**********@yahoo.comwrote:
Is there a way to find the number of processors on a machine (on linux/
windows/macos/cygwin) using python code (using the same code/cross
platform code)?
From processing <http://cheeseshop.python.org/pypi/processing/0.34:
A few reiterated notes inline, from my previous message.
def cpuCount():
'''
Returns the number of CPUs in the system
'''
if sys.platform == 'win32':
try:
num = int(os.environ['NUMBER_OF_PROCESSORS'])
The user can do bad things to this environment variable, but it's
probably ok most of the time. (Hey, they change it, they pay the
consequences).
else:
try:
num = os.sysconf('SC_NPROCESSORS_ONLN')
This is really bad on linux. You really want to parse /proc/cpuinfo
because HT 'cpus' are almost useless, and can be bad in situations
where you try to treat them as general purpose cpus.

--
Nick
Oct 8 '07 #11
Nicholas Bastin <ni*********@gmail.comwrote:
This is really bad on linux. You really want to parse /proc/cpuinfo
because HT 'cpus' are almost useless, and can be bad in situations
where you try to treat them as general purpose cpus.
I'm not the author of the library. I think you should address these bug
reports elsewhere...

--
Lawrence, oluyede.org - neropercaso.it
"It is difficult to get a man to understand
something when his salary depends on not
understanding it" - Upton Sinclair
Oct 8 '07 #12

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Paul Tremblay | last post by:
Hi All, I am writing an application framework which consists of classes mainApp and childApp (parent, child resp). My intention is to make childApp a base class to be extended by applications...
1
by: Ian | last post by:
I'm trying to restore a database backup taken from a server running DB2 8.1 FP6a on Windows 2000 Server on to a new machine running Windows Server 2003 (also 8.1 FP6a), but I'm getting the...
35
by: Alex Martelli | last post by:
Having fixed a memory leak (not the leak of a Python reference, some other stuff I wasn't properly freeing in certain cases) in a C-coded extension I maintain, I need a way to test that the leak is...
42
by: =?ISO-8859-1?Q?Tom=E1s_=D3_h=C9ilidhe?= | last post by:
I'm currently writing a program and I've got in mind to keep it as portable as possible. In particular I want it to run on Linux and Windows, but I'm also keeping an open mind to any machine that...
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.