Help me use my Dual Core CPU!

[Simon Wittber]

I've just bought a new notebook, which has a dual core CPU.

I write cross platform games in Python, and I'd really like to be able
to use this second core (on my machine, and on user's machines) for any
new games I might write.

I know threads won't help (in CPython at least) so I'm investigating
other types of concurrency which I might be able to use. I really like
the PyLinda approach, however I need to be able to pass around all the
simple python types, which PyLinda won't help me with. Also, PyLinda is
more focused on distributing computing; I really only want to have 2
processes cooperating (or 4, if I had 4 CPUs/cores etc).

Is there any cross platform way to share python objects across
processes? (I've found POSH, but it's old, and doesn't appear to be
maintained). I could implement my own object space using shared memory,
but from what I can see, this is not available on Win32.

Are there any other concurrency options I've not discovered yet?
-Sw.

[Paul Rubin]

"Simon Wittber" <si**********@g mail.comwrites:

Are there any other concurrency options I've not discovered yet?

I've been wondering about the different Python MPI bindings that are
out there, and whether they might make sense for general purpose
concurrency when they were designed mostly for parallel numerical
computation. Better MPI libraries should at least be able to
communicate by shared memory--I don't know if Pylinda does that.

[Michael]

Simon Wittber wrote:

I've just bought a new notebook, which has a dual core CPU.

I write cross platform games in Python, and I'd really like to be able
to use this second core (on my machine, and on user's machines) for any
new games I might write.

I know threads won't help (in CPython at least) so I'm investigating
other types of concurrency which I might be able to use. I really like
the PyLinda approach, however I need to be able to pass around all the
simple python types, which PyLinda won't help me with. Also, PyLinda is
more focused on distributing computing; I really only want to have 2
processes cooperating (or 4, if I had 4 CPUs/cores etc).

Is there any cross platform way to share python objects across
processes? (I've found POSH, but it's old, and doesn't appear to be
maintained). I could implement my own object space using shared memory,
but from what I can see, this is not available on Win32.

Are there any other concurrency options I've not discovered yet?

We *haven't* implemented process based components for Kamaelia yet, however
if a process component base class (which is something we want to do) was
created that might serve as a possiblity. If you're interested in this I
could chat with you about the ideas we've had about how this would work
(#kamaelia on freenode or here). (We are planning on doing this when we
manage to get sufficient round tuits).

It's probably worth noting that we wouldn't be thinking of using shared
objects, but piping data between the two processes - with the various
options including pickling to memory mapped files (however there's security
issues there aside from anything else...).

Also, Paul Boddie posted a module for parallel systems a while back as well
which might be useful (at least for ideas):
* http://cheeseshop.python.org/pypi/parallel

I'd be interested in helping out BTW :)
Michael.
--
http://kamaelia.sourceforge.net/Home
http://yeoldeclue.com/blog

[John Henry]

I don't know what CPython is but I have developed a Python application
under Windows that utilize the Dure Core CPU when it's present.

I don't know that I can say for sure that "threads won't help". Have
you done some testing before using other approaches to see if it indeed
won't help?
Simon Wittber wrote:

I've just bought a new notebook, which has a dual core CPU.

I write cross platform games in Python, and I'd really like to be able
to use this second core (on my machine, and on user's machines) for any
new games I might write.

I know threads won't help (in CPython at least) so I'm investigating
other types of concurrency which I might be able to use. I really like
the PyLinda approach, however I need to be able to pass around all the
simple python types, which PyLinda won't help me with. Also, PyLinda is
more focused on distributing computing; I really only want to have 2
processes cooperating (or 4, if I had 4 CPUs/cores etc).

Is there any cross platform way to share python objects across
processes? (I've found POSH, but it's old, and doesn't appear to be
maintained). I could implement my own object space using shared memory,
but from what I can see, this is not available on Win32.

Are there any other concurrency options I've not discovered yet?
-Sw.

[Brian L. Troutwine]

John Henry wrote:

I don't know what CPython is but I have developed a Python application
under Windows that utilize the Dure Core CPU when it's present.

It's the default python implementation, the one you find at python.org.
It happens to be written in C. Other python implementations included
IronPython, PyPy and Jython in .Net, CPython and Java, respectively.

[Simon Wittber]

Michael wrote:

Also, Paul Boddie posted a module for parallel systems a while back as well
which might be useful (at least for ideas):
* http://cheeseshop.python.org/pypi/parallel

I've checked this out, it looks like a good idea which I could build
further on.

I've just noticed that os.fork is not available on Win32. Ouch.

Does that mean there is _no_ way for a single Python program to use
multiple CPU/core systems on Windows? (other than writing an extension
module in C (which is completely unacceptable for me!))

-Sw.

[Paul Rubin]

"Simon Wittber" <si**********@g mail.comwrites:

I've just noticed that os.fork is not available on Win32. Ouch.

Use the subprocess module.

[Simon Wittber]

Paul Rubin wrote:

"Simon Wittber" <si**********@g mail.comwrites:

I've just noticed that os.fork is not available on Win32. Ouch.

Use the subprocess module.

I can't see how subprocess.Pope n can replace a fork. Using a manually
started process is not really viable, as it does not automatically
share pre-built (read-only) data between the processes. If it can, I'd
really like to know how...

Yikes. This is a bummer. The conclusion seems to be, I cannot use any
common cross platform, true concurrency strategies in my games. On top
of that, I can't really use any form of concurrency on Win32.

Lets hope we get some super fast SMP friendly backends for PyPy sooner
rather than later!

-Sw.

[Paul Rubin]

"Simon Wittber" <si**********@g mail.comwrites:

I can't see how subprocess.Pope n can replace a fork. Using a manually
started process is not really viable, as it does not automatically
share pre-built (read-only) data between the processes. If it can, I'd
really like to know how...

Either with sockets or mmap.

[Paul Boddie]

Simon Wittber wrote:

Michael wrote:

Also, Paul Boddie posted a module for parallel systems a while back as well
which might be useful (at least for ideas):
* http://cheeseshop.python.org/pypi/parallel

I've checked this out, it looks like a good idea which I could build
further on.

Feel free to expand on what I've done. The tricky part was making some
kind of communications mechanism with asynchronous properties, and
although I suppose I could have used asyncore, Medusa or Twisted, I was
interested in the exercise of learning more about the poll system call
(and without adding dependencies which dwarf the pprocess module
itself).

I've just noticed that os.fork is not available on Win32. Ouch.

Sorry! My motivation was to support operating systems that I personally
care about and which also have solutions for transparent process
migration, which I believe could be used in conjunction with the
pprocess module for better-than-thread parallelisation . In other words,
you get processes running independently on potentially many CPUs with
only the communications as overhead, and with access to shared data
either through the communications channels or via global variables
which are read only due to the nature of the fork semantics. This
doesn't necessarily play well with threaded-style programs wanting to
modify huge numbers of shared objects, but I'd argue that the benefits
of parallelisation (for performance) are somewhat reduced in such
programs anyway.

Does that mean there is _no_ way for a single Python program to use
multiple CPU/core systems on Windows? (other than writing an extension
module in C (which is completely unacceptable for me!))

Rumour has it that recent versions of Windows provide fork-like
semantics through a system call. Your mission is to integrate this
transparently into the standard library's os.fork function. ;-)

Paul