473,287 Members | 1,419 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,287 software developers and data experts.

multiprocessing module (PEP 371)

I sometimes read python-dev, but never contribute. So I'll post my
rant here instead.

I completely support adding this module to the standard lib. Get it in
as soon as possible, regardless of PEP deadlines or whatever.

I don't see pyprocessing as a drop-in replacement for the threading
module. Multi-threading and multi-processing code tend to be
different, unless something like mutable objects in shared memory is
used as well (cf. Python Shared Objects). If this limitation can
educate Python programmers to use queues instead of locks and mutable
objects, even multi-threaded Python programs may actually benefit.
Some API differences between threading and multiprocessing do not
matter. Programmers should not consider processes as a drop-in
replacement for threads.

One limitation not discussed on python-dev is the lack of fork on
Win32. This makes the pyprocessing module particularly inefficient at
creating processes on this platform, as it depends on serializing
(pickling and de-pickling) a lot of Python objects. Even a non-COWfork
would be preferred. I will strongly suggest something is done to add
support for os.fork to Python on Windows. Either create a full cow
fork using ZwCreateProcess (ntdll.dll does support COWforking, but
Win32 API does not expose it), or do the same as Cygwin is doing to
fork a process without COW. Although a non-cow fork a la Cygwin is not
as efficient as a fork on Linux/FreeBSD/Unix, it is still better than
what pyprocessing is doing.


Jun 27 '08 #1
6 1611
On 4 Jun, 20:06, sturlamolden <sturlamol...@yahoo.nowrote:
>
Even a non-COWfork
would be preferred. I will strongly suggest something is done to add
support for os.fork to Python on Windows. Either create a full cow
fork using ZwCreateProcess (ntdll.dll does support COWforking, but
Win32 API does not expose it), or do the same as Cygwin is doing to
fork a process without COW. Although a non-cow fork a la Cygwin is not
as efficient as a fork on Linux/FreeBSD/Unix, it is still better than
what pyprocessing is doing.
You seem to know more about this matter than the average person, I
would wager, so it might be an idea if you more than "strongly
suggest" something. ;-) I've looked at this situation briefly, I've
seen the different Cygwin-based techniques, and I've even gone as far
to investigate whether it's possible to write the necessary code using
the mingw32 stuff, although I don't think it actually worked when I
tested the executable on Windows. COW (copy-on-write, for those still
thinking that we're talking about dairy products) would be pretty
desirable if it's feasible, though.

Having said all this, I don't care about Windows myself, and my own
contribution to the collection of available libraries in this domain
has never been targeted at standard library adoption (nor thread API
compatibility) and thus has no need to run on Windows without Cygwin.

Paul
Jun 27 '08 #2
On Jun 4, 11:29 pm, Paul Boddie <p...@boddie.org.ukwrote:
tested the executable on Windows. COW (copy-on-write, for those still
thinking that we're talking about dairy products) would be pretty
desirable if it's feasible, though.
There is a well known C++ implementation of cow-fork on Windows, which
I have slightly modified and ported to C. But as the new WDK (Windows
driver kit) headers are full of syntax errors, the compiler choke on
it. :( I am seriously considering re-implementing the whole cow fork
in pure Python using ctypes.

If you pass NULL as section handle to ZwCreateProcess (or
NtCreateProcess) you do get a rudimentary cow fork. But the new
process image has no context and no active threads. The NT kernel is
designed to support several subsystems. Both the OS/2 and SUA
subsystems provide a functional COW fork, but the Win32 subsystem do
not expose the functionality. I honestly don't understand why, but
maybe it is backwards compatibility that prevents it (it's backlog
goes back to DOS, in which forking was impossible due to single-
tasking.)

But anyway ... what I am trying to say is that pyprocessing is
somewhat inefficient (and limited) on Windows due to lack of a fork
(cow or not).

Jun 27 '08 #3
sturlamolden schrieb:
There is a well known C++ implementation of cow-fork on Windows, which
I have slightly modified and ported to C. But as the new WDK (Windows
driver kit) headers are full of syntax errors, the compiler choke on
it. :( I am seriously considering re-implementing the whole cow fork
in pure Python using ctypes.
Can you provide a C implementation that compiles under VS 2008? Python
2.6 and 3.0 are using my new VS 2008 build system and we have dropped
support for 9x, ME and NT4. If you can provide us with an implementation
we *might* consider using it.

Christian

Jun 27 '08 #4
In article <877a5774-d3cc-49d3-bb64-5cab8505a419
@m3g2000hsc.googlegroups.com>, st**********@yahoo.no says...
I don't see pyprocessing as a drop-in replacement for the threading
module. Multi-threading and multi-processing code tend to be
different, unless something like mutable objects in shared memory is
used as well (cf. Python Shared Objects). If this limitation can
educate Python programmers to use queues instead of locks and mutable
objects, even multi-threaded Python programs may actually benefit.
Some API differences between threading and multiprocessing do not
matter. Programmers should not consider processes as a drop-in
replacement for threads.
This is probably not very central to the main intention of your post,
but I see a terminology problem coming up here. It is possible for
python objects to share a reference to some other object. This has
nothing to do with threads or processes, although it can be used as a
*mechanism* for threads and processes to share data. Another mechanism
would be some copying and synchronization scheme, which is what posh
seems to do. Or maybe not, I haven't used posh yet, I just read some
docs (and I already hate the "if process.fork():" idiom, what are they
trying to do, reintroduce c-style assignment and swiching?).

By the way I haven't done much thread and process programming, but the
things I *have* done often combine threads and processes, like starting
a console oriented program in a background process, redirecting the IO
and communicate with it using an event loop in a thread. I gets more
complicated when a gui thread is also involved, for example when
retrofitting a gui interface to an existing terminal based chess or go
playing program.

P.
Jun 27 '08 #5
On Jun 5, 11:02 am, pataphor <patap...@gmail.comwrote:
This is probably not very central to the main intention of your post,
but I see a terminology problem coming up here. It is possible for
python objects to share a reference to some other object. This has
nothing to do with threads or processes, although it can be used as a
*mechanism* for threads and processes to share data.
It is complicated in the case of processes, because the object must be
kept in shared memory. The complicating factor is that the base
address of the memory mapping, which is not guaranteed to be the same
in the virtual address space of different processes.

Jun 27 '08 #6
sturlamolden wrote:
On Jun 5, 11:02 am, pataphor <patap...@gmail.comwrote:
>This is probably not very central to the main intention of your post,
but I see a terminology problem coming up here. It is possible for
python objects to share a reference to some other object. This has
nothing to do with threads or processes, although it can be used as a
*mechanism* for threads and processes to share data.

It is complicated in the case of processes, because the object must be
kept in shared memory. The complicating factor is that the base
address of the memory mapping, which is not guaranteed to be the same
in the virtual address space of different processes.
Introducing shared memory in Python would be a terrible idea,
for many reasons, including the need for interprocess garbage
collection and locking. Don't go there. Use message passing instead.

John Nagle
Jun 27 '08 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
by: Petri Savolainen | last post by:
I was trying to roll my own python code importer, but in the end, it seems that no matter what you try, it is always necessary to supply a REAL file object for python lower-level import machinery...
1
by: Peter Åstrand | last post by:
There's a new PEP available: PEP 324: popen5 - New POSIX process module A copy is included below. Comments are appreciated. ---- PEP: 324 Title: popen5 - New POSIX process module
18
by: Steven Bethard | last post by:
In the "empty classes as c structs?" thread, we've been talking in some detail about my proposed "generic objects" PEP. Based on a number of suggestions, I'm thinking more and more that instead of...
2
by: Kenneth McDonald | last post by:
I'd like to propose a new PEP , for a standard library module that deals with files and file paths in an object oriented manner. I believe this module should be included as part of the standard...
5
by: pigmartian | last post by:
I recently learned (from I response on this newsgroup to an earlier query) of the processing module for working with subprocesses in a similar manner to threading. For what I needed to do, it...
1
by: redbaron | last post by:
I stuck in new multiprocessing module (ex. processing). I dont' understand why queue.get_nowait() never returns data, but always raises Empty, even if it is guaranteed that queue is not empty. ...
9
by: YouCanCallMeAl | last post by:
It seems that the multiprocessing module in 2.6 is broken for *BSD; I've seen issue 3770 regarding this. I'm curious if there are more details on this issue since the posts in 3770 were a bit...
1
by: davy zhang | last post by:
I mean every process attach like thread in wingide like thread or tasklet in wingide :) maybe I asked toooo much:D
1
by: Jeffrey Barish | last post by:
skip@pobox.com wrote: So I thought at first, but then I saw this statement in the documentation: It is possible to run a manager server on one machine and have clients use it from other...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: marcoviolo | last post by:
Dear all, I would like to implement on my worksheet an vlookup dynamic , that consider a change of pivot excel via win32com, from an external excel (without open it) and save the new file into a...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.