Lately I've been tinkering around with Erlang and have begun to sorely want
some of its features in Python, mostly the ease at which new processes can be
forked off for computation. To that end I've coded up a class I call,
boringly enough, Process. It takes a function, its args and keywords and runs
the function in another process using os.fork. Processes can be treated as
callables to retrieve the return value of the passed in function.
The code is pasted here: http://deadbeefbabe.org/paste/4972. A simple
exposition of Process is included at the end of the code in some __main__
magic. (Note: The code will not run on any system which lacks os.fork and
spawns 100 child processes while running.)
I'd very much appreciate people to look the code over and give me their
reactions to it, or suggests for improvement. As it stands, I see three major
defects with the current implementation:
1) Process hangs forever if its child errors out.
2) Process isn't portable because of its use of os.fork().
3) Process should use AF_UNIX when available instead of TCP.
---
blt 5 1321
On 1 Jun, 19:34, "Brian L. Troutwine" <goofyheadedp.. .@gmail.com>
wrote:
Lately I've been tinkering around with Erlang and have begun to sorely want
some of its features in Python, mostly the ease at which new processes can be
forked off for computation. To that end I've coded up a class I call,
boringly enough, Process. It takes a function, its args and keywords and runs
the function in another process using os.fork. Processes can be treated as
callables to retrieve the return value of the passed in function.
This sounds familiar... http://wiki.python.org/moin/ParallelProcessing
Do you have any opinions about those projects listed on the above page
that are similar to your own? My contribution (pprocess), along with
others (processing, pp...), can offer similar facilities, but the
styles of interfacing with spawned processes may be somewhat
different.
Paul http://wiki.python.org/moin/ParallelProcessing
Ah, I'd forgotten about that page of the wiki; I hadn't seen it for a few
months.
Do you have any opinions about those projects listed on the above page
that are similar to your own? My contribution (pprocess), along with
others (processing, pp...), can offer similar facilities, but the
styles of interfacing with spawned processes may be somewhat
different.
The interface was my most important design goal, in that I wanted it to be a
dead simple "Drop in some code and forget about it for a while. Retrieve the
results later as if you'd called the function yourself." sort of thing.
Secondly I wanted share nothing parallelism in order to avoid the nastier
bits of implementing concurrent code.
delegate doesn't fit the bill because it returns its results in a dictionary.
pp is rather more feature-full and rather less simple as a result. processing
is a clone of threading, more or less, and voids itself for not being simple
and having shared objects between processes (a nifty trick to be sure). POSH
uses a shared memory approach and hasn't been updated, it would seem, since
2003. pprocess isn't as simple as I wanted, though rather more simple than
all the others. remoteD uses shared memory.
I suppose, then, my opinion is that they're not brain-dead simple enough to
fulfill my desired style of process creation. (I'm smitten with Erlang.)
On Friday 01 June 2007 10:48:10 Paul Boddie wrote:
On 1 Jun, 19:34, "Brian L. Troutwine" <goofyheadedp.. .@gmail.com>
wrote:
Lately I've been tinkering around with Erlang and have begun to sorely
want some of its features in Python, mostly the ease at which new
processes can be forked off for computation. To that end I've coded up a
class I call, boringly enough, Process. It takes a function, its args and
keywords and runs the function in another process using os.fork.
Processes can be treated as callables to retrieve the return value of the
passed in function.
This sounds familiar...
http://wiki.python.org/moin/ParallelProcessing
Do you have any opinions about those projects listed on the above page
that are similar to your own? My contribution (pprocess), along with
others (processing, pp...), can offer similar facilities, but the
styles of interfacing with spawned processes may be somewhat
different.
Paul
On Jun 1, 2:20 pm, "Brian L. Troutwine" <goofyheadedp.. .@gmail.com>
wrote: http://wiki.python.org/moin/ParallelProcessing
Ah, I'd forgotten about that page of the wiki; I hadn't seen it for a few
months.
Do you have any opinions about those projects listed on the above page
that are similar to your own? My contribution (pprocess), along with
others (processing, pp...), can offer similar facilities, but the
styles of interfacing with spawned processes may be somewhat
different.
The interface was my most important design goal, in that I wanted it to be a
dead simple "Drop in some code and forget about it for a while. Retrieve the
results later as if you'd called the function yourself." sort of thing.
Secondly I wanted share nothing parallelism in order to avoid the nastier
bits of implementing concurrent code.
Funny, I've been working on a similar library these days with the same
primary goal, a minimal intuitive API. The closest to what I had in
mind that I found in the parallel processing wiki was the
Scientific.Dist ributedComputin g package (actually there are no
dependencies with the rest Scientific.* packages, the library
comprises two modules and one script all in all). It's worth checking
out and there's at least one idea I plan to copy (the "watchdog"
thread monitoring if a process has died), but overall I found the API
a bit clunky for my taste.
So I rolled yet another parallel processing package and I'm glad to
have a first working version ready at this point, exposing a single
lightweight API and three distinct platform-independent
implementations : one using multiple threads, one using multiple
processes in one or more hosts (through PYRO) and one singlethreaded
(for the sake of completeness, probably not very useful). I'll write
up some docs and I'll announce it, hopefully within the week.
George
Brian L. Troutwine <go************ *@gmail.comwrot e:
Lately I've been tinkering around with Erlang and have begun to sorely want
some of its features in Python, mostly the ease at which new processes can be
forked off for computation. To that end I've coded up a class I call,
boringly enough, Process. It takes a function, its args and keywords and runs
the function in another process using os.fork. Processes can be treated as
callables to retrieve the return value of the passed in function.
The code is pasted here: http://deadbeefbabe.org/paste/4972. A simple
exposition of Process is included at the end of the code in some __main__
magic. (Note: The code will not run on any system which lacks os.fork and
spawns 100 child processes while running.)
I'd very much appreciate people to look the code over and give me their
reactions to it, or suggests for improvement. As it stands, I see three major
defects with the current implementation:
1) Process hangs forever if its child errors out.
2) Process isn't portable because of its use of os.fork().
3) Process should use AF_UNIX when available instead of TCP.
You could use os.pipe() or socket.socketpa ir(). It would simplify the
code a lot too!
The biggest problem I see is that there is no non-blocking way of
seeing whether the Process() has finished or not, and no way to wait
on more than one Process() at once.
If there is an exception then you should return it to the parent (see
the subprocess module for an example).
--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Alex |
last post by:
Hi all,
We're looking at a vendor who uses the InterSystems Cache Database
Platform, but our IT department has zero experience with this system.
This software package will have a pivotal and mission critical roll in
our organization, so I'd like some comments on what others think of
this database platform.
Mainly I'm curious how easy/difficult it is to query a Cache Database,
and does it use standard SQL calls like Oracle and MS SQL? ...
|
by: Wendy Shuya |
last post by:
I am in the process of translating a large set of XML files. I've run
into some problems and thought someone may be able to assist.
Problem 1: Comments
-------------------
I need to match the comments (<!-- -->)in the original XML file, and
output them to the translated file. The code I have to do this:
<xsl:template match="comment()">
<xsl:comment><xsl:value-of select="."/></xsl:comment>
</xsl:template>
|
by: Brad Jones |
last post by:
<Previously posted in microsoft.public.dotnet.framework.windowsforms>
Hi all. Any suggestions here would be appreciated. Thanks for reading.
I'm primarly a C++ developer but I've been trying to complete development on
a C# application with the following basic requirements:
- The app runs in the background is initially hidden from the user,
including the Windows Taskbar.
- The app needs to receive a registered Windows message
- The app...
|
by: Jonnie |
last post by:
I am in the process of building an employee database for use by the
management team here. They current have Access 97 (I am working on
getting them to upgrade to at least 2000). For the most part I have it
under control. There are several points that have popped up as wanted
features and I need to verify somethings.
1. We now want to add the ability to add comments/notes with time and
date stamps to an employees record. These comments...
|
by: Philippe Bertrand |
last post by:
I tried following the pattern in MSDN and it just plain doesn't work.
I replaced the comments for a class with an include tag but did not change
any of the comments on the members of the class (without the include tag my
docs were building fine).
/// <include file='Command2.xml' path='MyDocs/MyMembers/*'/>
public class MyClass : ...
with Command2.xml looking like:
<MyDocs>
| |
by: Richard Buckle |
last post by:
Hi fellow Pythonistas,
I've been using Python in anger for some time, and I must say, as I
wrote in <http://macprang.sourceforge.net/>:
"It's refreshing beyond words to use a language that so freely combines
such a small, clean syntax with such a powerful synthesis of
procedural, object-oriented and functional techniques."
My (hardcore C++) workplace is now very much converted to Python for
|
by: tjb |
last post by:
I often see code like this:
/// <summary>
/// Removes a node.
/// </summary>
/// <param name="node">The node to remove.</param>
public void RemoveNode(Node node) {
<...>
}
|
by: Kay Schluehr |
last post by:
Every once in a while Erlang style message passing concurrency
is discussed for Python which does not only imply Stackless tasklets
but also some process isolation semantics that lets the runtime
easily distribute tasklets ( or logical 'processes' ) across physical
processes. Syntactically a tasklet might grow out of a generator by
reusing the yield keyword for sending messages:
yield_expr : 'yield' ( | testlist 'to' testlist)
...
|
by: process |
last post by:
In erlang you can cons like this: . i tried this in python and it
didnt raise an error but i dont know what the result do
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
| |
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |