473,236 Members | 1,477 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,236 software developers and data experts.

Erlang style processes for Python

Every once in a while Erlang style [1] message passing concurrency [2]
is discussed for Python which does not only imply Stackless tasklets
[3] but also some process isolation semantics that lets the runtime
easily distribute tasklets ( or logical 'processes' ) across physical
processes. Syntactically a tasklet might grow out of a generator by
reusing the yield keyword for sending messages:

yield_expr : 'yield' ([testlist] | testlist 'to' testlist)

where the second form is specific for tasklets ( one could also use a
new keyword like "emit" if this becomes confusing - the semantics is
quite different ) and the addition of a new keyword for assigning the
"mailbox" e.g:

required_stmt: 'required' ':' suite

So tasklets could be identified on a lexical level ( just like
generators today ) and compiled accordingly. I just wonder about
sharing semantics. Would copy-on-read / copy-on-write and new opcodes
be needed? What would happen when sharing isn't dropped at all but
when the runtime moves a tasklet around into another OS level thread /
process it will be pickled and just separated on need? I think it
would be cleaner to separate it completely but what are the costs?

What do you think?

[1] http://en.wikipedia.org/wiki/Erlang_...mming_language
[2] http://en.wikipedia.org/wiki/Actor_model
[3] http://www.stackless.com/

May 10 '07 #1
6 1807
On Wed, 09 May 2007 18:16:32 -0700, Kay Schluehr wrote:
Every once in a while Erlang style [1] message passing concurrency [2]
is discussed for Python which does not only imply Stackless tasklets [3]
but also some process isolation semantics that lets the runtime easily
distribute tasklets ( or logical 'processes' ) across physical
processes. Syntactically a tasklet might grow out of a generator by
reusing the yield keyword for sending messages:

yield_expr : 'yield' ([testlist] | testlist 'to' testlist)

where the second form is specific for tasklets ( one could also use a
new keyword like "emit" if this becomes confusing - the semantics is
quite different ) and the addition of a new keyword for assigning the
"mailbox" e.g:

required_stmt: 'required' ':' suite

So tasklets could be identified on a lexical level ( just like
generators today ) and compiled accordingly. I just wonder about sharing
semantics. Would copy-on-read / copy-on-write and new opcodes be needed?
What would happen when sharing isn't dropped at all but when the runtime
moves a tasklet around into another OS level thread / process it will be
pickled and just separated on need? I think it would be cleaner to
separate it completely but what are the costs?

What do you think?

[1] http://en.wikipedia.org/wiki/Erlang_...mming_language [2]
http://en.wikipedia.org/wiki/Actor_model [3] http://www.stackless.com/
Funny enough, I'm working on a project right now that is designed for
exactly that: PARLEY, http://osl.cs.uiuc.edu/parley . (An announcement
should show up in clp-announce as soon as the moderators release it). My
essential thesis is that syntactic sugar should not be necessary -- that a
nice library would be sufficient. I do admit that Erlang's pattern
matching would be nice, although you can get pretty far by using uniform
message formats that can easily be dispatched on -- the tuple
(tag, sender, args, kwargs)
in the case of PARLEY, which maps nicely to instance methods of a
dispatcher class.

The questions of sharing among multiple physical processes is interesting.
Implicit distribution of actors may not even be necessary if it is easy
enough for two hosts to coordinate with each other. In terms of the
general question of assigning actors to tasklets, threads, and processes,
there are added complications in terms of the physical limitations of
Python and Stackless Python:
- because of the GIL, actors in the same process do not gain the
advantag of true parallel computation
- all tasklet I/O has to be non-blocking
- tasklets are cooperative, while threads are preemptive
- communication across processes is slower, has to be serialized, etc.
- using both threads and tasklets in a single process is tricky

PARLEY currently only works within a single process, though one can choose
to use either tasklets or threads. My next goal is to figure out I/O, at
which point I get to tackle the fun question of distribution.

So far, I've not run into any cases where I've wanted to change the
interpreter, though I'd be interested in hearing ideas in this direction
(especially with PyPy as such a tantalizing platform!).

--
Jacob Lee <ar*****@freeshell.org>
May 10 '07 #2
On May 10, 8:31 am, Jacob Lee <artd...@freeshell.orgwrote:
Funny enough, I'm working on a project right now that is designed for
exactly that: PARLEY,http://osl.cs.uiuc.edu/parley. (An announcement
should show up in clp-announce as soon as the moderators release it). My
essential thesis is that syntactic sugar should not be necessary -- that a
nice library would be sufficient.
Synsugar is helpfull when you want to control compiler actions. Of
course you can do this also by means of __special__ attributes but I
guess this becomes clutter when you work with certain exposed sections
in the code.
I do admit that Erlang's pattern
matching would be nice, although you can get pretty far by using uniform
message formats that can easily be dispatched on -- the tuple
(tag, sender, args, kwargs)
in the case of PARLEY, which maps nicely to instance methods of a
dispatcher class.
Yes, I do think so too. It is more interesting to think about what
might be qualify as a message. Destructuring it is not hard in anyway
and I do also have a few concerns with naive pattern matching:

http://www.fiber-space.de/EasyExtend...itch-statement
The questions of sharing among multiple physical processes is interesting.
Implicit distribution of actors may not even be necessary if it is easy
enough for two hosts to coordinate with each other. In terms of the
general question of assigning actors to tasklets, threads, and processes,
there are added complications in terms of the physical limitations of
Python and Stackless Python:
- because of the GIL, actors in the same process do not gain the
advantag of true parallel computation
- all tasklet I/O has to be non-blocking
- tasklets are cooperative, while threads are preemptive
- communication across processes is slower, has to be serialized, etc.
- using both threads and tasklets in a single process is tricky
Actors don't need locking primitives since their data is locked by
virtue of the actors definition. That's also why I'm in favour for a
runtime / compiler based solution. Within the shiny world of actors
and actresses the GIL has no place. So a thread that runs actors only,
does not need to be blocked or block other threads - at least not for
data locking purposes. It is used much like an OS level process with
better sharing capabilities ( for mailbox addresses and messages ).
Those threads shall not take part of the access/release GIL game. They
might also not be triggered explicitely using the usual threading
API.
PARLEY currently only works within a single process, though one can choose
to use either tasklets or threads. My next goal is to figure out I/O, at
which point I get to tackle the fun question of distribution.

So far, I've not run into any cases where I've wanted to change the
interpreter, though I'd be interested in hearing ideas in this direction
(especially with PyPy as such a tantalizing platform!).
--
Jacob Lee <artd...@freeshell.org>
I guess you mean tantalizing in both of its meanings ;)

Good luck and inform us when you find interesting results.

Kay

May 10 '07 #3
jkn
Have you seen Candygram?

http://candygram.sourceforge.net/
jon N

May 10 '07 #4
On Thu, 10 May 2007 01:03:39 -0700, jkn wrote:
Have you seen Candygram?

http://candygram.sourceforge.net/
jon N
I did look at Candygram. I wasn't so keen on the method of dispatch (a
dictionary of lambdas that is passed to the receive function). It also
only works with threads and doesn't communicate across processes.

I definitely used Candygram as a reference point when determining what
features to hoist from Erlang.

--
Jacob Lee <ar*****@freeshell.org>
May 10 '07 #5
On Thu, 10 May 2007 00:19:11 -0700, Kay Schluehr wrote:
[snip]
>I do admit that Erlang's pattern
matching would be nice, although you can get pretty far by using uniform
message formats that can easily be dispatched on -- the tuple
(tag, sender, args, kwargs)
in the case of PARLEY, which maps nicely to instance methods of a
dispatcher class.

Yes, I do think so too. It is more interesting to think about what
might be qualify as a message. Destructuring it is not hard in anyway
and I do also have a few concerns with naive pattern matching:

http://www.fiber-space.de/EasyExtend...itch-statement
Interesting. Scala's pattern matching also looks nice. They have a
construct called a "case class" which is sort of like an algebraic data
type in that == compares the actual internal structure of the objects...
come to think of it, it reminds me of the proposal for named tuples that
floated around one of the python lists recently.
[snip]
Actors don't need locking primitives since their data is locked by
virtue of the actors definition. That's also why I'm in favour for a
runtime / compiler based solution. Within the shiny world of actors
and actresses the GIL has no place. So a thread that runs actors only,
does not need to be blocked or block other threads - at least not for
data locking purposes. It is used much like an OS level process with
better sharing capabilities ( for mailbox addresses and messages ).
Those threads shall not take part of the access/release GIL game. They
might also not be triggered explicitely using the usual threading
API.
There are also a lot of places where Python implicitly shares data, though.
Global variables are one -- if you disallow those, then each actor has
to have its own copy of all imported modules. I think the GC is also not
at all threadsafe. I'm not familiar enough with how the interpreter works
to judge whether disallowing shared memory would make any of the existing
obstacles to removing the GIL easier to deal with.

Certainly, if it's doable, it would be a big win to tackle these problems.
>PARLEY currently only works within a single process, though one can choose
to use either tasklets or threads. My next goal is to figure out I/O, at
which point I get to tackle the fun question of distribution.

So far, I've not run into any cases where I've wanted to change the
interpreter, though I'd be interested in hearing ideas in this direction
(especially with PyPy as such a tantalizing platform!).
--
Jacob Lee <artd...@freeshell.org>

I guess you mean tantalizing in both of its meanings ;)

Good luck and inform us when you find interesting results.

Kay
Thanks!

--
Jacob Lee <ar*****@freeshell.org>
May 10 '07 #6
Jacob Lee wrote:
Funny enough, I'm working on a project right now that is designed for
exactly that: PARLEY, http://osl.cs.uiuc.edu/parley .
Have you seen Kamaelia? Some people have noted that Kamaelia seems to have a
number of similarities to Erlang's model, which seems to come from a common
background knowledge. (Kamaelia's model is based on a blending of what I
know from a very basic recasting of CSP, Occam, unix pipelines and async
hardware verification).

Home:
http://kamaelia.sourceforge.net/Home

Intros:
http://kamaelia.sourceforge.net/Introduction
http://kamaelia.sourceforge.net/t/TN...t-Kamaelia.pdf
http://www.bbc.co.uk/rd/pubs/whp/whp113.shtml
* http://kamaelia.sourceforge.net/t/TN...ToKamaelia.pdf
http://kamaelia.sourceforge.net/Docs...isualisingAxon

The one *'d is perhaps the best at the moment.

Detail:
http://kamaelia.sourceforge.net/Cookbook
http://kamaelia.sourceforge.net/Components
Michael.

May 11 '07 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Guy | last post by:
Hi I've done some projects in python for my work which invole some test scripts etc. All quite basic stuff, I find python a good language to work in but have come up against a brick wall. I'm...
3
by: Sylwia | last post by:
Hi! How to make python programs show up by name is ps (e.g ps -e). By default if we have many different python processes running simultaneously, then we can only see python python python for...
6
by: Bob Swerdlow | last post by:
My application starts up a number of processes for various purposes using: self.popen = popen2.Popen3("/usr/local/bin/python -O "myscript.py") and then shuts them down when appropriate with...
1
by: Sori Schwimmer | last post by:
Hi, I am working on an application which involves interprocess communication. More to the point, processes should be able to notify other processes about certain situations, so the "notifyees"...
3
by: adsheehan | last post by:
I have been reading many of the posting on the GIL and impact on threading etc. I have found is confusing and would welcome some clarity on this. I understand that embedding the interpreter in a...
35
by: Carl J. Van Arsdall | last post by:
Alright, based a on discussion on this mailing list, I've started to wonder, why use threads vs processes. So, If I have a system that has a large area of shared memory, which would be better? ...
5
by: Brian L. Troutwine | last post by:
Lately I've been tinkering around with Erlang and have begun to sorely want some of its features in Python, mostly the ease at which new processes can be forked off for computation. To that end...
1
by: | last post by:
Hi! I'm running a Python program on M$ Windows 2000 as a test monitor. The program should close various processes, mostly "Application error"-windows, as they are created. This works fine until...
2
by: process | last post by:
In erlang you can cons like this: . i tried this in python and it didnt raise an error but i dont know what the result do
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.