Advanced concurrancy

[Peter Tillotson]

Hi,

I'm looking for an advanced concurrency module for python and don't seem
to be able to find anything suitable. Does anyone know where I might
find one? I know that there is CSP like functionality built into
Stackless but i'd like students to be able to use a standard python build.

I'm trying to develop distributed / Grid computing modules based on
python. The aim is to be able to use barriers for synchronisation and
channels for communication between processes running on a single box.
Then the jump to multiple processes on multiple boxes and eventually to
MPI implementations . Hopefully, each jump should not be that big a leap.

Of course it would be nice if there was a robust way of managing
concurrency in python aswell ;-)

p

[Stefan Rank]

on 04.08.2005 00:36 Michael Sparks said the following:

Peter Tillotson wrote:

I'm quite interested in the mini version and also using the modules as
mobile code rather than installing it formally.

I'll document it slightly better and post up on the website in the next 48
hours or so. In the meantime, the optimisation test which includes a
slightly modified mini-axon can be found here:

http://cvs.sourceforge.net/viewcvs.p...imisationTest/

Specifically the file "simplegame.py" .

Hi, sorry to intrude here.
I looked at Kamaelia (Axon actually) recently and I liked it,
what I did miss in the code (maybe I just did not find it) is a blocking
way for send-ing and recv-ing from in/outboxes that the scheduler is
aware of.
It seems as though all components basically have to do busy waiting now.

So do you plan on including a kind of scheduler-aware blocking
communication (like the `channels` of the `tasklets` in stackless)?

(as you have a postman already - not a bad idea i think if you compare
with multi-agent-systems theory (see jade) - it could be responsible for
alerting the scheduler of box.has_data)

Cheers,
Stefan

[Matt Hammond]

Hi Stefan,

It seems as though all components basically have to do busy waiting now.
You are right - components are, for the most part, busy-waiting. Which
is not a good thing!
So do you plan on including a kind of scheduler-aware blocking
communication (like the `channels` of the `tasklets` in stackless)?

(as you have a postman already - not a bad idea i think if you compare
with multi-agent-systems theory (see jade) - it could be responsible for
alerting the scheduler of box.has_data)

There is basic support for components to block, by calling the
self.pause() method. From the next yield, this blocks further execution
until
data arrives on any inbox.

The un-pausing action is actually performed by code within the
component
itself, though this is in effect, as you suggested it might be,
instigated by
the postman microprocess.

A simple example would be an improved 'consoleEcho' component:

class consoleEchoer(A xon.Component.c omponent):
....
def main(self):
while 1:
yield 1
while self.dataReady( "inbox"):
print self.recv("inbo x")
self.pause()

That said, if you look at the code, you'll see that probably the
majority of components do not make good use of this yet!

There are quite probably better mechanisms around - hence the
distinction, in the code, between microprocesses and components.
We'd be very interested if yourself or other people want to play with
alternative communication & blocking mechanisms.

I'd hope that even if the components and postboxes model doesn't
work out in the long run, the underlying generator based
microprocesses code could still be of some value to others!
regards
Matt

[Matt Hammond]

Ooops! In case you're wondering who I am, I'm working
on Kamaelia with Michael Sparks at the BBC.
Matt

[Stefan Rank]

on 04.08.2005 11:15 Matt Hammond said the following:

Hi Stefan,

It seems as though all components basically have to do busy waiting now.
You are right - components are, for the most part, busy-waiting. Which
is not a good thing!

So do you plan on including a kind of scheduler-aware blocking
communicati on (like the `channels` of the `tasklets` in stackless)?

[snip]
There is basic support for components to block, by calling the
self.pause() method. From the next yield, this blocks further execution
until
data arrives on any inbox.
[snip]

ah, i was searching for blocking on a specific inbox, but this is
actually a saner approach i think. why should you block on one inbox and
ignore data coming in on another...
There are quite probably better mechanisms around - hence the
distinction, in the code, between microprocesses and components.
We'd be very interested if yourself or other people want to play with
alternative communication & blocking mechanisms.
i think in-out channels is probably the best way for easy to manage
comunicating processes.
I'd hope that even if the components and postboxes model doesn't
work out in the long run, the underlying generator based
microprocesses code could still be of some value to others!

yes both levels will (it will take some time before i can use it
here...), i think of it as a more complete version of some of the
stackless features in standard python. (thanks again by the way)
the magic.greenlets have 'cooperative microprocesses' , but lack the more
pipe-like communication facilities and the independency of the
generators+sche duler based approach.
the coroutine support in future pythons (see the generator PEPs) will
probably make this easier... maybe::

next_inbox_to_r ead = yield blockon(inbox1, inbox2)

this communication infrastructure is definitely a Good Thing (TM), the
same holds for a standard environment or registry (wassitsname, your CAT
thing) as you need at least one well-known point of reference in a
'distributed' system.

<dreamingmode >
now if someone could only cut the gordian knot of the
GlobalInterpret erLock to make all this really concurrent... and then,
when she's at it, allow transparent distribution of components in a
network... ;-)
</dreamingmode>

[Matt Hammond]

Hi Stefan,

On Fri, 05 Aug 2005 07:20:36 +0100, Stefan Rank <st*********@of ai.at>
wrote:

So do you plan on including a kind of scheduler-aware blocking
communication (like the `channels` of the `tasklets` in stackless)?

Not at the moment. The kinds of applications we've mainly been looking at,
whilst developing, have been multimedia data-flow oriented, so blocking
communication would be used as a rate limiting mechanism. However, we've
found we can achieve that by putting explicit feedback links in.

I think we are also trying to stick with the asynchronous way, if
possible, because it may well make things simpler and more efficient when
it comes to distributing components across processes / networked
computers. Synchronous links will lead to handshaking latencies, more data
needing to flow etc. IMO that kind of facility could always be built on
top - for example - using a feedback inbox->outbox path to 'request' new
data ... if you really need it.

.... it also simplifies the scheduler. We've tried, so far, to keep the
underlying microprocess model decoupled from the components and
inbox/outbox layer. That said, that may have to change eventually!
this communication infrastructure is definitely a Good Thing (TM), the
same holds for a standard environment or registry (wassitsname, your CAT
thing) as you need at least one well-known point of reference in a
'distributed' system.
yep its a CAT (Coordinating Assistant Tracker). Nothing to do with the
project logo ... nooo! :-)
<dreamingmode >
now if someone could only cut the gordian knot of the
GlobalInterpret erLock to make all this really concurrent... and then,
when she's at it, allow transparent distribution of components in a
network... ;-)
</dreamingmode>

I'll second that!

regards
Matt
--

| Matt Hammond
| R&D Engineer, BBC Research and Development, Tadworth, Surrey, UK.

[travlr]

Hi, I've been on the research pursuit regarding these subjects and I
wonder why Erlang doesn't fit well into this category of concurrency.

Also, from what I have seen, there hasn't been a recent interest in an
updated interface. Maybe I'm mistaken and Erlang's features don't apply
well. Thanks for your comments.

Peter