Opposite of yield?

[achrist]

The yield statement looks to be a big step toward some kind of
lightweight concurrency-oriented programming in python. Is there
any similarly nice way to do the opposite of yield, just sit around
(perhaps in the middle of a loop) and wait until some other routine
(unknown to the waiting module) hurls a value in?

If this is not implemented now, is there any chance that it's on the
list of things to come?
Al

[Paul Rubin]

ac*****@easystreet.com writes:

The yield statement looks to be a big step toward some kind of
lightweight concurrency-oriented programming in python. Is there
any similarly nice way to do the opposite of yield, just sit around
(perhaps in the middle of a loop) and wait until some other routine
(unknown to the waiting module) hurls a value in?

If this is not implemented now, is there any chance that it's on the
list of things to come?

Not much chance. Even Stackless can't do that any more, if I
understand correctly.

[Erik Max Francis]

ac*****@easystreet.com wrote:

The yield statement looks to be a big step toward some kind of
lightweight concurrency-oriented programming in python. Is there
any similarly nice way to do the opposite of yield, just sit around
(perhaps in the middle of a loop) and wait until some other routine
(unknown to the waiting module) hurls a value in?

Yep:

generator.next()

--
Erik Max Francis && ma*@alcyone.com && http://www.alcyone.com/max/
__ San Jose, CA, USA && 37 20 N 121 53 W && &tSftDotIotE
/ \ Dead men have no victory.
\__/ Euripides

[Bob Gailer]

At 01:40 PM 9/10/2003, ac*****@easystreet.com wrote:

The yield statement looks to be a big step toward some kind of
lightweight concurrency-oriented programming in python. Is there
any similarly nice way to do the opposite of yield, just sit around
(perhaps in the middle of a loop) and wait until some other routine
(unknown to the waiting module) hurls a value in?

Sounds like a task for a thread that sleeps a while then checks for the data.

Also reminiscent (a long time ago) someone proposed a COME FROM statement
for FORTRAN to be the inverse of GO TO.

Perhaps a new Python statement "suck"?

Bob Gailer
bg*****@alum.rpi.edu
303 442 2625
---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.506 / Virus Database: 303 - Release Date: 8/1/2003

[Dave Benjamin]

"Bob Gailer" <bg*****@alum.rpi.edu> wrote in message
news:ma*********************************@python.or g...

At 01:40 PM 9/10/2003, ac*****@easystreet.com wrote:

The yield statement looks to be a big step toward some kind of
lightweight concurrency-oriented programming in python. Is there
any similarly nice way to do the opposite of yield, just sit around
(perhaps in the middle of a loop) and wait until some other routine
(unknown to the waiting module) hurls a value in?

...

Perhaps a new Python statement "suck"?

Oh come on, now, just because Java does it...

[Peter Hansen]

ac*****@easystreet.com wrote:

The yield statement looks to be a big step toward some kind of
lightweight concurrency-oriented programming in python. Is there
any similarly nice way to do the opposite of yield, just sit around
(perhaps in the middle of a loop) and wait until some other routine
(unknown to the waiting module) hurls a value in?

If this is not implemented now, is there any chance that it's on the
list of things to come?

Queue.Queue.get() ...

-Peter

[Christos TZOTZIOY Georgiou]

On Wed, 10 Sep 2003 12:40:37 -0700, rumours say that
ac*****@easystreet.com might have written:

The yield statement looks to be a big step toward some kind of
lightweight concurrency-oriented programming in python. Is there
any similarly nice way to do the opposite of yield, just sit around
(perhaps in the middle of a loop) and wait until some other routine
(unknown to the waiting module) hurls a value in?
Almost. Use a Queue.Queue() and threads. That is, some other thread(s)
does/do .put() and you wait with a .get().
Without threads, I don't believe there is such a mechanism, not even in
Stackless, as Paul said.
If this is not implemented now, is there any chance that it's on the
list of things to come?

I believe such a chance would be a candidate for dietary products
advertising (that is, slim :).
--
TZOTZIOY, I speak England very best,
Microsoft Security Alert: the Matrix began as open source.

[Christos TZOTZIOY Georgiou]

On Wed, 10 Sep 2003 20:19:35 GMT, rumours say that "Dave Benjamin"
<da**@3dex.com> might have written:

Perhaps a new Python statement "suck"?

Oh come on, now, just because Java does it...

QOTW +1 :)
--
TZOTZIOY, I speak England very best,
Microsoft Security Alert: the Matrix began as open source.

[achrist]

Peter Hansen wrote:

Queue.Queue.get() ...

That looks good. But that needs a thread to block, right?
A generator manages to run without being a thread (at least
overtly). If we have the suck() statement, we also need
something that's the opposite of a generator (a Sucker?)
and something that's the opposite of an iterator (a Suckee?).
I'm starting to get an idea why this is completely not all
there.

The main question this raises is "How lightweight are threads?"
Can I program with dozens or hundreds of python threads in a
program (for example under Windows) and not notice that this is
an inefficient or inept coding style?
Al

[achrist]

Erik Max Francis wrote:

Yep:

generator.next()

This style supports only output-driven processes. The generator,
being on the input side, is always the server, and the output
side is always the client. It doesn't fit the situation where the
input is heterogeneous and input has to get spewed in several
directions, depending on its content.
Al

[Peter Hansen]

ac*****@easystreet.com wrote:

Peter Hansen wrote:

Queue.Queue.get() ...

That looks good. But that needs a thread to block, right?

Yep.
A generator manages to run without being a thread (at least
overtly). If we have the suck() statement, we also need
something that's the opposite of a generator (a Sucker?)
and something that's the opposite of an iterator (a Suckee?).
I'm starting to get an idea why this is completely not all
there.

The main question this raises is "How lightweight are threads?"
Can I program with dozens or hundreds of python threads in a
program (for example under Windows) and not notice that this is
an inefficient or inept coding style?

The real question might be why would you want to?

If you don't want a thread, but you want something which takes
input from elsewhere and does some processing, then returns
control to some other place until more data is available (which
is what one might assume if threads aren't good for you), there
is already a convenient solution: the subroutine. ;-)

Seriously, what's the requirement driving the need for this?
(I suspect Erik Max's answer is still what you need, and you
misunderstand the nature of what he was suggesting, but if you'll
explain the specific case you have in mind we'll know for sure.)

-Peter

[Chris Liechti]

ac*****@easystreet.com wrote in news:3F***************@easystreet.com:

Peter Hansen wrote:

Queue.Queue.get() ...

That looks good. But that needs a thread to block, right?

yes
A generator manages to run without being a thread (at least
overtly). If we have the suck() statement, we also need
something that's the opposite of a generator (a Sucker?)
and something that's the opposite of an iterator (a Suckee?).
I'm starting to get an idea why this is completely not all
there.
basicaly you poll every generator (or more precicly "iterable" as that is
what a generator returns), one after the other... enjoy the link below.
The main question this raises is "How lightweight are threads?"
Can I program with dozens or hundreds of python threads in a
program (for example under Windows) and not notice that this is
an inefficient or inept coding style?

this one is for you :-)
http://www-106.ibm.com/developerwork.../l-pythrd.html

you wont get happy with hundrets of native threads, but with the light ones
from above you can have thousands...

generaly you will find interesting stuff here (one line):
http://www-
106.ibm.com/developerworks/views/linux/articles.jsp?sort_order=desc&expand=
&sort_by=Date&show_abstract=true&view_by=Search&se arch_by=charming+python%3
A

chris
--
Chris <cl******@gmx.net>

[Bengt Richter]

On Wed, 10 Sep 2003 14:39:14 -0700, ac*****@easystreet.com wrote:

Erik Max Francis wrote:

Yep:

generator.next()

This style supports only output-driven processes. The generator,
being on the input side, is always the server, and the output
side is always the client. It doesn't fit the situation where the
input is heterogeneous and input has to get spewed in several
directions, depending on its content.

Could you make some fantasy-code to illustrate what you mean?

Regards,
Bengt Richter

[Stuart D. Gathman]

On Wed, 10 Sep 2003 17:39:14 -0400, achrist wrote:

Erik Max Francis wrote:

Yep:

generator.next()

This style supports only output-driven processes. The generator, being
on the input side, is always the server, and the output side is always
the client. It doesn't fit the situation where the input is
heterogeneous and input has to get spewed in several directions,
depending on its content.


It does if you add a dispatcher framework. There is a python project for
just such a framework, but I can't remember the name. The idea is dirt
simple, but the implementation gets knarly with exceptions and such.

Found it - Charming Python 
http://www-106.ibm.com/developerwork.../l-pythrd.html

[Erik Max Francis]

ac*****@easystreet.com wrote:

This style supports only output-driven processes. The generator,
being on the input side, is always the server, and the output
side is always the client. It doesn't fit the situation where the
input is heterogeneous and input has to get spewed in several
directions, depending on its content.

The distinction you're making is push vs. pull processing. To get what
you want, you should use some sort of asynchronous mechanism with
synchronization when data is transferred. Threading with the use of the
Queue module comes to mind immediately. Other forms of asynchronous
processing are certainly possible, as well.

That being said, calling the next method of a generator _is_ the obvious
inverse process of a generator yielding a value.

--
Erik Max Francis && ma*@alcyone.com && http://www.alcyone.com/max/
__ San Jose, CA, USA && 37 20 N 121 53 W && &tSftDotIotE
/ \ Forever we / Infinitely
\__/ Sandra St. Victor

[achrist]

Peter Hansen wrote:

The real question might be why would you want to?

If you don't want a thread, but you want something which takes
input from elsewhere and does some processing, then returns
control to some other place until more data is available (which
is what one might assume if threads aren't good for you), there
is already a convenient solution: the subroutine. ;-)

Seriously, what's the requirement driving the need for this?
(I suspect Erik Max's answer is still what you need, and you
misunderstand the nature of what he was suggesting, but if you'll
explain the specific case you have in mind we'll know for sure.)

No, I didn't say that I didn't want a thread. If there are no
serious drawbacks to relying heavily on threads, I'd be glad to
use them. This would be deluxe.

There really isn't any need for a generator, but it is a very
nice way to construct the input side of a program (the caller
doesn't have to worry about expicitly calling initialization and
finalization). Some kind of through-the-looking-glass version
of a generator would be a very nice way to construct the output
side.

I decided to test threads to see what happens:

-------------Warning, Code Follows---------------------

import Queue
import thread
import time

queues = [Queue.Queue()]
threads = []

appendLock = thread.allocate_lock()

def TransferData(queues, atPos):
while len(queues) < atPos + 1:
pass
inputQueue = queues[atPos]
outputQueue = Queue.Queue()
appendLock.acquire()
queues.append(outputQueue)
appendLock.release()
while 1:
data = inputQueue.get(True)
outputQueue.put(data, True)

startTime = time.time()
for i in range(1000):
print i,
threads.append(thread.start_new_thread(TransferDat a, (queues, i)))
queues[0].put(i, True)
appendLock.acquire()
print queues[-1].get(True)
appendLock.release()

print time.time() - startTime
--------------Warning Discussion Follows -------------------------

Running this on my Windows NT 200 NHz machine gave decent results.
With 1000 threads and 1000 Queues, memory used was 24MB. Python
takes almost 10 MB to start, so the overhead per thread + Queue
is only a little over 10 kb. Passing all the messages through
the chain took 2.5 minutes, so we are down around a quarter millisec
for each put or get.

That should be pretty good for many uses.
Al

[Jan Decaluwe]

ac*****@easystreet.com wrote:

Peter Hansen wrote:

Queue.Queue.get() ...

That looks good. But that needs a thread to block, right?
A generator manages to run without being a thread (at least
overtly). If we have the suck() statement, we also need
something that's the opposite of a generator (a Sucker?)
and something that's the opposite of an iterator (a Suckee?).
I'm starting to get an idea why this is completely not all
there.

The main question this raises is "How lightweight are threads?"
Can I program with dozens or hundreds of python threads in a
program (for example under Windows) and not notice that this is
an inefficient or inept coding style?

Al

You may want to check out how such things are done with my MyHDL
package. The target application is hardware design, but the idea
is more probably general.

http://jandecaluwe.com/Tools/MyHDL/m...model-obj.html

Jan

--
Jan Decaluwe - Resources bvba - http://jandecaluwe.com
Losbergenlaan 16, B-3010 Leuven, Belgium
Bored with EDA the way it is? Check this:
http://jandecaluwe.com/Tools/MyHDL/Overview.html

[Oren Tirosh]

On Wed, Sep 10, 2003 at 12:40:37PM -0700, ac*****@easystreet.com wrote:

The yield statement looks to be a big step toward some kind of
lightweight concurrency-oriented programming in python. Is there
any similarly nice way to do the opposite of yield, just sit around
(perhaps in the middle of a loop) and wait until some other routine
(unknown to the waiting module) hurls a value in?

Here is a way of converting a generator function into a consumer using
a single item list as a mailbox:

from __future__ import generators

def consumer():
print "This code runs before any data is fed to the consumer"
mailbox = []; yield mailbox
while mailbox:
print "Here's what I got:", mailbox.pop()
yield mailbox
print "This code runs after end of data is signalled"

# How to feed a consumer:

i = 0
for mailbox in consumer():
if i < 5:
mailbox.append(i) # feed next item
i += 1
Output:
This code runs before any data is fed to the consumer
Here's what I got: 0
Here's what I got: 1
Here's what I got: 2
Here's what I got: 3
Here's what I got: 4
This code runs after end of data is signalled
For a more comprehensive solution try Christian Tismer's Stackless Python.

Oren

[Greg Chapman]

On Wed, 10 Sep 2003 12:40:37 -0700, ac*****@easystreet.com wrote:

The yield statement looks to be a big step toward some kind of
lightweight concurrency-oriented programming in python. Is there
any similarly nice way to do the opposite of yield, just sit around
(perhaps in the middle of a loop) and wait until some other routine
(unknown to the waiting module) hurls a value in?

If this is not implemented now, is there any chance that it's on the
list of things to come?

You can use a generator as the body of a function. That is, the generator is
only ever accessed through the function, and the function and the generator have
some agreed on way to pass parameters. I.e.:

def getline(output):
'''getline accumulates text until it has a complete line (signaled
by either a newline (\\n) or a null (\\0)). It then passes the
line (without trailing \n) to output'''

genargs = [None]
def callgen(text):
genargs[0] = text
return gennext()
def gen():
line = ''
while True:
text, = genargs
if text:
lines = text.splitlines(True)
if line:
lines[0] = line+lines[0]
for line in lines[:-1]:
output(line[:-1])
line = lines[-1]
if line[-1] in '\n\0':
if line[-1] == '\n':
line = line[:-1]
output(line)
line = ''
yield None
gennext = gen().next
return callgen

def printer(text):
print text

lineprinter = getline(printer)

lineprinter('hello') #nothing output
lineprinter(' world\n') #prints 'hello world'
---
Greg Chapman

[Hung Jung Lu]

ac*****@easystreet.com wrote in message news:<3F***************@easystreet.com>...

The yield statement looks to be a big step toward some kind of
lightweight concurrency-oriented programming in python. Is there
any similarly nice way to do the opposite of yield, just sit around
(perhaps in the middle of a loop) and wait until some other routine
(unknown to the waiting module) hurls a value in?

This sounds like event-driven programming to me.

(1) Usage of threads in principle is optional, but at Python level
event-driven programming is probably always done in multi-threaded
environment.

(2) Also, at software level, interruptions are always emulated by
polling (i.e. event loop.) You will have a loop somewhere.

(3) I won't get into details of the event-handler implementation, as
it is well-known and you probably already know it or can find it
easily. But I'd like to mention a simple implementation using
exception handling. For instance:

while 1:
try:
poll_keyboard()
poll_mouse()
poll_microphone()
poll_temperature_sensor()
except IncomingSignal, e:
... handle event ...

You raise the IncomingSignal event inside the various polling
routines.

regards,

Hung Jung Lu

[Isaac To]

>>>>> "Hung" == Hung Jung Lu <hu********@yahoo.com> writes:

Hung> This sounds like event-driven programming to me.

Hung> (1) Usage of threads in principle is optional, but at Python level
Hung> event-driven programming is probably always done in multi-threaded
Hung> environment.

That does not stop somebody who wants something cheaper than threads when he
know that at most one place of his program needs being serviced at each
time.

Hung> (2) Also, at software level, interruptions are always emulated by
Hung> polling (i.e. event loop.) You will have a loop somewhere.

At "software level", most event loops are waited by calling the OS. The OS
can deal with the hardware interrupts caused by devices or timers, or the
signalling caused by other processes. In these ways the OS will not need to
do polling most of the time.

Hung> (3) I won't get into details of the event-handler implementation,
Hung> as it is well-known and you probably already know it or can find
Hung> it easily. But I'd like to mention a simple implementation using
Hung> exception handling. For instance:

Hung> while 1: try: poll_keyboard() poll_mouse() poll_microphone()
Hung> poll_temperature_sensor() except IncomingSignal, e: ... handle
Hung> event ...

Hung> You raise the IncomingSignal event inside the various polling
Hung> routines.

Never do that! You have a lot of ways to avoid holding up the CPU without
doing anything real. No event loop in real systems acts like that, except
for embedded systems that has exactly one program loaded into the memory at
the same time. Instead, use something like select.poll() to wait from more
than one source of input/output, or use a thread as suggested by the parent
thread.

Regards,
Isaac.

[Hung Jung Lu]

Isaac To <kk**@csis.hku.hk> wrote in message

At "software level", most event loops are waited by calling the OS. The OS
can deal with the hardware interrupts caused by devices or timers, or the
signalling caused by other processes. In these ways the OS will not need to
do polling most of the time.
OSes do polling most of the time: on their idle event loop. Yes,
"idle" is considered an event, too.

Also, events are not restricted to hardware inputs. Users can generate
their own events. Like in the original poster's example: once he
computes a result, he may want to generate an event, and let the
program handle the generated result through an event handler.
Never do that! You have a lot of ways to avoid holding up the CPU without
doing anything real. No event loop in real systems acts like that, except
for embedded systems that has exactly one program loaded into the memory at
the same time. Instead, use something like select.poll() to wait from more
than one source of input/output, or use a thread as suggested by the parent
thread.

Take it easy. :) It's just a sample code that shows some fun usage of
exception handling in Python. What the original poster asked was
"opposite of yield", and I only wanted to point out that what he
wanted was probably just "event-driven programming", which I was
surprised that no one has mentioned, at all.

My example is just a typical dirty trick of execution flow control in
Python via the exception handling. It's a well-known trick.

I did not mention the details of typical event-driven programming
implementation, because it's known by most programmers. No need for me
to show it here.

regards,

Hung Jung

[Jacek Generowicz]

Bob Gailer <bg*****@alum.rpi.edu> writes:

Also reminiscent (a long time ago) someone proposed a COME FROM
statement for FORTRAN to be the inverse of GO TO.

http://c2.com/cgi/wiki?ComeFrom