473,703 Members | 2,564 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

fiber(cooperati ve multi-threading)


Hi all.

I found cooperative multi-threading(only one thread runs at once,
explicit thread switching) is useful for writing some simulators.
With it, I'm able to be free from annoying mutual exclusion, and make
results deterministic.

For this purpose, and inspired by Ruby(1.9) fiber, I wrote my own
version of fiber in Python.

It just works, but using native Python threads for non-preemptive
threading is not cost-effective. Python has generator instead but it
seemed to be very restricted for general scripting. I wish I could
write nested (generator) functions easily at least.

Is there any plan of implementing real (lightweight) fiber in Python?

------------------------------------------------------------
import threading

class Fiber(threading .Thread):
def __init__(self):
threading.Threa d.__init__(self )

self.semaphore_ running = threading.Semap hore(0)
self.semaphore_ finish = None
self.val = None

self.setDaemon( True)
self.start()
self.start = self.start_fibe r

def start_fiber(sel f):
self.semaphore_ finish = threading.Semap hore(0)
self.semaphore_ running.release ()
self.semaphore_ finish.acquire( )

def run(self): # override
self.semaphore_ running.acquire ()
self.main()
if self.semaphore_ finish is not None:
self.semaphore_ finish.release( )

def switchto(self, fiber, val=None):
fiber.val = val
fiber.semaphore _running.releas e()
self.semaphore_ running.acquire ()
return self.val

def main(self): # should be overridden
pass

class F1(Fiber):
def main(self):
print "f1 start"
self.switchto(f 2)
print "f1 foo"
v = self.switchto(f 2)
print "f1 v=%s world" % v
self.switchto(f 2, "OK")
print "f1 end"

class F2(Fiber):
def main(self):
print "f2 start"
self.switchto(f 1)
print "f2 bar"
result = self.switchto(f 1, "Hello, ")
print "f2 result=%s" % result
print "f2 end"
self.switchto(f 1)

f1 = F1()
f2 = F2()

print "start"
f1.start()
print "end"

-- kayama
Dec 22 '07 #1
14 3449
On Dec 22, 12:10*pm, Akihiro KAYAMA <kay...@st.rim. or.jpwrote:
Hi all.

I found cooperative multi-threading(only one thread runs at once,
explicit thread switching) is useful for writing some simulators.
With it, I'm able to be free from annoying mutual exclusion, and make
results deterministic.

For this purpose, and inspired by Ruby(1.9) fiber, I wrote my own
version of fiber in Python.

It just works, but using native Python threads for non-preemptive
threading is not cost-effective. Python has generator instead but it
seemed to be very restricted for general scripting. I wish I could
write nested (generator) functions easily at least.

Is there any plan of implementing real (lightweight) fiber in Python?

------------------------------------------------------------
import threading

class Fiber(threading .Thread):
* * def __init__(self):
* * * * threading.Threa d.__init__(self )

* * * * self.semaphore_ running = threading.Semap hore(0)
* * * * self.semaphore_ finish = None
* * * * self.val = None

* * * * self.setDaemon( True)
* * * * self.start()
* * * * self.start = self.start_fibe r

* * def start_fiber(sel f):
* * * * self.semaphore_ finish = threading.Semap hore(0)
* * * * self.semaphore_ running.release ()
* * * * self.semaphore_ finish.acquire( )

* * def run(self): * * * * * * *# override
* * * * self.semaphore_ running.acquire ()
* * * * self.main()
* * * * if self.semaphore_ finish is not None:
* * * * * * self.semaphore_ finish.release( )

* * def switchto(self, fiber, val=None):
* * * * fiber.val = val
* * * * fiber.semaphore _running.releas e()
* * * * self.semaphore_ running.acquire ()
* * * * return self.val

* * def main(self): * * * * * * # should be overridden
* * * * pass

class F1(Fiber):
* * def main(self):
* * * * print "f1 start"
* * * * self.switchto(f 2)
* * * * print "f1 foo"
* * * * v = self.switchto(f 2)
* * * * print "f1 v=%s world" % v
* * * * self.switchto(f 2, "OK")
* * * * print "f1 end"

class F2(Fiber):
* * def main(self):
* * * * print "f2 start"
* * * * self.switchto(f 1)
* * * * print "f2 bar"
* * * * result = self.switchto(f 1, "Hello, ")
* * * * print "f2 result=%s" % result
* * * * print "f2 end"
* * * * self.switchto(f 1)

f1 = F1()
f2 = F2()

print "start"
f1.start()
print "end"

-- kayama
I am not really familiar with ruby but these fibers seem to be some
sort of coroutines. Since python 2.5, generators can be sent values,
this can be used to implement what you want. I have had a got at it
for fun and this is what I came up with: note that there is no need
for a specific class, or for threads. I have tested this only on your
example, but it gives the same result as your code :)

-------------------------------------

# A 'fiber' is a generator function that yields tuples (fiber, value)
# or (fiber,). The 'run' function starts the the fiber and carries on
# with the fiber it yields. If the fiber also yields a value, then
# this value is sent to the 'next' fiber.

def run(fiber):
it, val = fiber(), []
fibers = { fiber: it }
try:
while True:
n = it.send(*val) if val else it.next()
co, val = n[0], n[1:]
it = fibers.get(co)
if it is None:
fibers[co] = it = co()
except StopIteration:
return val[0] if val else None

# Now the two 'fibers'

def f1():
print "f1 start"
yield f2,
print "f1 foo"
v = yield f2,
print "f1 v=%s world" % v
yield f2, "OK"
print "f1 end"

def f2():
print "f2 start"
yield f1,
print "f2 bar"
result = yield f1, "Hello, "
print "f2 result=%s" % result
print "f2 end"
yield f1,

print "start"
run(f1)
print "end"

--------------------------------------

HTH

--
Arnaud

Dec 22 '07 #2
Arnaud Delobelle <ar*****@google mail.comwrote:
I am not really familiar with ruby but these fibers seem to be some
sort of coroutines. Since python 2.5, generators can be sent values,
this can be used to implement what you want. I have had a got at it
for fun and this is what I came up with: note that there is no need
for a specific class, or for threads. I have tested this only on your
example, but it gives the same result as your code :)
I think it was Microsoft who invented the term 'fiber' as a synonym for
coroutine. See http://msdn2.microsoft.com/en-us/library/ms682661.aspx

Unfortunately generators only save a single level of stack-frame, so they
are not really a replacement for fibers/coroutines. The OP should perhaps
look at Stackless Python or Greenlets. See
http://codespeak.net/py/dist/greenlet.html

Dec 22 '07 #3
On Dec 22, 2:37*pm, Duncan Booth <duncan.bo...@i nvalid.invalidw rote:
Arnaud Delobelle <arno...@google mail.comwrote:
I am not really familiar with ruby but these fibers seem to be some
sort of coroutines. *Since python 2.5, generators can be sent values,
this can be used to implement what you want. *I have had a got at it
for fun and this is what I came up with: note that there is no need
for a specific class, or for threads. *I have tested this only on your
example, but it gives the same result as your code :)

I think it was Microsoft who invented the term 'fiber' as a synonym for
coroutine. Seehttp://msdn2.microsoft .com/en-us/library/ms682661.aspx
OK
Unfortunately generators only save a single level of stack-frame, so they
are not really a replacement for fibers/coroutines. The OP should perhaps
look at Stackless Python or Greenlets. Seehttp://codespeak.net/py/dist/greenlet.html
Still what I am proposing offers a level of cooperative multi-
threading which may be enough for the OP. In fact it is not clear to
me at the moment what can be done (sensibly :) with the OP's Fiber
class that cannot be achieved with the run() function I suggested.
This makes me think that there is something important that I am
failing to take into consideration; I would be grateful if it was
pointed out to me.

--
Arnaud

Dec 22 '07 #4

Thanks for your replies.

In article <37************ *************** *******@i72g200 0hsd.googlegrou ps.com>,
Arnaud Delobelle <ar*****@google mail.comwrites:

arnodeldef f1():
arnodel print "f1 start"
arnodel yield f2,
arnodel print "f1 foo"
arnodel v = yield f2,
arnodel print "f1 v=%s world" % v
arnodel yield f2, "OK"
arnodel print "f1 end"
arnodel>
arnodeldef f2():
arnodel print "f2 start"
arnodel yield f1,
arnodel print "f2 bar"
arnodel result = yield f1, "Hello, "
arnodel print "f2 result=%s" % result
arnodel print "f2 end"
arnodel yield f1,

This is the most simple example. In real programming, things are more
complicate so I will want to refactor it like below:

def foo(fiber, s, arg=None)
print s
return yield fiber, arg

def f1():
foo(f2, "start") # XXX returns generator object
v = foo(f2, "foo")
foo(f2, "v=%s world" % v, "OK")

But current Python generator specification requires me:

def f1():
for x in foo(f2, "foo"): yield x
for x in foo(f2, "foo"): yield x
# XXX v = ... (I don't know how to do this)
for x in foo(f2, "v=%s world" % v, "OK"): yield x

I think it is not straitforward. Single level function which generator
impose is impractical for real use.

In article <Xn************ *************@1 27.0.0.1>,
Duncan Booth <du**********@i nvalid.invalidw rites:

duncan.boothUnf ortunately generators only save a single level of stack-frame, so they
duncan.boothare not really a replacement for fibers/coroutines. The OP should perhaps
duncan.boothloo k at Stackless Python or Greenlets. See
duncan.boothhttp://codespeak.net/py/dist/greenlet.html

I am happy if I could use convenient coroutine features via standard
or simple extension library. py.magic.greenl et may be what I'm
looking for, but I wonder why this is named "magic" :-)

-- kayama
Dec 23 '07 #5
Akihiro KAYAMA <ka****@st.rim. or.jpwrote:
Thanks for your replies.

But current Python generator specification requires me:

def f1():
for x in foo(f2, "foo"): yield x
for x in foo(f2, "foo"): yield x
# XXX v = ... (I don't know how to do this)
for x in foo(f2, "v=%s world" % v, "OK"): yield x

I think it is not straitforward. Single level function which generator
impose is impractical for real use.
Not just impractical, there are plenty of situations where you simply
cannot do that at all.

Here's a greenlet example (from
http://socal-piggies.org/presentatio.../2005_07_21/):

from py.magic import greenlet
import xml.parsers.exp at

def send(arg):
greenlet.getcur rent().parent.s witch(arg)

def start_element(n ame, attrs):
send(('START', name, attrs))
def end_element(nam e):
send(('END', name))
def char_data(data) :
data = data.strip()
if data: send(('DATA', data))

def greenparse(xmld ata):
p = xml.parsers.exp at.ParserCreate ()
p.StartElementH andler = start_element
p.EndElementHan dler = end_element
p.CharacterData Handler = char_data
p.Parse(xmldata , 1)

def iterxml(xmldata ):
g = greenlet(greenp arse)
data = g.switch(xmldat a)
while data is not None:
yield data
data = g.switch()

if __name__ == "__main__":
for data in iterxml("somexm ldata"):
# do something with data
The greenlet here calls expat, but the 'yield' happens inside an expat
callback. To get this to work you would need to rewrite expat to use its
callback as a generator. Expat is coded in C, so you also need to find
some way to get it to save the state of the C functions when it has
yielded.

I think this example should answer Arnaud's question ("In fact it is not
clear to me at the moment what can be done (sensibly :) with the OP's
Fiber class that cannot be achieved with the run() function I
suggested.") Arnaud's run function cannot be used by anything which needs
to save C stack state.
I am happy if I could use convenient coroutine features via standard
or simple extension library. py.magic.greenl et may be what I'm
looking for, but I wonder why this is named "magic" :-)
I think it is called "magic" because it does something which is at first
glance impossible: saving the C stack state as well as the Python stack.
Possibly it does something underhand to achieve an impressive effect.
I haven't looked at the implementation to see how it does it: creating
threads and then scheduling them cooperatively would be one way but I
expect it just switches the C stack around between different areas of
memory.

A long time ago I ported the BCPL coroutine library and there is one
function in the middle (the one which does the stack switch) that
definitely felt like magic.

One interesting question raised by all this is whether those of us using
Windows can usefully call Microsoft's Fiber functions through ctypes and
get the benefit of coroutines without having to load any additional C
extensions at all.
Dec 23 '07 #6
Duncan Booth wrote:
Unfortunately generators only save a single level of stack-frame, so they
are not really a replacement for fibers/coroutines. The OP should perhaps
look at Stackless Python or Greenlets. See
On the surface of things, the single level aspect *LOOKS* like a problem,
but in fact is actually really useful. The reason is because it encourages
a generator to be relatively simple and focused encouraging reuse.

cf the components listed here:
* http://kamaelia.sourceforge.net/Components

Also, because they're simple, you can link every component in that list
either directly with all the others or via a trivial filtering component.

Also the fact that you have to build your own scheduler means you can do the
equivalent of saying to the scheduler "Don't run me next, run this next".
This gives you pretty much the full power of co-routines but allows for
very heavy reusability and independence of testing.

An example application that makes moderate use of this "bounce back"
coroutines is a greylisting server you can find here:
* http://kamaelia.sourceforge.net/KamaeliaGrey

I'd (almost) argue that the single level aspect of generators is perhaps
their best feature.
Michael.
--
http://yeoldeclue.com/blog
http://kamaelia.sourceforge.net/Developers/
Dec 23 '07 #7
Hi,

It just works, but using native Python threads for non-preemptive
threading is not cost-effective. Python has generator instead but it
seemed to be very restricted for general scripting. I wish I could
write nested (generator) functions easily at least.

Is there any plan of implementing real (lightweight) fiber in Python?
Please take a look at Kamaelia. Generators work extremely well, and are
extremely useful to work with:

* http://kamaelia.sourceforge.net/Home

(I need to update/improve the website...)

If you *must* have nested generators (it's rarely useful) you can do that,
but just need to have your "thing running the generators" understand
a "don't run me next, run this next" message. Our message is called
WaitComplete, and you can see it used here (in a non-trivial useful
system):

https://kamaelia.svn.sourceforge.net...greylisting.py

(I could point at trivial examples, but prefer something real)

The shortest overall description is here:
* http://kamaelia.sourceforge.net/t/TN...ToKamaelia.pdf

Cookbook examples:
* http://kamaelia.sourceforge.net/Cookbook

List of dozens of reusable (all with each other) components:
* http://kamaelia.sourceforge.net/Components

How to build your own (non-optimised) core:
* http://kamaelia.sourceforge.net/MiniAxon/

Ruby (mini) version: (for comparison :-)
https://kamaelia.svn.sourceforge.net...by/miniaxon.rb

Experimental process based (as well as thread & generator based) version:

http://yeoldeclue.com/cgi-bin/blog/b...eid=1196129474

(Going to move to process based components for static segmentation of an
application across process boundaries to give explicit multicore support.
May move to automatic distribution at some point, but static is an easy win
as the above example shows :-)

It's not as lightweight as stackless python's microthreads, but it's pretty
close. The scaling style is pretty similar as well - cf:

http://www.rhonabwy.com/wp/2007/11/1...ing-in-python/
I also blogged about that here:
http://yeoldeclue.com/cgi-bin/blog/b...eid=1195688924

(Someone curious about comparing Kamaelia to stackless. Interesting to see
the same scaling curve, but unsurprisingly stackless wins :-) Kamaelia
however works with standard python from version 2.2.late onwards :-) )

Interestingly recently discovered that Kamaelia's been slowly reinventing a
system the UK's MOD have been using for around 30 years to make concurrent
systems easier to build.... (mascot) :-)

Merry Christmas :-)
Michael
--
http://yeoldeclue.com/blog
http://kamaelia.sourceforge.net/Developers/

Dec 23 '07 #8
Michael Sparks <ms@cerenity.or gwrote:
Duncan Booth wrote:
>Unfortunatel y generators only save a single level of stack-frame, so
they are not really a replacement for fibers/coroutines. The OP
should perhaps look at Stackless Python or Greenlets. See

On the surface of things, the single level aspect *LOOKS* like a
problem, but in fact is actually really useful. The reason is because
it encourages a generator to be relatively simple and focused
encouraging reuse.
Ah, perhaps Python should similarly limit function call nesting to one
level so as to keep things simple and encourage reuse.
Dec 23 '07 #9
Is there any plan of implementing real (lightweight) fiber in Python?

I have no such plan, and I don't know of anybody else's plan, either.

Regards,
Martin
Dec 24 '07 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
5078
by: searcher1234 | last post by:
Hi, I just start learning network programming. How can I write a web server with "C" so that it can connect to web browser and read the request, accept the request, open requested html page, send it back to the web browser for user viewing. One more is how can i allow multi-users send request to the web server?
51
4050
by: thesushant | last post by:
hi, can anyone of C masters siting out there suggest any 1 example of re-entrnt function just to show what is the significance of that property and how we can exploit it .... sushant
6
1841
by: Kevin | last post by:
Hi All, Please could someone explain to me what an iterator is and how it may be used in a real world situation. I am new to C#, and I can't find a decent,plain english explanation for iterators. TIA Kevin
3
1036
by: Ioannis Vranos | last post by:
I am finishing learning the fundamentals of Managed Extensions, however my current book has not anything about multithreading. I am thinking to read a chapter about that of another VC++ 2003 oriented book. However I would like to know, will multithreading in VC++ 2005 with the new C++/CLI stuff, be significantly different than what is in 2003 version? If yes, I'll wait for the next release to read about the subject, if not
4
35957
Niheel
by: Niheel | last post by:
http://bytes.com/images/howtos/technologies_cut_cost_slow_economy.jpgThere's no escaping the slowing economy. Layoffs are being announced and companies are in cost cutting mode. Whether you are a tech at a big company or a small one man operation it's time to cut the fat before you get cut out of work. Unnecessary expenses have to go and inefficient ways of working need to be analyzed and improved. If done right, the use of some...
0
8756
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9251
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9117
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
9014
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
1
6585
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
4429
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
3122
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2446
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2063
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.