467,223 Members | 1,377 Online
Bytes | Developer Community
Ask Question

Home New Posts Topics Members FAQ

Post your question to a community of 467,223 developers. It's quick & easy.

Twisted (or for loops ?) madness

Hi,
Probably not the best group to post my question but I'm sure there is
some people here that use Twisted.
First here is the beginning of my source code:

from twisted.internet import reactor, defer, threads
import time

class CompilerThread(object):
def __init__(self, task, delay):
self.task = task
self.delay = delay

def _processing(self, delay):
print 'Start :', self.task
# Simulate delayed result, to fire immediately use
self.d.callback(self.task)
time.sleep(delay)
return self.task

def compile(self):
print 'Compile :', self.task
print self
# Create Deferred in another thread and add callback
self.d = threads.deferToThread(self._processing,
self.delay).addCallback(self.print_result)
# Return the deferred, this way you could add callback later
return self.d

def print_result(self, result):
# Print result
print 'Compiler result :', result, self.task
# MUST return result otherwise next callback receive None
return result

# Create Compiler objects
ct1 = CompilerThread('*OBJECT 1*', 2)
ct2 = CompilerThread('*OBJECT 2*', 3)
ct3 = CompilerThread('*OBJECT 3*', 5)

# Use succeed to create a deferred already fired
d = defer.succeed(None)

Now my problem:
With this code everything work fine:

d.addCallback(lambda result: ct1.compile())
d.addCallback(lambda result: ct2.compile())
d.addCallback(lambda result: ct3.compile())

reactor.callLater(20, reactor.stop)
reactor.run()

Output:

Compile : *OBJECT 1*
<__main__.CompilerThread object at 0x00BAD070>
Start : *OBJECT 1*
Compiler result : *OBJECT 1* *OBJECT 1*
Compile : *OBJECT 2*
<__main__.CompilerThread object at 0x00BAD050>
Start : *OBJECT 2*
Compiler result : *OBJECT 2* *OBJECT 2*
Compile : *OBJECT 3*
<__main__.CompilerThread object at 0x00CDA4B0>
Start : *OBJECT 3*
Compiler result : *OBJECT 3* *OBJECT 3*
But when I try to replace this code with a for loops, something goes
wrong:

l = [ct1, ct2, ct3]
for c in l:
d.addCallback(lambda result: c.compile())

reactor.callLater(20, reactor.stop)
reactor.run()

Output:

Compile : *OBJECT 1*
<__main__.CompilerThread object at 0x00BAD030>
Start : *OBJECT 1*
Compiler result : *OBJECT 1* *OBJECT 1*
Compile : *OBJECT 3*
<__main__.CompilerThread object at 0x00CD9470>
Start : *OBJECT 3*
Compiler result : *OBJECT 3* *OBJECT 3*
Compile : *OBJECT 3*
<__main__.CompilerThread object at 0x00CD9470>
Start : *OBJECT 3*
Compiler result : *OBJECT 3* *OBJECT 3*

OBJECT 3 run 2 times and OBJECT 2 never ?!?

Any idea ? Maybe something related to Threads ?
Thanks for your help.

Oct 15 '07 #1
  • viewed: 1248
Share:
6 Replies
On Oct 15, 9:46 am, looping <kad...@gmail.comwrote:
l = [ct1, ct2, ct3]
for c in l:
d.addCallback(lambda result: c.compile())

reactor.callLater(20, reactor.stop)
reactor.run()

Output:

Compile : *OBJECT 1*
<__main__.CompilerThread object at 0x00BAD030>
Start : *OBJECT 1*
Compiler result : *OBJECT 1* *OBJECT 1*
Compile : *OBJECT 3*
<__main__.CompilerThread object at 0x00CD9470>
Start : *OBJECT 3*
Compiler result : *OBJECT 3* *OBJECT 3*
Compile : *OBJECT 3*
<__main__.CompilerThread object at 0x00CD9470>
Start : *OBJECT 3*
Compiler result : *OBJECT 3* *OBJECT 3*

OBJECT 3 run 2 times and OBJECT 2 never ?!?

Any idea ? Maybe something related to Threads ?
Thanks for your help.
After further tests, it look like it is the lambda that cause the
problem:
-Adding a parameter result to compile(self, result)
-Changing d.addCallback(lambda result: c.compile()) for
d.addCallback(c.compile)

And everything run fine.

Why lambda doesn't work ? (variable scope problem ?)

Oct 15 '07 #2
looping wrote:
On Oct 15, 9:46 am, looping <kad...@gmail.comwrote:
>l = [ct1, ct2, ct3]
for c in l:
d.addCallback(lambda result: c.compile())

reactor.callLater(20, reactor.stop)
reactor.run()

Output:

Compile : *OBJECT 1*
<__main__.CompilerThread object at 0x00BAD030>
Start : *OBJECT 1*
Compiler result : *OBJECT 1* *OBJECT 1*
Compile : *OBJECT 3*
<__main__.CompilerThread object at 0x00CD9470>
Start : *OBJECT 3*
Compiler result : *OBJECT 3* *OBJECT 3*
Compile : *OBJECT 3*
<__main__.CompilerThread object at 0x00CD9470>
Start : *OBJECT 3*
Compiler result : *OBJECT 3* *OBJECT 3*

OBJECT 3 run 2 times and OBJECT 2 never ?!?

Any idea ? Maybe something related to Threads ?
Thanks for your help.

After further tests, it look like it is the lambda that cause the
problem:
-Adding a parameter result to compile(self, result)
-Changing d.addCallback(lambda result: c.compile()) for
d.addCallback(c.compile)

And everything run fine.

Why lambda doesn't work ? (variable scope problem ?)
Yes. When defined, the lambda inherits the surrounding scope - which
contains the name c, that is bound to the last instance of the iteration.

What you need to do is to introduce a scoped variable inside the lambda at
creation time - e.g. like this:

for a in 1,2,3:
d.addCallback(lambda arg=a: work(a))

Diez
Oct 15 '07 #3
On Oct 15, 12:10 pm, looping <kad...@gmail.comwrote:
>
Why lambda doesn't work ? (variable scope problem ?)
This is a well known issue of for loops. Some believe it to be a bug
but Guido says it
is a design decision, in the sense that Python always do late binding.
If you
browse this list you will find many discussions. Basically if you do

funclist = []
for i in 1,2,3:
def f():
print i
funclist.append(f)

you will get funclist[0]() == funclist[1]() == funclist[2]() == 3 (you
get the latest
binding of "i"). As you see, it has nothing to do with lambdas.
Michele Simionato

Oct 15 '07 #4
On Oct 15, 12:33 pm, Michele Simionato <michele.simion...@gmail.com>
wrote:
is a design decision, in the sense that Python always do late binding.
If you
you will get funclist[0]() == funclist[1]() == funclist[2]() == 3 (you
get the latest
binding of "i"). As you see, it has nothing to do with lambdas.
Thanks Diez, replacing my addCallback with d.addCallback(lambda
result, comp=c: comp.compile()) do the trick.

So if I understand what Michele wrote (thanks too), when a function is
defined (with def), no scope is saved and every variable value not
passed in parameter is lost ? It means that variable value come from
the outer scope when the function is called ?
Oct 15 '07 #5
On Oct 15, 1:01 pm, looping <kad...@gmail.comwrote:
So if I understand what Michele wrote (thanks too), when a function is
defined (with def), no scope is saved and every variable value not
passed in parameter is lost ? It means that variable value come from
the outer scope when the function is called ?
Yes, in my example you get the value of "i" at the function *calling*
time,
not at the function definition time. If you want to store the value at
the
definition time, you must use the default argument trick:

funclist = []
for i in 1,2,3:
def f(i=i):
print i
funclist.append(f)

Michele Simionato

Oct 15 '07 #6
On Oct 15, 1:51 pm, Michele Simionato <michele.simion...@gmail.com>
wrote:
On Oct 15, 1:01 pm, looping <kad...@gmail.comwrote:
So if I understand what Michele wrote (thanks too), when a function is
defined (with def), no scope is saved and every variable value not
passed in parameter is lost ? It means that variable value come from
the outer scope when the function is called ?

Yes, in my example you get the value of "i" at the function *calling*
time,
not at the function definition time. If you want to store the value at
the
definition time, you must use the default argument trick:

funclist = []
for i in 1,2,3:
def f(i=i):
print i
funclist.append(f)

Michele Simionato
Thanks Michele, now I understand how it works and I learned something
new. Not a bad day...

Oct 15 '07 #7

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

4 posts views Thread by Paul Moore | last post: by
1 post views Thread by Fazer | last post: by
11 posts views Thread by mir nazim | last post: by
2 posts views Thread by Taki Jeden | last post: by
13 posts views Thread by Charlotte | last post: by
6 posts views Thread by Kartic | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.