yield, curry, mix-in, new.function, global, closure, .... what will work?

[ecir.hana]

Dear list,

maybe I'm overlooking something obvious or this is not possible at all
or I don't know. Please, consider the following code:

## insert here anything you like

def changer():
change_i('changed_i')
change_j('changed_j')

def run(i='', j=''):

## insert here anything you like

return i, j

run() == 'changed_i', 'changed_j'

Let me explain: First, changer() is kind of templating language so it
should be written down in this form - however, it can change during
run-time as you like. Basically, it is just ordinary python code which
changes (should change) the local variables of another function,
run(). Oh, and it has to be *thread-safe*.

Here's what I tried and didn't work (maybe I just haven't tried hard
enough):
- pass i, j around: changer(i, j) and change_i(i, arg) - easiest, most
obvious solution - can't do it, certainly not during writing of the
code - have to preserve changer()'s form
- global variable - not thread-safe
- in run(), create dummy closure for changer() and call it -
new.function(blah, blah, blah, dummy.func_closure) - not thread safe,
i guess, and besides change_i() and change_j() would also need a such
a trick - I am not sure how to alter the codeobject of changer()
- new.function(new.code(... - oh my, this won't work
- in run(), modify change_i() and change_j() so it actually yield-s
the changes, collect them here - something like: change_i.__call__ =
return yield ('changed_i', 'comes_from_changer_i()') - doesn't work,
of course
- update locals() - yes!! - no!! - doen't work, it's just a copy
- curry, mix-in, ... - hmm....
- eval(blah, blah, locals()) - not safe, ugly - maybe? what do you
think?
- find out which file changer() is written in, which line it starts
and ends, read that file's section, parse, edit, append the needed
local variable's declarations, compile to changer_new() and changer =
changer_new - there has to be something better! :)

So, to wrap up, the question is: how can I pass a variable from one
function to another, without writing down function's argument during
coding, and still be thread-safe?

I can only hope you are still with me and not very confused....

Thanks for any feedback!!

[Paul Rubin]

ec*******@gmail.com writes:

Let me explain: First, changer() is kind of templating language so it
should be written down in this form - however, it can change during
run-time as you like. Basically, it is just ordinary python code which
changes (should change) the local variables of another function,
run(). Oh, and it has to be *thread-safe*.

That is total madness. Just use a normal object or dictionary with a lock.

[ecir.hana]

On Apr 15, 8:07 pm, Paul Rubin <http://[e-mail]:

That is total madness. Just use a normal object or dictionary with a lock.

Please, can you elaborate further, I'm not sure if I understood.
Should I lock global variables i, j during the execution of run()? In
that case I have to apologize, I showed rather simplified version of
the actual problem I have - in fact changer() and run() will be a bit
more complex thus executing a bit longer and perhaps causing a dead-
lock.

[Paul Rubin]

ec*******@gmail.com writes:

On Apr 15, 8:07 pm, Paul Rubin <http://[e-mail]:

That is total madness. Just use a normal object or dictionary with a lock.

Please, can you elaborate further, I'm not sure if I understood.
Should I lock global variables i, j during the execution of run()? In
that case I have to apologize, I showed rather simplified version of
the actual problem I have - in fact changer() and run() will be a bit
more complex thus executing a bit longer and perhaps causing a dead-lock.

Put both variables into one shared object with a lock (see the docs for
threading.RLock()). Acquire the lock before modifying or reading the
variables, and release it afterwards. That is the traditional way.
Another way popular in Python is to have separate threads for changer
and run, and have them communicate through Queue.Queue objects. In no
case should you attempt to write a function that messes with another
function's local variables. Local variables are called "local" for a
reason ;-).

Basically this stuff takes more understanding and getting used to than
I can really convey in a newsgroup post. You might look at some
Wikipedia articles or ASPN recipes about concurrent programming, or
get the Python Cookbook.

[ecir.hana]

On Apr 16, 3:05 am, Paul Rubin <http://[e-mail]:

[e-mail] writes:

Please, can you elaborate further, I'm not sure if I understood.
Should I lock global variables i, j during the execution of run()? In
that case I have to apologize, I showed rather simplified version of
the actual problem I have - in fact changer() and run() will be a bit
more complex thus executing a bit longer and perhaps causing a dead-lock.

Put both variables into one shared object with a lock (see the docs for
threading.RLock()). Acquire the lock before modifying or reading the
variables, and release it afterwards. That is the traditional way.

Thanks for the reply! And at the same time, please bear with me.

If I understand correctly: when one thread acquires the lock, every
other thread has to wait. If so, this is not exacly what I would like
to have since the thread might take a bit longer to finish.

The reason why I try so hard to use local variables is that they are
inherently thread-safe. So I don't even mind to copy changer() every
time run() is called - run() has it's own local variables i, j, no one
has to touch them except it's ("local") function changer(). But the
problem is, I don't know how to propagate run()'s variables into
changer() without declarating them as changer()'s arguments (it would
be ok to append the declaration during run-time, though, if I only
knew how).

[Jason]

On Apr 16, 7:28 am, [e-mail] wrote:

On Apr 16, 3:05 am, Paul Rubin <http://[e-mail]:

[e-mail] writes:

Please, can you elaborate further, I'm not sure if I understood.
Should I lock global variables i, j during the execution of run()? In
that case I have to apologize, I showed rather simplified version of
the actual problem I have - in fact changer() and run() will be a bit
more complex thus executing a bit longer and perhaps causing a dead-lock.

Put both variables into one shared object with a lock (see the docs for
threading.RLock()). Acquire the lock before modifying or reading the
variables, and release it afterwards. That is the traditional way.

Thanks for the reply! And at the same time, please bear with me.

If I understand correctly: when one thread acquires the lock, every
other thread has to wait. If so, this is not exacly what I would like
to have since the thread might take a bit longer to finish.

The reason why I try so hard to use local variables is that they are
inherently thread-safe. So I don't even mind to copy changer() every
time run() is called - run() has it's own local variables i, j, no one
has to touch them except it's ("local") function changer(). But the
problem is, I don't know how to propagate run()'s variables into
changer() without declarating them as changer()'s arguments (it would
be ok to append the declaration during run-time, though, if I only
knew how).

In Python, names are bound to objects. The parameter names passed to
a function *are not inherently thread safe*! Python parameters are
not passed-by-value. To show you what I mean:

>>spam = ["delicious"]
def test(meal):

.... global spam
.... if spam is meal:
.... print "Spam is the same object as meal"
....

>>test(spam)

Spam is the same object as meal

(While the "global spam" statement is optional in this case, I wanted
to make it painfully obvious where the "spam" name in function test is
coming from.)

It is thread-safe to rebind the name "meal" in the function test (ie,
meal = "Green eggs"). It is not thread-safe to mutate or modify the
object that meal is bound to. In the example given above, appending
data to the list, removing data, changing elements, and other
operations will cause potential race conditions across multiple
threads.

Follow Paul's advice and get acquainted with the issues of concurrent
and threaded programming. Judicious locking will help avoid most race
conditions. If you don't want to keep other threads waiting, make a
copy of your data then release the data lock.

Depending on the data, you can usually have multiple threads "reading"
the data, as long as no other threads write to the data while there
are any readers. A writer can be allowed to change the data, but only
if there are no readers and no other writers. (This is commonly known
as a read/write lock.) I didn't see a read/write lock in the Python
documentation with some casual browsing, but one can be implemented
from the existing thread locking mechanisms.

Your description of what you want to do is rather vague, so I can't
get too specific. You've described how you want to do things, but I
don't know what you're trying to accomplish. Where possible, simplify
your design.

--Jason

[ecir.hana]

On Apr 16, 5:36 pm, "Jason" <[e-mail]:

On Apr 16, 7:28 am, [e-mail] wrote:

On Apr 16, 3:05 am, Paul Rubin <http://[e-mail]:

[e-mail] writes:

Please, can you elaborate further, I'm not sure if I understood.
Should I lock global variables i, j during the execution of run()? In
that case I have to apologize, I showed rather simplified version of
the actual problem I have - in fact changer() and run() will be a bit
more complex thus executing a bit longer and perhaps causing a dead-lock.

Put both variables into one shared object with a lock (see the docs for
threading.RLock()). Acquire the lock before modifying or reading the
variables, and release it afterwards. That is the traditional way.

Thanks for the reply! And at the same time, please bear with me.

If I understand correctly: when one thread acquires the lock, every
other thread has to wait. If so, this is not exacly what I would like
to have since the thread might take a bit longer to finish.

The reason why I try so hard to use local variables is that they are
inherently thread-safe. So I don't even mind to copy changer() every
time run() is called - run() has it's own local variables i, j, no one
has to touch them except it's ("local") function changer(). But the
problem is, I don't know how to propagate run()'s variables into
changer() without declarating them as changer()'s arguments (it would
be ok to append the declaration during run-time, though, if I only
knew how).

In Python, names are bound to objects. The parameter names passed to
a function *are not inherently thread safe*! Python parameters are
not passed-by-value. To show you what I mean:

>spam = ["delicious"]
def test(meal):

... global spam
... if spam is meal:
... print "Spam is the same object as meal"
...>>test(spam)

Spam is the same object as meal

(While the "global spam" statement is optional in this case, I wanted
to make it painfully obvious where the "spam" name in function test is
coming from.)

It is thread-safe to rebind the name "meal" in the function test (ie,
meal = "Green eggs"). It is not thread-safe to mutate or modify the
object that meal is bound to. In the example given above, appending
data to the list, removing data, changing elements, and other
operations will cause potential race conditions across multiple
threads.

Follow Paul's advice and get acquainted with the issues of concurrent
and threaded programming. Judicious locking will help avoid most race
conditions. If you don't want to keep other threads waiting, make a
copy of your data then release the data lock.

Depending on the data, you can usually have multiple threads "reading"
the data, as long as no other threads write to the data while there
are any readers. A writer can be allowed to change the data, but only
if there are no readers and no other writers. (This is commonly known
as a read/write lock.) I didn't see a read/write lock in the Python
documentation with some casual browsing, but one can be implemented
from the existing thread locking mechanisms.

Your description of what you want to do is rather vague, so I can't
get too specific. You've described how you want to do things, but I
don't know what you're trying to accomplish. Where possible, simplify
your design.

--Jason

All I was trying to do, was to get rid of those 'k's in changer():

def change_i(k, arg):
k[0] = arg

def change_j(k, arg):
k[1] = arg

def changer(k):
change_i(k, 'changed_i')
change_j(k, 'changed_j')

def run(i='', j=''):
k = [i, j]
changer(k)
[i, j] = k
return i, j

print run() == ('changed_i', 'changed_j')

Maybe I made a mistake, I should have asked this first, sorry. If the
only way to accomplish this is through locks, then I guess I better
use those 'k's, what do you think?

Thanks Jason, thanks Paul!

[ecir.hana]

I'm reading the docs now and I stumbled upon something:
section "15.3 threading -- Higher-level threading interface" mensions
a class "local", in which "... Thread-local data are data whose values
are thread specific. ..."

Does it mean, I can create global variables whose changing is thread-
safe?
More specific:

import threading

def change_i(arg):
global k
k.i = arg

def change_j(arg):
global k
k.j = arg

def changer():
change_i('changed_i')
change_j('changed_j')

def run(i='', j=''):
global k
k = threading.local()
k.i = i
k.j = j
changer()
i = k.i
j = k.j
return i, j

print run() == ('changed_i', 'changed_j')

Is this ok?

[ecir.hana]

On Apr 17, 3:51 am, [e-mail] wrote:

I'm reading the docs now and I stumbled upon something:
section "15.3 threading -- Higher-level threading interface" mensions
a class "local", in which "... Thread-local data are data whose values
are thread specific. ..."

Does it mean, I can create global variables whose changing is thread-
safe?
More specific:

import threading

def change_i(arg):
global k
k.i = arg

def change_j(arg):
global k
k.j = arg

def changer():
change_i('changed_i')
change_j('changed_j')

def run(i='', j=''):
global k
k = threading.local()
k.i = i
k.j = j
changer()
i = k.i
j = k.j
return i, j

print run() == ('changed_i', 'changed_j')

Is this ok?

I was too quick, "k = threading.local()" has to by outside of run(),
right?