How safe is modifying locals()?

[Paul Paterson]

I am trying to find a way to mimic by-reference argument passing for
immutables in Python. I need to do this because I am writing an
automated VB to Python converter.

Here's an example of the VB code:

Sub Change(ByVal x, ByRef y)
x = x+1
y = y+1
End Sub

x = 0: y = 0
Change x, y
' Now x should be 0 and y should be 1

One approach that has been suggested is to use locals() and indirect
references to the immutable via the resulting dictionary,

def change(x, y, refs):
x = x + 1
refs[y] = refs[y] + 1

x = 0; y = 0;
change(x, 'y', locals())

The Python documentation gives a warning that modifying the contents of
locals() may not affect local values. Despite the warning, the code
seems to work as desired, at least on Python 2.2.

Does anyone know how safe this approach is, particularly in a
multithreaded environment? Will this work on Jython?

Does anyone have any alternative strategies for trying to achieve the
objective? A "Pythonic" approach such as,

def change(x, y):
x = x + 1
y = y + 1
return y

x = 0; y = 0
y = change(x, y)

Works in a single threaded environment but, because the value of 'y'
changes at the wrong time, with multiple threads there is a possibility
that two users of 'y' could dissagree on its value.

Paul
Paul Paterson
(pa**********@u sers.sourceforg e.net)

vb2py :: A Visual Basic to Python Conversion Toolkit
http://vb2py.sourceforge.net

[Terry Reedy]

"Paul Paterson" <pa**********@u sers.sourceforg e.net> wrote in message
news:BD******** *************@t wister.austin.r r.com...

I am trying to find a way to mimic by-reference argument passing for
immutables in Python. I need to do this because I am writing an
automated VB to Python converter.

Here's an example of the VB code:

Sub Change(ByVal x, ByRef y)
x = x+1
y = y+1
End Sub

x = 0: y = 0
Change x, y
' Now x should be 0 and y should be 1

One approach that has been suggested is to use locals() and indirect
references to the immutable via the resulting dictionary,

def change(x, y, refs):
x = x + 1
refs[y] = refs[y] + 1

x = 0; y = 0;
change(x, 'y', locals())

The Python documentation gives a warning that modifying the contents of locals() may not affect local values. Despite the warning, the code
seems to work as desired, at least on Python 2.2.

At module scope, locals() == globals(). Within a function, locals()
currently is a *copy* of the function's local namespace. So above
only works because you made the call from the global (module) context
and would not if you tested by making call from within a function.

Terry J. Reedy

[Paul Paterson]

Terry Reedy wrote:

"Paul Paterson" <pa**********@u sers.sourceforg e.net> wrote in message

The Python documentation gives a warning that modifying the contents

of

locals() may not affect local values. Despite the warning, the code
seems to work as desired, at least on Python 2.2.

At module scope, locals() == globals(). Within a function, locals()
currently is a *copy* of the function's local namespace. So above
only works because you made the call from the global (module) context
and would not if you tested by making call from within a function.

Thanks Terry! I adjusted my tests and now see this behaviour exactly.
The function itself works but the changes do not propogate out to the
calling scope. So this approach of using locals() appears to be dead.

Are there any other approaches?

Paul

[Steven Taschuk]

Quoth Paul Paterson:
[...]

Does anyone have any alternative strategies for trying to achieve the
objective? A "Pythonic" approach such as,

def change(x, y):
x = x + 1
y = y + 1
return y

x = 0; y = 0
y = change(x, y)

Works in a single threaded environment but, because the value of 'y'
changes at the wrong time, with multiple threads there is a possibility
that two users of 'y' could dissagree on its value.

Even if the simple
y = y + 1
could change the caller's binding, there would be a race condition
-- you'll need a lock anyway. (I'd be a bit surprised if this
were not so in VB as well. Are statements atomic in VB?)

--
Steven Taschuk w_w
st******@telusp lanet.net ,-= U
1 1

[Shane Hathaway]

Paul Paterson wrote:

I am trying to find a way to mimic by-reference argument passing for
immutables in Python. I need to do this because I am writing an
automated VB to Python converter.

Here's an example of the VB code:

Sub Change(ByVal x, ByRef y)
x = x+1
y = y+1
End Sub

x = 0: y = 0
Change x, y
' Now x should be 0 and y should be 1

I might put all of the "ByRef" variables in a dictionary that gets
passed back to the caller through a hidden argument.

def change(x, y, _byref):
x = x + 1
y = _byref['y'] = y + 1

x = 0; y = 0
__byref = {'y': y}; change(x, y, __byref); y = __byref['y']

At least this relies on nothing magical. One thing that's still wrong,
though, is that you can't embed the change() call in an expression.
AFAIK, in Python, assignment to a local variable absolutely requires a
statement. You can get around this by splitting apart the expression
and storing the result of the call in another hidden temporary variable.

Even if you do that, though, exceptions will behave differently in
Python than they do in VB. If an exception occurs in change() after a
value has been assigned to 'y', the outer code won't get the value, and
it just might affect the behavior of the outer code. You could fix that
with a try: / finally:

__byref = {'y': y}
try:
change(x, y, __byref)
finally:
y = __byref['y']

Suddenly it's awfully verbose and ugly, but if variables by reference
are rare (as they should be), maybe it's not so bad. :-)

You could go the extreme route and store all variables in dictionaries
rather than use local variables, which would allow "true" references,
but then you'd kill speed and readability. Better not do that.

Shane

[Ian Bicking]

On Fri, 2003-07-25 at 12:28, Paul Paterson wrote:

Thanks Terry! I adjusted my tests and now see this behaviour exactly.
The function itself works but the changes do not propogate out to the
calling scope. So this approach of using locals() appears to be dead.

Are there any other approaches?

Think about what you are trying to do, and try to identify a (mutable)
object that can encapsulate that. Then pass the object, and modify its
instance variables, like:

class Point:
def __init__(self, x, y):
self.x = x
self.y = y

def change(p):
p.y = 10

obj = Point(0, 0)
change(obj)
assert obj.y == 10
Maybe this mutable object will simply be the main application object,
and instead of functions you will use methods of that application
object.

Ian

[Paul Paterson]

Ian Bicking wrote:

On Fri, 2003-07-25 at 12:28, Paul Paterson wrote:

Thanks Terry! I adjusted my tests and now see this behaviour exactly.
The function itself works but the changes do not propogate out to the
calling scope. So this approach of using locals() appears to be dead.

Are there any other approaches?

Think about what you are trying to do, and try to identify a (mutable)
object that can encapsulate that. Then pass the object, and modify its
instance variables, like:

class Point:
def __init__(self, x, y):
self.x = x
self.y = y

def change(p):
p.y = 10

obj = Point(0, 0)
change(obj)
assert obj.y == 10
Maybe this mutable object will simply be the main application object,
and instead of functions you will use methods of that application
object.

Ian

This is a very interesting (and Pythonic) approach, thanks for
suggesting it! This is certainly what I would try to do if I were
writing the code from scratch. It may be possible to construct a
"namespace" type object which gets passed to the function.

VB does have multiple namespaces so a single App object is probably not
feasible but since my parser knows the App structure it can determine
which namespace a variable will be resolved from anyway and just
translate all attempts to access it to the relevant namespace object lookup.

Paul

[Paul Paterson]

Shane Hathaway wrote:

Paul Paterson wrote:

I am trying to find a way to mimic by-reference argument passing for
immutables in Python. I need to do this because I am writing an
automated VB to Python converter.

Here's an example of the VB code:

Sub Change(ByVal x, ByRef y)
x = x+1
y = y+1
End Sub

x = 0: y = 0
Change x, y
' Now x should be 0 and y should be 1

I might put all of the "ByRef" variables in a dictionary that gets
passed back to the caller through a hidden argument.

def change(x, y, _byref):
x = x + 1
y = _byref['y'] = y + 1

x = 0; y = 0
__byref = {'y': y}; change(x, y, __byref); y = __byref['y']

The problem here is that you still have the delayed change to 'y' in the
caller's scope. Other posts in this thread have made me wonder whether
this is a problem sepcific to the ByRef topic.

The interesting thing about your approach is that you have an explicit
step for transmitting changes to y back to the calling scope - which is
quite nice. In an automated translation, having this explicit is a good
sign-post for the user to let them know that something important is
going on.
At least this relies on nothing magical. One thing that's still wrong,
though, is that you can't embed the change() call in an expression.
AFAIK, in Python, assignment to a local variable absolutely requires a
statement. You can get around this by splitting apart the expression
and storing the result of the call in another hidden temporary variable.

Even if you do that, though, exceptions will behave differently in
Python than they do in VB.
Matching VB's exception behaviour is my worst nightmare - but that's
another story ...
If an exception occurs in change() after a
value has been assigned to 'y', the outer code won't get the value, and
it just might affect the behavior of the outer code. You could fix that
with a try: / finally:

__byref = {'y': y}
try:
change(x, y, __byref)
finally:
y = __byref['y']

Suddenly it's awfully verbose and ugly, but if variables by reference
are rare (as they should be), maybe it's not so bad. :-)
Unfortunately (and to my shock), it seems that ByRef is the *default*!
It may be the case that using the implications of ByRef are rare and it
may be possible for me to detect when this is occuring.
You could go the extreme route and store all variables in dictionaries
rather than use local variables, which would allow "true" references,
but then you'd kill speed and readability. Better not do that.

At the end of the day (see my response to Ian) this may in fact be the
safest approach if you want to match the exact behaviour of the original
VB code. There will almost certainly be a switch where you can turn this
off, but then it is up to the programmer to go and fix things which get
broken. Hopefully it will be possible to try to identify likely hot
spots automatically.

Paul

[Stefan Schwarzer]

Hi Paul

Paul Paterson wrote:

This is a very interesting (and Pythonic) approach, thanks for
suggesting it! This is certainly what I would try to do if I were
writing the code from scratch. It may be possible to construct a
"namespace" type object which gets passed to the function.

I think the most common way to make a namespace in Python is

class my_namespace: pass
....
my_namespace.a = 'foo'
my_namespace.b = 'bar'

Such namespaces are very similar to dictionaries:

my_namespace = {}
....
my_namespace['a'] = 'foo'
my_namespace['b'] = 'bar'

but the first approach is a bit more readable.

If you need many containers, you could use:

class Container:
pass

c1 = Container()
c2 = Container()
....
c1.foo = 'foo'
c2.foo = 'bar'

Of course, if you have a better name for your container, that's even
better. Think of Ian's Point example. :-)

Stefan

[Paul Paterson]

Stefan Schwarzer wrote:

Hi Paul

Paul Paterson wrote:

This is a very interesting (and Pythonic) approach, thanks for
suggesting it! This is certainly what I would try to do if I were
writing the code from scratch. It may be possible to construct a
"namespace" type object which gets passed to the function.

I think the most common way to make a namespace in Python is

<snip nice example of namespace class>

Thanks, the bare namespace class is essentially what I am going to try.

Of course, if you have a better name for your container, that's even
better. Think of Ian's Point example. :-)

I think you may have missed my original post; I am writing a generic VB
to Python converter so generating good names based on code intent would
be pretty tough! Perhaps in v2.0 ;)

My current thinking is (for the orignal example of changing one
variable, 'y' but not another, 'x'),

class Namespace: pas

# ByVal arguments passed directly, ByRef via a namespace
def change(x, byrefs):
x = x + 1
byrefs.y = byrefs.y + 1

ns = Namespace() # The local namespace
ns.x = 0
ns.y = 0
change(ns.x, ns)
# Now ns.x = 0, ns.y = 1
Paul