By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
458,249 Members | 1,633 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 458,249 IT Pros & Developers. It's quick & easy.

changing local namespace of a function

P: n/a
Dear list,

I have many dictionaries with the same set of keys and I would like to
write a function to calculate something based on these values. For
example, I have

a = {'x':1, 'y':2}
b = {'x':3, 'y':3}

def fun(dict):
dict['z'] = dict['x'] + dict['y']

fun(a) and fun(b) will set z in each dictionary as the sum of x and y.

My function and dictionaries are a lot more complicated than these so I
would like to set dict as the default namespace of fun. Is this
possible? The ideal code would be:

def fun(dict):
# set dict as local namespace
# locals() = dict?
z = x + y

Many thanks in advance.
Bo
Jul 18 '05 #1
Share this Question
Share on Google+
31 Replies


P: n/a
Hello Bo,
Don't use dict it is a builtin ;)

Also try this stuff out in an interpreter session it is easy and fast
to get your own answers.
def fun(d):

.... __dict__ = d
.... return __dict__
hth,
M.E.Farmer

Jul 18 '05 #2

P: n/a
Bo Peng wrote:
Dear list,

I have many dictionaries with the same set of keys and I would like to
write a function to calculate something based on these values. For
example, I have

a = {'x':1, 'y':2}
b = {'x':3, 'y':3}

def fun(dict):
dict['z'] = dict['x'] + dict['y']

fun(a) and fun(b) will set z in each dictionary as the sum of x and y.

My function and dictionaries are a lot more complicated than these so I
would like to set dict as the default namespace of fun. Is this
possible? The ideal code would be:

def fun(dict):
# set dict as local namespace
# locals() = dict?
z = x + y

As you no doubt have discovered from the docs and this group, that isn't doable
with CPython.

If you must write your functions as real functions, then you might do something
like this:
a = {'x':1, 'y':2}
b = {'x':3, 'y':3} ... def funa(x,y, **kw): ... del kw #Careful of unwanted names in locals with this approach
... z = x + y
... return locals()
... a.update(funa(**a))
b.update(funa(**b))
a {'y': 2, 'x': 1, 'z': 3} b {'y': 3, 'x': 3, 'z': 6}

Alternatively, you could use exec:
a = {'x':1, 'y':2}
b = {'x':3, 'y':3}
exec "z = x + y" in globals(), a
a {'y': 2, 'x': 1, 'z': 3} exec "z = x + y" in globals(), b
b {'y': 3, 'x': 3, 'z': 6}


Michael

Jul 18 '05 #3

P: n/a
Bo Peng wrote:
My function and dictionaries are a lot more complicated than these so I
would like to set dict as the default namespace of fun.


This sounds to me like you're trying to re-implement object orientation.

Turn all of those functions into methods on a class, and instead of
creating dictionaries that'll be passed into functions, create class
instances.

class MyClass(object):
def __init__(self, **kwargs):
for key, val in kwargs:
setattr(self, key, val)
def fun(self):
self.z = self.y + self.x

a = MyClass(x=1, y=2)
a.fun()
print a.z

Jeff Shannon
Technician/Programmer
Credit International
Jul 18 '05 #4

P: n/a
M.E.Farmer wrote:
def fun(d):
... __dict__ = d
... return __dict__
hth,

Does not work?
a = { 'x':1, 'y':2}
b = { 'x':2, 'y':9}
def fun(d): .... __dict__ = d
.... print locals()
.... z = x + y fun(a)

{'__dict__': {'y': 2, 'x': 1}, 'd': {'y': 2, 'x': 1}}
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/tmp/python-6377tax.py", line 4, in fun
NameError: global name 'x' is not defined
Jul 18 '05 #5

P: n/a
Michael Spencer wrote:
As you no doubt have discovered from the docs and this group, that isn't
doable with CPython.
Too bad to know this.
>>> a = {'x':1, 'y':2}
>>> b = {'x':3, 'y':3} ... >>> def funa(x,y, **kw):

... del kw #Careful of unwanted names in locals with this approach
... z = x + y
... return locals()


There are hundreds of items in the dictionary (that will be needed in
the calculation) so passing the whole dictionary is a lot better than
passing individual items.
Alternatively, you could use exec:


Yes. I thought of using exec or eval. If there are a dozen statements,

def fun(d):
exec 'z = x + y' in globals(), d

seems to be more readable than

def fun(d):
d['z'] = d['x'] + d['y']

But how severe will the performance penalty be?

Bo

Jul 18 '05 #6

P: n/a
Jeff Shannon wrote:

This sounds to me like you're trying to re-implement object orientation.


I have no control over the big dictionaries. All I need to do is
processing them in situ --- that is to say, go into each map and
manipulate numbers. Parameter passing should be avoid whenever possible
since involved number of items are huge.

It is basically like

exec 'statements' in d

but a function with local dictionary d would be best.

Bo
Jul 18 '05 #7

P: n/a
Bo Peng wrote:
Michael Spencer wrote:
There are hundreds of items in the dictionary (that will be needed in
the calculation) so passing the whole dictionary is a lot better than
passing individual items. ....
def fun(d):
exec 'z = x + y' in globals(), d

seems to be more readable than

def fun(d):
d['z'] = d['x'] + d['y']

But how severe will the performance penalty be?
Try it and see.
Bo


Compare it with Jeff Shannon's suggestion, and with a lazy dict-wrapper like this:
class wrapbigdict(object): ... """Lazy attribute access to dictionary keys. Will not access
... keys that are not valid attribute names!"""
... def __init__(self, mydict):
... object.__setattr__(self, "mydict",mydict)
... def __getattr__(self, attrname):
... return self.mydict[attrname]
... def __setattr__(self, attrname, value):
... self.mydict[attrname] = value
...
... a = {'x':1, 'y':2}
b = {'x':3, 'y':3} ... w_a = wrapbigdict(a)
w_b = wrapbigdict(b) ... def fun(d): ... d.z = d.x + d.y
... fun(w_a)
fun(w_b) ... w_a.mydict {'y': 2, 'x': 1, 'z': 3} w_b.mydict {'y': 3, 'x': 3, 'z': 6}

Jul 18 '05 #8

P: n/a
Michael Spencer wrote:
def fun(dict):
# set dict as local namespace
# locals() = dict?
z = x + y


As you no doubt have discovered from the docs and this group, that isn't
doable with CPython.


Not entirely impossible:

Py> def f(d):
.... exec "locals().update(d)"
.... return x + y
....
Py> f(dict(x=1, y=2))
3

Due to the way 'exec' is implemented, modifications to locals() inside an exec
statement actually take effect (basically, they're freeloading on the code which
allows 'exec "x = 1"' to work properly).

This is an evil, evil hack and almost certainly not what anyone should be doing.
Also, variables created this way will be slower than normal variables due to the
way the associated code works.

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net
Jul 18 '05 #9

P: n/a
Bo Peng wrote:
Jeff Shannon wrote:

This sounds to me like you're trying to re-implement object orientation.

I have no control over the big dictionaries. All I need to do is
processing them in situ --- that is to say, go into each map and
manipulate numbers. Parameter passing should be avoid whenever possible
since involved number of items are huge.


No, all parameter passing does is pass a pointer to the dictionary - it doesn't
copy the dictionary itself.
It is basically like

exec 'statements' in d

but a function with local dictionary d would be best.


Why? How do you know that the performance is unacceptable?

If you *really* want to execute a function with different locals, then you can do:

Py> def f():
.... print x + y
....
Py> exec f.func_code in dict(x=1, y=2)
3

This is unreliable though, since name binding doesn't work correctly:

Py> def f():
.... z = x + y
....
Py> d = dict(x=1, y=2)
Py> exec f.func_code in d
Py> d['z]
Py> d['z']
Traceback (most recent call last):
File "<stdin>", line 1, in ?
KeyError: 'z'
Py> exec "z = x + y" in d
Py> d['z']
3

An approach like Michael's is going to be much easier to debug, much easier to
understand later, and far more reliable all the way along. The performance is
almost certainly going to be acceptable.

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net
Jul 18 '05 #10

P: n/a
Nick Coghlan wrote:
Michael Spencer wrote:
def fun(dict):
# set dict as local namespace
# locals() = dict?
z = x + y

As you no doubt have discovered from the docs and this group, that
isn't doable with CPython.

Not entirely impossible:

Py> def f(d):
... exec "locals().update(d)"
... return x + y
...
Py> f(dict(x=1, y=2))
3

Due to the way 'exec' is implemented, modifications to locals() inside
an exec statement actually take effect (basically, they're freeloading
on the code which allows 'exec "x = 1"' to work properly).

This is an evil, evil hack and almost certainly not what anyone should
be doing. Also, variables created this way will be slower than normal
variables due to the way the associated code works.

Cheers,
Nick.

Oooh - evil indeed, but thanks for the pointer.

I debated including a link to one of the 'writable locals' threads, when I
settled on not 'doable', but gambled on being probably useful rather than
certainly accurate. Just goes to show you can't get away with anything in this
NG ;-)

Cheers

Michael

Jul 18 '05 #11

P: n/a
Thank all for your suggestions. I have tried all methods and compared
their performance.
import profile
a = {'x':1, 'y':2}
N = 100000
# solution one: use dictionary directly .... def fun1(d):
.... for i in xrange(0,N):
.... d['z'] = d['x'] + d['y']
.... # solution two: use exec .... def fun2(d):
.... for i in xrange(0,N):
.... exec 'z = x + y' in globals(), d
.... # solution three: update local dictionary .... # Note that locals() is *not* d after update() so
.... # z = x + y
.... # does not set z in d
.... def fun3(d):
.... exec "locals().update(d)"
.... for i in xrange(0,N):
.... d['z'] = x + y
.... # solution four: use dict wrapper .... # this makes code easier to write and read
.... class wrapdict(object):
.... """Lazy attribute access to dictionary keys. Will not access
.... keys that are not valid attribute names!"""
.... def __init__(self, mydict):
.... object.__setattr__(self, "mydict",mydict)
.... def __getattr__(self, attrname):
.... return self.mydict[attrname]
.... def __setattr__(self, attrname, value):
.... self.mydict[attrname] = value
.... # use wrapper .... def fun4(d):
.... wd = wrapdict(d)
.... for i in xrange(0,N):
.... wd.z = wd.x + wd.y
.... profile.run('fun1(a)') 3 function calls in 0.070 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.060 0.060 <string>:1(?)
1 0.010 0.010 0.070 0.070 profile:0(fun1(a))
0 0.000 0.000 profile:0(profiler)
1 0.060 0.060 0.060 0.060 python-4645vcY.py:2(fun1)

profile.run('fun2(a)') 100003 function calls (3 primitive calls) in 5.890 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
100001/1 0.440 0.000 5.890 5.890 <string>:1(?)
1 0.000 0.000 5.890 5.890 profile:0(fun2(a))
0 0.000 0.000 profile:0(profiler)
1 5.450 5.450 5.890 5.890 python-46458me.py:2(fun2)

profile.run('fun3(a)') 4 function calls (3 primitive calls) in 0.060 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
2/1 0.000 0.000 0.060 0.060 <string>:1(?)
1 0.000 0.000 0.060 0.060 profile:0(fun3(a))
0 0.000 0.000 profile:0(profiler)
1 0.060 0.060 0.060 0.060 python-4645Jxk.py:5(fun3)

profile.run('fun4(a)')

300004 function calls in 3.910 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 3.910 3.910 <string>:1(?)
1 0.000 0.000 3.910 3.910 profile:0(fun4(a))
0 0.000 0.000 profile:0(profiler)
100000 0.530 0.000 0.530 0.000
python-4645W7q.py:10(__setattr__)
1 0.000 0.000 0.000 0.000 python-4645W7q.py:6(__init__)
200000 0.960 0.000 0.960 0.000
python-4645W7q.py:8(__getattr__)
1 2.420 2.420 3.910 3.910 python-4645jFx.py:1(fun4)

Exec is slow since compiling the string and calls to globals() use a lot
of time. The last one is most elegant but __getattr__ and __setattr__
are costly. The 'evil hack' solution is good since accessing x and y
takes no additional time.

I guess I will go with solution 3. It is evil but it is most close to my
original intention. It leads to most readable code (except for the first
line to do the magic and the last line to return result) and fastest
performance.

Thank again for everyone's help. I have learned a lot from the posts,
especially the wrapdict class.

Bo
Jul 18 '05 #12

P: n/a
I quote myself :
Also try this stuff out in an interpreter session it is easy and fast
to get your own answers.

Sorry I guess I should have added a nudge and a <.5 wink> at the end.
Sometimes misinformation is just what we need to get our own answers!
The absurdity of what you are doing led me to give you that piece of
advice.
It seems if I had just told you to RTF you would not have learned as
much.
I have been out so did not respond earlier, but I see others have given
you other useful information. There is no reason to make these local
epecially if they are large , leave them in the dictionary they come in
and acess the members as needed. I really don't see your need.
M.E.Farmer

Jul 18 '05 #13

P: n/a
M.E.Farmer wrote:
I really don't see your need.


Maybe it is just my laziness. It is almost intolerable for me to write
lines and lines of code like

d['z'] = func(d['x']+d['y']+d['whatever']['as']+d[a][0] )

It is ugly, unreadable and error prone. If I have to use this code, I
would write

_z = func(_x + _y + _whatever['as'] + _a[0])

and use a perl script to generate the real code. (See, I am not lazy :-)

Bo

Jul 18 '05 #14

P: n/a
> Maybe it is just my laziness. It is almost intolerable for me to write
lines and lines of code like

d['z'] = func(d['x']+d['y']+d['whatever']['as']+d[a][0] )


By the way, will 'with statement', like the one in pascal and many other
languages, be a good addition to python? For example,

with d do:
z = x + y

would be equivalent to d['z']=d['x']+d['y'] or d.z = d.x + d.y in some
other cases.

This would absolutely be the *best* solution to my problem.

Bo
Jul 18 '05 #15

P: n/a
>It is ugly, unreadable and error prone. If I have to use this code, I
would write
_z = func(_x + _y + _whatever['as'] + _a[0])
and use a perl script to generate the real code. (See, I am not lazy

:-)
Ok laziness is an acceptable answer ;)
This is starting to make sense , you've been reading Perl's 'dense'
code and are not used to Python's explicitness.
*Explicit is better than implicit!*
It is a Zen of Python. Also I notice you say you will probably use the
'evil hack'.
Uhmmm, it is labeled evil hack for a reason! So go ahead and make life
hard and save a few keystrokes. There is a reason for the verbosity of
Python.
I really don't completly follow what you are trying to construct here
but it seems like you need to create an adapter class of some sort and
use that ( much like Jeff Shannon or Michael Spencer mentioned )
As a general rule class abstraction is your friend in Python.
It looks like ultimatly you are just using dictionary's and want to do
inplace operations on some of the data and store the results in other
parts of the dictionary... If that is the case then why not creat a few
helper functions that do exactly one thing each and use them as your
interface.

Py> def fun_helper(d):
.... d['z'] = func(d['x']+d['y']+d['whatever']['as']+d[a][0])
.... return d

and use them as needed.

M.E.Farmer

Jul 18 '05 #16

P: n/a
Bo Peng wrote:
By the way, will 'with statement', like the one in pascal and many other
languages, be a good addition to python? For example,

with d do:
z = x + y

would be equivalent to d['z']=d['x']+d['y'] or d.z = d.x + d.y in some
other cases.

This would absolutely be the *best* solution to my problem.

Bo


Guido van Rossum has stated that he wants Python to eventually have a 'with'
statement of the form:

with d:
.z = .x + .y

(The leading '.' being required to avoid ambiguity in name lookups)

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net
Jul 18 '05 #17

P: n/a
Bo Peng wrote:
I guess I will go with solution 3. It is evil but it is most close to my
original intention. It leads to most readable code (except for the first
line to do the magic and the last line to return result) and fastest
performance.
Thousands of programs use Python's class attribute access syntax day-in and
day-out and find the performance to be acceptable.

Premature optimization is the root of much evil - the exec + locals() hack if
very dependent on the exact Python version you are using. Expect some hard to
find bugs if you actually use the hack.

And code which relies on an evil hack in order to have the desired effect
doesn't count as readable in my book - I would be very surprised if many people
reading your code knew that updates to locals() can sometimes be made to work by
performing them inside a bare exec statement.
Thank again for everyone's help. I have learned a lot from the posts,
especially the wrapdict class.


It sounds like the dictionaries you are working with actually have some
meaningful content that you want to manipulate.

I'd seriously suggest creating a class which is able to extract the relevant
data from the dictionaries, supports the desired manipulations, and then be
instructed to update the original dictionaries.

For example, an approach based on Michael's dictionary wrapper class

class DataAccessor(object):
# Just a different name for Michael's "wrapbigdict"
def __init__(self, data):
object.__setattr__(self, "_data", data)
def __getattr__(self, attrname):
return self._data[attrname]
def __setattr__(self, attrname, value):
self._data[attrname] = value

class DataManipulator(DataAccessor):
def __init__(self, data):
DataAccessor.__init__(self, data)
def calc_z(self):
self.z = self.x + self.y

In action:

Py> data = DataManipulator(dict(x=1, y=2))
Py> data.z
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 6, in __getattr__
KeyError: 'z'
Py> data.calc_z()
Py> data.z
3
Py> data._data
{'y': 2, 'x': 1, 'z': 3}

The class based approach also gives you access to properties, which can be used
to make that first call to 'z' automatically calculate the desired result
instead of giving a KeyError:

class EnhancedDataManipulator(DataAccessor):
def __init__(self, data):
DataAccessor.__init__(self, data)

class _utils(object):

@staticmethod
def make_prop(name, calculator):
def get(self, _name=name, _calculator=calculator):
try:
return self._data[_name]
except KeyError:
val = _calculator(self)
self._data[_name] = val
return val
def set(self, val, _name=name):
self._data[_name] = val
def delete(self, _name=name):
del self._data[_name]
return property(get, set, delete)

@staticmethod
def calc_z(self):
return self.x + self.y

z = _utils.make_prop('z', _utils.calc_z)

Note the nested _utils class isn't strictly necessary. I just like to use it to
separate out the information which the class uses to construct itself from those
which are provided to clients of the class.

Anyway, the enhanced version in action:

Py> data = EnhancedDataManipulator(dict(x=1,y=2))
Py> data.x
1
Py> data.y
2
Py> data.z
3
Py> data.x = 30
Py> data.z
3
Py> data.z = 10
Py> data.z
10
Py> del data.z
Py> data.z
32

If you want to add more calculated properties to the data manipulator, simply
define additional calculator methods, and define the attribute with make_prop.

Use a class. Giving meaningful form to a collection of attributes is what
they're for, and Python provides many features designed to make that process easier.

Trying to fake it with function namespaces is just going to make trouble for you
in the future.

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net
Jul 18 '05 #18

P: n/a
Bo Peng <bp***@rice.edu> wrote:
M.E.Farmer wrote:
I really don't see your need.


Maybe it is just my laziness. It is almost intolerable for me to write
lines and lines of code like

d['z'] = func(d['x']+d['y']+d['whatever']['as']+d[a][0] )

It is ugly, unreadable and error prone. If I have to use this code, I
would write

_z = func(_x + _y + _whatever['as'] + _a[0])


So, what about having as the real code
d.z = func(d.x + d.y + d.whatever['as'] + d.a[0])
Doesn't seem too different from what you would write, would it?

To wrap a dictionary into an object whose attributes are mapped to items
of the dictionary is not hard: just start your function with

d = type('wrapadict', (), {})()
d.__dict__ = the_dict_to_wrap

and there you are.

Affecting *BARE* names is a completely different order of magnitude of
complication away from affecting *COMPOUND* names. Barenames are,
almost inevitably, handled with deep magic, essentially for the purposes
of can't-do-without optimization; compound names are _way_ more
malleable...
Alex
Jul 18 '05 #19

P: n/a
Bo Peng <bp***@rice.edu> wrote:
...
Thank again for everyone's help. I have learned a lot from the posts,
especially the wrapdict class.


Hmmm, you do realize that wrapdict uses a lot of indirection while my
equivalent approach, just posted, is very direct, right? To reiterate
the latter, and dress it up nicely too, it's

class wrapwell(object):
def __init__(self, somedict):
self.__dict__ = somedict

and use d=wrapwell(whateverdict) to make the wrapper. Now, reading or
writing d.x is just like reading or writing whateverdict['x'], etc etc.
Alex
Jul 18 '05 #20

P: n/a
Alex Martelli wrote:
Bo Peng <bp***@rice.edu> wrote:
...
Thank again for everyone's help. I have learned a lot from the posts,
especially the wrapdict class.

Hmmm, you do realize that wrapdict uses a lot of indirection while my
equivalent approach, just posted, is very direct, right? To reiterate
the latter, and dress it up nicely too, it's

class wrapwell(object):
def __init__(self, somedict):
self.__dict__ = somedict

and use d=wrapwell(whateverdict) to make the wrapper. Now, reading or
writing d.x is just like reading or writing whateverdict['x'], etc etc.


I was wondering if that would work, but never got around to trying it. It can be
used with the property-based auto calculation approach, too (the calculated
properties just store the results in self.__dict__ instead of self._data):

class DataManipulator(object):
def __init__(self, data):
self.__dict__ = data

class _utils(object):
@staticmethod
def make_prop(name, calculator):
def get(self, _name=name, _calculator=calculator):
try:
return self.__dict__[_name]
except KeyError:
val = _calculator(self)
self.__dict__[_name] = val
return val
def set(self, val, _name=name):
self.__dict__[_name] = val
def delete(self, _name=name):
del self.__dict__[_name]
return property(get, set, delete)

@staticmethod
def calc_z(self):
return self.x + self.y

z = _utils.make_prop('z', _utils.calc_z)

Py> d = DataManipulator(dict(x=1, y=2))
Py> d.x
1
Py> d.y
2
Py> d.z
3
Py> d.x = 10
Py> d.z
3
Py> del d.z
Py> d.z
12

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net
Jul 18 '05 #21

P: n/a
Bo Peng wrote:
Dear list,

I have many dictionaries with the same set of keys and I would like to
write a function to calculate something based on these values. For
example, I have

a = {'x':1, 'y':2}
b = {'x':3, 'y':3}

def fun(dict):
dict['z'] = dict['x'] + dict['y']

fun(a) and fun(b) will set z in each dictionary as the sum of x and y.

My function and dictionaries are a lot more complicated than these so I
would like to set dict as the default namespace of fun. Is this
possible? The ideal code would be:

def fun(dict):
# set dict as local namespace
# locals() = dict?
z = x + y


You can part way there using keyword arguments. You just have to use dictionary syntax for changing
values in the dictionary:
def f(d, x=None, y=None): ... d['z'] = x + y
... a = {'x':1, 'y':2}
b = {'x':3, 'y':3}

f(a, **a)
a {'y': 2, 'x': 1, 'z': 3} f(b, **b)
b

{'y': 3, 'x': 3, 'z': 6}

Kent
Jul 18 '05 #22

P: n/a
Nick Coghlan wrote:

If you want to add more calculated properties to the data manipulator,
simply define additional calculator methods, and define the attribute
with make_prop.


This has became really appealing....

You know, I have a deep root in C/C++ so performance is the king and
hacking is part of my daily life. Time to change now. :)

My original problem was that I need to manipulate and create dictionary
items in situ and since there are so many "d['key']"s, I would like to
make my code easier to read (and save some key strokes) by using "d.key"
or simply "key", without sacrificing performance. OOP seemed to be
irrelevant.

But this on-demand calculation (quote: make first call to z automatic)
is *really* good! I can calculate self._data.stat_a = self._data.stat_b
+ 1 without worrying about the status of stat_b and users can access any
value in the dictionary, original or calculated, in a uniform and
easier way! This makes my program instantly easier to use.

I can not say enough thank you for this.

Bo
Jul 18 '05 #23

P: n/a
Bo Peng wrote:
Yes. I thought of using exec or eval. If there are a dozen statements,

def fun(d):
exec 'z = x + y' in globals(), d

seems to be more readable than

def fun(d):
d['z'] = d['x'] + d['y']

But how severe will the performance penalty be?


You can precompile the string using compile(), you only have to do this once.
def makeFunction(funcStr, name): ... code = compile(funcStr, name, 'exec')
... def f(d):
... exec code in d
... del d['__builtins__'] # clean up extra entry in d
... return f
... f = makeFunction('z = x + y', 'f')
a = {'x':1, 'y':2}
b = {'x':3, 'y':3}
f(a)
a {'y': 2, 'x': 1, 'z': 3} f(b)
b

{'y': 3, 'x': 3, 'z': 6}
Jul 18 '05 #24

P: n/a
Bo Peng wrote:
I can not say enough thank you for this.


Don't thank me, thank Guido. He created the property machinery - I just let you
know it was there :)

But yes, Python's OO is OO the way it should be - something that helps you get
the job done quickly and cleanly, rather than making you jump through irrelevant
hoops.

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net
Jul 18 '05 #25

P: n/a
Something I forgot to mention. . .

Bo Peng wrote:
You know, I have a deep root in C/C++ so performance is the king and
hacking is part of my daily life. Time to change now. :)


The entire design of C++ is in many ways a regrettable monument to the idea that
premature optimisation is evil - far too many of the language's default
behaviours are 'fast in particular cases, but quite simply wrong in most cases,
and the compiler often can't tell you which case applies'. So you can write
buggy warning free code just by failing to override the unsafe default behaviours.

That said, one of the reasons I like CPython is that it lets you drop into C/C++
really easily when you need to (generally for hardware interface or performance
reasons). The builtin profiler can also give some decent hints on the
bottlenecks where you need to focus your performance improvements.

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net
Jul 18 '05 #26

P: n/a
> Alex Martelli wrote:
Hmmm, you do realize that wrapdict uses a lot of indirection while my
equivalent approach, just posted, is very direct, right? To reiterate
the latter, and dress it up nicely too, it's

class wrapwell(object):
def __init__(self, somedict):
self.__dict__ = somedict

Bad mistake on my part, sorry!
Nick Coghlan wrote:
... a class that combined property access with the above...

In a similar vein to Nick's solution:

class AutoProperty(object):
def __init__(self, meth):
self.meth = meth
self.name = meth.__name__
self.__doc__ = meth.__doc__
def __get__(self, obj, cls):
if isinstance(obj, cls):
return obj.__dict__.setdefault(self.name, self.meth(obj))
else:
return self.__doc__
# You could define __set__ and __del__ but they don't seem
# necessary based on what you've said so far
class DataManipulator(object):
def __init__(self, data):
self.__dict__ = data

class Model(DataManipulator):
def z(self):
"""x+y"""
return self.x+self.y
z = AutoProperty(z)

def z1(self):
"""Or any other useful information"""
return self.z + self.x
z1 = AutoProperty(z1)

# You could automate these calls to AutoProperty in a metaclass
a = {'x':1, 'y':2}
b = {'x':3, 'y':3}
d = Model(a)
d.z 3 d.z1 4 a {'y': 2, 'x': 1, 'z': 3, 'z1': 4} d= Model(b)
d.z1 9 b {'y': 3, 'x': 3, 'z': 6, 'z1': 9}


Michael
Jul 18 '05 #27

P: n/a
Kent Johnson wrote:
You can part way there using keyword arguments. You just have to use
dictionary syntax for changing values in the dictionary:
>>> def f(d, x=None, y=None): ... d['z'] = x + y
... >>> a = {'x':1, 'y':2}
>>> b = {'x':3, 'y':3}
>>>
>>> f(a, **a)
>>> a {'y': 2, 'x': 1, 'z': 3} >>> f(b, **b)
>>> b {'y': 3, 'x': 3, 'z': 6}


This is not possible in my case since my dictionary have many more items
than just x and y. So, if there is are other items in the dictionary
a = {'x':1, 'y':2, 'else':4}
f(a,**a)

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/tmp/python-10176RtT.py", line 1, in ?
f(a,**a)
TypeError: f() got an unexpected keyword argument 'else'

Bo
Jul 18 '05 #28

P: n/a
> Exec is slow since compiling the string and calls to globals() use a lot
of time. The last one is most elegant but __getattr__ and __setattr__
are costly. The 'evil hack' solution is good since accessing x and y
takes no additional time.


Previous comparison was not completely fair since I could pre-compile
fun2 and I used indirect __setattr__. Here is the new one:

import profile
a = {'x':1, 'y':2}
N = 100000
# solution one: use dictionary directly .... def fun1(d):
.... for i in xrange(0,N):
.... d['z'] = d['x'] + d['y']
.... # solution two: use exec .... def makeFunction(funcStr, name):
.... code = compile(funcStr, name, 'exec')
.... def f(d):
.... exec code in d
.... return f
.... def fun2(d): .... myfun = makeFunction('z = x + y', 'myfun')
.... for i in xrange(0,N):
.... myfun(d)
.... del d['__builtins__']
....
.... # solution three: update local dictionary # Note that locals() is NOT d after update() so .... # z = x + y
.... # does not set z in d
.... def fun3(d):
.... exec "locals().update(d)"
.... for i in xrange(0,N):
.... d['z'] = x + y
.... # solution four: use dict wrapper .... # this makes code easier to write and read
.... class wrapdict(object):
.... """Lazy attribute access to dictionary keys. Will not access
.... keys that are not valid attribute names!"""
.... def __init__(self, mydict):
.... self.__dict__ = mydict
....
.... # use wrapper def fun4(d): .... wd = wrapdict(d)
.... for i in xrange(0,N):
.... wd.z = wd.x + wd.y
.... profile.run('fun1(a)') 3 function calls in 0.060 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.060 0.060 <string>:1(?)
1 0.000 0.000 0.060 0.060 profile:0(fun1(a))
0 0.000 0.000 profile:0(profiler)
1 0.060 0.060 0.060 0.060 python-10176FWs.py:2(fun1)

profile.run('fun2(a)') 200004 function calls in 2.130 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 2.130 2.130 <string>:1(?)
100000 0.520 0.000 0.520 0.000 myfun:1(?)
1 0.000 0.000 2.130 2.130 profile:0(fun2(a))
0 0.000 0.000 profile:0(profiler)
1 0.590 0.590 2.130 2.130 python-10176EqB.py:1(fun2)
1 0.000 0.000 0.000 0.000
python-10176Sgy.py:2(makeFunction)
100000 1.020 0.000 1.540 0.000 python-10176Sgy.py:4(f)

profile.run('fun3(a)') 4 function calls (3 primitive calls) in 0.070 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
2/1 0.000 0.000 0.070 0.070 <string>:1(?)
1 0.000 0.000 0.070 0.070 profile:0(fun3(a))
0 0.000 0.000 profile:0(profiler)
1 0.070 0.070 0.070 0.070 python-10176R0H.py:4(fun3)

profile.run('fun4(a)')

4 function calls in 0.100 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.100 0.100 <string>:1(?)
1 0.000 0.000 0.100 0.100 profile:0(fun4(a))
0 0.000 0.000 profile:0(profiler)
1 0.000 0.000 0.000 0.000
python-10176e-N.py:6(__init__)
1 0.100 0.100 0.100 0.100 python-10176rIU.py:1(fun4)
Since

d['x'] is fast but cumbersome
exec "z=x+y' is still slow.
exec "locals().update(d)" is evil
d.x is elegant and only a little slower than d['x']

I am announcing the winner of the contest: dictwrap! (applause)

Bo

Jul 18 '05 #29

P: n/a
Bo Peng wrote:
Exec is slow since compiling the string and calls to globals() use a
lot of time. The last one is most elegant but __getattr__ and
__setattr__ are costly. The 'evil hack' solution is good since
accessing x and y takes no additional time.

Previous comparison was not completely fair since I could pre-compile
fun2 and I used indirect __setattr__. Here is the new one:
>>> # solution two: use exec ... def makeFunction(funcStr, name):
... code = compile(funcStr, name, 'exec')
... def f(d):
... exec code in d
... return f
... >>> def fun2(d):

... myfun = makeFunction('z = x + y', 'myfun')
... for i in xrange(0,N):
... myfun(d)
... del d['__builtins__']


You are still including the compile overhead in fun2. If you want to see how fast the compiled code
is you should take the definition of myfun out of fun2:

myfun = makeFunction('z = x + y', 'myfun')
def fun2(d):
for i in xrange(0,N):
myfun(d)
del d['__builtins__']

Kent
Jul 18 '05 #30

P: n/a
Kent Johnson wrote:
Bo Peng wrote:
Exec is slow since compiling the string and calls to globals() use a
lot of time. The last one is most elegant but __getattr__ and
__setattr__ are costly. The 'evil hack' solution is good since
accessing x and y takes no additional time.


Previous comparison was not completely fair since I could pre-compile
fun2 and I used indirect __setattr__. Here is the new one:
>>> # solution two: use exec

... def makeFunction(funcStr, name):
... code = compile(funcStr, name, 'exec')
... def f(d):
... exec code in d
... return f
...
>>> def fun2(d):

... myfun = makeFunction('z = x + y', 'myfun')
... for i in xrange(0,N):
... myfun(d)
... del d['__builtins__']

You are still including the compile overhead in fun2. If you want to see
how fast the compiled code is you should take the definition of myfun
out of fun2:

myfun = makeFunction('z = x + y', 'myfun')
def fun2(d):
for i in xrange(0,N):
myfun(d)
del d['__builtins__']

Kent


I assumed that most of the time will be spent on N times execution of
myfunc.
myfun = makeFunction('z = x + y', 'myfun')
def fun2(d): .... # myfun = makeFunction('z = x + y', 'myfun')
.... for i in xrange(0,N):
.... myfun(d)
.... del d['__builtins__'] def fun21(d): .... myfun = makeFunction('z = x + y', 'myfun')
.... for i in xrange(0,N):
.... myfun(d)
.... del d['__builtins__']
profile.run('fun2(a)') 200003 function calls in 1.930 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 1.930 1.930 <string>:1(?)
100000 0.460 0.000 0.460 0.000 myfun:1(?)
1 0.000 0.000 1.930 1.930 profile:0(fun2(a))
0 0.000 0.000 profile:0(profiler)
100000 0.980 0.000 1.440 0.000 python-162616Io.py:4(f)
1 0.490 0.490 1.930 1.930 python-16261Ud0.py:1(fun2)
profile.run('fun21(a)')

200004 function calls in 1.920 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 1.920 1.920 <string>:1(?)
100000 0.390 0.000 0.390 0.000 myfun:1(?)
1 0.000 0.000 1.920 1.920 profile:0(fun21(a))
0 0.000 0.000 profile:0(profiler)
1 0.000 0.000 0.000 0.000
python-162616Io.py:2(makeFunction)
100000 0.930 0.000 1.320 0.000 python-162616Io.py:4(f)
1 0.600 0.600 1.920 1.920 python-16261huu.py:1(fun21)
Jul 18 '05 #31

P: n/a
Bo Peng wrote:
Kent Johnson wrote:
You are still including the compile overhead in fun2. If you want to
see how fast the compiled code is you should take the definition of
myfun out of fun2:


I assumed that most of the time will be spent on N times execution of
myfunc.


Doh! Right.

Kent
Jul 18 '05 #32

This discussion thread is closed

Replies have been disabled for this discussion.