obj.__dict__ expected behavior or bug?

[Ed Young]

Here is an example of the behavior:
------- code start -----------------------------------
#!/usr/bin/python
#bugtest - test of class attribute initiation
class Config:
a = 1
b = 2
c = 3
d = None
e = None
h = {'d' : 22, 'e' : 33}

def __init__(self, factor):
for attr in self.h.keys():
self.__dict__[attr] = self.h[attr] * factor

def moda(self):
self.a *= 5
c = Config(2)
print c.a, c.b, c.c, c.d, c.e
for attr in c.__dict__:
print 'c.%s = %s' % (attr, c.__dict__[attr])
print

c.moda()
print c.a, c.b, c.c, c.d, c.e
for attr in c.__dict__:
print 'c.%s = %s' % (attr, c.__dict__[attr])
print
------- code ends -----------------------------------
------- output starts -------------------------------
$ bugtest
1 2 3 44 66
c.e = 66
c.d = 44

5 2 3 44 66
c.a = 5
c.e = 66
c.d = 44
------- output ends ---------------------------------
What happened to c.a, c.b, and c.c when iterating thru
c.__dict__ ?

It appears that __dict__ members are not instantiated
until they are changed.

This precludes using __dict__ as the dictionary in
a formatted print statement. e.g.

print "c.a=%(a)s, c.b=%(b)s, c.c=%(c)s" % c.__dict__

Is this a bug or expected behavior?

[Erik Max Francis]

Ed Young wrote:

What happened to c.a, c.b, and c.c when iterating thru
c.__dict__ ?

It appears that __dict__ members are not instantiated
until they are changed.

This precludes using __dict__ as the dictionary in
a formatted print statement. e.g.

print "c.a=%(a)s, c.b=%(b)s, c.c=%(c)s" % c.__dict__

Is this a bug or expected behavior?

Expected behavior. What you're missing is the general way that Python
does attribute lookup. When c is an instance and you say c.x, Python
looks in c's __dict__ for an 'x' entry, then it looks in c's class's
__dict__ for an 'x' entry, then it looks (in a well-defined way) through
c's class's base classes, if any, for an 'x' entry in their __dict__
members.

When you defined

class C:
a = ...
b = ...

and so on, these are all _class_ attributes. When you instantiate a C
and then wrote self.a = ... in its methods, you instantiated _instance_
attributes on that instance. Class attributes are analogous to static
members/fields in other languages:

class C: # class with two class attributes .... a = 1
.... b = 2
.... c = C()
d = C()
c.a 1 d.a 1 c.a = 10 # change an instance attribute
c.a 10 d.a 1 C.b = 20 # change a class attribute
c.b 20 d.b 20 C.__dict__ {'a': 1, '__module__': '__main__', 'b': 20, '__doc__': None} c.__dict__ {'a': 10} d.__dict__

{}

--
Erik Max Francis && ma*@alcyone.com && http://www.alcyone.com/max/
__ San Jose, CA, USA && 37 20 N 121 53 W && &tSftDotIotE
/ \ Nobody's on nobody's side
\__/ Florence, _Chess_

[Raymond Hettinger]

"Ed Young

Here is an example of the behavior:
------- code start -----------------------------------
#!/usr/bin/python
#bugtest - test of class attribute initiation
class Config:
a = 1
b = 2
c = 3
d = None
e = None
h = {'d' : 22, 'e' : 33}

def __init__(self, factor):
for attr in self.h.keys():
self.__dict__[attr] = self.h[attr] * factor

def moda(self):
self.a *= 5
c = Config(2)
print c.a, c.b, c.c, c.d, c.e
for attr in c.__dict__:
print 'c.%s = %s' % (attr, c.__dict__[attr])
print

c.moda()
print c.a, c.b, c.c, c.d, c.e
for attr in c.__dict__:
print 'c.%s = %s' % (attr, c.__dict__[attr])
print
------- code ends -----------------------------------
------- output starts -------------------------------
$ bugtest
1 2 3 44 66
c.e = 66
c.d = 44

5 2 3 44 66
c.a = 5
c.e = 66
c.d = 44
------- output ends ---------------------------------
What happened to c.a, c.b, and c.c when iterating thru
c.__dict__ ?
They are up in C.__dict__
This precludes using __dict__ as the dictionary in
a formatted print statement. e.g.

print "c.a=%(a)s, c.b=%(b)s, c.c=%(c)s" % c.__dict__

Not really. Use a wrapper to forward dict lookup requests
to getattr() which knows how/where to search for attributes:

class AttrDict:
def __init__(self, obj):
self.obj = obj
def __getitem__(self, key):
return getattr(self.obj, key)

print "c.a=%(a)s, c.b=%(b)s, c.c=%(c)s" % AttrDict(c)

Is this a bug or expected behavior?

Expected.

Raymond Hettinger

[Sean Ross]

Hi.
It's expected behaviour.

Let's go through your code with a few additional print statements to see if
we can demonstrate what's happening;

class Config:
a = 1
b = 2
c = 3
d = None
e = None
h = {'d' : 22, 'e' : 33}

def __init__(self, factor):
for attr in self.h.keys():
self.__dict__[attr] = self.h[attr] * factor

def moda(self):
self.a *= 5
c = Config(2)

# Here's what to pay attention to ...........
print "c.__dict__: ", c.__dict__ #
this is the instance dictionary
print "c.__class__.__dict__: ", c.__class__.__dict__ # this is the class
dictionary
print c.a, c.b, c.c, c.d, c.e
for attr in c.__dict__:
print 'c.%s = %s' % (attr, c.__dict__[attr])
print

c.moda()

print "c.moda() --------------"
print "c.__dict__: ", c.__dict__
print "c.__class__.__dict__: ", c.__class__.__dict__

print c.a, c.b, c.c, c.d, c.e
for attr in c.__dict__:
print 'c.%s = %s' % (attr, c.__dict__[attr])
print

Now, here's the output with #annotations:
c.__dict__: {'e': 66, 'd': 44}
c.__class__.__dict__: {'a': 1, 'moda': <function moda at 0x015209B0>,
'__module__': '__main__', 'b': 2, 'e': None, 'd': None, 'h': {'e': 33, 'd':
22}, 'c': 3, '__init__': <function __init__ at 0x01520B30>, '__doc__': None}
1 2 3 44 66
c.e = 66
c.d = 44

# Okay. We can see that the values for 'a', 'b', 'c' were all found
# in the class dictionary of instance c, while 'd', and 'e' were
# found in the instance dictionary of c. More on this later....
# Now we're about to call moda() ....
c.moda() --------------

# What's changed?
c.__dict__: {'a': 5, 'e': 66, 'd': 44}
c.__class__.__dict__: {'a': 1, 'moda': <function moda at 0x015209B0>,
'__module__': '__main__', 'b': 2, 'e': None, 'd': None, 'h': {'e': 33, 'd':
22}, 'c': 3, '__init__': <function __init__ at 0x01520B30>, '__doc__': None}
5 2 3 44 66
c.a = 5
c.e = 66
c.d = 44

# This time only 'b' and 'c''s values were pulled from instance c's class'
dictionary.
# What about 'a'? 'a' was pulled from c's instance dictionary. Nothing's
changed
# for 'd' and 'e'.
Okay then. What's going on?

class Config:
a = 1
b = 2
c = 3
d = None
e = None
h = {'d' : 22, 'e' : 33}

The code above adds class variables a - h to the class Config. So, if you
have an instance c of Config,
variables a-h are stored in c's class dictionary (c.__class__.__dict__) and
NOT c's instance dictionary
(c.__dict__). Moving on...

def __init__(self, factor):
for attr in self.h.keys():
self.__dict__[attr] = self.h[attr] * factor

Inside the constructor, you call self.h.keys(). To find self.h, Python looks
first in self.__dict__. But 'h' isn't there.
Next it looks in self.__class__.__dict__. That's were 'h' is! Now this:

self.__dict__[attr] = self.h[attr] * factor

Here, you're assigning NEW attributes 'd' and 'e' to self's __dict__. What
you are not doing is assigning new values to class variables 'd' and 'e' in
self.__class__.__dict__ .
def moda(self):
self.a *= 5

Something similar is happening in here. This one is a bit more complicated.

self.a *= 5

is the same as

self.a = self.a * 5

What does this really mean? Well,

self.a = ....

is equivalent to

self.__dict__['a'] = ....

But

self.a = self.a ....

is not necessarily equivalent to

self.__dict__['a'] = self.__dict__['a']
because the self.a on the right hand side of the assignment has to be looked
up by Python. And, as we showed earlier,
look up starts with self.__dict__. But 'a' is not yet a key in that
dictionary, so we move up to self.__class__.__dict__.
That's where 'a' is! It's value is '1', so we get

self.__dict__['a'] = 1*5
^
self.__class__.__dict__['a']

We finish the evaluation, and assign 5 to self.__dict__['a'], creating a new
instance variable.
The class variable 'a' is unchanged. If you call c.moda() again later then,
that time, Python's lookup
would find 'a' in self.__dict__, and the expression self.a *= 5 would be
equivalent to

self.__dict__['a'] = 5*5
^
self.__dict__['a']

So, the thing is, yes the behaviour is expected, if you know what behaviour
to expect ...
Okay, then. Hopefully that was helpful.
Sean

[Ed Young]

Thank you all for the kind and detailed explanations.
I now have a thorough understanding of the mechanism
behind attribute lookup.