Class Variable Access and Assignment

[Graham]

This has to do with class variables and instances variables.

Given the following:

<code>

class _class:
var = 0
#rest of the class

instance_b = _class()

_class.var=5

print instance_b.var # -> 5
print _class.var # -> 5

</code>

Initially this seems to make sense, note the difference between to last
two lines, one is refering to the class variable 'var' via the class
while the other refers to it via an instance.

However if one attempts the following:

<code>

instance_b.var = 1000 # -> _class.var = 5
_class.var = 9999 # -> _class.var = 9999

</code>

An obvious error occurs. When attempting to assign the class variable
via the instance it instead creates a new entry in that instance's
__dict__ and gives it the value. While this is allowed because of
pythons ability to dynamically add attributes to a instance however it
seems incorrect to have different behavior for different operations.

There are two possible fixes, either by prohibiting instance variables
with the same name as class variables, which would allow any reference
to an instance of the class assign/read the value of the variable. Or
to only allow class variables to be accessed via the class name itself.

Many thanks to elpargo and coke. elpargo assisted in fleshing out the
best way to present this.

perhaps this was intended, i was just wondering if anyone else had
noticed it, and if so what form would you consider to be 'proper'
either referring to class variables via the class itself or via
instances of that class. Any response would be greatly appreciated.
Graham

[Mike Meyer]

bo**@oz.net (Bengt Richter) writes:

On 04 Nov 2005 17:53:34 -0800, Paul Rubin <http://ph****@NOSPAM.i nvalid> wrote:

bo**@oz.net (Bengt Richter) writes:

Hm, "the" fix? Why wouldn't e.g. treating augassign as shorthand for
a source transformation (i.e., asstgt <op>= expr becomes by simple
text substitution asstgt = asstgt <op> expr) be as good a fix? Then
we could discuss what

Consider "a[f()] += 3". You don't want to eval f() twice.

Well, if you accepted macro semantics IWT you _would_ want to ;-)

Another one of those throw-away lines.

I'd say that was true only if the macro was poorly written. Unless a
macros is intended as a tool to repeatedly evaluate an argument, it
should only evaluate it at most once.

Of course, if you're using some rock-stupid textual macro system, you
really don't have much choice in the matter.

<mike
--
Mike Meyer <mw*@mired.or g> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Mike Meyer]

Paul Rubin <http://ph****@NOSPAM.i nvalid> writes:

Steven D'Aprano <st***@REMOVETH IScyber.com.au> writes:

It also allows you to do something like this:
class ExpertGame(Game ):
current_level = 100

and then use ExpertGame anywhere you would have used Game with no problems.

Well, let's say you set, hmm, current_score = 100 instead of current_level.
Scores in some games can get pretty large as you get to the higher
levels, enough so that you start needing long ints, which maybe are
used elsewhere in your game too, like for the cryptographic signatures
that authenticate the pieces of treasure in the dungeon. Next you get
some performance gain by using gmpy to handle the long int arithmetic,
and guess what? Eventually a version of your game comes along that
enables the postulated (but not yet implemented) mutable int feature
of gmpy for yet more performance gains. So now, current_score += 3000
increments the class variable instead of creating an instance
variable, and whoever maintains your code by then now has a very weird
bug to track down and fix.

I'd say that's a wart with +=, not with Python's inheritance
mechanisms. += is neither + nor =, but takes on different aspects of
each depending on what it's operating on. While it's true that python
is dynamic enough that the you can create classes that make this true
for any operator, += is the only one that acts like that on the
builtin types.

<mike
--
Mike Meyer <mw*@mired.or g> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Steve Holden]

Paul Rubin wrote:

Steven D'Aprano <st***@REMOVETH IScyber.com.au> writes:

It also allows you to do something like this:

class ExpertGame(Game ):
current_level = 100

and then use ExpertGame anywhere you would have used Game with no problems.

Well, let's say you set, hmm, current_score = 100 instead of current_level.
Scores in some games can get pretty large as you get to the higher
levels, enough so that you start needing long ints, which maybe are
used elsewhere in your game too, like for the cryptographic signatures
that authenticate the pieces of treasure in the dungeon. Next you get
some performance gain by using gmpy to handle the long int arithmetic,
and guess what? Eventually a version of your game comes along that
enables the postulated (but not yet implemented) mutable int feature
of gmpy for yet more performance gains. So now, current_score += 3000
increments the class variable instead of creating an instance
variable, and whoever maintains your code by then now has a very weird
bug to track down and fix.

Anyway, I'm reacting pretty badly to the construction you're
describing. I haven't gotten around to looking at the asyncore code
but will try to do so.

I wouldn't bother. From memory it's just using a class variable as an
initialiser.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

[Steven D'Aprano]

On Fri, 04 Nov 2005 20:41:31 -0800, Paul Rubin wrote:

Steven D'Aprano <st***@REMOVETH IScyber.com.au> writes:

It also allows you to do something like this:

class ExpertGame(Game ):
current_level = 100

and then use ExpertGame anywhere you would have used Game with no problems.

Well, let's say you set, hmm, current_score = 100 instead of current_level.

Right. Because inheriting scores makes so much sense. But that's okay.
Assume I think of some use for inheriting scores and implement this.
Scores in some games can get pretty large as you get to the higher
levels, enough so that you start needing long ints, which maybe are
used elsewhere in your game too, like for the cryptographic signatures
that authenticate the pieces of treasure in the dungeon.
Python already converts ints to long automatically, but please, do go on.
Next you get
some performance gain by using gmpy to handle the long int arithmetic,
Then whatever happens next will be my own stupid fault for prematurely
optimising code.

and guess what? Eventually a version of your game comes along that
enables the postulated (but not yet implemented) mutable int feature
of gmpy for yet more performance gains.
This would be using Python3 or Python4?
So now, current_score += 3000 increments the class variable instead of
creating an instance variable, and whoever maintains your code by then
now has a very weird bug to track down and fix.

That's a lot of words to say "If ints become mutable when you expect
them to be immutable, things will go badly for you." Well duh.

What exactly is your point? That bugs can happen if the behaviour of your
underlying libraries changes? If list.sort suddenly starts randomizing the
list instead of sorting it, I'll have bugs too. Should I avoid using sort
just in case?
--
Steven.

[Paul Rubin]

Steven D'Aprano <st***@REMOVETH IScyber.com.au> writes:

Next you get some performance gain by using gmpy to handle the long int arithmetic,
Then whatever happens next will be my own stupid fault for prematurely optimising code.

Huh? There's nothing premature about using gmpy if you need better long int performance.
It was written for a reason, after all.

and guess what? Eventually a version of your game comes along
that enables the postulated (but not yet implemented) mutable int
feature of gmpy for yet more performance gains.

This would be using Python3 or Python4?

No, it would be a gmpy feature, not a Python feature. So it could be
used with any version of Python.
What exactly is your point? That bugs can happen if the behaviour of your
underlying libraries changes?

That your initialization scheme is brittle--the idea of data
abstraction is being able to change object behaviors -without- making
surprising bugs like that one. You don't even need the contrived gmpy
example. You might replace the level number with, say, a list of
levels that have been visited.

I don't think the culprit is the mutable/immutable distinction +=
uses, though that is certainly somewhat odd. I think Antoon is on the
right track: namespaces in Python live in sort of a ghetto unbecoming
of how the Zen list describes them as a "honking great idea". These
things we call variables are boxed objects where the namespace is the
box. So having x+=y resolve x to a slot in a namespace before
incrementing that same slot by y, maybe better uses the notion of
namespaces than what happens now. I'm too sleepy to see for sure
whether it gets rid of the mutable/immutable weirdness.

[Bengt Richter]

On Fri, 04 Nov 2005 21:14:17 -0500, Mike Meyer <mw*@mired.or g> wrote:

bo**@oz.net (Bengt Richter) writes:

On Thu, 03 Nov 2005 13:37:08 -0500, Mike Meyer <mw*@mired.or g> wrote:
[...]

I think it even less sane, if the same occurce of b.a refers to two
different objects, like in b.a += 2

That's a wart in +=, nothing less. The fix to that is to remove +=
from the language, but it's a bit late for that.
Hm, "the" fix? Why wouldn't e.g. treating augassign as shorthand for a source transformation
(i.e., asstgt <op>= expr becomes by simple text substitution asstgt = asstgt <op> expr)
be as good a fix? Then we could discuss what

b.a = b.a + 2

should mean ;-)

The problem with += is how it behaves, not how you treat it. But you
can't treat it as a simple text substitution, because that would imply
that asstgt gets evaluated twice, which doesn't happen.

I meant that it would _make_ that happen, and no one would wonder ;-)

BTW, if b.a is evaluated once each for __get__ and __set__, does that not
count as getting evaluated twice?

class shared(object): ... def __init__(self, v=0): self.v=v
... def __get__(self, *any): print '__get__'; return self.v
... def __set__(self, _, v): print '__set__'; self.v = v
... class B(object): ... a = shared(1)
... b=B()
b.a __get__
1 b.a += 2 __get__
__set__ B.a __get__
3

Same number of get/sets:
b.a = b.a + 10 __get__
__set__ b.a __get__
13

I posted the disassembly in another part of the thread, but I'll repeat:
def foo(): ... a.b += 2
... a.b = a.b + 2
... import dis
dis.dis(foo) 2 0 LOAD_GLOBAL 0 (a)
3 DUP_TOP
4 LOAD_ATTR 1 (b)
7 LOAD_CONST 1 (2)
10 INPLACE_ADD
11 ROT_TWO
12 STORE_ATTR 1 (b)

3 15 LOAD_GLOBAL 0 (a)
18 LOAD_ATTR 1 (b)
21 LOAD_CONST 1 (2)
24 BINARY_ADD
25 LOAD_GLOBAL 0 (a)
28 STORE_ATTR 1 (b)
31 LOAD_CONST 0 (None)
34 RETURN_VALUE

It looks like the thing that's done only once for += is the LOAD_GLOBAL (a)
but DUP_TOP provides the two copies of the reference which are
used either way with LOAD_ATTR followed by STORE_ATTR, which UIAM
lead to the loading of the (descriptor above) attribute twice -- once each
for the __GET__ and __SET__ calls respectively logged either way above.

OTOH, we could discuss how you can confuse yourself with the results of b.a += 2
after defining a class variable "a" as an instance of a class defining __iadd__ ;-)
You may confuse yourself that way, I don't have any problems with it
per se.

I should have said "one can confuse oneself," sorry ;-)
Anyway, I wondered about the semantics of defining __iadd__, since it seems to work just
like __add__ except for allowing you to know what source got you there. So whatever you
return (unless you otherwise intercept instance attribute binding) will get bound to the
instance, even though you internally mutated the target and return None by default (which
gives me the idea of returning NotImplemented, but (see below) even that gets bound :-(

BTW, semantically does/should not __iadd__ really implement a _statement_ and therefore
have no business returning any expression value to bind anywhere?
class DoIadd(object): ... def __init__(self, v=0, **kw):
... self.v = v
... self.kw = kw
... def __iadd__(self, other):
... print '__iadd__(%r, %r) => '%(self, other),
... self.v += other
... retv = self.kw.get('re tv', self.v)
... print repr(retv)
... return retv
... class B(object): ... a = DoIadd(1)
... b=B()
b.a <__main__.DoIad d object at 0x02EF374C> b.a.v 1

The normal(?) mutating way: b.a += 2 __iadd__(<__mai n__.DoIadd object at 0x02EF374C>, 2) => 3 vars(b) {'a': 3} B.a <__main__.DoIad d object at 0x02EF374C> B.a.v 3

Now fake attempt to mutate self without returning anything (=> None) B.a = DoIadd(1, retv=None) # naive default
b.a 3
Oops, remove instance attr del b.a
b.a <__main__.DoIad d object at 0x02EF3D6C> b.a.v 1
Ok, now try it b.a +=2 __iadd__(<__mai n__.DoIadd object at 0x02EF3D6C>, 2) => None vars(b) {'a': None}
Returned value None still got bound to instance B.a.v 3
Mutation did happen as planned

Now let's try NotImplemented as a return B.a = DoIadd(1, retv=NotImpleme nted) # mutate but probably do __add__ too
del b.a
b.a <__main__.DoIad d object at 0x02EF374C> b.a.v 1 b.a +=2 __iadd__(<__mai n__.DoIadd object at 0x02EF374C>, 2) => NotImplemented
__iadd__(<__mai n__.DoIadd object at 0x02EF374C>, 2) => NotImplemented vars(b) {'a': NotImplemented} B.a.v

5

No problem with that? ;-)

I'd say it looks like someone got tired of implementing __iadd__ since
it's too easy to work around the problem. If _returning_ NotImplemented
could have the meaning that return value processing (binding) should not
be effected, then mutation could happen without a second evaluation of
b.a as a target. ISTM a return value for __iadd__ is kind of strange in any case,
since it's a statement implementation, not an expression term implementation.

Or point out that you can define descriptors (or use property to make it easy)
to control what happens, pretty much in as much detail as you can describe requirements ;-)

I've already pointed that out.

Sorry, missed it. Big thread ;-)

Regards,
Bengt Richter

[Bengt Richter]

On Sat, 05 Nov 2005 14:37:19 +1100, Steven D'Aprano <st***@REMOVETH IScyber.com.au> wrote:

On Sat, 05 Nov 2005 00:25:34 +0000, Bengt Richter wrote:

On Fri, 04 Nov 2005 02:59:35 +1100, Steven D'Aprano <st***@REMOVETH IScyber.com.au> wrote:

On Thu, 03 Nov 2005 14:13:13 +0000, Antoon Pardon wrote:

Fine, we have the code:

b.a += 2

We found the class variable, because there is no instance variable,
then why is the class variable not incremented by two now? Because the class variable doesn't define a self-mutating __iadd__
(which is because it's an immutable int, of course). If you want
b.__dict__['a'] += 2 or b.__class__.__d ict__['a'] += 2 you can
always write it that way ;-)

(Of course, you can use a descriptor to define pretty much whatever semantics
you want, when it comes to attributes).

Because b.a += 2 expands to b.a = b.a + 2. Why would you want b.a =

No, it doesn't expand like that. (Although, BTW, a custom import could
make it so by transforming the AST before compiling it ;-)

Note BINARY_ADD is not INPLACE_ADD:

Think about *what* b.a += 2 does, not *how* it does it. Perhaps for some

what it does, or what in the abstract it was intended to do? (which we need
BDFL channeling to know for sure ;-)

It looks like it means, "add two to <whatever b.a is>". I think Antoon
is unhappy that <whatever b.a is> is not determined once for the one b.a
expression in the statement. I sympathize, though it's a matter of defining
what b.a += 2 is really intended to mean.
The parses are certainly distinguishable :

import compiler
compiler.parse( 'b.a +=2','exec').no de Stmt([AugAssign(Getat tr(Name('b'), 'a'), '+=', Const(2))]) compiler.parse( 'b.a = b.a + 2','exec').node

Stmt([Assign([AssAttr(Name('b '), 'a', 'OP_ASSIGN')], Add((Getattr(Na me('b'), 'a'), Const(2))))])

Which I think leads to the different (BINARY_ADD vs INPLACE_ADD) code, which probably really
ought to have a conditional STORE_ATTR for the result of INPLACE_ADD, so that if __iadd__
was defined, it would be assumed that the object took care of everything (normally mutating itself)
and no STORE_ATTR should be done. But that's not the way it works now. (See also my reply to Mike).

Perhaps all types that want to be usable with inplace ops ought to inherit from some base providing
that, and there should never be a return value. This would be tricky for immutables though, since
re-binding is necessary, and the __iadd__ method would have to be passed the necessary binding context
and methods. Probably too much of a rewrite to be practical.
other data type it would make a difference whether the mechanism was
BINARY_ADD (__add__) or INPLACE_ADD (__iadd__), but in this case it does
not. Both of them do the same thing. Unfortunately you seem to be right in this case.
Actually, no "perhaps" about it -- we've already discussed the case of
lists. Well, custom objects have to be considered too. And where attribute access
is involved, descriptors.

Sometimes implementation makes a difference. I assume BINARY_ADD and
INPLACE_ADD work significantly differently for lists, because their
results are significantly (but subtly) different:

py> L = [1,2,3]; id(L)
-151501076
py> L += [4,5]; id(L)
-151501076
py> L = L + []; id(L)
-151501428
Yes.
But all of this is irrelevant to the discussion about binding b.a
differently on the left and right sides of the equals sign. We have
discussed that the behaviour is different with mutable objects, because
they are mutable -- if I recall correctly, I was the first one in this
thread to bring up the different behaviour when you append to a list
rather than reassign, that is, modify the class attribute in place.

I'll admit that my choice of terminology was not the best, but it wasn't
misleading. b.a += 2 can not modify ints in place, and so the
effect of b.a += 2 is the same as b.a = b.a + 2, regardless of what
byte-codes are used, or even what C code eventually implements that
add-and-store. It is so currently, but that doesn't mean that it couldn't be otherwise.
I think there is some sense to the idea that b.a should be re-bound in
the same namespace where it was found with the single apparent evaluation
of "b.a" in "b.a += 2" (which incidentally is Antoon's point, I think).
This is just for augassign, of course.

OTOH, this would be find-and-rebind logic for attributes when augassigned,
and that would enable some tricky name-collision bugs for typos, and code
that used instance.attr += incr depending on current behavior would break.

In the case of lists, setting Class.a = [] and then calling instance.a +=
[1] would not exhibit the behaviour Antoon does not like, because the
addition is done in place. But calling instance.a = instance.a + [1]
would.

My question still stands: why would you want instance.a = <something>
to operate as instance.__clas s__.a = <something>?

Because in the case of instance.a += <increment>, "instance.a "
is a short spelling for "instance.__cla ss__.a" (in the limited case we are discussing),
and that spelling specifies _both_ source and target in a _single_ expression,
unlike instance.a = instance.a + <incr> where two expressions are used, which
one should expect to have their meaning accoring to the dynamic moment and
context of their evaluation.

If 'a' in vars(instance) then instance.a has the meaning instance.__dict __['a']
for both source and target of +=.

I think you can argue for the status quo or find-and-rebind, but since there
are adequate workarounds to let you do what you want, I don't expect a change.
I do think that returning NotImplemented from __iadd__ to indicate no binding
of return value desired (as opposed to __iadd__ itself not implemented, which
is detected before the call) might make things more controllable for custom objects.

Sorry about cramming too much into sentences ;-/

Regards,
Bengt Richter

[Steven D'Aprano]

On Sat, 05 Nov 2005 21:26:22 +0000, Bengt Richter wrote:

BTW, semantically does/should not __iadd__ really implement a _statement_ and therefore
have no business returning any expression value to bind anywhere?

We get to practicality versus purity here.

Consider x += y for some object type x. If x is a mutable object, then
__iadd__ could be a statement, because it can/should/must modify x in
place. That is the pure solution.

But do you want x += y to work for immutable objects as well? Then
__iadd__ cannot be a statement, because x can't be modified in place.
Our pure "add in place" solution fails in practice, unless we needlessly
restrict what can use it, or have the same syntactical expression (x +=
y) bind to two different methods (__iadd__ statement, and __riadd__
function, r for return). Either pure solution is yucky. (That's a
technical term for "it sucks".) So for practical reasons, __iadd__ can't
be a statement, it needs to return an object which gets bound to x.

Fortunately, that behaviour works for mutables as well, because __iadd__
simply returns self, which gets re-bound to x.

While I am enjoying the hoops people are jumping through to modify the
language so that b.a += 2 assigns b.a in the same scope as it was
accessed, I'm still rather perplexed as to why you would want that
behaviour. It seems to me like spending many hours building a wonderfully
polished, ornate, exquisite device for building hiking boots for mountain
goats.
--
Steven.

[Steven D'Aprano]

On Fri, 04 Nov 2005 22:19:39 -0800, Paul Rubin wrote:

Steven D'Aprano <st***@REMOVETH IScyber.com.au> writes:

> Next you get some performance gain by using gmpy to handle the long int arithmetic,
Then whatever happens next will be my own stupid fault for prematurely optimising code.

Huh? There's nothing premature about using gmpy if you need better long int performance.
It was written for a reason, after all.

Sure, but I would be willing to bet that incrementing a counter isn't it.

What exactly is your point? That bugs can happen if the behaviour of your
underlying libraries changes?

That your initialization scheme is brittle--the idea of data
abstraction is being able to change object behaviors -without- making
surprising bugs like that one. You don't even need the contrived gmpy
example. You might replace the level number with, say, a list of
levels that have been visited.

Do you expect level += 1 to still work when you change level to a list of
levels?

The problem with data abstraction is if you take it seriously, it means
"You should be able to do anything with anything". If I change
object.__dict__ to None, attribute lookup should work, yes? No? Then
Python isn't sufficiently abstract.

As soon as you accept that there are some things you can't do with some
data, you have to stop abstracting. *Prematurely* locking yourself into
one *specific* data structure is bad: as a basic principle, data
abstraction is very valuable -- but in practice there comes a time where
you have to say "Look, just choose a damn design and live with it." If you
choose sensibly, then it won't matter if your counter is an int or a long
or a float or a rational -- but you can't sensibly expect to change your
counter to a binary tree without a major redesign of your code.

I've watched developers with an obsession with data abstraction in
practice. I've watched one comp sci graduate, the ink on his diploma not
even dry yet, spend an hour mapping out state diagrams for a factorial
function.

Hello McFly? The customer is paying for this you know. Get a move on. I've
written five different implementations of factorial in ten minutes, and
while none of them worked with symbolic algebra I didn't need symbolic
algebra support, so I lost nothing by not supporting it.

So I hope you'll understand why I get a bad taste in my mouth when people
start talking about data abstraction.
I don't think the culprit is the mutable/immutable distinction +=
uses, though that is certainly somewhat odd. I think Antoon is on the
right track: namespaces in Python live in sort of a ghetto unbecoming
of how the Zen list describes them as a "honking great idea". These
things we call variables are boxed objects where the namespace is the
box. So having x+=y resolve x to a slot in a namespace before
incrementing that same slot by y, maybe better uses the notion of
namespaces than what happens now.
Perhaps it does, but it breaks inheritance, which is more important than
purity of namespace resolution. Practicality beats purity.

I'm too sleepy to see for sure
whether it gets rid of the mutable/immutable weirdness.

What weirdness? What would be weird is if mutable and immutable objects
worked the same as each other. They behave differently because they are
different. If you fail to see that, you are guilty of excessive data
abstraction.
--
Steven.

[Paul Rubin]

Steven D'Aprano <st***@REMOVETH IScyber.com.au> writes:

But do you want x += y to work for immutable objects as well? Then
__iadd__ cannot be a statement, because x can't be modified in place.
It never occurred to me that immutable objects could implement __iadd__.
If they can, I'm puzzled as to why.
While I am enjoying the hoops people are jumping through to modify the
language so that b.a += 2 assigns b.a in the same scope as it was
accessed, I'm still rather perplexed as to why you would want that
behaviour.

Weren't you the one saying += acting differently for mutables and
immutables was a wart? If it's such a wart, why are do you find it so
important to be able to rely on the more bizarre consequences of the
wartiness? Warts should be (if not fixed) avoided, not relied on.