Static variable vs Class variable

[minkoo.seo]

Hi.

I've got a question on the differences and how to define static and
class variables. AFAIK, class methods are the ones which receives the
class itself as an argument, while static methods are the one which
runs statically with the defining class.

Hence, my understanding is that static variables must be bound to the
class defining the variables and shared by children of parent class
where the variable is defined. But, please have a look at this code in
which a guy told me that the variable a is static:

>>class Foo:

a = 1
@classmethod
def increment(cls):
cls.a += 1
print cls.a

Here, I am defining variable a which, I believe is class variable,
i.e., variable that is not bound to Foo itself. Rather, a is bound to
the class which is accessing the variable. The code that corroborates
this idea is as follows:

>>class Child1(Foo):

pass

>>Child1.increment()

4

>>class Child2(Foo):

pass

>>Child2.increment()

4

This means that Child1 and Child2 does not share variable a which
means that variable a is class variable rather than static variable.

Could you please comment on this? Is a static or class variable?
What's the most recent way of defining 'class' and 'static' variables?

Thanks.
- Minkoo

[Laszlo Nagy]

mi********@gmail.com wrote:

Hi.

I've got a question on the differences and how to define static and
class variables. AFAIK, class methods are the ones which receives the
class itself as an argument, while static methods are the one which
runs statically with the defining class.

Hence, my understanding is that static variables must be bound to the
class defining the variables and shared by children of parent class
where the variable is defined. But, please have a look at this code in
which a guy told me that the variable a is static:

>>>class Foo:

a = 1
@classmethod
def increment(cls):
cls.a += 1
print cls.a

In your increment() method, you do this:

cls.a += 1

It does the following thing:

#1. read cls.a
#2. add one
#3. assign this value to cls.a

In point #3, you really bind a name to a value. As you probably know, in
Python, there are names and objects. The initial value of the name 'a'
is 1. It is an immutable object. The "+=" operator usually increments a
value of an object. However, because the 'int' type is immutable, the +=
operator will rather rebind this variable to a newly created value. I
believe this is what is happening here. Your question "is variable a
static or class variable?" has no real answer. After running the
increment() method on a descendant class, e.g. Child1 will rebind the
name Child1.a, creating a new name in the namespace of the class. So the
variable Foo.a is still there, but you are accessing Child1.a instead.

If you want to HANDLE a as a static variable, you can handle it with a
static method. That won't bind a new name in the descendant class.
(However, you can still rebind it, e.g. "Child.a=42")
Now here is a good question: how do you handle a variable as static,
from a class (not static) method? Here is an example:

>>class Foo(object):

.... a = 1
.... @classmethod
.... def increment(cls):
.... Foo.a += 1
.... print cls.a
....

>>class Child1(Foo):

.... pass
....

>>Child1.increment()

2

>>Child1.increment()

3

>>Foo.a

3

>>Child1.a = 10
Child1.increment()

10

>>Child1.increment()

10

>>Child1.increment()

10

>>Foo.a

6

>>>

However, the question is: why would you do this? :-)

BTW you should use new style classes whenever it is possible. Old style
classes will have gone...

Hope this helps,

Laszlo

[Marc 'BlackJack' Rintsch]

On Tue, 09 Oct 2007 09:16:12 -0700, mi********@gmail.com wrote:

I've got a question on the differences and how to define static and
class variables.

First you have to define what you mean by "static".

AFAIK, class methods are the ones which receives the
class itself as an argument, while static methods are the one which
runs statically with the defining class.

`classmethod`\s receive the class as first arguments, `staticmethod`\s are
just functions bound to the class object.

Hence, my understanding is that static variables must be bound to the
class defining the variables and shared by children of parent class
where the variable is defined. But, please have a look at this code in
which a guy told me that the variable a is static:

Ask the guy what he means by "static".

>>>class Foo:

a = 1
@classmethod
def increment(cls):
cls.a += 1
print cls.a

Here, I am defining variable a which, I believe is class variable,
i.e., variable that is not bound to Foo itself.

No you define a class attribute that *is* bound to the class `Foo`.

Rather, a is bound to the class which is accessing the variable. The code
that corroborates this idea is as follows:

>>>class Child1(Foo):

pass

>>>Child1.increment()

4

Four!? Hard to believe.

>>>class Child2(Foo):

pass

>>>Child2.increment()

4

This means that Child1 and Child2 does not share variable a which means
that variable a is class variable rather than static variable.

Could you please comment on this? Is a static or class variable? What's
the most recent way of defining 'class' and 'static' variables?

There is no such thing as a "static" variable. Think of attributes that
are bound to objects. All dynamically.

What happens is: you bind a 1 to the attribute `Foo.a` in the `Foo` class
definition.

When you call `Child1.increment()` the class method will be called with
`Child1` as first argument. Now ``cls.a += 1`` is executed which is
somewhat like a short form of ``cls.a = cls.a + 1``. So this is reading
the attribute `a` from `Child1` and then bind the result to `Child1`.
`Child1` doesn't have an attribute `a`, so it is looked up in the parent
class. But the result is then bound to `Child1`. So you are reading from
`Foo` and writing to `Child1`. That's it.

Ciao,
Marc 'BlackJack' Rintsch

[Laszlo Nagy]

Your question "is variable a
static or class variable?" has no real answer. After running the
increment() method on a descendant class, e.g. Child1 will rebind the
name Child1.a, creating a new name in the namespace of the class. So the
variable Foo.a is still there, but you are accessing Child1.a instead.

Please notice, that theoretically there is no way to "change the value"
of Foo.a in any way, because this is a NAME that references to an
IMMUTABLE OBJECT. It means that you can only rebind the variable. You
cannot change its value, because when we are talking about its value, we
mean the state of the referenced object. The object referenced by the
name is an integer instance, namely it is "one". The object "one" always
remains "one", this cannot be changed. You can add one to one, and you
will get two, but that is another object.

(I'm sorry, probably you already knew this.)

Laszlo

[Matimus]

On Oct 9, 9:16 am, "[e-mail]" <[e-mail]>
wrote:

Hi.

I've got a question on the differences and how to define static and
class variables. AFAIK, class methods are the ones which receives the
class itself as an argument, while static methods are the one which
runs statically with the defining class.

Hence, my understanding is that static variables must be bound to the
class defining the variables and shared by children of parent class
where the variable is defined. But, please have a look at this code in
which a guy told me that the variable a is static:

>class Foo:

a = 1
@classmethod
def increment(cls):
cls.a += 1
print cls.a

Here, I am defining variable a which, I believe is class variable,
i.e., variable that is not bound to Foo itself. Rather, a is bound to
the class which is accessing the variable. The code that corroborates
this idea is as follows:

>class Child1(Foo):

pass

>Child1.increment()

4

>class Child2(Foo):

pass

>Child2.increment()

4

This means that Child1 and Child2 does not share variable a which
means that variable a is class variable rather than static variable.

Could you please comment on this? Is a static or class variable?
What's the most recent way of defining 'class' and 'static' variables?

In Python `a' is considered a class variable. By modifying `a' using a
class method though, you are essentially re-binding `a' to the class
that it is called from. So, it will be shared by multiple instances of
the same class, but not derivatives. The re-binding only occurs when
`increment' is called. This makes for some very confusing behavior.
I'm not aware of a way, in python, to explicitly specify a variable as
static. For the behavior I think you are looking for you just need to
modify it carefully. This can be done with a static, class or instance
method. Though, using a class method is kind of silly. I'm sure some
would argue that using an instance method is also silly.

code:

class C(object):
a = 0
def inc(self):
C.a += 1 # directly modify the variable on the base class that it is
attached to
return C.a

# or you could use a static method, which is fine since the instance
isn't being used. This
# will also allow the method to be called without instantiating the
class.

class C(object):
a = 0
@staticmethod
def inc():
C.a += 1
return C.a

Matt

[Diez B. Roggisch]

In point #3, you really bind a name to a value. As you probably know, in

Python, there are names and objects. The initial value of the name 'a'
is 1. It is an immutable object. The "+=" operator usually increments a
value of an object. However, because the 'int' type is immutable, the +=
operator will rather rebind this variable to a newly created value. I
believe this is what is happening here.

Your believes aside, this is simply wrong. The statement

a += x

always leads to a rebinding of a to the result of the operation +. I
presume you got confused by the somewhat arbitrary difference between

__add__

and
__iadd__

that somehow suggest there is an in-place-modification going on in case
of mutables.

but as the following snippet shows - that's not the case:
class Foo(object):
def __add__(self, o):
return "__add__"

def __iadd__(self, o):
return "__iadd__"
a = Foo()
a += 1
print a

a = Foo()
b = Foo()
c = a + b
print c

So you see, the first += overrides a with the returned value of __iadd__.

The reason for the difference though is most probably what you yourself
expected: thus it's possible to alter a mutable in place.

Diez

[Steven D'Aprano]

On Tue, 09 Oct 2007 19:23:37 +0200, Diez B. Roggisch wrote:

Your believes aside, this is simply wrong. The statement

a += x

always leads to a rebinding of a to the result of the operation +.

Not true.

>>L = []
id(L)

3083496716L

>>L += [1]
id(L)

3083496716L

It's the same L, not rebound at all.

I presume you got confused by the somewhat arbitrary difference between

__add__

and

__iadd__

that somehow suggest there is an in-place-modification going on in case
of mutables.

The __iFOO__ methods are supposed to do in-place modification if
possible, but it is up to the class writer to make them do so. In the
case of your example, you specifically created an __iadd__ method that
didn't even attempt in-place modification. What did you expect it to do?

but as the following snippet shows - that's not the case:
class Foo(object):
def __add__(self, o):
return "__add__"

def __iadd__(self, o):
return "__iadd__"
a = Foo()
a += 1
print a

a = Foo()
b = Foo()
c = a + b
print c

So you see, the first += overrides a with the returned value of
__iadd__.

That's because you told it to do that. If you told it to do something
more sensible, it would have done so. Lists know how to do in-place
modification:

>>id(L) # from above

3083496716L

>>L *= 5
id(L)

3083496716L

>>L = L*5
id(L)

3083496972L
--
Steven.

[Laszlo Nagy]

Steven D'Aprano wrote:

On Tue, 09 Oct 2007 19:23:37 +0200, Diez B. Roggisch wrote:

>Your believes aside, this is simply wrong. The statement

a += x

always leads to a rebinding of a to the result of the operation +.

Not true.

Hmm. Or you can write __iadd__ to rebind the name to the same object
again? I think usually there is not much difference between "rebinding
to the same object" and "not rebinding". But as we have seen, sometimes
there is a difference. I think that Diez made his statement because it
is possible to rebind from __iadd__ to a different object. So it is more
safe to think that "inplace operations" will always rebind, than to
think they will never rebind.

I'm not sure about the language level. It is told that "+=" is an
inplace operator, but how "in place" is defined? If we say "an inplace
operator will change the state of an object, and will never create a new
object" then "+=" is NOT an inplace operator. Well, at least not when
you watch it from the language level. But what about the implementation
level? Here is an example:

a = 1
a+= 1 # The compiler will probably optimize this and the Python bytecode
interpreter will not rebind 'a' here, just increment the integer in memory.

This topic starts to be very interesting. Language lawyers and
developers, please help us! How "in place operator" is defined in
Python? What makes it an "in place operator"? Is it the syntax, or what?

Laszlo

[Marc 'BlackJack' Rintsch]

On Tue, 09 Oct 2007 18:08:34 +0000, Steven D'Aprano wrote:

>>>L = []
id(L)

3083496716L

>>>L += [1]
id(L)

3083496716L

It's the same L, not rebound at all.

It *is* rebound. To the same object, but it *is* assigned to `L` and not
just mutated in place.

In [107]: class A:
.....: a = list()
.....:

In [108]: class B(A):
.....: pass
.....:

In [109]: B.a += [42]

In [110]: A.a
Out[110]: [42]

In [111]: B.a
Out[111]: [42]

If it was just mutation then `B.a` would have triggered an `AttributeError`.

Ciao,
Marc 'BlackJack' Rintsch

[Diez B. Roggisch]

Steven D'Aprano schrieb:

On Tue, 09 Oct 2007 19:23:37 +0200, Diez B. Roggisch wrote:

>Your believes aside, this is simply wrong. The statement

a += x

always leads to a rebinding of a to the result of the operation +.

Not true.

Yes, it is.

>>>L = []
id(L)

3083496716L

>>>L += [1]
id(L)

3083496716L

It's the same L, not rebound at all.

Just because the __iadd__-operation in the list implemented chose to
returen a reference to itself - certainly a sensible choice, I never
doubted that - doesn't change the fact that python rebinds the name on
the left with whatever __iadd__ returned.

Diez

[Steven D'Aprano]

On Tue, 09 Oct 2007 19:46:35 +0000, Marc 'BlackJack' Rintsch wrote:

On Tue, 09 Oct 2007 18:08:34 +0000, Steven D'Aprano wrote:

>>>>L = []
id(L)

3083496716L

>>>>L += [1]
id(L)

3083496716L

It's the same L, not rebound at all.

It *is* rebound. To the same object, but it *is* assigned to `L` and
not just mutated in place.

Picky picky.

Yes, technically there is an assignment of L to itself. I was sloppy to
say "not rebound at all", because when you write an augmented assignment
method you have to return self if you want to implement in-place
mutation. But I hardly call "rebinding to itself" any sort of rebinding
worth the name :)

Diez is still wrong though, even though I overstated my case. See my
reply to his post.

[snip code]

If it was just mutation then `B.a` would have triggered an
`AttributeError`.

Why? Don't Python classes have inheritance?

(That's a rhetorical question. Yes they do, and no B.a would not raise
AttributeError because it would inherit from A.)
--
Steven.

[Steven D'Aprano]

On Tue, 09 Oct 2007 22:27:47 +0200, Diez B. Roggisch wrote:

Steven D'Aprano schrieb:

>On Tue, 09 Oct 2007 19:23:37 +0200, Diez B. Roggisch wrote:

>>Your believes aside, this is simply wrong. The statement

a += x

always leads to a rebinding of a to the result of the operation +.

Not true.

Yes, it is.

I'm afraid not.

As I admitted in my reply to Marc, I overstated my case by saying that L
isn't rebound at all. Of course it is rebound, but to itself.

However, it is not true that += "always leads to a rebinding of a to the
result of the operation +". The + operator for lists creates a new list.
+= for lists does an in-place modification:

>>L = []
M = L
L += [1]
M

[1]

Compare with:

>>L = []
M = L
L = L + [1]
M

[]

You said:

"I presume you got confused by the somewhat arbitrary difference between
__add__ and __iadd__ that somehow suggest there is an in-place-
modification going on in case of mutables but as the following snippet
shows - that's not the case: ..."

That's an explicit denial that in-place modification takes place, and
that's *way* off the mark. I was concentrating so hard on showing in-
place modification that I glossed over the "return self" part.

--
Steven.

[Steven D'Aprano]

On Tue, 09 Oct 2007 21:25:38 +0200, Laszlo Nagy wrote:

a = 1
a+= 1 # The compiler will probably optimize this and the Python bytecode
interpreter will not rebind 'a' here, just increment the integer in
memory.

No. This is Python, not C. You can't increment integers in memory.
Integers are immutable objects, not raw bytes:

>>x = 1
id(x)

150830896

>>x += 1
id(x)

150830884


--
Steven.

[Marc 'BlackJack' Rintsch]

On Tue, 09 Oct 2007 22:43:16 +0000, Steven D'Aprano wrote:

On Tue, 09 Oct 2007 19:46:35 +0000, Marc 'BlackJack' Rintsch wrote:

>On Tue, 09 Oct 2007 18:08:34 +0000, Steven D'Aprano wrote:

>>>>>L = []
>id(L)
3083496716L
>L += [1]
>id(L)
3083496716L

It's the same L, not rebound at all.

It *is* rebound. To the same object, but it *is* assigned to `L` and
not just mutated in place.

Picky picky.

Yes, technically there is an assignment of L to itself. I was sloppy to
say "not rebound at all", because when you write an augmented assignment
method you have to return self if you want to implement in-place
mutation. But I hardly call "rebinding to itself" any sort of rebinding
worth the name :)

Maybe picky but that detail was the source of the OPs confusion because it
introduced a new attribute on the subclass.

Ciao,
Marc 'BlackJack' Rintsch

[Bruno Desthuilliers]

mi********@gmail.com a écrit :

Hi.

I've got a question on the differences and how to define static and
class variables.

What's a "static" variable ? A variable that doesn't move ?-)

Hence, my understanding is that static variables must be bound to the
class defining the variables

You mean "class attributes". Thinking in terms of another language won't
help understanding Python's objec model.

and shared by children of parent class
where the variable is defined. But, please have a look at this code in
which a guy told me that the variable a is static:

Then don't listen what this guy says.

>>>>class Foo:

a = 1
@classmethod
def increment(cls):
cls.a += 1
print cls.a

Here, I am defining variable a which, I believe is class variable,

a class attribute

i.e., variable that is not bound to Foo itself.

Yes it is. It's an attribute of the class object Foo.

Rather, a is bound to
the class which is accessing the variable.

False.

>>dir(Foo)

['__doc__', '__module__', 'a', 'increment']

>>Foo.__dict__

{'a': 1, '__module__': '__main__', '__doc__': None, 'increment':
<classmethod object at 0x402d141c>}

The code that corroborates
this idea is as follows:

>>>>class Child1(Foo):

pass

>>>>Child1.increment()

4

???

>

>>>>class Child2(Foo):

pass

>>>>Child2.increment()

4

I don't have the same result here. But anyway: the increment method is
rebinding 'a', so the first time it's called on a child class, it
creates an attribute 'a' for this class. Just change your code this way
and you'll see what happens:

def increment(cls):
print dir(cls)
cls.a += 1
print dir(cls)
FWIW, if you don't get this, then you may encounter another common gotcha:

class Truc:
a = 1
def gotcha(self):
self.a += 1
print self.a

print Truc.a
t = Truc()
print t.a
t.gotcha()
print Truc.a
t2 = Truc()
print t2.a
t2.gotcha()
print Truc.a
t.gotcha()
print t.a
print t2.a
print Truc.a

Enjoy !-)

Now back to your snippet : if you don't want this behaviour, wrap 'a'
into a mutable container:

class Foo:
a = [1]

@classmethod
def increment(cls):
cls.a[0] += 1
print cls.a[0]
class Child(Foo):
pass

And if you want to hide that, use properties (warning: you need nex
style classes here):

class Foo(object):
_a = [1]

@classmethod
def increment(cls):
cls._a[0] += 1
print cls._a[0]

@apply
def a():
def fget(self):
return type(self)._a[0]
def fset(self, val):
type(self)._a[0] = val
return property(**locals())

class Child(Foo):
pass

This means that Child1 and Child2 does not share variable a which
means that variable a is class variable rather than static variable.

Nope. This only means that you don't understand the (somewhat peculiar)
semantics of bindings, immutable objects and attribute lookup rules in
Python.

Could you please comment on this? Is a static or class variable?

It's a class attribute.

What's the most recent way of defining 'class' and 'static' variables?

No 'static variable' in Python. And what you call a 'class variable' is
nothing else than an ordinary of the class object

[Bruno Desthuilliers]

Marc 'BlackJack' Rintsch a écrit :

On Tue, 09 Oct 2007 18:08:34 +0000, Steven D'Aprano wrote:

>>>>>L = []
>id(L)

3083496716L

>>>>>L += [1]
>id(L)

3083496716L

It's the same L, not rebound at all.

It *is* rebound. To the same object, but it *is* assigned to `L` and not
just mutated in place.

In [107]: class A:
.....: a = list()
.....:

In [108]: class B(A):
.....: pass
.....:

In [109]: B.a += [42]

In [110]: A.a
Out[110]: [42]

In [111]: B.a
Out[111]: [42]

If it was just mutation then `B.a` would have triggered an `AttributeError`.

Nope.

>>class A:

.... l = []
....

>>class B(A): pass

....

>>A.l

[]

>>A.l += [1]
A.l

[1]

>>B.l

[1]

>>>
B.l is A.l

True
And it is *not* rebound:

>>B.l += [2]
A.l

[1, 2]

>>B.l

[1, 2]

>>A.l is B.l

True

>>>

[Bruno Desthuilliers]

Bruno Desthuilliers a écrit :
(snip)

And it is *not* rebound:

Doh. Stupid me. Of course it is - but to a ref to the same object...

>>class A:

.... l = []
....

>>class B(A): pass

....

>>B.__dict__

{'__module__': '__main__', '__doc__': None}

>>B.l

[]

>>B.l += [1]
B.__dict__

{'__module__': '__main__', '__doc__': None, 'l': [1]}

>>>

Thanks Diez for pointing this out. And as far as I'm concerned: time to
bed :(

[Diez B. Roggisch]

>Yes, it is.

>
I'm afraid not.

As I admitted in my reply to Marc, I overstated my case by saying that L
isn't rebound at all. Of course it is rebound, but to itself.

However, it is not true that += "always leads to a rebinding of a to the
result of the operation +". The + operator for lists creates a new list.
+= for lists does an in-place modification:

It still is true.

a += b

rebinds a. Period. Which is the _essential_ thing in my post, because
this rebinding semantics are what confused the OP.

>>>L = []
M = L
L += [1]
M

[1]

Compare with:

>>>L = []
M = L
L = L + [1]
M

[]

You said:

"I presume you got confused by the somewhat arbitrary difference between
__add__ and __iadd__ that somehow suggest there is an in-place-
modification going on in case of mutables but as the following snippet
shows - that's not the case: ..."

Admittedly, I miss _one_ word here: necessarily before the "an".

That's an explicit denial that in-place modification takes place, and
that's *way* off the mark. I was concentrating so hard on showing in-
place modification that I glossed over the "return self" part.

And I was concetrating so hard on the rebinding-part, I glossed over the
in-place-modification part.

Diez

[Diez B. Roggisch]

Diez B. Roggisch schrieb:

>>Yes, it is.

I'm afraid not.

As I admitted in my reply to Marc, I overstated my case by saying that
L isn't rebound at all. Of course it is rebound, but to itself.

However, it is not true that += "always leads to a rebinding of a to
the result of the operation +". The + operator for lists creates a new
list. += for lists does an in-place modification:

It still is true.

a += b

rebinds a. Period. Which is the _essential_ thing in my post, because
this rebinding semantics are what confused the OP.

Which I just came around to show in a somewhat enhanced example I could
have used the first time:

class Foo(object):

a = []

@classmethod
def increment(cls):
print cls
cls.a += [1]

class Bar(Foo):
pass

Foo.increment()
Bar.increment()

print Foo.a
print Bar.a
Bar.a = []
print Foo.a
print Bar.a

192:~/projects/SoundCloud/ViewAnimationTest deets$ python /tmp/test.py
<class '__main__.Foo'>
<class '__main__.Bar'>
[1, 1]
[1, 1]
[1, 1]
[]
Which makes the rebinding-part of __iadd__ pretty much an issue, don't
you think?
Diez

[Laszlo Nagy]

Steven D'Aprano írta:

On Tue, 09 Oct 2007 21:25:38 +0200, Laszlo Nagy wrote:

>a = 1
a+= 1 # The compiler will probably optimize this and the Python bytecode
interpreter will not rebind 'a' here, just increment the integer in
memory.

No. This is Python, not C. You can't increment integers in memory.

What? Please read my post. I was talking about __implementation level__,
clearly. :-)

[paul.melis]

On Oct 10, 8:23 am, "Diez B. Roggisch" <[e-mail]:

However, it is not true that += "always leads to a rebinding of a to the
result of the operation +". The + operator for lists creates a new list.
+= for lists does an in-place modification:

It still is true.

a += b

rebinds a. Period. Which is the _essential_ thing in my post, because
this rebinding semantics are what confused the OP.

Doesn't this depend on wether "a" supports __iadd__ or not? Section
3.4.7 of the docs say

"""
If a specific method is not defined, the augmented operation falls
back to the normal methods. For instance, to evaluate the expression x
+=y, where x is an instance of a class that has an __iadd__() method,
x.__iadd__(y) is called. If x is an instance of a class that does not
define a __iadd__() method, x.__add__(y) and y.__radd__(x) are
considered, as with the evaluation of x+y.
"""

So if a.__iadd__ exists, a += b is executed as a.__iadd__(b), in which
case there's no reason to rebind a.

However, this confuses the heck out of me:

>>class A:

.... l = []
....

>>class B(A): pass

....

>>B.__dict__

{'__module__': '__main__', '__doc__': None}

>>B.l

[]

>>B.l.append('1')
B.l

['1']

>>B.__dict__

{'__module__': '__main__', '__doc__': None}

>>B.l.__iadd__('2')

['1', '2']

>>B.l

['1', '2']

>>B.__dict__

{'__module__': '__main__', '__doc__': None}

>>B.l += '3'
B.__dict__

{'__module__': '__main__', '__doc__': None, 'l': ['1', '2', '3']}

Why is B.l set for the += case only? B.l.__iadd__ obviously exists.

Paul

[Marc 'BlackJack' Rintsch]

On Wed, 17 Oct 2007 00:33:59 -0700, paul.melis wrote:

On Oct 10, 8:23 am, "Diez B. Roggisch" <[e-mail]:

However, it is not true that += "always leads to a rebinding of a to the
result of the operation +". The + operator for lists creates a new list.
+= for lists does an in-place modification:

It still is true.

a += b

rebinds a. Period. Which is the _essential_ thing in my post, because
this rebinding semantics are what confused the OP.

Doesn't this depend on wether "a" supports __iadd__ or not?

No. As shown several times in this thread already.

Section 3.4.7 of the docs say

"""
If a specific method is not defined, the augmented operation falls
back to the normal methods. For instance, to evaluate the expression x
+=y, where x is an instance of a class that has an __iadd__() method,
x.__iadd__(y) is called. If x is an instance of a class that does not
define a __iadd__() method, x.__add__(y) and y.__radd__(x) are
considered, as with the evaluation of x+y.
"""

So if a.__iadd__ exists, a += b is executed as a.__iadd__(b), in which
case there's no reason to rebind a.

`__iadd__` *may* doing the addition in place and return `self` but it is
also allowed to return a different object. So there is always a rebinding.

However, this confuses the heck out of me:

>>>class A:

... l = []
...

>>>class B(A): pass

...

>>>B.__dict__

{'__module__': '__main__', '__doc__': None}

>>>B.l

[]

>>>B.l.append('1')
B.l

['1']

>>>B.__dict__

{'__module__': '__main__', '__doc__': None}

>>>B.l.__iadd__('2')

['1', '2']

Here you see that the method actually returns an object!

>>>B.l

['1', '2']

>>>B.__dict__

{'__module__': '__main__', '__doc__': None}

>>>B.l += '3'
B.__dict__

{'__module__': '__main__', '__doc__': None, 'l': ['1', '2', '3']}

Why is B.l set for the += case only? B.l.__iadd__ obviously exists.

Because there is always a rebinding involved.

Ciao,
Marc 'BlackJack' Rintsch

[Duncan Booth]

pa********@gmail.com wrote:

Curious, do you have the relevant section in the docs that describes
this behaviour?

Yes, but mostly by implication. In section 3.4.7 of the docs, the sentence
before the one you quoted says:

These methods should attempt to do the operation in-place (modifying
self) and return the result (which could be, but does not have to be,
self).

The 'does not have to be self' tells you that the result of __iadd__ is
used, i.e there is still an assignment going on.

Just read all of that paragraph carefully. It says that if there is no
__iadd__ method it considers calling __add__/__radd__. Nowhere does it say
that it handles the result of calling the methods differently.

[paul.melis]

On Oct 17, 11:08 am, Duncan Booth <[e-mail]>
wrote:

[e-mail] wrote:

Curious, do you have the relevant section in the docs that describes
this behaviour?

Yes, but mostly by implication. In section 3.4.7 of the docs, the sentence
before the one you quoted says:

These methods should attempt to do the operation in-place (modifying
self) and return the result (which could be, but does not have to be,
self).

The 'does not have to be self' tells you that the result of __iadd__ is
used, i.e there is still an assignment going on.

Just read all of that paragraph carefully. It says that if there is no
__iadd__ method it considers calling __add__/__radd__. Nowhere does it say
that it handles the result of calling the methods differently.

Right, the paragraph is actually pretty clear after a second reading.
I find it surprising nonetheless, as it's easy to forget to return a
result when you're implementing a method that does an in-place
operation, like __iadd__:

>>class C:

.... def __init__(self, v):
.... self.v = v
.... def __iadd__(self, other):
.... self.v += other
....

>>c=C(1)
c.v

1

>>c += 3
c
c is None

True
Paul

[Hrvoje Niksic]

pa********@gmail.com writes:

Right, the paragraph is actually pretty clear after a second
reading. I find it surprising nonetheless, as it's easy to forget
to return a result when you're implementing a method that does an
in-place operation, like __iadd__:

I've recently been bitten by that, and I don't understand the
reasoning behind __iadd__'s design. I mean, what is the point of an
*in-place* add operation (and others) if it doesn't always work
in-place?

[Duncan Booth]

Hrvoje Niksic <hn*****@xemacs.orgwrote:

pa********@gmail.com writes:

>Right, the paragraph is actually pretty clear after a second
reading. I find it surprising nonetheless, as it's easy to forget
to return a result when you're implementing a method that does an
in-place operation, like __iadd__:

I've recently been bitten by that, and I don't understand the
reasoning behind __iadd__'s design. I mean, what is the point of an
*in-place* add operation (and others) if it doesn't always work
in-place?

A very common use case is using it to increment a number:

x += 1

If += always had to work inplace then this would throw an exception: an
inplace addition would be meaningless for Python numbers.

[Hrvoje Niksic]

Duncan Booth <du**********@invalid.invalidwrites:

Hrvoje Niksic <hn*****@xemacs.orgwrote:

>I've recently been bitten by [rebinding the var to what __iadd__
returns], and I don't understand the reasoning behind __iadd__'s
design. I mean, what is the point of an *in-place* add operation
(and others) if it doesn't always work in-place?

A very common use case is using it to increment a number:

I'm aware of that; but remember that there's still __add__. It would
be sufficient for numbers not to implement __iadd__. And, in fact,
they already don't:

>>1 .__add__(1)

2

>>1 .__iadd__(1)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'int' object has no attribute '__iadd__'

The current implementation of += uses __add__ for addition and
__iadd__ for addition that may or may not be in-place. I'd like to
know the rationale for that design.

[Marc 'BlackJack' Rintsch]

On Wed, 17 Oct 2007 13:41:06 +0200, Hrvoje Niksic wrote:

Duncan Booth <du**********@invalid.invalidwrites:

>Hrvoje Niksic <hn*****@xemacs.orgwrote:

>>I've recently been bitten by [rebinding the var to what __iadd__
returns], and I don't understand the reasoning behind __iadd__'s
design. I mean, what is the point of an *in-place* add operation
(and others) if it doesn't always work in-place?

A very common use case is using it to increment a number:

I'm aware of that; but remember that there's still __add__. It would
be sufficient for numbers not to implement __iadd__. And, in fact,
they already don't:

>>>1 .__add__(1)

2

>>>1 .__iadd__(1)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'int' object has no attribute '__iadd__'

The current implementation of += uses __add__ for addition and
__iadd__ for addition that may or may not be in-place. I'd like to
know the rationale for that design.

Simply not to introduce special cases I guess. If you write ``x.a += b``
then `x.a` will be rebound whether an `a.__iadd__()` exists or not.
Otherwise one would get interesting subtle differences with properties for
example. If `x.a` is a property that checks if the value satisfies some
constraints ``x.a += b`` would trigger the set method only if there is no
`__iadd__()` involved if there's no rebinding.

Ciao,
Marc 'BlackJack' Rintsch

[Steven D'Aprano]

On Wed, 17 Oct 2007 13:41:06 +0200, Hrvoje Niksic wrote:

The current implementation of += uses __add__ for addition and __iadd__
for addition that may or may not be in-place. I'd like to know the
rationale for that design.

Everything you need is in the PEP:

http://www.python.org/dev/peps/pep-0203/

--
Steven.

[Duncan Booth]

Hrvoje Niksic <hn*****@xemacs.orgwrote:

>
The current implementation of += uses __add__ for addition and
__iadd__ for addition that may or may not be in-place. I'd like to
know the rationale for that design.

Apart from the obvious short answer of being consistent (so you don't
have to guess whether or not a+=b is going to do an assignment), I think
the decision was partly to keep the implementation clean.

Right now an inplace operation follows one of three patterns:

load a value LOAD_FAST or LOAD_GLOBAL
do the inplace operation (INPLACE_ADD etc)
store the result (STORE_FAST or STORE_GLOBAL)

or:

compute an expression
DUP_TOP
LOAD_ATTR
do the inplace operation
ROT_TWO
STORE_ATTR

or:

compute two expressions
DUP_TOPX 2
BINARY_SUBSCR
do the inplace operation
ROT_THREE
STORE_SUBSCR

I'm not sure I've got all the patterns here, but we have at least three
different cases with 0, 1, or 2 objects on the stack and at least 4
different store opcodes.

If the inplace operation wasn't always going to store the result, then
either we would need 4 times as many INPLACE_XXX opcodes, or it would
have to do something messy to pop the right number of items off the
stack and skip the following 1 or 2 instructions.

I see from PEP203 that ROT_FOUR was added as part of the implementation,
so I guess I must have missed at least one other bytecode pattern for
augmented assignment (which turns out to be slice assignment with a
stride).

[Duncan Booth]

Marc 'BlackJack' Rintsch <bj****@gmx.netwrote:

Simply not to introduce special cases I guess. If you write ``x.a +=
b`` then `x.a` will be rebound whether an `a.__iadd__()` exists or
not. Otherwise one would get interesting subtle differences with
properties for example. If `x.a` is a property that checks if the
value satisfies some constraints ``x.a += b`` would trigger the set
method only if there is no `__iadd__()` involved if there's no
rebinding.

Unfortunately that doesn't work very well. If the validation rejects the
new value the original is still modified:

>>class C(object):

def setx(self, value):
if len(value)>2:
raise ValueError
self._x = value
def getx(self):
return self._x
x = property(getx, setx)

>>o = C()
o.x = []
o.x += ['a']
o.x += ['b']
o.x += ['c']

Traceback (most recent call last):
File "<pyshell#27>", line 1, in <module>
o.x += ['c']
File "<pyshell#22>", line 4, in setx
raise ValueError
ValueError

>>o.x

['a', 'b', 'c']

>>>

[Paul Melis]

On Oct 17, 2:39 pm, Duncan Booth <[e-mail]:

Marc 'BlackJack' Rintsch <[e-mail]:

Simply not to introduce special cases I guess. If you write ``x.a +=
b`` then `x.a` will be rebound whether an `a.__iadd__()` exists or
not. Otherwise one would get interesting subtle differences with
properties for example. If `x.a` is a property that checks if the
value satisfies some constraints ``x.a += b`` would trigger the set
method only if there is no `__iadd__()` involved if there's no
rebinding.

Unfortunately that doesn't work very well. If the validation rejects the
new value the original is still modified:

>class C(object):

def setx(self, value):
if len(value)>2:
raise ValueError
self._x = value
def getx(self):
return self._x
x = property(getx, setx)

>o = C()
o.x = []
o.x += ['a']
o.x += ['b']
o.x += ['c']

Traceback (most recent call last):
File "<pyshell#27>", line 1, in <module>
o.x += ['c']
File "<pyshell#22>", line 4, in setx
raise ValueError
ValueError

>o.x

['a', 'b', 'c']

Now that's really interesting. I added a print "before" and print
"after" statement just before and after the self._x = value and these
*do not get called* after the exception is raised when the third
element is added.

[Marc 'BlackJack' Rintsch]

On Wed, 17 Oct 2007 05:57:50 -0700, Paul Melis wrote:

On Oct 17, 2:39 pm, Duncan Booth <[e-mail]:

>>class C(object):

def setx(self, value):
if len(value)>2:
raise ValueError
self._x = value
def getx(self):
return self._x
x = property(getx, setx)

>>o = C()
o.x = []
o.x += ['a']
o.x += ['b']
o.x += ['c']

Traceback (most recent call last):
File "<pyshell#27>", line 1, in <module>
o.x += ['c']
File "<pyshell#22>", line 4, in setx
raise ValueError
ValueError

>>o.x

['a', 'b', 'c']

Now that's really interesting. I added a print "before" and print
"after" statement just before and after the self._x = value and these
*do not get called* after the exception is raised when the third
element is added.

Well, of course not. Did you really expect that!? Why?

Ciao,
Marc 'BlackJack' Rintsch

[Paul Melis]

On Oct 17, 3:20 pm, Marc 'BlackJack' Rintsch <[e-mail]:

On Wed, 17 Oct 2007 05:57:50 -0700, Paul Melis wrote:

On Oct 17, 2:39 pm, Duncan Booth <[e-mail]:

>class C(object):

def setx(self, value):
if len(value)>2:
raise ValueError
self._x = value
def getx(self):
return self._x
x = property(getx, setx)

>o = C()
o.x = []
o.x += ['a']
o.x += ['b']
o.x += ['c']

Traceback (most recent call last):
File "<pyshell#27>", line 1, in <module>
o.x += ['c']
File "<pyshell#22>", line 4, in setx
raise ValueError
ValueError

>o.x
['a', 'b', 'c']

Now that's really interesting. I added a print "before" and print
"after" statement just before and after the self._x = value and these
*do not get called* after the exception is raised when the third
element is added.

Well, of course not. Did you really expect that!? Why?

Because o.x *is* updated, i.e. the net result of self._x = value is
executed.

Paul

[Paul Melis]

On Oct 17, 3:41 pm, Paul Melis <[e-mail]:

On Oct 17, 3:20 pm, Marc 'BlackJack' Rintsch <[e-mail]:

On Wed, 17 Oct 2007 05:57:50 -0700, Paul Melis wrote:

On Oct 17, 2:39 pm, Duncan Booth <[e-mail]:
>>class C(object):

> def setx(self, value):
> if len(value)>2:
> raise ValueError
> self._x = value
> def getx(self):
> return self._x
> x = property(getx, setx)

>>o = C()
>>o.x = []
>>o.x += ['a']
>>o.x += ['b']
>>o.x += ['c']

>Traceback (most recent call last):
> File "<pyshell#27>", line 1, in <module>
> o.x += ['c']
> File "<pyshell#22>", line 4, in setx
> raise ValueError
>ValueError

>>o.x
>['a', 'b', 'c']

Now that's really interesting. I added a print "before" and print
"after" statement just before and after the self._x = value and these
*do not get called* after the exception is raised when the third
element is added.

Well, of course not. Did you really expect that!? Why?

Because o.x *is* updated, i.e. the net result of self._x = value is
executed.

Argh, the getter is of course used to append the third element, after
which the rebinding triggers the exception. Got it now...

Paul

[Hrvoje Niksic]

Marc 'BlackJack' Rintsch <bj****@gmx.netwrites:

Simply not to introduce special cases I guess. If you write ``x.a
+= b`` then `x.a` will be rebound whether an `a.__iadd__()` exists
or not. Otherwise one would get interesting subtle differences with
properties for example. If `x.a` is a property that checks if the
value satisfies some constraints ``x.a += b`` would trigger the set
method only if there is no `__iadd__()` involved if there's no
rebinding.

Note that such constraint is easily invalidated simply with:

foo = x.a
foo += b

[Hrvoje Niksic]

Duncan Booth <du**********@invalid.invalidwrites:

Hrvoje Niksic <hn*****@xemacs.orgwrote:

>The current implementation of += uses __add__ for addition and
__iadd__ for addition that may or may not be in-place. I'd like to
know the rationale for that design.

Apart from the obvious short answer of being consistent (so you don't
have to guess whether or not a+=b is going to do an assignment), I think
the decision was partly to keep the implementation clean.

The implementation, by necessity, supports a lot of historical cruft,
so keeping it clean was not a priority. (This is not a criticism, I
like how they kept backward compatibility.)

Right now an inplace operation follows one of three patterns:

[...]

Thanks for pointing it out; I didn't realize just how much work went
into the new assignment opcodes. Given the complexity, it's a wonder
they're there at all!

[bambam]

"Steven D'Aprano" <st***@REMOVE-THIS-cybersource.com.auwrote in message
news:13*************@corp.supernews.com...

On Wed, 17 Oct 2007 13:41:06 +0200, Hrvoje Niksic wrote:

>The current implementation of += uses __add__ for addition and __iadd__
for addition that may or may not be in-place. I'd like to know the
rationale for that design.

Everything you need is in the PEP:

http://www.python.org/dev/peps/pep-0203/

--
Steven.

Which illustrates that the proposal, while simplified for implementation,
is not exactly what was desired*

""" is both more readable and less error prone, because it is
instantly obvious to the reader that it is <xthat is being
changed, and not <xthat is being replaced
"""

As we see from this thread, it is not instantly obvious to the reader;
the meaning of "changed, not replaced" is ambiguous.

[david]

* That is, unless ambiguity was the desideratum