Bizarre method keyword-arg bug.

[Jasper]

I'm stumped. I'm calling a method that has keyword args, but not
setting them, and yet one of them starts off with data?!

The class definition begins like so:

class BattleIntentionAction( BattleAction ):
def __init__( self, factionName, location, tactic='hold',
targetFacName='', terrainArgs=[], garrisonIds=[] ):
self.terrainArgs = terrainArgs
print terrainArgs

The constructor is called somewhere else, like so:
act = BattleIntentionAction( facName, self.location )
During this object's construction, terrainArgs is set to a list with
values corresponding to a previously created BattleIntentionAction!
Even more bizarre, the terrainArgs param is a testing formality, and
doesn't actually get used anywhere in my code -- the corresponding
attribute is always modified after object creation. Furthermore, this
doesn't happen with the other keyword args...

Obviously, I'm glossing over a ton of code here, but I'm having a
tough time isolating this problem, as it seems to be very dependent on
events leading up to it. It feels like the sort of memory stomping
bug I remember seeing from days of yore when I hacked C++. :-(
I frankly don't understand how "terrainArgs" can have a value if
nothing is passed for it on the calling invocation, short of some
obscure compiler bug (this is Python 2.4.3). Am I being naive? Is
there some way I could be bringing this about myself?

I can easily work around this weirdness by having the caller set
terrainArgs explicitly, but I can't shake the sensation that this
"fix" just masks some deeper flaw in my code.
Arg!
-Jasper

[Simon Brunning]

2008/8/18 Jasper <ja****@peak.org>:

I'm stumped. I'm calling a method that has keyword args, but not
setting them, and yet one of them starts off with data?!

<http://www.python.org/doc/faq/general/#why-are-default-values-shared-between-objects>

--
Cheers,
Simon B.
si***@brunningonline.net
http://www.brunningonline.net/simon/blog/
GTalk: simon.brunning | MSN: small_values | Yahoo: smallvalues | Twitter: brunns

[Diez B. Roggisch]

Jasper schrieb:

I'm stumped. I'm calling a method that has keyword args, but not
setting them, and yet one of them starts off with data?!

The class definition begins like so:

class BattleIntentionAction( BattleAction ):
def __init__( self, factionName, location, tactic='hold',
targetFacName='', terrainArgs=[], garrisonIds=[] ):
self.terrainArgs = terrainArgs
print terrainArgs

The constructor is called somewhere else, like so:
act = BattleIntentionAction( facName, self.location )
During this object's construction, terrainArgs is set to a list with
values corresponding to a previously created BattleIntentionAction!
Even more bizarre, the terrainArgs param is a testing formality, and
doesn't actually get used anywhere in my code -- the corresponding
attribute is always modified after object creation. Furthermore, this
doesn't happen with the other keyword args...

Obviously, I'm glossing over a ton of code here, but I'm having a
tough time isolating this problem, as it seems to be very dependent on
events leading up to it. It feels like the sort of memory stomping
bug I remember seeing from days of yore when I hacked C++. :-(
I frankly don't understand how "terrainArgs" can have a value if
nothing is passed for it on the calling invocation, short of some
obscure compiler bug (this is Python 2.4.3). Am I being naive? Is
there some way I could be bringing this about myself?

I can easily work around this weirdness by having the caller set
terrainArgs explicitly, but I can't shake the sensation that this
"fix" just masks some deeper flaw in my code.

This is a FAQ:

http://effbot.org/pyfaq/why-are-defa...en-objects.htm

Diez

[Fredrik Lundh]

Jasper wrote:

I'm stumped. I'm calling a method that has keyword args, but not
setting them, and yet one of them starts off with data?!

The class definition begins like so:

class BattleIntentionAction( BattleAction ):
def __init__( self, factionName, location, tactic='hold',
targetFacName='', terrainArgs=[], garrisonIds=[] ):
self.terrainArgs = terrainArgs
print terrainArgs

The constructor is called somewhere else, like so:
act = BattleIntentionAction( facName, self.location )

During this object's construction, terrainArgs is set to a list with
values corresponding to a previously created BattleIntentionAction!

default argument values are evaluated when the function object is
created (by the "def" statement, that is), not when the resulting
function is called. if you mutate the default values, the mutations
will stick.

this is explained in the FAQ, the tutorial, and the reference manual,
and hopefully in your favourite python book as well; see e.g.

http://docs.python.org/tut/node6.htm...00000000000000
http://docs.python.org/ref/function.html

</F>

[Fredrik Lundh]

Fredrik Lundh wrote:

default argument values are evaluated when the function object is
created (by the "def" statement, that is), not when the resulting
function is called. if you mutate the default values, the mutations
will stick.

and yes, workarounds and further details are provided here:

http://effbot.org/zone/default-values.htm

</F>

[Jasper]

On Aug 18, 1:49 am, "Simon Brunning" <[e-mail]:

2008/8/18 Jasper <[e-mail]>:

I'm stumped. I'm calling a method that has keyword args, but not
setting them, and yet one of them starts off with data?!

<http://www.python.org/doc/faq/general/#why-are-default-values-shared-...>

--
Cheers,
Simon B.

Uggg! /That's/ an intuitive side-effect/wart. :-/

Thanks for sorting me out!

-Jasper

[Fredrik Lundh]

Jasper wrote:

Uggg! /That's/ an intuitive side-effect/wart. :-/

it's done that way on purpose, of course, because evaluating a full
closure for each default argument at every call would greatly hurt
performance (and lead to another set of surprises, of course).

please don't label things that you don't understand and haven't spent
any time reflecting over as bugs or warts; that's disrespectful to the
designers and probably not good for your blood pressure.

</F>

[Jasper]

On Aug 18, 2:40 am, Fredrik Lundh <[e-mail]:

Jasper wrote:

Uggg! /That's/ an intuitive side-effect/wart. :-/

it's done that way on purpose, of course, because evaluating a full
closure for each default argument at every call would greatly hurt
performance (and lead to another set of surprises, of course).

please don't label things that you don't understand and haven't spent
any time reflecting over as bugs or warts; that's disrespectful to the
designers and probably not good for your blood pressure.

</F>

I understand it's done that way on purpose, and that there are
tradeoffs
involved, but frankly your /guess/ that I don't understand is wrong.
Having
used Python for some 15 years, I'm hardly a neophyte -- it's pure
serendipity
that this hasn't bitten me before.

I can see the elegance from a language design perspective, the speed
advantage,
etc. Nonetheless, it's an unintuitive wart, hurting Python's clarity
-- as evidence
I'll point out all the warnings that need to be sprinkled through the
various docs.
And no, the alternative /does not/ have an equivalent set of surprises
-- it's not
like Python is unique in having default arguments.

Frankly, if I wanted speed, I wouldn't be using python, and if I
wanted clever tricks,
I'd use Perl. Surprise caching as a side-effect is /very/ Perl-like.

-Jasper

[Paul Boddie]

On 18 Aug, 11:40, Fredrik Lundh <[e-mail]:

Jasper wrote:

Uggg! */That's/ an intuitive side-effect/wart. *:-/

it's done that way on purpose, of course, because evaluating a full
closure for each default argument at every call would greatly hurt
performance (and lead to another set of surprises, of course).

Having had the opportunity to reflect on this recently, I'd agree that
the current behaviour is probably the better outcome in many cases,
although one usually only sees people having problems with this when
using literals (lists mostly, and often empty lists), so there's
always the question of how people perceive those literals, whether
they consider them sufficiently "low cost" to be evaluated for each
call, and so on. Indeed, issues of binding don't apply to such
literals, and I imagine that this conceals the possibility of
surprising behaviour (in the general case with names which could refer
to different things at different times) and the rationale for
implementing a mechanism which is consequently less complicated (both
for the developers and in terms of predicting program behaviour).

please don't label things that you don't understand and haven't spent
any time reflecting over as bugs or warts; that's disrespectful to the
designers and probably not good for your blood pressure.

Well, in the page of "Python warts" that I compiled when it was
claimed that Python 3000 addresses such issues in Python 2.x, the
"Mutable default arguments" entry lists at least one experienced
Python author who agrees with the inquirer's assertion:

http://wiki.python.org/moin/PythonWarts

Paul

[Jasper]

On Aug 18, 3:04 am, Paul Boddie <[e-mail]:

>
Well, in the page of "Python warts" that I compiled when it was
claimed that Python 3000 addresses such issues in Python 2.x, the
"Mutable default arguments" entry lists at least one experienced
Python author who agrees with the inquirer's assertion:

http://wiki.python.org/moin/PythonWarts

Paul

Not surprising, as it's fairly non-standard. I'd even argue that
calling them "default arguments" is a misnomer -- they're more akin to
static variables.

It doesn't help that the solution to get the expected behavior
involves adding boiler-plate code all over.

-Jasper

[Paul Boddie]

On 18 Aug, 12:20, Jasper <[e-mail]:

>
Not surprising, as it's fairly non-standard. *I'd even argue that
calling them "default arguments" is a misnomer -- they're more akin to
static variables.

Indeed, default parameter values are occasionally suggested for that
purpose, although it has often been said that one shouldn't really use
them for that, either because there are often superior alternatives to
static variables, or because (as at least claimed in years gone by)
the behaviour may change one day. I think the latter explanation has
little substance now, at least for implementations compatible with
CPython.

It doesn't help that the solution to get the expected behavior
involves adding boiler-plate code all over.

Yes, it's a case of avoiding the full extent of the feature because it
doesn't do what one might expect. Personally, I only really use None,
numbers and strings as defaults, with the boilerplate you mention in
the function to get the initialisation that would have been provided
by the defaults. There is, however, a useful pattern which arises from
such a conservative approach: adopting None as a default (or perhaps a
special value) means that one has a way of explicitly indicating that
the default is desired, rather than omitting a parameter - something
which may not always be convenient. The boilerplate then loads the
appropriate default which may be stored in a more convenient location:
as a module global or in a class or instance attribute.

Ultimately, I suppose one could enforce some kind of least surprising
"best practice" by limiting default parameter values to being literals
of immutable objects or names, as opposed to expressions, thus
eliminating some potential confusion. Perhaps the various static code
checking tools provide guidance on this matter.

Paul

[Steven D'Aprano]

On Mon, 18 Aug 2008 04:07:14 -0700, Paul Boddie wrote:

Ultimately, I suppose one could enforce some kind of least surprising
"best practice" by limiting default parameter values to being literals
of immutable objects or names, as opposed to expressions, thus
eliminating some potential confusion.

-1

Firstly, I *like* the ability to use mutable objects as default
arguments. I don't do it often, but when I do, I do it deliberately. I
find it useful.

Secondly, I think forbidding expressions as default arguments would be
far worse than the so-called "problem" you wish to fix. It would make
such simple default arguments as these unnecessarily complicated:

def foo(x=2**64, sentinel=object())

Please don't try to "fix" this feature.
--
Steven

[Steven D'Aprano]

On Mon, 18 Aug 2008 03:20:11 -0700, Jasper wrote:

It doesn't help that the solution to get the expected behavior involves
adding boiler-plate code all over.

Expected by who?

Please don't assume that everyone has had their intuition shaped by
exposure to the same languages yours has been shaped by. What surprises
you is obvious to me.

In a previous post, you asserted that the alternative behaviour (having
default arguments re-evaluated each time the function is called) can't
possibly be surprising. You wrote:

"And no, the alternative /does not/ have an equivalent set of surprises
-- it's not like Python is unique in having default arguments."

That's simply not true. I would find this behaviour very surprising, and
I bet you would too:

>>x = "parrot"
def foo(obj=x):

.... print obj
....

>>foo() # this is the current behaviour

parrot

>>x = "shrubbery"
foo() # but this is not

shrubbery

>>del x
foo() # nor is this

Traceback (most recent call last):
...
NameError: name 'x' is not defined

--
Steven

[Robert Brown]

Steven D'Aprano <st***@REMOVE-THIS-cybersource.com.auwrites:

On Mon, 18 Aug 2008 03:20:11 -0700, Jasper wrote:
"And no, the alternative /does not/ have an equivalent set of surprises
-- it's not like Python is unique in having default arguments."

That's simply not true. I would find this behaviour very surprising, and
I bet you would too:

>>>x = "parrot"
def foo(obj=x):

... print obj
...

>>>foo() # this is the current behaviour

parrot

>>>x = "shrubbery"
foo() # but this is not

shrubbery

>>>del x
foo() # nor is this

Traceback (most recent call last):
NameError: name 'x' is not defined

You may find the above surprising, but Common Lisp users expect the default
argument expression to be evaluated anew when need by a function call:

* (defvar *x* "parrot")
*X*

* (defun foo (&optional (obj *x*)) ;; optional arg, default is *x*
obj)
FOO

* (foo)
"parrot"

* (setf *x* "shrubbery")
"shrubbery"

* (foo)
"shrubbery"

* (makunbound '*x*)
*X*

* (foo)
debugger invoked on a UNBOUND-VARIABLE in thread #<THREAD "initial thread"
RUNNING {10023EDE81}>:
The variable *X* is unbound.

I find the Lisp approach more reasonable. Also, an argument based on
performance for Python's current behavior seems dubious, given the
language's other performance robbing design choices.

bob

[Fredrik Lundh]

Jasper wrote:

Having used Python for some 15 years, I'm hardly a neophyte -- it's pure
serendipity that this hasn't bitten me before.

Using languages and spending time reflecting over how they're put
together are two very different things.

</F>

[Fredrik Lundh]

Steven D'Aprano wrote:

>It doesn't help that the solution to get the expected behavior involves
adding boiler-plate code all over.

Expected by who?

Blub programmers.

</F>

[Fredrik Lundh]

Robert Brown wrote:

You may find the above surprising, but Common Lisp users expect the default
argument expression to be evaluated anew when need by a function call:

I find the Lisp approach more reasonable. Also, an argument based on
performance for Python's current behavior seems dubious, given the
language's other performance robbing design choices.

well, I'd say an argument based on "Common Lisp users" is a lot more
dubious ;-)

(and some of us are actually capable of writing pretty fast code in
Python. slowing the language down (or crippling it) because some blub
programmers screams "bug!" instead of looking things up in the handbook
when they stumble upon something they haven't seen before doesn't strike
me as a good use of anyone's time.)

</F>

[Robert Brown]

Fredrik Lundh <fr*****@pythonware.comwrites:

Robert Brown wrote:

>You may find the above surprising, but Common Lisp users expect the
default argument expression to be evaluated anew when needed by a
function call:

well, I'd say an argument based on "Common Lisp users" is a lot more
dubious ;-)

Actually, it's really not dubious. Because Lisp is extensible, Lisp *users*
have evolved the language considerably over the years. It's an excellent
place to look for alternative design ideas. For instance, Lisp users
experimented with several ways (LOOPS, Flavors, etc.) of supporting the
object oriented style of programming before CLOS became part of the Common
Lisp standard. If you're designing a language feature, it's often the case
that Lisp users have tried several alternatives over the last few decades
and have decided what works best, for them of course.

In any case, chances are high that Lisp's way of handling default arguments
would have been changed had it been shown to cause performance problems.
We're talking about a language used to implement operating systems --
performance is always a consideration.

[Steven D'Aprano]

On Wed, 20 Aug 2008 13:09:21 -0400, Robert Brown wrote:

In any case, chances are high that Lisp's way of handling default
arguments would have been changed had it been shown to cause performance
problems.

But nobody is suggesting that it would cause performance problems in
*Lisp*. It might, or it might not. Who cares? We're not talking about
Lisp, we're talking about *Python*, and evaluating default arguments
every time the function is called would certainly cause a performance hit
in Python.
--
Steven

[Robert Brown]

Steven D'Aprano <st***@REMOVE-THIS-cybersource.com.auwrites:

On Wed, 20 Aug 2008 13:09:21 -0400, Robert Brown wrote:

>In any case, chances are high that Lisp's way of handling default
arguments would have been changed had it been shown to cause performance
problems.

But nobody is suggesting that it would cause performance problems in
*Lisp*. It might, or it might not. Who cares? We're not talking about
Lisp, we're talking about *Python*, and evaluating default arguments
every time the function is called would certainly cause a performance hit
in Python.

Please explain why it's a performance problem for Python but not for other
languages.

[Steven D'Aprano]

On Wed, 20 Aug 2008 15:58:44 -0400, Robert Brown wrote:

Steven D'Aprano <st***@REMOVE-THIS-cybersource.com.auwrites:

>On Wed, 20 Aug 2008 13:09:21 -0400, Robert Brown wrote:

>>In any case, chances are high that Lisp's way of handling default
arguments would have been changed had it been shown to cause
performance problems.

But nobody is suggesting that it would cause performance problems in
*Lisp*. It might, or it might not. Who cares? We're not talking about
Lisp, we're talking about *Python*, and evaluating default arguments
every time the function is called would certainly cause a performance
hit in Python.

Please explain why it's a performance problem for Python but not for
other languages.

Because other languages are implemented differently from Python, just as
other languages are implemented differently from Lisp, C, Basic, Forth,
Smalltalk, Whitespace, Ruby, PHP, Java, and quite a few others.

If you mean, why is it a problem for Python but not Lisp, I'm afraid I'll
have to defer to others who are more knowledgeable about the
implementation than I am. All I know is that Fredrik Lundh, who has
forgotten more about Python than I know, has stated:

"it's done that way on purpose, of course, because evaluating a full
closure for each default argument at every call would greatly hurt
performance"

But if you want a simple example, consider this function definition:

def parrot(x=expr)

Evaluating expr will take arbitrary time. If you don't believe me,
consider the pathological case of x=time.sleep(10**6). The current
semantics pays that cost once, when the def statement is executed, and
from that point on retrieving the default value of x is almost free.
Evaluating expr every time the function is called will pay that cost
every time.

Frankly, I can't see how any language, including Lisp, could possibly
avoid that runtime cost.
--
Steven