mutable default parameter problem [Prothon]

[Mark Hahn]

As we are addressing the "warts" in Python to be fixed in Prothon, we have
come upon the
mutable default parameter problem. For those unfamiliar with the problem,
it can be seen in this Prothon code sample where newbies expect the two
function calls below to both print [ 1 ] :

def f( list=[ ] ):
print list.append!(1)

f() # prints [ 1 ]
f() # prints [ 1, 1 ]

It is more than just a newbie problem. Even experts find themselves having
to do things like this which is a waste of programming effort:

def f( list = None ):
if list == None: list = [ ]

We have three proposals in the Prothon mailing list right now to fix this.
I'd like to bounce these off of the Python list also since this will
possibly make a big difference in Python code ported over to Prothon and we
can always use more advice.

1) Only allow immutable objects as default values for formal parameters. In
Prothon immutable objects are well-defined since they have an immutable flag
that write-protects them. This (as the other solutions below) would only
solve the problem in a shallow way as one could still have something like a
tuple of mutable objects and see the problem at a deeper level. If the
newbie is going to be dealing with something this complex though then they
are dealing with the overall problem of references versus copies and that is
a bigger overall issue.

2) Evaluate the default expression once at each call time when the default
value is needed. The default expression would be evaluated in the context
of the function definition (like a closure).

3) Evaluate the expression at definition time as it is done now, but at call
time do a defaultValue.co py() operation. This would be a shallow copy so
again it would be a shallow solution.

Choice 2 is my favorite in that it matches the dynamic nature of Prothon,
but it is an expensive solution. Choice 1 is the least expensive solution
but it is limiting to the user. Choice 1 does not help the second code
sample above. Choice 3 is a good compromise since an object.copy() is
pretty fast in Prothon.

Comments? How much Python code would these different proposals break?

[Pierre-Frédéric Caillaud]

Nice !
I wish Python had this, too.

Also, re-evaluating each time will allow one to use a global variable
whose value could change as a default parameter. Which looke suspicious.
Or something like that, which is GOOD :

class youpi( object ):
def mymethod( self, a=self.a, b=self.b ):
...

Pierre-Frédéric Caillaud wrote:

2) Evaluate the default expression once at each call time when the
default
value is needed. The default expression would be evaluated in the
context
of the function definition (like a closure).

I like Choice 2 because I've always wanted to do the following :

def func( x, y=2*x ):

It looks like you will get your wish. The voting has been pretty much
unanimous for option 2.

--
Using Opera's revolutionary e-mail client: http://www.opera.com/m2/

[Hung Jung Lu]

Rob Williscroft <rt*@freenet.co .uk> wrote:

But python has static variables.

def another( x ):
y = getattr( another, 'static', 10 )
another.static = x
return y

print another(1), another(2), another(4)

What about the following case:

def f():
f.static = getattr(f, 'static', 0)
f.static += 1
return f.static

print f(), f(), f() # prints 1 2 3

As opposed to C++, you now have a line of code that is always executed
in subsequent calls, for no good reason. This is worse than:

def f(static=[0]):
static[0] += 1
return static[0]

in the sense that you have a wasteful call ("getattr") that doesn't do
anything productive in subsequent calls. (You could change that to
f.static = getattr(f, 'static', 0) + 1, but the "getattr" is surely
inefficient compared to f.static += 1, and internally likely incurs
some conditional statement at some level.)

Maybe one can do instead:

def f():
global f
def f():
f.static += 1
return f.static
f.static = 0 # initialization
return f()

print f(), f(), f() # prints 1 2 3

The advantage is there is no more wasteful statements in subsequent
calls. No "if" conditional statement. The above is of course a toy
example to illustrate the case of a function that needs to perform
something special the first time it is called. (I am well aware of the
outside assignment like:

def f():
f.static += 1
return f.static
f.static = 0

mentioned in this thread, but I am talking about something more
general. Notice that in the latter case, f.static=0 is done before f()
is called, which may not be what one wants. E.g.: if f() is never
called, this assignment is wasteful. Not a problem in this case, for
if complicated initialization is needed, like requiring a timestamp,
it may not be a good idea.)

In code refactoring, the equivalent is to replace conditional
statements by polymorphism. In terms of codeblocks, what I mean is
dynamic hook-on and hook-off of codeblocks. If the underlying language
is powerful enough, one should be able to achieve runtime
restructuring of code, without performance impact for subsequent
calls.

regards,

Hung Jung

[Mark Hahn]

Hung Jung Lu wrote:

In code refactoring, the equivalent is to replace conditional
statements by polymorphism. In terms of codeblocks, what I mean is
dynamic hook-on and hook-off of codeblocks. If the underlying language
is powerful enough, one should be able to achieve runtime
restructuring of code, without performance impact for subsequent
calls.

What about this Prothon code:

def f():
if 'static' not in f.attrs_:
f.static = 0
f.static += 1

tmp = f.static
def outer.f():
f.static += 1
return f.static
f.static = tmp

return f.static

print f() # 1
print f() # 2
print f() # 3

I'm sure Python can do this better so please don't barrage me with replies
saying so. I'm not going to brag this time :-)

[Peter Hansen]

Mark Hahn wrote:

I also said I was
sorry in my very first posting. If you want me to repeat this a third time
I will.

It's likely that Christos' Usenet link (if that's how he gets
this group) is much slower than yours or mine... he may not
have received (or at least read) either of the other replies
by the time he sent his own...

Or, he might just want you to repeat it a third time. ;-)

-Peter

[Rob Williscroft]

Hung Jung Lu wrote in news:8e******** *************** ***@posting.goo gle.com
in comp.lang.pytho n:

Rob Williscroft <rt*@freenet.co .uk> wrote:

But python has static variables.

def another( x ):
y = getattr( another, 'static', 10 )
another.static = x
return y

print another(1), another(2), another(4)
What about the following case:

def f():
f.static = getattr(f, 'static', 0)
f.static += 1
return f.static

print f(), f(), f() # prints 1 2 3

Yep this would be the way I would instinctivly write it. However
Andrea suggested:

def f():
if not hasattr( f, 'static', 0 ):
f.static = 0
f.static += 1
return f.static

Which my primitive timing tests show to be faster.

As opposed to C++, you now have a line of code that is always executed
in subsequent calls, for no good reason. This is worse than:
Well C++ has `if ( __compiler_gene rated_bool__ )` that will always
be executed ( except in the case of POD's (char, int, double etc) ).

def f(static=[0]):
static[0] += 1
return static[0]

I don't know but surely the interpreter is doing some kind of
extra if/attribute lookup in there, its just faster than the
other versions.

I've timed both, and this was by far the fastest. I don't think
the problems is the `if` though. I don't know but I suspect this
version finds the local/paramiter 'static' with __slots__ like
performance, all other version's suffer from attribute lookup
problems.

in the sense that you have a wasteful call ("getattr") that doesn't do
anything productive in subsequent calls. (You could change that to
f.static = getattr(f, 'static', 0) + 1, but the "getattr" is surely
inefficient compared to f.static += 1, and internally likely incurs
some conditional statement at some level.)

Maybe one can do instead:

def f():
global f
def f():
f.static += 1
return f.static
f.static = 0 # initialization
return f()

print f(), f(), f() # prints 1 2 3

The advantage is there is no more wasteful statements in subsequent
calls. No "if" conditional statement.
Intresting, but it looses (in my timing tests) to the f(static=[0])
version, static[0] must be faster than f.static I guess.
The above is of course a toy
example to illustrate the case of a function that needs to perform
something special the first time it is called. (I am well aware of the
outside assignment like:

def f():
f.static += 1
return f.static
f.static = 0

mentioned in this thread, but I am talking about something more
general. Notice that in the latter case, f.static=0 is done before f()
is called, which may not be what one wants. E.g.: if f() is never
called, this assignment is wasteful. Not a problem in this case, for
if complicated initialization is needed, like requiring a timestamp,
it may not be a good idea.)
Indeed, the fast version is:

_f_static = 0
def f():
global _f_static
_f_static += 1
return _f_static

This is directly equivalent to
<c++>
static int f_static = 0;
int f() { return ++f_static; }
</c++>

C++ name hiding is with the static keyword, python name hiding
is with a leading underscore.

In code refactoring, the equivalent is to replace conditional
statements by polymorphism. In terms of codeblocks, what I mean is
dynamic hook-on and hook-off of codeblocks. If the underlying language
is powerful enough, one should be able to achieve runtime
restructuring of code, without performance impact for subsequent
calls.

Nice.

<c++>
static int f_static;
static int f_init();
static int f_run();
int (*f)() = f_init();

static int f_init()
{
f_static = /* dynamic value */ 0;
f = f_run;
return f();
}
static int f_run()
{
return ++f_static;
}
</c++>

In C++ the (performance) cost is visible and only f() and its callers
pay for it, in Python the the (performance) cost is invisible and
everybody pays for it.

In C++ I write:

int f()
{
static int var = 0;
return ++var;
}

And I let the compiler worry about the best way to implement it,
in Python I write:

class f( object ):
def __init__( self ):
self.var = 0;
def run( self ):
self.var += 1
return self.var

Though in a simple case (as all the examples have been) I
might write:

def f():
if not hasattr( f, 'static' ):
f.static = 0
f.static += 1
return f.static

Clarity (my clairty of purpose, as a programmer) wins in
both languages.

I won't be writing:

def f(static=[0]):
#etc

It simply isn't clear.

Rob.
--
http://www.victim-prime.dsl.pipex.com/

[Christos TZOTZIOY Georgiou]

On Fri, 18 Jun 2004 20:38:51 -0400, rumours say that Peter Hansen
<pe***@engcorp. com> might have written:

I also said I was
sorry in my very first posting. If you want me to repeat this a third time
I will.

It's likely that Christos' Usenet link (if that's how he gets
this group) is much slower than yours or mine... he may not
have received (or at least read) either of the other replies
by the time he sent his own...

Or, he might just want you to repeat it a third time. ;-)

I am actually reading through Usenet... the newsserver link is fast, but
the arrival order of posts is not guaranteed, so I don't read messages
in the same order as people using the mail list. Often I post a reply
to a single message, only to find in the next synchronisation that
others did reply even earlier than me saying more or less the same
things. Other times I don't bother, saying "somebody else will reply to
this trivial question", and next day I find out that everybody else
thought the same as I did. That's Usenet (sigh).
--
TZOTZIOY, I speak England very best,
"Tssss!" --Brad Pitt as Achilles in unprecedented Ancient Greek

[Mark Hahn]

"Dave Brueck" <da**@pythonapo crypha.com> wrote

FYI: It's not that the exclamation mark causes append to return the
sequence. The exclamation mark is always there and the sequence is always returned. The exclamation mark is the universal symbol for in-place
modification. This is straight from Ruby and solves the problem that caused

Guido to not allow sequences to be returned. And, yes, I do think that's worth bragging about ;-)

Wait, so is the exclamation point required or not? IOW, say you have a

class like this:

class List(list):
def append(self, what):
list.append(sel f, what)
return self

a = List()
b = a.append(5)

So in the call to append, is it a.append!(5) or just a.append(5) ? If it's
the former, then does the compiler detect that it's required because the
function returns 'self' or is the determining factor something else?

Or, does the append method not really return anything, and the language
takes care of substituting in the object? (in which case, does that mean you can override the return value by adding '!' - such that z=foo.bar!() ignores the return value of bar and z references foo?)
As I said above: It's not that the exclamation mark that causes append to
return the sequence. The exclamation mark is always there and the sequence
is always returned. In Prothon (and Ruby and other languages before) the
exclamation mark is just part of the method name and is there to warn you
that in-place modification is happening.
Personally, I don't like the modify-in-place-and-return-the-object
'feature' - it's not needed _that_ often, but more importantly, it makes the code harder to read (to me at least).

If you use the Prothon append!() exactly as you use the Python append() you
will get the exact same results. This is just an extra feature for those
that want it.

Guido avoided returning values from in-place modification functions because
of the confusion as to whether in-place mods were happening or not. We have
solved that confusion with the exclamation mark. Our code is very readable
because of this.

[Mark Hahn]

Christos TZOTZIOY Georgiou wrote:

Or, he might just want you to repeat it a third time. ;-)

I am actually reading through Usenet... the newsserver link is fast,
but the arrival order of posts is not guaranteed, so I don't read
messages in the same order as people using the mail list. Often I
post a reply to a single message, only to find in the next
synchronisation that others did reply even earlier than me saying
more or less the same things. Other times I don't bother, saying
"somebody else will reply to this trivial question", and next day I
find out that everybody else thought the same as I did. That's
Usenet (sigh).

I'm sorry if I was rude. It is always when I make a stupid remark on a
public forum that I am asked to repeat it :-) No one ever asks me to repeat
a witty gem. (Of course maybe that never occurs).

[Christos TZOTZIOY Georgiou]

On Thu, 24 Jun 2004 17:06:33 -0700, rumours say that "Mark Hahn"
<ma**@prothon.o rg> might have written:

No one ever asks me to repeat
a witty gem.

You can say that again :)
--
TZOTZIOY, I speak England very best,
"Tssss!" --Brad Pitt as Achilles in unprecedented Ancient Greek

[Dave Brueck]

Mark wrote:

Wait, so is the exclamation point required or not? IOW, say you have a class

like this:

class List(list):
def append(self, what):
list.append(sel f, what)
return self

a = List()
b = a.append(5)

So in the call to append, is it a.append!(5) or just a.append(5) ? If it's the former, then does the compiler detect that it's required because the
function returns 'self' or is the determining factor something else?

Or, does the append method not really return anything, and the language
takes care of substituting in the object? (in which case, does that mean

you

can override the return value by adding '!' - such that z=foo.bar!()

ignores

the return value of bar and z references foo?)

As I said above: It's not that the exclamation mark that causes append to
return the sequence. The exclamation mark is always there and the

sequence is always returned. In Prothon (and Ruby and other languages before) the
exclamation mark is just part of the method name and is there to warn you
that in-place modification is happening.
Ahh...so the method name is just _spelled_ with an exclamation point? IOW,
the ! is a token you can use at the end of an identifier, but it is not
actually used by the language itself - it's some sort of pseudo-syntax? I
think I understand now. But is it truly part of the name in that you are
required to include the ! when calling the method? (I'm still thinking of
the confusion I'd experience with something like w = x.y.z!() )

So if I want a reference to one of those methods I could end up doing

ref = obj.method! or ref! = obj.method!

and the program runs the same either way, it's just that in one case the
code is misleading?

If it's up to the programmer to remember to add it (meaning that it doesn't
cause an error to forget to use it), and if is really just part of the name,
then it's just a naming convention, right? Wouldn't you get the same result
by establishing the convention that e.g. method names ending in a single
underscore signify in-place modification (foo.append_() ) ? Seems like a
waste to reserve a symbol for something so rarely needed.

Personally, I don't like the modify-in-place-and-return-the-object
'feature' - it's not needed _that_ often, but more importantly, it makes

the

code harder to read (to me at least).

If you use the Prothon append!() exactly as you use the Python append()

you will get the exact same results. This is just an extra feature for those
that want it.

Guido avoided returning values from in-place modification functions because of the confusion as to whether in-place mods were happening or not. We have solved that confusion with the exclamation mark. Our code is very readable because of this.

Clearly, readability is in the eye of the beholder. :)

-Dave