lambda

Op 2005-01-13, Tim Leslie schreef <ti********@gmail.com>:

Because if it takes more than a single line it deserves a name.

So if I have a call with an expression that takes more than
one line, I should assign the expression to a variable and
use the variable in the call?

But wait if I do that, people will tell me how bad that it
is, because it will keep a reference to the value which
will prevent the garbage collector from harvesting this
memory.

Besides python allows more than one statement on the same line.

--
Antoon Pardon

Jul 18 '05 #4

hanz

Antoon Pardon wrote:

So if I have a call with an expression that takes more than
one line, I should assign the expression to a variable and
use the variable in the call?
Yes, that's sometimes a good practice and can clarify
the call.
But wait if I do that, people will tell me how bad that it
is, because it will keep a reference to the value which
will prevent the garbage collector from harvesting this
memory.
Nobody will tell you that it's bad. Python was never
about super performance, but about readability.
Besides, using such temporaries won't consume much
memory (relatively).
Besides python allows more than one statement on the same line.

But it's discouraged in general.

Jul 18 '05 #5

Paul Rubin

"hanz" <ha******@yahoo.com.au> writes:

But wait if I do that, people will tell me how bad that it is,
because it will keep a reference to the value which will prevent
the garbage collector from harvesting this memory.

Nobody will tell you that it's bad. Python was never about super
performance, but about readability. Besides, using such temporaries
won't consume much memory (relatively).

That completely depends on the objects in question. Compare

temp = all_posters[:]
temp.sort()
top_five_posters = temp[-5:]

to:

top_five_posters = all_posters.sorted()[-5:]

which became possible only when .sorted() was added to Python 2.4.

This is another reason it would be nice to be able to create very
temporary scopes.

Jul 18 '05 #6

Paul Rubin wrote:

That completely depends on the objects in question. Compare

temp = all_posters[:]
temp.sort()
top_five_posters = temp[-5:]

to:

top_five_posters = all_posters.sorted()[-5:]

which became possible only when .sorted() was added to Python 2.4.

I believe you mean "when sorted() was added to Python 2.4":

py> ['d', 'b', 'c', 'a'].sorted()
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
AttributeError: 'list' object has no attribute 'sorted'
py> sorted(['d', 'b', 'c', 'a'])
['a', 'b', 'c', 'd']

Note that sorted is a builtin function, not a method of a list object.
It takes any iterable and creates a sorted list from it. Basically the
equivalent of:

def sorted(iterable):
result = list(iterable)
result.sort()
return result

Steve

Jul 18 '05 #7

Paul Rubin

Steven Bethard <st************@gmail.com> writes:

Note that sorted is a builtin function, not a method of a list
object.

Oh, same difference. I thought it was a method because I'm not using
2.4 yet. The result is the same, other than that having it as a
function instead of a method is another inconsistency to remember.

Jul 18 '05 #8

Fredrik Lundh

Paul Rubin wrote:

Note that sorted is a builtin function, not a method of a list
object.
Oh, same difference. I thought it was a method because I'm not using
2.4 yet. The result is the same

nope. sorted works on any kind of sequence, including forward-only
iterators. sorted(open(filename)) works just fine, for example.
other than that having it as a function instead of a method is another
inconsistency to remember

I suspect that you don't really understand how sequences and iterators
work in Python...

</F>

Jul 18 '05 #9

Paul Rubin

"Fredrik Lundh" <fr*****@pythonware.com> writes:

Oh, same difference. I thought it was a method because I'm not using
2.4 yet. The result is the same

nope. sorted works on any kind of sequence, including forward-only
iterators. sorted(open(filename)) works just fine, for example.

Oh cool. However I meant the result is the same in my example, where
assigning the temporary result to a variable stopped memory from
getting reclaimed until after the function exits.

Jul 18 '05 #10

Terry Reedy

"Egor Bolonev" <eb******@mail.ru> wrote in message
news:op**************@theurs.octopusnet.lan...

why functions created with lambda forms cannot contain statements?

Because lambda was only ever intended to be an inline abbreviation of
simple one-use functions whose body consists of 'return <expression>'. It
is clear to me that the word 'lambda' was a mistake since it engenders
expectations of more than that from too many people, such as you.

Terry J. Reedy

Jul 18 '05 #11

Terry Reedy

"Paul Rubin" <"http://phr.cx"@NOSPAM.invalid> wrote in message
news:7x************@ruckus.brouhaha.com...

Steven Bethard <st************@gmail.com> writes:
Note that sorted is a builtin function, not a method of a list
object.

Oh, same difference. I thought it was a method because I'm not using
2.4 yet. The result is the same, other than that having it as a
function instead of a method is another inconsistency to remember.

No, not same difference. A list method would only operate on lists, as is
true of all list methods. Being a function lets it work for any iterable,
as is true of any function of iterable. Big difference. And consistent.
One could argue though that it should have been put into itermethods module
instead of builtins.

Terry J. Reedy

Jul 18 '05 #12

Op 2005-01-13, hanz schreef <ha******@yahoo.com.au>:

Antoon Pardon wrote:
So if I have a call with an expression that takes more than
one line, I should assign the expression to a variable and
use the variable in the call?

Yes, that's sometimes a good practice and can clarify
the call.
But wait if I do that, people will tell me how bad that it
is, because it will keep a reference to the value which
will prevent the garbage collector from harvesting this
memory.

Nobody will tell you that it's bad.

Sorry, someone already did. If I recall correctly it
was Alex Martelli.

--
Antoon Pardon

Jul 18 '05 #13

Antoon Pardon wrote:

Op 2005-01-13, hanz schreef <ha******@yahoo.com.au>:
Antoon Pardon wrote:
So if I have a call with an expression that takes more than
one line, I should assign the expression to a variable and
use the variable in the call?

Yes, that's sometimes a good practice and can clarify
the call.

But wait if I do that, people will tell me how bad that it
is, because it will keep a reference to the value which
will prevent the garbage collector from harvesting this
memory.

Of course, unless that reference is in the global scope of the __main__
module its lifetime will be transient anyway. If the reference is stored
in a function's local variable then unless its value is returned from
the function it will become available for garbage collection when the
function returns.
Nobody will tell you that it's bad.

Sorry, someone already did. If I recall correctly it
was Alex Martelli.

"A foolish consistency is the hobgoblin of little minds". Rules are made
to be broken. Besides which, if you don't understand the language
environment, rules alone will do you very little good. Try to focus a
little more on principles and a little less on minutiae.

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119

Jul 18 '05 #14

Op 2005-01-14, Steve Holden schreef <st***@holdenweb.com>:

Antoon Pardon wrote:
Op 2005-01-13, hanz schreef <ha******@yahoo.com.au>:
Antoon Pardon wrote:

So if I have a call with an expression that takes more than
one line, I should assign the expression to a variable and
use the variable in the call?

Yes, that's sometimes a good practice and can clarify
the call.
But wait if I do that, people will tell me how bad that it
is, because it will keep a reference to the value which
will prevent the garbage collector from harvesting this
memory.
Of course, unless that reference is in the global scope of the __main__
module its lifetime will be transient anyway. If the reference is stored
in a function's local variable then unless its value is returned from
the function it will become available for garbage collection when the
function returns.
Nobody will tell you that it's bad.

Sorry, someone already did. If I recall correctly it
was Alex Martelli.

"A foolish consistency is the hobgoblin of little minds". Rules are made
to be broken.

Like only use immutables as dictionary keys.
Besides which, if you don't understand the language
environment, rules alone will do you very little good. Try to focus a
little more on principles and a little less on minutiae.

And what are the difference between those two?

Sometimes I get the impression that everything is a principle until
one personnaly finds the need to break it. After that it is a rule.

--
Antoon Pardon

Jul 18 '05 #15

Antoon Pardon wrote:
[...]

"A foolish consistency is the hobgoblin of little minds". Rules are made
to be broken.

Like only use immutables as dictionary keys.

Fair enough, but don;t go advising newbies to do this.
Besides which, if you don't understand the language
environment, rules alone will do you very little good. Try to focus a
little more on principles and a little less on minutiae.

And what are the difference between those two?

Sometimes I get the impression that everything is a principle until
one personnaly finds the need to break it. After that it is a rule.

Principle: "Ten angels can dance on the head of a pin".

Minutiae: "Well, if they'd all recently been on a diet and they hold on
to each other very carefully you can get 10.3. I got eleven once by
adding a hash function".

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119

Jul 18 '05 #16

Op 2005-01-17, Steve Holden schreef <st***@holdenweb.com>:

Antoon Pardon wrote:
[...]
"A foolish consistency is the hobgoblin of little minds". Rules are made
to be broken.

Like only use immutables as dictionary keys.

Fair enough, but don;t go advising newbies to do this.

How about something like this.

Because of the extra precautions one has to take when
using mutables as hash keys, we advise newbies
to stick with immutable keys until they have gathered
enough knowledge and experience to adequatly weight
the pro and cons of a mutable key solution against
an immutable key solution.

--
Antoon Pardon

Jul 18 '05 #17

Antoon Pardon wrote:

Op 2005-01-17, Steve Holden schreef <st***@holdenweb.com>:
Antoon Pardon wrote:
[...]
"A foolish consistency is the hobgoblin of little minds". Rules are made
to be broken.
Like only use immutables as dictionary keys.

Fair enough, but don;t go advising newbies to do this.

How about something like this.

Because of the extra precautions one has to take when
using mutables as hash keys, we advise newbies
to stick with immutable keys until they have gathered
enough knowledge and experience to adequatly weight
the pro and cons of a mutable key solution against
an immutable key solution.

There you go with the minutiae again. How about:

"Don't use mutables as hash keys"?

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119

Jul 18 '05 #18

Op 2005-01-17, Steve Holden schreef <st***@holdenweb.com>:

Antoon Pardon wrote:
Op 2005-01-17, Steve Holden schreef <st***@holdenweb.com>:
Antoon Pardon wrote:
[...]

>"A foolish consistency is the hobgoblin of little minds". Rules are made
>to be broken.
Like only use immutables as dictionary keys.
Fair enough, but don;t go advising newbies to do this.

How about something like this.

Because of the extra precautions one has to take when
using mutables as hash keys, we advise newbies
to stick with immutable keys until they have gathered
enough knowledge and experience to adequatly weight
the pro and cons of a mutable key solution against
an immutable key solution.

There you go with the minutiae again. How about:

"Don't use mutables as hash keys"?

That sounds too dogmatic to my ears. I also find it
too selective. The problem with mutables as dictionary
keys is not specific to dictionaries. Everywhere you
have mutables in a container, it is possible that
mutating the object in the container will cause
problem. Heck even using mutables as arguments can
cause trouble. Why else the specific advice against

def foo(p = [])

type of arguments. So should we adopt the principles:

Don't use mutables in containers

Don't use mutables as default values for parameters

Don't use mutables as arguments.

Don't assign one mutable to an other.
I don't see a big difference between these principles
and the hash key principle, so in the end may be we
should just stick with the more general principle:

Don't use mutables!
and be done with it.

--
Antoon Pardon

Jul 18 '05 #19

Just

In article <sl********************@rcpc42.vub.ac.be>,
Antoon Pardon <ap*****@forel.vub.ac.be> wrote:

Op 2005-01-17, Steve Holden schreef <st***@holdenweb.com>:
There you go with the minutiae again. How about:

"Don't use mutables as hash keys"?

That sounds too dogmatic to my ears. I also find it
too selective. The problem with mutables as dictionary
keys is not specific to dictionaries. Everywhere you
have mutables in a container, it is possible that
mutating the object in the container will cause
problem.

The main difference is: if you mutate a dict key you *always* have a
problem. So don't do that. Mutating (say) a list item *only* is a
problem if you (say) need that list to remain sorted. Lists don't
magically remain sorted, so people generally sort it before they do some
operation that expects a sorted list.
Heck even using mutables as arguments can
cause trouble. Why else the specific advice against

def foo(p = [])

type of arguments. So should we adopt the principles:

Don't use mutables in containers
Nonsense.
Don't use mutables as default values for parameters
That's good advice in general.
Don't use mutables as arguments.
Nonsense.
Don't assign one mutable to an other.
Nonsense. Some newbies get surprised by Python's assignment-doesn't-copy
semantics, but it's such basic knowledge (as well as a useful feature)
that I simply don't understand you saying this.
I don't see a big difference between these principles
and the hash key principle,
Than you haven't looked hard enough.
so in the end may be we
should just stick with the more general principle:

Don't use mutables!
and be done with it.

Ok, so you're indeed a troll.

Just

Jul 18 '05 #20

Antoon Pardon wrote:

Op 2005-01-17, Steve Holden schreef <st***@holdenweb.com>:
Antoon Pardon wrote:

Op 2005-01-17, Steve Holden schreef <st***@holdenweb.com>:
Antoon Pardon wrote:
[...]
>>"A foolish consistency is the hobgoblin of little minds". Rules are made
>>to be broken.
>
>
>Like only use immutables as dictionary keys.
>

Fair enough, but don;t go advising newbies to do this.
How about something like this.

Because of the extra precautions one has to take when
using mutables as hash keys, we advise newbies
to stick with immutable keys until they have gathered
enough knowledge and experience to adequatly weight
the pro and cons of a mutable key solution against
an immutable key solution.

There you go with the minutiae again. How about:

"Don't use mutables as hash keys"?

That sounds too dogmatic to my ears. I also find it
too selective. The problem with mutables as dictionary
keys is not specific to dictionaries. Everywhere you
have mutables in a container, it is possible that
mutating the object in the container will cause
problem. Heck even using mutables as arguments can
cause trouble. Why else the specific advice against

def foo(p = [])

type of arguments. So should we adopt the principles:

Don't use mutables in containers

Don't use mutables as default values for parameters

Don't use mutables as arguments.

Don't assign one mutable to an other.
I don't see a big difference between these principles
and the hash key principle, so in the end may be we
should just stick with the more general principle:

Don't use mutables!
and be done with it.

http://redwing.hutman.net/~mreed/war...ssrebutter.htm

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119

Jul 18 '05 #21

Op 2005-01-17, Just schreef <ju**@xs4all.nl>:

In article <sl********************@rcpc42.vub.ac.be>,
Antoon Pardon <ap*****@forel.vub.ac.be> wrote:
Op 2005-01-17, Steve Holden schreef <st***@holdenweb.com>:
> There you go with the minutiae again. How about:
>
> "Don't use mutables as hash keys"?

That sounds too dogmatic to my ears. I also find it
too selective. The problem with mutables as dictionary
keys is not specific to dictionaries. Everywhere you
have mutables in a container, it is possible that
mutating the object in the container will cause
problem.

The main difference is: if you mutate a dict key you *always* have a
problem. So don't do that. Mutating (say) a list item *only* is a
problem if you (say) need that list to remain sorted.

That is not true. It is a problem every time I expect the list
items to remain the same.
Lists don't
magically remain sorted, so people generally sort it before they do some
operation that expects a sorted list.
Heck even using mutables as arguments can
cause trouble. Why else the specific advice against

def foo(p = [])

type of arguments. So should we adopt the principles:

Don't use mutables in containers

Nonsense.
Don't use mutables as default values for parameters

That's good advice in general.
Don't use mutables as arguments.

Nonsense.
Don't assign one mutable to an other.

Nonsense. Some newbies get surprised by Python's assignment-doesn't-copy
semantics, but it's such basic knowledge (as well as a useful feature)
that I simply don't understand you saying this.

Well it is this same sematics that causes the problems with
mutable dictionary keys. If it is such basic knowledge then what
is the problem with mutable dictionary keys?

I don't see a big difference between these principles
and the hash key principle,

Than you haven't looked hard enough.

All of these can get unexpected behaviour because of the
assignment-doesn't-copy semantics. The same semantics
that can cause problems if you work with mutable dictionary
keys.

so in the end may be we
should just stick with the more general principle:

Don't use mutables!

and be done with it.

Ok, so you're indeed a troll.

The problems with mutables as dictionary keys is just one
particulary case of the problems you can have when your
assignmentxs semantics just creates a new reference to the
same object. As such it is no different from the problem
of mutating one object and finding out this object was
also reference through an other name you expected to remain
the same or finding out this object was also in a list you
expected to remain stable etc.

--
Antoon Pardon

Jul 18 '05 #22

Op 2005-01-17, Steve Holden schreef <st***@holdenweb.com>:

Antoon Pardon wrote:
I don't see a big difference between these principles
and the hash key principle, so in the end may be we
should just stick with the more general principle:

Don't use mutables!
and be done with it.

http://redwing.hutman.net/~mreed/war...ssrebutter.htm

regards
Steve

I you want to play it like this

http://redwing.hutman.net/~mreed/war...tm/crybaby.htm

--
Antoon Pardon

Jul 18 '05 #23

John Lenton

On Mon, Jan 17, 2005 at 11:41:20AM +0000, Antoon Pardon wrote:

Op 2005-01-17, Steve Holden schreef <st***@holdenweb.com>:
Antoon Pardon wrote:
[...]
"A foolish consistency is the hobgoblin of little minds". Rules are made
to be broken.

Like only use immutables as dictionary keys.

Fair enough, but don;t go advising newbies to do this.

How about something like this.

Because of the extra precautions one has to take when
using mutables as hash keys, we advise newbies
to stick with immutable keys until they have gathered
enough knowledge and experience to adequatly weight
the pro and cons of a mutable key solution against
an immutable key solution.

knowledgeable and experienced users know when to ignore the rules.

--
John Lenton (jo**@grulic.org.ar) -- Random fortune:
Una buena gran parte del arte del bien hablar consiste en saber mentir con
gracia.
-- Erasmo de Rotterdam. (1469-1536).

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)

iD8DBQFB69asgPqu395ykGsRAkNNAJwPy4YOsAUoUHWTJc6M1v WqdgquIwCdFCaf
LokGmkEUZL/0ezwbzMpRScc=
=XKjw
-----END PGP SIGNATURE-----

Jul 18 '05 #24

Just

In article <sl********************@rcpc42.vub.ac.be>,
Antoon Pardon <ap*****@forel.vub.ac.be> wrote:

I don't see a big difference between these principles
and the hash key principle,

Than you haven't looked hard enough.

All of these can get unexpected behaviour because of the
assignment-doesn't-copy semantics. The same semantics
that can cause problems if you work with mutable dictionary
keys.

Again, the difference is:

1. assigning mutable objects *can* cause unexpected behavior
(however, it's a useful feature, everyone using Python
for longer than a day or two knows this, and then it's
*expected* behavior.

2. mutating dict keys *does* *always* cause problems.
(unless you use an identity hash/cmp)

It's nonsense to forbid 1) since it's a useful feature. It's useful to
forbid ("discourage") 2) since mutating dict keys is seldom useful (and
when it is, Python lets you support it in your own objects).

Just

Jul 18 '05 #25

Op 2005-01-17, John Lenton schreef <jo**@grulic.org.ar>:

--vni90+aGYgRvsTuO
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Mon, Jan 17, 2005 at 11:41:20AM +0000, Antoon Pardon wrote:
Op 2005-01-17, Steve Holden schreef <st***@holdenweb.com>:
> Antoon Pardon wrote:
> [...]
>>>"A foolish consistency is the hobgoblin of little minds". Rules are ma= de=20 >>>to be broken.
>>=20
>>=20
>> Like only use immutables as dictionary keys.
>>=20
> Fair enough, but don;t go advising newbies to do this.

=20
How about something like this.
=20
Because of the extra precautions one has to take when
using mutables as hash keys, we advise newbies
to stick with immutable keys until they have gathered
enough knowledge and experience to adequatly weight
the pro and cons of a mutable key solution against
an immutable key solution.

knowledgeable and experienced users know when to ignore the rules.

Then why seems there to be so few acknowledgement that these rules
may indeed be broken by users. My experience is that anyone who suggests
so runs the risk of being branded a (python) heretic.

--
Antoon Pardon

Jul 18 '05 #26

Antoon Pardon wrote:

Op 2005-01-17, Steve Holden schreef <st***@holdenweb.com>:
Antoon Pardon wrote:

I don't see a big difference between these principles
and the hash key principle, so in the end may be we
should just stick with the more general principle:

Don't use mutables!
and be done with it.

http://redwing.hutman.net/~mreed/war...ssrebutter.htm

regards
Steve

I you want to play it like this

http://redwing.hutman.net/~mreed/war...tm/crybaby.htm

Hey, I'm not crying, or trying to play it any particular way. It's a
free Internet, and you are (more than) welcome to your opinions.
Personally I'd say I'm actually more of a combination of
http://redwing.hutman.net/~mreed/war...m/diplomat.htm and
http://redwing.hutman.net/~mreed/war.../evilclown.htm, with added
touches of http://redwing.hutman.net/~mreed/warriorshtm/garble.htm and
possibly http://redwing.hutman.net/~mreed/war...oxicgranny.htm

Whereas I also detect touches of
http://redwing.hutman.net/~mreed/war...rouscranus.htm in your
posts. It must be nice to *never* be wrong :-). Your resemblance to
http://redwing.hutman.net/~mreed/war...filibuster.htm, however,
makes me wish you could be right rather more succinctly.

Mostly, though, I was trying to say that I found your nitpicking
insistence on terminological exactitude, even when giving advice to
those new to the language, both inappropriate and tedious in the extreme.

Have a nice day :-)

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119

Jul 18 '05 #27

Craig Ringer

On Mon, 2005-01-17 at 12:15 -0300, John Lenton wrote:

knowledgeable and experienced users know when to ignore the rules.

+1 QOTW

One of the nice things is that Python permits you to do exactly that
where appropriate while avoiding forcing you to do gruesome things to
get a job done.

I think the classic example of your statement is the use of 'goto' in C
and C++ code. Don't use goto - except when there's no other sensible way
to make your code clear. For example, goto and Py_XECREF seem to be very
handy for cleanup after detecting an exception when working with the
Python/C API.

That said, I do think "the rules" deserve consideration and respect -
they're usually there because of many others' experience over time. It's
interesting to learn those lessons first hand, but it's nice to be able
to avoid repeating every single one of them.

--
Craig Ringer

Jul 18 '05 #28

Op 2005-01-17, Just schreef <ju**@xs4all.nl>:

In article <sl********************@rcpc42.vub.ac.be>,
Antoon Pardon <ap*****@forel.vub.ac.be> wrote:
>> I don't see a big difference between these principles
>> and the hash key principle,
>
> Than you haven't looked hard enough.
All of these can get unexpected behaviour because of the
assignment-doesn't-copy semantics. The same semantics
that can cause problems if you work with mutable dictionary
keys.

Again, the difference is:

1. assigning mutable objects *can* cause unexpected behavior
(however, it's a useful feature, everyone using Python
for longer than a day or two knows this, and then it's
*expected* behavior.

2. mutating dict keys *does* *always* cause problems.
(unless you use an identity hash/cmp)

3 mutating an item in a sorted list *does* *always* cause problems

4 mutating an item in a heap queue *does* *always* cause problems
It's nonsense to forbid 1) since it's a useful feature. It's useful to
forbid ("discourage") 2) since mutating dict keys is seldom useful (and
when it is, Python lets you support it in your own objects).

Yes mutating dict keys is seldom usefull. But again you are conflating
mutable dict keys with mutating dict keys. No mutables as dict keys
means you can't mutate the object even if it is not a key. But whether
assigning mutable objects is usefull or not doesn't depend on whether
the object will also be usefull as a key or not. You treat this as
if an obejct is always a key or never and then decide mutable objects
as dict keys is not usefull because mutating keys is not usefull.

More specific the Decimal class is mutable and usable as dict key.

--
Antoon Pardon

Jul 18 '05 #29

Op 2005-01-17, Steve Holden schreef <st***@holdenweb.com>:

Antoon Pardon wrote:

Mostly, though, I was trying to say that I found your nitpicking
insistence on terminological exactitude, even when giving advice to
those new to the language, both inappropriate and tedious in the extreme.

I think it is appropiate because not all new to the language are new to
programming and even newbees have a right to know how it really is. Otherwise
after some time you get experienced users who don't know the fact. In
this case for example there are a number of people who flat out assert that
muatble dict keys in pyhthon is impossible.

--
Antoon Pardon

Jul 18 '05 #30

Antoon Pardon wrote:

In this case for example there are a number of people who flat out
assert that muatble dict keys in pyhthon is impossible.

If you run into any of these folks, please point them to:

http://www.python.org/moin/DictionaryKeys

It's a pretty good summary of one of the more recent threads on this
topic by Peter Maas and myself. If you feel like your point is not
covered[1], feel free to add some additional material. (That is, after
all, the point of a wiki.)

Steve

[1] I assume that you feel this way because you have repeated
essentially the same point about 5 times in this thread. The wiki is a
chance to make your statement once, and then simply post a pointer once
per appropriate thread. It might save you some time...

Jul 18 '05 #31

Simon Brunning

On 18 Jan 2005 07:51:00 GMT, Antoon Pardon <ap*****@forel.vub.ac.be> wrote:

3 mutating an item in a sorted list *does* *always* cause problems
No, it doesn't. It might cause the list no longer to be sorted, but
that might or might no be a problem.
More specific the Decimal class is mutable and usable as dict key.

Decimal objects are immutable, so far as I know.

from decimal import Decimal
spam = Decimal('1.2')
eggs = spam
eggs is spam True eggs += 1
eggs is spam

False

--
Cheers,
Simon B,
si***@brunningonline.net,
http://www.brunningonline.net/simon/blog/

Jul 18 '05 #32

Op 2005-01-18, Simon Brunning schreef <si************@gmail.com>:

On 18 Jan 2005 07:51:00 GMT, Antoon Pardon <ap*****@forel.vub.ac.be> wrote:
3 mutating an item in a sorted list *does* *always* cause problems

No, it doesn't. It might cause the list no longer to be sorted, but
that might or might no be a problem.

Than in the same vain I can say that mutating a key in a dictionary
doesn't always cause problems either. Sure it may probably make a
key unaccessible directly, but that might or might not be a problem.

More specific the Decimal class is mutable and usable as dict key.

Decimal objects are immutable, so far as I know.
from decimal import Decimal
spam = Decimal('1.2')
eggs = spam
eggs is spam True eggs += 1
eggs is spam False

from decimal import Decimal
spam = Decimal('1.2')
egg = spam
spam._int = (1, 3)
spam Decimal("1.3") spam is egg

True

--
Antoon Pardon

Jul 18 '05 #33

Nick Coghlan

Antoon Pardon wrote:

More specific the Decimal class is mutable and usable as dict key.

It's *meant* to be immutable though. The fact that we used __slots__ instead of
__setattr__ to implement the immutability, so you *can* overwrite the slot
variables if you really want to is merely an artifact of the current Python
implementation.

The limited mutability bug will disappear in Python 2.5, so it's not a good
example of a 'mutable' dict key (especially given that the only way to mutate it
is to modify private variables directly).

And, as I've stated previously, if the issue of sane mutable keys in
dictionaries and sets really bugs you so much - implement identity_dict and
identity_set in C and lobby for their inclusion in the collections module.

On the more general point of "don't use mutable objects with non-identity based
comparisons as dictionary keys", try teaching students for a while (or listen to
those who have):

When stating useful general principles, it is never, ever worth it to get into
the quibbly little details about exceptions to the principle. If students ask,
admit that they exist, but point out that the exceptions are rare, and not worth
worrying about at that point in their learning.

Only when one is aware of the reasons for a principle, can one be aware of good
reasons not to follow it :)

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

Jul 18 '05 #34

Op 2005-01-18, Nick Coghlan schreef <nc******@iinet.net.au>:

Antoon Pardon wrote:
More specific the Decimal class is mutable and usable as dict key.
It's *meant* to be immutable though. The fact that we used __slots__ instead of
__setattr__ to implement the immutability, so you *can* overwrite the slot
variables if you really want to is merely an artifact of the current Python
implementation.

The limited mutability bug will disappear in Python 2.5, so it's not a good
example of a 'mutable' dict key (especially given that the only way to mutate it
is to modify private variables directly).

And, as I've stated previously, if the issue of sane mutable keys in
dictionaries and sets really bugs you so much - implement identity_dict and
identity_set in C and lobby for their inclusion in the collections module.

I'm not bugged by its absence in Python. I'm bugged by the attitude
about them. But anyway I'm thinking about implementing a safe dictionary
that will copy whenever it would otherwise allow the risk of mutating a
key.
On the more general point of "don't use mutable objects with non-identity based
comparisons as dictionary keys", try teaching students for a while (or listen to
those who have):
What kind of students?

I have implemented a hash table when I was a student and its
implementation allowed the use of 'mutable' objects as a key
without a problem. It simply always made copies when appropiate
and didn't allow external access to the keys. So although the
key objects were 'mutable' there was no way a user could accidently
mutate a key.

So don't use a mutable as a dictionary key isn't so much a dictionary
limitation in general but a specific limitation of the python implementation.

And yes I understand, the current implenatation is the result of the
fact that the same dictionaries are used internally for variables in
scopes and attributes in objects and the fact that no copies are
involved gives a boost to performance. But it could be argued that
providing these same dictionaries with those semantics to the users
was a premature optimisation.
When stating useful general principles, it is never, ever worth it to get into
the quibbly little details about exceptions to the principle. If students ask,
admit that they exist, but point out that the exceptions are rare, and not worth
worrying about at that point in their learning.

But don't use mutable keys is not a general principle. It is a principle
introduced by the limitations of the python implementations.

I don't like it when a good rule of thumb because of implementation
limitations is sold as a general principle.

--
Antoon Pardon

Jul 18 '05 #35

John Lenton

On Mon, Jan 17, 2005 at 03:20:01PM +0000, Antoon Pardon wrote:

Op 2005-01-17, John Lenton schreef <jo**@grulic.org.ar>:

knowledgeable and experienced users know when to ignore the rules.

Then why seems there to be so few acknowledgement that these rules
may indeed be broken by users. My experience is that anyone who suggests
so runs the risk of being branded a (python) heretic.

First you learn the basics, then you think you're knowledgeable and
experienced, then you learn the rules, then you become one with the
rules, and then you may break them.

Most people suggesting these things haven't gotten past step #3. Using
Craig's parallel to C's goto, every and all newbie using gotos should
be lambasted: even if the use might be correct for the problem they
are trying to solve, the reasons for its correctness are far too
complex for them to grasp. But really, in practically any system, the
rules are generalizations, and they exist because the particulars are
too delicate to trust the unexperienced. The small print is
unprintable.

He dicho.

--
John Lenton (jo**@grulic.org.ar) -- Random fortune:
Yo siempre seré el futuro Nobel. Debe ser una tradición escandinava.
-- Jorge Luis Borges. (1899-1986) Escritor argentino.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)

iD8DBQFB7RuVgPqu395ykGsRAi6fAJ42W5rGkkqc6NaQUjxDPb sjUGZragCgs27C
sr5B5HIe+LXvYaxjqyc2PQg=
=L6xP
-----END PGP SIGNATURE-----

Jul 18 '05 #36

Antoon Pardon wrote:

Op 2005-01-18, Nick Coghlan schreef <nc******@iinet.net.au>: [...] But don't use mutable keys is not a general principle. It is a principle
introduced by the limitations of the python implementations.
Sorry, but it *is* a general principle, adduced from the potential
pitfalls available to inexperienced programmers when breaking the principle.
I don't like it when a good rule of thumb because of implementation
limitations is sold as a general principle.

So, since you are so good at nit-picking, perhaps you will explain the
difference between "rule of thumb" and "general principle".

preferably-in-less-than-three-thousand-words-ly y'rs - steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119

Jul 18 '05 #37

Op 2005-01-18, Steve Holden schreef <st***@holdenweb.com>:

Antoon Pardon wrote:
Op 2005-01-18, Nick Coghlan schreef <nc******@iinet.net.au>:

[...]
But don't use mutable keys is not a general principle. It is a principle
introduced by the limitations of the python implementations.

Sorry, but it *is* a general principle, adduced from the potential
pitfalls available to inexperienced programmers when breaking the principle.

But if these inexperienced programmers would have used dictionaries
that were implemented more safely, those pitfalls would have been
avoided. So are the pitfalls caused by breaking the principle
or by using an unsafe dictionary implementation?

I don't like it when a good rule of thumb because of implementation
limitations is sold as a general principle.

So, since you are so good at nit-picking, perhaps you will explain the
difference between "rule of thumb" and "general principle".

A rule of thumb is context sensitive. If circumstances change,
so do the rules of thumb. Principles have a broader field
of application.

IMO there is nothing principally wrong with using a mutable object
as a dictionary key. But avoiding doing so is a good rule of
thumb if you have a python-like implementation of a dictionary.

--
Antoon Pardon

Jul 18 '05 #38

David Bolen

Antoon Pardon <ap*****@forel.vub.ac.be> writes:

Op 2005-01-18, Simon Brunning schreef <si************@gmail.com>:
On 18 Jan 2005 07:51:00 GMT, Antoon Pardon <ap*****@forel.vub.ac.be> wrote:
3 mutating an item in a sorted list *does* *always* cause problems

No, it doesn't. It might cause the list no longer to be sorted, but
that might or might no be a problem.

Than in the same vain I can say that mutating a key in a dictionary
doesn't always cause problems either. Sure it may probably make a
key unaccessible directly, but that might or might not be a problem.

Well, I'd definitely consider an inaccessible key as constituting a
problem, but I don't think that's a good analogy to the list case.

With the dictionary, the change can (though I do agree it does not
have to) interfere with proper operation of the dictionary, while a
list that is no longer sorted still functions perfectly well as a
list. That is, I feel "problems" are more guaranteed with a
dictionary since we have affected base object behavior, whereas sorted
is not an inherent attribute of the base list type but something the
application is imposing at a higher level.

For example, I may choose to have an object type that is mutable (and
not worthy for use as a dictionary key) but maintains a logical
ordering so is sortable. I see no problem with sorting a list of such
objects, and then walking that list to perform some mutation to each
of the objects, even if along the way the mutation I am doing results
in the items so touched no longer being in sorted order. The act of
sorting was to provide me with a particular sequence of objects, but
aside from that fact, the list continues to perform perfectly well as
a list even after the mutations - just no longer delivering objects in
sorted order.

-- David

Jul 18 '05 #39

Bengt Richter

On 18 Jan 2005 13:28:00 GMT, Antoon Pardon <ap*****@forel.vub.ac.be> wrote:

Op 2005-01-18, Nick Coghlan schreef <nc******@iinet.net.au>:
Antoon Pardon wrote:
More specific the Decimal class is mutable and usable as dict key.
It's *meant* to be immutable though. The fact that we used __slots__ instead of
__setattr__ to implement the immutability, so you *can* overwrite the slot
variables if you really want to is merely an artifact of the current Python
implementation.

The limited mutability bug will disappear in Python 2.5, so it's not a good
example of a 'mutable' dict key (especially given that the only way to mutate it
is to modify private variables directly).

And, as I've stated previously, if the issue of sane mutable keys in
dictionaries and sets really bugs you so much - implement identity_dict and
identity_set in C and lobby for their inclusion in the collections module.

I'm not bugged by its absence in Python. I'm bugged by the attitude
about them. But anyway I'm thinking about implementing a safe dictionary
that will copy whenever it would otherwise allow the risk of mutating a
key.

I encourage you to do it. It should be a rich learning experience.
My bet is you will discover something about the real requirements of
what you are "thinking about implementing" ;-)

IANAP, but ISTM that after a certain point, trying to get made-my-point satisfaction
in a thread like this is more like OCD symptomatology than n.g. dialog ;-)

On the more general point of "don't use mutable objects with non-identity based
comparisons as dictionary keys", try teaching students for a while (or listen to
those who have):
What kind of students?

I have implemented a hash table when I was a student and its
implementation allowed the use of 'mutable' objects as a key
without a problem. It simply always made copies when appropiate
and didn't allow external access to the keys. So although the
key objects were 'mutable' there was no way a user could accidently
mutate a key.

This is word play IMO. What you describe is effectively using the mutables
to construct immutable keys for actual use, not using immutable keys. Having
the immutable (because of access restriction) constructed keys allows
you to check on their consistency with their source any time you want,
but that doesn't change the fact that you have created an alternative dictionary
implementation with immutable keys made immutable by copying and access restriction.
If you update your dictionary when a key no longer matches its source data, you
are just deleting an immutable-by-special-means key and entering the associated
value in association with a new immutable-by-special-means key.

I can produce OCD symptoms too ;-)

So don't use a mutable as a dictionary key isn't so much a dictionary
limitation in general but a specific limitation of the python implementation.

And yes I understand, the current implenatation is the result of the
fact that the same dictionaries are used internally for variables in
scopes and attributes in objects and the fact that no copies are
involved gives a boost to performance. But it could be argued that
providing these same dictionaries with those semantics to the users
was a premature optimisation.
If you see a sign like

+--------------------------------->
| WILD GOOSE CHASE THIS WAY ->
+----------------------------->
||
|| \|/
|| =o=
|| /|\
..`.||,..__|_,.

Do you feel that you must make sure?
The sign painter may only be indicating what his own experience was, after all.
But if it was a core developer's experience, it may be prudent to bet on another trail
-- unless you have an extremely interesting hunch. Then you should follow your passion
and have your own experience, and you may wind up being able to contribute something
(even if only an exclamation point on the sign ;-)

When stating useful general principles, it is never, ever worth it to get into
the quibbly little details about exceptions to the principle. If students ask,
admit that they exist, but point out that the exceptions are rare, and not worth
worrying about at that point in their learning.
But don't use mutable keys is not a general principle. It is a principle
introduced by the limitations of the python implementations.

That is so in your mind, and it is so given certain interpretations of the words
you are using, but to others "mutable keys" may be a contradiction in terms, because
they do not use the words in the same sense as you. If you can't see alternative conceptual
constructs you are stuck, and if we can't agree on the meaning of words for a given dialog,
we won't be communicating very well. The difficulty is getting people to recognize their
implicit assumptions and definitions ;-)

I don't like it when a good rule of thumb because of implementation
limitations is sold as a general principle.

People take short cuts in expressing themselves, just as you do.
E.g., you say "mutable key" when you really mean a value that will
remain constant while you are using it as a dictionary key. The "safe" dictionary
that you are "thinking about" will apparently get its "safety" by ensuring that
the "mutable key" can't be "mutated" -- or do you want to discuss which copy
is the real "key" ;-) (I don't ;-)

Regards,
Bengt Richter

Jul 18 '05 #40

Op 2005-01-18, David Bolen schreef <db**@fitlinxx.com>:

Antoon Pardon <ap*****@forel.vub.ac.be> writes:
Op 2005-01-18, Simon Brunning schreef <si************@gmail.com>:
> On 18 Jan 2005 07:51:00 GMT, Antoon Pardon <ap*****@forel.vub.ac.be> wrote:
>> 3 mutating an item in a sorted list *does* *always* cause problems
>
> No, it doesn't. It might cause the list no longer to be sorted, but
> that might or might no be a problem.
Than in the same vain I can say that mutating a key in a dictionary
doesn't always cause problems either. Sure it may probably make a
key unaccessible directly, but that might or might not be a problem.

Well, I'd definitely consider an inaccessible key as constituting a
problem, but I don't think that's a good analogy to the list case.

With the dictionary, the change can (though I do agree it does not
have to) interfere with proper operation of the dictionary, while a
list that is no longer sorted still functions perfectly well as a
list.

I think we are talking past each other. I'm talking about the concept
the programmer has in his mind. If the prgrammer has a sorted list in
his mind and an element in that list gets mutated that *does* *always*
cause problems. I'm not talking about a list in which the elements
happen to be in a particular order, but about a list that will be given
to algorithms that depend on the list being ordered.
That is, I feel "problems" are more guaranteed with a
dictionary since we have affected base object behavior, whereas sorted
is not an inherent attribute of the base list type but something the
application is imposing at a higher level.
But if the program uses one of the buildin sorts, isn't it then obvious
that this is an inherent attribute we want to impose? So is sorting
a list with mutable object not also more guaranteed to cause problems.
So shouldn't we limit the sort algorithms to containers with immutable
elements?
For example, I may choose to have an object type that is mutable (and
not worthy for use as a dictionary key) but maintains a logical
ordering so is sortable. I see no problem with sorting a list of such
objects, and then walking that list to perform some mutation to each
of the objects, even if along the way the mutation I am doing results
in the items so touched no longer being in sorted order. The act of
sorting was to provide me with a particular sequence of objects,
Personnaly I don't see what the sorting was good for in this case.
And I think such programming might be slightly dangerous if you
cooperate with other people. They may not understand the particular
nature of your mutation and rely on the fact that your list has been
sorted to conclude that it still is sorted.
but
aside from that fact, the list continues to perform perfectly well as
a list even after the mutations - just no longer delivering objects in
sorted order.

Which may be a problem for those who rely on it.

--
Antoon Pardon

Jul 18 '05 #41

Op 2005-01-18, Bengt Richter schreef <bo**@oz.net>:

On 18 Jan 2005 13:28:00 GMT, Antoon Pardon <ap*****@forel.vub.ac.be> wrote:
I have implemented a hash table when I was a student and its
implementation allowed the use of 'mutable' objects as a key
without a problem. It simply always made copies when appropiate
and didn't allow external access to the keys. So although the
key objects were 'mutable' there was no way a user could accidently
mutate a key. This is word play IMO. What you describe is effectively using the mutables
to construct immutable keys for actual use, not using immutable keys.

No it is not wordplay. There is a difference between inaccessible and
immutable. The difference is that the user can mutate his own objects
which he wants to use as a key, which wouldn't be the case if he was
using immutables as keys.
Having
the immutable (because of access restriction) constructed keys allows
you to check on their consistency with their source any time you want,
but that doesn't change the fact that you have created an alternative dictionary
implementation with immutable keys made immutable by copying and access restriction.
If you update your dictionary when a key no longer matches its source data, you
are just deleting an immutable-by-special-means key and entering the associated
value in association with a new immutable-by-special-means key.
The immutable vs mutable as keys IMO refers to the class the key belongs
to. Not the accessibiliy of individual instances.

So don't use a mutable as a dictionary key isn't so much a dictionary
limitation in general but a specific limitation of the python implementation.

And yes I understand, the current implenatation is the result of the
fact that the same dictionaries are used internally for variables in
scopes and attributes in objects and the fact that no copies are
involved gives a boost to performance. But it could be argued that
providing these same dictionaries with those semantics to the users
was a premature optimisation.

If you see a sign like

+--------------------------------->
| WILD GOOSE CHASE THIS WAY ->
+----------------------------->
||
|| \|/
|| =o=
|| /|\
.`.||,..__|_,.

Do you feel that you must make sure?

Talk to my wife. She regularly complains that I don't accept
her word often enough on what she says. :-)
The sign painter may only be indicating what his own experience was, after all.
But if it was a core developer's experience, it may be prudent to bet on another trail
-- unless you have an extremely interesting hunch. Then you should follow your passion
and have your own experience, and you may wind up being able to contribute something
(even if only an exclamation point on the sign ;-)
When stating useful general principles, it is never, ever worth it to get into
the quibbly little details about exceptions to the principle. If students ask,
admit that they exist, but point out that the exceptions are rare, and not worth
worrying about at that point in their learning.
But don't use mutable keys is not a general principle. It is a principle
introduced by the limitations of the python implementations.

That is so in your mind, and it is so given certain interpretations of the words
you are using,

As far as I understand I using them in a standard python interpretation.
but to others "mutable keys" may be a contradiction in terms, because
they do not use the words in the same sense as you.
I don't think so. I have seen defenders of only immutables as keys, use
the argument that using mutables as keys would require a copy of the key.
Therefore using only immutables allows for optimisations in the implementation.
I saw nothing in there discourse that implied they saw this copy as
immutable.
If you can't see alternative conceptual
constructs you are stuck, and if we can't agree on the meaning of words for a given dialog,
we won't be communicating very well. The difficulty is getting people to recognize their
implicit assumptions and definitions ;-)

I can be wrong, but until now I have seen no indication that I was
using mutable and immutable differently than other people. AFAICT
we all refer to whether an object belongs to a mutable or immutable
class.

I don't like it when a good rule of thumb because of implementation
limitations is sold as a general principle.

People take short cuts in expressing themselves, just as you do.
E.g., you say "mutable key" when you really mean a value that will
remain constant while you are using it as a dictionary key.

What I mean is a key that belongs to a mutable class. Whether I
actually mutate it in certain circumstances or not doesn't change
that.

I also don't want my values to change when I have sorted a list
and still need to apply a number of algorithms that rely on
that. Nobody seems to have problems with the possibility that
the list items are mutable (and called such).

--
Antoon Pardon

Jul 18 '05 #42

Nick Coghlan

Antoon Pardon wrote:

A rule of thumb is context sensitive. If circumstances change,
so do the rules of thumb. Principles have a broader field
of application.

IMO there is nothing principally wrong with using a mutable object
as a dictionary key. But avoiding doing so is a good rule of
thumb if you have a python-like implementation of a dictionary.

For a 'mutable key' to make sense, the following:

lst = []
dct = {l: "Hi!"}
print dct[[]]
print dct[lst]
lst.append(1)
print dct[[1]]
print dct[lst]

Should print:
Hi
Hi
Hi
Hi

That's completely impractical though - for any sane implementation, at least one
of the above print statements will throw a KeyError.

Your example of a 'safe_dict' that copies mutable keys means that the final
print statement is the one that fails.

My suggestion of an "identity_dict" means that both of the same-value based
lookups would fail.

Notice that both of these approaches take a mutable key and make it immutable
(either by holding the sole reference to a copy, or retrieving an immutable
property of the mutable object).

There's also your solution of "I promise not to mutate the key while it is in
the dictionary". Workable, but opens the door to some entertaining bug hunts
when the promise is broken.

Which solution is the best default behaviour?

Well, the Zen of Python states "In the face of ambiguity, refuse the temptation
to guess".

So that's the policy the builtin dict follows - it doesn't try to guess when to
make a copy, or whether or not to use identity based semantics in the face of
mutability. Instead, it raises an exception at key entry time, asking the
programmer to clarify their intent.

The principle is *certainly* that hash tables should contain immutable keys.
Your own 'safe_dict' example implicitly concedes this point by making copies
'when required'. 'When required' for what? When required to preserve
immutability, perhaps?

In short, sane hash tables require immutable keys, and how mutable keys acquire
the requisite immutability is going to be application dependent.

Provision of a safe_dict or identity_dict would merely allow a programmer to
state their intent at the time the container is created, rather than having to
state it whenever keys are generated or referenced.

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

Jul 18 '05 #43

Op 2005-01-19, Nick Coghlan schreef <nc******@iinet.net.au>:

Antoon Pardon wrote:
A rule of thumb is context sensitive. If circumstances change,
so do the rules of thumb. Principles have a broader field
of application.

IMO there is nothing principally wrong with using a mutable object
as a dictionary key. But avoiding doing so is a good rule of
thumb if you have a python-like implementation of a dictionary.
For a 'mutable key' to make sense, the following:

As you said yourself, people use shortcuts when they express themselves.
With 'mutable key' I mean a mutable object that is used as a key.
That doesn't contradict that the key itself can't change because
it is inaccessible.
lst = []
dct = {l: "Hi!"}
print dct[[]]
print dct[lst]
lst.append(1)
print dct[[1]]
print dct[lst]

Should print:
Hi
Hi
Hi
Hi That's completely impractical though - for any sane implementation, at least one
of the above print statements will throw a KeyError.

Your example of a 'safe_dict' that copies mutable keys means that the final
print statement is the one that fails.

My suggestion of an "identity_dict" means that both of the same-value based
lookups would fail.

Notice that both of these approaches take a mutable key and make it immutable
(either by holding the sole reference to a copy, or retrieving an immutable
property of the mutable object).

There's also your solution of "I promise not to mutate the key while it is in
the dictionary". Workable, but opens the door to some entertaining bug hunts
when the promise is broken.
That is usually the case when promisses are broken. To mention something
different, I guess a number of equally entertaining bug hunts can be
caused by the fact that the value in the dictionary is changed.
Which solution is the best default behaviour?
IMO a dictionary establisches a relationship between values. The most
general implementation that protects this relationship from external
tampering is copying the key and value for use in the dictionary.
Well, the Zen of Python states "In the face of ambiguity, refuse the temptation
to guess".
But python does guess in the case of the value. If I want to establish a
relationship between "foo" and [3, 5, 8] then I can be in big trouble
if that list gets mutated in something else.
So that's the policy the builtin dict follows - it doesn't try to guess when to
make a copy, or whether or not to use identity based semantics in the face of
mutability. Instead, it raises an exception at key entry time, asking the
programmer to clarify their intent.

The principle is *certainly* that hash tables should contain immutable keys.
Your own 'safe_dict' example implicitly concedes this point by making copies
'when required'. 'When required' for what? When required to preserve
immutability, perhaps?
But they don't have to be immutable within the dictionary itself, that
is just an implementation detail. The only thing that matters is that
the dictionary can reproduce the relationship. How that is done is
immaterial.
In short, sane hash tables require immutable keys, and how mutable keys acquire
the requisite immutability is going to be application dependent.
No they don't, that is just the most easy implementation. If some
implementation would constantly change the internal objects but
consulting the dictionary wouldn't show any different effect then
that would be fine too. And even in this case a mutation
from the outside would probably cause havoc because it would ruin
the sanity of the internal structure.
Provision of a safe_dict or identity_dict would merely allow a programmer to
state their intent at the time the container is created, rather than having to
state it whenever keys are generated or referenced.

Well that would be a good thing IMO. Good enough in any case for me to
spend some time analyzing the requirements. Whether I'll actually
implement something depends on how much time I have and how often I
need it.

--
Antoon Pardon

Jul 18 '05 #44

Just

In article <ma**************************************@python.o rg>,
Nick Coghlan <nc******@iinet.net.au> wrote:

Antoon Pardon wrote:
A rule of thumb is context sensitive. If circumstances change,
so do the rules of thumb. Principles have a broader field
of application.

IMO there is nothing principally wrong with using a mutable object
as a dictionary key. But avoiding doing so is a good rule of
thumb if you have a python-like implementation of a dictionary.

For a 'mutable key' to make sense, the following:

lst = []
dct = {l: "Hi!"}
print dct[[]]
print dct[lst]
lst.append(1)
print dct[[1]]
print dct[lst]

Should print:
Hi
Hi
Hi
Hi

That's completely impractical though - for any sane implementation, at least
one
of the above print statements will throw a KeyError.

Your example of a 'safe_dict' that copies mutable keys means that the final
print statement is the one that fails.

My suggestion of an "identity_dict" means that both of the same-value based
lookups would fail.

Notice that both of these approaches take a mutable key and make it immutable
(either by holding the sole reference to a copy, or retrieving an immutable
property of the mutable object).

There's also your solution of "I promise not to mutate the key while it is in
the dictionary". Workable, but opens the door to some entertaining bug hunts
when the promise is broken.

Which solution is the best default behaviour?

Well, the Zen of Python states "In the face of ambiguity, refuse the
temptation
to guess".

So that's the policy the builtin dict follows - it doesn't try to guess when
to
make a copy, or whether or not to use identity based semantics in the face of
mutability. Instead, it raises an exception at key entry time, asking the
programmer to clarify their intent.

The principle is *certainly* that hash tables should contain immutable keys.
Your own 'safe_dict' example implicitly concedes this point by making copies
'when required'. 'When required' for what? When required to preserve
immutability, perhaps?

In short, sane hash tables require immutable keys, and how mutable keys
acquire
the requisite immutability is going to be application dependent.

Provision of a safe_dict or identity_dict would merely allow a programmer to
state their intent at the time the container is created, rather than having
to
state it whenever keys are generated or referenced.

FWIW, the Cocoa framework on OSX supports mutable objects as keys.
However, it makes an immutable copy. This is cheap for immutable objects
-- Cocoa has both a mutable array as well as an immutable array,
likewise for dicts -- since the "copy" will then simply be the same
object. Thanks to PyObjC we can explore this from Python:

Python 2.3 (#1, Sep 13 2003, 00:49:11)
[GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

from Foundation import *
lst = []
dct = NSMutableDictionary.alloc().init()
dct {} dct[lst] = "Hi!"
dct {(1) = "Hi!"; }

(Note that Python lists get wrapped as native Cocoa arrays. '(1)' is
Cocoa's repr for the wrapped array.)
dct[lst] u'Hi!' lst.append(1)
dct[lst] Traceback (most recent call last):
File "<stdin>", line 1, in ?
File
"build/lib.darwin-7.6.0-Power_Macintosh-2.3/objc/_convenience.py", line
77, in __getitem__objectForKey
KeyError: [1, 1] dct.keys() ((1)) dct[[1]] u'Hi!' k = dct.keys()[0]
k (1) k.append(2) Traceback (most recent call last):
File "<stdin>", line 1, in ?
File
"build/lib.darwin-7.6.0-Power_Macintosh-2.3/objc/_convenience.py", line
205, in <lambda>
objc.error: NSInternalInconsistencyException - *** -[NSCFArray
addObject:]: mutating method sent to immutable object

This is not all that different from Python actually, in that Cocoa tries
very hard to make it impossible to mutate dict keys.

Just

Jul 18 '05 #45

Nick Coghlan

Antoon Pardon wrote:

I can be wrong, but until now I have seen no indication that I was
using mutable and immutable differently than other people. AFAICT
we all refer to whether an object belongs to a mutable or immutable
class.
The difference is that when you take a copy of the key and stick it in the
dictionary, such that the dictionary now holds the *sole* reference to that key,
you have made that key *effectively* immutable.

This is why no-one really batted an eyelid when you mentioned that mutable keys
can be used safely by making a copy - doing so makes the key *effectively*
immutable, and means that modifying the original key object (i.e. the
application's copy, not the dict's copy), and reusing it to look up in the
dictionary is likely to give a KeyError.

These semantics would be understandably surprising to many users, and hence, are
not supplied by default.

Additionally, a dictionary's keys are accessible via its API. Accordingly, to
preserve this 'effective immutability', making a copy on key input is
insufficient - keys must be copied on *output* as well (that is, dict.keys,
dict.items etc must return *copies* of the key objects, not the key objects
themselves).

Since there is no reliable way in Python to tell if an object is mutable or not
(the closest equivalent is the presence of __hash__, which clearly can't be used
in this example), this copying would need to be done for *every* object.

Alternately, the dictionary can say to the API clients, "make an immutable copy
and supply that instead. It is left to API clients to decide how best to make
the immutable version". The API client copies the key once (to make the
immutable version), and everyone lives happily ever after.

For example:

Py> class mylist(list):
.... def __init__(self, arg):
.... super(mylist, self).__init__(arg)
.... self._tuple = None
.... def frozen(self):
.... if self._tuple is None:
.... self._tuple = tuple(self)
.... return self._tuple
.... def unfreeze(self):
.... self._tuple = None
....
Py> x = mylist(range(10))
Py> x
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Py> x.frozen()
(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
Py> x.append(10)
Py> x.frozen()
(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
Py> x.unfreeze()
Py> x.frozen()
(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

This is much safer than having a list subclass that implements __hash__, and
still lets you avoid redundant copy operations.

Nicely, so long as you don't unfreeze the object you used to key the dictionary,
reusing the same object will always get you the correct dictionary entry, and
two lists that compare equal at the time they are frozen will also get you the
same dictionary entry. The equivalent tuples can be used for the lookup, too.
I also don't want my values to change when I have sorted a list
and still need to apply a number of algorithms that rely on
that. Nobody seems to have problems with the possibility that
the list items are mutable (and called such).

OK, to make this comparison of sorted lists and dictionaries fair:

Write a sorted_list class that is like a regular Python list, but maintains as
an invariant that the list contents will stay sorted.

See how well you go maintaining that invariant while allowing mutable objects in
the list. The only way would be to copy them in when they're supplied, and copy
them out again when you're done. Otherwise, there is no way the class can keep
its promise. The performance will be lousy, since __setitem__ and __getitem__
will be making copies all the time.

Alternatively, the class could declare itself to work reliably only with
immutable objects. Performance will improve, since copies need only be made when
an object *actually* changes (and the old immutable copy is deleted and the new
version inserted in its place).

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

Jul 18 '05 #46

Op 2005-01-19, Nick Coghlan schreef <nc******@iinet.net.au>:

Antoon Pardon wrote:
> I can be wrong, but until now I have seen no indication that I was
using mutable and immutable differently than other people. AFAICT
we all refer to whether an object belongs to a mutable or immutable
class.
The difference is that when you take a copy of the key and stick it in the
dictionary, such that the dictionary now holds the *sole* reference to that key,
you have made that key *effectively* immutable.

This is why no-one really batted an eyelid when you mentioned that mutable keys
can be used safely by making a copy - doing so makes the key *effectively*
immutable, and means that modifying the original key object (i.e. the
application's copy, not the dict's copy), and reusing it to look up in the
dictionary is likely to give a KeyError.

These semantics would be understandably surprising to many users, and hence, are
not supplied by default.

This seems like circle reasoning. The reason why these semantics would
be surprising is that those are not the semantics supplied.
Additionally, a dictionary's keys are accessible via its API. Accordingly, to
preserve this 'effective immutability', making a copy on key input is
insufficient - keys must be copied on *output* as well (that is, dict.keys,
dict.items etc must return *copies* of the key objects, not the key objects
themselves).
Yes I haven been thinking about this too. I also think not making a copy
on output is less dangerous than not making a copy on input.
Since there is no reliable way in Python to tell if an object is mutable or not
(the closest equivalent is the presence of __hash__, which clearly can't be used
in this example), this copying would need to be done for *every* object.

Alternately, the dictionary can say to the API clients, "make an immutable copy
and supply that instead. It is left to API clients to decide how best to make
the immutable version". The API client copies the key once (to make the
immutable version), and everyone lives happily ever after.

For example:

Py> class mylist(list):
... def __init__(self, arg):
... super(mylist, self).__init__(arg)
... self._tuple = None
... def frozen(self):
... if self._tuple is None:
... self._tuple = tuple(self)
... return self._tuple
... def unfreeze(self):
... self._tuple = None
...
Py> x = mylist(range(10))
Py> x
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Py> x.frozen()
(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
Py> x.append(10)
Py> x.frozen()
(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
Py> x.unfreeze()
Py> x.frozen()
(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

This is much safer than having a list subclass that implements __hash__, and
still lets you avoid redundant copy operations.

Nicely, so long as you don't unfreeze the object you used to key the dictionary,
reusing the same object will always get you the correct dictionary entry, and
two lists that compare equal at the time they are frozen will also get you the
same dictionary entry. The equivalent tuples can be used for the lookup, too.

Interesting idea. But I think you are wrong when you say that two lists
that compare equal at the time they are frozen, will get the same
dictionary entry. The problem is an object must compare equal to
the key in the dictionary to get at the same entry. So if you freeze
a list and its copy but then mutate them differently, they no longer
are equal and so wont get you at the same entry.

[ Rest snipped, this is getting into a semantic debate on what
'mutable' means. This may be interesting enough on its own
but I feel it detracts too much from what I consider the
more important issue. ]

--
Antoon Pardon

Jul 18 '05 #47

Nick Coghlan wrote:

For a 'mutable key' to make sense, the following:

lst = []
dct = {l: "Hi!"}
print dct[[]]
print dct[lst]
lst.append(1)
print dct[[1]]
print dct[lst]

Should print:
Hi
Hi
Hi
Hi

And here's an implementation that does so:

py> class sillydict(dict):
.... def __getitem__(self, key):
.... items = self.items()
.... self.clear()
.... self.update(items)
.... return super(sillydict, self).__getitem__(key)
....
py> class sillylist(list):
.... def __hash__(self):
.... return hash(tuple(self))
....
py> lst = sillylist([])
py> dct = sillydict({lst: "Hi!"})
py> print dct[sillylist([])]
Hi!
py> print dct[lst]
Hi!
py> lst.append(1)
py> print dct[sillylist([1])]
Hi!
py> print dct[lst]
Hi!

Of course, as noted by the classes themselves, they're silly. ;) But as
long as you don't mind rehashing the dict each time the dict is accessed
;) you can get sane behavior even in the presence of mutable keys.

Steve

Jul 18 '05 #48

David Eppstein

In article <ma**************************************@python.o rg>,
Nick Coghlan <nc******@iinet.net.au> wrote:

For a 'mutable key' to make sense, the following:

lst = []
dct = {l: "Hi!"}
print dct[[]]
print dct[lst]
lst.append(1)
print dct[[1]]
print dct[lst]

Should print:
Hi
Hi
Hi
Hi

Yes, and what should the following do?

lst1 = [1]
lst2 = [2]
dct = {lst1: "1", lst2: "2"}
lst2[0]=1
lst1[0]=2
print dct[[1]]
print dct[[2]]

--
David Eppstein
Computer Science Dept., Univ. of California, Irvine
http://www.ics.uci.edu/~eppstein/

Jul 18 '05 #49