magical expanding hash

[braver]

I need a magical expanding hash with the following properties:

* it creates all intermediate keys

meh['foo']['bar] = 1

-- works even if meh['foo'] didn't exist before

* allows pushing new elements to leaves which are arrays

meh['foo']['list] << elem1
meh['foo']['list] << elem2

* allows incrementing numeric leaves

meh['foo']['count'] += 7

* serializable

I have such a class in ruby. Can python do that?

[braver]

Well, I know some python, but since there are powerful and magical
features in it, I just wonder whether there're some which address this
issue better than others.

[James Stroud]

braver wrote:

Well, I know some python, but since there are powerful and magical
features in it, I just wonder whether there're some which address this
issue better than others.

In python, += is short, of course, for

a = a + 1

But if we haven't already assigned a, how does the interpreter know that
we want an int, float, complex, long, or some other data-type that
defines "+"?

Better, clearer or more pythonic would be:

a = 0.0 # we want a float, compiler didn't have to read mind
b = 0 # now we want an int, saving compiler lots of guesswork
a += 1 # incrementing a float by one

The "<<" operator corresponds to the __lshift__ magic method. You can
make a custom data-type here:

class lshiftinglist(l ist):
def __lshift__(self , value):
list.append(sel f, value)

class meh(dict):
def __getitem__(sel f, item):
return dict.setdefault (self, item, meh())

m = meh()
m['bob']['carol'] = 1
m['bob']['carol'] += 1
m['bob']['ted'] = lshiftinglist()
m['bob']['ted'] << 42
m['bob']['ted'] << 43

print m # {'bob': {'carol': 2, 'ted': [42, 43]}}
Other than magically reading mind of programmer, this works pretty much
according to specification.

If you really want a lot of mindreading abilities, you have to write
your own mindreading code. Here is a tiny example:

class meh(dict):
def __getitem__(sel f, item):
return dict.setdefault (self, item, meh())
def __getattr__(sel f, attr):
return self.ga(attr)
def __lshift__(self , value):
print "You are thinking of '%s'." % value
def __iadd__(self, other):
# don't try this on a populated meh!!!!!
return other

m = meh()

# mindreading way
m['carol'] += 4
m['carol'] += 5
m['bob'] << 44 # "You are thinking of '44'."

# better, not mindreading way
m['alice'] = [10]
m['alice'].append(11)
m['ted'] = 18
m['ted'] += 1

print m # "{'carol': 9, 'ted': 19, 'bob': {}, 'alice': [10, 11]}"
It would take a lot of coding to make that << work right. Better is the
pythonic

m[key] = [value]

Its really only one more keystroke than

m[key] << value
James

[Steven Bethard]

Steve Holden wrote:

Steven Bethard wrote:

Agreed. I really hope that Python 3.0 applies Raymond Hettinger's
suggestion "Improved default value logic for Dictionaries" from
http://wiki.python.org/moin/Python3%2e0Suggestions

This would allow you to make the setdefault() call only once, instead
of on every lookup:

class meh(dict):
def __init__(self, *args, **kwargs):
super(meh, self).__init__( *args, **kwargs)
self.setdefault (function=meh)

STeVe

In fact, why not go one better and also add a "default" keyword
parameter to dict()?

It's not backwards compatible:

dict(default=4)

{'default': 4}

And I use the **kwargs form of the dict constructor often enough to hope
that it doesn't go away in Python 3.0.

STeVe

[braver]

Thanks, James! This is really helpful.

: It would take a lot of coding to make that << work right. Better is
the pythonic
:
: m[key] = [value]
:
: Its really only one more keystroke than
:
: m[key] << value

But it's only for the first element, right? I'd have to say
meh[key1]...[keyN].append(elem2) after that, while I want an operator
to look the same.

Also, what's the shortest python idiom for get_or_set in expression?
E.g., when creating a numeric leaf, I'd say

if meh.has_key('a' ): meh['a'] += 7
else: meh['a'] = 7

-- but I'd have to do it everywhere! That's why I'd like to override
+= to do the check/init for me.

[James Stroud]

braver wrote:

Thanks, James! This is really helpful.

: It would take a lot of coding to make that << work right. Better is
the pythonic
:
: m[key] = [value]
:
: Its really only one more keystroke than
:
: m[key] << value

But it's only for the first element, right? I'd have to say
meh[key1]...[keyN].append(elem2) after that, while I want an operator
to look the same.

Yes, being explicit is only for the first element with the "<<". If you
use the lshiftinglist I provided, you could easily do
class lshiftinglist(l ist):
def __lshift__(self , value):
list.append(sel f, value)

class meh(dict):
def __getitem__(sel f, item):
return dict.setdefault (self, item, meh())
def __getattr__(sel f, attr):
return self.ga(attr)
def __lshift__(self , value):
print "You are thinking of '%s'." % value
def __iadd__(self, other):
# don't try this on a populated meh!!!!!
return other

m = meh()

m['fred'] = lshiftinglist([18])
m['fred'] << 25
m['barney'] += 1
m['barney'] += 1

print m # {'barney': 2, 'fred': [18, 25]}
And so-on. More pythonic, of course is

m['fred'] = [18]
m['key'].append(25)
m['barney'] = 1
m['barney'] += 1
Now the reason "m['barney'] += 1" works in the former is becasue "+="
actually returns a value to which the name on the left gets re-assigned.
"<<" does not work this way, so it can't be done as easily.

You might want to make a named method that thinks for you. The resulting
code is less terse but more clear (i.e. pythonic):
def meh_append(ameh , key, value):
if not ameh.has_key(ke y):
ameh[key] = [value]
else:
ameh[key].append(value)

def meh_addleaf(ame h, key, value={}):
if value == {}:
ameh[key] = {}
else:
ameh[key] = value

m = meh()

meh_addleaf(m['bob'], 'carol', None)
meh_append(m['ted'], 'alice', 14)
meh_append(m, 'fred', 1)
meh_append(m, 'fred', 2)

print m # {'bob': {'carol': None},
# 'ted': {'alice': [14]}, 'fred': [1, 2]}

But now its getting very pythonic. And the only magic we need is the
original __getattr__ modification, which we could find a way of
eliminating if we tried.

James

[Steve Holden]

Steven Bethard wrote:

Steve Holden wrote:

Steven Bethard wrote:

Agreed. I really hope that Python 3.0 applies Raymond Hettinger's
suggestion "Improved default value logic for Dictionaries" from
http://wiki.python.org/moin/Python3%2e0Suggestions

This would allow you to make the setdefault() call only once, instead
of on every lookup:

class meh(dict):
def __init__(self, *args, **kwargs):
super(meh, self).__init__( *args, **kwargs)
self.setdefault (function=meh)

STeVe

In fact, why not go one better and also add a "default" keyword
parameter to dict()?

It's not backwards compatible:

>>> dict(default=4)

{'default': 4}

And I use the **kwargs form of the dict constructor often enough to hope
that it doesn't go away in Python 3.0.

Nyargle. Thanks, you're quite right, of course: I was focussing on the
list-of-pairs argument style when I wrote that. So the best we could do
is provide a subtype, defaultdict(def ault, *args, *kw).

It still seems to me that would be better than having to call a method
(though I don't object to the method for use if the defaut must change
dynamically). Maybe I just liked Icon tables too much.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

[Steven D'Aprano]

On Tue, 17 Jan 2006 18:00:00 -0700, Steven Bethard wrote:

Steve Holden wrote:

Steven Bethard wrote:

Agreed. I really hope that Python 3.0 applies Raymond Hettinger's
suggestion "Improved default value logic for Dictionaries" from
http://wiki.python.org/moin/Python3%2e0Suggestions

This would allow you to make the setdefault() call only once, instead
of on every lookup:

class meh(dict):
def __init__(self, *args, **kwargs):
super(meh, self).__init__( *args, **kwargs)
self.setdefault (function=meh)

STeVe

In fact, why not go one better and also add a "default" keyword
parameter to dict()?

It's not backwards compatible:

>>> dict(default=4)

{'default': 4}

And I use the **kwargs form of the dict constructor often enough to hope
that it doesn't go away in Python 3.0.

I don't like the idea of all dicts having default values. Sometimes you
don't want a default value, you want an exception when the key isn't in
the dict.

And even if you do want defaults, sometimes you want a default which is
global to the dict, and sometimes you want a default which depends on the
key. More of a "missing value" than a default.

I vote to leave dict just as it is, and add a subclass, either in a module
or as a built in (I'm not fussed either way) for dicts-with-defaults.

--
Steven.

[Giovanni Bajo]

braver wrote:

Also, what's the shortest python idiom for get_or_set in expression?

dict.setdefault , as I already explained to you.

Again, I'd like to point out that what you're doing is *not* the correct
Pythonic way of doing things. In Python, there is simply no implicit
sub-dicts creation, nor implicit type inference from operators. And there
are very good reason for that. Python is a strongly typed languages: objects
have a type and keep it, they don't change it when used with different
operators. setdefault() is you get'n'set, everything else has to be made
explicit for a good reason. Strong typing has its virtues, let me give you a
link about this:

http://wingware.com/python/success/astra
See specifically the paragraph "Python's Error Handling Improves Robustness"

I believe you're attacking the problem from a very bad point of view.
Instead of trying to write a Python data structure which behaves like
Perl's, convert a Perl code snippet into Python, using the *Pythonic* way of
doing it, and then compare things. Don't try to write Perl in Python, just
write Python and then compare the differences.
--
Giovanni Bajo

[braver]

Giovanni Bajo wrote,

dict.setdefault , as I already explained to you.

I wonder about numerics too. Say we have a = None somewhere.

I want to increment it, so I'd say a += 8. Now if this is a parsing
app, the increment may happen everywhere -- so I'd write a function to
do it if I worry about it being initialized before. Is there any
operator/expression level support for what in ruby looks like

a ||= 0