How can I make a dictionary that marks itself when it's modified?

[sandravandale]

It's important that I can read the contents of the dict without
flagging it as modified, but I want it to set the flag the moment I add
a new element or alter an existing one (the values in the dict are
mutable), this is what makes it difficult. Because the values are
mutable I don't think you can tell the difference between a read and a
write without making some sort of wrapper around them.

Still, I'd love to hear how you guys would do it.

Thanks,
-Sandra

[garabik-news-2005-05]

sa***********@yahoo.com wrote:

It's important that I can read the contents of the dict without
flagging it as modified, but I want it to set the flag the moment I add
a new element or alter an existing one (the values in the dict are
mutable), this is what makes it difficult. Because the values are
mutable I don't think you can tell the difference between a read and a
write without making some sort of wrapper around them.

Still, I'd love to hear how you guys would do it.

if the dictionary is small and speed not important, you can wrap it
in a class catching __getitem__ and __setitem__ and testing
if repr(self) changes.
--
-----------------------------------------------------------
| Radovan GarabÃ*k http://kassiopeia.juls.savba.sk/~garabik/ |
| __..--^^^--..__ garabik @ kassiopeia.juls.savba.sk |
-----------------------------------------------------------
Antivirus alert: file .signature infected by signature virus.
Hi! I'm a signature virus! Copy me into your signature file to help me spread!

[Steve Holden]

ga******************@kassiopeia.juls.savba.sk wrote:

sa***********@yahoo.com wrote:

It's important that I can read the contents of the dict without
flagging it as modified, but I want it to set the flag the moment I add
a new element or alter an existing one (the values in the dict are
mutable), this is what makes it difficult. Because the values are
mutable I don't think you can tell the difference between a read and a
write without making some sort of wrapper around them.

Still, I'd love to hear how you guys would do it.

if the dictionary is small and speed not important, you can wrap it
in a class catching __getitem__ and __setitem__ and testing
if repr(self) changes.

d = {1: [a, b],
2: [b, c]}

d[1][0] = 3

How would this work? __getitem__() will be called once, to get a
reference to the list, and so there's no opportunity to compare the
'before' and 'after' repr() values.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

[Michael Spencer]

sa***********@yahoo.com wrote:

It's important that I can read the contents of the dict without
flagging it as modified, but I want it to set the flag the moment I add
a new element or alter an existing one (the values in the dict are
mutable), this is what makes it difficult. Because the values are
mutable I don't think you can tell the difference between a read and a
write without making some sort of wrapper around them.

Still, I'd love to hear how you guys would do it.

Thanks,
-Sandra

Sandra

You appear to want:

D = {"a":obj1, "b":obj2}

when obj1 or obj2 changes, to notify D

I think that's hard to do in the general case (i.e., for any value of obj). You
will have to design/modify obj1 and obj2 or wrap them it to achieve this. If
the objects are not homogeneous, I'd give up ;-)

If you can design obj for the purpose it shouldn't be do hard, using the
observer pattern:

Here's a sketch:

class Observable(object):
"""An object that notifies observers when it is changed"""
def __init__(self):
self._observers = []
def add_observer(self, observer):
self._observers.append(observer)
def _notify(self, attrname, newval):
for observer in self._observers:
observer.notify(self, attrname, newval)
def __setattr__(self, attrname, val):
object.__setattr__(self, attrname, val)
self._notify(attrname, val)

class DictObserver(dict):
"""A dict that observes Observable instances"""
def __setitem__(self, key, value):
assert isinstance(value, Observable)
dict.__setitem__(self, key, value)
value.add_observer(self)
def notify(self, obj, attr, newval):
print "%s.%s = %s" % (obj, attr, newval)

a = Observable()
b = Observable()
D = DictObserver()
D["a"] = a
D["b"] = b
a.new_attribute = 4 <observer_dict.Observable object at 0x01AE9210>.new_attribute = 4 b.new_attribute = 6 <observer_dict.Observable object at 0x01AE98F0>.new_attribute = 6 a.new_attribute = 5 <observer_dict.Observable object at 0x01AE9210>.new_attribute = 5

Michael

[Bengt Richter]

On Thu, 12 Jan 2006 18:28:54 +0000, Steve Holden <st***@holdenweb.com> wrote:

ga******************@kassiopeia.juls.savba.sk wrote:

sa***********@yahoo.com wrote:

It's important that I can read the contents of the dict without
flagging it as modified, but I want it to set the flag the moment I add
a new element or alter an existing one (the values in the dict are
mutable), this is what makes it difficult. Because the values are
mutable I don't think you can tell the difference between a read and a
write without making some sort of wrapper around them.

Still, I'd love to hear how you guys would do it.

if the dictionary is small and speed not important, you can wrap it
in a class catching __getitem__ and __setitem__ and testing
if repr(self) changes.

d = {1: [a, b],
2: [b, c]}

d[1][0] = 3

How would this work? __getitem__() will be called once, to get a
reference to the list, and so there's no opportunity to compare the
'before' and 'after' repr() values.

You are right, but OTOH the OP speaks of a "flagging" the dict as modified.
If she made e.g., "modified" a property of the dict subclass, then
retrieving the the "modified" "flag" could dynamically check current state repr
vs some prior state repr. Then the question becomes "modified w.r.t. what prior state?"

This lets the OP ask at any time whether the dict is/has_been modified, but it's not a basis
for e.g., a modification-event callback registry or such.

Regards,
Bengt Richter

[Christian Tismer]

Steve Holden wrote:

ga******************@kassiopeia.juls.savba.sk wrote:

sa***********@yahoo.com wrote:

It's important that I can read the contents of the dict without
flagging it as modified, but I want it to set the flag the moment I add
a new element or alter an existing one (the values in the dict are
mutable), this is what makes it difficult. Because the values are
mutable I don't think you can tell the difference between a read and a
write without making some sort of wrapper around them.

Still, I'd love to hear how you guys would do it.
if the dictionary is small and speed not important, you can wrap it
in a class catching __getitem__ and __setitem__ and testing
if repr(self) changes.

IMHO, that's too much.
d = {1: [a, b],
2: [b, c]}

d[1][0] = 3

How would this work? __getitem__() will be called once, to get a
reference to the list, and so there's no opportunity to compare the
'before' and 'after' repr() values.

I think tracking whether a dict gets modified should in fact only
trace changes to the dict, not to elements contained in the dict.
So __repr__ is probably too much, and also not the intent.

I'd just overwrite __setitem__ for write access to the dict.
Enforcing tracking of all contents is hard (as you showed above)
and probably not really needed.

If further tracking is reasonable, then one would continue and
patch contained lists as well.

my 0.2 € - chris

--
Christian Tismer :^) <mailto:ti****@stackless.com>
tismerysoft GmbH : Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/
14109 Berlin : PGP key -> http://wwwkeys.pgp.net/
work [phone] mobile [phone] fax [phone]
PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04
whom do you want to sponsor today? http://www.stackless.com/

[Christian Tismer]

Just to add a word that I forgot:

Adhering to the subject line, the intent is to track modifications
of a dict.
By definition, modification of a member of a dict without replacing
the value is not considered a dict change.

I'd stick with the shallow approach.
Asking to track mutation of an element in the general case
is causing much trouble.
Support for element tracking can probably provided by overriding
the dict's getattr and recording the element in some extra
candidate list.
If the element itself is modified, it then could be looked up
as a member of that dict, given that the element's setattr
is traced, too.

ciao - chris

--
Christian Tismer :^) <mailto:ti****@stackless.com>
tismerysoft GmbH : Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/
14109 Berlin : PGP key -> http://wwwkeys.pgp.net/
work [phone] mobile [phone] fax [phone]
PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04
whom do you want to sponsor today? http://www.stackless.com/

[Steve Holden]

Christian Tismer wrote:

Just to add a word that I forgot:

Adhering to the subject line, the intent is to track modifications
of a dict.
By definition, modification of a member of a dict without replacing
the value is not considered a dict change.
Well, I agree. But I suppose much depends on exactly what the OP meant
by "... add a new element or alter an existing one". The post did follow
that with "(the values in the dict are mutable)", which is presumably
why garabik-2500 proposed catching __getitem__ as well as __setitem__.

I merely wanted to point out (not to you!) that there was no effective
way to capture a change to a mutable item without, as you say, modifying
the element classes.
I'd stick with the shallow approach.
Asking to track mutation of an element in the general case
is causing much trouble.
Support for element tracking can probably provided by overriding
the dict's getattr and recording the element in some extra
candidate list.
If the element itself is modified, it then could be looked up
as a member of that dict, given that the element's setattr
is traced, too.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

[sandravandale]

Thanks to everyone who posted comments or put some thought into this
problem.

I should have been more specific with what I want to do, from your
comments the general case of this problem, while I hate to say
impossible, is way more trouble than it's worth.

By modified I meant that the dictionary has been changed from its
original (constructed) state by either:
1) A new element was added
2) An existing element was changed

I want to wrap a dictionary of cookies in mod_python. Cookies are
represented as objects with no methods. In light of this and Michael's
excellent post, I came up with the following code.

class CookieCollection(dict):
def __init__(self, d):
for k, v in d:
v.__setattr__ = self.__wrap_setattr(v.__setattr__)
self[k] = v

self.modified = False

def __setitem__(self, key, value):
super(dict, self).__setitem__(key, value)
self.modified = True # we don't have to wrap this item, the dict
is modified, and can't ever be unmodified

def __wrap_setattr(self, real_setattr):
def setattr(attrname, val):
self.modified = True
real_setattr(attrname, val)
return setattr

def send(self, req):
if self.modified:
for cookie in req.cookies.values():
Cookie.add_cookie(req, cookie)

The purpose of which is to be able to store any cookies sent with the
request in the CookieCollection, and then automatically call send()
before the headers are finished being sent and it will only do
something if the cookies have been altered (otheriwse they don't need
to be sent again.) I haven't tested the code yet, but the general idea
should be correct at the very least I think.

Thanks again,
-Sandra

[Christian Tismer]

Steve Holden wrote:

Christian Tismer wrote:

Just to add a word that I forgot:

Adhering to the subject line, the intent is to track modifications
of a dict.
By definition, modification of a member of a dict without replacing
the value is not considered a dict change.
Well, I agree. But I suppose much depends on exactly what the OP meant
by "... add a new element or alter an existing one". The post did follow
that with "(the values in the dict are mutable)", which is presumably
why garabik-2500 proposed catching __getitem__ as well as __setitem__.

Yes, I understood this after reading more. Probably easier to
solve if the problem is spelled more specifically.
I merely wanted to point out (not to you!) that there was no effective
way to capture a change to a mutable item without, as you say, modifying
the element classes.

You are completely right. This is asking for too much, unless one is
prepared to heavily modify the interpreter for debugging purposes,
which actually might be a way to solve the specific problem, once.

cheers - chris

--
Christian Tismer :^) <mailto:ti****@stackless.com>
tismerysoft GmbH : Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/
14109 Berlin : PGP key -> http://wwwkeys.pgp.net/
work [phone] mobile [phone] fax [phone]
PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04
whom do you want to sponsor today? http://www.stackless.com/

[Raymond Hettinger]

[sa***********@yahoo.com]

It's important that I can read the contents of the dict without
flagging it as modified, but I want it to set the flag the moment I add
a new element or alter an existing one (the values in the dict are
mutable), this is what makes it difficult. Because the values are
mutable I don't think you can tell the difference between a read and a
write without making some sort of wrapper around them.

Still, I'd love to hear how you guys would do it.

Take a look at shelve.py to see how Alex Martelli solved a similar
problem. His solution kicks in when writeback=True. The module
docstring elaborates on the problem and solution.

[Duncan Booth]

Bengt Richter wrote:

You are right, but OTOH the OP speaks of a "flagging" the dict as
modified. If she made e.g., "modified" a property of the dict
subclass, then retrieving the the "modified" "flag" could dynamically
check current state repr vs some prior state repr. Then the question
becomes "modified w.r.t. what prior state?"

This lets the OP ask at any time whether the dict is/has_been
modified, but it's not a basis for e.g., a modification-event callback
registry or such.

Good point. So the following matches what was asked for, although
depending on the actual use pattern it may or may not match what is
required:

---------- tracked.py -------------
import cPickle, md5
class TrackedDict(dict):
def __init__(self, *args, **kw):
dict.__init__(self, *args, **kw)
self.resetModified()

def __getstate__(self):
return dict(self)
def __setstate__(self, d):
self.update(d)

def _gethash(self):
pickle = cPickle.dumps(self)
hash = md5.new(pickle).digest()
return hash

@property
def modified(self):
return self._hash != self._gethash()

def resetModified(self):
self._hash = self._gethash()

if __name__=='__main__':
d = TrackedDict(x=[])
assert not d.modified
d['a'] = [1, 2, 3]
assert d.modified
d.resetModified()
assert not d.modified
d['a'].append(4)
assert d.modified
assert d== {'x':[], 'a':[1, 2, 3, 4]}
assert d==cPickle.loads(cPickle.dumps(d))
-----------------------------------

[Kent Johnson]

sa***********@yahoo.com wrote:

Thanks to everyone who posted comments or put some thought into this
problem.

I should have been more specific with what I want to do, from your
comments the general case of this problem, while I hate to say
impossible, is way more trouble than it's worth.

By modified I meant that the dictionary has been changed from its
original (constructed) state by either:
1) A new element was added
2) An existing element was changed

I want to wrap a dictionary of cookies in mod_python. Cookies are
represented as objects with no methods. In light of this and Michael's
excellent post, I came up with the following code.

class CookieCollection(dict):
def __init__(self, d):
for k, v in d:
v.__setattr__ = self.__wrap_setattr(v.__setattr__)
self[k] = v

I don't think this will work if v is an instance of a new-style class -
for new-style classes, special methods are always looked up on the
class, you can't override them in an instance. For example:

class foo(object): ... def __setattr__(self, attr, value):
... print 'foo.__setattr__'
... object.__setattr__(self, attr, value)
... f=foo()
f.x = 3 foo.__setattr__ def new_setattr(attr, value): ... print 'new_setattr'
... f.__setattr__ = new_setattr foo.__setattr__ f.y=3

foo.__setattr__

still calls foo.__setattr__()

Kent