Delayed evaluation and setdefault()

[Leo Breebaart]

Hi all,

I have a question about Python and delayed evaluation.

Short-circuiting of Boolean expressions implies that in:

if a() and b():
any possible side-effects the call to b() might have will not
happen of a() returns true, because then b() will never be
executed.

However, if I go looking for dictionary keys like this:
d = {}
d.setdefault('foo', b())

then b() *does* get executed regardless of whether or not the
value it returns is actually used ('foo' was not found) or not
('foo' was found).

If b() has side-effects, this may not be what you want at all. It
would seem to me not too exotic to expect Python to have some
sort of construct or syntax for dealing with this, just as it
supports Boolean short-circuiting.

So I guess my question is twofold: one, do people think
differently, and if so why?; and two: is there any elegant (or
other) way in which I can achieve the delayed evaluation I desire
for setdefault, given a side-effect-having b()?

--
Leo Breebaart <le*@lspace.org>

[anton muhin]

Leo Breebaart wrote:

Hi all,

I have a question about Python and delayed evaluation.

Short-circuiting of Boolean expressions implies that in:

>>> if a() and b():
any possible side-effects the call to b() might have will not
happen of a() returns true, because then b() will never be
executed.

However, if I go looking for dictionary keys like this:
>>> d = {}
>>> d.setdefault('foo', b())

then b() *does* get executed regardless of whether or not the
value it returns is actually used ('foo' was not found) or not
('foo' was found).

If b() has side-effects, this may not be what you want at all. It
would seem to me not too exotic to expect Python to have some
sort of construct or syntax for dealing with this, just as it
supports Boolean short-circuiting.

So I guess my question is twofold: one, do people think
differently, and if so why?; and two: is there any elegant (or
other) way in which I can achieve the delayed evaluation I desire
for setdefault, given a side-effect-having b()?

It's normal---setdefault is just a method and its parameters get
evaluted before the call occur. If you want this kind of lazy
evaluation, I'd suggest something like this ('foo' in d) or
(d.setdefault('foo', b()). If you want your value too, it gets even
trickier.

Actually, if you really need this kind of functionality, it might be
worth defining a function.

def lazysetdefault(d, key, func):
if key not in d:
d[key] = func()()
return d[key]

lazysetdefault(d, 'foo', lambda: b())

However, are you real sure you need it?

regards,
anton.

[Peter Hansen]

Leo Breebaart wrote:

>>> d.setdefault('foo', b()) then b() *does* get executed regardless of whether or not the
value it returns is actually used ('foo' was not found) or not
('foo' was found).

So I guess my question is twofold: one, do people think
differently, and if so why?; and two: is there any elegant (or
other) way in which I can achieve the delayed evaluation I desire
for setdefault, given a side-effect-having b()?

def lazysetdefault(dict, key, ref):
if not dict.has_key(key):
dict[key] = ref()
return dict[key]

d = {}
lazysetdefault(d, 'foo', b)

-Peter

[Erik Max Francis]

Leo Breebaart wrote:

So I guess my question is twofold: one, do people think
differently, and if so why?; and two: is there any elegant (or
other) way in which I can achieve the delayed evaluation I desire
for setdefault, given a side-effect-having b()?

The setdefault method of dictionaries is just a normal method in Python;
only the builtin `and' and `or' operators have special short-circuiting
behavior. That is to say, there are no builtin functions or methods in
Python which are "special forms."

It's true that the kind of thing you want to do (lazy evaluation in some
incidental context) is common, but it would not be a good idea to start
selectively adding special forms to the language since it would greatly
complicate things. Right now if I see

this.that(something, theOther())

I can immediately tell what gets evaluated or called and when. I don't
need to know that there are special cases in some things that look like
method/function calls but actually might be special forms, and that's a
good thing.

As for your particular solution, you're much better off implementing the
lazy evaluation you want yourself, rather than wishing there were
special language support for it for that particular method (or a very
small subset of methods/functions).

--
__ Erik Max Francis && ma*@alcyone.com && http://www.alcyone.com/max/
/ \ San Jose, CA, USA && 37 20 N 121 53 W && &tSftDotIotE
\__/ The critical period in matrimony is breakfast-time.
-- A.P. Herbert

[Peter Otten]

Leo Breebaart wrote:

Hi all,

I have a question about Python and delayed evaluation.

Short-circuiting of Boolean expressions implies that in:

>>> if a() and b():
any possible side-effects the call to b() might have will not
happen of a() returns true, because then b() will never be
executed.

However, if I go looking for dictionary keys like this:
>>> d = {}
>>> d.setdefault('foo', b())

then b() *does* get executed regardless of whether or not the
value it returns is actually used ('foo' was not found) or not
('foo' was found).

If b() has side-effects, this may not be what you want at all. It
would seem to me not too exotic to expect Python to have some
sort of construct or syntax for dealing with this, just as it
supports Boolean short-circuiting.

So I guess my question is twofold: one, do people think
differently, and if so why?; and two: is there any elegant (or
other) way in which I can achieve the delayed evaluation I desire
for setdefault, given a side-effect-having b()?

Lazy evaluation is dangerous as the state of mutable objects may have
changed in the mean time. In your case

if "foo" not in d:
d["foo"] = b()

should do.

However, if you absolutely want it, here's a slightly more general approach:

<defer.py>
class Defer:
def __init__(self, fun, *args):
self.fun = fun
self.args = args
def __call__(self):
""" Calculate a deferred function the first time it is
invoked, return the stored result for subsequent calls
"""
try:
return self.value
except AttributeError:
self.value = self.fun(*self.args)
return self.value

def defer(fun, *args):
""" Use, e. g., defer(average, 1, 2) to delay the calculation
of the average until the first time undefer() is called
with the Defer instance.
"""
return Defer(fun, *args)

def undefer(value):
""" Check if value is deferred, if so calculate it, otherwise
return it unchanged
"""
if isinstance(value, Defer):
return value()
return value

#example usage:

class Dict(dict):
def setdefault(self, key, value):
try:
return self[key]
except KeyError:
# it might make sense to further delay the
# calculation until __getitem__()
self[key] = value = undefer(value)
return value

def b(value):
print "side effect for", value
return value
dv = defer(b, "used three times")

d = Dict()
d.setdefault("x", defer(b, "a1"))
d.setdefault("x", defer(b, "a2"))
d.setdefault("y", "b")
d.setdefault("a", dv)
d.setdefault("b", dv)
d.setdefault("c", dv)
print ",\n".join(repr(d).split(","))
assert d["a"] is d["b"]

</defer.py>

You will still have to bother with undeferring in *many* places in your
program, so the above would go into the "or other" category. For lazy
evaluation to take off, I think it has to be build into the language. I
doubt, though, that much effort will be saved, as a human is normally lazy
enough to not calculate things until absolutely needed.

Peter

[Aahz]

In article <40***************@engcorp.com>,
Peter Hansen <pe***@engcorp.com> wrote:

Leo Breebaart wrote:

>>> d.setdefault('foo', b())

then b() *does* get executed regardless of whether or not the
value it returns is actually used ('foo' was not found) or not
('foo' was found).

So I guess my question is twofold: one, do people think
differently, and if so why?; and two: is there any elegant (or
other) way in which I can achieve the delayed evaluation I desire
for setdefault, given a side-effect-having b()?

def lazysetdefault(dict, key, ref):
if not dict.has_key(key):
dict[key] = ref()
return dict[key]

First of all, Peter made a boo-boo in naming a parameter ``dict``.

If the key will be in the dict more than 90% (*very* roughly) of the time
you call this, you can do this instead:

def lazysetdefault(d, key, ref):
try:
return d[key]
except KeyError:
d[key] = ref()
return d[key]

That minimizes the bytecode and function calls.
--
Aahz (aa**@pythoncraft.com) <*> http://www.pythoncraft.com/

A: No.
Q: Is top-posting okay?

[Peter Hansen]

Aahz wrote:

In article <40***************@engcorp.com>,
Peter Hansen <pe***@engcorp.com> wrote:

Leo Breebaart wrote:

>>> d.setdefault('foo', b())
then b() *does* get executed regardless of whether or not the
value it returns is actually used ('foo' was not found) or not
('foo' was found).

So I guess my question is twofold: one, do people think
differently, and if so why?; and two: is there any elegant (or
other) way in which I can achieve the delayed evaluation I desire
for setdefault, given a side-effect-having b()?

def lazysetdefault(dict, key, ref):
if not dict.has_key(key):
dict[key] = ref()
return dict[key]

First of all, Peter made a boo-boo in naming a parameter ``dict``.

Oops... I was, uh, writing pseudo-code, not Python! Yeah, that's it...

-but-at-least-it-would-have-executed-ly yr's,
Peter

[Aahz]

In article <40***************@engcorp.com>,
Peter Hansen <pe***@engcorp.com> wrote:

Aahz wrote:

First of all, Peter made a boo-boo in naming a parameter ``dict``.

Oops... I was, uh, writing pseudo-code, not Python! Yeah, that's it...

Python *is* pseudo-code. That's why we like it, right?
--
Aahz (aa**@pythoncraft.com) <*> http://www.pythoncraft.com/

A: No.
Q: Is top-posting okay?