is decorator the right thing to use?

[Dmitry S. Makovey]

Hi,

after hearing a lot about decorators and never actually using one I have
decided to give it a try. My particular usecase is that I have class that
acts as a proxy to other classes (i.e. passes messages along to those
classes) however hand-coding this type of class is rather tedious, so I
decided to use decorator for that. Can somebody tell me if what I'm doing
is a potential shot-in-the-foot or am I on the right track? (Note, It's
rather rudimentary proof-of-concept implementation and not the final
solution I'm about to employ so there are no optimizations or
signature-preserving code there yet, just the idea).

Here's the code:

class A:
b=None
def __init__(self,b ):
self.val='aval'
self.b=b
b.val='aval'

def mymethod(self,a ):
print "A::mymetho d, ",a

def mymethod2(self, a):
print "A::another method, ",a
def Aproxy(fn):
def delegate(*args, **kw):
print "%s::%s" % (args[0].__class__.__na me__,fn.__name_ _)
args=list(args)
b=getattr(args[0],'b')
fnew=getattr(b, fn.__name__)
# get rid of original object reference
del args[0]
fnew(*args,**kw )
setattr(A,fn.__ name__,delegate )
return fn

class B:
def __init__(self):
self.val='bval'

@Aproxy
def bmethod(self,a) :
print "B::bmethod "
print a, self.val

@Aproxy
def bmethod2(self,a ):
print "B::bmethod 2"
print a, self.val

b=B()
b.bmethod('foo' )
a=A(b)
b=B()
b.val='newval'
a.bmethod('bar' )
a.bmethod2('zam ')

[George Sakkis]

On Sep 25, 3:36 pm, "Dmitry S. Makovey" <dmi...@athabas cau.cawrote:

Aaron "Castironpi " Brady wrote:

I kept this part of the problem out of this discussion as I'm pretty sure
I can fill those in once I figure out the basic problem of
auto-population of proxy methods since for each class/method those are
going to be nearly identical. If I can autogenerate those on-the-fly I'm
pretty sure I can add some extra-logic to them as well including
signature change where A::bmethod(self ,c,x) would become
A::bmethod(self ,x) etc.

Do you want to couple instances or classes together?

It would be nice to have objects of B, C and D classes not knowing that they
are proxied (as they are used on their own too, not only inside of A
objects).

I'm not sure if the approach below deals with all the issues, but one
thing it does is decouple completely the proxied objects from the
proxy:

#======== usage
=============== =============== =============== =============== ====

from proxies import Proxy

class B(object):
def __init__(self): self.val = 'bval'
def bmethod(self,n) : print "B::bmethod ",n
def bmethod2(self,n ,m): print "B::bmethod2",n ,m

class C(object):
def __init__(self): self.val = 'cval'
def cmethod(self,x) : print "C::cmethod ",x
def cmethod2(self,x ,y): print "C::cmethod2",x ,y
cattr = 4

class A(Proxy):
DelegateMap = {
'bmethod' : B,
'bmethod2': B,
'cmethod': C,
# do NOT delegate C.cmethod2
#'cmethod2': C,
'cattr' : C,
}

def __init__(self, b, c):
print "init A()"
# must call Proxy.__init__( *delegates)
super(A,self)._ _init__(b,c)

def amethod(self,a) :
print "A::mymetho d",a
if __name__ == '__main__':
a = A(B(), C())
a.amethod('foo' )

# test bounded methods
a.bmethod('foo' )
a.bmethod2('bar ','baz')
a.cmethod('foo' )
try: a.cmethod2('bar ','baz')
except Exception, ex: print ex

# works for unbound methods too
A.bmethod(a,'fo o')
A.bmethod2(a,'b ar','baz')
A.cmethod(a, 'foo')
try: A.cmethod2(a,'b ar','baz')
except Exception, ex: print ex

# non callable attributes
print A.cattr

#====== output =============== =============== ====
init A()
A::mymethod foo
B::bmethod foo
B::bmethod2 bar baz
C::cmethod foo
'A' object has no attribute 'cmethod2'
B::bmethod foo
B::bmethod2 bar baz
C::cmethod foo
type object 'A' has no attribute 'cmethod2'
4

#======== proxies.py =============== ==========

class _ProxyMethod(ob ject):
def __init__(self, name):
self._name = name
def unbound(proxy, *args, **kwds):
method = proxy._get_targ et_attr(name)
return method(*args, **kwds)
self._unbound = unbound

def __get__(self, proxy, proxytype):
if proxy is not None:
return proxy._get_targ et_attr(self._n ame)
else:
return self._unbound
class _ProxyMeta(type ):
def __new__(meta, name, bases, namespace):
for attrname,cls in namespace.get(' DelegateMap',
{}).iteritems() :
if attrname not in namespace:
attr = getattr(cls, attrname)
if callable(attr):
namespace[attrname] = _ProxyMethod(at trname)
else:
namespace[attrname] = attr
return super(_ProxyMet a,meta).__new__ (meta, name, bases,
namespace)
class Proxy(object):
__metaclass__ = _ProxyMeta

def __init__(self, *delegates):
self._cls2deleg ate = {}
for delegate in delegates:
cls = type(delegate)
if cls in self._cls2deleg ate:
raise ValueError('Mor e than one %s delegates were
given' % cls)
self._cls2deleg ate[cls] = delegate

def _get_target_att r(self, name):
try:
cls = self.DelegateMa p[name]
delegate = self._cls2deleg ate[cls]
return getattr(delegat e, name)
except (KeyError, AttributeError) :
raise AttributeError( '%r object has no attribute %r' %
(self.__class__ .__name__, name))

HTH,
George

[Dmitry S. Makovey]

George Sakkis wrote:

I'm not sure if the approach below deals with all the issues, but one
thing it does is decouple completely the proxied objects from the
proxy:

<snip/>

class _ProxyMeta(type ):

<snip/>

It smelled to me more and more like metaclass too, I was just trying to
avoid them :)

Your code looks awefully close to what I'm trying to do, except it looks bit
heavier than decorators. Seems like decorators are not going to happen in
this part of project for me anyway, however the whole discussion gave me a
lot to think about. Thank you Bruno, Aaron, Diez and George.

Thanks for the concrete code with metaclass. I'm going to study it
thoroughly to see if I can spot caveats/issues for my usecases however it
seems to put me on the right track. I never used metaclasses before and
decorators seemed to be bit more straight-forward to me :) ..oh well.

[Bruno Desthuilliers]

Dmitry S. Makovey a écrit :
(snip)

I never used metaclasses before and
decorators seemed to be bit more straight-forward to me :) ..oh well.

Don't be afraid !-) While it's true that they can be a bit confusing at
first, metaclasses are just classes - whose instances happens to be
classes themselves.

And while you're certainly right to not jump on metaclasses *before*
hanving spent a bit time thinking of other possible solutions, there's
nothing wrong with using metaclasses when that's really what you need...

[Paul McGuire]

On Sep 25, 10:41*am, "Dmitry S. Makovey" <dmi...@athabas cau.cawrote:

Diez B. Roggisch wrote:

Dmitry S. Makovey schrieb:

Dmitry S. Makovey wrote:
In my real-life case A is a proxy to B, C and D instances/objects, not
just one.

forgot to mention that above would mean that I need to have more than one
decorator function like AproxyB, AproxyC and AproxyD or make Aproxy
smarter about which property of A has instance of which class etc.

__getattr__?

see, in your code you're assuming that there's only 1 property ( 'b' )
inside of A that needs proxying. In reality I have several. So in your code
self._delegate should be at least a tupple or a list. Plus what you're
doing - you just promiscuously passing any method not found in Proxy to
self._delegate which is not what I need as I need to pass only a subset of
calls, so now your code needs to acquire dictionary of "allowed" calls, and
go over all self._delegates to find if any one has it which is not
efficient since there IS a 1:1 mapping of A::method -B::method so lookups
shouldn't be necessary IMO (for performance reasons).

No, really, Diez has posted the canonical Proxy form in Python, using
__getattr__ on the proxy, and then redirecting to the contained
delegate object. This code does *not* assume that only one property
('b'? where did that come from?) is being redirected - __getattr__
will intercept all attribute lookups and redirect them to the
delegate.

If you need to get fancier and support this single-proxy-to-multiple-
delegates form, then yes, you will need some kind of map that says
which method should delegate to which object. Or, if it is just a
matter of precedence (try A, then try B, then...), then use hasattr to
see if the first delegate has the given attribute, and if not, move on
to the next.

>
I'm trying to figure out if there
is another way to use decorators for my scenario or is there another way of
achieving the same thing without using decorators and without bloating up
the code with alternative solution.

Another way could be to use Metaclass to populate class with method upon
declaration but that presents quite a bit of "special" cruft which is more
than I have to do with decorators :) (but maybe it's all *necessary* ? )

Your original question was "is decorator the right thing to use?" For
this application, the answer is "no". It sounds like you are trying
to force this particular to solution to your problem, but you are
probably better off giving __getattr__ intercepting another look.

-- Paul

[Dmitry S. Makovey]

Paul McGuire wrote:

>see, in your code you're assuming that there's only 1 property ( 'b' )
inside of A that needs proxying. In reality I have several.

<snip/>

>
No, really, Diez has posted the canonical Proxy form in Python, using
__getattr__ on the proxy, and then redirecting to the contained
delegate object. This code does *not* assume that only one property
('b'? where did that come from?) is being redirected - __getattr__
will intercept all attribute lookups and redirect them to the
delegate.

If you need to get fancier and support this single-proxy-to-multiple-
delegates form, then yes, you will need some kind of map that says
which method should delegate to which object. Or, if it is just a
matter of precedence (try A, then try B, then...), then use hasattr to
see if the first delegate has the given attribute, and if not, move on
to the next.

that is what I didn't like about it - I have to iterate over delegates when
I can build direct mapping once and for all and tie it to class
definition ;)

Your original question was "is decorator the right thing to use?" For
this application, the answer is "no".

yeah. seems that way. in the other fork of this thread you'll find my
conclusion which agrees with that :)

It sounds like you are trying
to force this particular to solution to your problem, but you are
probably better off giving __getattr__ intercepting another look.

__getattr__ implies constant lookups and checks (for filtering purposes) - I
want to do them once, attach generated methods as native methods and be
done with it. That is why I do not like __getattr__ in this particular
case. Otherwise - you're right.

[Dmitry S. Makovey]

Bruno Desthuilliers wrote:

Hem... I'm afraid you don't really take Python's dynamic nature into
account here. Do you know that even the __class__ attribute of an
instance can be rebound at runtime ? What about 'once and for all' then ?

must've been wrong wording on my part. Dynamic nature is exactly what I
wanted to use :) except that I do not expect clients to take advantage of
it while using my classes ;)

>>Your original question was "is decorator the right thing to use?" For
this application, the answer is "no".

yeah. seems that way. in the other fork of this thread you'll find my
conclusion which agrees with that :)

>>It sounds like you are trying
to force this particular to solution to your problem, but you are
probably better off giving __getattr__ intercepting another look.

__getattr__ implies constant lookups and checks (for filtering purposes)

Unless you cache the lookups results...

sure

>- I
want to do them once, attach generated methods as native methods

What is a "native method" ? You might not be aware of the fact that
method objects are usually built anew from functions on each method
call...

again, wrong wording on my part. by native I meant: make use as much as
possible of existing machinery and override default behavior only when it's
absolutely necessary. (hopefully my wording is not off this time ;) )

>and be
done with it. That is why I do not like __getattr__ in this particular
case.

There's indeed an additional penalty using __getattr__, which is that
it's only called as a last resort. Now remember that premature
optimization is the root of evil... Depending on effective use (ie : how
often a same 'proxied' method is called on a given Proxy instance, on
average), using __getattr__ to retrieve the appropriate bound method on
the delegate then adding it to the proxy instance *as an instance
attribute* might be a way better (and simpler) optimization.

I actually ended up rewriting things (loosely based on George's suggested
code) with descriptors and not using metaclasses or decorators (so much for
my desire to use them).

With following implementation (unpolished at this stage but already
functional) I can have several instances of B objects inside of A object
and proxy certain methods to one or another object (I might be having a
case when I have A.b1 and A.b2 and passing some methods to b1 and others to
b2 having both of the same class B, maybe even multiplexing). This one
seems to be fairly light as well without need to scan instances (well,
except for one getattr, but I couldn't get around it). Maybe I didn't
account for some shoot-in-the-foot scenarios but I can't come up with any.
Last time I played with __getattr__ I shot myself in the foot quite well
BTW :)

class ProxyMethod(obj ect):

def __init__(self,o b_name,meth):
self.ob_name=ob _name
self.meth=meth

def my_call(self,in stance,*argv,** kw):
ob=getattr(inst ance,self.ob_na me)
cls=self.meth.i m_class
return self.meth.__get __(ob,cls)(*arg v,**kw)

def __get__(self,in stance,owner):
if not instance:
return self.my_call
ob=getattr(inst ance,self.ob_na me)
cls=self.meth.i m_class
return self.meth.__get __(ob,cls)

class B:
def __init__(self):
self.val='bval'

def bmethod(self,a) :
print "B::bmethod ",
print a, self.val

class A:
b=None

def __init__(self,b =None):
self.val='aval'
self.b=b
b.val='aval-b'

def mymethod(self,a ):
print "A::mymetho d, ",a

bmethod=ProxyMe thod('b',B.bmet hod)

b=B()
b.bmethod('foo' )
a=A(b)
b=B()
b.val='newval'
a.mymethod('baz ')
a.bmethod('bar' )
A.bmethod(a,'zo om')

[Aaron \Castironpi\ Brady]

On Sep 26, 3:03*pm, Bruno Desthuilliers
<bdesth.quelque [e-mail] quepart.frwrote :

Dmitry S. Makovey a écrit :

Paul McGuire wrote:

>see, in your code you're assuming that there's only 1 property ( 'b' )
inside of A that needs proxying. In reality I have several.

<snip/>

No, really, Diez has posted the canonical Proxy form in Python, using
__getattr__ on the proxy, and then redirecting to the contained
delegate object. *This code does *not* assume that only one property
('b'? where did that come from?) is being redirected - __getattr__
will intercept all attribute lookups and redirect them to the
delegate.

If you need to get fancier and support this single-proxy-to-multiple-
delegates form, then yes, you will need some kind of map that says
which method should delegate to which object. *Or, if it is just a
matter of precedence (try A, then try B, then...), then use hasattr to
see if the first delegate has the given attribute, and if not, move on
to the next.

that is what I didn't like about it - I have to iterate over delegates when
I can build direct mapping once and for all and tie it to class
definition ;)

Hem... I'm afraid you don't really take Python's dynamic nature into
account here. Do you know that even the __class__ attribute of an
instance can be rebound at runtime ? What about 'once and for all' then ?

But anyway:

Your original question was "is decorator the right thing to use?" *For
this application, the answer is "no". *

yeah. seems that way. in the other fork of this thread you'll find my
conclusion which agrees with that :)

It sounds like you are trying
to force this particular to solution to your problem, but you are
probably better off giving __getattr__ intercepting another look.

__getattr__ implies constant lookups and checks (for filtering purposes)

Unless you cache the lookups results...

- I
want to do them once, attach generated methods as native methods

What is a "native method" ? You might not be aware of the fact that
method objects are usually built anew from functions on each method call....

and be
done with it. That is why I do not like __getattr__ in this particular
case.

There's indeed an additional penalty using __getattr__, which is that
it's only called as a last resort. Now remember that premature
optimization is the root of evil... Depending on effective use (ie : how
often a same 'proxied' method is called on a given Proxy instance, on
average), using __getattr__ to retrieve the appropriate bound method on
the delegate then adding it to the proxy instance *as an instance
attribute* might be a way better (and simpler) optimization.

That prohibits using a descriptor in the proxied classes, or at least
the proxied functions, since you break descriptor protocol and only
call __get__ once. Better to cache and get by name. It's only slower
by the normal amount, and technically saves space, strings vs.
instancemethod objects (except for really, really long function names).

[Dmitry S. Makovey]

Aaron "Castironpi " Brady wrote:

That prohibits using a descriptor in the proxied classes, or at least
the proxied functions, since you break descriptor protocol and only
call __get__ once. Better to cache and get by name. It's only slower
by the normal amount, and technically saves space, strings vs.
instancemethod objects (except for really, really long function names).

that is an interesting point since I didn't think about having descriptors
in proxied classes. my reworked code clearly breaks when descriptors are
thrown at it. It will break with methods in proxied objects that are
implemented as objects too. Now I adjusted constructor a bit to account for
that (I just can't figure out case when I'll be proxying descriptors unless
they return function but than I don't see benefit in using descriptor for
that, probably because I haven't used them much).

class ProxyMethod(obj ect):
def __init__(self,o b_name,meth):
self.ob_name=ob _name
if not hasattr(meth,'i m_class'):
if hasattr(meth,'_ _call__'):
self.meth=getat tr(meth,'__call __')
else:
raise ValueError("Met hod should be either a class method or
a callable class")
else:
self.meth=meth

[George Sakkis]

On Sep 26, 6:38*pm, "Dmitry S. Makovey" <dmi...@athabas cau.cawrote:

I actually ended up rewriting things (loosely based on George's suggested
code) with descriptors and not using metaclasses or decorators (so much for
my desire to use them).

With following implementation (unpolished at this stage but already
functional) I can have several instances of B objects inside of A object
and proxy certain methods to one or another object (I might be having a
case when I have A.b1 and A.b2 and passing some methods to b1 and others to
b2 having both of the same class B, maybe even multiplexing).

You seem to enjoy pulling the rug from under our feet by changing the
requirements all the time :)

This one
seems to be fairly light as well without need to scan instances (well,
except for one getattr, but I couldn't get around it). Maybe I didn't
account for some shoot-in-the-foot scenarios but I can't come up with any..
Last time I played with __getattr__ I shot myself in the foot quite well
BTW :)

[code snipped]

class A:
* * b=None

^^^^^^ you don't need this

* * def __init__(self,b =None):
* * * * self.val='aval'
* * * * self.b=b
* * * * b.val='aval-b'

* * def mymethod(self,a ):
* * * * print "A::mymetho d, ",a

* * bmethod = ProxyMethod('b' ,B.bmethod)

Although this works, the second argument to ProxyMethod shouldn't be
necessary, it's semantically redundant; ideally you would like to
write it as "bmethod = ProxyMethod('b' )". As before, I don't think
that's doable without metaclasses (or worse, stack frame hacking).
Below is the update of my original recipe; interestingly, it's
(slightly) simpler than before:

#======= usage =============== =============== ==========

from proxies import Proxy

class B(object):
def __init__(self, val): self.val = val
def bmethod(self,n) : print "B::bmethod ", self.val, n
def bmethod2(self,n ,m): print "B::bmethod 2", self.val, n, m

class C(object):
def __init__(self, val): self.val = val
def cmethod(self,x) : print "C::cmethod ", self.val, x
def cmethod2(self,x ,y): print "C::cmethod2",s elf.val, x, y
cattr = 4

class A(Proxy):
# DelegateMap:
# Maps each delegate method to the proxy attribute that refers to
the
# respective delegate object
DelegateMap = {
'bmethod' : 'b1',
'bmethod2': 'b2',
'cmethod' : 'c',
# do NOT delegate C.cmethod2
#'cmethod2': 'c',
}

def __init__(self, b1, b2, c):
print "init A()"
# must call Proxy.__init__
super(A,self)._ _init__(b1=b1, b2=b2, c=c)

def amethod(self,a) :
print "A::mymetho d",a

if __name__ == '__main__':
a = A(B(10), B(20), C(30))
a.amethod('foo' )

print "bound proxy calls"
a.bmethod('foo' )
a.bmethod2('bar ','baz')
a.cmethod('foo' )
try: a.cmethod2('bar ','baz')
except Exception, ex: print ex

print "unbound proxy calls"
A.bmethod(a,'fo o')
A.bmethod2(a,'b ar','baz')
A.cmethod(a, 'foo')
try: A.cmethod2(a,'b ar','baz')
except Exception, ex: print ex

#======= output =============== =============== ==========

init A()
A::mymethod foo
bound proxy calls
B::bmethod 10 foo
B::bmethod2 20 bar baz
C::cmethod 30 foo
'A' object has no attribute 'cmethod2'
unbound proxy calls
B::bmethod 10 foo
B::bmethod2 20 bar baz
C::cmethod 30 foo
type object 'A' has no attribute 'cmethod2'

#====== proxies.py =============== =============== ========

class _ProxyMeta(type ):
def __new__(meta, name, bases, namespace):
for methodname in namespace.get(' DelegateMap', ()):
if methodname not in namespace:
namespace[methodname] = _ProxyMethod(me thodname)
return super(_ProxyMet a,meta).__new__ (meta, name, bases,
namespace)
class _ProxyMethod(ob ject):
def __init__(self, name):
self._name = name

def __get__(self, proxy, proxytype):
if proxy is not None:
return proxy._get_targ et_attr(self._n ame)
else:
return self._unbound_m ethod

def _unbound_method (self, proxy, *args, **kwds):
method = proxy._get_targ et_attr(self._n ame)
return method(*args, **kwds)
class Proxy(object):
__metaclass__ = _ProxyMeta

def __init__(self, **attr2delegate ):
self.__dict__.u pdate(attr2dele gate)

def _get_target_att r(self, name):
try:
delegate = getattr(self, self.DelegateMa p[name])
return getattr(delegat e, name)
except (KeyError, AttributeError) :
raise AttributeError( '%r object has no attribute %r' %
(self.__class__ .__name__, name))
HTH,
George

[Dmitry S. Makovey]

George Sakkis wrote:

You seem to enjoy pulling the rug from under our feet by changing the
requirements all the time :)

but that's half the fun! ;)

Bit more seriously - I didn't know I had those requirements until now :) I'm
kind of exploring where can I get with those ideas. Initial post was based
exactly on what I had in my code with desire to make it more
automated/{typo,fool}-proof/robust, elegant and possibly faster. Hopefully
such behavior is not off-topic on this list (if it is - let me know and
I'll go exploring solo).

Although this works, the second argument to ProxyMethod shouldn't be
necessary, it's semantically redundant; ideally you would like to
write it as "bmethod = ProxyMethod('b' )".

since I'm already on exploratory trail (what about that rug being pulled
from under....?) With my code I can do really dirty tricks like this (not
necessary that I'm going to):

class B_base:
def bmethod(self):
print 'B_base'

class B(B_base):
def bmethod(self):
print 'B'

class A:
bmethod=ProxyMe thod('b',B_base .bmethod)

As before, I don't think
that's doable without metaclasses (or worse, stack frame hacking).
Below is the update of my original recipe; interestingly, it's
(slightly) simpler than before:

Out of curiosity (and trying to understand): why do you insist on
dictionaries with strings contents ( {'bmethod' : 'b1' } ) rather than
something more explicit ? Again, I can see that your code is working and I
can even understand what it's doing, just trying to explore alternatives :)

I guess my bias is towards more explicit declarations thus

bmethod=ProxyMe thod('b',B.bmet hod)

looks more attractive to me, but I stand to be corrected/educated why is
that not the right thing to do?

Another thing that turns me away from string dictionaries is that those are
the ones causing me more trouble hunting down typos. Maybe it's just "my
thing" so I'm not going to insist on it. I'm open to arguments against that
theory.

One argument I can bring in defence of more explicit declarations is IDE
parsing when autocompletion for B.bme... pops up (suggesting bmethod and
bmethod2) and with 'b':'bmethod' it never happens. However that's not good
enough reason to stick to it if it introduces other problems. What kind of
problems could those be?

P.S.
so far I find this discussion quite educating BTW. I am more of a "weekend
programmer" with Python - using basic language for most of things I need,
however I decided to explore further to educate myself and to write more
efficient code.