Decorator for Enforcing Argument Types

[Chris]

I'm not sure if this has been done before, but I couldn't easily find
any prior work on Google, so here I present a simple decorator for
documenting and verifying the type of function arguments.
Feedback/suggestions/criticism is welcome.

'''
2006.12.21 Created.
'''

import unittest
import inspect

def arguments(*args):
'''A simple decorator for formally documenting
and verifying argument types.

usage:

@arguments(type1, type2, [type3.1, type3.2, ..., type3.N],
type4, ..., typeN)
def somefunc(arg1, arg2, arg3, arg4, ..., argN):
do stuff
return

'''
return lambda f:_Arguments(f, *args)

class _Arguments(object):
# todo: extend to verify Zope interfaces
def __init__(self, fn, *args):
self.fn = fn

# create argument type list
self.arguments = []
for arg in args:
if not isinstance(arg, list):
arg = list([arg])
arg = set(arg)
self.arguments.append(arg)

# create name-to-index lookup
argNames, varArgName, varkwName, defaults =
inspect.getargspec(fn)
assert len(argNames) == len(self.arguments), 'list of argument
types must match the number of arguments'
self.argNameToIndex = {}
for i,name in enumerate(argNames):
self.argNameToIndex[name] = i
if defaults and i >= len(self.arguments)-len(defaults):
# add default type to allowable types

self.arguments[i].add(type(defaults[i-(len(self.arguments)-len(defaults))]))

def verify(self, value, i):
'''Returns true if the value matches the allowable types
for the ith argument.'''
if not isinstance(i, int):
if i not in self.argNameToIndex:
raise Exception, 'unknown argument name: %s' % i
i = self.argNameToIndex[i]
return type(value) in self.arguments[i]

def verifyAll(self, *values, **kvalues):
'''Returns true if all values matche the allowable types
for their corresponding arguments.'''
for i,value in enumerate(values):
if not self.verify(value, i):
return False
for name,value in kvalues.iteritems():
if not self.verify(value, name):
return False
return True

def __call__(self, *args, **kargs):
assert self.verifyAll(*args, **kargs), 'argument types must be
in the form of %s' % self.arguments
return self.fn(*args, **kargs)

class Test(unittest.TestCase):

def test(self):

@arguments(str, [int, float], list)
def foo(abc, xyz, big=None):
return '%s %s' % (abc, xyz)

self.assertEqual(type(foo), _Arguments)
self.assertEqual(len(foo.arguments), 3)
self.assertEqual(foo.arguments[2], set([list, type(None)]))
self.assertEqual(foo.verify('how', 0), True)
self.assertEqual(foo.verify(123, 0), False)
self.assertEqual(foo.verify(123, 1), True)
self.assertEqual(foo.verify(1.23, 1), True)
self.assertEqual(foo.verifyAll('how',123), True)
self.assertEqual(foo.verifyAll(123,'how'), False)
self.assertEqual(foo.verifyAll(abc='how',xyz=123), True)
self.assertEqual(foo.verifyAll('how',xyz=123), True)
self.assertEqual(foo.verifyAll('how',xyz='oeuuo'), False)
self.assertEqual(foo.verifyAll('how',xyz=123,big=N one), True)
self.assertEqual(foo.verifyAll('how',xyz=123,big=[1,2,3]),
True)
self.assertEqual(foo.verifyAll('how',123,[1,2,3]), True)
self.assertEqual(foo.verifyAll('how',123,'asoenhua s'), False)
self.assertTrue(foo('how',123))
self.assertTrue(foo(abc='how',xyz=123,big=None))

if __name__ == '__main__':

unittest.main()

[Paul McGuire]

"Chris" <ch*******@gmail.comwrote in message
news:11**********************@79g2000cws.googlegro ups.com...

I'm not sure if this has been done before, but I couldn't easily find
any prior work on Google, so here I present a simple decorator for
documenting and verifying the type of function arguments.
Feedback/suggestions/criticism is welcome.

They Python wiki has a page for decorator ideas/submissions, compare yours
to this one:
http://wiki.python.org/moin/PythonDe...348f78db34303e

-- Paul

[Chris]

On Dec 21, 3:57 pm, "Paul McGuire" <[e-mail]._bogus_.comwrote:

"Chris" <[e-mail] in messagenews:11**********************@79g2000cws.go oglegroups.com...

I'm not sure if this has been done before, but I couldn't easily find
any prior work on Google, so here I present a simple decorator for
documenting and verifying the type of function arguments.
Feedback/suggestions/criticism is welcome.They Python wiki has a page for decorator ideas/submissions, compare yours

to this one:http://wiki.python.org/moin/PythonDe...-308f2b3507ca9...

Thanks, I was unaware of that page. I designed mine allow for
introspection, so the program can explicitly "see" what types a
function takes and test argument combinations if need be, which that
example doesn't seem to allow. Mine also supports multiple types per
argument. However, I like how that one also checks the return value.

Chris

[Fredrik Lundh]

Chris wrote:

I'm not sure if this has been done before

see example 4 in the original specification:

http://www.python.org/dev/peps/pep-0318/#examples

</F>

[bearophileHUGS]

Chris wrote:

I'm not sure if this has been done before,

I think this is the best implementation:
http://oakwinter.com/code/typecheck/

I have never used it, but it seems well done.

Bye,
bearophile

[Bruno Desthuilliers]

Chris a écrit :

I'm not sure if this has been done before,

It has...

but I couldn't easily find
any prior work on Google, so here I present a simple decorator for
documenting and verifying the type of function arguments.
Feedback/suggestions/criticism is welcome.

<my humble opinion>
Python is dynamic, and fighting against the language is IMHO a really
bad idea. The only places where theres a real need for this kind of
stuff are when dealing with the "outside world" (IOW : inputs and
outputs). And then packages like formencode can do much more than mere
type-checking
</my humble opinion>

[John Machin]

Bruno Desthuilliers wrote:

<my humble opinion>
Python is dynamic, and fighting against the language is IMHO a really
bad idea. The only places where theres a real need for this kind of
stuff are when dealing with the "outside world" (IOW : inputs and
outputs). And then packages like formencode can do much more than mere
type-checking
</my humble opinion>

Agreed. The worst case I have seen:

An elaborate decorator (similar to the OP's) laboriously checks arg
types. That's IMHO not "consenting adults" territory. But it gets a
whole lot worse when it's applied to a method whose body is like this:

if isinstance(....
action_for_type1(...
# big snip
elif isinstance(...
action_typeN( ...
# no else statement

[George Sakkis]

John Machin wrote:

Bruno Desthuilliers wrote:

<my humble opinion>
Python is dynamic, and fighting against the language is IMHO a really
bad idea. The only places where theres a real need for this kind of
stuff are when dealing with the "outside world" (IOW : inputs and
outputs). And then packages like formencode can do much more than mere
type-checking
</my humble opinion>

Agreed. The worst case I have seen:

An elaborate decorator (similar to the OP's) laboriously checks arg
types. That's IMHO not "consenting adults" territory. But it gets a
whole lot worse when it's applied to a method whose body is like this:

if isinstance(....
action_for_type1(...
# big snip
elif isinstance(...
action_typeN( ...
# no else statement

Ouch.. someone must have skipped his/her OO class...

[John Machin]

George Sakkis wrote:

John Machin wrote:

Bruno Desthuilliers wrote:

<my humble opinion>
Python is dynamic, and fighting against the language is IMHO a really
bad idea. The only places where theres a real need for this kind of
stuff are when dealing with the "outside world" (IOW : inputs and
outputs). And then packages like formencode can do much more than mere
type-checking
</my humble opinion>

Agreed. The worst case I have seen:

An elaborate decorator (similar to the OP's) laboriously checks arg
types. That's IMHO not "consenting adults" territory. But it gets a
whole lot worse when it's applied to a method whose body is like this:

if isinstance(....
action_for_type1(...
# big snip
elif isinstance(...
action_typeN( ...
# no else statement

Ouch.. someone must have skipped his/her OO class...

Quite possibly :-) but that's not the problem here.

The method in question might be called say emit_generic(self,
any_type_of obj) and so one bunch of isinstance calls is actually
needed, but not two bunches. So: lose the decorator and add an else and
a raise at the end.

There is a secondary problem: annoying the crap out of callers who
often know what type they have and want an emit_real which would take
an int, a long, or a float and an emit_strg and ... (yes, almost all
the possible types are that simple) which wouldn't go through even one
chain of isinstance calls.

[Duncan Booth]

"John Machin" <sj******@lexicon.netwrote:

if isinstance(....
action_for_type1(...
# big snip
elif isinstance(...
action_typeN( ...
# no else statement

Ouch.. someone must have skipped his/her OO class...

Quite possibly :-) but that's not the problem here.

The method in question might be called say emit_generic(self,
any_type_of obj) and so one bunch of isinstance calls is actually
needed, but not two bunches. So: lose the decorator and add an else and
a raise at the end.

There is a secondary problem: annoying the crap out of callers who
often know what type they have and want an emit_real which would take
an int, a long, or a float and an emit_strg and ... (yes, almost all
the possible types are that simple) which wouldn't go through even one
chain of isinstance calls.

I think the point that was being made was that the method's body should be
something like:

actions[type(arg)](...)

which not only avoids all of the isinstance calls but also the else and the
raise at the end.

[John Machin]

Duncan Booth wrote:

"John Machin" <sj******@lexicon.netwrote:

if isinstance(....
action_for_type1(...
# big snip
elif isinstance(...
action_typeN( ...
# no else statement

Ouch.. someone must have skipped his/her OO class...

Quite possibly :-) but that's not the problem here.

The method in question might be called say emit_generic(self,
any_type_of obj) and so one bunch of isinstance calls is actually
needed, but not two bunches. So: lose the decorator and add an else and
a raise at the end.

There is a secondary problem: annoying the crap out of callers who
often know what type they have and want an emit_real which would take
an int, a long, or a float and an emit_strg and ... (yes, almost all
the possible types are that simple) which wouldn't go through even one
chain of isinstance calls.

I think the point that was being made was that the method's body should be
something like:

actions[type(arg)](...)

which not only avoids all of the isinstance calls but also the else and the
raise at the end.

You are saying that you think that George thinks that they are teaching
efficient coding methods in OO classes??

[Bruno Desthuilliers]

George Sakkis a écrit :

John Machin wrote:

>>Bruno Desthuilliers wrote:

>>><my humble opinion>
Python is dynamic, and fighting against the language is IMHO a really
bad idea. The only places where theres a real need for this kind of
stuff are when dealing with the "outside world" (IOW : inputs and
outputs). And then packages like formencode can do much more than mere
type-checking
</my humble opinion>

Agreed. The worst case I have seen:

An elaborate decorator (similar to the OP's) laboriously checks arg
types. That's IMHO not "consenting adults" territory. But it gets a
whole lot worse when it's applied to a method whose body is like this:

if isinstance(....
action_for_type1(...
# big snip
elif isinstance(...
action_typeN( ...
# no else statement

Ouch.. someone must have skipped his/her OO class...

Depends... I've sometimes had to write such kind of horrors - at least
once: a wrapper class to ensure that all strings accessible directly *or
indirectly* (via a dict, list, method call, whatever...) would be
unicode strings...

But of course I didn't use decorators to type-check annything !-)

[Duncan Booth]

"John Machin" <sj******@lexicon.netwrote:

You are saying that you think that George thinks that they are
teaching efficient coding methods in OO classes??

No, but I hope they teach how to recognise patterns, and I imagine they
also teach something about refactoring to remove switches or long if/elif
chains. (My imagination may not, of course, bear any resemblance to real
life.)

[John Machin]

Duncan Booth wrote:

"John Machin" <sj******@lexicon.netwrote:

You are saying that you think that George thinks that they are
teaching efficient coding methods in OO classes??

No, but I hope they teach how to recognise patterns, and I imagine they
also teach something about refactoring to remove switches or long if/elif
chains. (My imagination may not, of course, bear any resemblance to real
life.)

My point is that efficent coding methods, recognition of patterns and
refactoring are *general* comp sci concepts, they are not special to
OO. Why do you hope / imagine such things about OO classes?

[Duncan Booth]

"John Machin" <sj******@lexicon.netwrote:

>
Duncan Booth wrote:

>"John Machin" <sj******@lexicon.netwrote:

You are saying that you think that George thinks that they are
teaching efficient coding methods in OO classes??

No, but I hope they teach how to recognise patterns, and I imagine
they also teach something about refactoring to remove switches or
long if/elif chains. (My imagination may not, of course, bear any
resemblance to real life.)

My point is that efficent coding methods, recognition of patterns and
refactoring are *general* comp sci concepts, they are not special to
OO. Why do you hope / imagine such things about OO classes?

Both things are applicable outside OO, but the OO community is where they
evolved.

The concept of design patterns was popularized by the Gang of Four in
"Design Patterns: Elements of Reusable Object-Oriented Software", and I
believe that the concept of refactoring also grew out of the OO community.
According to Wikipedia, which is occasionally not far wrong:

The first known use of the term "refactoring" in the published
literature was in the article, Refactoring: An Aid in Designing
Application Frameworks and Evolving Object-Oriented Systems,
Proceedings of the Symposium on Object Oriented Programming
Emphasizing Practical Applications (SOOPPA) September, 1990, ACM by
William F. Opdyke and Ralph E. Johnson.

[Peter Wang]

Bruno Desthuilliers wrote:

<my humble opinion>
Python is dynamic, and fighting against the language is IMHO a really
bad idea. The only places where theres a real need for this kind of
stuff are when dealing with the "outside world" (IOW : inputs and
outputs). And then packages like formencode can do much more than mere
type-checking
</my humble opinion>

I don't think proper use of type checking is "fighting against the
language". The goal of any language is to enable programmers to
express their intent in a form that executes correctly.

Python is extremely expressive but there is a trade-off with
correctness - you can easily say something that you don't mean. Unit
testing is sometimes sufficient, but it can never span the infinite
space of potential errors. Type-checking method signatures guarantees
a certain amount of low-level correctness, and most type-checking
mechanisms also serve as documentation aids.

I think that with a sufficiently sophisticated type checking syntax,
one can get the best of both worlds. If the type checker understood
interfaces (like PyProtocols) and its syntax had the flexibility to
indicate sets of allowed arguments and aggregates of allowed
types/interfaces, it would cover the majority of cases without limiting
expressive power.

I understand that we're all adults, but it's still nice to have the
computer tell us when we're being childish. :)
-peter

[George Sakkis]

Duncan Booth wrote:

"John Machin" <sj******@lexicon.netwrote:

if isinstance(....
action_for_type1(...
# big snip
elif isinstance(...
action_typeN( ...
# no else statement

Ouch.. someone must have skipped his/her OO class...

Quite possibly :-) but that's not the problem here.

The method in question might be called say emit_generic(self,
any_type_of obj) and so one bunch of isinstance calls is actually
needed, but not two bunches. So: lose the decorator and add an else and
a raise at the end.

There is a secondary problem: annoying the crap out of callers who
often know what type they have and want an emit_real which would take
an int, a long, or a float and an emit_strg and ... (yes, almost all
the possible types are that simple) which wouldn't go through even one
chain of isinstance calls.

I think the point that was being made was that the method's body should be
something like:

actions[type(arg)](...)

which not only avoids all of the isinstance calls but also the else and the
raise at the end.

Actually my first thought was that the objects being typechecked belong
(or could be refactored so that they belong) in a hierarchy so that
each action_type_i() could be just an action() method of a subclass
(I've actually seen C++ code - apparently from [ex-]C programmers -
that uses a private 'kind' field and then builds long switch statements
based on 'kind' instead of proper subclassing).

But you're right, if the objects in question don't have (and cannot or
should not grow) an action() method (e.g. builtins), a dict dispatch
based on type is a reasonable choice, at least if you don't care about
handling automatically the derived classes. Otherwise you have to
simulate the mro(), using something like:

class TypeDispatcher(dict):

def __call__(self, obj, *args, **kwds):
try: # handle both old and new style classes
objtype = obj.__class__
except AttributeError:
objtype = type(obj)
for t in itermro(objtype):
if t in self:
return self[t](obj, *args, **kwds)
raise TypeError('No handler for %r' % objtype.__name__)
def itermro(cls):
from collections import deque
visited = set()
queue = deque([cls])
while queue:
cls = queue.popleft()
yield cls
for cls in cls.__bases__:
if cls not in visited:
visited.add(cls)
queue.append(cls)
if __name__ == '__main__':
class A(object):
def foo(self, *args, **kwds):
return "A.foo dispatcher(%s,%s)" % (args,kwds)

class B(A):
def foo(self, *args, **kwds):
return "B.foo dispatcher(%s,%s)" % (args,kwds)

d = TypeDispatcher()
d[A] = lambda self,*args,**kwds: self.foo(*args,**kwds)
d[int] = lambda self,*args,**kwds: "int dispatcher(%s, %s, %s)" %
(self, args, kwds)

for obj in A(), B(), 3:
print d(obj, "hello", "world", x=3, y=3.5)
George

[Fredrik Lundh]

Peter Wang wrote:

I understand that we're all adults, but it's still nice to have the
computer tell us when we're being childish. :)

like this?

def accepts(*types):
raise RuntimeError("don't be childish!")

def returns(rtype):
raise RuntimeError("don't be childish!")

</F>

[John Machin]

Peter Wang wrote:

Bruno Desthuilliers wrote:

<my humble opinion>
Python is dynamic, and fighting against the language is IMHO a really
bad idea. The only places where theres a real need for this kind of
stuff are when dealing with the "outside world" (IOW : inputs and
outputs). And then packages like formencode can do much more than mere
type-checking
</my humble opinion>

I don't think proper use of type checking is "fighting against the
language". The goal of any language is to enable programmers to
express their intent in a form that executes correctly.

Python is extremely expressive but there is a trade-off with
correctness - you can easily say something that you don't mean. Unit
testing is sometimes sufficient, but it can never span the infinite
space of potential errors. Type-checking method signatures guarantees
a certain amount of low-level correctness, and most type-checking
mechanisms also serve as documentation aids.

I think that with a sufficiently sophisticated type checking syntax,
one can get the best of both worlds. If the type checker understood
interfaces (like PyProtocols) and its syntax had the flexibility to
indicate sets of allowed arguments and aggregates of allowed
types/interfaces, it would cover the majority of cases without limiting
expressive power.

I understand that we're all adults, but it's still nice to have the
computer tell us when we're being childish. :)

Your comments on the following cut-down and disguised version of a
*real-world* example would be appreciated:

@accepts(object, (int, float))
def tally(self, anobj):
self.total += anobj

I assure you that the comments of a caller whose code did this:
fubar.tally(someobj)
and got this:
AssertionError: arg 12345678901L does not match (<type 'int'>,
<type 'float'>)
are definitely *not* repeatable in front of the children.

[George Sakkis]

John Machin wrote:

Peter Wang wrote:

Bruno Desthuilliers wrote:

<my humble opinion>
Python is dynamic, and fighting against the language is IMHO a really
bad idea. The only places where theres a real need for this kind of
stuff are when dealing with the "outside world" (IOW : inputs and
outputs). And then packages like formencode can do much more than mere
type-checking
</my humble opinion>

I don't think proper use of type checking is "fighting against the
language". The goal of any language is to enable programmers to
express their intent in a form that executes correctly.

Python is extremely expressive but there is a trade-off with
correctness - you can easily say something that you don't mean. Unit
testing is sometimes sufficient, but it can never span the infinite
space of potential errors. Type-checking method signatures guarantees
a certain amount of low-level correctness, and most type-checking
mechanisms also serve as documentation aids.

I think that with a sufficiently sophisticated type checking syntax,
one can get the best of both worlds. If the type checker understood
interfaces (like PyProtocols) and its syntax had the flexibility to
indicate sets of allowed arguments and aggregates of allowed
types/interfaces, it would cover the majority of cases without limiting
expressive power.

I understand that we're all adults, but it's still nice to have the
computer tell us when we're being childish. :)

Your comments on the following cut-down and disguised version of a
*real-world* example would be appreciated:

@accepts(object, (int, float))
def tally(self, anobj):
self.total += anobj

I assure you that the comments of a caller whose code did this:
fubar.tally(someobj)
and got this:
AssertionError: arg 12345678901L does not match (<type 'int'>,
<type 'float'>)
are definitely *not* repeatable in front of the children.

I guess this is not the 'accepts' decorator of the typecheck module;
with that you'd rather write it as @accepts(Self(), Number) and prevent
this error (http://oakwinter.com/code/typecheck/.../methods.html).

I have also a very recent real-world example to share, from the other
side of the fence this time. It's even worse because it's an error that
passes silently. Cut-down version follows:

@cherrypy.expose
def retrieve(self, **kwds):
queries = kwds['q']
rows = self._selectRows(*queries)
# more stuff

'q' here is a multiselect field that is binded to a list of selected
strings. Or so I thought, until someone noticed bizarre results in some
cases. Turns out that if you select a single item from the select box,
'q' is binded to a string instead of a list of length 1, so instead of
retrieving 'apple', she got back the results for 'a', 'p', 'p',
'l','e'.

Bottom line, type checking is a tricky business. In some sense it's
similar to the recall/precision tradeoff in information retrieval. With
stricter type checking, you can increase the precision but may hurt the
recall, i.e. valid code is rejected, as in your example. And vice
versa, loosing up the type checks increases the recall but may hurt the
precision, as in my example.

George

[Bruno Desthuilliers]

George Sakkis a écrit :

John Machin wrote:

>>Peter Wang wrote:

>>>Bruno Desthuilliers wrote:

<my humble opinion>
Python is dynamic, and fighting against the language is IMHO a really
bad idea. The only places where theres a real need for this kind of
stuff are when dealing with the "outside world" (IOW : inputs and
outputs). And then packages like formencode can do much more than mere
type-checking
</my humble opinion>

(snip)

I have also a very recent real-world example to share, from the other
side of the fence this time. It's even worse because it's an error that
passes silently. Cut-down version follows:

@cherrypy.expose
def retrieve(self, **kwds):
queries = kwds['q']
rows = self._selectRows(*queries)
# more stuff

'q' here is a multiselect field that is binded to a list of selected
strings. Or so I thought, until someone noticed bizarre results in some
cases. Turns out that if you select a single item from the select box,
'q' is binded to a string instead of a list of length 1, so instead of
retrieving 'apple', she got back the results for 'a', 'p', 'p',
'l','e'.

This is a typical case of converting/validating data from the "outside
world" - something well covered by formencode.

Bottom line, type checking is a tricky business.

Indeed. In fact, proper "type checking" would first require some
agreement on what's a "type"...