Unittest - How do I code lots of simple tests

[Paul Moore]

One of the things I really dislike about Unittest (compared, say, to a
number of adhoc testing tricks I've used in the past, and to Perl's
"standard" testing framework) is that the testcase-as-a-class model
tends to imply a relatively high granularity in testing.

A good example of this comes from "Dive Into Python"
(http://diveintopython.org) section 7.3. Here, the author has written
a module which converts numbers to Roman numerals. The test case is
then

class KnownValues(uni ttest.TestCase) :
knownValues = ( (1, 'I'), ... # 50-plus specific cases

def testToRomanKnow nValues(self):
"""toRoman should give known result with known input"""
for integer, numeral in self.knownValue s:
result = roman.toRoman(i nteger)
self.assertEqua l(numeral, result)

Now, to my mind, the big problem here is that this *isn't* one test,
but rather 50 (or more). OK, the same checks are done if everything
succeeds, but

1. If a test fails, the rest are skipped! If there's a pattern to the
failures (the code handles numbers ending in 4 and 9 wrongly, for
example) it's much easier to find if all of the checks are
reported.
2. Psychologically , "52 tests succeeded" is a much bigger boost than
"1 test succeeded" (albeit this test covered 52 cases). And it's
not just psychology - we really *did* test 52 distinct conditions.

The only way I can see of producing the "true" number of tests is via
some form of ugly hack like

test_values = ((1, 'I'), ...)

class KnownValues(uni ttest.TestCase) :
pass

for arabic, roman in test_values:
def test(self):
result = roman.toRoman(a rabic)
self.assertEqua l(roman, result)
setattr(KnownVa lues, 'test_%s_%s' % (arabic, roman), test)

But I can't really see that as the "right approach".

Can anyone suggest a more reasonable way of running this sort of
table-driven test via unittest?

Maybe I should use doctest instead, but to be honest, I prefer the
overall infrastructure of unittest (for real tests). It's just this
particular issue that really bugs me...

Paul.
--
This signature intentionally left blank

[Peter Hansen]

Paul Moore wrote:

Can anyone suggest a more reasonable way of running this sort of
table-driven test via unittest?

Why not just extend self.assertEqua l() and use your own check, with
additional logic as required to increment counters or add items
to the list of passing tests. Then put a final check of the number
of passing tests or something like that at the end to make sure
things worked overall.
For example:

class KnownValues(uni ttest.TestCase) :
def setUp(self):
self.passCount = 0

def checkValue(self , expected, result):
if expected == result:
self.passCount += 1
else:
# left as exercise to the reader, but pass would work...

def testToRomanKnow nValues(self):
for integer, numeral in self.knownValue s:
result = roman.toRoman(i nteger)
self.checkValue (numeral, result)
self.assertEqua l(len(self.know nValues), self.passCount)

No, you don't get the psychologically affirming "52 tests passed!"
without changes to the TestRunner, but I assume the non-cosmetic part
of this is more your concern right now...

-Peter

[David Goodger]

Paul Moore wrote:

Can anyone suggest a more reasonable way of running this sort of
table-driven test via unittest?

Take a look at Docutils' test/DocutilsTestSup port.py for ideas. The
CustomTestSuite and CustomTestCase classes provide support for named
data-driven tests. Most of Docutils' tests are data-driven.

Ian Bicking wrote: unittest is not written with subclassing in mind, except for the
limited subclassing that is documented. (And it uses
double-underscore variables, like it's just *trying* to piss me off!
Double-underscore variables are so arrogant and patronizing.

All very true. Double-underscores ought to be banned from the
standard library. They inevitably get in the way because no matter
how well a class is written, somebody is going to want to subclass it
in a way the original author never considered.

--
David Goodger http://starship.python.net/~goodger
For hire: http://starship.python.net/~goodger/cv
Docutils: http://docutils.sourceforge.net/
(includes reStructuredTex t: http://docutils.sf.net/rst.html)

[Jeremy Fincher]

Paul Moore <pf******@yahoo .co.uk> wrote in message news:<br******* ***@yahoo.co.uk >...

1. If a test fails, the rest are skipped! If there's a pattern to the
failures (the code handles numbers ending in 4 and 9 wrongly, for
example) it's much easier to find if all of the checks are
reported.

That's true, but most of the time when I test, I prefer that behavior.
Many of my asserts depend on the success of the assert prior to
them, so I want the test to fail as soon as one of the asserts has
failed.

It'd be nice to have both possibilities, though.

Jeremy

[Miki Tebeka]

Hello Paul,

test_values = ((1, 'I'), ...)

class KnownValues(uni ttest.TestCase) :
pass

for arabic, roman in test_values:
def test(self):
result = roman.toRoman(a rabic)
self.assertEqua l(roman, result)
setattr(KnownVa lues, 'test_%s_%s' % (arabic, roman), test)

But I can't really see that as the "right approach".

On reason that it won't do what you want :-)
All the tests will check the last value in test_values. (Something to
do with binding rules, can't recall how to solve)

Try:
--- i2r.py ---
#!/usr/bin/env python
from unittest import TestCase, makeSuite, main

class Roman:
def toRoman(self, i):
return { 1 : "I",
2 : "II",
3 : "III",
4 : "IV",
5 : "V"}[i]
roman = Roman()

class KnownValues(Tes tCase):
pass

test_values = ((1, "I"), (2, "II"), (3, "III"), (4, "IV"), (5, "V"))
for a, r in test_values:
def test(self):
print a, r
result = roman.toRoman(a )
self.assertEqua l(r, result)
setattr(KnownVa lues, "test_%s_%s " % (a, r), test)

test_suite = makeSuite(Known Values, "test_")

if __name__ == "__main__":
main()
--- i2r.py ---

$ python i2r.py
5 V
..5 V
..5 V
..5 V
..5 V
..
----------------------------------------------------------------------
Ran 5 tests in 0.000s

OK
HTH.
Miki

[Duncan Booth]

Paul Moore <pf******@yahoo .co.uk> wrote in news:br******** **@yahoo.co.uk:

But I can't really see that as the "right approach".

Can anyone suggest a more reasonable way of running this sort of
table-driven test via unittest?

Ok, how about the file below.
It uses a metaclass to generate dynamic tests from a table. I deliberately
wrote the tests so that one fails, when run it will tell you that
test_roman_v failed so you can see that the names get handled properly.

Output:
D:\temp>tablete st.py --verbose
testMe (__main__.MyTes ts) ... ok
test_roman_i (__main__.MyTes ts) ... ok
test_roman_v (__main__.MyTes ts) ... FAIL
test_roman_x (__main__.MyTes ts) ... ok

=============== =============== =============== =============== ==========
FAIL: test_roman_v (__main__.MyTes ts)
----------------------------------------------------------------------
Traceback (most recent call last):
File "D:\temp\tablet est.py", line 21, in test
self.assertEqua ls(roman, 'x')
File "D:\Python23\li b\unittest.py", line 302, in failUnlessEqual
raise self.failureExc eption, \
AssertionError: 'v' != 'x'

----------------------------------------------------------------------
Ran 4 tests in 0.020s

FAILED (failures=1)
Note that the metaclass itself is reusable. The factory
function tableDrivenTest s, although defined within the class is a function,
not a method, and cannot access any members of the class (since the class
does not yet exist at the time it is called). The table itself either has
to be created inside the tableDrivenTest s function, or global.
The factory function simply returns a dictionary of functions which are
added to the class, note that the keyname in the dictionary is important so
far as the unittest code is concerned, not the original name of the
function.

Also be sure to pass parameters into the test function using default
parameters as nested scopes will get the values left at the end of the loop
(so you might end up with lots of tests that all do the same thing). I
mention that here because I did exactly that writing the code.

---- begin tabletest.py ----
class MetaTableTest(t ype):
def __new__(metacls , name, bases, dict):
factory = dict['tableDrivenTes ts']
dict.update(fac tory())
return super(MetaTable Test, metacls).__new_ _(metacls, name, bases,
dict)

import unittest

class MyTests(unittes t.TestCase):
__metaclass__ = MetaTableTest
def tableDrivenTest s():
'''Return a dictionary of additional test functions'''
knownValues = (1,'i'), (5, 'v'), (10, 'x')
table = {}
for arabic, roman in knownValues:
def test(self, arabic=arabic, roman=roman):
if arabic==1:
self.assertEqua ls(roman, 'i')
else:
self.assertEqua ls(roman, 'x')

table['test_roman_%s' % roman] = test
return table

def testMe(self):
self.assert_(Tr ue)
if __name__=='__ma in__':
unittest.main()

---- end tabletest.py ----

--
Duncan Booth du****@rcp.co.u k
int month(char *p){return(1248 64/((p[0]+p[1]-p[2]&0x1f)+1)%12 )["\5\x8\3"
"\6\7\xb\1\x9\x a\2\0\4"];} // Who said my code was obscure?

[Peter Otten]

Miki Tebeka wrote:

Hello Paul,

test_values = ((1, 'I'), ...)

class KnownValues(uni ttest.TestCase) :
pass

for arabic, roman in test_values:
def test(self):
result = roman.toRoman(a rabic)
self.assertEqua l(roman, result)
setattr(KnownVa lues, 'test_%s_%s' % (arabic, roman), test)

But I can't really see that as the "right approach". On reason that it won't do what you want :-)
All the tests will check the last value in test_values. (Something to
do with binding rules, can't recall how to solve)

Try:
--- i2r.py ---
#!/usr/bin/env python
from unittest import TestCase, makeSuite, main

class Roman:
def toRoman(self, i):
return { 1 : "I",
2 : "II",
3 : "III",
4 : "IV",
5 : "V"}[i]
roman = Roman()

class KnownValues(Tes tCase):
pass

test_values = ((1, "I"), (2, "II"), (3, "III"), (4, "IV"), (5, "V"))
for a, r in test_values:
def test(self):

Change the above line to

def test(self, a=a, r=r):

or you will perform the test five times with (5, "V").
print a, r
result = roman.toRoman(a )
self.assertEqua l(r, result)
setattr(KnownVa lues, "test_%s_%s " % (a, r), test)

test_suite = makeSuite(Known Values, "test_")

if __name__ == "__main__":
main()
--- i2r.py ---

I like the idea, once the little error is removed. In general, I think the
unit test code should be as simple as possible. Otherwise we would need
unit tests for unit tests for...

Peter

[Paul Moore]

Peter Hansen <pe***@engcorp. com> wrote in message news:<3F******* ********@engcor p.com>...

Paul Moore wrote:

Can anyone suggest a more reasonable way of running this sort of
table-driven test via unittest?

Why not just extend self.assertEqua l() and use your own check, with
additional logic as required to increment counters or add items
to the list of passing tests. Then put a final check of the number
of passing tests or something like that at the end to make sure
things worked overall.
For example:

class KnownValues(uni ttest.TestCase) :
def setUp(self):
self.passCount = 0

def checkValue(self , expected, result):
if expected == result:
self.passCount += 1
else:
# left as exercise to the reader, but pass would work...

def testToRomanKnow nValues(self):
for integer, numeral in self.knownValue s:
result = roman.toRoman(i nteger)
self.checkValue (numeral, result)
self.assertEqua l(len(self.know nValues), self.passCount)

No, you don't get the psychologically affirming "52 tests passed!"
without changes to the TestRunner, but I assume the non-cosmetic part
of this is more your concern right now...

I don't see the remainder of the problem as "non-cosmetic". The error
report I get (or rather the information it offers) is

1 test failed - pass count is 25 instead of 52.

But that doesn't tell me *which* tests failed.

The key point here is that I'm NOT running one test - I really am
running 52 distinct tests. OK, they are all very similar, differing
only in the data - but for pity's sake, isn't that what an object
oriented structure is supposed to make easy???

Paul.

[Paul Moore]

David Goodger <go*****@python .org> wrote in message news:<ma******* *************** *************** @python.org>...

Paul Moore wrote:

>>> Can anyone suggest a more reasonable way of running this sort of
>>> table-driven test via unittest?

Take a look at Docutils' test/DocutilsTestSup port.py for ideas. The
CustomTestSuite and CustomTestCase classes provide support for named
data-driven tests. Most of Docutils' tests are data-driven.

Urk. That's hairy stuff. Thanks for the pointer, I'll do some research.

But I still think that this sort of thing should be easy :-(

Paul.

[Anthony Briggs]

At 3:11 AM -0700 10/22/03, Paul Moore wrote:

Peter Hansen <pe***@engcorp. com> wrote in message
news:<3F****** *********@engco rp.com>...

> > No, you don't get the psychologically affirming "52 tests passed!" without changes to the TestRunner, but I assume the non-cosmetic part
of this is more your concern right now...

I don't see the remainder of the problem as "non-cosmetic". The error
report I get (or rather the information it offers) is

1 test failed - pass count is 25 instead of 52.

But that doesn't tell me *which* tests failed.

Are you running the tests verbosely? eg. with a -v argument under
UNIX, or as specified in the docs
<http://www.python.org/doc/current/lib/minimal-example.html>? I get
the following output when using Miki's code (with a deliberate error
thrown in, and the print statement commented out):

bash-2.05b$ ./test.py -v
test_1_I (__main__.Known Values) ... ok
test_2_II (__main__.Known Values) ... ok
test_3_II (__main__.Known Values) ... FAIL
test_4_IV (__main__.Known Values) ... ok
test_5_V (__main__.Known Values) ... ok

=============== =============== =============== =============== ==========
FAIL: test_3_II (__main__.Known Values)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./test.py", line 21, in test
self.assertEqua l(r, result)
File "/usr/local/lib/python2.3/unittest.py", line 292, in failUnlessEqual
raise self.failureExc eption, \
AssertionError: 'II' != 'III'

----------------------------------------------------------------------
Ran 5 tests in 0.055s

FAILED (failures=1)
bash-2.05b$
The key point here is that I'm NOT running one test - I really am
running 52 distinct tests. OK, they are all very similar, differing
only in the data - but for pity's sake, isn't that what an object
oriented structure is supposed to make easy???

Well, you're testing one aspect of the code. It's really just a
question of how you think about your tests.

Anthony
--
----------------------------------------------------
HyPEraCtiVE? HeY, WhO aRE YoU cALliNg HypERaCtIve?!
aB*****@wEStNeT .cOm.aU
----------------------------------------------------