By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
435,530 Members | 2,181 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 435,530 IT Pros & Developers. It's quick & easy.

unittest: collecting tests from many modules?

P: n/a
I have a set of tests in different modules: test_foo.py, test_bar.py and so
on. All of these use the simplest possible internal layout: a number of
classes containing test*() methods, and the good old lines at the end:

if __name__ == "__main__":
unittest.main()

This is great, because each of the modules are runnable, and I can select
classes or tests to run on the commandline if I want to. However, running
all the tests from e.g. a Makefile is not that fun; I don't get a single
pass/fail summary across the modules.

What's the best way of creating a test.py which
- aggregates the tests from all the test_*.py modules?
- doesn't require me to enumerate all the test classes in test.py
(forcing each module to define test_foo.theSuite or someting would
be OK though)
- retains the ability to select tests and verbosity (-q, -v) from the
command line?

Something like:

import unittest
import test_foo
import test_bar

if __name__ == "__main__":
unittest.main(modules = ['test_foo',
'test_bar'])

Seems to me this should be possible, since all the logic for doing it /is/
there for the local module; I'd assume there would be a way to make unittest
search additional modules for test classes. But my head starts spinning
when I read the source code ...

/Jorgen

--
// Jorgen Grahn <jgrahn@ Ph'nglui mglw'nafh Cthulhu
\X/ algonet.se> R'lyeh wgah'nagl fhtagn!
Jul 19 '05 #1
Share this Question
Share on Google+
7 Replies


P: n/a
"Jorgen Grahn" <jg*********@algonet.se> wrote in message
news:sl************************@frailea.sa.invalid ...
I have a set of tests in different modules: test_foo.py, test_bar.py and so
on. All of these use the simplest possible internal layout: a number of
classes containing test*() methods, and the good old lines at the end:

if __name__ == "__main__":
unittest.main()

This is great, because each of the modules are runnable, and I can select
classes or tests to run on the commandline if I want to. However, running
all the tests from e.g. a Makefile is not that fun; I don't get a single
pass/fail summary across the modules.

What's the best way of creating a test.py which
- aggregates the tests from all the test_*.py modules?
- doesn't require me to enumerate all the test classes in test.py
(forcing each module to define test_foo.theSuite or someting would
be OK though)
- retains the ability to select tests and verbosity (-q, -v) from the
command line?
I use your second point: I build a TestAll module. However,
I'm going to look at building the test suite using the TestLoader
class in the next version of PyFit. It sholdn't be all that difficult
to find all the Test*.py modules in a directory, import them and
use the TestLoader class to add them to the test suite.

Or, for that matter, to use reflection to find all the classes that
derive from TestCase and add them to the suite manually. That
has the advantage that one could then select classes according
to some parameter.

John Roth


Something like:

import unittest
import test_foo
import test_bar

if __name__ == "__main__":
unittest.main(modules = ['test_foo',
'test_bar'])

Seems to me this should be possible, since all the logic for doing it /is/
there for the local module; I'd assume there would be a way to make
unittest
search additional modules for test classes. But my head starts spinning
when I read the source code ...

/Jorgen

--
// Jorgen Grahn <jgrahn@ Ph'nglui mglw'nafh Cthulhu
\X/ algonet.se> R'lyeh wgah'nagl fhtagn!


Jul 19 '05 #2

P: n/a
"Jorgen Grahn" wrote:
I have a set of tests in different modules: test_foo.py, test_bar.py and so
on. All of these use the simplest possible internal layout: a number of
classes containing test*() methods, and the good old lines at the end:

if __name__ == "__main__":
unittest.main()

This is great, because each of the modules are runnable, and I can select
classes or tests to run on the commandline if I want to. However, running
all the tests from e.g. a Makefile is not that fun; I don't get a single
pass/fail summary across the modules.

What's the best way of creating a test.py which
- aggregates the tests from all the test_*.py modules?
- doesn't require me to enumerate all the test classes in test.py
(forcing each module to define test_foo.theSuite or someting would
be OK though)
- retains the ability to select tests and verbosity (-q, -v) from the
command line?

Something like:

import unittest
import test_foo
import test_bar

if __name__ == "__main__":
unittest.main(modules = ['test_foo',
'test_bar'])

Seems to me this should be possible, since all the logic for doing it /is/
there for the local module; I'd assume there would be a way to make unittest
search additional modules for test classes. But my head starts spinning
when I read the source code ...


I had written a script to do something close to this; currently it
doesn't do any kind of aggregation, but it should be easy to extend it
as you like. What I don't like is the way it currently works: it
replaces sys.modules['__main__'] for each unit test and then it
execfile()s it, which seems like a hack. I didn't look into unittest's
internals in case there is a more elegant way around this; if there
isn't, a future version of unittest should address the automatic
aggregation of tests, as py.test does already.

The link to the script is http://rafb.net/paste/results/V0y16g97.html.

Hope this helps,
George

Jul 19 '05 #3

P: n/a
Jorgen Grahn wrote:
I have a set of tests in different modules: test_foo.py, test_bar.py and so
on. All of these use the simplest possible internal layout: a number of
classes containing test*() methods, and the good old lines at the end:
if __name__ == "__main__":
unittest.main()
....

What's the best way of creating a test.py which
- aggregates the tests from all the test_*.py modules?
- doesn't require me to enumerate all the test classes in test.py
(forcing each module to define test_foo.theSuite or someting would
be OK though)
- retains the ability to select tests and verbosity (-q, -v) from the
command line? Something like:

import unittest
import test_foo
import test_bar

if __name__ == "__main__":
unittest.main(modules = ['test_foo',
'test_bar'])

Seems to me this should be possible, since all the logic for doing it /is/
there for the local module; I'd assume there would be a way to make unittest
search additional modules for test classes. But my head starts spinning
when I read the source code ...

/Jorgen

How about some variant of:

import unittest
import test_foo, test_bar, ...

def set_globals(modules):
glbl = globals()
for module in modules:
modprefix = module.__name__[5:] + '__'
for element in dir(module):
data = getattr(module, element)
if isinstance(data, type) and issubclass(data,
unittest.TestCase):
glbl[modprefix + element] = data

if __name__ == "__main__":
module = type(unittest)
set_globals([mod for name, mod in globals().items()
if name.lower().beginswith('test')
and isinstance(mod, module)])
unittest.main()
--Scott David Daniels
Sc***********@Acm.Org
Jul 19 '05 #4

P: n/a
On 12 Jun 2005 08:06:18 -0700, "George Sakkis" <gs*****@rutgers.edu> wrote:
"Jorgen Grahn" wrote:
I have a set of tests in different modules: test_foo.py, test_bar.py and so
on. All of these use the simplest possible internal layout: a number of
classes containing test*() methods, and the good old lines at the end:

if __name__ == "__main__":
unittest.main()

This is great, because each of the modules are runnable, and I can select
classes or tests to run on the commandline if I want to. However, running
all the tests from e.g. a Makefile is not that fun; I don't get a single
pass/fail summary across the modules.

What's the best way of creating a test.py which
- aggregates the tests from all the test_*.py modules?
- doesn't require me to enumerate all the test classes in test.py
(forcing each module to define test_foo.theSuite or someting would
be OK though)
- retains the ability to select tests and verbosity (-q, -v) from the
command line?

Something like:

import unittest
import test_foo
import test_bar

if __name__ == "__main__":
unittest.main(modules = ['test_foo',
'test_bar'])

Seems to me this should be possible, since all the logic for doing it /is/
there for the local module; I'd assume there would be a way to make unittest
search additional modules for test classes. But my head starts spinning
when I read the source code ...


I had written a script to do something close to this; currently it
doesn't do any kind of aggregation, but it should be easy to extend it
as you like. What I don't like is the way it currently works: it
replaces sys.modules['__main__'] for each unit test and then it
execfile()s it, which seems like a hack. I didn't look into unittest's
internals in case there is a more elegant way around this; if there
isn't, a future version of unittest should address the automatic
aggregation of tests, as py.test does already.

The link to the script is http://rafb.net/paste/results/V0y16g97.html.

I think if you execfile a script and supply the global dict initialized
to {'__name__':'__main__'} then I think that will satisfy the if __name__ ... condition
and the whole thing will run as if executed interactively from the command line.
So IWT you could just loop through a list of test modules doing that.
Haven't tried it though. E.g., maybe there are nasty reload problems for
shared modules among different tests?

Regards,
Bengt Richter
Jul 19 '05 #5

P: n/a
"Jorgen Grahn" wrote:
What's the best way of creating a test.py which
- aggregates the tests from all the test_*.py modules?


You might want to check out the test runner in Zope 3
(svn://svn.zope.org/repos/main/Zope3/trunk/src/zope/app/testing)

It aggregates test reporting, does code coverage, lets you select
subsets of the tests to run, as well as control verbosity.

And, if you feel experimental you might want to preview the new Zope
test runner currently under development
(svn://svn.zope.org/repos/main/zope.testing).
--
Benji York
Jul 19 '05 #6

P: n/a
"Bengt Richter" <bo**@oz.net> wrote:
On 12 Jun 2005 08:06:18 -0700, "George Sakkis" <gs*****@rutgers.edu> wrote:
I had written a script to do something close to this; currently it
doesn't do any kind of aggregation, but it should be easy to extend it
as you like. What I don't like is the way it currently works: it
replaces sys.modules['__main__'] for each unit test and then it
execfile()s it, which seems like a hack. I didn't look into unittest's
internals in case there is a more elegant way around this; if there
isn't, a future version of unittest should address the automatic
aggregation of tests, as py.test does already.

I think if you execfile a script and supply the global dict initialized
to {'__name__':'__main__'} then I think that will satisfy the if __name__ ... condition
and the whole thing will run as if executed interactively from the command line.


Yes, execfile is called with {'__name__':'__main__'} for globals but
that's not enough because __main__ is already bound to the test.py
script, not the tested module. Even worse, after each test,
sys.modules['__main__'] seems to have to be reset to the original
__main__; otherwise os (and I guess other globals) is bound to None !
I'm not sure if this is normal or I missed something basic.

Regards,
George
PS: I reposted the script at
http://rafb.net/paste/results/3f1PIZ70.html.

Jul 19 '05 #7

P: n/a
On 12 Jun 2005 10:14:50 GMT, Jorgen Grahn <jg*********@algonet.se> wrote:

[regarding module unittest]
What's the best way of creating a test.py which
- aggregates the tests from all the test_*.py modules?
- doesn't require me to enumerate all the test classes in test.py
(forcing each module to define test_foo.theSuite or someting would
be OK though)
- retains the ability to select tests and verbosity (-q, -v) from the
command line?


Thanks for all the input. Three weeks later I stumble across this thread
again and notice I didn't report what I ended up with. I ended up doing the
thing I wanted to avoid in the first place: hardcoding everything.

import unittest

import test_bibdb
import test_citefmt
import test_person
import test_refsfmt

suite = unittest.TestSuite()
suite.addTest(test_bibdb.suite())
suite.addTest(test_citefmt.suite())
suite.addTest(test_person.suite())
suite.addTest(test_refsfmt.suite())

if __name__ == "__main__":
unittest.TextTestRunner(verbosity=2).run(suite)

My test modules are, after all, a fairly fixed set, and if I want to run
specific tests, or with a specific verbosity, I can execute the individual
test modules.

BR,
/Jorgen

--
// Jorgen Grahn <jgrahn@ Ph'nglui mglw'nafh Cthulhu
\X/ algonet.se> R'lyeh wgah'nagl fhtagn!
Jul 21 '05 #8

This discussion thread is closed

Replies have been disabled for this discussion.