473,324 Members | 2,581 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,324 software developers and data experts.

Art of Unit Testing

In August 2001, there was a thread about the "Art of Unit Testing":
http://groups.google.com/group/comp....a29faf0a0485d5

Paul Moore asked the legitimate question why there is no hook for a
"global" fixture code that is run only once for the whole TestCase, as
opposed to the normal "setUp" and "tearDown" code that is run for every
single test in the TestCase. A "global fixture" would be preferable
whenever creating the fixture is time consuming, e.g. you have to create
a database connection or even a whole database.

What would be the preferred solution for such global fixtures? Simply
create the database, run the TestCase, and drop the database?

Would it make sense to add "globaleSetup" and "globalTearDown" methods
to the TestCase class? I think at least it would not harm anybody. Where
should such proposals be submitted?

-- Christoph Zwerschke
Aug 2 '05 #1
38 3343
The py.test module offers setup/teardown hooks at the method, class and
module level. The scenario you're describing would be covered at the
module level. See the py.test documentation for more details:
<http://codespeak.net/py/current/doc/test.html#managing-test-state-across-test-modules-classes-and-methods>

Grig

====
http://agiletesting.blogspot.com

Aug 2 '05 #2
Christoph Zwerschke <zw*******@zuv.uni-heidelberg.de> writes:
Would it make sense to add "globaleSetup" and "globalTearDown" methods
to the TestCase class? I think at least it would not harm
anybody. Where should such proposals be submitted?


In general that's not such a good idea. If you build your tests like
that, it gets hard to know which test really went wrong, and you might
get the situation where the whole set of tests works, but depend on each
other in some way. (This can also happen for more obscure reasons, and
is worth looking out for whichever way you do it.)

So, rebuilding the environment for the each before every single test is
generally worth the overhead.

--
Björn Lindström <bk**@stp.lingfil.uu.se>
Student of computational linguistics, Uppsala University, Sweden
Aug 2 '05 #3
""Björn Lindström"" <bk**@stp.lingfil.uu.se> wrote in message
news:87************@lucien.dreaming...
Christoph Zwerschke <zw*******@zuv.uni-heidelberg.de> writes:
Would it make sense to add "globaleSetup" and "globalTearDown" methods
to the TestCase class? I think at least it would not harm
anybody. Where should such proposals be submitted?
In general that's not such a good idea. If you build your tests like
that, it gets hard to know which test really went wrong, and you might
get the situation where the whole set of tests works, but depend on each
other in some way. (This can also happen for more obscure reasons, and
is worth looking out for whichever way you do it.)

So, rebuilding the environment for the each before every single test is
generally worth the overhead.


Generally is not always. There are configuration issues that are
best dealt with once at the beginning of the test, and once at
the end, and that have absolutely nothing to do with the order
in which each elementary test runs.

When your customers keep asking for something, and you
keep telling them that they really don't want what they're
asking for, who's lisening, and who's being stubborn?

John Roth
Python Fit.

--
Björn Lindström <bk**@stp.lingfil.uu.se>
Student of computational linguistics, Uppsala University, Sweden


Aug 2 '05 #4
Thanks for the link, Grig. I wasn't aware of the py lib so far. The
possibility to create fixtures at the three different scopes is exactly
what I was looking for.

Anyway, I think it would be nice to have that feature in the standard
lib unittest as well. It should not be too hard to add setUpOnce and
tearDownOnce methods in addition to setUp and tearDown. Actually, I am
wondering that there doesn't seem to be any development progress since
unittest was included in the standard lib of Python 2.1 in August 2001.
I had expected that such an important module would be continually
improved and maintained. How come? So few people using unit tests? Or do
most people write their own testing code or use py.test?

-- Christoph
Aug 2 '05 #5
Björn Lindström wrote:
Would it make sense to add "globaleSetup" and "globalTearDown" methods
to the TestCase class?

In general that's not such a good idea.


I completely agree and I think it makes a lot of sense that unittest
calls setUp and tearDown for every single test. However, the fact that
this is *generally* the best way doesn't exclude the fact that there are
*exceptions* when it makes sense to setUp and tearDown not for every
test, e.g. when it is absolutely sure that the fixture cannot be
destroyed by the individual tests or when creating the fixture takes too
much time. I already gave the example of creating database connections
or even creating/importing whole databases. My question was, how do I
handle these cases with the standard lib unittest?

According to the "extreme programming" paradigm, testing should be done
several times a day. So a requirement for extreme programm is that tests
are fast enough. If the testing needs too much time, people are
discouraged to test often.

-- Christoph
Aug 2 '05 #6
Christoph Zwerschke wrote:
Thanks for the link, Grig. I wasn't aware of the py lib so far. The
possibility to create fixtures at the three different scopes is exactly
what I was looking for.

Anyway, I think it would be nice to have that feature in the standard
lib unittest as well. It should not be too hard to add setUpOnce and
tearDownOnce methods in addition to setUp and tearDown. Actually, I am
wondering that there doesn't seem to be any development progress since
unittest was included in the standard lib of Python 2.1 in August 2001.
I had expected that such an important module would be continually
improved and maintained. How come? So few people using unit tests? Or do
most people write their own testing code or use py.test?


Or because it is already close to perfection (at least in what it is
intended to do).

The unittest module is a 'port' of the JUnit framework for Java which has a
certain wellknown API and semantics. The same API is available for many
other languages, so it is probably a good idea to stick with it in order to
make people coming from other language feel more comfortable with Python.

Some (many?) people don't like the unittest module, because it is not very
pythonic - nothing to wonder as it has its root in the Java world. That's
probably one of the reasons why there are other (more pythonic) unittesting
frameworks for Python out there.

I prefer to use unittest (because this was the API the textbook was using
that introduced me to this topic) and I also had the problem of heavy setup
costs. I deal with it by using a script around my testsuite (instead of
calling just unittest.main()) that does the setup/teardown of the
environment.

--
Benjamin Niemann
Email: pink at odahoda dot de
WWW: http://www.odahoda.de/
Aug 2 '05 #7
Christoph Zwerschke wrote:
Björn Lindström wrote:

I already gave the example of creating database connections
or even creating/importing whole databases. My question was, how do I
handle these cases with the standard lib unittest?
I do not get why a unit test whould create/import a whole database. In
order to unit test a function / method one should use mock objects,
otherwise you may not be sure that your code is faulty. A mock object is
an object that let the object uder test think it is connected to a
database, but it is not really connected to one. This object is
programmed to return well know values (correct, incorrect and
exceptional ones). What you are talking about is more related to
interaction / load tests and not unit tests (at least to me who am not
an expert of unit testing).
According to the "extreme programming" paradigm, testing should be done
several times a day. So a requirement for extreme programm is that tests
are fast enough. If the testing needs too much time, people are
discouraged to test often.


Well as I said above, a unit test is dedicated to a single function or
method. So wehn you update a function / method, you should test the
whole class or module, but maybe not the whole application (or the
development task may have not been properly organized --splited between
several developers). As someones may not write several thousand lines a
day, unit tests should not be that long to run (well from my point of
view, which is again not the one of an expert of unit testing).

I do agree with Benjamin Niermann when he says: "Or because it is
already close to perfection (at least in what it is intended to do)."

unittest is for unittesting :-)

my 2 cents

--
rafi

"Imagination is more important than knowledge."
(Albert Einstein)
Aug 2 '05 #8
rafi wrote:
In order to unit test a function / method one should use mock objects,
otherwise you may not be sure that your code is faulty.


'should' may be too strong, 'may' may be better. In the meantime I found:

http://python-mock.sourceforge.net/

my 2 cents

--
rafi

"Imagination is more important than knowledge."
(Albert Einstein)
Aug 2 '05 #9
Christoph Zwerschke wrote:
I completely agree and I think it makes a lot of sense that unittest
calls setUp and tearDown for every single test. However, the fact that
this is *generally* the best way doesn't exclude the fact that there are
*exceptions* when it makes sense to setUp and tearDown not for every
test, e.g. when it is absolutely sure that the fixture cannot be
destroyed by the individual tests or when creating the fixture takes too
much time. I already gave the example of creating database connections
or even creating/importing whole databases. My question was, how do I
handle these cases with the standard lib unittest?
What's wrong with using Python's existing "global" support, and just
having your test case setUp() call a global setup routine which checks
whether that global setup work has already been done and, if not, does
it once and sets a flag to say that it has now been done? I've done
this easily in the few cases where I've wanted this behaviour. It
doesn't seem complex enough to warrant adding to the standard unit test
support.
According to the "extreme programming" paradigm, testing should be done
several times a day. So a requirement for extreme programm is that tests
are fast enough. If the testing needs too much time, people are
discouraged to test often.


If you're going to quote XP rules of thumb, the tests should be
independent and very fast, and if you have a setup code that is taking a
long time, it's likely a "code smell" of some kind, and you should be
fixing the design which prevents you writing these tests with minimal
and quick setup. Are these really like "acceptance" tests? If they
were unit tests, they should take only a few minutes to run, total, and
you should be running them all *many* times a day, not twice.

Still, if you're stuck, and really do need a lengthy setup to execute
once as an optimization, it's not hard to do with globals.

-Peter
Aug 2 '05 #10
Benjamin Niemann wrote:
The unittest module is a 'port' of the JUnit framework for Java which has a
certain wellknown API and semantics. The same API is available for many
other languages, so it is probably a good idea to stick with it in order to
make people coming from other language feel more comfortable with Python.
Ok, that's a good reason. I just had a look at the JUnit homepage. It
seems like unittest has only implemented a part of JUnit. But you're
right, even JUnit lacks setUpOnce and tearDownOnce. I found the
following excuse and workaround in the JUnit FAQ:
http://junit.sourceforge.net/doc/faq/faq.htm#organize_3
Some (many?) people don't like the unittest module, because it is not very
pythonic - nothing to wonder as it has its root in the Java world. That's
probably one of the reasons why there are other (more pythonic) unittesting
frameworks for Python out there.
So I think it would have been better that "unittest" had been named
"PUnit" to make clear that it is a JUnit port and to allow a more
pythonic testing framework to be added to the Python's standard lib.
I prefer to use unittest (because this was the API the textbook was using
that introduced me to this topic) and I also had the problem of heavy setup
costs. I deal with it by using a script around my testsuite (instead of
calling just unittest.main()) that does the setup/teardown of the
environment.


Yes, I think that's the simplest solution.

-- Christoph
Aug 3 '05 #11
Peter Hansen wrote:
What's wrong with using Python's existing "global" support, and just
having your test case setUp() call a global setup routine which checks
whether that global setup work has already been done and, if not, does
it once and sets a flag to say that it has now been done? I've done
this easily in the few cases where I've wanted this behaviour. It
doesn't seem complex enough to warrant adding to the standard unit test
support.
Actually I already thought about doing it that way, but then I thought
it is so ugly, there must be a simpler solution ;-)
If you're going to quote XP rules of thumb, the tests should be
independent and very fast, and if you have a setup code that is taking a
long time, it's likely a "code smell" of some kind, and you should be
fixing the design which prevents you writing these tests with minimal
and quick setup. Are these really like "acceptance" tests? If they
were unit tests, they should take only a few minutes to run, total, and
you should be running them all *many* times a day, not twice.


You're right. I think wanting to have a more global initialization
indicates that you are acutally not wanting to do a "unit" test, but a
more global test of the overall system, something like an acceptance or
integration test, i.e. you are trying to abuse unittest for something it
was not intended to be used for.

Maybe since unittest is the only testing framework included with the
standard lib, people tend to use it for all testing purposes. If you
only have a hammer, everything looks like a nail.

-- Christoph
Aug 3 '05 #12
Christoph Zwerschke wrote:
Benjamin Niemann wrote:
Some (many?) people don't like the unittest module, because it is not
very pythonic - nothing to wonder as it has its root in the Java world.
That's probably one of the reasons why there are other (more pythonic)
unittesting frameworks for Python out there.


So I think it would have been better that "unittest" had been named
"PUnit" to make clear that it is a JUnit port and to allow a more
pythonic testing framework to be added to the Python's standard lib.


It was called PyUnit before it was integrated into the stdlib. Dunno why it
was renamed...

--
Benjamin Niemann
Email: pink at odahoda dot de
WWW: http://www.odahoda.de/
Aug 3 '05 #13
rafi wrote:
'should' may be too strong, 'may' may be better. In the meantime I found:
http://python-mock.sourceforge.net/


Thanks for the link. Björn also pointed to http://pmock.sourceforge.net

Using mock objects sounds like a good idea.

A problem with mock objects may be that they make writing tests for the
occasional programmer yet another bit more difficult, and that you
always have to ensure your mock objects really mock the real objects
perfectly, so you have to write another test for that. The behavior and
the API of the real objects may change every now and then.

Let me summarize some good answers in this thread:

- unittest is deliberately intended to be a JUnit implementation
- unittest is for *unit* testing (only) ;-)
- use mock objects to mimic the behaviour of external components like
databases
- other more pythonic testing frameworks are out there

I think it would be good to have these remarks added to the unittest
documentation in the Python library reference.

Also, if one of the more pythonic unit testing modules will be mature
enough and widely accepted, I think it would be good to make it Python's
standard (lib) testing framework and rename the current unittest back to
pyunit or punit.

-- Christoph
Aug 3 '05 #14
On Tue, 02 Aug 2005 21:11:52 +0200, Christoph Zwerschke <ci**@online.de> wrote:
Thanks for the link, Grig. I wasn't aware of the py lib so far. The
possibility to create fixtures at the three different scopes is exactly
what I was looking for.

Anyway, I think it would be nice to have that feature in the standard
lib unittest as well. It should not be too hard to add setUpOnce and
tearDownOnce methods in addition to setUp and tearDown. Actually, I am
wondering that there doesn't seem to be any development progress since
unittest was included in the standard lib of Python 2.1 in August 2001.
I had expected that such an important module would be continually
improved and maintained. How come? So few people using unit tests? Or do
most people write their own testing code or use py.test?


I can answer this question, at least for myself.

I use regression testing a lot, and have done so since before
Python's unittest was written.

Originally I just used my own code, which didn't use classes just
functions that called other functions. later on I decided to have a
look at unittest. I found two problems with it.

Firstly it didn't
stop when it reached an error, it continued doing the rewsr of the
tests. This wasn't what i wanted: most of the time when a test
fails, I want to look at what's happening. I don't want unnecessary
information about other tests, I want to concentrate on that one
thing.

Also, the log of information sent to stdout about each test was less
informative than I wanted (and which my old system provided).

So I had a look at unittest to see if I could modify it to fix these
problems. However, I found the code to be rather complex and hard to
understand so I decided it would be quicker to write my own testing
framework. Which I did.
--
Email: zen19725 at zen dot co dot uk
Aug 3 '05 #15
On Wed, 03 Aug 2005 09:35:08 +0200, Christoph Zwerschke <ci**@online.de> wrote:
Some (many?) people don't like the unittest module, because it is not very
pythonic - nothing to wonder as it has its root in the Java world. That's
probably one of the reasons why there are other (more pythonic) unittesting
frameworks for Python out there.


So I think it would have been better that "unittest" had been named
"PUnit" to make clear that it is a JUnit port and to allow a more
pythonic testing framework to be added to the Python's standard lib.


Not a bad idea.
--
Email: zen19725 at zen dot co dot uk
Aug 3 '05 #16
On Tue, 02 Aug 2005 21:26:28 +0200, Christoph Zwerschke <ci**@online.de> wrote:
Björn Lindström wrote:
Would it make sense to add "globaleSetup" and "globalTearDown" methods
to the TestCase class?

In general that's not such a good idea.


I completely agree and I think it makes a lot of sense that unittest
calls setUp and tearDown for every single test. However, the fact that
this is *generally* the best way doesn't exclude the fact that there are
*exceptions* when it makes sense to setUp and tearDown not for every
test, e.g. when it is absolutely sure that the fixture cannot be
destroyed by the individual tests or when creating the fixture takes too
much time. I already gave the example of creating database connections
or even creating/importing whole databases. My question was, how do I
handle these cases with the standard lib unittest?

According to the "extreme programming" paradigm, testing should be done
several times a day. So a requirement for extreme programm is that tests
are fast enough. If the testing needs too much time, people are
discouraged to test often.


Indeed. Running the tests should ideally take less than a few
seconds. Any longer, and people won't use them so often.

--
Email: zen19725 at zen dot co dot uk
Aug 3 '05 #17
On Tue, 02 Aug 2005 19:02:09 +0200, Björn Lindström <bk**@stp.lingfil.uu.se> wrote:
Christoph Zwerschke <zw*******@zuv.uni-heidelberg.de> writes:
Would it make sense to add "globaleSetup" and "globalTearDown" methods
to the TestCase class? I think at least it would not harm
anybody. Where should such proposals be submitted?
In general that's not such a good idea. If you build your tests like
that, it gets hard to know which test really went wrong,


No it isn't, provided you've coded your setup code and tests well.

And if you haven't coded your setup code and tests well, you're
fucked anyway.

Programming languages and libraries should assume the programmers is
competent and knows what he's doing. People who disagree should
stick to Java or Pascal or something.
and you might
get the situation where the whole set of tests works, but depend on each
other in some way. (This can also happen for more obscure reasons, and
is worth looking out for whichever way you do it.)
Sure. And if the programmer is competent, he'll bear that in mind
when he's writing the tests.
So, rebuilding the environment for the each before every single test is
generally worth the overhead.


Note "generally". Sometimes it isn't.

--
Email: zen19725 at zen dot co dot uk
Aug 3 '05 #18
On Tue, 02 Aug 2005 23:13:08 +0200, rafi <ra**@free.fr> wrote:
According to the "extreme programming" paradigm, testing should be done
several times a day. So a requirement for extreme programm is that tests
are fast enough. If the testing needs too much time, people are
discouraged to test often.


Well as I said above, a unit test is dedicated to a single function or
method. So wehn you update a function / method, you should test the
whole class or module, but maybe not the whole application


Often this is true. However sometimes the application is complex
with all sorts of dependencies between different parts of it (I
know, these should be kept to a minimum), and changes in one thing
can break stuff in another module. When that happens, you want to
know about it sooner rather than later.

--
Email: zen19725 at zen dot co dot uk
Aug 3 '05 #19
On Tue, 02 Aug 2005 18:44:01 +0200, Christoph Zwerschke <zw*******@zuv.uni-heidelberg.de> wrote:
In August 2001, there was a thread about the "Art of Unit Testing":
http://groups.google.com/group/comp....a29faf0a0485d5

Paul Moore asked the legitimate question why there is no hook for a
"global" fixture code that is run only once for the whole TestCase, as
opposed to the normal "setUp" and "tearDown" code that is run for every
single test in the TestCase. A "global fixture" would be preferable
whenever creating the fixture is time consuming, e.g. you have to create
a database connection or even a whole database.

What would be the preferred solution for such global fixtures? Simply
create the database, run the TestCase, and drop the database?

Would it make sense to add "globaleSetup" and "globalTearDown" methods
to the TestCase class? I think at least it would not harm anybody. Where
should such proposals be submitted?


I think this cowuld be very useful. As to the name, do other xUnit
testing frameworks have a common name? If not, may I suggest
setupAll() and tearDownAll().
--
Email: zen19725 at zen dot co dot uk
Aug 3 '05 #20
On Wed, 03 Aug 2005 10:19:05 +0200, Christoph Zwerschke <ci**@online.de> wrote:
rafi wrote:
'should' may be too strong, 'may' may be better. In the meantime I found:
http://python-mock.sourceforge.net/


Thanks for the link. Björn also pointed to http://pmock.sourceforge.net

Using mock objects sounds like a good idea.

A problem with mock objects may be that they make writing tests for the
occasional programmer yet another bit more difficult, and that you
always have to ensure your mock objects really mock the real objects
perfectly, so you have to write another test for that. The behavior and
the API of the real objects may change every now and then.

Let me summarize some good answers in this thread:

- unittest is deliberately intended to be a JUnit implementation
- unittest is for *unit* testing (only) ;-)


I use regression testing (using my "lintest" module, which is
functionally similar to unittest) on the whole application. I really
see no reason why the concept should be limited to testing
individual modules (or roughly similar-sized pieces).

--
Email: zen19725 at zen dot co dot uk
Aug 3 '05 #21
On Tue, 02 Aug 2005 17:18:51 -0400, Peter Hansen <pe***@engcorp.com> wrote:

If you're going to quote XP rules of thumb, the tests should be
independent and very fast, and if you have a setup code that is taking a
long time, it's likely a "code smell" of some kind, and you should be
fixing the design which prevents you writing these tests with minimal
and quick setup. Are these really like "acceptance" tests? If they
were unit tests, they should take only a few minutes to run, total,


Eek! Seconds, more like.

--
Email: zen19725 at zen dot co dot uk
Aug 3 '05 #22
Christoph Zwerschke wrote:
- unittest is for *unit* testing (only) ;-)
Why would you say that? We've used it extensively for a wide ranging of
testing, not limited to unit tests.
- use mock objects to mimic the behaviour of external components like
databases
....when doing unit testing. If you mock while doing higher level
testing, you'll eventually regret it in most cases.
Also, if one of the more pythonic unit testing modules will be mature
enough and widely accepted, I think it would be good to make it Python's
standard (lib) testing framework and rename the current unittest back to
pyunit or punit.


What, and break all the code that currently does "import unittest"?

-Peter
Aug 3 '05 #23
Christoph Zwerschke wrote:
I think wanting to have a more global initialization
indicates that you are acutally not wanting to do a "unit" test, but a
more global test of the overall system, something like an acceptance or
integration test, i.e. you are trying to abuse unittest for something it
was not intended to be used for.


As I mentioned in another post just now, we use unittest for acceptance
testing and other kinds of high level testing. We also try very very
hard to follow XP rules of thumb in terms of keeping our setup code
short and fast. As a result, we generally don't need to do what you are
finding you need to do, and unittest is quite adequate. Don't try to
make the point that unittest cannot do acceptance testing; it's _your_
acceptance tests which are the issue here, and I agree unittest was not
designed specifically to do exactly what you want. It also wasn't
designed to prevent it, but really just to be uninvolved in this issue.

The whole JUnit family wasn't designed to solve all testing problems,
but (I infer, from long use) to be a fairly simple and therefore general
approach. It doesn't have a huge array of features supporting all
possible approaches to testing, but it's simple enough not to prevent
you from doing what you need to do in most cases.

If you want a test framework with built-in support for what you need,
that's fine. In programming, one size doesn't fit all.

-Peter
Aug 3 '05 #24
phil hunt wrote:
On Tue, 02 Aug 2005 21:26:28 +0200, Christoph Zwerschke <ci**@online.de> wrote:
According to the "extreme programming" paradigm, testing should be done
several times a day. So a requirement for extreme programm is that tests
are fast enough. If the testing needs too much time, people are
discouraged to test often.


Indeed. Running the tests should ideally take less than a few
seconds. Any longer, and people won't use them so often.


That's probably too general a statement to be very useful, and I don't
think it's probably helpful to a newcomer who is seeking guidance.

If one makes the XP-style distinction between "unit" and "acceptance"
tests, then the acceptance test *suite* should take no more than some
number of minutes. I believe ten minutes absolute max is a commonly
offered rule of thumb, but I've found even eight minutes a little too
long. Five seems to be just fine, and less if of course wonderful.

For unit tests, the entire suite should take only a minute or two, max,
if possible, and preferably well under a minute. If you're doing
test-driven development (TDD), you will want to be able to run unit
tests (though maybe not all of them) sometimes a couple of times per
minute! If the overhead is fifteen seconds, you can't type much new
code in between running the tests.

Still, in a large project (and especially one written in Python, with
the overhead of interpreter startup and the cost of executing bytecode)
the suite can get fairly long if you have many hundreds of tests. In
that case, there's nothing that says you can't run individual test
files, or even individual test cases, when you are focused on one small
area of the code. That should let you run tests in only a couple of
seconds, as phil just recommended, in almost any case.

Having your code organized nicely into packages can help as well. If
you have a helpful "test runner" utility which scans subdirectories for
test cases, you can easily run all the tests in a given package
independently of the rest of the app while you are working on that one
package, reducing the risk inherent in running only a subset of your
full test suite.

-Peter
Aug 3 '05 #25
"Christoph Zwerschke" <ci**@online.de> wrote in message
news:dc**********@online.de...
Peter Hansen wrote:
What's wrong with using Python's existing "global" support, and just
having your test case setUp() call a global setup routine which checks
whether that global setup work has already been done and, if not, does it
once and sets a flag to say that it has now been done? I've done this
easily in the few cases where I've wanted this behaviour. It doesn't
seem complex enough to warrant adding to the standard unit test support.
Actually I already thought about doing it that way, but then I thought it
is so ugly, there must be a simpler solution ;-)
If you're going to quote XP rules of thumb, the tests should be
independent and very fast, and if you have a setup code that is taking a
long time, it's likely a "code smell" of some kind, and you should be
fixing the design which prevents you writing these tests with minimal and
quick setup. Are these really like "acceptance" tests? If they were
unit tests, they should take only a few minutes to run, total, and you
should be running them all *many* times a day, not twice.


You're right. I think wanting to have a more global initialization
indicates that you are acutally not wanting to do a "unit" test, but a
more global test of the overall system, something like an acceptance or
integration test, i.e. you are trying to abuse unittest for something it
was not intended to be used for.

Maybe since unittest is the only testing framework included with the
standard lib, people tend to use it for all testing purposes. If you only
have a hammer, everything looks like a nail.


An excellent point, even though it's somewhat blunted by the
inclusion of doctest in the core.

Another point to ponder is that the originators of xUnit, Kent
Beck and Erich Gamma, did not intend it to be used for acceptance
testing. That ecological niche is filled by FIT. The Python port of
Fit can be acquired from the file libraries of the FitNessse or
Extremeprogramming Yahoo mailing lists. I'll get it onto the
CheeseShop (what used to be the Python Package Index) one
of these days Real Soon Now. (Arlo gave me a good shove at
Agile2005 in that direction.)

The FIT developers are working on standardizing the specs,
and expensive setups and teardown are definitely on the agenda.
In fact, that's one of my work items...

Unit testing usually does not require expensive setups and teardowns,
at least if you're not working on real tangled legacy code.
Acceptance testing does.

Meanwhile, there's a book and two web sites: fit.c2.com and
www.fitnesse.org to look at.

John Roth
Python Fit
-- Christoph


Aug 3 '05 #26
On Wed, 03 Aug 2005 09:51:49 -0400, Peter Hansen <pe***@engcorp.com> wrote:
phil hunt wrote:
On Tue, 02 Aug 2005 21:26:28 +0200, Christoph Zwerschke <ci**@online.de> wrote:
According to the "extreme programming" paradigm, testing should be done
several times a day. So a requirement for extreme programm is that tests
are fast enough. If the testing needs too much time, people are
discouraged to test often.
Indeed. Running the tests should ideally take less than a few
seconds. Any longer, and people won't use them so often.


That's probably too general a statement to be very useful, and I don't
think it's probably helpful to a newcomer who is seeking guidance.


I did say "ideally"; if the test have to take longer, so be it.
If one makes the XP-style distinction between "unit" and "acceptance"
tests, then the acceptance test *suite* should take no more than some
number of minutes. I believe ten minutes absolute max is a commonly
offered rule of thumb, but I've found even eight minutes a little too
long. Five seems to be just fine, and less if of course wonderful.
I think we might be talking at cross purposes here. To me
"acceptance test suite" means a test suite that has to be passed
each time before a new version of the software is released to the
users. I don't see that 10 minutes is a sensible limit here, unless
you are releasing more often that once a day. (I once had an
acceptance test suite that ran for 12 hours; I used to run it
nightly).
If you're doing
test-driven development (TDD), you will want to be able to run unit
tests (though maybe not all of them) sometimes a couple of times per
minute!
That's what I often do.
Still, in a large project (and especially one written in Python, with
the overhead of interpreter startup and the cost of executing bytecode)
the suite can get fairly long if you have many hundreds of tests.


I'm currently running one with 227 assertions in 38 test functions.
It runs in <1 second on an AMD 3000+.
--
Email: zen19725 at zen dot co dot uk
Aug 3 '05 #27
>> - unittest is for *unit* testing (only) ;-)
Why would you say that? We've used it extensively for a wide ranging...
That was actually only a quote from this thread that summarizes some of
the answers I got: unittest has no support for "global" fixtures,
because it is intended for unit testing, not for global tests.
What, and break all the code that currently does "import unittest"?


Just insert the two words "punit as" between "import" and "unittest",
and the code will run as before...

-- Christoph
Aug 3 '05 #28
Benjamin Niemann wrote:
Christoph Zwerschke wrote:

Benjamin Niemann wrote:
Some (many?) people don't like the unittest module, because it is not
very pythonic - nothing to wonder as it has its root in the Java world.
That's probably one of the reasons why there are other (more pythonic)
unittesting frameworks for Python out there.


So I think it would have been better that "unittest" had been named
"PUnit" to make clear that it is a JUnit port and to allow a more
pythonic testing framework to be added to the Python's standard lib.

It was called PyUnit before it was integrated into the stdlib. Dunno why it
was renamed...


unittest describes exactly what it does.

pyunit says that it is in Python (duh), and that it has something to do
with units, which could be a whole number of things.

I'm thankful that logging is called logging as well, rather than log4py.
--
Michael Hoffman
Aug 3 '05 #29
Christoph Zwerschke wrote:
- unittest is for *unit* testing (only) ;-)


Why would you say that? We've used it extensively for a wide ranging...


That was actually only a quote from this thread that summarizes some of
the answers I got: unittest has no support for "global" fixtures,
because it is intended for unit testing, not for global tests.


Fair enough, as a quote, but it's still an inaccurate statement as I
hoped I've made clear in the last several posts. The need to do
"global" fixtures is most definitely not inherent in acceptance-type
testing, so it's incorrect to say "unittest" is limited to unit testing.
What, and break all the code that currently does "import unittest"?


Just insert the two words "punit as" between "import" and "unittest",
and the code will run as before...


No thanks. I have hundreds upon hundreds of test files and no desire to
change them. Luckily there's roughly a zero chance of this actually
happening, so no problem. :-)

-Peter
Aug 3 '05 #30
phil hunt wrote:
I think we might be talking at cross purposes here. To me
"acceptance test suite" means a test suite that has to be passed
each time before a new version of the software is released to the
users. I don't see that 10 minutes is a sensible limit here, unless
you are releasing more often that once a day. (I once had an
acceptance test suite that ran for 12 hours; I used to run it
nightly).


We're clearly on different wavelengths. I thought Extreme Programming
was sort of the context, where the acceptance tests actually *are* run
many times during the day, as well as before each new release. (A new
release could technically occur more than once in a day as well, so it's
a good thing if the tests take less than 12 hours.)
Still, in a large project (and especially one written in Python, with
the overhead of interpreter startup and the cost of executing bytecode)
the suite can get fairly long if you have many hundreds of tests.


I'm currently running one with 227 assertions in 38 test functions.
It runs in <1 second on an AMD 3000+.


That's a nice start. ;-)

-Peter
Aug 3 '05 #31
Michael Hoffman wrote:
Benjamin Niemann wrote:
Christoph Zwerschke wrote:

Benjamin Niemann wrote:

Some (many?) people don't like the unittest module, because it is not
very pythonic - nothing to wonder as it has its root in the Java world.
That's probably one of the reasons why there are other (more pythonic)
unittesting frameworks for Python out there.

So I think it would have been better that "unittest" had been named
"PUnit" to make clear that it is a JUnit port and to allow a more
pythonic testing framework to be added to the Python's standard lib.

It was called PyUnit before it was integrated into the stdlib. Dunno why
it was renamed...


unittest describes exactly what it does.

pyunit says that it is in Python (duh), and that it has something to do
with units, which could be a whole number of things.


XUnit (with X being the preferred prefix for the programming language) is a
common and wellknown name for a certain kind of unittesting framework. Of
course there are some people around who know what unittesting is but never
heard of JUnit and it decendents. But a quick textsearch on the TOC of the
library reference would reveal it.
Anyway, it too late now.
I'm thankful that logging is called logging as well, rather than log4py.


If someone is heavily using log4j and thinks about moving to Python, she
will be happy to see that her preferred logging API is available for Python
without even leaving the TOC of the library reference. log4X is a similar
case as XUnit.
(Is logging an implementation of the log4X API? Never used log4X...)

--
Benjamin Niemann
Email: pink at odahoda dot de
WWW: http://www.odahoda.de/
Aug 3 '05 #32
Christoph Zwerschke wrote:
I wasn't aware of the py lib so far. The
possibility to create fixtures at the three different scopes is exactly
what I was looking for.

Anyway, I think it would be nice to have that feature in the standard
lib unittest as well. It should not be too hard to add setUpOnce and
tearDownOnce methods in addition to setUp and tearDown. Actually, I am
wondering that there doesn't seem to be any development progress since
unittest was included in the standard lib of Python 2.1 in August 2001.
I had expected that such an important module would be continually
improved and maintained. How come? So few people using unit tests? Or do
most people write their own testing code or use py.test?


Voice of Mr. Unittest: """
The details of the implementation are not nearly as important as the
test cases. If you can support a set of test cases like the ones given
here, then you can write tests that are isolated and can be composed,
and you will be on your way to being able to develop test-first.
. . .
The spirit of xUnit is simplicity.
. . .
Some of the implementations have gotten a little complicated for my
taste. """ -- Kent Beck, Test Driven Development, p. 119.

Python's unittest module is already a bit to complicated. Building it
out further would be a step in the wrong direction. As it stands now,
it is somewhat powerful and flexible without being overly difficult to
learn.

FWIW, it does not take much skill to use the existing unittest module
to meet advanced testing needs. Lib/test/test_decimal.py has few lines
that dynamically build a whole suite of tests from a directory of text
resource files. Lib/test/test_compiler.py has just a few lines to
shorten testing time by running a random subset of tests.
Lib/test/test_deque.py has a handful of lines to run the suite multiple
times to check reference counts and it incorporates a doctest suite to
verify the examples in the documentation.

Your own feature request for setUpOnce() and tearDownOnce() is
trivially coded using a global or class variable to restrict running to
a single occasion. If that seems unpleasant, then encapsulate the
logic in a subclass of TestCase, in a decorator, or in a metaclass.

If you want better, try py lib's py.test module which is both more
minimalistic and more powerful.

Raymond
P.S. The above is based on the experience of writing thousands of
lines of test code. The existing unittest module is just fine without
further buildout. If it weren't for backwards compatibility issues, I
would recommend pairing it down further -- only a small subset of its
features are necessary to meet most needs.

Aug 4 '05 #33
Benjamin Niemann <pi**@odahoda.de> writes:
Some (many?) people don't like the unittest module, because it is not very
pythonic - nothing to wonder as it has its root in the Java world. That's
probably one of the reasons why there are other (more pythonic) unittesting
frameworks for Python out there.


I knew there was some other one before unittest came along but I thought
unittest was supposed to replace the older stuff.

What's the preferred one, Pythonically speaking?

I've been using unittest a little, but it's pretty clumsy, and the docs
aren't great.
Aug 4 '05 #34

"Paul Rubin" <"http://phr.cx"@NOSPAM.invalid> wrote in message
news:7x************@ruckus.brouhaha.com...
I knew there was some other one before unittest came along but I thought
unittest was supposed to replace the older stuff.
I believe unittest was an alternative rather than replacement for doctest.
What's the preferred one, Pythonically speaking?


py.test was written, apparently, by pypy folks to replace unittest for pypy
testing. To me, it is more Pythonic in spirit, and I plan to try it for an
upcoming TDD project.

Terry J. Reedy

Aug 4 '05 #35
On Wed, 03 Aug 2005 21:19:42 +0200, Benjamin Niemann <pi**@odahoda.de>
declaimed the following in comp.lang.python:


If someone is heavily using log4j and thinks about moving to Python, she
will be happy to see that her preferred logging API is available for Python
without even leaving the TOC of the library reference. log4X is a similar
case as XUnit.
(Is logging an implementation of the log4X API? Never used log4X...)
So does it mean anything if more recent Java releases come with
a built-in logging facility rather than an add-in...

java.util.logging

-- ================================================== ============ <
wl*****@ix.netcom.com | Wulfraed Dennis Lee Bieber KD6MOG <
wu******@dm.net | Bestiaria Support Staff <
================================================== ============ <
Home Page: <http://www.dm.net/~wulfraed/> <
Overflow Page: <http://wlfraed.home.netcom.com/> <

Aug 4 '05 #36
> Your own feature request for setUpOnce() and tearDownOnce() is
trivially coded using a global or class variable to restrict running to
a single occasion. If that seems unpleasant, then encapsulate the
logic in a subclass of TestCase, in a decorator, or in a metaclass.


Ok, you can have a setUpOnce() at module level that way, but how do you
realize tearDownOnce() that way? I think what you really have to do is
to modify the TestSuite class, not the TestCase class. Then you don't
have to fool around with global or class variables.

-- Christoph

import unittest

class MyTestCase(unittest.TestCase):
def setUp(self):
print "setUp",
def tearDown(self):
print "tearDown",
def test1(self):
print "test1"
def test2(self):
print "test2"

class MyTestSuite(unittest.TestSuite):
def setUp(self):
print "setUpAll",
def tearDown(self):
print "tearDownAll",
def run(self, result):
self.setUp()
unittest.TestSuite.run(self, result)
self.tearDown()

if __name__ == '__main__':
suite = unittest.makeSuite(MyTestCase, suiteClass=MyTestSuite)
unittest.TextTestRunner().run(suite)
Aug 4 '05 #37
"Terry Reedy" <tj*****@udel.edu> writes:
"Paul Rubin" <"http://phr.cx"@NOSPAM.invalid> wrote in message
news:7x************@ruckus.brouhaha.com...
I knew there was some other one before unittest came along but I thought
unittest was supposed to replace the older stuff.
I believe unittest was an alternative rather than replacement for doctest.


Around the time pyunit got added to the stdlib (as unittest) there
were some other candidates (one written by AMK et al at the MEMS
Exchange -- Sancho or something like that?), and pyunit got chosen by
the python-dev cabal, for reasons I don't recall now. It's probably
in the archives.
What's the preferred one, Pythonically speaking?


py.test was written, apparently, by pypy folks to replace unittest for pypy
testing.


That is a teeny bit inaccurate -- it's mostly Holger Krekel's work,
though his work on pypy was quite a lot of the inspiration. Armin
Rigo helped a lot, the other PyPy people less so, on average.
To me, it is more Pythonic in spirit, and I plan to try it for an
upcoming TDD project.


It's very cool, indeed.

Cheers,
mwh

--
Darn! I've only got 10 minutes left to get falling-down drunk! I
suppose I'll have to smoke crack instead now.
-- Tim Peters is checking things in on 2002-12-31
Aug 5 '05 #38
I had tested the above only with Python 2.4 but I just noticed it does
not work with Python 2.3. The following works also with Python 2.3:

import unittest

class MyTestCase(unittest.TestCase):
def setUp(self):
print "setUp",
def tearDown(self):
print "tearDown",
def test1(self):
print "test1"
def test2(self):
print "test2"

class MyTestSuite(unittest.TestSuite):
def setUp(self):
print "setUpAll",
def tearDown(self):
print "tearDownAll",
def __call__(self, result):
self.setUp()
unittest.TestSuite.__call__(self, result)
self.tearDown()

if __name__ == '__main__':
suite = unittest.makeSuite(MyTestCase, suiteClass=MyTestSuite)
unittest.TextTestRunner().run(suite)
Aug 5 '05 #39

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: Hugh Cowan | last post by:
Hello, I don't program full-time (anymore), but I do try and stay on-top of the latest technologies and like most are always trying to upgrade my skills and remain current (as much as is...
11
by: rhat | last post by:
Hi Everyone, I've recently been reading some articles about unit-testing in Python , but I am a bit confused: where do I go to get started with this? I tried googling for "unittest" but all I've...
14
by: | last post by:
Hi! I'm looking for unit-testing tools for .NET. Somthing like Java has --> http://www.junit.org regards, gicio
4
by: Peter Rilling | last post by:
Does VS.NET 2005 Professional support integrated unit testing, or is that only with the team system?
72
by: Jacob | last post by:
I have compiled a set og unit testing recommendations based on my own experience on the concept. Feedback and suggestions for improvements are appreciated: ...
4
by: Dat AU DUONG | last post by:
Hi, I am new to Unit testing, could you tell me where I could find information (hopefully step by step) and what is the benefit of unit testing. I am a sole developer in a company, therefore I...
5
by: shuisheng | last post by:
Dear All, I was told that unit test is a powerful tool for progamming. If I am writing a GUI code, is it possible to still using unit test? I have a little experience in using unittest++. But...
176
by: nw | last post by:
Hi, I previously asked for suggestions on teaching testing in C++. Based on some of the replies I received I decided that best way to proceed would be to teach the students how they might write...
48
by: Ark Khasin | last post by:
Unit testing is an integral component of both "formal" and "agile" models of development. Alas, it involves a significant amount of tedious labor. There are test automation tools out there but...
5
by: Ben Finney | last post by:
Howdy all, PEP 299 <URL:http://www.python.org/dev/peps/pep-0299details an enhancement for entry points to Python programs: a module attribute (named '__main__') that will be automatically called...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.