unit testing guidelines

[Jacob]

I have compiled a set og unit testing
recommendations based on my own experience
on the concept.

Feedback and suggestions for improvements
are appreciated:

http://geosoft.no/development/unittesting.html

Thanks.

[Hendrik Maryns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
NotDashEscaped: You need GnuPG to verify this message

Jacob uitte de volgende tekst op 03/18/2006 12:03 AM:

I have compiled a set og unit testing
recommendations based on my own experience
on the concept.

Feedback and suggestions for improvements
are appreciated:

http://geosoft.no/development/unittesting.html

Nice work.

I don't totally agree with point 16: a throws statement means an
exception *might* be thrown, and the circumstances under which this can
happen should be documented. It is seldom that an exception must be thrown.

You might want to give some explanation about what you assertX methods do.

H.
--
Hendrik Maryns

==================
www.lieverleven.be
http://aouw.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD4DBQFEHEUxe+7xMGD3itQRAsFwAKCBXX77fVjeMGJz3+AU0B xe/pYnoQCY5z2F
Ti5C4PCNnHJBSHbz3tp2jQ==
=VsD2
-----END PGP SIGNATURE-----

[Daniel T.]

In article <dv*************@news.t-online.com>,
Hendrik Maryns <he************@despammed.com> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
NotDashEscaped: You need GnuPG to verify this message

Jacob uitte de volgende tekst op 03/18/2006 12:03 AM:

I have compiled a set og unit testing
recommendations based on my own experience
on the concept.

Feedback and suggestions for improvements
are appreciated:

http://geosoft.no/development/unittesting.html

Nice work.

I don't totally agree with point 16: a throws statement means an
exception *might* be thrown, and the circumstances under which this can
happen should be documented. It is seldom that an exception must be thrown.

I agree. Only test what you actually want the client code to rely on.
Now if you want the client code to rely on the method throwing an
exception...
--
Magic depends on tradition and belief. It does not welcome observation,
nor does it profit by experiment. On the other hand, science is based
on experience; it is open to correction by observation and experiment.

[Ian Collins]

Jacob wrote:

I have compiled a set og unit testing
recommendations based on my own experience
on the concept.

Feedback and suggestions for improvements
are appreciated:

http://geosoft.no/development/unittesting.html

I'd add point 0 - write the tests first.

8 - names should be more expressive, rather than testSaveAs(), how about
a series of tests, testSaveAsCreatesANewFile(),
testSaveAsSavesCuentdataInNewFile() etc. Often tests with a broad name
attempt to test too much ad don't express their intent.

Point 0 covers point 11.

13 - take care with random numbers, they can lead to failures that are
hard to reproduce. I'd use a pseudo-random sequence that is repeatable
with a given seed.

0 and 8 covers 14.

0 covers 17.

0 covers 20.

--
Ian Collins.

[Timo Stamm]

Jacob schrieb:

I have compiled a set og unit testing
recommendations based on my own experience
on the concept.

Feedback and suggestions for improvements
are appreciated:

| 7. Keep tests close to the class being tested
|
| If the class to test is Foo the test class should be called FooTest
| and kept in the same package (directory) as Foo. The build environment
| must be configured so that the test classes doesn't make its way into
| production code.

It is necessary to have test classes in the same package as the tested
class in order to test package private methods.

But you don't have to put the classes in the same directory. Most IDEs
support several source folders. You can setup two source folders. For
example: "src" for your application source, "test" for your test source.
If you use the same package structure in the test source folder, you can
test package private methods and it is very easy to deploy only
application code.
Timo

[Ian Collins]

Timo Stamm wrote:

Jacob schrieb:

I have compiled a set og unit testing
recommendations based on my own experience
on the concept.

Feedback and suggestions for improvements
are appreciated:

| 7. Keep tests close to the class being tested
|
| If the class to test is Foo the test class should be called FooTest
| and kept in the same package (directory) as Foo. The build environment
| must be configured so that the test classes doesn't make its way into
| production code.

It is necessary to have test classes in the same package as the tested
class in order to test package private methods.

Another view that tests that require access to private methods are a
design smell. Often these can be refactored into objects that can be
tested in isolation.

In C++, it's very tempting to make the test class a friend of the class
under test. I've found that I end up with a better design by resisting
this temptation.

--
Ian Collins.

[Hendrik Maryns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
NotDashEscaped: You need GnuPG to verify this message

Ian Collins uitte de volgende tekst op 03/19/2006 12:13 AM:

Timo Stamm wrote:

Jacob schrieb:

I have compiled a set og unit testing
recommendations based on my own experience
on the concept.

Feedback and suggestions for improvements
are appreciated:

| 7. Keep tests close to the class being tested
|
| If the class to test is Foo the test class should be called FooTest
| and kept in the same package (directory) as Foo. The build environment
| must be configured so that the test classes doesn't make its way into
| production code.

It is necessary to have test classes in the same package as the tested
class in order to test package private methods.

Another view that tests that require access to private methods are a
design smell. Often these can be refactored into objects that can be
tested in isolation.

I was about to answer the same: shouldn't problems in package private
methods spill through to public methods? Then why test them separately?
Find an error in a public method and retrace it with you favorite
debugger to the package private method, I'd say (without much
experience, so correct me if I'm wrong).

H.

--
Hendrik Maryns

==================
www.lieverleven.be
http://aouw.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFEHJyze+7xMGD3itQRAuV5AJ9RMb89yiGxxknFqT7XCa TjHnJJ9QCfXrZ3
7iuZP99HYR0uR9FuffXLeLk=
=KKUv
-----END PGP SIGNATURE-----

[Bent C Dalager]

In article <dv*************@news.t-online.com>,
Hendrik Maryns <he************@despammed.com> wrote:

I was about to answer the same: shouldn't problems in package private
methods spill through to public methods? Then why test them separately?
It makes it more time-consuming to find out where the error is.
Find an error in a public method and retrace it with you favorite
debugger to the package private method, I'd say (without much
experience, so correct me if I'm wrong).

I prefer my unit tests to have obvious failure modes so that I can
basically tell from which test failed, exactly where in my source the
bug is. This means I don't have to muck around with a debugger, I can
just fix it and get on with things.

For this to be the case, however, the methods that I test need to be
reasonably small and not do a whole lot. These are my private helper
methods that I invoke from my more involved algorithm methods. Many
are one or two liners and they generally don't make sense to have
publicly accessible since they're really just internal building blocks
for constructing other more interesting methods.

Cheers
Bent D
--
Bent Dalager - bc*@pvv.org - http://www.pvv.org/~bcd
powered by emacs

[Timo Stamm]

Ian Collins schrieb:

Timo Stamm wrote:

It is necessary to have test classes in the same package as the tested
class in order to test package private methods.

Another view that tests that require access to private methods are a
design smell. Often these can be refactored into objects that can be
tested in isolation.

Not "private", but "package private".

Package private classes are only visible within the same package (same
directory). They are useful in large APIs where you have a lot of
functionality, but only want to expose a small interface.
Timo

[Timo Stamm]

Timo Stamm schrieb:

| 7. Keep tests close to the class being tested
|
| If the class to test is Foo the test class should be called FooTest
| and kept in the same package (directory) as Foo. The build environment
| must be configured so that the test classes doesn't make its way into
| production code.

It is necessary to have test classes in the same package as the tested
class in order to test package private methods.

But you don't have to put the classes in the same directory. Most IDEs
support several source folders. You can setup two source folders. For
example: "src" for your application source, "test" for your test source.
If you use the same package structure in the test source folder, you can
test package private methods and it is very easy to deploy only
application code.

Oops, I didn't realize that the guidelines aren't java-specific and that
this thread is on c.l.java.p as well as c.l.c++.

My objection is specific to java. I doubt that the same applies to c++.

[Ian Collins]

Timo Stamm wrote:

Ian Collins schrieb:

Timo Stamm wrote:

It is necessary to have test classes in the same package as the
tested class in order to test package private methods.

Another view that tests that require access to private methods are a
design smell. Often these can be refactored into objects that can be
tested in isolation.

Not "private", but "package private".

Package private classes are only visible within the same package (same
directory). They are useful in large APIs where you have a lot of
functionality, but only want to expose a small interface.

I see, a concept not shared with C++.

--
Ian Collins.

[Ian Collins]

Hendrik Maryns wrote:

| 7. Keep tests close to the class being tested
|
| If the class to test is Foo the test class should be called FooTest
| and kept in the same package (directory) as Foo. The build environment
| must be configured so that the test classes doesn't make its way into
| production code.

It is necessary to have test classes in the same package as the tested
class in order to test package private methods.

Another view that tests that require access to private methods are a
design smell. Often these can be refactored into objects that can be
tested in isolation.

I was about to answer the same: shouldn't problems in package private
methods spill through to public methods? Then why test them separately?
Find an error in a public method and retrace it with you favorite
debugger to the package private method, I'd say (without much
experience, so correct me if I'm wrong).

As Brent said, you are testing too much with your tests. A golden rule
is not to rely on indirect tests.

I've recently come to the conclusion (while working with PHP which
doesn't have a handy debugger) that resorting to the debugger is a
strong indicator that your tests aren't fine grained enough. Try working
without one for a while and see your tests improve!

--
Ian Collins.

[Adam Maass]

"Jacob" <ja***@yahoo.com> wrote:

I have compiled a set og unit testing
recommendations based on my own experience
on the concept.

Feedback and suggestions for improvements
are appreciated:

http://geosoft.no/development/unittesting.html

Thanks.

I strongly object to number 13. Unit-tests, especially in an automated
framework, should be repeatable. (When a test fails, you need to know on
what inputs it failed. Once you fix the failure, you should hard-code the
inputs it failed on so that subsequent changes do not cause a regression of
the error.)

I don't necessarily object to looping over large numbers of inputs and
testing each one for expected outputs. But a unit test should contain no
randomness at all. (Or at least should have a way of specifying the seed for
the randomness generator(s).)
-- Adam Maass

[Jacob]

Hendrik Maryns wrote:

I don't totally agree with point 16: a throws statement means an
exception *might* be thrown, and the circumstances under which this can
happen should be documented. It is seldom that an exception must be thrown.

I assume the conditions for when an exception is thrown
is deterministic and well documented (though it to a large
extent depend on *documentation* rather than language syntax
which is a problem as documentation is inherently inaccurate).

The simple example is the java List.get(int index) method that
is documented to throw an exception if index < 0. This is the
contract, and this is one of the things I want to test in a
unit test.

Recommendation 16 just indicate how this is done in practice.

[Jacob]

Ian Collins wrote:

I'd add point 0 - write the tests first.
Personaly find the XP approach to unit testing a bit too restrictive
and therefore left the issue intentionally open. I really like
more feedback on it though, as though I have practiced unit testing
for years, I never adopted this practice myself.
8 - names should be more expressive, rather than testSaveAs(), how about
a series of tests, testSaveAsCreatesANewFile(),
testSaveAsSavesCuentdataInNewFile() etc. Often tests with a broad name
attempt to test too much ad don't express their intent.
Agree. I think this is basically what's in #8 without being to
verbose.
Point 0 covers point 11.
I am not sure it does, and I wanted to define the two concepts
"execution coverage" and "test coverage" anyway. There is a blurred
distinction between the two in the literature as far as I have been
able to dig up.
13 - take care with random numbers, they can lead to failures that are
hard to reproduce. I'd use a pseudo-random sequence that is repeatable
with a given seed.

0 and 8 covers 14.
To some degree, but I'd include them even if #0 was there. I don't
see "testing first" as a silver bullet, but more as a different
process aproach.
0 covers 17.
Not necesserily. #0 states when to write the tests. #17 states that
the *code* should be written so that the workload of the unit testing
is minimized.
0 covers 20.

Yes, assuming everything is tested always. But in that case
it is covered without #0 as well. What I see in the industry today
is a major shift in adding unit testing to legacy code. I added
#20 as a suggestion to start this work at the bottom level.

[Jacob]

Ian Collins wrote:

Another view that tests that require access to private methods are a
design smell. Often these can be refactored into objects that can be
tested in isolation.

In C++, it's very tempting to make the test class a friend of the class
under test. I've found that I end up with a better design by resisting
this temptation.

This is my experience as well, and the reason why I added
recommendation #9 "Test public API".

That something is technically feasable (private method testing through
reflection or by other means) doesn't necesserily mean it is a good idea.

You need to draw the line somewhere, and the public API seems quite
natural in this case. This is also more robust agains changes, in that it
will be more stable and require less testing maintainance during code
refactoring.

[Jacob]

Timo Stamm wrote:

Ian Collins schrieb:

Timo Stamm wrote:

It is necessary to have test classes in the same package as the
tested class in order to test package private methods.

Another view that tests that require access to private methods are a
design smell. Often these can be refactored into objects that can be
tested in isolation.

Not "private", but "package private".

Package private classes are only visible within the same package (same
directory). They are useful in large APIs where you have a lot of
functionality, but only want to expose a small interface.

I regard this as "private" in this context. An error in the
inner logic between classes of the same package (or *friends*
in C++ syntax) will eventually reveal itself through the public
API.

I want to keep test classes close to the class being tested for
practical reasons rather than technical reasons.

I understand the objection of "testing too large chunks of code"
(Ian C.), but test code adds complexity and workload to your system
afterall, and I really want to keep it to a minimum. That's why I
reduce the public API of classes as much as possible (by heavy
use of package private methods for instance) and insist on testing
public API only.

But I don't clain that this is the only way, and it might well
depend on the nature of the project being tested.

[Jacob]

Adam Maass wrote:

I strongly object to number 13. Unit-tests, especially in an automated
framework, should be repeatable. (When a test fails, you need to know on
what inputs it failed. Once you fix the failure, you should hard-code the
inputs it failed on so that subsequent changes do not cause a regression of
the error.)

I understand your objection, but this is actually one of the
mechanisms that have helped me found some of the hardest to
trace and most subtle errors in the code. It has proven to be
extremely helpful. Also, it gives me lots of confidence
knowing that my test suite of several thousand tests
are executed every hour with different input each time.
It is like adding another dimension to unit testing.

But as tests must be reproducable I agree, I added #15 to ensure
that when a test fails, the test report will include the input
parameters if failed with exactly. Then you can add a test with
this explicit input and debug it from there.

[Ian Collins]

Jacob wrote:

Ian Collins wrote:

I'd add point 0 - write the tests first.

Personaly find the XP approach to unit testing a bit too restrictive
and therefore left the issue intentionally open. I really like
more feedback on it though, as though I have practiced unit testing
for years, I never adopted this practice myself.

TDD is more than an approach to unit testing, it is an approach to the
full design-test-code cycle.

8 - names should be more expressive, rather than testSaveAs(), how
about a series of tests, testSaveAsCreatesANewFile(),
testSaveAsSavesCuentdataInNewFile() etc. Often tests with a broad
name attempt to test too much ad don't express their intent.

Agree. I think this is basically what's in #8 without being to
verbose.

Point 0 covers point 11.

I am not sure it does, and I wanted to define the two concepts
"execution coverage" and "test coverage" anyway. There is a blurred
distinction between the two in the literature as far as I have been
able to dig up.

TDD done well will give you 100% execution coverage for free. How good
your test coverage is depends on how good you are at thinking up edge
cases to test.

13 - take care with random numbers, they can lead to failures that are
hard to reproduce. I'd use a pseudo-random sequence that is
repeatable with a given seed.

0 and 8 covers 14.

To some degree, but I'd include them even if #0 was there. I don't
see "testing first" as a silver bullet, but more as a different
process aproach.

Simple, incremental tests are the essence of good TDD.

0 covers 17.

Not necesserily. #0 states when to write the tests. #17 states that
the *code* should be written so that the workload of the unit testing
is minimized.

If you start with the tests,the code will have to be written that way.

0 covers 20.

Yes, assuming everything is tested always. But in that case
it is covered without #0 as well. What I see in the industry today
is a major shift in adding unit testing to legacy code. I added
#20 as a suggestion to start this work at the bottom level.

Very true.

--
Ian Collins.

[Andrew McDonagh]

Ian Collins wrote:

Jacob wrote:

Ian Collins wrote:

I'd add point 0 - write the tests first.

Personaly find the XP approach to unit testing a bit too restrictive
and therefore left the issue intentionally open. I really like
more feedback on it though, as though I have practiced unit testing
for years, I never adopted this practice myself.

TDD is more than an approach to unit testing, it is an approach to the
full design-test-code cycle.

More fundamentally, TDD is Design Methodology, Not a Testing Methodology.

It just happens to use Unit tests as its means of describing the design,
much like RUP uses UML.

Indeed, some TDD practitioners are starting to call it BDD - as in

http://www.google.co.uk/search?hl=en...e+Search&meta=

8 - names should be more expressive, rather than testSaveAs(), how
about a series of tests, testSaveAsCreatesANewFile(),
testSaveAsSavesCuentdataInNewFile() etc. Often tests with a broad
name attempt to test too much ad don't express their intent.

Agree. I think this is basically what's in #8 without being to
verbose.

Point 0 covers point 11.

I am not sure it does, and I wanted to define the two concepts
"execution coverage" and "test coverage" anyway. There is a blurred
distinction between the two in the literature as far as I have been
able to dig up.

TDD done well will give you 100% execution coverage for free.

I'd clarify that with 'TDD done *Correctly will give you 100% execution
coverage'

*Correctly = Write 1 failing Testcase,
Write only enough code to make test Pass,
Refactor to Remove Duplication,
Repeat

More commonly referred to as Red, Green, Refactor.
How good your test coverage is depends on how good you are at thinking up edge
cases to test.
Always starting with the test first, only allows for 100%.

13 - take care with random numbers, they can lead to failures that
are hard to reproduce. I'd use a pseudo-random sequence that is
repeatable with a given seed.

0 and 8 covers 14.

To some degree, but I'd include them even if #0 was there. I don't
see "testing first" as a silver bullet, but more as a different
process aproach.

Simple, incremental tests are the essence of good TDD.

These kind of tests are unit tests as in the TDD usage - they are stress
tests that happen to be written in the same framework as the TDD unit
tests.

However, looping over a random set of numbers isn't the best approach to
this style of testing. If the OP wants to do this style, then using one
of the various Agitating frameworks/products will give a better result.

These tools tend to use byte code manipulation to random change various
values which aren't just numbers, but anything: int, long, float,
Integer, Double, String, Boolean, boolean, introducing Nulls, etc.

See http://www.agitar.com/

[Phlip]

Andrew McDonagh wrote:

Ian Collins wrote:

Jacob wrote:

Ian Collins wrote:

I'd add point 0 - write the tests first.
Personaly find the XP approach to unit testing a bit too restrictive

I find debugging a bit too restrictive. I can't just use Undo to make the
bug go >poof<.

Imagine if you had such a button on your debugger! You would hit it all the
time!

You have such a button; it's just a little more expensive than raw code. The
cost savings - no more debugging - overwhelmingly offsets that cost.
TDD is more than an approach to unit testing, it is an approach to the
full design-test-code cycle.

More fundamentally, TDD is Design Methodology, Not a Testing Methodology.

It just happens to use Unit tests as its means of describing the design,
much like RUP uses UML.

Indeed, some TDD practitioners are starting to call it BDD - as in

http://www.google.co.uk/search?hl=en...e+Search&meta=

And some call it Test First Programming, because TDD is position to replace
the hideous name "eXtreme Programming".

And it doesn't create "unit tests", which are a different topic entirely.

The failure of a unit test implicates only one unit - such as the Ariane V
engine controller.

The failure of a _Developer_ Test implicates the developer's last edit. Time
to hit Undo.

TDD done well will give you 100% execution coverage for free.

That's not exhaustive.

TDD done well will reduce the _odds_ that you need exhaustive unit testing.

--
Phlip
http://www.greencheese.org/ZeekLand <-- NOT a blog!!!

[Andrew McDonagh]

Phlip wrote:

Andrew McDonagh wrote:

Ian Collins wrote:

Jacob wrote:
Ian Collins wrote:

> I'd add point 0 - write the tests first.

Personaly find the XP approach to unit testing a bit too restrictive

Phlip, your mail client must have removed to much cause Ian wrote this
'too restrictive' part not me.

I find debugging a bit too restrictive. I can't just use Undo to make the
bug go >poof<.

Imagine if you had such a button on your debugger! You would hit it all the
time!

You have such a button; it's just a little more expensive than raw code. The
cost savings - no more debugging - overwhelmingly offsets that cost.
100% agree and its how I work too - make a change, it doesn't work, undo
change, make a different change, rinse & repeat.

TDD is more than an approach to unit testing, it is an approach to the
full design-test-code cycle.

More fundamentally, TDD is Design Methodology, Not a Testing Methodology.

It just happens to use Unit tests as its means of describing the design,
much like RUP uses UML.

Indeed, some TDD practitioners are starting to call it BDD - as in

http://www.google.co.uk/search?hl=en...e+Search&meta=

And some call it Test First Programming, because TDD is position to replace
the hideous name "eXtreme Programming".

;-) If only Kent had his way

[Phlip]

Andrew McDonagh wrote:

Phlip wrote:

Andrew McDonagh wrote:

Ian Collins wrote:
Jacob wrote:
> Ian Collins wrote:
>
>> I'd add point 0 - write the tests first.
>
> Personaly find the XP approach to unit testing a bit too restrictive
Phlip, your mail client must have removed to much cause Ian wrote this
'too restrictive' part not me.
When I snip, I leave in the replied-to person, and sometimes leave in the
previous repliers. But I don't change the >>>, so count them.

(We must hope readers understand the ticks and don't come after you for
others' heretical statements...)
100% agree and its how I work too - make a change, it doesn't work, undo
change, make a different change, rinse & repeat.
If we could put that feeling into a dartgun...
And some call it Test First Programming, because TDD is position to
replace the hideous name "eXtreme Programming".

;-) If only Kent had his way

Uh, it's his idea, and it's why XPX2 uses TFP instead of TDD.

--
Phlip
http://www.greencheese.org/ZeekLand <-- NOT a blog!!!

[Ian Collins]

Andrew McDonagh wrote:

I am not sure it does, and I wanted to define the two concepts
"execution coverage" and "test coverage" anyway. There is a blurred
distinction between the two in the literature as far as I have been
able to dig up.

TDD done well will give you 100% execution coverage for free.

I'd clarify that with 'TDD done *Correctly will give you 100% execution
coverage'

*Correctly = Write 1 failing Testcase,
Write only enough code to make test Pass,
Refactor to Remove Duplication,
Repeat

More commonly referred to as Red, Green, Refactor.

How good your test coverage is depends on how good you are at thinking
up edge cases to test.

Always starting with the test first, only allows for 100%.

I was using the OP's definition of "test coverage". It might just be
me, but I've always had testers or users (normally testers) find some
bizarre use case that wasn't catered for in the original user stories or
unit tests.

--
Ian Collins.

[Phlip]

Ian Collins wrote:

I was using the OP's definition of "test coverage". It might just be me,
but I've always had testers or users (normally testers) find some bizarre
use case that wasn't catered for in the original user stories or unit
tests.

That's why, regardless of your unit testing strategy, you work to lower the
cost of acceptance tests, so anyone can write them, and they come up with
all sorts of things.

Hence all of XP is driven by tests.

--
Phlip
http://www.greencheese.org/ZeekLand <-- NOT a blog!!!

[Andrew McDonagh]

Ian Collins wrote:

Andrew McDonagh wrote:

Always starting with the test first, only allows for 100%.

I was using the OP's definition of "test coverage". It might just be
me, but I've always had testers or users (normally testers) find some
bizarre use case that wasn't catered for in the original user stories or
unit tests.

But we are talking about unit testing here - developers write and run
unit tests.

Users/testers don't unit test - they Acceptance (integration, System) Test.

Everyone runs the acceptance tests.

[n2xssvv g02gfr12930]

Jacob wrote:

I have compiled a set og unit testing
recommendations based on my own experience
on the concept.

Feedback and suggestions for improvements
are appreciated:

http://geosoft.no/development/unittesting.html

Thanks.

At no point do you suggest that both the code design and testing should
be subject to independent review, preferably by another programmer and a
user. This can often help spot potential problems early on. The
developer can frequently miss something, if only because their too close
and as the cliche goes "You can't see the wood for the trees".

Hope you approve, from an unemployed C++ developer

JB

[Adam Maass]

"Jacob" <ja***@yahoo.com> wrote:

Adam Maass wrote:

I strongly object to number 13. Unit-tests, especially in an automated
framework, should be repeatable. (When a test fails, you need to know on
what inputs it failed. Once you fix the failure, you should hard-code the
inputs it failed on so that subsequent changes do not cause a regression
of the error.)

I understand your objection, but this is actually one of the
mechanisms that have helped me found some of the hardest to
trace and most subtle errors in the code. It has proven to be
extremely helpful. Also, it gives me lots of confidence
knowing that my test suite of several thousand tests
are executed every hour with different input each time.
It is like adding another dimension to unit testing.

There is sometimes value in testing on large numbers of random inputs. But
this isn't *unit* testing; it's more akin to a system or stress test. It's
something you hope your QAs will do for you; test on inputs that you weren't
necessarily expecting and see what breaks. Unit testing is about the
correctness of code for known inputs. If you come across a failure for a
novel set of inputs in system or stress testing, by all means, take that
input and add it to your unit test suite.

Note that test frameworks can be used both for unit tests as well as other
kinds of tests. (Simply because it's called 'JUnit', for example, does not
necessarily mean that all the test cases are, in fact, unit tests.)

-- Adam Maass

[Jacob]

Adam Maass wrote:

There is sometimes value in testing on large numbers of random inputs. But
this isn't *unit* testing; it's more akin to a system or stress test. It's
something you hope your QAs will do for you; test on inputs that you weren't
necessarily expecting and see what breaks. Unit testing is about the
correctness of code for known inputs.

Which definition of unit testing is this? I have searched the
net but hasn't been able to find any backing for this?

If I write a method void setLength(double length), who define
the input "necesserily expected", and why isn't this the entire
double range? I'd claim the latter and to cover as many inputs
as possible I use the random trick.

I don't have a problem with defining this kind of testing
differently, for instance "stress testing", but on the other
hand there isn't really any more "stress" is calling
setLength(1.23076e+307) than setLength(2.0) as long as the
method accepts a double as input?

And why do you care about "known" input as long as the
actual (failing) input can be traced afterwards anyway?

You define this as a unit test:

for (int i = 0; i < 1000; i++)
testMyIntMethod(i);

while this is not:

for (int i = 0; i < 1000; i++)
testMyIntMethod(getRandomInt())

even if an error on input=42 will produce identical error reports
in both cases. Only the latter will (eventually) reveal the
error for input=-100042.

Also, if I have a setLength() method which cover the "typical"
input cases just fine, but is in general crap (a common scenario),
then a testSetLength() method that verifies that setLength() work
fine for "typical" input isn't worth a lot. What you need is a test
method that test the non-typical inputs. From a black-box perspective
you don't really know what is typical or non-typical, so why not just
throw a random number genrator at it?

[Ben Pope]

Jacob wrote:

Adam Maass wrote:

There is sometimes value in testing on large numbers of random inputs.
But this isn't *unit* testing; it's more akin to a system or stress
test. It's something you hope your QAs will do for you; test on inputs
that you weren't necessarily expecting and see what breaks. Unit
testing is about the correctness of code for known inputs.
Which definition of unit testing is this? I have searched the
net but hasn't been able to find any backing for this?

If I write a method void setLength(double length), who define
the input "necesserily expected", and why isn't this the entire
double range? I'd claim the latter and to cover as many inputs
as possible I use the random trick.

The programmer specifies the preconditions. If the preconditions are
not met, there is no reason for it to produce valid results. Random is
not repeatable, and is not predictable.
I don't have a problem with defining this kind of testing
differently, for instance "stress testing", but on the other
hand there isn't really any more "stress" is calling
setLength(1.23076e+307) than setLength(2.0) as long as the
method accepts a double as input?
No, but the point is that when you unit test, you need to make informed
choices about the inputs you choose. It's usually wise to throw in a
couple of "normal", "everyday" values, but also explicitly check
boundary cases and out of range.
And why do you care about "known" input as long as the
actual (failing) input can be traced afterwards anyway?
Repeatability. It's no use relying on randomness to thoroughly test.
You have to design your test cases.
You define this as a unit test:

for (int i = 0; i < 1000; i++)
testMyIntMethod(i);
Not really. What are you testing? That it doesn't crash? Presumably
you need to check the output against an array of 1000 pre-computed
values? Not much fun.
while this is not:

for (int i = 0; i < 1000; i++)
testMyIntMethod(getRandomInt())
How can you possibly check the output is correct for a random input?
even if an error on input=42 will produce identical error reports
in both cases. Only the latter will (eventually) reveal the
error for input=-100042.
I don't understand, are you checking for crashing?
Also, if I have a setLength() method which cover the "typical"
input cases just fine, but is in general crap (a common scenario),
then a testSetLength() method that verifies that setLength() work
fine for "typical" input isn't worth a lot. What you need is a test
method that test the non-typical inputs. From a black-box perspective
you don't really know what is typical or non-typical, so why not just
throw a random number genrator at it?

You know your preconditions. You know your postconditions.

If anything can happen on invalid input, then no point in testing. If
you want a default output, exception or whatever for out-of-range, then
check it with a unit test. You get to choose your input and you get to
check your output.

Randomness just doesn't cut it, and I don't understand how you can check
the output is correct, without knowing the input.

Ben Pope
--
I'm not just a number. To many, I'm known as a string...

[Jacob]

Ben Pope wrote:

Randomness just doesn't cut it, and I don't understand how you can check
the output is correct, without knowing the input.

You *do* know the input!

Consider testing this method:

double square(double v)
{
return v * v;
}

Below is a typical unit test that verifies that the
method behaves correctly on typical input:

double v = 2.0;
double v2 = square(v); // You know the input: It is 2.0!
assertEquals(v2, 4.0);

The same test using random input:

double v = getRandomDouble();
double v2 = square(v); // You know the input: It is v!
assertEquals(v2, v*v);

If the test fails, all the details will be in the error
report.

And this method actually *do* fail for a mjority of all
possible inputs (abs of v exceeding sqrt(maxDouble)).
This will be revealed instantly using the random approach.

For an experienced programmer the limitation of square()
might be obvious so border cases are probably covered
sufficiently in both the code and the test. But for more
complex logic this might not be this apparent and throwing
in random input (in ADDITION to the typical cases and all
obvious border cases) has proven quite helpful, at least
to me.

[Tom Leylan]

Jacob:

You've chosen a trivial example where your assert can compute the results of
the Square() function you are calling. That is hardly a typical situation
or there would be no reason for the function to have been created.

double v = getRandomDouble();
double v2 = AccountBalance( v );
assertEquals( v2, ? );

So explain how you get the value to type in ? given you don't know what the
input will be. Perhaps you would do the computations you read about in the
AccountBalance() method inline to see if those and yours matched?
"Jacob" <ja***@yahoo.com> wrote in message
news:Jt********************@telenor.com...

Ben Pope wrote:

Randomness just doesn't cut it, and I don't understand how you can check
the output is correct, without knowing the input.

You *do* know the input!

Consider testing this method:

double square(double v)
{
return v * v;
}

Below is a typical unit test that verifies that the
method behaves correctly on typical input:

double v = 2.0;
double v2 = square(v); // You know the input: It is 2.0!
assertEquals(v2, 4.0);

The same test using random input:

double v = getRandomDouble();
double v2 = square(v); // You know the input: It is v!
assertEquals(v2, v*v);

If the test fails, all the details will be in the error
report.

And this method actually *do* fail for a mjority of all
possible inputs (abs of v exceeding sqrt(maxDouble)).
This will be revealed instantly using the random approach.

For an experienced programmer the limitation of square()
might be obvious so border cases are probably covered
sufficiently in both the code and the test. But for more
complex logic this might not be this apparent and throwing
in random input (in ADDITION to the typical cases and all
obvious border cases) has proven quite helpful, at least
to me.

[Jacob]

Tom Leylan wrote:

You've chosen a trivial example where your assert can compute the results of
the Square() function you are calling. That is hardly a typical situation
or there would be no reason for the function to have been created.

double v = getRandomDouble();
double v2 = AccountBalance( v );
assertEquals( v2, ? );

So explain how you get the value to type in ? given you don't know what the
input will be. Perhaps you would do the computations you read about in the
AccountBalance() method inline to see if those and yours matched?

I chose a fairly typical example of a basic unit requiring
unit testing and I proved that by using random input it
easily identified an error that otherwise could slip through.

I never said that using random input was useful in all
cases and perhaps it isn't in your specific example. On the
other hand, how do you know what goes into "?" given you
know the input? There must be some sort of reasoning behind
your result as well.

Below is a different example which might not be as trivial
as my previous. It "proves" that encoding + decoding (according
to some procedure) of any string should give back the original
string:

String text = getRandomString(0,1000000); // 0 - 1MB
String encoded = Encoder.encode(text);
String decoded = Encoder.decode(encoded);
assertEquals(text, decoded);

[Andrew McDonagh]

Below is a different example which might not be as trivial
as my previous. It "proves" that encoding + decoding (according
to some procedure) of any string should give back the original
string:

String text = getRandomString(0,1000000); // 0 - 1MB
String encoded = Encoder.encode(text);
String decoded = Encoder.decode(encoded);
assertEquals(text, decoded);

This only proves that the encoding & decoding scheme is the same. I can
make this test pass like this...

public String encode(String text) {
return text;
}

public String decode(String encoded) {
return encoded;
}

Here, the unit test is testing every line of code, yet its a worthless
implementation.

The unit test is not testing the encoding mechanism as it should be
implemented as its using the decode() for the assertion.

In this case, its prudent/better to have separate unit tests for both
the encode & decode, an doing the opposite conversion locally within the
unit test itself as this prevents errors of implementation but more
importantly, forces the implementation to use the correct algorithms.

public void testEncoder() {
String text = getRandomString(0,1000000); // 0 - 1MB
String encoded = Encoder.encode(text);
String decoded = MD5.decode(encoded);
assertEquals(text, decoded);
}

public void testDecoder() {
String text = getRandomString(0,1000000); // 0 - 1MB
String encoded = MD5.encode(text);
String decoded = Encoder.decode(encoded);
assertEquals(text, decoded);
}

Here we have separated the logic needed for the test from the unit under
test.

[Tom Leylan]

Forgive me but you are terming it "fairly typical" and it isn't typical of
anything I have seen. So rather than you or I decide (since we don't agree)
let others weigh in on how typical a function is that an easily-contrived
formula produces the exact same answer. Show me your assertEquals for
IsPrime() for instance.

The discussion hasn't been (as I read it) that random input is of no value
in all cases. It was illustrated by others that "unit tests" imply one does
know the answer and that random input means you can rarely know the answer.
To answer your question as to how one might know the value of ? it would be
"computed" by whatever method was required. A test for IsPrime() would be
fed known prime and non-prime values safely knowing which ones should return
true and which should return false. A test for AccountBalance() would
similarly have inputs and outputs which have been determined to test the
functionality.

I think Andrews response does a great job of pointing out how relying on the
two functions to test each other is a mistake.
"Jacob" <ja***@yahoo.com> wrote in message
news:4p********************@telenor.com...

Tom Leylan wrote:

You've chosen a trivial example where your assert can compute the results
of the Square() function you are calling. That is hardly a typical
situation or there would be no reason for the function to have been
created.

double v = getRandomDouble();
double v2 = AccountBalance( v );
assertEquals( v2, ? );

So explain how you get the value to type in ? given you don't know what
the input will be. Perhaps you would do the computations you read about
in the AccountBalance() method inline to see if those and yours matched?

I chose a fairly typical example of a basic unit requiring
unit testing and I proved that by using random input it
easily identified an error that otherwise could slip through.

I never said that using random input was useful in all
cases and perhaps it isn't in your specific example. On the
other hand, how do you know what goes into "?" given you
know the input? There must be some sort of reasoning behind
your result as well.

Below is a different example which might not be as trivial
as my previous. It "proves" that encoding + decoding (according
to some procedure) of any string should give back the original
string:

String text = getRandomString(0,1000000); // 0 - 1MB
String encoded = Encoder.encode(text);
String decoded = Encoder.decode(encoded);
assertEquals(text, decoded);

[Alex Hunsley]

Andrew McDonagh wrote:

Below is a different example which might not be as trivial
as my previous. It "proves" that encoding + decoding (according
to some procedure) of any string should give back the original
string:

String text = getRandomString(0,1000000); // 0 - 1MB
String encoded = Encoder.encode(text);
String decoded = Encoder.decode(encoded);
assertEquals(text, decoded);
This only proves that the encoding & decoding scheme is the same. I can
make this test pass like this...

public String encode(String text) {
return text;
}

public String decode(String encoded) {
return encoded;
}

Here, the unit test is testing every line of code, yet its a worthless
implementation.

No, an encoding that doesn't do any encoding is still an encoding. It
just happens to be the 'identity' encoding (in the same way that 1 is
the identity for multiplication and 0 is the identity for addition). If
an encoding and decoding function are presented as a pair of opposite
functions, then you are perfectly justified in testing that one is the
inverse of the other:

Decode(Encode(x)) = x

Testing that an encoding is *any good* by whatever means you judge
'good' is entirely a different matter to testing the reversibility of an
encode/decode pair (and harder to do, to boot, although testing that the
encoding did something, anything, to the data is trivial).




The unit test is not testing the encoding mechanism as it should be
implemented as its using the decode() for the assertion.

In this case, its prudent/better to have separate unit tests for both
the encode & decode, an doing the opposite conversion locally within the
unit test itself as this prevents errors of implementation but more
importantly, forces the implementation to use the correct algorithms.

public void testEncoder() {
String text = getRandomString(0,1000000); // 0 - 1MB
String encoded = Encoder.encode(text);
String decoded = MD5.decode(encoded);
assertEquals(text, decoded);
}

public void testDecoder() {
String text = getRandomString(0,1000000); // 0 - 1MB
String encoded = MD5.encode(text);
String decoded = Encoder.decode(encoded);
assertEquals(text, decoded);
}

Here we have separated the logic needed for the test from the unit under
test.

[Jacob]

Andrew McDonagh wrote:

Here, the unit test is testing every line of code, yet its a worthless
implementation.

The test is still useful. It can't prove that the code is correct,
but if it fails, it can prove that the code is wrong.

And as stated several times already: The test comes in ADDITION to
the test for typical cases and all obvious boundary cases, which in
this particular case would have been written quite differently.

[Andrew McDonagh]

Jacob wrote:

Andrew McDonagh wrote:

Here, the unit test is testing every line of code, yet its a worthless
implementation.

The test is still useful. It can't prove that the code is correct,
but if it fails, it can prove that the code is wrong.

But it cant tell you which of the collaborating en/de coding methods is
the cause.

[Jacob]

Tom Leylan wrote:

Forgive me but you are terming it "fairly typical" and it isn't typical of
anything I have seen.
The most typical methods around are getters and setters which
are even less complex than the square example I used previously:

String name = getRandomString(0,1000);
A.setName(name);
assertEquals(A.getName(), name);

They are not the most interesting ones to test, but they should
still be tested, and using random input increase the test coverage.
Show me your assertEquals for IsPrime() for instance.

Not the best example I could come up with, but it indicates
the principle:

for (int i = 0; i < 1000; i++) {
int v1 = getRandomInt();
if (isPrime(v1)) {
for (int j = 0; j < 1000; j++) {
int v2 = getRandomInt();
if (isPrime(v2)) {
assertNotEquals(v2 % v1, 0);
assertNotEquals(v1 % v2, 0);
}
}
}
}

Again: It doesn't prove that isPrime() is correct, but it may be able
to prove that it is wrong.

[Jacob]

Andrew McDonagh wrote:

Jacob wrote:

Andrew McDonagh wrote:

Here, the unit test is testing every line of code, yet its a
worthless implementation.

The test is still useful. It can't prove that the code is correct,
but if it fails, it can prove that the code is wrong.

But it cant tell you which of the collaborating en/de coding methods is
the cause.

Given an explicit input for which the operation fails should
give you enough information to be able to track this down yourself.

[Jacob]

Tom Leylan wrote:

Forgive me but you are terming it "fairly typical" and it isn't typical of anything I have seen.
The most typical methods around are getters and setters which
are even less complex than the square example I used previously:

String name = getRandomString(0,1000);
A.setName(name);
assertEquals(A.getName(), name);

They are not the most interesting ones to test, but they should
still be tested, and using random input increase the test coverage.
Show me your assertEquals for IsPrime() for instance.

Not the best example I could come up with, but it indicates
the principle:

for (int i = 0; i < 1000; i++) {
int v1 = getRandomInt();
if (isPrime(v1)) {
for (int j = 0; j < 1000; j++) {
int v2 = getRandomInt();
if (isPrime(v2)) {
assertNotEquals(v2 % v1, 0);
assertNotEquals(v1 % v2, 0);
}
}
}
}

Again: It doesn't prove that isPrime() is correct, but it may be able
to prove that it is wrong.

[Alex Hunsley]

Andrew McDonagh wrote:

Jacob wrote:

Andrew McDonagh wrote:

Here, the unit test is testing every line of code, yet its a
worthless implementation.

The test is still useful. It can't prove that the code is correct,
but if it fails, it can prove that the code is wrong.

But it cant tell you which of the collaborating en/de coding methods is
the cause.

But it is telling you that you have a problem, which is a good start.
You can then invetigate further, or back up with other complementary tests.

[Tom Leylan]

Of course one can insert getRandomString() into a test when the actual
string value has no known limits. Put it into your US State or US Zip
Get/Set... Any field which validates it's entry should fail upon assignment
and your assertion doesn't get a chance to run does it? Again nobody
claimed that there is anything wrong with random string testing. It was
pointed out that it shouldn't form the basis of one's unit tests.

It seems to me the purpose of unit testing is to verify that values known to
be good, pass and that values known to be bad, fail. If a bad value makes
it through it can be added to the test suite. That is different than
"generating an additional random value."

So you should continue to include random string tests (to your unit tests)
and I (and a few others) will probably recommend against it. There is no
problem with differing viewpoints.
"Jacob" <ja***@yahoo.com> wrote in message
news:2Y********************@telenor.com...

Tom Leylan wrote:

Forgive me but you are terming it "fairly typical" and it isn't typical
of anything I have seen.

The most typical methods around are getters and setters which
are even less complex than the square example I used previously:

String name = getRandomString(0,1000);
A.setName(name);
assertEquals(A.getName(), name);

They are not the most interesting ones to test, but they should
still be tested, and using random input increase the test coverage.

Show me your assertEquals for IsPrime() for instance.

Not the best example I could come up with, but it indicates
the principle:

for (int i = 0; i < 1000; i++) {
int v1 = getRandomInt();
if (isPrime(v1)) {
for (int j = 0; j < 1000; j++) {
int v2 = getRandomInt();
if (isPrime(v2)) {
assertNotEquals(v2 % v1, 0);
assertNotEquals(v1 % v2, 0);
}
}
}
}

Again: It doesn't prove that isPrime() is correct, but it may be able
to prove that it is wrong.

[Ben Pope]

Jacob wrote:

Ben Pope wrote:

Randomness just doesn't cut it, and I don't understand how you can
check the output is correct, without knowing the input.
You *do* know the input!

Consider testing this method:

double square(double v)
{
return v * v;
}

Below is a typical unit test that verifies that the
method behaves correctly on typical input:

double v = 2.0;
double v2 = square(v); // You know the input: It is 2.0!
assertEquals(v2, 4.0);

The same test using random input:

double v = getRandomDouble();
double v2 = square(v); // You know the input: It is v!
assertEquals(v2, v*v);

If the test fails, all the details will be in the error
report.

And how exactly did you come up with v*v as the value to test against?
Did you copy it from the function you're testing? Do you expect that to
fail?

Did you get somebody else to write the code? Do you implement all the
code twice, independently and check them against each other?
And this method actually *do* fail for a mjority of all
possible inputs (abs of v exceeding sqrt(maxDouble)).
This will be revealed instantly using the random approach.

For an experienced programmer the limitation of square()
might be obvious so border cases are probably covered
sufficiently in both the code and the test. But for more
complex logic this might not be this apparent and throwing
in random input (in ADDITION to the typical cases and all
obvious border cases) has proven quite helpful, at least
to me.

I fail to see how you are going to automatically test this complicated
logic.

Ben Pope
--
I'm not just a number. To many, I'm known as a string...

[Adam Maass]

"Jacob" <ja***@yahoo.com> wrote:

Adam Maass wrote:

There is sometimes value in testing on large numbers of random inputs.
But this isn't *unit* testing; it's more akin to a system or stress test.
It's something you hope your QAs will do for you; test on inputs that you
weren't necessarily expecting and see what breaks. Unit testing is about
the correctness of code for known inputs.
Which definition of unit testing is this? I have searched the
net but hasn't been able to find any backing for this?

If I write a method void setLength(double length), who define
the input "necesserily expected", and why isn't this the entire
double range? I'd claim the latter and to cover as many inputs
as possible I use the random trick.

I don't have a problem with defining this kind of testing
differently, for instance "stress testing", but on the other
hand there isn't really any more "stress" is calling
setLength(1.23076e+307) than setLength(2.0) as long as the
method accepts a double as input?

And why do you care about "known" input as long as the
actual (failing) input can be traced afterwards anyway?

You define this as a unit test:

for (int i = 0; i < 1000; i++)
testMyIntMethod(i);

while this is not:

for (int i = 0; i < 1000; i++)
testMyIntMethod(getRandomInt())

even if an error on input=42 will produce identical error reports
in both cases. Only the latter will (eventually) reveal the
error for input=-100042.

If you don't care about the result for input 100042, then the "random"
version is flawed. In unit testing, you want to select several typical
inputs, as well as boundary and out-of-range inputs. This is sufficient to
obtain a general sense that the code is correct for the general case. It
also requires the test-writer to /think/ about what the boundary conditions
are. There may be several of these, at many points in the domain.

Also, if I have a setLength() method which cover the "typical"
input cases just fine, but is in general crap (a common scenario),
then a testSetLength() method that verifies that setLength() work
fine for "typical" input isn't worth a lot. What you need is a test
method that test the non-typical inputs. From a black-box perspective
you don't really know what is typical or non-typical, so why not just
throw a random number genrator at it?

My objection to random inputs is that unit-tests must be 100% repeatable for
every run of the test suite. I don't ever want to see a failure of a unit
test that doesn't reappear on the next run of the suite unless something
significant -- either the test case or the code under test -- has changed.
Random inputs are likely to skip those inputs that cause failures, even if
every once in a while they do uncover a failure.

Note too that unit-testing is not black-box testing. Good unit tests usually
have pretty good knowledge of the underlying algorithm under test.
-- Adam Maass

[Timbo]

Adam Maass wrote:

My objection to random inputs is that unit-tests must be 100% repeatable for
every run of the test suite. I don't ever want to see a failure of a unit
test that doesn't reappear on the next run of the suite unless something
significant -- either the test case or the code under test -- has changed.
Random inputs are likely to skip those inputs that cause failures, even if
every once in a while they do uncover a failure.

Agreed. A potential problem with randomly generated inputs is that
the person fixing the fault has to write a unit test to reproduce
the bug. Some people are lazy and will just fix the bug, run the
random unit tests, see them pass (because the randomly generated
input is not tested the next time), and recommit the new version.

Also, I've never seen anything to indicate that random tests are
any more likely to uncover a fault than properly selected test cases.

[Roedy Green]

On Mon, 27 Mar 2006 07:41:54 -0800, "Adam Maass"
<ad***************@comcast.net> wrote, quoted or indirectly quoted
someone who said :

In unit testing, you want to select several typical
inputs, as well as boundary and out-of-range inputs.

a term you will also hear is "corner cases".
--
Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.

[Jacob]

Adam Maass wrote:

In unit testing, you want to select several typical
inputs, as well as boundary and out-of-range inputs. This is sufficient to
obtain a general sense that the code is correct for the general case. It
also requires the test-writer to /think/ about what the boundary conditions
are. There may be several of these, at many points in the domain.
You describe an ideal world where the unit test writer thinks
of every possible scenario beforehand. In such a regime you don't
need unit testing in the first place.

My experience is that you tend to "forget" certain scenarios
when you write the code, and then "forget" the exact same cases
in the test. The result is a test that works fine in normal cases,
but fails to reveal the flaw in the code for the not-so-normal
cases. This is a useless and costly excercise. Random inputs may
cover some of the cases that was forgotten in this process.
My objection to random inputs is that unit-tests must be 100% repeatable for
every run of the test suite. I don't ever want to see a failure of a unit
test that doesn't reappear on the next run of the suite unless something
significant -- either the test case or the code under test -- has changed.
If I have a flaw in my code I'd be more happy with a test that
indicates this *sometime* rather than *never*. Of course *always*
is even better, but then we're back to Utopia.

BTW: You can acheieve repeatability by specifying the random
seed in the test setup. My personal approach is of course to seed
with a maximum of randomness (using current time millis :-)
Note too that unit-testing is not black-box testing. Good unit tests usually
have pretty good knowledge of the underlying algorithm under test.

Again you add definition to unit testing without further reference. Unit
testing is *in practice* white-box testing since the tests are normally
written by the target code developer, but it is actually beneficial to
treat it as a black-box test: Look at the class from the public API,
consider the requirements, and then try to tear it appart without thinking
too much about the code internals. This is at least my personal approach
when writing unit tests for my own code.

[Jacob]

Ben Pope wrote:

And how exactly did you come up with v*v as the value to test against?
Did you copy it from the function you're testing? Do you expect that to
fail?

The unit test reflects the requirements and for a square()
method the requirement is to return the square of its argument: v*v.

That this happens to be identical to the code implementation is
purly coincidental and a result of picking a (too?) simple example.
The square method may well be implemented by establishing a socket
connection to the math query engine at the MIT, a fancy caching
mechanism or some advanced bit operation.

[Noah Roberts]

Jacob wrote:

Adam Maass wrote:

In unit testing, you want to select several typical
inputs, as well as boundary and out-of-range inputs. This is sufficient to
obtain a general sense that the code is correct for the general case. It
also requires the test-writer to /think/ about what the boundary conditions
are. There may be several of these, at many points in the domain.
You describe an ideal world where the unit test writer thinks
of every possible scenario beforehand. In such a regime you don't
need unit testing in the first place.

Sure you do. Unit tests can stop a lot of bugs before they happen and
before tracking them down gets difficult. The ones that remain mean
that you have to track them down as you normally would, write a test
for the condition that causes the bug to replicate, and then fix your
code until all tests pass.

This means that changes you make to the code later in refactoring or
adding features do not reintroduce bugs you have fixed before. Think
about how many times you have fixed a bug only for it to turn up later
because of changes you or someone else made to the code.
My experience is that you tend to "forget" certain scenarios
when you write the code, and then "forget" the exact same cases
in the test.
It helps to write the test first and to write the test independant of
the code in question. For instance, my latest batch of additions to
our code base involved adding features that were available in a
different code base...one we are depricating. My tests simply verify
that the same results result from the same inputs since at this time I
want the answers to be the same. I chose those values randomly but I
put them in as static values in my tests.

Forgetting is also important as I described above in which bugs
reappear after being fixed ages ago because you or someone else forgot
what caused them and put that problem back when altering the code.

The result is a test that works fine in normal cases, but fails to reveal the flaw in the code for the not-so-normal
cases. This is a useless and costly excercise. Random inputs may
cover some of the cases that was forgotten in this process.

Random inputs are difficult to regenerate. It might be beneficial to
initially create some random inputs but always put those as static
values in your test. This may cover some forgotten conditions yet
remain predictable and traceable. Remember, unit tests should be
completely automatic.

Note too that unit-testing is not black-box testing. Good unit tests usually
have pretty good knowledge of the underlying algorithm under test.

Again you add definition to unit testing without further reference. Unit
testing is *in practice* white-box testing since the tests are normally
written by the target code developer, but it is actually beneficial to
treat it as a black-box test: Look at the class from the public API,
consider the requirements, and then try to tear it appart without thinking
too much about the code internals. This is at least my personal approach
when writing unit tests for my own code.

Yes, that is how unit tests should be performed. The don't test the
code, they test the interface to make sure the code conforms to that
interface and that the interface is what is needed. They also serve to
document your code base fairly well.