473,903 Members | 4,199 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

unit testing guidelines

I have compiled a set og unit testing
recommendations based on my own experience
on the concept.

Feedback and suggestions for improvements
are appreciated:

http://geosoft.no/development/unittesting.html

Thanks.
Mar 17 '06
72 5298
Timbo wrote:
Also, I've never seen anything to indicate that random tests are any
more likely to uncover a fault than properly selected test cases.


"Properly selected" is fine. If you miss some of those (there may
be MANY remember), the random cases *may* catch them.

That's it. You are not supposed to replace any of the good stuff
you are already doing. It's just a simple tool for making the whole
package even better.
Mar 27 '06 #51
On Mon, 27 Mar 2006 22:12:41 +0200, Jacob <ja***@yahoo.co m> wrote,
quoted or indirectly quoted someone who said :
My experience is that you tend to "forget" certain scenarios
when you write the code, and then "forget" the exact same cases
in the test. The result is a test that works fine in normal cases,
but fails to reveal the flaw in the code for the not-so-normal
cases. This is a useless and costly excercise. Random inputs may
cover some of the cases that was forgotten in this process.


the other way to get coverage is to get same some tests written by
people unfamiliar with the inner workings. The will test things that
"don't need" testing.

--
Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.
Mar 27 '06 #52
Jacob wrote:
Adam Maass wrote:
In unit testing, you want to select several typical inputs, as well as
boundary and out-of-range inputs. This is sufficient to obtain a
general sense that the code is correct for the general case. It also
requires the test-writer to /think/ about what the boundary conditions
are. There may be several of these, at many points in the domain.
You describe an ideal world where the unit test writer thinks
of every possible scenario beforehand. In such a regime you don't
need unit testing in the first place.

My experience is that you tend to "forget" certain scenarios
when you write the code, and then "forget" the exact same cases
in the test. The result is a test that works fine in normal cases,
but fails to reveal the flaw in the code for the not-so-normal
cases. This is a useless and costly excercise. Random inputs may
cover some of the cases that was forgotten in this process.


There is where TDD comes in.

If we write one test at a time .
Write Just Enough Code to make the test pass.
Refactor to improve the current state of the design

We are only writing code for tests we already have. The next test is
only needed if we need to code something or to strengthen the
corner-case tests of the code that we have just made.

This way - there is no forgetting.

To make this achievable, each test case (method) should :
1) only test one aspect of the code
2) Have as few asserts as possible (1 being the best)
3) Be small (like any method) ~ 10(or what ever your favourite number
is) lines of code.
4) be fast - the faster they run, the more we run them continuosly,
the sooner we find problems.
5) Do not use/touch: Files, Networks, dbs - these are slow compared to
in memory fake data/objects.

My objection to random inputs is that unit-tests must be 100%
repeatable for every run of the test suite. I don't ever want to see a
failure of a unit test that doesn't reappear on the next run of the
suite unless something significant -- either the test case or the code
under test -- has changed.


If I have a flaw in my code I'd be more happy with a test that
indicates this *sometime* rather than *never*. Of course *always*
is even better, but then we're back to Utopia.

BTW: You can acheieve repeatability by specifying the random
seed in the test setup. My personal approach is of course to seed
with a maximum of randomness (using current time millis :-)


you might want to google 'seeding with time' to see why its not a great
idea.... especially when unit tests are concerned.
Note too that unit-testing is not black-box testing. Good unit tests
usually have pretty good knowledge of the underlying algorithm under
test.


Again you add definition to unit testing without further reference. Unit
testing is *in practice* white-box testing since the tests are normally
written by the target code developer, but it is actually beneficial to
treat it as a black-box test: Look at the class from the public API,
consider the requirements, and then try to tear it appart without thinking
too much about the code internals. This is at least my personal approach
when writing unit tests for my own code.


white box /black box.... all the same really from a testing PoV... the
only difference is how tolerable the test case is to the code design
changing. White box..not terribly tolerant. Black box...tolerant.

With TDD, its better to consider the unit tests to be 'Behavior
Specification Tests'. They are validating that the specified Behavior
exists within the code under test. But each specification test is
specifying a small part of the code under test, as we have multiple
small test cases. Not few large testcases.

For example, we have Calculator class that can Add, Subtract, Multiply &
Divide Integers.

So we'd have the following tests...

testAddingZeros ()
testAddingPosit iveNumbers()
testAddingNegat iveNumbers()
testAddingNegat iveWithPositive Numbers()
testAddingPosit iveWithNegative Numbers();

testDividingByZ ero()
testDividingPos itiveNumberByNe gative()
.....
I Don't need to have tests for different values within the Integer Range
within each test case, as I have separate testcases for the different
boundaries. One benefit of having separate named testcases rather than
lumping them all in a single testAdd() method, is that I can write Just
Enought code to make each test pass. However, the biggest benefit comes
later when I or someone else modifies the code and one or two Named
testcase fail rather than a single test case. Immediately - with having
debug! I can see what has broken.

"typing.... run all tests ... bang!
...
testAddingNegat iveWithPositive Numbers() failed - expected -10, got -30)
"

I know I've broken the negative with Positive code somehow, but I also
know I Have Not broken any other conditions (testcases).

if all of those asserts were in one testAdd() method, then any asserts
after the one testing -10 + 20 would NOT be run, so I would know if I've
broken anything else.

This might seem like a small thing, but when your application has 1700s
unit tests, its so much easier to see whats happening quickly with this
apporach.
Now each of these test cases my end up being the same apart from the
values passed to the Calc object and the expected output.

In that case I'd do one of two things:
1) refactor the tests to use a private helper method
private void testWith(Intege r num1, Integer num2, Integer expected)..

2) Apply the 'ParameterisedT estcase pattern.

Andrew
Mar 27 '06 #53
Noah Roberts wrote:
....
Random inputs are difficult to regenerate.


Whether or not pseudo-random inputs are difficult to regenerate depends
on the design of the test framework.

I suggest the following requirements:

1. Each pseudo-random test must support both an externally supplied seed
and a system time based seed.

2. The seed is part of the output on any pseudo-random test failure.

Given those properties, I think one can set up a test regime that gets
the benefits of random testing without the costs.

All tests in the regression test suite that is run for each code change
must be effectively non-random. That includes random tests bound to a
fixed seed. This is important, because any failure in this context
should be due to the most recent code change.

Running with system time seeds is an additional test activity. If it
finds an error, the first step towards a fix is to add the failing
test/seed combination to the regression test suite, so that it fails.

Whether the system time seed testing is considered "unit test" is a
matter of how "unit test" is defined.

Patricia
Mar 27 '06 #54
On Mon, 27 Mar 2006 20:46:04 GMT, Patricia Shanahan <pa**@acm.org >
wrote, quoted or indirectly quoted someone who said :
Running with system time seeds is an additional test activity. If it
finds an error, the first step towards a fix is to add the failing
test/seed combination to the regression test suite, so that it fails.


Good thinking. It would be so frustrating to discover an error you
can't reproduce.
--
Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.
Mar 28 '06 #55

"Jacob" <ja***@yahoo.co m> wrote:
Adam Maass wrote:
In unit testing, you want to select several typical inputs, as well as
boundary and out-of-range inputs. This is sufficient to obtain a general
sense that the code is correct for the general case. It also requires the
test-writer to /think/ about what the boundary conditions are. There may
be several of these, at many points in the domain.
You describe an ideal world where the unit test writer thinks
of every possible scenario beforehand. In such a regime you don't
need unit testing in the first place.


Well, no. You still need the unit tests for regression testing purposes.
(Make a change; does the code still obey the contract on it as expressed by
its test regime? If a unit test fails, it means that the code no longer
meets it contract.)

Unit tests are also a really good /development/ aide, if you write the test
cases first. Express your preconditions and postconditions, then write the
code to make the pre- and post- conditions hold true. The test cases are
often easier to write than the code that implements the logic required by
them.
My experience is that you tend to "forget" certain scenarios
when you write the code, and then "forget" the exact same cases
in the test. The result is a test that works fine in normal cases,
but fails to reveal the flaw in the code for the not-so-normal
cases. This is a useless and costly excercise. Random inputs may
cover some of the cases that was forgotten in this process.
Which is why no test regime is complete if it relies solely on unit-testing.
You want to expend some effort exposing the code to novel inputs -- just to
see what happens. My argument is that these novel inputs do not belong in
/unit/ testing.
My objection to random inputs is that unit-tests must be 100% repeatable
for every run of the test suite. I don't ever want to see a failure of a
unit test that doesn't reappear on the next run of the suite unless
something significant -- either the test case or the code under test --
has changed.


If I have a flaw in my code I'd be more happy with a test that
indicates this *sometime* rather than *never*. Of course *always*
is even better, but then we're back to Utopia.


See above. No testing regime is complete if it relies solely on unit tests.
By all means, run your code through random inputs if you think it will
discover failures. But do not make it a main feature of your unit test
suite, because a unit test must be 100% repeatable from run to run. (Else
how do you know that you've really fixed any failure you've discovered?)

If other kinds of testing show a failure, by all means add that case to your
unit test suite [when it makes sense] so that it doesn't happen again.
BTW: You can acheieve repeatability by specifying the random
seed in the test setup. My personal approach is of course to seed
with a maximum of randomness (using current time millis :-)
[Unimpressed.] Yes, you *could* do that. But another important feature of a
unit-test suite should be that it is easy to run, not requiring any special
setup. In short, it shouldn't require any parameters, and yet still be 100%
repeatable from run to run. That means hard-coded inputs.
Note too that unit-testing is not black-box testing. Good unit tests
usually have pretty good knowledge of the underlying algorithm under
test.


Again you add definition to unit testing without further reference. Unit
testing is *in practice* white-box testing since the tests are normally
written by the target code developer, but it is actually beneficial to
treat it as a black-box test: Look at the class from the public API,
consider the requirements, and then try to tear it appart without thinking
too much about the code internals. This is at least my personal approach
when writing unit tests for my own code.


My experience in many different organizations is that the QA teams expect
code to be unit-tested by the developers before being turned over to QA.
Developers writing unit tests means that the unit tests are white-box, of
necessity.


Story time! Consider your reaction to a failing test case.

"Gee, that's odd. The tests passed last time..."

"What's different this time?"

"Well, I just modified the file FooBar.java. The failure must have something
to do with the change I just made there."

"But the test case that is failing is called 'testBamBazzAdd 1'. How could a
change to FooBar.java cause that case to fail?"

[Many hours later...]

"There is no possible way that FooBar.java has anything to do with the
failing test case."

"Ohhhh.... you know, we saw a novel input in the test case testBamBazzAdd1 .
I wonder how that happened?"

"Well, let's fix the code to account for the novel input..."

[Make some changes, but do not add a new test case. The change doesn't
actually fix the error.]

"Well, that's a relief... the test suite now runs to completion without
error."
These are harried, busy developers working on a codebase that has thousands
of classes, and they're under the gun to get code out the door... they cut
corners here (bad developers!) but I think we can all relate to them.

Random inputs in a unit-test case can:

1. Mislead developers when a failure suddenly appears on novel inputs. If
they aren't working on the piece of code that the random inputs test, they
have to switch gears to understand what's going on;

2. Mislead developers into believing the code is actually fixed, when in
fact it is not, when the failure disappears on the next run of the test
suite.

3. Can create an air of suspicion around the unit-test suite. (To make
errors go away, just run the suite multiple times until you get a run
without errors.)


-- Adam Maass
Mar 28 '06 #56
Jacob wrote:

My experience is that you tend to "forget" certain scenarios
when you write the code, and then "forget" the exact same cases
in the test. The result is a test that works fine in normal cases,
but fails to reveal the flaw in the code for the not-so-normal
cases. This is a useless and costly excercise.


An observation; not written in stone; a subejective view.

Ignoring TDD, no unit test ever has and no unit test ever will verify a
requirement or testify to completeness of behaviour. You seem to think
that unit testing is to help find all possible inputs for a given
behaviour; I don't think this is true.

Unit tests are regression tests.

When you introduce new feature X in an iteration 5, you write unit tests
to show some confidence that the feature works; you're not guaranteeing
it works for any subset, or for the entire range, of input
possibilities. You could easily have a flaw in the program that gives
the correct output for a given input, but for entirely the wrong reason,
as would be apparent if you used input+1; but you didn't. The unit tests
you write in iteration 5 are, in fact, a cost without a return*.

When you introduce feature Y in iteration 6 is when you see the returns
for your iteration 5 unit tests. As when you run these again, and they
all pass, then you know that whatever you did in iteration 6 didn't
break those parts of iteration 5 that seen to run before. But they still
don't guarantee that feature X is fully tested. If you missed a test in
iteration 5, then re-running the tests in iteration 6 won't help. And
you could still have that bug iteration 5. Unit testing will never
uncover it. All they do is show that whatever you did in iteration 6
didn't change much.

Think of iteration tests like a camera. Before you go on holiday, you
take a snapshot of your treasury (you do have a treasury, don't you?) so
that you can quickly identify any that's stolen. When you come back from
you holiday, the police are there saying that there's been a break-in.
You take another snapshot of your treasury and compare the two photos:
damn it, they got the Ark of the Covenant. Again.

This does not, however, show you any objects that were stolen before you
took that first photograph.

By comparison, manual testing can be seen as taking an inventory before
you go and when you come back, based on the list of items (the
requirements) that have been updated ever since you had the treasury
installed.

[*] Actually, regression testing is useful even during feature X's
design phase, so there is some benefit accrued.

--
www.EdmundKirwan.com - Home of The Fractal Class Composition.

Download Fractality, free Java code analyzer:
http://www.EdmundKirwan.com/servlet/...c-page130.html
Mar 28 '06 #57
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
NotDashEscaped: You need GnuPG to verify this message

Jacob schreef:
My experience is that you tend to "forget" certain scenarios
when you write the code, and then "forget" the exact same cases
in the test. The result is a test that works fine in normal cases,
but fails to reveal the flaw in the code for the not-so-normal
cases. This is a useless and costly excercise. Random inputs may
cover some of the cases that was forgotten in this process.


This discussion about whether or not to use random inputs in tests makes
me curious: is it that important at all? The code I am working with now
uses almost no primitive types, except the occasional naming string and
perhaps an int or two. In other words, it is impossible to use random
input.

Is this such unusual? Is so much code working on ints and doubles that
it is possible to use random inputs?

Curious,
H.
--
Hendrik Maryns

=============== ===
www.lieverleven.be
http://aouw.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFEKPjRe+7 xMGD3itQRAoQiAJ 9levnvjpByW6AWh dGuOin62ltqHQCf fIjo
EJv4GfKu/b8p4LZI6a3gulI=
=04fH
-----END PGP SIGNATURE-----
Mar 28 '06 #58
Hendrik Maryns wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
NotDashEscaped: You need GnuPG to verify this message

Jacob schreef:
My experience is that you tend to "forget" certain scenarios
when you write the code, and then "forget" the exact same cases
in the test. The result is a test that works fine in normal cases,
but fails to reveal the flaw in the code for the not-so-normal
cases. This is a useless and costly excercise. Random inputs may
cover some of the cases that was forgotten in this process.

This discussion about whether or not to use random inputs in tests makes
me curious: is it that important at all? The code I am working with now
uses almost no primitive types, except the occasional naming string and
perhaps an int or two. In other words, it is impossible to use random
input.

Is this such unusual? Is so much code working on ints and doubles that
it is possible to use random inputs?

I don't believe so. Very little of what I write can be tested
randomly.

Another problem is -- how does one determine the expected output
of a randomly generated test case? This requires the
implementation of a test oracle that reproduces the behaviour of
the code under test. If the code under test has some complex data
types that are used for efficiency, and can be replicated using
something similar, this may be useful, but more often than not,
this isn't the case.
Mar 28 '06 #59
Hendrik Maryns wrote:
Is this such unusual? Is so much code working on ints and doubles that
it is possible to use random inputs?


A lot depends on what you are doing. (As an aside, I think a lot of
programmers underestimate how much variety there is in /other/ programmers'
typical tasks.) For some people working with, say, double[] arrays is the
norm, others would hardly ever see a primitive type except that the language
forces us to use them.

Regarding random testing, it seems to me to be a compromise forced on us by the
fact that machines have limited speed. If computers were infinitely fast then
no one would ever consider random testing -- we'd use a brute-force exploration
of the /entire/ problem space instead. Random testing is one way (only one
way) of trying for a computationally feasible approximation to that ideal. But
I don't think the idea of "exhaustive " testing even makes sense in many
contexts, so random testing doesn't make sense in those contexts either.

For instance, I have some code for manipulating string data in variety of byte
encoding (not written in Java). At one level everything's wrapped up in nice
objects, and exhaustive testing makes no sense (all possible strings ? All
possible operations on strings ??). OTOH, I need to handle byte-encodings too,
such as Java's weird not-entirely-unlike-UTF-8 byte encoding, and that is
happening (in a sense) below the level of objects. I would dearly love to be
able to run some tests on every possible sequence of Unicode characters.
Obviously that's out, but in practical terms, it would almost certainly suffice
to test all sequences up to, say, 8 characters long (in order to avoid edge
effects). But even that isn't feasible. So I plan to do exhaustive testing of
all possible sequences of 1 Unicode character, and random testing of /lots/ of
somewhat longer sequences. There will be other tests too, of course, but I
wouldn't even consider going live with code of this nature without some attempt
to test the /entire/ problem domain.

-- chris

Mar 28 '06 #60

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

14
2768
by: | last post by:
Hi! I'm looking for unit-testing tools for .NET. Somthing like Java has --> http://www.junit.org regards, gicio
5
6537
by: shuisheng | last post by:
Dear All, I was told that unit test is a powerful tool for progamming. If I am writing a GUI code, is it possible to still using unit test? I have a little experience in using unittest++. But I can not work out a way to use it to test GUI code. Thanks a lot!
0
9845
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10872
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10981
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10499
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
9675
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
8047
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
7205
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
6085
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4725
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.