473,372 Members | 953 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,372 software developers and data experts.

Unit-testing single function with large number of different inputs


Hi all ya unit-testing experts there :)

Code I'm working on has to parse large and complex files and detect
equally complex and large amount of errors before the contents of the file
is fed to module interpreting it. First I created a unit-test class named
TestLoad which loaded, say, 40 files of which about 10 are correct and
other 30 files contained over 20 different types of errors. Different
methods on the TestLoad class were coded so that they expected the
spesific error to occur. They looked like the following:

def test_xxx_err(self):
for fname in xxx_err_lst:
self.assertEqual(get_load_status(fname), xxx)

However, now we've added several more tests and we have added another
function to the system - the ability to try loading the file without
actually executing instructions in it. So, I added another class
TestValidate, which has exactly the same tests as the TestLoad class, the
only difference being in the routine get_load_status, which is not
get_load_status but get_validate_status.

I thought I'd simplify the process of adding new tests - if it is too
cumbersome to add new tests, we'll end up with having few tests which in
turn will result in more buggy software system.

Thus I came up with the simple idea of adding two directories:
test/correct-files and test/incorrect-files. Every time the test suite is
run, it goes through every file in the test/correct-files and
test/incorrect-files and expects to receive status 0 from all files under
correct-files, and appropriate error code from all files under
test/incorrect-files. How does it deduce the correct error code? Well,
I've forced things to that any file in the test/incorrect-files must
begin with code prefix 'xx-' in the file name, and the test suite reads
expected error code from the file name prefix.

All this allows me to have only two methods (in addition to setUp and
tearDown methods) in TestLoad and TestValidate classes, and what's more
convenient - I don't have to alter unit test suite at all when I add new
file tests to the system; all I have to do is to put it under either
directory, and if the file contains an error, I prefix the file name with
appropriate code.

The system has but one annoyance: first, if I use self.assertEqual in the
test_load method, it aborts on the first error found and as such doesn't
test all the other files. This would be bad, because when errors occur in
the code, I have to fix them one by one running test suite after every fix
to find other potential errors in the system.

So I changed the test_load routine by replacing self.assertEqual with my
own code which won't abort the test if errors are found, but it will
report them with sufficient verbosity and continue testing. But this
approach isn't optimal either: because there is no self.assertEqual in the
test anymore, there may be many errors in the test class but after running
all the methods, the test suite will report OK status, even though errors
did occur.

The problem I described above shows probably one of the best reasons why
each test method should test only one (type of) thing, but in my special
case I don't find that appropriate due to aforementioned reasons. Any
suggestions? I was wondering about creating test methods dynamically
according to the files found in the test directories (which would solve
all the problems), but I find it a tad too complex way of solving the
problem.

--
# Edvard Majakari Software Engineer
# PGP PUBLIC KEY available Soli Deo Gloria!

$_ = '456476617264204d616a616b6172692c20612043687269737 469616e20'; print
join('',map{chr hex}(split/(\w{2})/)),uc substr(crypt(60281449,'es'),2,4),"\n";

Jul 18 '05 #1
2 2064
I'm not a unit-testing expert, and I don't necessarily recommend what
I'm about to suggest, but...

To solve the problem of having a single "test" actually run many
"internal" tests, report on the results of those many tests, and fail
(in the PyUnit sense) if any of those internal tests fail, you could
add a flag, AllTestsPassed, to the TestLoad and TestValidate classes.
Initialize that flag to True. Then, in your version of assertEqual,
if any test fails set the flag to False. Finally, after you've run
all your tests, add

self.failUnless(self.AllTestsPassed, 'Some tests failed.')

HTH,

-robert
Edvard Majakari <ed*********@majakari.net> wrote in message news:<87************@titan.staselog.com>...
Hi all ya unit-testing experts there :)

Code I'm working on has to parse large and complex files and detect
equally complex and large amount of errors before the contents of the file
is fed to module interpreting it. First I created a unit-test class named
TestLoad which loaded, say, 40 files of which about 10 are correct and
other 30 files contained over 20 different types of errors. Different
methods on the TestLoad class were coded so that they expected the
spesific error to occur. They looked like the following:

def test_xxx_err(self):
for fname in xxx_err_lst:
self.assertEqual(get_load_status(fname), xxx)

However, now we've added several more tests and we have added another
function to the system - the ability to try loading the file without
actually executing instructions in it. So, I added another class
TestValidate, which has exactly the same tests as the TestLoad class, the
only difference being in the routine get_load_status, which is not
get_load_status but get_validate_status.

I thought I'd simplify the process of adding new tests - if it is too
cumbersome to add new tests, we'll end up with having few tests which in
turn will result in more buggy software system.

Thus I came up with the simple idea of adding two directories:
test/correct-files and test/incorrect-files. Every time the test suite is
run, it goes through every file in the test/correct-files and
test/incorrect-files and expects to receive status 0 from all files under
correct-files, and appropriate error code from all files under
test/incorrect-files. How does it deduce the correct error code? Well,
I've forced things to that any file in the test/incorrect-files must
begin with code prefix 'xx-' in the file name, and the test suite reads
expected error code from the file name prefix.

All this allows me to have only two methods (in addition to setUp and
tearDown methods) in TestLoad and TestValidate classes, and what's more
convenient - I don't have to alter unit test suite at all when I add new
file tests to the system; all I have to do is to put it under either
directory, and if the file contains an error, I prefix the file name with
appropriate code.

The system has but one annoyance: first, if I use self.assertEqual in the
test_load method, it aborts on the first error found and as such doesn't
test all the other files. This would be bad, because when errors occur in
the code, I have to fix them one by one running test suite after every fix
to find other potential errors in the system.

So I changed the test_load routine by replacing self.assertEqual with my
own code which won't abort the test if errors are found, but it will
report them with sufficient verbosity and continue testing. But this
approach isn't optimal either: because there is no self.assertEqual in the
test anymore, there may be many errors in the test class but after running
all the methods, the test suite will report OK status, even though errors
did occur.

The problem I described above shows probably one of the best reasons why
each test method should test only one (type of) thing, but in my special
case I don't find that appropriate due to aforementioned reasons. Any
suggestions? I was wondering about creating test methods dynamically
according to the files found in the test directories (which would solve
all the problems), but I find it a tad too complex way of solving the
problem.

Jul 18 '05 #2
fe*****@diablotech.com (Robert Ferrell) writes:
add a flag, AllTestsPassed, to the TestLoad and TestValidate classes.
Initialize that flag to True. Then, in your version of assertEqual,
if any test fails set the flag to False. Finally, after you've run
all your tests, add

self.failUnless(self.AllTestsPassed, 'Some tests failed.')


But that doesn't give instant feedback (running ValidateTest takes over a
minute), neither does it report exact cause of errors. Hmm. I think I'll
look into Python's ability to create methods on the fly.

Thanks anyway!

PS. Sorry for the provoking .signature, it dates back to the Old Times
when I didn't know of Python ;)

--
#!/usr/bin/perl -w
$h={23,69,28,'6e',2,64,3,76,7,20,13,61,8,'4d',24,7 3,10,'6a',12,'6b',21,68,14,
72,16,'2c',17,20,9,61,11,61,25,74,4,61,1,45,29,20, 5,72,18,61,15,69,20,43,26,
69,19,20,6,64,27,61,22,72};$_=join'',map{chr hex $h->{$_}}sort{$a<=>$b}
keys%$h;m/(\w).*\s(\w+)/x;$_.=uc substr(crypt(join('',60,28,14,49),join'',
map{lc}($1,substr $2,4,1)),2,4)."\n"; print;
Jul 18 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
by: Bob | last post by:
Are they different names for the same concept ?
3
by: Praveen | last post by:
How to get rid of this error
1
by: Shan | last post by:
Is there any tool available for generating unit test code in .NET. This is reverse engineering process, which we have to unit test methods in a class. Tool such as Nunit can test the methods only...
2
by: Ben Finney | last post by:
Howdy all, My practice when writing unit tests for a project is to make 'test/' subdirectories for each directory containing modules I want to test. project-foo/ +-- lib/ | +-- test/ +--...
4
by: Stephan Rose | last post by:
In one of the applications I am developing I have need of working with numbers in varying unit system. I might be adding 5 mil to 1 inch or whatever. What I have come up with works really well,...
4
by: AbrahamLincolnIllinois | last post by:
Greetings! I am working with a researcher who would be very happy if he could include units in his calculations. Then, if his complicated expression didn't result in "kg/yr" the program would...
2
by: shuisheng | last post by:
Dear All, I am using visual studipo 2005 (standard version which do not provide unit test tool) to develop some c++ code. I want to do unit test while I am coding. Anybody can suggest me an...
5
by: Lee | last post by:
Hi, Using the propertygrid, is it possible to have a user selectable 'unit of measure' for a field. For example, I am looking at using a propertygrid for defining inputs into a calculation. ...
1
by: batvanio | last post by:
Hi, I am writing unit tests in VS2005 and am having the following problem: I am trying to test a timeout property of one of my methods. This timeout exhibits itself in an exceptioin - i.e. I am...
5
by: =?Utf-8?B?cmFuZHkxMjAw?= | last post by:
I'm working in Visual Studio 2005 Team Edition for Software Developers I've used the wizard under the Test menu to create a bunch of unit tests. When I click "Test -Start Selected Test Project...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...
0
by: ryjfgjl | last post by:
In our work, we often need to import Excel data into databases (such as MySQL, SQL Server, Oracle) for data analysis and processing. Usually, we use database tools like Navicat or the Excel import...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: aa123db | last post by:
Variable and constants Use var or let for variables and const fror constants. Var foo ='bar'; Let foo ='bar';const baz ='bar'; Functions function $name$ ($parameters$) { } ...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.