Requesting critique of a C unit test environment

[Ark Khasin]

[First, I apologize for cross-posting. I just think a wider audience can
critique from different vantage points.]

Unit testing is an integral component of both "formal" and "agile"
models of development. Alas, it involves a significant amount of tedious
labor.

There are test automation tools out there but from what limited exposure
I've had, they are pricey, reasonably buggy, and require compiler/target
adaptation.

Out of my frustration with two out of two of them came my own. Its
instrumentation approach is based solely on profound abuse of the C
preprocessor (and in this respect it is equally applicable to C++).

I would like to ask to evaluate the approach
- whether it has gaping holes in ideology or implementation
- whether in your opinion it has merits

A preliminary draft description is at
http://www.macroexpressions.com/dl/C...shoestring.pdf

A reference implementation (with a C99 accent) with a runnable example is at
http://www.macroexpressions.com/dl/maestra.zip

Please reply to a newsgroup or via email as you find convenient.
Thank you for your anticipated feedback,

-- Ark

[Ian Collins]

Ark Khasin wrote:

[First, I apologize for cross-posting. I just think a wider audience can
critique from different vantage points.]

Unit testing is an integral component of both "formal" and "agile"
models of development. Alas, it involves a significant amount of tedious
labor.

There are test automation tools out there but from what limited exposure
I've had, they are pricey, reasonably buggy, and require compiler/target
adaptation.

Out of my frustration with two out of two of them came my own. Its
instrumentation approach is based solely on profound abuse of the C
preprocessor (and in this respect it is equally applicable to C++).

I would like to ask to evaluate the approach
- whether it has gaping holes in ideology or implementation
- whether in your opinion it has merits

Why not just use one of the free frameworks such as CppUnit?

It works well with both C (with a little fiddling like you do in your
paper for "static") and C++. I'm sure the same applies for other
frameworks.

--
Ian Collins.

[Ark Khasin]

Ian Collins wrote:
<snip>

Why not just use one of the free frameworks such as CppUnit?

It works well with both C (with a little fiddling like you do in your
paper for "static") and C++. I'm sure the same applies for other
frameworks.

Ian,
Thank you for your response.

Please correct me if I am wrong, but AFAIK CppUnit doesn't provide a
code execution trace, so it's pretty darn hard to prove code coverage.
[There must be reasons why testing tools vendors command big money.]

Also, if I use C in non-C++ compatible way (e.g. tentative definitions),
my source won't even compile for CppUnit.

And finally there is a port issue (it's an embedded type talking :)). I
am proposing something that requires only the compiler.

Regards,
Ark

[Ian Collins]

Ark Khasin wrote:

Ian Collins wrote:
<snip>

>Why not just use one of the free frameworks such as CppUnit?

It works well with both C (with a little fiddling like you do in your
paper for "static") and C++. I'm sure the same applies for other
frameworks.

Ian,
Thank you for your response.

Please correct me if I am wrong, but AFAIK CppUnit doesn't provide a
code execution trace, so it's pretty darn hard to prove code coverage.
[There must be reasons why testing tools vendors command big money.]

If you develop your software test first, you get all the code coverage
you need.

Also, if I use C in non-C++ compatible way (e.g. tentative definitions),
my source won't even compile for CppUnit.

If you mean K&R style prototypes, don't use them. Write and compile
your tests in C++ and your code in C. Don't attempt to compile your C
with a C++ compiler.

And finally there is a port issue (it's an embedded type talking :)). I
am proposing something that requires only the compiler.

Shouldn't matter for unit testing, develop and test on a hosted system.
If you require bits of the target environment, mock (simulate) them.

--
Ian Collins.

[Colin Paul Gloster]

On 2007-08-27, Ark Khasin <ak*****@macroexpressions.comwrote:

|--------------------------------------------------------------------------------------|
|"[..] |
| |
|Unit testing is an integral component of [..] "formal" [..] |
|models of development. [..] |
| |
|[..]" |
|--------------------------------------------------------------------------------------|

Testing is not an intgeral component of formal methods intended to
reduce testing.

Regards,
Colin Paul Gloster

[Phlip]

Colin Paul Gloster wrote:

Testing is not an intgeral component of formal methods intended to
reduce testing.

Colin Paul Gloster

Why would a formal method intend to reduce a Good Thing??

--
Phlip
http://www.oreilly.com/catalog/9780596510657/
^ assert_xpath
http://tinyurl.com/23tlu5 <-- assert_raise_message

[=?ISO-8859-1?Q?Erik_Wikstr=F6m?=]

On 2007-08-27 09:08, Phlip wrote:

Colin Paul Gloster wrote:

>Testing is not an intgeral component of formal methods intended to
reduce testing.

Colin Paul Gloster

Why would a formal method intend to reduce a Good Thing??

Testing is used to find errors, while formal methods are used to prove
that there are no errors, at least that's the goal. So if you can prove
that there are no errors why test for them?

--
Erik Wikström

[Richard Heathfield]

Erik Wikström said:

<snip>

Testing is used to find errors, while formal methods are used to prove
that there are no errors, at least that's the goal. So if you can
prove that there are no errors why test for them?

"Beware of bugs in the above code; I have only proved it correct, not
tried it." - Donald E Knuth.

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999

[Phlip]

Erik Wikström wrote:

Testing is used to find errors, while formal methods are used to prove
that there are no errors, at least that's the goal. So if you can prove
that there are no errors why test for them?

I use testing as a formal method, to prevent errors. I have not tried
the "proof" systems (and please don't try to tell the mathematicians I used
to hang out with that they are really "proofs").

You are describing writing and running tests in isolation from the
development process. Don't do that.

--
Phlip
http://www.oreilly.com/catalog/9780596510657/
^ assert_xpath
http://tinyurl.com/23tlu5 <-- assert_raise_message

[Richard]

Richard Heathfield <rj*@see.sig.invalidwrites:

Erik Wikström said:

<snip>

>Testing is used to find errors, while formal methods are used to prove
that there are no errors, at least that's the goal. So if you can
prove that there are no errors why test for them?

"Beware of bugs in the above code; I have only proved it correct, not
tried it." - Donald E Knuth.

Proof of Correctness depends very much on how and when "correct" is
used. It is a crock of shit in most day to day SW development.

[Ben Bacarisse]

Richard Heathfield <rj*@see.sig.invalidwrites:

Erik Wikström said:

<snip>

>Testing is used to find errors, while formal methods are used to prove
that there are no errors, at least that's the goal. So if you can
prove that there are no errors why test for them?

"Beware of bugs in the above code; I have only proved it correct, not
tried it." - Donald E Knuth.

But this was a "by hand" proof in 1977. A machine assisted proof of
the actual code could be expected to inspire a little more confidence.

--
Ben.

[Richard Heathfield]

Ben Bacarisse said:

Richard Heathfield <rj*@see.sig.invalidwrites:

>Erik Wikström said:

<snip>

>>Testing is used to find errors, while formal methods are used to
prove that there are no errors, at least that's the goal. So if you
can prove that there are no errors why test for them?

"Beware of bugs in the above code; I have only proved it correct, not
tried it." - Donald E Knuth.

But this was a "by hand" proof in 1977. A machine assisted proof of
the actual code could be expected to inspire a little more confidence.

Why? Presumably the machine that is doing the assisting is itself a
computer program. What makes you think the assistance program is
correct?

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999

[Ben Bacarisse]

Richard Heathfield <rj*@see.sig.invalidwrites:

Ben Bacarisse said:

>Richard Heathfield <rj*@see.sig.invalidwrites:

>>Erik Wikström said:

<snip>

Testing is used to find errors, while formal methods are used to
prove that there are no errors, at least that's the goal. So if you
can prove that there are no errors why test for them?

"Beware of bugs in the above code; I have only proved it correct, not
tried it." - Donald E Knuth.

But this was a "by hand" proof in 1977. A machine assisted proof of
the actual code could be expected to inspire a little more confidence.

Why? Presumably the machine that is doing the assisting is itself a
computer program. What makes you think the assistance program is
correct?

What do you test your software with if not more software?

If you think that a machine assisted proof would not inspire "a little
more" confidence than a hand proof, then I won't try to persuade you
(it was a modest enough claim) but the fact that a proof system is
software does not invalidate the method any more than testing is
invalidated by being done in software.

--
Ben.

[Richard Heathfield]

Ben Bacarisse said:

Richard Heathfield <rj*@see.sig.invalidwrites:

>Ben Bacarisse said:

<snip>

>>>
A machine assisted proof of
the actual code could be expected to inspire a little more
confidence.

Why? Presumably the machine that is doing the assisting is itself a
computer program. What makes you think the assistance program is
correct?

What do you test your software with if not more software?

A rolling pin. Any software that can withstand the pastry test is likely
to be able to withstand anything else too.

If you think that a machine assisted proof would not inspire "a little
more" confidence than a hand proof, then I won't try to persuade you
(it was a modest enough claim)

Yes, on reflection I see that I'm guilty of (accidentally) extending
your claim, which was indeed modest enough.

<snip>

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999

[Ben Pfaff]

Ben Bacarisse <be********@bsb.me.ukwrites:

Richard Heathfield <rj*@see.sig.invalidwrites:

>Erik Wikström said:

>>Testing is used to find errors, while formal methods are used to prove
that there are no errors, at least that's the goal. So if you can
prove that there are no errors why test for them?

"Beware of bugs in the above code; I have only proved it correct, not
tried it." - Donald E Knuth.

But this was a "by hand" proof in 1977. A machine assisted proof of
the actual code could be expected to inspire a little more confidence.

YMMV of course, but if I could get Donald Knuth to prove my
programs correct "by hand", I'd feel no need for additional
confidence.
--
Ben Pfaff
http://benpfaff.org

[Phlip]

Flash Gordon wrote:

You have missed out internal formal testing which in many environments
is far more complete than acceptance testing. For example, I've worked
on projects where a formal test literally takes a week to complete but
the customer acceptance testing takes only a few hours.

What did y'all do if the "formal" test failed?

What I look for is this: Replicate the failure as a short unit test. Not a
proof - just a stupid test that fails because the code change needed to fix
that formal test isn't there.

The point is to make the fast tests higher value as you go...

--
Phlip
http://www.oreilly.com/catalog/9780596510657/

[Ben Bacarisse]

Ben Pfaff <bl*@cs.stanford.eduwrites:

Ben Bacarisse <be********@bsb.me.ukwrites:

>Richard Heathfield <rj*@see.sig.invalidwrites:

>>Erik Wikström said:
Testing is used to find errors, while formal methods are used to prove
that there are no errors, at least that's the goal. So if you can
prove that there are no errors why test for them?

"Beware of bugs in the above code; I have only proved it correct, not
tried it." - Donald E Knuth.

But this was a "by hand" proof in 1977. A machine assisted proof of
the actual code could be expected to inspire a little more confidence.

YMMV of course, but if I could get Donald Knuth to prove my
programs correct "by hand", I'd feel no need for additional
confidence.

No, MMIS[1]. I did not intend to disparage Prof. Knuth's "hand
proofs" (what a thought!) but rather to say that the problem he is
referring to is as likely to be that one proves something other than
the program one has written (or later writes) as it is to be that ones
proof is (internally) flawed.

I suspect that he is not entirely happy with the way that quip is used
so often to suggest the pointlessness of proofs[2] (after all, what
did he choose to do with his "Notes on van Emde Boas constriction of
priority deques" -- a proof rather than a test implementation!).

[1] "My mileage is similar".
[2] This not one of those times -- RH was just countering the much
stronger assertion that proof =no need to test.

--
Ben.

[user923005]

On Aug 27, 4:17 pm, Ben Bacarisse <[e-mail]:
[snip]

I suspect that he is not entirely happy with the way that quip is used
so often to suggest the pointlessness of proofs[2] (after all, what
did he choose to do with his "Notes on van Emde Boas constriction of
priority deques" -- a proof rather than a test implementation!).

Aside:
Working vEB tree implementation here:
http://www.itu.dk/people/kokholm/veb/

[Ian Collins]

Flash Gordon wrote:

Ian Collins wrote, On 27/08/07 21:51:

>Flash Gordon wrote:

>>Ian Collins wrote, On 27/08/07 08:14:
Ark Khasin wrote:
Ian Collins wrote:
<snip>

>>And finally there is a port issue (it's an embedded type talking
>>:)). I
>>am proposing something that requires only the compiler.
>>>
>Shouldn't matter for unit testing, develop and test on a hosted
>system.
> If you require bits of the target environment, mock (simulate) them.
>>
The farthest I can go away from the target is a software simulator of
the instruction set. Same compiler, same version, perhaps, more
"memory". I think I am not alone in this...
>
Why?
There are many possible *valid* reasons for this. One is that if you are
not using the same version of the same compiler with the same switches
then the code you are testing is not the same as the code that will be
run. Since compilers *do* have bugs it is possible that the bug will be
triggered in the real environment but not in the test environment unless
you ensure that they are the same.

But one has to differentiate between developer unit testing (the subject
of this post) and QA (customer) acceptance testing.

You have missed out internal formal testing which in many environments
is far more complete than acceptance testing. For example, I've worked
on projects where a formal test literally takes a week to complete but
the customer acceptance testing takes only a few hours.

If performed, internal formal testing is still a step away from
developer testing.

>The former can be
performed on any environment the developer chooses,

Informal testing can be run in any environment the developer has
available. Formal testing, which is the only sort of testing that you
can guarantee will be available and working for those maintaining later,
is another matter.

How so? A unit test suite doesn't just vanish when the code is
released, it is an essential part of the code base.

>>
Again, this is different from developer unit testing, I don't think
anyone would be daft enough to release a product that hadn't been
through acceptance testing on the target platform.

Acceptance testing has very little to do with proving whether the system
works, it is just to give the customer some confidence.

That depends on your definition of Acceptance tests. In our case, they
are the automated suite of tests that have to pass before the product is
released to customers.

The real worth
while formal testing has to be completed *before* doing customer
acceptance testing and done with the correct compiler. At least, this is
the case in many environments, including all the projects where I have
been involved in QA, and on the safety critical project I was involved in.

Again, that depends on your process.

If your customer acceptance testing is sufficient to prove the SW is
sufficiently correct then your customer has either very little trust in
your company or a lot of time to waste. If your customer acceptance
testing is the only testing done with the correct compiler and it is not
sufficient to prove your SW is sufficiently correct then your SW is not
tested properly. At least, not according to any standard of testing I
have come across.

Why? Our acceptance test are very comprehensive, written by
professional testers working with a product manager (the customer).

It sounds like you don't have fully automated acceptance tests. Where
ever possible, all tests should be fully automated.

--
Ian Collins.

[Richard Heathfield]

Ben Bacarisse said:

<snip>

>
I suspect that [DEK] is not entirely happy with the way that quip
is used so often to suggest the pointlessness of proofs[2]

<snip>

[2] This not one of those times -- RH was just countering the much
stronger assertion that proof =no need to test.

Right. One problem is that they don't always prove what you asked them
to prove. What you actually want to know is "does this program properly
do what I need it to do?", but what a prover actually tells you is
whether program X conforms to a particular expression of specification
Y. It makes no comment whatsoever on whether specification Y
corresponds to wishlist Z. And, very often, such correspondence is far
from perfect.

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999

[Jonathan Kirwan]

On Mon, 27 Aug 2007 19:14:04 +1200, Ian Collins <ia******@hotmail.com>
wrote:

><snip>

>The farthest I can go away from the target is a software simulator of
the instruction set. Same compiler, same version, perhaps, more
"memory". I think I am not alone in this...

Why?

Perhaps because this is an example of a medical product where that is
felt to be required. There is a list of various kinds of "structural
coverages," those selected commensurate with the level of risk posed
by the software. (Saying something has "coverage," I think, always
implies 100% coverage, too. Not partial. So you either have coverage
or you don't.)

Borrowing from one of the US CDRH PDFs I have laying about:

· Statement Coverage – This criteria requires sufficient test cases
for each program statement to be executed at least once; however,
its achievement is insufficient to provide confidence in a
software product's behavior.

· Decision (Branch) Coverage – This criteria requires sufficient test
cases for each program decision or branch to be executed so that
each possible outcome occurs at least once. It is considered to be
a minimum level of coverage for most software products, but
decision coverage alone is insufficient for high-integrity
applications.

· Condition Coverage – This criteria requires sufficient test cases
for each condition in a program decision to take on all possible
outcomes at least once. It differs from branch coverage only when
multiple conditions must be evaluated to reach a decision.

· Multi-Condition Coverage – This criteria requires sufficient test
cases to exercise all possible combinations of conditions in a
program decision.

· Loop Coverage – This criteria requires sufficient test cases for
all program loops to be executed for zero, one, two, and many
iterations covering initialization, typical running and termination
(boundary) conditions.

· Path Coverage – This criteria requires sufficient test cases for
each feasible path, basis path, etc., from start to exit of a
defined program segment, to be executed at least once. Because of
the very large number of possible paths through a software program,
path coverage is generally not achievable. The amount of path
coverage is normally established based on the risk or criticality
of the software under test.

· Data Flow Coverage – This criteria requires sufficient test cases
for each feasible data flow to be executed at least once. A number
of data flow testing strategies are available.

For potentially high risk software, you may not just use a different
compiler or a different operating system environment or change even
the optimization options. As the OP mentioned, it's probably going to
enough just justifying an instruction simulator.

I can easily see a desire for an automated way of demonstrating that
structural testing has achieved one or more of these cases. If I read
the OP right about this, anyway.

Jon

[Richard Bos]

Ben Bacarisse <be********@bsb.me.ukwrote:

Richard Heathfield <rj*@see.sig.invalidwrites:

Ben Bacarisse said:

Richard Heathfield <rj*@see.sig.invalidwrites:

Erik Wikström said:

Testing is used to find errors, while formal methods are used to
prove that there are no errors, at least that's the goal. So if you
can prove that there are no errors why test for them?

"Beware of bugs in the above code; I have only proved it correct, not
tried it." - Donald E Knuth.

But this was a "by hand" proof in 1977. A machine assisted proof of
the actual code could be expected to inspire a little more confidence.

Why? Presumably the machine that is doing the assisting is itself a
computer program. What makes you think the assistance program is
correct?

What do you test your software with if not more software?

Sed quis custodiet ipsos custodes?

Richard

[Ian Collins]

Jonathan Kirwan wrote:

On Mon, 27 Aug 2007 19:14:04 +1200, Ian Collins <ia******@hotmail.com>
wrote:

><snip>

>>The farthest I can go away from the target is a software simulator of
the instruction set. Same compiler, same version, perhaps, more
"memory". I think I am not alone in this...

Why?

Perhaps because this is an example of a medical product where that is
felt to be required. There is a list of various kinds of "structural
coverages," those selected commensurate with the level of risk posed
by the software. (Saying something has "coverage," I think, always
implies 100% coverage, too. Not partial. So you either have coverage
or you don't.)

The why was prompted by the posting subject "critique of a C unit test
environment". To my way of thinking (TDD), unit tests are developer
tool, not formal product tests.
<interesting stuff snipped>
--
Ian Collins.

[Colin Paul Gloster]

On 2007-08-27, Ben Pfaff <bl*@cs.stanford.eduwrote:

|--------------------------------------------------------------------------|
|"[..] |
| |
|YMMV of course, but if I could get Donald Knuth to prove my |
|programs correct "by hand", I'd feel no need for additional |
|confidence." |
|--------------------------------------------------------------------------|

Such as the way Donald E. Knuth told Leslie Lamport that TeX would
hardly change at all? From
HTTP://research.Microsoft.com/users/...-interview.pdf
:"[..]
[..] When Don
was writing TEX80, he announced that it would be a
reimplementation of TEX78, but he was not going to
add new features. I took him seriously and asked for
almost no changes to TEX itself. [..] However, there were many other
im-
provements that I could have suggested but didn't. In
the end, Don wound up making very big changes to
TEX78. But they were all incremental, and there was
never a point where he admitted that he was willing
to make major changes. Had I known at the begin-
ning how many changes he would be making, I would
have tried to participate in the redesign. [..]
[..]"

Regards,
Colin Paul Gloster

[Colin Paul Gloster]

On 2007-08-27, Richard Heathfield <rj*@see.sig.invalidwrote:

|------------------------------------------------------------------------|
|"Ben Bacarisse said: |
| |
|Richard Heathfield <rj*@see.sig.invalidwrites: |
| |
|>Erik Wikström said: |
|> |
|><snip |
|> |
|>>Testing is used to find errors, while formal methods are used to |
|>>prove that there are no errors, at least that's the goal. So if you |
|>>can prove that there are no errors why test for them? |
|> |
|>"Beware of bugs in the above code; I have only proved it correct, not|
|>tried it." - Donald E Knuth. |
| |
|But this was a "by hand" proof in 1977. A machine assisted proof of |
|the actual code could be expected to inspire a little more confidence.|
| |
|Why? Presumably the machine that is doing the assisting is itself a |
|computer program. What makes you think the assistance program is |
|correct?" |
|------------------------------------------------------------------------|

Full points to Mister Heathfield.

[Phlip]

Colin Paul Gloster wrote:

Such as the way Donald E. Knuth told Leslie Lamport that TeX would
hardly change at all? From

HTTP://research.Microsoft.com/users/...-interview.pdf

:"[..]
[..] When Don
was writing TEX80, he announced that it would be a
reimplementation of TEX78, but he was not going to
add new features. I took him seriously and asked for
almost no changes to TEX itself. [..] However, there were many other
im-
provements that I could have suggested but didn't. In
the end, Don wound up making very big changes to
TEX78. But they were all incremental, and there was
never a point where he admitted that he was willing
to make major changes. Had I known at the begin-
ning how many changes he would be making, I would
have tried to participate in the redesign. [..]

"Principle 4

" * Level out the workload (heijunka). (Work like the tortoise, not the
hare).

"This helps achieve the goal of minimizing waste (muda), not overburdening
people or the equipment (muri), and not creating uneven production levels
(mura)."

http://en.wikipedia.org/wiki/The_Toyota_Way

--
Phlip

[Walter Banks]

I will second that, well put.

The Achilles heel for either Testing or formal methods is
contaminating the evaluation process with information
from the implementation.

I have seen unit tests contaminated from just knowing
the application area the code was going to be used.

w..
Colin Paul Gloster wrote:

On 2007-08-27, Richard Heathfield <rj*@see.sig.invalidwrote:

|------------------------------------------------------------------------|
|"Ben Bacarisse said: |
| |
|Richard Heathfield <rj*@see.sig.invalidwrites: |
| |
|>Erik Wikström said: |
|> |
|><snip |
|> |
|>>Testing is used to find errors, while formal methods are used to |
|>>prove that there are no errors, at least that's the goal. So if you |
|>>can prove that there are no errors why test for them? |
|> |
|>"Beware of bugs in the above code; I have only proved it correct, not|
|>tried it." - Donald E Knuth. |
| |
|But this was a "by hand" proof in 1977. A machine assisted proof of |
|the actual code could be expected to inspire a little more confidence.|
| |
|Why? Presumably the machine that is doing the assisting is itself a |
|computer program. What makes you think the assistance program is |
|correct?" |
|------------------------------------------------------------------------|

Full points to Mister Heathfield.

[Phlip]

Flash Gordon wrote:

>We all work from our own point of reference, in mine, units tests are a
developer tool so that's why I answered as I did.

You should try to avoid assuming everyone works the same way. In the
defence industry at least it is very common for there to be a lot of
formal unit tests.

I think Ian refers to "developer tests". Giving them different definitions
helps. They have overlapping effects but distinct motivations.

The failure of a unit test implicates only one unit in the system, so the
search for a bug should be very easy. The failure of a developer test
implicates the last edit - not that it inserted a bug, but only that it
failed the test suite! Finding and reverting that edit is easier than
debugging.

I fully understand the use of them. However, it is not always either
practical or cost effective.

They are more cost effective than endless debugging!!!

--
Phlip
http://www.oreilly.com/catalog/9780596510657/
"Test Driven Ajax (on Rails)"
assert_xpath, assert_javascript, & assert_ajax

[Ark Khasin]

Thank you folks for the fruitful discussion :)

[Ian Collins]

Ark Khasin wrote:

Thank you folks for the fruitful discussion :)

What was your conclusion?

--
Ian Collins.

[Ark Khasin]

Ian Collins wrote:

The last sentence is important, so I'll repeat it - the unit test are
built and run each time the module is compiled.

Assuming that the test code itself must be reasonably dumb (so that
/its/ errors immediately stand out), that's not terribly realistic:
imagine a sweep over, say, "int24_t" range. One could only hope to run
automated tests overnight - on a long night :).
--
Ark

[Ian Collins]

Ark Khasin wrote:

Ian Collins wrote:

>The last sentence is important, so I'll repeat it - the unit test are
built and run each time the module is compiled.

Assuming that the test code itself must be reasonably dumb (so that
/its/ errors immediately stand out), that's not terribly realistic:
imagine a sweep over, say, "int24_t" range. One could only hope to run
automated tests overnight - on a long night :).

It may not appear that way, but it is the reality on any project I
manage. In all (C++) cases, the tests take less time to run than the
code takes to build (somewhere between 50 and 100 tests per second,
unoptimised).

--
Ian Collins.

[Ian Collins]

Ark Khasin wrote:

Ian Collins wrote:

>>
Well TDD done correctly will give you the required coverage, might be
interesting proving it to the certifying agency though (a branch won't
be there unless a test requires it).

The mantra is, if a branch is to never be executed, it shall not be in
the production code. If it is executable, I must demonstrate how I
exercised it.

With TDD, if the branch isn't required to pass a test, it wont be there
at all.

--
Ian Collins.

[Keith Thompson]

Ark Khasin <ak*****@macroexpressions.comwrites:

Ian Collins wrote:

>The last sentence is important, so I'll repeat it - the unit test are
built and run each time the module is compiled.

Assuming that the test code itself must be reasonably dumb (so that
/its/ errors immediately stand out), that's not terribly realistic:
imagine a sweep over, say, "int24_t" range. One could only hope to run
automated tests overnight - on a long night :).

If every unit test has to check every possible value over a large
range, then yes, things could take a while. I just wrote a program
that iterated over a range of 2**24 in under a second, but if a
function takes three 32-bit arguments an exhaustive test starts to be
impractical.

But presumably in that case you'd just test a carefully chosen subset
of the possible argument values.

--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <* <http://users.sdsc.edu/~kst>
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"

[Phlip]

Ian Collins wrote:

Well TDD done correctly will give you the required coverage, might be
interesting proving it to the certifying agency though

Put the test into FIT, and let the certifying agency change the input
variables and watch the output responses change.

Oh, are you going to say there are "certifying agencies" out there which
_don't_ expect literate acceptance tests to cover their requirements??

Sure explains KBR, huh? (-;

--
Phlip
http://www.oreilly.com/catalog/9780596510657/
"Test Driven Ajax (on Rails)"
assert_xpath, assert_javascript, & assert_ajax

[Phlip]

Ark Khasin wrote:

Ian Collins wrote:

>The last sentence is important, so I'll repeat it - the unit test are
built and run each time the module is compiled.

Assuming that the test code itself must be reasonably dumb (so that /its/
errors immediately stand out), that's not terribly realistic: imagine a
sweep over, say, "int24_t" range. One could only hope to run automated
tests overnight - on a long night :).

Under TDD, you might only need enough tests to establish a linear function
along that int24_t (two tests), and enough to establish its extrema (two
more tests). These run fast.

If you then need more tests, you add them. There is no TDD "or" a formal
test campaign. You are allowed to use "unit test" techniques. For example,
you might calculate a sweep that covers a representative subset of all
possible integers.

If your tests run too long to help development, you push the slow ones out
into a slow suite, and run this on a test server. You can use one of many
Continuous Integration tools to trigger that batch run each time developers
commit. (And they should commit every 5 to 15 minutes.)

You leave every TDD test in the developers' suite.

If the long suites fail, you treat the failure the same as a bug reported by
users. You determine the fix, then write a trivial TDD test which fails
because the fix is not there. Note the test does not exhaustively prove the
fix is correct; it's just enough to pin the fix down. You leave this test
case with the developers' suite.

A program that uses a wide range of an int24_t's possible states is a
program with a high dimensional space. All programs have such a space! The
exhaustive test for such spaces would take forever to run. However, high
dimension spaces are inherently sparse. Tests only need to constrain
specific points and lines within that space.

One way to determine these points and lines is to analyze that space to
determine the minimum set of sweeps of inputs that will cover the whole
space. You test a carefully choses subset of the possible input values.

Another way is to write the tests first. Then you have a test for two points
on every line, and each endpoint to each line, and so on.

If you write tests first, you also get to avoid a lot of debugging. If your
tests fail unexpectedly, you have the option to revert back to the last
state where all tests passed, and try again.

This leads to some curious effects. Firstly, you can make much more savage
changes between each test run. You can even change code that someone else
wrote! a long time ago!! that everything else uses!!!

Next, if your tests are cheap and sloppy, but the code can't exist without
them, you get the ultimate in Design for Testing. You get Design BY Testing.
That means your tests might fail even if your code had no bug.

Again: Your tests might fail even if your code had no bug.

That means your code occupies a high-dimension space that is easy for your
tests to cover. So instead of analyzing your code and wrapping your tests
around it, you used Adaptive Planning with both the code and tests to
simplify that high-dimension space.

--
Phlip
http://www.oreilly.com/catalog/9780596510657/
"Test Driven Ajax (on Rails)"
assert_xpath, assert_javascript, & assert_ajax

[Everett M. Greene]

Ark Khasin <ak*****@macroexpressions.comwrites:

Ian Collins wrote:

The last sentence is important, so I'll repeat it - the unit test are
built and run each time the module is compiled.

Assuming that the test code itself must be reasonably dumb (so that
/its/ errors immediately stand out), that's not terribly realistic:
imagine a sweep over, say, "int24_t" range. One could only hope to run
automated tests overnight - on a long night :).

Is it really necessary to test all 2**24 values? It would seem that
testing the minimum, maximum, zero, and some representative values in
between would suffice. The "representative values" should be numerically
irrational (pi and e are good, for instance) so as to catch cases of
certain bits not being handled properly; 1 and 2 are not good choices
although they need to work properly as well.

In the area of branch testing, one has to test loops for proper
termination. [I just found some bugs last evening that involved
some simple counting loops that didn't terminate due to doing a
check for <0 on an unsigned value -- oops.]

[Ark Khasin]

Everett M. Greene wrote:

Is it really necessary to test all 2**24 values? It would seem that
testing the minimum, maximum, zero, and some representative values in
between would suffice. The "representative values" should be numerically
irrational (pi and e are good, for instance) so as to catch cases of
certain bits not being handled properly; 1 and 2 are not good choices
although they need to work properly as well.

The example of "gullibility measurement and conversion" in
http://www.macroexpressions.com/dl/C...shoestring.pdf
may be reasonably convincing

>
In the area of branch testing, one has to test loops for proper
termination. [I just found some bugs last evening that involved
some simple counting loops that didn't terminate due to doing a
check for <0 on an unsigned value -- oops.]

IMHO, unsigned<0 condition doesn't rise to testing: Lint will find it
before you compile

[Phlip]

Ark Khasin wrote:

[If we agree that a test is a contraption to check if the code works as
expected:]

The weakest possible such contraption - yes.

If we don't know what to expect ("research"), we cannot write a test. [Or
again I am missing something]

If you can think of the next line of code to write, you must perforce be
able to think of a test case that will fail because the line is not there.

Next, if you are talking about research to generate algorithms for some
situation, then you aren't talking about production code. Disposable code
doesn't need TDD. Once you have a good algorithm, it will have details that
lead to simple test cases.

E.g. if I'm writing a version control system, I know exactly what _has_ to
happen, and I can write the tests.
If e.g. I'm writing a monitoring code for the car's wheel speed sensors, I
may have a rock solid idea that e.g. whatever the speeds, the wheels
always remain in the vertices of a rectangle of the original size. Enter
sensor noise, wheel spinning, tire inflation and what not. I need lots of
code just to study what's going on before I can arrive at a sensible
algorithm.

That's an acceptance test. TDD tests don't give a crap if your code is
acceptable - if it targets wheels or wings. It's just a system to match
lines of code to trivial, nearly useless test cases.

--
Phlip
http://www.oreilly.com/catalog/9780596510657/
"Test Driven Ajax (on Rails)"
assert_xpath, assert_javascript, & assert_ajax

[Phlip]

Everett M. Greene wrote:

Have you ever worked in a product R&D environment?

Yes. I helped teach TDD and Python to polymaths who were into declaring
multi-dimensional arrays as void **.

A lot of concepts
are taken for a test drive without ever seeing the light of day outside
the lab. If the product does make it out the door, the original
concept proving/testing work is probably a very small portion of the
final product. You want to spend a lot of time and effort producing
more formalized testing processes for something that has a very low
probability of ever being used in a production environment?

TDD is faster and easier than debugger-oriented programming.

--
Phlip
http://www.oreilly.com/catalog/9780596510657/
"Test Driven Ajax (on Rails)"
assert_xpath, assert_javascript, & assert_ajax

[Ark Khasin]

Phlip wrote:

Next, if you are talking about research to generate algorithms for some
situation, then you aren't talking about production code. Disposable code
doesn't need TDD. Once you have a good algorithm, it will have details that
lead to simple test cases.

That's the whole point. I end up with some working prototype code for
which I need to create tests post factum.

[Ian Collins]

Phlip wrote:

Ark Khasin wrote:

>[If we agree that a test is a contraption to check if the code works as
expected:]

The weakest possible such contraption - yes.

>If we don't know what to expect ("research"), we cannot write a test. [Or
again I am missing something]

If you can think of the next line of code to write, you must perforce be
able to think of a test case that will fail because the line is not there.

Next, if you are talking about research to generate algorithms for some
situation, then you aren't talking about production code. Disposable code
doesn't need TDD. Once you have a good algorithm, it will have details that
lead to simple test cases.

I have found TDD to be a good tool for pointing me at a new algorithm.
It might just be the way a think, but given something I'd forgotten or
was too lazy to look up such as polynomial fitting, I start with a
simple flat line test, then a slope with two points, and so on until I
have a working general solution. I've found a dozen or so tests are
required to pop out a working solution. Given the working tests, the
algorithm can then be optimised.

--
Ian Collins.

[Ian Collins]

Everett M. Greene wrote:

"Phlip" <ph******@yahoo.comwrites:

>>
They don't. Now let's hear why these "research" environments are _allowed_
to write code without a failing test, first!

Have you ever worked in a product R&D environment? A lot of concepts
are taken for a test drive without ever seeing the light of day outside
the lab.

We call them spikes, or a proof of concept. Once the concept has been
proven, the code is put to one side and re-written using TDD.

Even these spikes can often be produced faster with TDD, the time saved
not debugging justifies the more formal approach.

It's unfortunate that us C and C++ programmers are spoiled rotten with
decent debuggers. Try developing something complex in an environment
without one and the benefits of TDD become clear. I do a lot of PHP and
I have never bothered looking for a PHP debugger.

--
Ian Collins.

[Phlip]

Ark Khasin wrote:

>Next, if you are talking about research to generate algorithms for some
situation, then you aren't talking about production code. Disposable code
doesn't need TDD. Once you have a good algorithm, it will have details
that lead to simple test cases.

That's the whole point. I end up with some working prototype code for
which I need to create tests post factum.

"Unit" tests post-factum. Not "developer tests" that support generating
production code.

Before you create this prototype code, do you _never_ debug it?

When researching, I frequently write disposable code test-free. When I
convert it to production code, I write the tests first. The result is much
cleaner for two reasons: It's a rewrite - that's always cleaner - and it's
super-easy to refactor. Without debugging.

--
Phlip
http://www.oreilly.com/catalog/9780596510657/
"Test Driven Ajax (on Rails)"
assert_xpath, assert_javascript, & assert_ajax

[Flash Gordon]

Ian Collins wrote, On 01/09/07 08:21:

Ark Khasin wrote:

>Ian Collins wrote:

>>The last sentence is important, so I'll repeat it - the unit test are
built and run each time the module is compiled.

Assuming that the test code itself must be reasonably dumb (so that
/its/ errors immediately stand out), that's not terribly realistic:
imagine a sweep over, say, "int24_t" range. One could only hope to run
automated tests overnight - on a long night :).

It may not appear that way, but it is the reality on any project I
manage. In all (C++) cases, the tests take less time to run than the
code takes to build (somewhere between 50 and 100 tests per second,
unoptimised).

This, however, is not always the case. I've written a function of about
20 lines that IIRC required something line 100-200 tests. If test took a
similar amount of time to run as the code took to compile. It was doing
maths and there where a *lot* of cases to consider.

For an audit of the entire piece of SW (rather than just that one
function) the customer insisted that we print out all of the module test
specs. The stack of A4 paper produced was a couple of feet tall! Running
that set of tests would take rather more than overnight.

On another project, doing a build of our piece of the SW took 8 hours.
Doing a build of all of the SW for the processor took 48 hours. Add
testing to that for each build...

Some projects are a lot harder than yours.
--
Flash Gordon

[Phlip]

Ian Collins wrote:

I have found TDD to be a good tool for pointing me at a new algorithm.
It might just be the way a think, but given something I'd forgotten or
was too lazy to look up such as polynomial fitting, I start with a
simple flat line test, then a slope with two points, and so on until I
have a working general solution. I've found a dozen or so tests are
required to pop out a working solution. Given the working tests, the
algorithm can then be optimised.

If you follow the exact refactoring rules, you'll remove all duplication
before adding the next line of code. I once tried that while generating an
algorithm to draw Roman Numerals, and I discovered that the outcome was
sensitive to one of my early refactors. The design I got sucked; it was
harder to code over time, not easier. I had to roll the entire process back
to that refactor, try it the other way, and _this_ time the correct
algorithm popped out.

TDD is a very good way to force a clean design to emerge, following simple
and known algorithms. But it's not a general-purpose algorithm generator.
Whoever discovers _that_ gets to go to the top of the food chain.

--
Phlip
http://www.oreilly.com/catalog/9780596510657/
"Test Driven Ajax (on Rails)"
assert_xpath, assert_javascript, & assert_ajax

[Ian Collins]

Phlip wrote:

Ian Collins wrote:

>I have found TDD to be a good tool for pointing me at a new algorithm.
It might just be the way a think, but given something I'd forgotten or
was too lazy to look up such as polynomial fitting, I start with a
simple flat line test, then a slope with two points, and so on until I
have a working general solution. I've found a dozen or so tests are
required to pop out a working solution. Given the working tests, the
algorithm can then be optimised.

If you follow the exact refactoring rules, you'll remove all duplication
before adding the next line of code. I once tried that while generating an
algorithm to draw Roman Numerals, and I discovered that the outcome was
sensitive to one of my early refactors. The design I got sucked; it was
harder to code over time, not easier. I had to roll the entire process back
to that refactor, try it the other way, and _this_ time the correct
algorithm popped out.

There you go, the solution is to hold off all but the most trivial
refactoring until the end!

--
Ian Collins.

[Ian Collins]

Flash Gordon wrote:

>
I know there is still SW that takes hours to build because within the
last few years I have done builds that have taken hours.

Must be huge, the biggest think I build regularly is the OpenSolaris
code base, which takes about 40 minutes on my box.

If a build takes too long, throw more cores at it. If the tools don't
support distributed building, change the tools.

--
Ian Collins.

[Ark Khasin]

Ark Khasin wrote:

Thank you folks for the fruitful discussion :)

Anyway, I posted a page, http://www.macroexpressions.com/maestra.html in
a hope that some people might find it useful.

-- Ark