Our OWN 'Deep C Secrets'

[Kamilche]

You know, all this furor over this book caused me to go look it up on
Amazon. I've never read this book... but from what I can see from the
legally available table of contents, excerpt, and index at Amazon, it
looks more like a "Teach me newbie C" book than a "UNCOVER DEEP
MYSTERIES OF C!!" sort of affair.

I've got a better idea! Let's discuss some 'Deep C Secrets' of our
own! I'll start.

Testing - If you haven't tested to prove it right, it can almost
certainly be proven to be WRONG. It's terribly easy to screw
something up in C, causing buffer overruns and undefined behavior.
It's gotten to the point where I don't feel comfortable calling coding
done until the tests are written. There's too many places C can bite
you.

I wasn't this paranoid in other programming languages... you know, the
ones that take 1/10 the time to write, and run 100x slower, like VB.
Because you didn't have fine-grained control over the code, there were
fewer places to screw up something fundamental. But you paid for it in
loss of control, loss of speed, and a less stable development
environment, because you had to rely on OTHER programmer's buggy code.
If it's YOUR buggy code, you at least have a fighting chance of
figuring out what's wrong with it and fixing it.

Uh, look there. A pearl of wisdom! Grab it and start your OWN book.
:-P

[Daniel Sjöblom]

Kamilche wrote:

You know, all this furor over this book caused me to go look it up on
Amazon. I've never read this book... but from what I can see from the
legally available table of contents, excerpt, and index at Amazon, it
looks more like a "Teach me newbie C" book than a "UNCOVER DEEP
MYSTERIES OF C!!" sort of affair.

I've got a better idea! Let's discuss some 'Deep C Secrets' of our
own! I'll start.

Testing - If you haven't tested to prove it right, it can almost
certainly be proven to be WRONG. It's terribly easy to screw
something up in C, causing buffer overruns and undefined behavior.

Except you can almost *never* prove anything right by testing it. That
is a very dangerous mind set that is bound to screw you over sooner or
later. The only sure way to prove a program correct is to write a
mathematical proof. Testing only works if the size of the input is
bounded and it can all be tested. (But tests are good because they can
prove something is *wrong* with a piece of code.)
--
Daniel Sjöblom
Remove _NOSPAM to reply by mail

[Martin Dickopp]

Daniel Sjöblom <ds******@mbnet .fi_NOSPAM> writes:

Except you can almost *never* prove anything right by testing it. That
is a very dangerous mind set that is bound to screw you over sooner or
later. The only sure way to prove a program correct is to write a
mathematical proof.
However, a mathematical proof created by a human can contain logical
fallacies, so you also have to prove the proof correct. And as the
proof of the proof can again contain logical fallacies, you also have
to prove it correct, and so on ad infinitum.
Testing only works if the size of the input is bounded and it can all
be tested. (But tests are good because they can prove something is
*wrong* with a piece of code.)

Given the choice between a 10 million lines of code program which comes
with a mathematical proof of correctness, but has not been tested, and
a thoroughly tested 10 million lines of code program which has not been
proven correct, I'd trust the latter more.

As Stanford mathematician Donald E. Knuth once put it [1]: "Beware of
bugs in the above code; I have only proved it correct, not tried it." :)

Martin
[1] http://www-cs-faculty.stanford.edu/~knuth/faq.html

--
,--. Martin Dickopp, Dresden, Germany ,= ,-_-. =.
/ ,- ) http://www.zero-based.org/ ((_/)o o(\_))
\ `-' `-'(. .)`-'
`-. Debian, a variant of the GNU operating system. \_/

[CBFalconer]

Daniel Sjöblom wrote:

Kamilche wrote:

.... snip ...

Testing - If you haven't tested to prove it right, it can almost
certainly be proven to be WRONG. It's terribly easy to screw
something up in C, causing buffer overruns and undefined behavior.

Except you can almost *never* prove anything right by testing it.
That is a very dangerous mind set that is bound to screw you over
sooner or later. The only sure way to prove a program correct is
to write a mathematical proof. Testing only works if the size of
the input is bounded and it can all be tested. (But tests are
good because they can prove something is *wrong* with a piece of
code.)

The technique of loop invariants can prove some assertions about
various routines. They can also help to document the
preconditions required. Unfortunately virtually nobody bothers.
Having proved those results, they can then be used in larger
areas, provided you keep the overall system well and cleanly
structured.

--
fix (vb.): 1. to paper over, obscure, hide from public view; 2.
to work around, in a way that produces unintended consequences
that are worse than the original problem. Usage: "Windows ME
fixes many of the shortcomings of Windows 98 SE". - Hutchison

[Kamilche]

Daniel Sjöblom <ds******@mbnet .fi_NOSPAM> wrote in message news:<40******* *************** @news.mbnet.fi> ...

Except you can almost *never* prove anything right by testing it. That
is a very dangerous mind set that is bound to screw you over sooner or
later. The only sure way to prove a program correct is to write a
mathematical proof. Testing only works if the size of the input is
bounded and it can all be tested. (But tests are good because they can
prove something is *wrong* with a piece of code.)

Yes, that's true - testing doesn't prove the absence of errors, it
only stops the stupidest ones. A programmer who does boundary
testing, tests for success, tests for failures, and stress testing,
can rest easier at night knowing they've made an honest effort.
Believe it or not, in my experience, most programmers DON'T unit test
their programs... they only do so if there's a bug so hard-to-find
that normal testing doesn't catch it. Sad, but true.

What's the testing policy in YOUR shop?

[John L]

"Kamilche" <kl*******@home .com> wrote in message news:88******** *************** ***@posting.goo gle.com...

Yes, that's true - testing doesn't prove the absence of errors, it
only stops the stupidest ones. A programmer who does boundary
testing, tests for success, tests for failures, and stress testing,
can rest easier at night knowing they've made an honest effort.
Believe it or not, in my experience, most programmers DON'T unit test
their programs... they only do so if there's a bug so hard-to-find
that normal testing doesn't catch it. Sad, but true.

One thing that can undermine testing is management merely
adding discovered flaws to a list of bugs (or "issues") rather
than immediately fixing them.

Another is not automating the analysis of test output. It can
be too easy to take any plausible output as being correct,
especially for our own code. A former colleague once asked
me to help track down an elusive bug in his hash table code.
We reran all his unit tests, one of which involved storing and
retrieving one word beginning with each letter of the alphabet.
Fortunately, I knew that there should have been 26 words output,
not 25. The programmer was a lot brighter than I am but was
faced with the classic problem of proofreading his own work:
he saw what he knew what should have been there.

--
John.

[Daniel Sjöblom]

Martin Dickopp wrote:

Daniel Sjöblom <ds******@mbnet .fi_NOSPAM> writes:

Except you can almost *never* prove anything right by testing it. That
is a very dangerous mind set that is bound to screw you over sooner or
later. The only sure way to prove a program correct is to write a
mathematica l proof.

However, a mathematical proof created by a human can contain logical
fallacies, so you also have to prove the proof correct.

Actually no. A proof cannot contain logical fallacies. If it did, it
wouldn't be a proof :-)

Testing only works if the size of the input is bounded and it can all
be tested. (But tests are good because they can prove something is
*wrong* with a piece of code.)

Given the choice between a 10 million lines of code program which comes
with a mathematical proof of correctness, but has not been tested, and
a thoroughly tested 10 million lines of code program which has not been
proven correct, I'd trust the latter more.

You read too much into my comment. I never said it was an either or
choice. I'd prefer both tested and proved code.
--
Daniel Sjöblom
Remove _NOSPAM to reply by mail

[Darrell Grainger]

On Sat, 8 May 2004, Kamilche wrote:

You know, all this furor over this book caused me to go look it up on
Amazon. I've never read this book... but from what I can see from the
legally available table of contents, excerpt, and index at Amazon, it
looks more like a "Teach me newbie C" book than a "UNCOVER DEEP
MYSTERIES OF C!!" sort of affair.

I've got a better idea! Let's discuss some 'Deep C Secrets' of our
own! I'll start.

Testing - If you haven't tested to prove it right, it can almost
certainly be proven to be WRONG. It's terribly easy to screw
something up in C, causing buffer overruns and undefined behavior.
It's gotten to the point where I don't feel comfortable calling coding
done until the tests are written. There's too many places C can bite
you.
I'd have to disagree with this. Far too often I have seen people use
testing to make themselves feel good about their code. I believe testing
is just a way to audit your design. It is a way to make sure you did the
right thing. If you did not design the right thing then how are you going
to test it? In other words, if I didn't think to code to prevent X then
why would I test for X?

Design and attention to detail are important. Empirical testing cannot
give 100% certainty. The number of combinations in just a simple project
is incredibly high. If there is user input then the numbers sky-rocket.

You need to think about all the things the program is capable of. Define
what you want it to do for all those capabilities. Then testing becomes a
way to confirm you did what you planned.

If you have introduced undefined behaviour into our program, testing might
not catch it. One of the most common mistake is that programmers test and
test and test and test. After testing to the point of insanity they decide
it is good and ship it. First problem, how many different configurations
did they test on? Probably one or two. Maybe ten to twenty different
configurations. Second problem, how many total person hours went into
testing? Let's say 10 people testing every day for 8 hours. They test for
a month (20 business days) for a total of 1600 hours. You ship it and sell
1000 copies. In 2 hours there has been more 'testing' by the customers
then you did in a month. There will be a lot of duplication with the
customers use but within a month or two they will find things you did not.

If the program was badly designed or requirements were not well defined,
I'd consider testing the same as using a bucket to keep a leaky boat from
sinking.

Some companies do a 'beta' release just to get more empirical testing on
the product. To me this is using a bigger bucket.
I wasn't this paranoid in other programming languages... you know, the
ones that take 1/10 the time to write, and run 100x slower, like VB.
Because you didn't have fine-grained control over the code, there were
fewer places to screw up something fundamental. But you paid for it in
loss of control, loss of speed, and a less stable development
environment, because you had to rely on OTHER programmer's buggy code.
If it's YOUR buggy code, you at least have a fighting chance of
figuring out what's wrong with it and fixing it.

Uh, look there. A pearl of wisdom! Grab it and start your OWN book.
:-P

--
Send e-mail to: darrell at cs dot toronto dot edu
Don't send e-mail to vi************@ whitehouse.gov

[August Derleth]

On Mon, 10 May 2004 16:38:13 +0300, Daniel Sjöblom wrote:

Martin Dickopp wrote:

Daniel Sjöblom <ds******@mbnet .fi_NOSPAM> writes:

Except you can almost *never* prove anything right by testing it. That
is a very dangerous mind set that is bound to screw you over sooner or
later. The only sure way to prove a program correct is to write a
mathematic al proof.

However, a mathematical proof created by a human can contain logical
fallacies, so you also have to prove the proof correct.

Actually no. A proof cannot contain logical fallacies. If it did, it
wouldn't be a proof :-)

Nonsense. If you call it a proof, I have to disprove your logic to
contradict you. If I don't follow your logic, or if your logical method is
wrong in a way we both miss, your proof is worthless yet we will both
believe it is valid.

Running a suite of tests is (or should be) trivial, and it should catch
all obvious possible errors and all errors related to outliers in the
possible input sets. If errors persist, the only reasonable way to catch
them is through real-life testing and debugging. Which is what really
counts anyway: Real-life performance, not ivory tower mathematics.

In any case, how do we test airframes and cars and other physical objects?
Crash testing, proving grounds, and other engineering methods. Mathematics
only describes what we already know, and it can exhaustively search spaces
we have already tested physically. In an absurd case, that might even be
useful. Actual testing, on the other hand, tells us something new.

--
yvoregnevna gjragl-guerr gjb-gubhfnaq guerr ng lnubb qbg pbz
To email me, rot13 and convert spelled-out numbers to numeric form.
"Makes hackers smile" makes hackers smile.

[Kamilche]

da*****@NOMORES PAMcs.utoronto. ca.com (Darrell Grainger) wrote in message news:<Pi******* *************** *********@drj.p f>...

I'd have to disagree with this. Far too often I have seen people use
testing to make themselves feel good about their code... ... Second problem, how many total person hours went into
testing? Let's say 10 people testing every day for 8 hours. They test for
a month (20 business days) for a total of 1600 hours. You ship it and sell
1000 copies. In 2 hours there has been more 'testing' by the customers
then you did in a month. There will be a lot of duplication with the
customers use but within a month or two they will find things you did not.

If the program was badly designed or requirements were not well defined,
I'd consider testing the same as using a bucket to keep a leaky boat from
sinking.

You can't artifically inflate the numbers to support your argument,
and use 'many customers will test for me' as a justification to get
out of doing unit testing. Unit testing is done by the programmer, not
users... and trying to 'pass the buck' to quality control or
end-users, is one of the reasons the software industry is in such bad
shape today. There's no way ten people SHOULD be testing every day for
8 hours, for a month, just to make up for a lazy programmer who
refuses to unit test. The programmer who wrote the code, should be the
programmer who writes the unit tests, and the tests should run
automatically, either at code compilation time, or once at startup.
Live human testers should be reserved for system integration tests,
the 'big picture' bugs that programmers don't often catch on their
own.

It sounds to me, like you're trying to feel good WITHOUT testing your
code... and you really shouldn't. You say your code works? Prove it.
When embedded in the source code, the test code serves double duty as
a usage example, as well, so it certainly won't be wasted effort.