Safer and Better C

[bazad]

Hi,

I am not using C all the time. I have a general understanding of C
and nothing else. The recent reply to use strlcpy and strlcat showed
me that I am not aware of the best and safe techniques. Is there any
place where I could learn more about safer and better C (on FreeBSD)?
Thank you

[Paul Hsieh]

CBFalconer <cb********@yah oo.com> wrote:

Paul Hsieh wrote:

CBFalconer <cb********@yah oo.com> wrote:

Paul Hsieh wrote:
strlcpy and strlcat just transform the way in which a buffer
overflow can happen. They don't address the cause (human error
w.r.t. length calculations.)

Please don't give such misinformation. Those routines prevent
buffer overflows, and report the condition. They DO address the
human cause by simplifying supplying the appropriate parameter
values. Read the bloody documentation.
These functions don't magically associate legal memory size with the
string. That's still the responsibility of the programmer. So its
changed an implicit requirement to an explicit one. But it doesn't
address the real problem, which is that programmers make mistakes
which causes memory limit to be inadequate for the desired string
operation -- and there following is the buffer overrun.

C in general cannot perform such protection, and the cause is
rooted in the flagrant bandying about of pointers, pointer
arithmetic, transformations such as casts, the use of variadic
functions, and more. Strings are simply one aspect of this. We
can all easily avoid these problems by switching to a language
designed with correctness in mind, such as Pascal, Modula, Ada.

Pascal has the same problem -- trying to dereference a nil pointer,
for example (I am not an expert in Modula-2 or Ada, but as I recall
neither is GC based, so are likely to have the same problem.) You
have to go to Java/Python/Perl if you want to bring yourself into a
more totally insulated programming environment. Another very
interesting approach is the Cyclone programming language -- though it
might be a little syntactically annoying, it gives near C level of
program control while being completely safe. These languages do total
abstraction for you. The question is -- can you add just enough
abstraction to C to gain enough safety that dealing with the
weaknesses of the C language doesn't become burdensome or unmanageable
in the long run. One way to do this is develop ADTs with full
closure. Bstrlib is about as close as you can come to an ADT with
full closure in C.

You speak in these generalities with your opinions about these things
yet you demonstrate so clearly that you have not honestly examined
Bstrlib. You don't understand the interesting line in the sand that I
have drawn -- the real question I have put forth to programming world.
If you can develop ADTs with safety, speed, functionality,
portability and interoperabilit y of Bstrlib then is the switch to
other programming languages really necessary?
Unfortunately the world is full of macho programmers who seem to
feel they can handle an error-prone language, such as C, without
ever getting burnt. I am often among them.
I don't think programming in C is about machismo. That would be
programming in ASM.
I think it is not a good idea to supply areas that avoid the
typical C problems, without handling the overall (impossible)
problem. It only gives the unwashed a false sense of security, and
encourages those to make even graver errors elsewhere.
What basis is there for this outrageous statement? All programming
abstractions are basically schemes for managing complexity which are
supposed to improve productivity or safety -- including switching to
other languages. Some of the *typical* problems of C are repetitive,
and solving them over and over again has no value whatsoever. I
reject the notion that allowing beginner programmers to make mistakes
with strings will help them make less grave errors in other
programming.

In fact Bstrlib includes a module called bsafe which forcably
deprecates some of C's unsafe functionality -- so it can be argued
that it reduces some errors outside of Bstrlib.
Note that providing routines that can be controlled is not such an avoidance.

Routines like strlcat, and strlcpy don't solve the real problem. All
of the exact same buffer overflow scenarios are still present in the
same way and same sense, just with a somewhat reduced likelihood of
happening.

A direct comparison with Bstrlib shows that so many of the typical
buffer overflow scenarios are just not possible using Bstrlib. But it
is not only just some safe programming library. Its an *EXAMPLE* (and
its source code is available) of how to improve safety through proper
abstraction without significantly compromising on anything.

--
Paul Hsieh
http://www.pobox.com/~qed/
http://bstring.sf.net/

[CBFalconer]

Paul Hsieh wrote:

CBFalconer <cb********@yah oo.com> wrote:
.... snip ...
You speak in these generalities with your opinions about these things
yet you demonstrate so clearly that you have not honestly examined
Bstrlib. You don't understand the interesting line in the sand that I
have drawn -- the real question I have put forth to programming world.
If you can develop ADTs with safety, speed, functionality,
portability and interoperabilit y of Bstrlib then is the switch to
other programming languages really necessary?
I readily concede that point. The fact that so far I have had no
noticeable problems using the existing system has something to do
with it.

Unfortunately the world is full of macho programmers who seem to
feel they can handle an error-prone language, such as C, without
ever getting burnt. I am often among them.

I don't think programming in C is about machismo. That would be
programming in ASM.

I think it is not a good idea to supply areas that avoid the
typical C problems, without handling the overall (impossible)
problem. It only gives the unwashed a false sense of security,
and encourages those to make even graver errors elsewhere.

What basis is there for this outrageous statement? ... snip ...

My own opinion. I got rid of the training wheels on my childrens
bicycles at the earliest opportunity.

--
"I support the Red Sox and any team that beats the Yankees"
"Any baby snookums can be a Yankee fan, it takes real moral
fiber to be a Red Sox fan" - "I listened to Toronto come back
from 3:0 in '42, I watched Boston come back from 3:0 in '04"

[Paul Hsieh]

CBFalconer <cb********@yah oo.com> wrote:

Paul Hsieh wrote:

CBFalconer <cb********@yah oo.com> wrote:
You speak in these generalities with your opinions about these things
yet you demonstrate so clearly that you have not honestly examined
Bstrlib. You don't understand the interesting line in the sand that I
have drawn -- the real question I have put forth to programming world.
If you can develop ADTs with safety, speed, functionality,
portability and interoperabilit y of Bstrlib then is the switch to
other programming languages really necessary?

I readily concede that point. The fact that so far I have had no
noticeable problems using the existing system has something to do
with it.

It means you don't measure performance, and you don't measure the
danger or effort required to deal with buffer overruns. You do it for
ego, yet you don't realize that you are implicitely subordinating your
ego to Thompson, Kernigan and Ritchie and their vision for how strings
should be implemented.

The C language and libraries implementation of strings is really poor
from every angle of consideration except for extremely small systems
(like 8bit systems with < 64K, or ROM programming.) Bstrlib makes the
effort to leave all the worst aspects of '\0' terminated char *
strings behind while retaining just the right amount of
interoperabilit y to leverage the incumbancy of them.

You've let K&R&T tell you that string and binary buffers are
necessarily distinct. Its insane things like this which make binary
preserving text or hex editors all the more complicated to implement.

I think it is not a good idea to supply areas that avoid the
typical C problems, without handling the overall (impossible)
problem. It only gives the unwashed a false sense of security,
and encourages those to make even graver errors elsewhere.

What basis is there for this outrageous statement? ... snip ...

My own opinion. I got rid of the training wheels on my childrens
bicycles at the earliest opportunity.

But what you don't realize is that you've also taken off the gears
with the greatest torque. The C way of doing things is not just
dangerous but its also *SLOWER* (there is an example included in the
Bstrlib downloads which has a benchmark -- see for yourself.) So
you've accepted a more difficult challenge in order to achieve an
inferior result.

--
Paul Hsieh
http://www.pobox.com/~qed/
http://bstring.sf.net/

[CBFalconer]

Paul Hsieh wrote:

CBFalconer <cb********@yah oo.com> wrote:

Paul Hsieh wrote:

CBFalconer <cb********@yah oo.com> wrote: .... snip ... I think it is not a good idea to supply areas that avoid the
typical C problems, without handling the overall (impossible)
problem. It only gives the unwashed a false sense of security,
and encourages those to make even graver errors elsewhere.

What basis is there for this outrageous statement? ... snip ...

My own opinion. I got rid of the training wheels on my childrens
bicycles at the earliest opportunity.

But what you don't realize is that you've also taken off the gears
with the greatest torque. The C way of doing things is not just
dangerous but its also *SLOWER* (there is an example included in the
Bstrlib downloads which has a benchmark -- see for yourself.) So
you've accepted a more difficult challenge in order to achieve an
inferior result.

No, when I want a better, more secure language I simply use
Pascal. When I want to stay very close to the machine I use
assembly. When I am willing to compromise or want to maximize
practical portability I use C. Apart from the assembly I am
usually on very firm ISO standardized ground.

Bear in mind that I have no objection to you, or anyone else, using
your Bstrlib system. I have simply found no need so far, and
consider such efforts better applied to languages with a secure
foundation. I reserve the right to change my mind in future.

--
"I support the Red Sox and any team that beats the Yankees"
"Any baby snookums can be a Yankee fan, it takes real moral
fiber to be a Red Sox fan" - "I listened to Toronto come back
from 3:0 in '42, I watched Boston come back from 3:0 in '04"

[Richard Bos]

we******@gmail. com (Paul Hsieh) wrote:

interoperabilit y to leverage the incumbancy of them.

You have just told us that you're a manager, and can conveniently be
ignored in a technical discussion. Thanks for the frankness.

Richard

[Alan Balmer]

On Fri, 22 Oct 2004 07:16:56 GMT, rl*@hoekstra-uitgeverij.nl (Richard
Bos) wrote:

we******@gmail .com (Paul Hsieh) wrote:

interoperabilit y to leverage the incumbancy of them.

You have just told us that you're a manager, and can conveniently be
ignored in a technical discussion. Thanks for the frankness.

I think he's an academic. No difference to your conclusion, of course.

--
Al Balmer
Balmer Consulting
re************* ***********@att .net

[Dave Thompson]

On 18 Oct 2004 13:57:17 -0700, qe*@pobox.com (Paul Hsieh) wrote:
<snip>

11. If you have algorithms that only make sense for certain modes of
some parameters, try to implement them in functions with static
declaration. External interfaces should accept any combination and
modes of parameters so long as they are legal with respect to their
own type. The idea is that a developer should be able to read a .h
file read the function names, and already have a good idea of how to
use the module. Typically what prevents this is that usage of
functions have non-obvious parameter restrictions which requires that
developers read through documentation (which may or may not exist, may
be of poor quality, have errors in it, etc) to figure out what is
going on.
I'm not sure what you mean by "mode" of a parameter. I have seen it
used (in COBOL, Pascal, and Ada, and informally in F9X) to mean the
parameter-passing mechanism or form (value, reference, in, out, etc.)
Since C only has by-value-initialized, you can't mean that. It sounds
like you mean values, or ranges, or combinations of such, that are
representable in the declared types but not valid for the callee.
There is some controversy here though. *Personally* I insist on
*supporting* aliased parameters to the maximum degree possible.
However, I have basically seen almost no libraries that are
implemented with this in mind (gmp is an example of a library which
takes my point of view, for functionality reasons -- but you can see
how supporting aliasing can be very well motivated.) The assumption
of no aliasing is usually implicit or specifically required, even
though this is rarely enforced by "restrict" (which is not in
widespread use since C99 has not been adopted by any mainstream
compiler vendor.)

Note that 'restrict' even when implemented does not ENFORCE anything,
or at least is not required to and actual checking in nontrivial cases
would be costly so implementors are unlikely to do it. This is unlike
the other/preexisting qualifiers const and volatile which are "safe"
(and sometimes annoying!) in that they cannot be "removed" from the
type without casting or cheating. All 'restrict' does for the caller
is DOCUMENT the requirement of nonaliasing, in a standardized (and
conceivably tool-processable) way, and in a place (the prototype)
where the programmer using it is almost certain to need to look. The
real benefit is supposed to be on the callee side, for optimization.

- David.Thompson1 at worldnet.att.ne t

[Paul Hsieh]

Dave Thompson <da************ *@worldnet.att. net> wrote:

On 18 Oct 2004 13:57:17 -0700, qe*@pobox.com (Paul Hsieh) wrote:

11. If you have algorithms that only make sense for certain modes of
some parameters, try to implement them in functions with static
declaration. External interfaces should accept any combination and
modes of parameters so long as they are legal with respect to their
own type. The idea is that a developer should be able to read a .h
file read the function names, and already have a good idea of how to
use the module. Typically what prevents this is that usage of
functions have non-obvious parameter restrictions which requires that
developers read through documentation (which may or may not exist, may
be of poor quality, have errors in it, etc) to figure out what is
going on.
I'm not sure what you mean by "mode" of a parameter.

Actually what I mean is things like "this integer parameter cannot be
negative" or "this pointer cannot be NULL" or "this integer parameter
must correspond to a lower bound for the space available for a given
buffer parameter" etc. Those are ok for module-internal statically
declared functions. But for stuff you expose via extern, you should
just accept any combination of whatever parameters so long as they are
legal with respect to their type (for example pointers should either
point to something valid or be NULL -- not just be randomly
unitialized.)

There is some controversy here though. *Personally* I insist on
*supporting* aliased parameters to the maximum degree possible.
However, I have basically seen almost no libraries that are
implemented with this in mind (gmp is an example of a library which
takes my point of view, for functionality reasons -- but you can see
how supporting aliasing can be very well motivated.) The assumption
of no aliasing is usually implicit or specifically required, even
though this is rarely enforced by "restrict" (which is not in
widespread use since C99 has not been adopted by any mainstream
compiler vendor.)

Note that 'restrict' even when implemented does not ENFORCE anything,
or at least is not required to and actual checking in nontrivial cases
would be costly so implementors are unlikely to do it.

I agree that enforcing it completely and pervasively is basically
infeasible. But obviously the compiler can and should check the most
obvious cases from the call sites (i.e., I'd at least like a warning
for strcat(p,p).)
[...] This is unlike
the other/preexisting qualifiers const and volatile which are "safe"
(and sometimes annoying!) in that they cannot be "removed" from the
type without casting or cheating. All 'restrict' does for the caller
is DOCUMENT the requirement of nonaliasing, in a standardized (and
conceivably tool-processable) way, and in a place (the prototype)
where the programmer using it is almost certain to need to look. The
real benefit is supposed to be on the callee side, for optimization.

I understand all this. But since "restrict" is not very good at
*enforcing* behavior it makes it significantly less useful than const
or volatile. Errors that arise from incorrect aliasing handling can
be difficult to debug, so without enforcement, we are left with a
performance hack (like register or inline) that is inevitably
superseded by better technology (like cross-file-inlining or similar
techniques from which true non-Aliasing properties can be sussed out.)

I don't think being in this situation of ambiguity is ultimately
productive, so I instead take the opposite tack. What if instead we
demand proper suppose for aliasing in our libraries/modules? This
leads us to considering for the following questions about
implementation:

1) Can we detect aliasing at runtime with high performance?
2) Is there a proper interpretation for functions that take aliased
parameters?
3) Can algorithms be written that are aliasing neutral?
4) Can we implement our ADTs to restrict aliasing to only trivial
kinds of aliasing? (i.e., identical -- but without partial
overlapping.)

The memmove() library call in the latest x86 compilers actually follow
these principles in enacting their solution. The idea is that the
memmove() function tries to switch to memcpy() in a maximal number of
cases, before using a slower aliasing neutral algorithm for
implementation. So memmove() is usually not really slower than
memcpy(), while being somewhat safer.

For bignum libraries like gmp, rather than having a myriad of
functions for implementing what seem like seperate operations: A = B +
C, A += B, A += A, the simpler solution is only to support A = B + C,
where any of A, B, or C can be aliased. This keeps the API managable,
while not seriously impacting performance (detection and branch is
very low overhead in comparison to the inner loops of any bignum
function.)

For my string library "the better string library"
(http://bstring.sf.net/) what I found is that under the assumption
that memmove() has nearly identical performance to memcpy(), there is
no performance difference in implementing a completely alias-safe
library versus one that is not. Having aliased parameters does not
lead to any ambiguous interpretation of what any Bstrlib function
*should* do semantically. This also means I don't have document
caveats like "Don't ever write bconcat(b,b) or binsert(b,0,b)" ,
because they work exactly as expected.

--
Paul Hsieh
http://www.pobox.com/~qed/
http://bstring.sf.net/

[Chris Torek]

(Almost-off-topic drift, and I am only going to respond to this
one item...)

Dave Thompson <da************ *@worldnet.att. net> wrote:

I'm not sure what you mean by "mode" of a parameter.

In article <news:d0******* *************** ****@posting.go ogle.com>
Paul Hsieh <we******@gmail .com> wrote:Actually what I mean is things like "this integer parameter cannot be
negative" or "this pointer cannot be NULL" or "this integer parameter
must correspond to a lower bound for the space available for a given
buffer parameter" etc. ...
A better word for this -- or at least one more commonly used -- is
"constraint s". This also happens to be the word used in the C
standards (not entirely coincidentally) for its own requirements
upon the programmer.
Those are ok for module-internal statically
declared functions. But for stuff you expose via extern, you should
just accept any combination of whatever parameters so long as they are
legal with respect to their type (for example pointers should either
point to something valid or be NULL -- not just be randomly
unitialized. )

While I agree that, in general, weaker constraints are "better"
for exposed interfaces than stronger ones, sometimes strong(ish)
constraints seem to make sense. An example we had earlier (though
I have no idea whether it was in this same thread) occurs with
strlen(NULL): while the C standards could require that strlen()
return 0 in this case, and perhaps that strcpy() do nothing if
either of its operands is NULL, and so on, NULL is not actually a
string, and claiming that it *is* a string of length 0 is clearly
not entirely correct either. I would not object to strlen(NULL)
returning 0, but I do not object to its being considered a
dreadful mistake either (as C works today). I find neither one
"clearly superior" to the other: there are tradeoffs either way.

Languages with exceptions (Ada, C++, and Eiffel all come to mind)
can handle this by rejecting the attempt at runtime with an error;
and indeed, in Eiffel one can even express many constraints directly
in a function interface, so that compilers can catch some of these
errors at compile time. Once one buys into the "exceptions " model,
it becomes clear what to do with constraint violations: instead
of allowing all possible inputs (and generating "the least garbagey
possible output" for garbage input), just throw an exception.
--
In-Real-Life: Chris Torek, Wind River Systems
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W) [phone]
email: forget about it http://web.torek.net/torek/index.html
Reading email is like searching for food in the garbage, thanks to spammers.

[Paul Hsieh]

Chris Torek <no****@torek.n et> wrote:

Paul Hsieh <we******@gmail .com> wrote:

Those are ok for module-internal statically
declared functions. But for stuff you expose via extern, you should
just accept any combination of whatever parameters so long as they are
legal with respect to their type (for example pointers should either
point to something valid or be NULL -- not just be randomly
unitialized. )
While I agree that, in general, weaker constraints are "better"
for exposed interfaces than stronger ones, sometimes strong(ish)
constraints seem to make sense. An example we had earlier (though
I have no idea whether it was in this same thread) occurs with
strlen(NULL): while the C standards could require that strlen()
return 0 in this case, and perhaps that strcpy() do nothing if
either of its operands is NULL, and so on, NULL is not actually a
string, and claiming that it *is* a string of length 0 is clearly
not entirely correct either. I would not object to strlen(NULL)
returning 0, but I do not object to its being considered a
dreadful mistake either (as C works today). I find neither one
"clearly superior" to the other: there are tradeoffs either way.

Ok, but that's because you are assuming the limits of size_t as output
and are just hacking in this extra condition. You see? You haven't
*DESIGNED* the right answer, you are just seeing if hacking in this
NULL <-> "" equivalence would work and are evaluating it from there.

Look, what happens if you change the definition of strlen like this:

int strlen (const char * s);

Which returns -1 if s is NULL, otherwise the same as strlen. But, now
we've limited the size of our strings to INT_MAX in length, rather
than the maximum of size_t. So if we think this is acceptable or can
ignore this for a second, then we have the return value telling us
about a typical error condition or else giving us the length in one
shot. This allows the user to treat it as the strlen they are used
to, or otherwise having at least some way of putting a low level
sanity check in there. More sophisticated platform specific debug
libraries could put more work into determining if the pointer s is
really a valid readable memory location or not and pile onto the -1
error condition.

Then of course we could change strcpy() to return NULL, if one of the
parameters is NULL as another indication of error.

Now the getting rid of half your integer range thing might be a
difficult pill to swallow, so as an alternative you can return
UINT_MAX (or whatever size_t's maximum value is -- why the hell isn't
there a SIZE_T_MAX in limits.h?!?! -- anyone who continues to put
K&R&T or anyone of the C standards committee up on a pedistal needs a
serious lobotomy) as the error value (thus eliminating only one
possible length.) If this is unsatisfactory then we always have:

int strlen (size_t * sz, const char * s);

but then it can't be used in an arithmetic expression.

If you haven't seen the punchline coming from a mile away yet, I'll be
very disappointed. Go look at http://bstring.sf.net/ to see how I
dealt with this exact issue. Strings in the bstring library can only
legally have lengths between 0 and INT_MAX. So my blength() function
returns an integer and happily returns -1 if you pass in NULL, or
other easily detectable flawed input that doesn't otherwise lead to
UB. So I give up the possibility of using monster long strings beyond
the size of INT_MAX -- and I don't have to worry about closure issues,
because its not feasible to construct a bstring that is longer in the
same way that a malloc'ed char * could be. I don't even consider that
a trade off -- for all the functionality, safety and speed I obtain, I
don't miss the possibility of generating certain incredibly long
strings that I have never encountered in my life of programming so
far.
Languages with exceptions (Ada, C++, and Eiffel all come to mind)
can handle this by rejecting the attempt at runtime with an error;
and indeed, in Eiffel one can even express many constraints directly
in a function interface, so that compilers can catch some of these
errors at compile time. Once one buys into the "exceptions " model,
it becomes clear what to do with constraint violations: instead
of allowing all possible inputs (and generating "the least garbagey
possible output" for garbage input), just throw an exception.

Dude -- you know the only reason C++ has exceptions is because there
is no program control for constructors or destructors (and therefore
nowhere for them to return an error code.) Anyhow, yes other
languages have other ways of obtaining closure. But now tell me who's
being off topic in c.l.c? So can I go back to posting how I think C
should be able to assume 2s complement arithmetic, have a widening
multiply, bit-scan, coroutines, and a seriously programmaticall y
enabled preprocessor?

Anyhow, if you view putting in code to handle NULL pointers as being
"garbage moderation" then I think you've missed the point. As you
point out, C doesn't have exception handling. So the next best thing
you've got is to just return error codes out of everything that might
screw up.

And you should be viewing this as a means of obtaining operational
closure, not just some interesting feature-add. Look, malloc can
return NULL, so pointers can be filled will NULL instead of pointing
to well formed blocks of memory. If you claim that you should always
test malloc for returning NULL and deal with failure cases, then why
not just stick with the typical malloc wrapper than exits the program
whenever NULL is returned? The easy answer: malloc() failing is not
necessarily fatal to the rest of your application. This is of
particular importance to Bstrlib which tries to allocate memory in
powers of 2; if it finds it cannot, then instead of failing
immediately, it tries to malloc the tightest possible fiting boundary
instead.

Just as malloc doesn't, similarly bstrlib doesn't set policy about
failed bstring construction -- if it fails (because there is no
memory) then it doesn't try to exit your program. It just returns
NULL then lets the programmer notice this or not. Then by having
closure (i.e., allowing parameters to be NULL, but returning an error
in such cases, so that the error continues to propogate) it gives the
programmer a myriad of options for dealing with such cases.

And of course to complete the picture, I also support parameter
aliasing, for a fairly generous definition of what aliasing could
legally mean.

This isn't just a "nice to have" -- its a powerful concept that takes
less code than you think and has scarcely any performance impact. The
reason why proponents of other languages poo poo C programmers is
because we don't have built-in safety nets in our code that keep us
from killing ourselves on UB at every turn. But my claim is that,
that might be more a matter of culture than what the real technical
limitations of C is.

People don't try to write such safe code in C because they think C is
supposed to be lean and mean and that doing so will bloat their code
or something. The minimal Bstrlib module is like 8K of footprint and
the C library has no chance of matching up to it in terms of
performance. People are just wrong. You can make libraries or just
code in general in C that are safe, fast, and as functional as any
higher level language.

You have to stop letting the short sightedness of Ritchie, Kernigan,
Thompson and the C standards committee keeping you from seeing C as a
language from which you can design sophisticated structures. Bstrlib
is not just some weird cases where everything seems to work out. Last
month I posted some abstracted linked list code which (while probably
not very fast) has a lot of the same safety, power, and simplicity of
Bstrlib. C is not just a fancy assembly language. At least I try not
to treat it so simply.

--
Paul Hsieh
http://www.pobox.com/~qed/
http://bstring.sf.net/