[RFC] C token counter

[Arthur J. O'Dwyer]

Request for comments on the following program:
http://www.contrib.andrew.cmu.edu/~a...kshop/tokens.c

It's a token counter for the C programming language, following
the outline kind-of-described here:
http://www.kochandreas.com/home/lang...sts/TOKENS.HTM
Basically, it's supposed to give a reasonable approximation of
the number of "atomic tokens" present in a C source file.

To those reading in c.l.c: Are there any glaring mistakes or
poor coding styles in the program itself? And did I miss any
corner cases --- does the program produce a wrong number of
tokens for any valid C99 program?

To those reading in c.l.m: Andreas, this is for you. (:
I'm just posting it generally in case anyone has any comments
along the lines of "gee, C is nifty!" or "gee, C can't do
anything!"

Please set follow-ups appropriately in your reply: comp.lang.c
probably doesn't care about Practical Language Comparison ;) and
comp.lang.misc definitely doesn't want long pedantic debates about
whether #define is one token or two.[1] ;)

-Arthur

[1] - I'm counting it as one token in my program, even though
it looks like technically it's not a "token" in any sense of the
word, Standard-ly speaking.

[Arthur J. O'Dwyer]

On Fri, 30 Apr 2004, Martin Dickopp wrote:

"Arthur J. O'Dwyer" <aj*@nospam.and rew.cmu.edu> writes:

Martin Dickopp wrote:

The goal is apparently to count preprocessing-tokens, right? In that
case, you seem to parse numbers on a too detailed level. Preprocessing-
numbers are defined in much more general terms, see 6.4.8 (of the C99
standard) for details.

It seems at a quick glance that the pp-number definition would
make
0xDE+0xAD
into one token,

Yes, it's one pp-token.

where a more comprehensive approach would suggest it's "really" three
tokens --- 0xDE, +, and 0xAD.

It's a pp-token which cannot be converted to a token. Compare the
following two programs (difference underlined):

#include <stdio.h>
int main (void) { printf ("%d\n", 0xDE + 0xAD); return 0; }
/* ^^^^^^^^^^^ */

and

#include <stdio.h>
int main (void) { printf ("%d\n", 0xDE+0xAD); return 0; }
/* ^^^^^^^^^ */

% gcc test.c
test.c: In function `main':
test.c:2: invalid suffix on integer constant

Wow. I learn something new every day!
This looks like *very* weird behavior to me... what's C's
rationale for parsing numbers this way rather than the "common-
sense" approach I *thought* it used? It would not be hard to
make 0xDE+0xAD the addition of the hex constants 0xDE and 0xAD,
rather than a syntax error; why did C decide to do the latter,
then? Just to make the formal lexing spec simpler?

-Arthur,
confused but slightly more enlightened

[Jeremy Yallop]

Arthur J. O'Dwyer wrote:

This looks like *very* weird behavior to me... what's C's rationale
for parsing numbers this way rather than the "common- sense"
approach I *thought* it used?
Dennis Ritchie's explanation seems plausible:

"In truth, I think that preprocessing numbers are the most
conspicuously incorrect thing X3J11 did. [...] Reportedly the idea
was worked out over lunch; it bears more signs of a late-night bar
session."
<news:Cy******* *@research.att. com>
It would not be hard to make 0xDE+0xAD the addition of the hex
constants 0xDE and 0xAD, rather than a syntax error; why did C
decide to do the latter, then? Just to make the formal lexing spec
simpler?

Yes. The C89 Rationale has the details:

3.1.8 Preprocessing numbers

The notion of preprocessing numbers has been introduced to simplify
the description of preprocessing. It provides a means of talking
about the tokenization of strings that look like numbers, or
initial substrings of numbers, prior to their semantic
interpretation. In the interests of keeping the description
simple, occasional spurious forms are scanned as preprocessing
numbers --- 0x123E+1 is a single token under the rules. The
Committee felt that it was better to tolerate such anomalies than
burden the preprocessor with a more exact, and exacting, lexical
specification. It felt that this anomaly was no worse than the
principle under which the characters a+++++b are tokenized as a ++
++ + b (an invalid expression), even though the tokenization a ++ +
++ b would yield a syntactically correct expression. In both cases,
exercise of reasonable precaution in coding style avoids surprises.

I don't see how it simplifies anything much, really, unless you're
writing a stand-alone preprocessor. Numbers have to be parsed
properly at some stage: the addition of preprocessing numbers means
that two number parsers (with conflicting behaviour) are needed rather
than one.

Jeremy.

[CBFalconer]

"Arthur J. O'Dwyer" wrote:

On Fri, 30 Apr 2004, Martin Dickopp wrote:

.... snip ...
It's a pp-token which cannot be converted to a token. Compare
the following two programs (difference underlined):

#include <stdio.h>
int main (void) { printf ("%d\n", 0xDE + 0xAD); return 0; }
/* ^^^^^^^^^^^ */

and

#include <stdio.h>
int main (void) { printf ("%d\n", 0xDE+0xAD); return 0; }
/* ^^^^^^^^^ */

% gcc test.c
test.c: In function `main':
test.c:2: invalid suffix on integer constant

Wow. I learn something new every day!
This looks like *very* weird behavior to me... what's C's
rationale for parsing numbers this way rather than the "common-
sense" approach I *thought* it used? It would not be hard to
make 0xDE+0xAD the addition of the hex constants 0xDE and 0xAD,
rather than a syntax error; why did C decide to do the latter,
then? Just to make the formal lexing spec simpler?

Consider 0xD. This is a hex value. The following E signifies a
floating point value, with the exponent following, which is
+0xAD. No different than 2e-23 in principle.

This is the sort of thing idiots who economize on blanks are
subject to, and that they impose on the poor suffering maintenance
programmer.

--
Chuck F (cb********@yah oo.com) (cb********@wor ldnet.att.net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home .att.net> USE worldnet address!

[Arthur J. O'Dwyer]

On Fri, 30 Apr 2004, CBFalconer wrote:

"Arthur J. O'Dwyer" wrote:

On Fri, 30 Apr 2004, Martin Dickopp wrote:

#include <stdio.h>
int main (void) { printf ("%d\n", 0xDE+0xAD); return 0; }

test.c:2: invalid suffix on integer constant

It would not be hard to
make 0xDE+0xAD the addition of the hex constants 0xDE and 0xAD,
rather than a syntax error; why did C decide to do the latter,
then? Just to make the formal lexing spec simpler?

Consider 0xD. This is a hex value. The following E signifies a
floating point value, with the exponent following, which is
+0xAD. No different than 2e-23 in principle.

Wrong. "E" (or "e") signifies a floating-point exponent *only*
when used with decimal and octal constants. The floating-point
exponent signifier for hexadecimal constants is "P" (or "p"),
because "E" is a hex digit itself. 0xDE+1 is just as invalid a
C construct as 0xDE+0xAD. See, isn't that weird, now? ;)

-Arthur

[Peter Nilsson]

"Arthur J. O'Dwyer" <aj*@nospam.and rew.cmu.edu> wrote in message
news:Pi******** *************** ************@un ix40.andrew.cmu .edu...

Combined response to Martin's and Jeremy's replies.
Thanks (so far...)!

On Fri, 30 Apr 2004, Arthur J. O'Dwyer wrote:

Request for comments on the following program:
http://www.contrib.andrew.cmu.edu/~a...kshop/tokens.c

If you're considering C99 digraphs, then...

g\u00E5

....should be reported as a single token.

--
Peter

[Arthur J. O'Dwyer]

On Sat, 1 May 2004, Peter Nilsson wrote:

"Arthur J. O'Dwyer" <aj*@nospam.and rew.cmu.edu> wrote...

Request for comments on the following program:
http://www.contrib.andrew.cmu.edu/~a...kshop/tokens.c

If you're considering C99 digraphs, then...

g\u00E5

...should be reported as a single token.

Request for clarification: First, universal character
names have nothing to do with digraphs, right? you just meant
that they're both obscure C99 features?
Second, are there any pitfalls involving these universal
characters? I just have to accept \u or \U in identifiers?
As I understand N869 6.4.3#2, I don't ever have to worry about
universal characters filling in for, say, digits or other
C tokens.
I don't think it's worth enumerating all those valid identifier
characters in Annex D of N869, for my program.

-Arthur

[Arthur J. O'Dwyer]

On Sat, 1 May 2004, Peter Nilsson wrote:

"Arthur J. O'Dwyer" <aj*@nospam.and rew.cmu.edu> wrote...

Request for comments on the following program:
http://www.contrib.andrew.cmu.edu/~a...kshop/tokens.c

If you're considering C99 digraphs, then...

g\u00E5

...should be reported as a single token.

Request for clarification: First, universal character
names have nothing to do with digraphs, right? you just meant
that they're both obscure C99 features?
Second, are there any pitfalls involving these universal
characters? I just have to accept \u or \U in identifiers?
As I understand N869 6.4.3#2, I don't ever have to worry about
universal characters filling in for, say, digits or other
C tokens.
I don't think it's worth enumerating all those valid identifier
characters in Annex D of N869, for my program.

-Arthur

[Peter Nilsson]

"Arthur J. O'Dwyer" <aj*@nospam.and rew.cmu.edu> wrote in message
news:Pi******** *************** ************@un ix49.andrew.cmu .edu...

On Sat, 1 May 2004, Peter Nilsson wrote:

"Arthur J. O'Dwyer" <aj*@nospam.and rew.cmu.edu> wrote...

>
> Request for comments on the following program:
> http://www.contrib.andrew.cmu.edu/~a...kshop/tokens.c
If you're considering C99 digraphs, then...

g\u00E5

...should be reported as a single token.

Request for clarification: First, universal character
names have nothing to do with digraphs, right? you just meant
that they're both obscure C99 features?

Yup. I should have typed: '(e.g. digraphs)', sorry.
Second, are there any pitfalls involving these universal
characters? I just have to accept \u or \U in identifiers?
The \u or \U must be followed by either 4 or 8 hex characters.

The only other constraint is...

A universal character name shall not specify a character whose
short identifier is less than 00A0 other than 0024 ($), 0040 (@),
or 0060 ('), nor one in the range D800 through DFFF inclusive.

[This differs fron N869.]
As I understand N869 6.4.3#2, I don't ever have to worry about
universal characters filling in for, say, digits or other
C tokens.
Yes.
I don't think it's worth enumerating all those valid identifier
characters in Annex D of N869, for my program.

Your call, but fair enough. [If you do, the standard (+TC1) didn't change any of the
characters listed in appendix D from N869.]

--
Peter

[Arthur J. O'Dwyer]

On Sun, 2 May 2004, Peter Nilsson wrote:

"Arthur J. O'Dwyer" <aj*@nospam.and rew.cmu.edu> wrote...

On Sat, 1 May 2004, Peter Nilsson wrote:

"Arthur J. O'Dwyer" <aj*@nospam.and rew.cmu.edu> wrote...
> >
> > Request for comments on the following program:
> > http://www.contrib.andrew.cmu.edu/~a...kshop/tokens.c

g\u00E5

<snip> Second, are there any pitfalls involving these universal
characters? I just have to accept \u or \U in identifiers?
The \u or \U must be followed by either 4 or 8 hex characters.

....but if it's not, then we have undefined behavior in the C program,
so I'm allowed to do whatever I like. So the semantics of the token
counter don't need to include the counting of hex digits.
The only other constraint is...

A universal character name shall not specify a character whose
short identifier is less than 00A0 other than 0024 ($), 0040 (@),
or 0060 ('), nor one in the range D800 through DFFF inclusive.

[This differs from N869.]

Eek! You scared me there for a moment! s/'/`/ in the above text!
If 0027 (') could be replaced by \u0027, *that* would have been really
icky. But 0060 (`) doesn't do anything in C, so it's okay.

I don't think it's worth enumerating all those valid identifier
characters in Annex D of N869, for my program.

Your call, but fair enough.

Again, as far as I can tell if the user enters an invalid universal
character name in the middle of an identifier, we get undefined
behavior. My current token-counter has gone ahead with the "pp-number"
semantics for counting numeric constants, so 0x0E4+D..P-xg27 counts as
one token; I don't see why FOO\u4D99BAR should be counted (or not) any
differently.

Thanks,
-Arthur

[Andreas Koch]

Jeremy Yallop wrote:

I think the task isn't well defined for a language with several
translation phases, like C, however. You have to decide whether to
count tokens or preprocessing tokens.

Good point. I'll add that. We want preprocessing tokens; neither
macros expanded nor files included etc.

--
Andreas
Andreas' practical language comparison:
http://www.kochandreas.com/home/language/lang.htm