[RFC] C token counter

[Arthur J. O'Dwyer]

Request for comments on the following program:
http://www.contrib.andrew.cmu.edu/~a...kshop/tokens.c

It's a token counter for the C programming language, following
the outline kind-of-described here:
http://www.kochandreas.com/home/lang...sts/TOKENS.HTM
Basically, it's supposed to give a reasonable approximation of
the number of "atomic tokens" present in a C source file.

To those reading in c.l.c: Are there any glaring mistakes or
poor coding styles in the program itself? And did I miss any
corner cases --- does the program produce a wrong number of
tokens for any valid C99 program?

To those reading in c.l.m: Andreas, this is for you. (:
I'm just posting it generally in case anyone has any comments
along the lines of "gee, C is nifty!" or "gee, C can't do
anything!"

Please set follow-ups appropriately in your reply: comp.lang.c
probably doesn't care about Practical Language Comparison ;) and
comp.lang.misc definitely doesn't want long pedantic debates about
whether #define is one token or two.[1] ;)

-Arthur

[1] - I'm counting it as one token in my program, even though
it looks like technically it's not a "token" in any sense of the
word, Standard-ly speaking.

[Nils M Holm]

In comp.lang.misc Arthur J. O'Dwyer <aj*@nospam.and rew.cmu.edu> wrote:

comp.lang.misc definitely doesn't want long pedantic debates about
whether #define is one token or two.[1] ;)

Why not? They /are/ two tokens:

# define x y

Nils

--
Nils M Holm <nm*@despammed. com> -- http://www.t3x.org/nmh/

[Martin Dickopp]

I'm reading this in comp.lang.c, followup set.

"Arthur J. O'Dwyer" <aj*@nospam.and rew.cmu.edu> writes:

Request for comments on the following program:
http://www.contrib.andrew.cmu.edu/~a...kshop/tokens.c

It's a token counter for the C programming language, following
the outline kind-of-described here:
http://www.kochandreas.com/home/lang...sts/TOKENS.HTM
Basically, it's supposed to give a reasonable approximation of
the number of "atomic tokens" present in a C source file.

To those reading in c.l.c: Are there any glaring mistakes or
poor coding styles in the program itself? And did I miss any
corner cases --- does the program produce a wrong number of
tokens for any valid C99 program?

I have just quickly skimmed your code, so please take my comments with a
grain of salt.

I prefer `int main (void)' over `int main ()', especially since you
don't use pre-C89-style definitions for all other functions.

When `getchar ()' returns `EOF', you should check if this is due to an
error or due to end-of-file condition.

The goal is apparently to count preprocessing-tokens, right? In that
case, you seem to parse numbers on a too detailed level. Preprocessing-
numbers are defined in much more general terms, see 6.4.8 (of the C99
standard) for details.

Martin
--
,--. Martin Dickopp, Dresden, Germany ,= ,-_-. =.
/ ,- ) http://www.zero-based.org/ ((_/)o o(\_))
\ `-' `-'(. .)`-'
`-. Debian, a variant of the GNU operating system. \_/

[Jeremy Yallop]

Arthur J. O'Dwyer wrote:

Request for comments on the following program:
http://www.contrib.andrew.cmu.edu/~a...kshop/tokens.c [...] To those reading in c.l.c: Are there any glaring mistakes or
poor coding styles in the program itself? And did I miss any
corner cases --- does the program produce a wrong number of
tokens for any valid C99 program?
It doesn't handle digraphs correctly: the following produces an error:

int main()<%%>

I think the task isn't well defined for a language with several
translation phases, like C, however. You have to decide whether to
count tokens or preprocessing tokens. In order to count tokens you
either need to specify that the input is already preprocessed, or
implement most of a preprocessor yourself. The following program has
35 preprocessing tokens, but only 11 tokens:

#define str(x) # x
int main() { puts(
str(
int main() { puts
("Hello, world.");
return 0;
}));}

Your program gives "34" for this (counting "#define" as a single
token), so it seems to be counting preprocessing tokens. However,
some valid preprocessing tokens, such as certain preprocessing
numbers, are rejected:

#define str(x) # x
int main() { str(3p+); }

I think all the programs above are strictly conforming.
comp.lang.misc definitely doesn't want long pedantic debates about
whether #define is one token or two.[1] ;) [...] [1] - I'm counting it as one token in my program, even though
it looks like technically it's not a "token" in any sense of the
word, Standard-ly speaking.

"#" is a punctuator, which is a token, and "define" is an identifier,
also a token.

Jeremy.

[Martin Dickopp]

Nils M Holm <nm*@despammed. com> writes:

In comp.lang.misc Arthur J. O'Dwyer <aj*@nospam.and rew.cmu.edu> wrote:

comp.lang.misc definitely doesn't want long pedantic debates about
whether #define is one token or two.[1] ;)

Since this posting might be the start of a long pedantic debate ;) about
`#define' being two tokens, I ignored the Followup-To: comp.lang.misc,
and set a Followup-To: comp.lang.c.
Why not? They /are/ two tokens:

# define x y

<pedantic>
While you're right, the fact that spaces are allowed between `#' and
`define' doesn't prove anything. ;) The language /could/ have defined
the sequence of `#', spaces, and `define' as a single preprocessing-
token. String literals are an example of preprocessing-tokens which may
include spaces.
</pedantic>

Martin
--
,--. Martin Dickopp, Dresden, Germany ,= ,-_-. =.
/ ,- ) http://www.zero-based.org/ ((_/)o o(\_))
\ `-' `-'(. .)`-'
`-. Debian, a variant of the GNU operating system. \_/

[Arthur J. O'Dwyer]

Combined response to Martin's and Jeremy's replies.
Thanks (so far...)!

On Fri, 30 Apr 2004, Arthur J. O'Dwyer wrote:

Request for comments on the following program:
http://www.contrib.andrew.cmu.edu/~a...kshop/tokens.c
Things I fixed:

int main(void) replaces int main(). Style issue.
Comments inside string and character literals weren't being handled.
Fixed now.
I had forgotten digraphs entirely. Fixed now.
Comments add spaces; e.g., makes a/**/b two tokens instead of one.
"# define" et al. are now two tokens instead of one. Jeremy's
argument convinced me, I guess. :) The fact that you can stick a
comment in between the "#" and the "define" contributed to my conviction,
too, although I knew that before.
Unrecognized or probably-invalid symbols are passed through anyway,
thus doing away with the NULL return from 'mystate'. This is to deal
with problems involving stringizing macros: basically *anything* could
be passed to them!
Martin Dickopp wrote:
The goal is apparently to count preprocessing-tokens, right? In that
case, you seem to parse numbers on a too detailed level. Preprocessing-
numbers are defined in much more general terms, see 6.4.8 (of the C99
standard) for details.
It seems at a quick glance that the pp-number definition would
make
0xDE+0xAD
into one token, where a more comprehensive approach would suggest it's
"really" three tokens --- 0xDE, +, and 0xAD. So while I agree I'm
doing too much with numbers, I don't yet see a better way that jibes
with the way I'm trying to define "tokens."
Jeremy Yallop wrote:
I think the task isn't well defined for a language with several
translation phases, like C, however. You have to decide whether to
count tokens or preprocessing tokens.

Right. At least, I have to make up a definition of "token"
that makes sense for most applications. Post-preprocessing tokens
are too complicated to handle on this program's scale; and you
pointed out the "stringizin g" issue, which I hadn't considered before
either.

Well, new version uploaded; same request for comments on this one. :)
Particularly, I'm not entirely sure that all numbers are handled
appropriately; and I'm not entirely sure that I did the digraph stuff
right --- especially 'PercentOp', which has to discriminate between
%, %:, %=, %>, and %:%:. FSMs are hard. ;-)

-Arthur

[Nils M Holm]

In comp.lang.misc Martin Dickopp <ex************ ****@zero-based.org> wrote:

While you're right, the fact that spaces are allowed between `#' and
`define' doesn't prove anything. ;) The language /could/ have defined
the sequence of `#', spaces, and `define' as a single preprocessing-
token.
While you are right, let's apply occam's razor: what is more likely,
that the parts of /some/ specific tokens can be separated by
(otherwise useless) spaces or that these space separate individual
tokens? :-)
String literals are an example of preprocessing-tokens which may
include spaces.

True. However, these spaces do have a purpose.

Nils

--
Nils M Holm <nm*@despammed. com> -- http://www.t3x.org/nmh/

[CBFalconer]

"Arthur J. O'Dwyer" wrote:

.... snip ...
Please set follow-ups appropriately in your reply: comp.lang.c
probably doesn't care about Practical Language Comparison ;) and
comp.lang.misc definitely doesn't want long pedantic debates about
whether #define is one token or two.[1] ;)

[1] - I'm counting it as one token in my program, even though
it looks like technically it's not a "token" in any sense of the
word, Standard-ly speaking.

I suggest you could have better set follow-ups yourself. How does
your code handle:

# if whatever
# define blah
# else
# undefine foo
# endif

which is perfectly legal, and necessary for some older compilers.

--
Chuck F (cb********@yah oo.com) (cb********@wor ldnet.att.net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home .att.net> USE worldnet address!

[Martin Dickopp]

"Arthur J. O'Dwyer" <aj*@nospam.and rew.cmu.edu> writes:

Martin Dickopp wrote:

The goal is apparently to count preprocessing-tokens, right? In that
case, you seem to parse numbers on a too detailed level. Preprocessing-
numbers are defined in much more general terms, see 6.4.8 (of the C99
standard) for details.
It seems at a quick glance that the pp-number definition would
make
0xDE+0xAD
into one token,

Yes, it's one pp-token.
where a more comprehensive approach would suggest it's "really" three
tokens --- 0xDE, +, and 0xAD.

It's a pp-token which cannot be converted to a token. Compare the
following two programs (difference underlined):

#include <stdio.h>
int main (void) { printf ("%d\n", 0xDE + 0xAD); return 0; }
/* ^^^^^^^^^^^ */

and

#include <stdio.h>
int main (void) { printf ("%d\n", 0xDE+0xAD); return 0; }
/* ^^^^^^^^^ */

The first one displays the number 395. The second one violates the
constraint of 6.4#2, so the implementation is free to interpret it in
any way it likes (after issuing at least one diagnostic).

Martin
PS: C specific discussion, therefore Followup-To: comp.lang.c.
--
,--. Martin Dickopp, Dresden, Germany ,= ,-_-. =.
/ ,- ) http://www.zero-based.org/ ((_/)o o(\_))
\ `-' `-'(. .)`-'
`-. Debian, a variant of the GNU operating system. \_/

[Martin Dickopp]

Nils M Holm <nm*@despammed. com> writes:

In comp.lang.misc Martin Dickopp <ex************ ****@zero-based.org> wrote:

While you're right, the fact that spaces are allowed between `#' and
`define' doesn't prove anything. ;) The language /could/ have defined
the sequence of `#', spaces, and `define' as a single preprocessing-
token.

While you are right, let's apply occam's razor: what is more likely,
that the parts of /some/ specific tokens can be separated by
(otherwise useless) spaces or that these space separate individual
tokens? :-)

There's not even a need to apply Occam's razor, since this aspect of the
C language is precisely defined: The likelihoods of the variants are
exactly 0 and 1, respectively. :)

Of course, I agree with your point: Defining the language such that the
sequence of `#', spaces, and `define' are a single token would have been
utterly stupid.

Martin
--
,--. Martin Dickopp, Dresden, Germany ,= ,-_-. =.
/ ,- ) http://www.zero-based.org/ ((_/)o o(\_))
\ `-' `-'(. .)`-'
`-. Debian, a variant of the GNU operating system. \_/