Deficiency of strtok

[Pilcrow]

Here is a quick program, together with its output, that illustrates what
I consider to be a deficiency of the standard function strtok from
<string.h>:

>#include <stdio.h>
#include <string.h>

char line1[] = "a,bbb,cc",
line2[] = ",pp,qqq",
line3[] = "r,,s",
line4[] = "r, ,s",
*token1, *token2, *token3;

int main()
{
puts("Line# Input Tokenized\n"
"----- -------- -------------");

printf("1: %8s", line1);
token1 = strtok(line1,", ");
token2 = strtok(NULL,"," );
token3 = strtok(NULL,"," );
printf(" %s-%s-%s\n",token1,to ken2,token3);

printf("2: %8s", line2);
token1 = strtok(line2,", ");
token2 = strtok(NULL,"," );
token3 = strtok(NULL,"," );
printf(" %s-%s-%s\n",token1,to ken2,token3);

printf("3: %8s", line3);
token1 = strtok(line3,", ");
token2 = strtok(NULL,"," );
token3 = strtok(NULL,"," );
printf(" %s-%s-%s\n",token1,to ken2,token3);

printf("4: %8s", line4);
token1 = strtok(line4,", ");
token2 = strtok(NULL,"," );
token3 = strtok(NULL,"," );
printf(" %s-%s-%s\n",token1,to ken2,token3);
return(0);
}
/* OUTPUT:

Line# Input Tokenized
----- -------- -------------
1: a,bbb,cc a-bbb-cc
2: ,pp,qqq pp-qqq-(null)
3: r,,s r-s-(null)
4: r, ,s r- -s

*/

I am using
C:\>gcc --version
gcc (GCC) 3.4.5 (mingw special)

I would like there to be a default, to be returned when two delimiters
(the commas, here) abut each other.

Is there a way of getting this effect without writing my own function?
Is there some other workaround?

I am new to C, being experienced in perl, where this would be duck soup.

[Martien Verbruggen]

On Thu, 07 Aug 2008 17:14:29 -0700,
Pilcrow <pi*****@pp.inf owrote:

>
Here is a quick program, together with its output, that illustrates what
I consider to be a deficiency of the standard function strtok from

[snip]

I would like there to be a default, to be returned when two delimiters
(the commas, here) abut each other.

I think I understood from the above that you have a problem with
strtok() seeing a sequence of separator characters as a single
separator?

Is there a way of getting this effect without writing my own function?
Is there some other workaround?

If you don't want to use C99 features, you could consider creating
something with strcspn(), which wouldn't be too hard, or you
could use strchr() if you only have the one separator character.

Martien
--
|
Martien Verbruggen | I used to have a Heisenbergmobil e. Every time
| I looked at the speedometer, I got lost.
|

[Pilcrow]

On Fri, 8 Aug 2008 11:04:49 +1000, Martien Verbruggen
<mg**@tradingpo st.com.auwrote:

>On Thu, 07 Aug 2008 17:14:29 -0700,
Pilcrow <pi*****@pp.inf owrote:

>>
Here is a quick program, together with its output, that illustrates what
I consider to be a deficiency of the standard function strtok from

[snip]

>I would like there to be a default, to be returned when two delimiters
(the commas, here) abut each other.

I think I understood from the above that you have a problem with
strtok() seeing a sequence of separator characters as a single
separator?

>Is there a way of getting this effect without writing my own function?
Is there some other workaround?

If you don't want to use C99 features, you could consider creating
something with strcspn(), which wouldn't be too hard, or you
could use strchr() if you only have the one separator character.

Martien

Thanks, looks like I could do one of these. I should have been able to
figure that out myself. BTW, what C99 features are you referring to?

[Martien Verbruggen]

On Thu, 07 Aug 2008 18:30:09 -0700,
Pilcrow <pi*****@pp.inf owrote:

On Fri, 8 Aug 2008 11:04:49 +1000, Martien Verbruggen
<mg**@tradingp ost.com.auwrote :

>>On Thu, 07 Aug 2008 17:14:29 -0700,
Pilcrow <pi*****@pp.inf owrote:

>>If you don't want to use C99 features, you could consider creating
something with strcspn(), which wouldn't be too hard, or you
could use strchr() if you only have the one separator character.

Thanks, looks like I could do one of these. I should have been able to
figure that out myself. BTW, what C99 features are you referring to?

I started out by writing a response using strpbrk(), but realised that
that wasn't the right function anyway. When I edited my post, I forgot
to remove that reference. Sorry for the confusion.

Martien
--
|
Martien Verbruggen | There are only 10 types of people in the
| world; those who understand binary and those
| who don't.

[Dann Corbit]

"Pilcrow" <pi*****@pp.inf owrote in message
news:i4******** *************** *********@4ax.c om...

>
Here is a quick program, together with its output, that illustrates what
I consider to be a deficiency of the standard function strtok from

[snip]

Here is a reentrant function called tokenize that accomplishes the same
thing as strtok() while remaining reentrant.
Modify it to do whatever you like.

#include <string.h>
#include <limits.h>
#include <stdlib.h>
#include <ctype.h>

/* The default delimiters are chosen as some ordinary white space
characters: */
static const char default_delimit ers[] =
{' ', '\n', '\t', '\r', '\f', 0};

/*
* The tokenize() function is similar to a reentrant version of strtok().
* It parses tokens from 's', where tokens are substrings separated by
* characters from 'delimiter_list '.
* To get the first token from 's', tokenize() is called with 's' as its
first
* parameter.
* Remaining tokens from 's' are obtained by calling tokenize() with NULL
for
* the first parameter.
* The s of delimiters, identified by 'delimiter_list ', can change from call
* to call.
* If the list of delimiters is NULL, then the standard list
'default_delimi ters'
* (see above) is used.
* tokenize() modifies the memory pointed to by 's', because it writes null
* characters into the buffer.
*/
char *tokenize(char *s, const char *delimiter_list , char **placeholder)
{
if (delimiter_list == NULL)
delimiter_list = default_delimit ers;

if (delimiter_list[0] == 0)
delimiter_list = default_delimit ers;

if (s == NULL)
s = *placeholder;

if (s == NULL)
return NULL;
/*
* The strspn() function computes the length of the initial segment of the
first
* string that consists entirely of characters contained in the second
string.
*/
s += strspn(s, delimiter_list) ;
if (!s[0]) {
*placeholder = s;
return NULL;
} else {
char *token;
token = s;
/*
* The strpbrk() function finds the first occurrence of any character
contained in
* the second string found in the first string.
*/
s = strpbrk(token, delimiter_list) ;
if (s == NULL)
*placeholder = token + strlen(token);
else {
*s++ = 0;
*placeholder = s;
}
return token;
}
}

#ifdef UNIT_TEST
char ts0[] =
"This is a test. This is only a test. If it were an actual emergency, you
would"
" be dead.";
char ts1[] =
"This is a also a test. This is only a test. If it were an actual emergency,
you"
" would be dead. 12345";
char ts2[] =
"The quick brown fox jumped over the lazy dog's back 1234567890 times.";
char ts3[] =
" \t\r\n\fThe quick brown fox jumped over the lazy dog's back 1234567890
times.";
char ts4[] =
"This is a test. This is only a test. If it were an actual emergency, you
would"
" be dead.";
char ts5[] =
"This is a also a test. This is only a test. If it were an actual emergency,
you"
" would be dead. 12345";
char ts6[] =
"The quick brown fox jumped over the lazy dog's back 1234567890 times.";
char ts7[] =
" \t\r\n\fThe quick brown fox jumped over the lazy dog's back 1234567890
times.";

#include <stdio.h>

char whitespace[UCHAR_MAX + 1];

/*
This test will create token separators as any whitespace or any punctuation
marks:
*/
void init_whitespace ()
{
int i;
int index = 0;
for (i = 0; i < UCHAR_MAX; i++) {
if (isspace(i)) {
whitespace[index++] = (char) i;
}
if (ispunct(i)) {
whitespace[index++] = (char) i;
}
}
}

/*
TNX Gerd.
*/
void spin_test(char *ts, char *white)
{
char *p = NULL;
char *token;
token = tokenize(ts, white, &p);
while (token) {
puts(token);
token = tokenize(NULL, white, &p);
}
}

int main(void)
{
init_whitespace ();
puts("Whitespac e is whitespace+punc tuation");
spin_test(ts0, whitespace);
spin_test(ts1, whitespace);
spin_test(ts2, whitespace);
spin_test(ts3, whitespace);
puts("Whitespac e is simple whitespace");
spin_test(ts4, NULL);
spin_test(ts5, NULL);
spin_test(ts6, NULL);
spin_test(ts7, NULL);
return 0;
}
#endif

** Posted from http://www.teranews.com **

[CBFalconer]

Pilcrow wrote:

>
Here is a quick program, together with its output, that illustrates
what I consider to be a deficiency of the standard function strtok
from <string.h>:

Here is my solution to that problem.

/* ------- file tknsplit.h ----------*/
#ifndef H_tknsplit_h
# define H_tknsplit_h

# ifdef __cplusplus
extern "C" {
# endif

#include <stddef.h>

/* copy over the next tkn from an input string, after
skipping leading blanks (or other whitespace?). The
tkn is terminated by the first appearance of tknchar,
or by the end of the source string.

The caller must supply sufficient space in tkn to
receive any tkn, Otherwise tkns will be truncated.

Returns: a pointer past the terminating tknchar.

This will happily return an infinity of empty tkns if
called with src pointing to the end of a string. Tokens
will never include a copy of tknchar.

released to Public Domain, by C.B. Falconer.
Published 2006-02-20. Attribution appreciated.
revised 2007-05-26 (name)
*/

const char *tknsplit(const char *src, /* Source of tkns */
char tknchar, /* tkn delimiting char */
char *tkn, /* receiver of parsed tkn */
size_t lgh); /* length tkn can receive */
/* not including final '\0' */

# ifdef __cplusplus
}
# endif
#endif
/* ------- end file tknsplit.h ----------*/

/* ------- file tknsplit.c ----------*/
#include "tknsplit.h "

/* copy over the next tkn from an input string, after
skipping leading blanks (or other whitespace?). The
tkn is terminated by the first appearance of tknchar,
or by the end of the source string.

The caller must supply sufficient space in tkn to
receive any tkn, Otherwise tkns will be truncated.

Returns: a pointer past the terminating tknchar.

This will happily return an infinity of empty tkns if
called with src pointing to the end of a string. Tokens
will never include a copy of tknchar.

A better name would be "strtkn", except that is reserved
for the system namespace. Change to that at your risk.

released to Public Domain, by C.B. Falconer.
Published 2006-02-20. Attribution appreciated.
Revised 2006-06-13 2007-05-26 (name)
*/

const char *tknsplit(const char *src, /* Source of tkns */
char tknchar, /* tkn delimiting char */
char *tkn, /* receiver of parsed tkn */
size_t lgh) /* length tkn can receive */
/* not including final '\0' */
{
if (src) {
while (' ' == *src) src++;

while (*src && (tknchar != *src)) {
if (lgh) {
*tkn++ = *src;
--lgh;
}
src++;
}
if (*src && (tknchar == *src)) src++;
}
*tkn = '\0';
return src;
} /* tknsplit */

#ifdef TESTING
#include <stdio.h>

#define ABRsize 6 /* length of acceptable tkn abbreviations */

/* ---------------- */

static void showtkn(int i, char *tok)
{
putchar(i + '1'); putchar(':');
puts(tok);
} /* showtkn */

/* ---------------- */

int main(void)
{
char teststring[] = "This is a test, ,, abbrev, more";

const char *t, *s = teststring;
int i;
char tkn[ABRsize + 1];

puts(teststring );
t = s;
for (i = 0; i < 4; i++) {
t = tknsplit(t, ',', tkn, ABRsize);
showtkn(i, tkn);
}

puts("\nHow to detect 'no more tkns' while truncating");
t = s; i = 0;
while (*t) {
t = tknsplit(t, ',', tkn, 3);
showtkn(i, tkn);
i++;
}

puts("\nUsing blanks as tkn delimiters");
t = s; i = 0;
while (*t) {
t = tknsplit(t, ' ', tkn, ABRsize);
showtkn(i, tkn);
i++;
}
return 0;
} /* main */

#endif
/* ------- end file tknsplit.c ----------*/

--
[mail]: Chuck F (cbfalconer at maineline dot net)
[page]: <http://cbfalconer.home .att.net>
Try the download section.

[Pilcrow]

On Fri, 08 Aug 2008 00:11:08 -0400, CBFalconer <cb********@yah oo.com>
wrote:

>Pilcrow wrote:

>>
Here is a quick program, together with its output, that illustrates
what I consider to be a deficiency of the standard function strtok
from <string.h>:

Here is my solution to that problem.

<snip>

Thanks so much to all of you, Martien, Dann, and Chuck. You have given
me so much material to pore over and digest. I've compiled and run the
programs Dann & Chuck gave me, and they ran wonderfully. Now to study
all of it, and absorb it.

- P.

[Richard]

CBFalconer <cb********@yah oo.comwrites:

Pilcrow wrote:

>>
Here is a quick program, together with its output, that illustrates
what I consider to be a deficiency of the standard function strtok
from <string.h>:

Here is my solution to that problem.

/* ------- file tknsplit.h ----------*/
#ifndef H_tknsplit_h
# define H_tknsplit_h

# ifdef __cplusplus
extern "C" {
# endif

#include <stddef.h>

/* copy over the next tkn from an input string, after
skipping leading blanks (or other whitespace?). The
tkn is terminated by the first appearance of tknchar,
or by the end of the source string.

The caller must supply sufficient space in tkn to
receive any tkn, Otherwise tkns will be truncated.

Returns: a pointer past the terminating tknchar.

This will happily return an infinity of empty tkns if
called with src pointing to the end of a string. Tokens
will never include a copy of tknchar.

released to Public Domain, by C.B. Falconer.
Published 2006-02-20. Attribution appreciated.
revised 2007-05-26 (name)
*/

const char *tknsplit(const char *src, /* Source of tkns */
char tknchar, /* tkn delimiting char */
char *tkn, /* receiver of parsed tkn */
size_t lgh); /* length tkn can receive */
/* not including final '\0' */

# ifdef __cplusplus
}
# endif
#endif
/* ------- end file tknsplit.h ----------*/

/* ------- file tknsplit.c ----------*/
#include "tknsplit.h "

/* copy over the next tkn from an input string, after
skipping leading blanks (or other whitespace?). The
tkn is terminated by the first appearance of tknchar,
or by the end of the source string.

The caller must supply sufficient space in tkn to
receive any tkn, Otherwise tkns will be truncated.

Returns: a pointer past the terminating tknchar.

This will happily return an infinity of empty tkns if
called with src pointing to the end of a string. Tokens
will never include a copy of tknchar.

A better name would be "strtkn", except that is reserved
for the system namespace. Change to that at your risk.

released to Public Domain, by C.B. Falconer.
Published 2006-02-20. Attribution appreciated.
Revised 2006-06-13 2007-05-26 (name)
*/

const char *tknsplit(const char *src, /* Source of tkns */
char tknchar, /* tkn delimiting char */
char *tkn, /* receiver of parsed tkn */
size_t lgh) /* length tkn can receive */
/* not including final '\0' */
{
if (src) {
while (' ' == *src) src++;

while (*src && (tknchar != *src)) {
if (lgh) {
*tkn++ = *src;
--lgh;
}
src++;
}
if (*src && (tknchar == *src)) src++;
}

I would replace the function with this more or less for the following
reasons:

Back to front comparisons are used in a minority of code and most people
hate them. And yes I do know "most" is not "all". Naming of variables
seems almost meaningles - lgh is what? it doesnt save much compiling
time to use meaningful names in a publicly released library. Oh and like
other functions assume it is called with meaningful data.

,----
|
|/* remove leading white space */
|while(*source= =' ')
| *source++;
|
|/* store string up to next token */
|while(maxLengt h-- && ((tokenChar = *source++) != endOfTokenChar) )
| *savedToken++=t okenChar;
|
|*savedToken='\ 0';
|
`----

*tkn = '\0';
return src;
} /* tknsplit */

#ifdef TESTING
#include <stdio.h>

#define ABRsize 6 /* length of acceptable tkn abbreviations */

/* ---------------- */

static void showtkn(int i, char *tok)
{
putchar(i + '1'); putchar(':');
puts(tok);
} /* showtkn */

/* ---------------- */

int main(void)
{
char teststring[] = "This is a test, ,, abbrev, more";

const char *t, *s = teststring;
int i;
char tkn[ABRsize + 1];

puts(teststring );
t = s;
for (i = 0; i < 4; i++) {
t = tknsplit(t, ',', tkn, ABRsize);
showtkn(i, tkn);
}

puts("\nHow to detect 'no more tkns' while truncating");
t = s; i = 0;
while (*t) {
t = tknsplit(t, ',', tkn, 3);
showtkn(i, tkn);
i++;
}

puts("\nUsing blanks as tkn delimiters");
t = s; i = 0;
while (*t) {
t = tknsplit(t, ' ', tkn, ABRsize);
showtkn(i, tkn);
i++;
}
return 0;
} /* main */

#endif
/* ------- end file tknsplit.c ----------*/

--

[Richard]

Richard<rg****@ gmail.comwrites :

CBFalconer <cb********@yah oo.comwrites:

,----
|
|/* remove leading white space */
|while(*source= =' ')
| *source++;

Whoops! ^ error above left to the inquisitive to spot and pick on ....

|
|/* store string up to next token */
|while(maxLengt h-- && ((tokenChar = *source++) != endOfTokenChar) )
| *savedToken++=t okenChar;
|
|*savedToken='\ 0';
|
`----

> *tkn = '\0';
return src;
} /* tknsplit */

#ifdef TESTING
#include <stdio.h>

#define ABRsize 6 /* length of acceptable tkn abbreviations */

/* ---------------- */

static void showtkn(int i, char *tok)
{
putchar(i + '1'); putchar(':');
puts(tok);
} /* showtkn */

/* ---------------- */

int main(void)
{
char teststring[] = "This is a test, ,, abbrev, more";

const char *t, *s = teststring;
int i;
char tkn[ABRsize + 1];

puts(teststring );
t = s;
for (i = 0; i < 4; i++) {
t = tknsplit(t, ',', tkn, ABRsize);
showtkn(i, tkn);
}

puts("\nHow to detect 'no more tkns' while truncating");
t = s; i = 0;
while (*t) {
t = tknsplit(t, ',', tkn, 3);
showtkn(i, tkn);
i++;
}

puts("\nUsing blanks as tkn delimiters");
t = s; i = 0;
while (*t) {
t = tknsplit(t, ' ', tkn, ABRsize);
showtkn(i, tkn);
i++;
}
return 0;
} /* main */

#endif
/* ------- end file tknsplit.c ----------*/

--

[santosh]

Richard wrote:

CBFalconer <cb********@yah oo.comwrites:

<snip>

>const char *tknsplit(const char *src, /* Source of tkns */
char tknchar, /* tkn delimiting char */
char *tkn, /* receiver of parsed tkn */
size_t lgh) /* length tkn can receive */
/* not including final '\0' */
{
if (src) {
while (' ' == *src) src++;

while (*src && (tknchar != *src)) {
if (lgh) {
*tkn++ = *src;
--lgh;
}
src++;
}
if (*src && (tknchar == *src)) src++;
}

I would replace the function with this more or less for the following
reasons:

Back to front comparisons are used in a minority of code and most
people hate them. And yes I do know "most" is not "all". Naming of
variables seems almost meaningles - lgh is what? it doesnt save much
compiling time to use meaningful names in a publicly released library.
Oh and like other functions assume it is called with meaningful data.

,----
|
|/* remove leading white space */
|while(*source= =' ')

Wouldn't you be better of using isspace()?

| *source++;

I suppose this should be source++?

|/* store string up to next token */
|while(maxLengt h-- && ((tokenChar = *source++) != endOfTokenChar) )
| *savedToken++=t okenChar;
|
|*savedToken='\ 0';
|
`----

<snip>