473,322 Members | 1,671 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,322 software developers and data experts.

using strtok

I cam across an interesting limitation to the use of strtok.

I have two strings on which I want strtok to operate.
However since strtok has only one memory of the residual string I must
complete one set of operations before starting on the second. This
is inconvenient in the context of my program!

So far the only solution I can see is to write a replacement for strtok
to use on one of the strings. Can anyone offer an alternative?

--
_ _________________________________________
/ \._._ |_ _ _ /' Orpheus Internet Services
\_/| |_)| |(/_|_|_/ 'Internet for Everyone'
_______ | ___________./ http://www.orpheusinternet.co.uk
Mar 10 '07 #1
14 3293
On Mar 10, 12:35 pm, Mr John FO Evans <m...@orpheusmail.co.ukwrote:
I cam across an interesting limitation to the use of strtok.

I have two strings on which I want strtok to operate.
However since strtok has only one memory of the residual string I must
complete one set of operations before starting on the second. This
is inconvenient in the context of my program!

So far the only solution I can see is to write a replacement for strtok
to use on one of the strings. Can anyone offer an alternative?
Not really, unless you'd be willing to use <OT>POSIX/SUS's strtok_r()</
OTand skip on the portability. Otherwise, look at CBFalconer's
replacement toksplit(), discussed at
http://groups.google.com/group/comp....58085dd57c3a5b.
--
WYCIWYG - what you C is what you get

Mar 10 '07 #2

Mr John FO Evans wrote:
I cam across an interesting limitation to the use of strtok.

I have two strings on which I want strtok to operate.
However since strtok has only one memory of the residual string I must
complete one set of operations before starting on the second. This
is inconvenient in the context of my program!

So far the only solution I can see is to write a replacement for strtok
to use on one of the strings. Can anyone offer an alternative?
POSIX specifies a strtok_r that was designed to work around the
reentrancy issue of strtok. If you want full portability however,
you'll have to roll your own version. It's not difficult and can be
done in completely standard C. CBFalconer periodically publishes his
toksplit function to this group. Use Google Group's search facility to
locate the source.

Mar 10 '07 #3
Mr John FO Evans wrote:
>
I cam across an interesting limitation to the use of strtok.

I have two strings on which I want strtok to operate.
However since strtok has only one memory of the residual string I
must complete one set of operations before starting on the second.
This is inconvenient in the context of my program!

So far the only solution I can see is to write a replacement for
strtok to use on one of the strings. Can anyone offer an
alternative?
Try this:

/* ------- file toksplit.c ----------*/
#include "toksplit.h"

/* copy over the next token from an input string, after
skipping leading blanks (or other whitespace?). The
token is terminated by the first appearance of tokchar,
or by the end of the source string.

The caller must supply sufficient space in token to
receive any token, Otherwise tokens will be truncated.

Returns: a pointer past the terminating tokchar.

This will happily return an infinity of empty tokens if
called with src pointing to the end of a string. Tokens
will never include a copy of tokchar.

A better name would be "strtkn", except that is reserved
for the system namespace. Change to that at your risk.

released to Public Domain, by C.B. Falconer.
Published 2006-02-20. Attribution appreciated.
Revised 2006-06-13
*/

const char *toksplit(const char *src, /* Source of tokens */
char tokchar, /* token delimiting char */
char *token, /* receiver of parsed token */
size_t lgh) /* length token can receive */
/* not including final '\0' */
{
if (src) {
while (' ' == *src) src++;

while (*src && (tokchar != *src)) {
if (lgh) {
*token++ = *src;
--lgh;
}
src++;
}
if (*src && (tokchar == *src)) src++;
}
*token = '\0';
return src;
} /* toksplit */

#ifdef TESTING
#include <stdio.h>

#define ABRsize 6 /* length of acceptable token abbreviations */

/* ---------------- */

static void showtoken(int i, char *tok)
{
putchar(i + '1'); putchar(':');
puts(tok);
} /* showtoken */

/* ---------------- */

int main(void)
{
char teststring[] = "This is a test, ,, abbrev, more";

const char *t, *s = teststring;
int i;
char token[ABRsize + 1];

puts(teststring);
t = s;
for (i = 0; i < 4; i++) {
t = toksplit(t, ',', token, ABRsize);
showtoken(i, token);
}

puts("\nHow to detect 'no more tokens' while truncating");
t = s; i = 0;
while (*t) {
t = toksplit(t, ',', token, 3);
showtoken(i, token);
i++;
}

puts("\nUsing blanks as token delimiters");
t = s; i = 0;
while (*t) {
t = toksplit(t, ' ', token, ABRsize);
showtoken(i, token);
i++;
}
return 0;
} /* main */

#endif
/* ------- end file toksplit.c ----------*/

/* ------- file toksplit.h ----------*/
#ifndef H_toksplit_h
# define H_toksplit_h

# ifdef __cplusplus
extern "C" {
# endif

#include <stddef.h>

/* copy over the next token from an input string, after
skipping leading blanks (or other whitespace?). The
token is terminated by the first appearance of tokchar,
or by the end of the source string.

The caller must supply sufficient space in token to
receive any token, Otherwise tokens will be truncated.

Returns: a pointer past the terminating tokchar.

This will happily return an infinity of empty tokens if
called with src pointing to the end of a string. Tokens
will never include a copy of tokchar.

released to Public Domain, by C.B. Falconer.
Published 2006-02-20. Attribution appreciated.
*/

const char *toksplit(const char *src, /* Source of tokens */
char tokchar, /* token delimiting char */
char *token, /* receiver of parsed token */
size_t lgh); /* length token can receive */
/* not including final '\0' */

# ifdef __cplusplus
}
# endif
#endif
/* ------- end file toksplit.h ----------*/

--
<http://www.cs.auckland.ac.nz/~pgut001/pubs/vista_cost.txt>
<http://www.securityfocus.com/columnists/423>

"A man who is right every time is not likely to do very much."
-- Francis Crick, co-discover of DNA
"There is nothing more amazing than stupidity in action."
-- Thomas Matthews

--
Posted via a free Usenet account from http://www.teranews.com

Mar 10 '07 #4

"matevzb" <ma*****@gmail.comwrote in message
news:11**********************@s48g2000cws.googlegr oups.com...
>So far the only solution I can see is to write a replacement for strtok
to use on one of the strings. Can anyone offer an alternative?
Not really, unless you'd be willing to use <OT>POSIX/SUS's strtok_r()</
OTand skip on the portability. Otherwise, look at CBFalconer's
replacement toksplit(), discussed at
are POSIX sources available?

Mar 11 '07 #5
"Servé Laurijssen" wrote:
"matevzb" <ma*****@gmail.comwrote in message
>>So far the only solution I can see is to write a replacement for
strtok to use on one of the strings. Can anyone offer an alternative?

Not really, unless you'd be willing to use <OT>POSIX/SUS's strtok_r()<
/OTand skip on the portability. Otherwise, look at CBFalconer's
replacement toksplit(), discussed at

are POSIX sources available?
Anything published here, as ready to go, will either run on POSIX
or there will be much wailing, teeth gnashing and berating from the
regulars. We deal with portable code here. Which, in turn, is why
POSIX is off-topic.

Please do not remove attribution lines for material you quote.
Those are the initial lines that say "Joe wrote:" or similar.

--
Chuck F (cbfalconer at maineline dot net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home.att.net>

--
Posted via a free Usenet account from http://www.teranews.com

Mar 11 '07 #6
On Mar 11, 3:59 pm, "Servé Laurijssen" <s...@n.tkwrote:
"matevzb" <mate...@gmail.comwrote in message

news:11**********************@s48g2000cws.googlegr oups.com...
So far the only solution I can see is to write a replacement for strtok
to use on one of the strings. Can anyone offer an alternative?
Not really, unless you'd be willing to use <OT>POSIX/SUS's strtok_r()</
OTand skip on the portability. Otherwise, look at CBFalconer's
replacement toksplit(), discussed at

are POSIX sources available?
POSIX/SUS, similar to ISO C, is a specification, so the answer would
be no. Source code for specific implementations may be available (e.g.
GNU libc), but whether or not they are portable and/or conform to
POSIX is another question. I'd say you're better off with toksplit().
--
WYCIWYG - what you C is what you get

Mar 11 '07 #7

"CBFalconer" <cb********@yahoo.comwrote in message
news:45***************@yahoo.com...
Anything published here, as ready to go, will either run on POSIX
or there will be much wailing, teeth gnashing and berating from the
regulars. We deal with portable code here. Which, in turn, is why
POSIX is off-topic.
I was just wondering why your function is not considered off-topic but every
time posix is mentioned its off topic. One could mention that to get a
portable version of strtok_r you can strip it from posix.
Please do not remove attribution lines for material you quote.
Those are the initial lines that say "Joe wrote:" or similar.
was mistake sorry
Mar 11 '07 #8
"Servé Laurijssen" <se*@n.tkwrites:
I was just wondering why your function is not considered off-topic but every
time posix is mentioned its off topic. One could mention that to get a
portable version of strtok_r you can strip it from posix.
POSIX is a standard. It's not a collection of source code from
which you can strip anything.
--
int main(void){char p[]="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuv wxyz.\
\n",*q="kl BIcNBFr.NKEzjwCIxNJC";int i=sizeof p/2;char *strchr();int putchar(\
);while(*q){i+=strchr(p,*q++)-p;if(i>=(int)sizeof p)i-=sizeof p-1;putchar(p[i]\
);}return 0;}
Mar 11 '07 #9
santosh wrote:
POSIX specifies a strtok_r that was designed to work around the
reentrancy issue of strtok. If you want full portability however,
you'll have to roll your own version. It's not difficult and can be
done in completely standard C.
#include <stddef.h>

char *str_tok_r(char *s1, const char *s2, char **s3);
char *str_chr(const char *s, int c);
size_t str_spn(const char *s1, const char *s2);
size_t str_cspn(const char *s1, const char *s2);

char *str_tok_r(char *s1, const char *s2, char **s3)
{
if (s1 != NULL) {
*s3 = s1;
}
s1 = *s3 + str_spn(*s3, s2);
if (*s1 == '\0') {
return NULL;
}
*s3 = s1 + str_cspn(s1, s2);
if (**s3 != '\0') {
*(*s3)++ = '\0';
}
return s1;
}

size_t str_spn(const char *s1, const char *s2)
{
size_t n;

for (n = 0; *s1 != '\0' && str_chr(s2, *s1) != NULL; ++s1) {
++n;
}
return n;
}

size_t str_cspn(const char *s1, const char *s2)
{
size_t n;

for (n = 0; str_chr(s2, *s1) == NULL; ++s1) {
++n;
}
return n;
}

char *str_chr(const char *s, int c)
{
while (*s != (char)c) {
if (*s == '\0') {
return NULL;
}
++s;
}
return (char *)s;
}

--
pete
Mar 11 '07 #10
In article <87************@blp.benpfaff.org>,
Ben Pfaff <bl*@cs.stanford.eduwrote:
>One could mention that to get a
portable version of strtok_r you can strip it from posix.
>POSIX is a standard. It's not a collection of source code from
which you can strip anything.
s/posix/one of the free posix implementations/

-- Richard
--
"Consideration shall be given to the need for as many as 32 characters
in some alphabets" - X3.4, 1963.
Mar 11 '07 #11
Mr John FO Evans <mi***@orpheusmail.co.ukwrote:
I cam across an interesting limitation to the use of strtok.

I have two strings on which I want strtok to operate.
However since strtok has only one memory of the residual string I must
complete one set of operations before starting on the second. This
is inconvenient in the context of my program!

So far the only solution I can see is to write a replacement for strtok
to use on one of the strings. Can anyone offer an alternative?
No. It's one of the many ways in which strtok() is unsuitable for most
of the jobs it was intended for. I've only come across a situation in
which strtok() was the right tool for the job, and I've since forgotten
what it was.

Richard
Mar 12 '07 #12
Servé Laurijssen wrote, On 11/03/07 19:02:
"CBFalconer" <cb********@yahoo.comwrote in message
news:45***************@yahoo.com...
>Anything published here, as ready to go, will either run on POSIX
or there will be much wailing, teeth gnashing and berating from the
regulars. We deal with portable code here. Which, in turn, is why
POSIX is off-topic.

I was just wondering why your function is not considered off-topic but every
time posix is mentioned its off topic. One could mention that to get a
portable version of strtok_r you can strip it from posix.
Because Chuck provides the source for it in standard C when he mentions
it, and code written in standard C is topical here. If you provide the
code for a POSIX function in standard C then we can talk about that, but
a number of things POSIX provides *cannot* be implemented in standard C.
--
Flash Gordon
Mar 12 '07 #13
Richard Bos wrote:
Mr John FO Evans <mi***@orpheusmail.co.ukwrote:
>I cam across an interesting limitation to the use of strtok.

I have two strings on which I want strtok to operate.
However since strtok has only one memory of the residual string I must
complete one set of operations before starting on the second. This
is inconvenient in the context of my program!

So far the only solution I can see is to write a replacement for strtok
to use on one of the strings. Can anyone offer an alternative?

No. It's one of the many ways in which strtok() is unsuitable for most
of the jobs it was intended for. I've only come across a situation in
which strtok() was the right tool for the job, and I've since forgotten
what it was.

Richard
Hear. strtok() bit me several years ago. I investigated and determined
why. As a result, I haven't used strtok() again.

--
Joe Wright
"Everything should be made as simple as possible, but not simpler."
--- Albert Einstein ---
Mar 13 '07 #14
On Tue, 13 Mar 2007 15:58:32 -0400, Joe Wright
<jo********@comcast.netwrote:
>No. It's one of the many ways in which strtok() is unsuitable for most
of the jobs it was intended for.
Nope, it's fine for the jobs it was intended for. It has problems when
used for jobs it wasn't intended for.
I've only come across a situation in
>which strtok() was the right tool for the job, and I've since forgotten
what it was.
I've used it a number of times. If you are reading, tokenizing and
discarding a line, and it's guaranteed not to have missing items
(consecutive delimiters), or you don't care if it does, it works just
fine.
>>
Richard

Hear. strtok() bit me several years ago. I investigated and determined
why. As a result, I haven't used strtok() again.
Perhaps you haven't had occasion to use it, but having determined what
it does, you certainly shouldn't be afraid to.

--
Al Balmer
Sun City, AZ
Mar 13 '07 #15

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: ern | last post by:
//I used strtok() like this: fgets(userCommand, MAX_COMMAND_SIZE, stdin); g_UserCommands = strtok(command, " "); while(g_UserCommands != NULL){ i+=1; g_UserCommands = strtok(NULL, " "); } ...
1
by: arssa2020 | last post by:
hello, i am writing a GPS app in evc++ 4.0, i am using c-style char* instead of CStrings, i want to parse NMEA sentence such as "GPZDA,162913.48,19,08,2005,,*63" problem is when i use strtok it...
20
by: bubunia2000 | last post by:
Hi all, I heard that strtok is not thread safe. So I want to write a sample program which will tokenize string without using strtok. Can I get a sample source code for the same. For exp:...
8
by: hu | last post by:
hi, everybody! I'm testing the fuction of strtok(). The environment is WinXP, VC++6.0. Program is simple, but mistake is confusing. First, the below code can get right outcome:"ello world, hello...
29
by: Pietro Cerutti | last post by:
Hello, here I have a strange problem with a real simple strtok example. The program is as follows: ### BEGIN STRTOK ### #include <string.h> #include <stdio.h>
11
by: Lothar Behrens | last post by:
Hi, I have selected strtok to be used in my string replacement function. But I lost the last token, if there is one. This string would be replaced select "name", "vorname", "userid",...
4
by: ohaqqi | last post by:
Hi everybody. I haven't programmed anything in about 8 years, I've read up a little bit on C and need to write a shell in C. I want to use strtok() to take an input from a user and parse it into the...
11
by: magicman | last post by:
can anyone point me out to its implementation in C before I roll my own. thx
4
by: spiralfire | last post by:
I wrote a translator, that reads a DIMACS graph format and writes to a simpler format... basically DIMACS format is: c comment p type nodes edges //type is alwats edge on my problems,...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.