473,326 Members | 2,192 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

strtok() and EOL

Hi,

I'm using strtok() in the following way:

void obtain_param(char *pmsg, CONF_PARAMS *cnf ) {
char *s1, *s2;
size_t msg_len;

s1 = strtok (pmsg,":");
if (s1) {
s2 = strtok (0, "");
msg_len = strcspn(s2, "\n");
if (memcmp(s1, "message", sizeof("message")) == 0) {
cnf->msg_length = msg_len;
cnf->message = malloc(msg_len);
memcpy(cnf->message, s2, msg_len);
cnf->message[msg_len]='\0';

....

I extract the information from a configuration file. For some reason,
sometimes I get for msg_len 15 for this message:
message:this is a test


(when 'this is a test\n' is only 14 characters long, w/o the '\n')

In other installations (Linux) works just fine, but in my
Eclipse/CDT/cygwin sometimes is not.

Am I using strtok incorrectly? is the 's2' pointer messing things up?

Thanks,

F~
Nov 29 '05 #1
7 5684
On Tue, 29 Nov 2005 15:16:17 GMT, Fernando Barsoba
<fb******@verizon.net> wrote in comp.lang.c:
Hi,

I'm using strtok() in the following way:

void obtain_param(char *pmsg, CONF_PARAMS *cnf ) {
char *s1, *s2;
size_t msg_len;

s1 = strtok (pmsg,":");
if (s1) {
s2 = strtok (0, "");
msg_len = strcspn(s2, "\n");
if (memcmp(s1, "message", sizeof("message")) == 0) {
cnf->msg_length = msg_len;
cnf->message = malloc(msg_len);
memcpy(cnf->message, s2, msg_len);
cnf->message[msg_len]='\0';

...

I extract the information from a configuration file. For some reason,
sometimes I get for msg_len 15 for this message:
>>message:this is a test


(when 'this is a test\n' is only 14 characters long, w/o the '\n')

In other installations (Linux) works just fine, but in my
Eclipse/CDT/cygwin sometimes is not.

Am I using strtok incorrectly? is the 's2' pointer messing things up?

Thanks,

F~


Where did the text string passed in 'pmsg' come from? Was it read
from a file? If it was read from a file, was the file opened in text
or binary mode? If you are reading a text file opened in binary mode
under Windows, you are getting the string as it appears in the file,
which would be:

"message:this is a test\r\n"

--
Jack Klein
Home: http://JK-Technology.Com
FAQs for
comp.lang.c http://www.eskimo.com/~scs/C-faq/top.html
comp.lang.c++ http://www.parashift.com/c++-faq-lite/
alt.comp.lang.learn.c-c++
http://www.contrib.andrew.cmu.edu/~a...FAQ-acllc.html
Nov 29 '05 #2

"Fernando Barsoba" <fb******@verizon.net> wrote in message
news:5h_if.6989$6e5.1530@trnddc09...
Hi,

I'm using strtok() in the following way:

void obtain_param(char *pmsg, CONF_PARAMS *cnf ) {
char *s1, *s2;
size_t msg_len;

s1 = strtok (pmsg,":");
if (s1) {
s2 = strtok (0, "");
msg_len = strcspn(s2, "\n");
if (memcmp(s1, "message", sizeof("message")) == 0) {
cnf->msg_length = msg_len;
cnf->message = malloc(msg_len);
memcpy(cnf->message, s2, msg_len);
cnf->message[msg_len]='\0';

...

I extract the information from a configuration file. For some reason,
sometimes I get for msg_len 15 for this message:
message:this is a test


(when 'this is a test\n' is only 14 characters long, w/o the '\n')

In other installations (Linux) works just fine, but in my
Eclipse/CDT/cygwin sometimes is not.

Am I using strtok incorrectly? is the 's2' pointer messing things up?


Are you perhaps opening your file in binary mode? Note that
some OS's store newline as more than one character in text files.
E.g. Windows uses CR/LF pair and typical Windows C implementations
represent '\n' internally as LF (this would give the behavior
you describe if file were opened in binary mode on such systems
-- the 'extra' CR character would count as part of length returned
by 'strcspn()').

If you're not using it already, try text mode.

-Mike
Nov 29 '05 #3

Fernando Barsoba wrote:
Hi,

I'm using strtok() in the following way:
Probably fine, but you do know the issues with strtok, right?
In clc it is not re-entrant. In (OT) land, it is often not threadsafe
either. It is probably fine as you use it below.

void obtain_param(char *pmsg, CONF_PARAMS *cnf ) {
char *s1, *s2;
size_t msg_len;

s1 = strtok (pmsg,":");
if (s1) {
s2 = strtok (0, "");
I'm not totally sure the empty string is legit here, perhaps others
can say. Works fine if you say ":" too.
msg_len = strcspn(s2, "\n");
if (memcmp(s1, "message", sizeof("message")) == 0) {
Do you intend to compare the '\0' as you are? sizeof("message") is 8.
Fine either way, strtok will have put one in.
cnf->msg_length = msg_len;
cnf->message = malloc(msg_len);
You don't allocate enough space here for the string including the '\0'
memcpy(cnf->message, s2, msg_len);
cnf->message[msg_len]='\0';
One byte overwrite here.

...

I extract the information from a configuration file. For some reason,
sometimes I get for msg_len 15 for this message:
>>message:this is a test


(when 'this is a test\n' is only 14 characters long, w/o the '\n')

In other installations (Linux) works just fine, but in my
Eclipse/CDT/cygwin sometimes is not.

Am I using strtok incorrectly? is the 's2' pointer messing things up?


I think you are using strtok mostly OK. I'd guess you are getting
messed up by DOS end of line vs unix here. Perhaps depends
on how the configuration file is read (binary vs text?).

-David

Nov 29 '05 #4
Fernando Barsoba <fb******@verizon.net> writes:
I'm using strtok() in the following way:


I don't generally recommend using strtok(). It has at least
these problems:

* It merges adjacent delimiters. If you use a comma as
your delimiter, then "a,,b,c" is three tokens, not
four. This is often the wrong thing to do. In fact,
it is only the right thing to do, in my experience,
when the delimiter set is limited to white space.

* The identity of the delimiter is lost, because it is
changed to a null terminator.

* It modifies the string that it tokenizes. This is bad
because it forces you to make a copy of the string if
you want to use it later. It also means that you can't
tokenize a string literal with it; this is not
necessarily something you'd want to do all the time but
it is surprising.

* It can only be used once at a time. If a sequence of
strtok() calls is ongoing and another one is started,
the state of the first one is lost. This isn't a
problem for small programs but it is easy to lose track
of such things in hierarchies of nested functions in
large programs. In other words, strtok() breaks
encapsulation.

--
"You call this a *C* question? What the hell are you smoking?" --Kaz
Nov 29 '05 #5
Jack Klein wrote:
On Tue, 29 Nov 2005 15:16:17 GMT, Fernando Barsoba
<fb******@verizon.net> wrote in comp.lang.c:
Hi,

I'm using strtok() in the following way:

void obtain_param(char *pmsg, CONF_PARAMS *cnf ) {
char *s1, *s2;
size_t msg_len;

s1 = strtok (pmsg,":");
(...) Where did the text string passed in 'pmsg' come from? Was it read
from a file? If it was read from a file, was the file opened in text
or binary mode? If you are reading a text file opened in binary mode
under Windows, you are getting the string as it appears in the file,
which would be:

"message:this is a test\r\n"

Thanks to all for your comments. Indeed, the string I'm getting is
"message:this is a test\r\n". And 'pmsg' comes also from a file. Oddly
enough, I thought I opened the file in text mode. Here's the sequence of
calls:

fp = open_file(argv[1], "r");
cnf = get_param(fp);

-------------
CONF_PARAMS *get_param(FILE *f_param) {
....
p_msg = fgets(param, PARAM_LENGTH, f_param);
....
}
--------
FILE * open_file(char *file_name, char *foptions) {
FILE *fp;
if ((fp = fopen(file_name, foptions)) == NULL) {
printf("Can't open %s\n", file_name);
exit(1);
}
return fp;
}

As the comments pointed out, this seems to be a Windows issue. That's
why in Linux works fine. I'm not opening the file in binary mode though..

F~
Nov 29 '05 #6
Jack Klein wrote:
On Tue, 29 Nov 2005 15:16:17 GMT, Fernando Barsoba
<fb******@verizon.net> wrote in comp.lang.c:


<snip>
>>message:this is a test


(when 'this is a test\n' is only 14 characters long, w/o the '\n')

In other installations (Linux) works just fine, but in my
Eclipse/CDT/cygwin sometimes is not.

Am I using strtok incorrectly? is the 's2' pointer messing things up?

Thanks,

F~


Where did the text string passed in 'pmsg' come from? Was it read
from a file? If it was read from a file, was the file opened in text
or binary mode? If you are reading a text file opened in binary mode
under Windows, you are getting the string as it appears in the file,
which would be:

"message:this is a test\r\n"


Note that since Cygwin is emulating some of Unix it may think of \n as
being the line terminator rather than \r\n even when you open a stream
in text mode, so you could still get a spurious \r. For the intricacies
of Cygwin line termination you will have to ask on a Cygwin mailing
list, but if this is your problem one option is to check if the last
character of the line you read is \r and if so strip it.
--
Flash Gordon
Living in interesting times.
Although my email address says spam, it is real and I read it.
Nov 29 '05 #7
Fernando Barsoba wrote:
Jack Klein wrote:
On Tue, 29 Nov 2005 15:16:17 GMT, Fernando Barsoba
<fb******@verizon.net> wrote in comp.lang.c:
<snip>
"message:this is a test\r\n"
Thanks to all for your comments. Indeed, the string I'm getting is
"message:this is a test\r\n". And 'pmsg' comes also from a file. Oddly
enough, I thought I opened the file in text mode. Here's the sequence of
calls:

fp = open_file(argv[1], "r");


Yes, that is opening in text mode.

<snip>
As the comments pointed out, this seems to be a Windows issue. That's
why in Linux works fine. I'm not opening the file in binary mode though..


See my previous post, it's a Cygwin issue. Specifically, Cygwin is
emulation Unix to a degree (that is its purpose) so by default it used
the Unix text file conventions. Ask on a Cygwin mailing list for the
details.
--
Flash Gordon
Living in interesting times.
Although my email address says spam, it is real and I read it.
Nov 29 '05 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Ram Laxman | last post by:
Hi all, I have written the following code: /* strtok example */ #include <stdio.h> #include <string.h> static const char * const resultFileName = "param.txt";
13
by: ern | last post by:
I'm using strtok( ) to capture lines of input. After I call "splitCommand", I call strtok( ) again to get the next line. Strtok( ) returns NULL (but there is more in the file...). That didn't...
20
by: bubunia2000 | last post by:
Hi all, I heard that strtok is not thread safe. So I want to write a sample program which will tokenize string without using strtok. Can I get a sample source code for the same. For exp:...
8
by: hu | last post by:
hi, everybody! I'm testing the fuction of strtok(). The environment is WinXP, VC++6.0. Program is simple, but mistake is confusing. First, the below code can get right outcome:"ello world, hello...
4
by: Michael | last post by:
Hi, I have a proble I don't understand when using strtok(). It seems that if I make a call to strtok(), then make a call to another function that also makes use of strtok(), the original call is...
3
by: nomad5000 | last post by:
Hi everybody! I'm having trouble using strtok to fill a matrix with int nrs. from a file. the code that is not working is the following: #include <iostream> #include <fstream> #include...
29
by: Pietro Cerutti | last post by:
Hello, here I have a strange problem with a real simple strtok example. The program is as follows: ### BEGIN STRTOK ### #include <string.h> #include <stdio.h>
11
by: Lothar Behrens | last post by:
Hi, I have selected strtok to be used in my string replacement function. But I lost the last token, if there is one. This string would be replaced select "name", "vorname", "userid",...
11
by: magicman | last post by:
can anyone point me out to its implementation in C before I roll my own. thx
12
by: Pilcrow | last post by:
Here is a quick program, together with its output, that illustrates what I consider to be a deficiency of the standard function strtok from <string.h>: I am using C:\>gcc --version gcc (GCC)...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.