Tokenizing a string using STRTOK() in g++

[wreckingcru]

I'm trying to tokenize a C++ string with the following code:

#include <iostream>
#include <stdio>
#include <string>

using namespace std;
int main(int argc, char* argv[])
{
char *temp_string;
char *new_string;
char *delim = " ";

cout <<"Please enter word: ";
cin >> new_string;

temp_string = strtok(new_string, delim);
while (temp_string != NULL)
{
cout<<temp_string<<endl;
temp_string = strtok(NULL, delim);
}
}
Here's my output:
Please enter word: This is a test
This

I never get more than the first word out!!! I'm using "g++ file.cpp"
with no flags.

I'm certain the flags are somehow wrong - I can't seem to figure out
anything wrong with my code.

Please advise?!

[Nitin Motgi]

wreckingcru wrote:

I'm trying to tokenize a C++ string with the following code:

If you want you can check
http://www.boost.org/libs/tokenizer/tokenizer.htm . It has a
implementation with iterators.

-- Nitin Motgi

[Nitin Motgi]

wreckingcru wrote:

I'm trying to tokenize a C++ string with the following code:

#include <iostream>
#include <stdio>
#include <string>

using namespace std;
int main(int argc, char* argv[])
{
char *temp_string;
char *new_string;
char *delim = " ";

cout <<"Please enter word: ";
cin >> new_string;

The call above only reads in in "This ". The new_string only contains
only "This" that's why you are seeing only that output. You can use
cin.getline(new_string, length);

-- Nitin Motgi

[red floyd]

wreckingcru wrote:

I'm trying to tokenize a C++ string with the following code:

#include <iostream>
#include <stdio>
#include <string>

using namespace std;
int main(int argc, char* argv[])
{
char *temp_string;
char *new_string;
char *delim = " ";

cout <<"Please enter word: ";
at this point new_string points to garbage. The next statement will
invoke UB. cin >> new_string; Congratulations, you have invoked UB.

temp_string = strtok(new_string, delim);
while (temp_string != NULL)
{
cout<<temp_string<<endl;
temp_string = strtok(NULL, delim);
}
}

Try this instead

#include <iostream>
#include <string>
#include <sstream>

int main()
{
std::cout << "Please enter some text: ";

std::string the_line;
if (std::getline(std::cin, the_line))
{
std::istringstream is(the_line);
std::string temp_str;
while (is >> temp_str)
std::cout << temp_str << std::endl;
}
}

[Sunil Varma]

wreckingcru wrote:

I'm trying to tokenize a C++ string with the following code:

#include <iostream>
#include <stdio>
#include <string>

using namespace std;
int main(int argc, char* argv[])
{
char *temp_string;
char *new_string;
char *delim = " ";

cout <<"Please enter word: ";
cin >> new_string;

-----> 1)Don't try to read data into an uninitialized pointer.
-----> 2)cin stops reading data when it encounters a space.
-----> 3)try cout<<new_string;
This will output "This" and not "This is a test"

temp_string = strtok(new_string, delim);
while (temp_string != NULL)
{
cout<<temp_string<<endl;
temp_string = strtok(NULL, delim);
}
}
Here's my output:
Please enter word: This is a test
This

I never get more than the first word out!!! I'm using "g++ file.cpp"
with no flags.

I'm certain the flags are somehow wrong - I can't seem to figure out
anything wrong with my code.

Please advise?!

[WittyGuy]

>char *temp_string;

char *new_string;

The above two variables need to be allocated memory dynamically using
new operator and also don't forget to free them at the end.

--Wg-

[wreckingcru]

Ok thanks for the help! I realize that cin stops at the first space.

Anyway, I don't even want it to take input from user prompt - I was
just using that for testing purposes.

Here's the problem though - the input will be coming from a string
object ..and strtok() only accepts char* - I tried typecasting it
explicitly but it doesn't like that for sure.

What's the fix for tokenizing this string object?

[Gavin Deane]

wreckingcru wrote:

Ok thanks for the help! I realize that cin stops at the first space.

Anyway, I don't even want it to take input from user prompt - I was
just using that for testing purposes.

Here's the problem though - the input will be coming from a string
object ..and strtok() only accepts char* - I tried typecasting it
explicitly but it doesn't like that for sure.
Never "try" casting to make something work. Only ever cast when you
know precisely why the code doesn't compile without the cast, precisely
what the cast will do *and* precisely why it is OK to overrule the
compiler in that way in that situation.
What's the fix for tokenizing this string object?

red floyd has already posted a solution using stringstreams.

If you really want to use strtok, you will need to copy the data from
the string into a char array. Note that just passing the result of the
string member function c_str() to strtok is not sufficient because
strtok needs to be able to modify its argument.

Rather than going through this process with every string you need to
tokenise, write a function to do the work that takes a string and
returns whatever you want to return.

Gavin Deane

[red floyd]

Gavin Deane wrote:

Rather than going through this process with every string you need to
tokenise, write a function to do the work that takes a string and
returns whatever you want to return.

Something like this:

#include <string>
#include <vector>
#include <sstring>

std::vector<std::string> tokenize(const std::string& the_str)
{
std::istringstream is(the_str);
std::vector<std::string> v;
std::string tmp;
while (is >> tmp)
v.push_back(tmp);
return v;
}

[Gavin Deane]

red floyd wrote:

Gavin Deane wrote:

Rather than going through this process with every string you need to
tokenise, write a function to do the work that takes a string and
returns whatever you want to return.

Something like this:

Something like that, yes ...
#include <string>
#include <vector>
#include <sstring>

.... except you meant <sstream> :-)

<snip>

Gavin Deane

[Marcelo Pinto]

wreckingcru wrote:

I'm trying to tokenize a C++ string with the following code:

#include <iostream>
#include <stdio>

[snip]

it should be
#include <cstdio>
or
#include <stdio.h>

HTH,

Marcelo Pinto

[red floyd]

Gavin Deane wrote:

[redacted]
... except you meant <sstream> :-)

Yes, I did. That's what happens when you type on the fly :(