467,169 Members | 1,034 Online
Bytes | Developer Community
Ask Question

Home New Posts Topics Members FAQ

Post your question to a community of 467,169 developers. It's quick & easy.

tokenize a string

hi:
in C, we can use strtok() to tokenize a char*
but i can't find any similar member function of string that can tokenize a
string
so how so i tokenize a string in C++?
do it the C way?

thanks
--
{ Kelvin@!!! }
remove the last .hk to reply
thanks
Jul 23 '05 #1
  • viewed: 2765
Share:
4 Replies
Kelvin@!!! wrote:
hi:
in C, we can use strtok() to tokenize a char*
but i can't find any similar member function of string that can tokenize a
string
so how so i tokenize a string in C++?
do it the C way?

thanks


Look up std::istringstream in your favorite reference book.
Jul 23 '05 #2
On Thu, 24 Feb 2005 06:24:31 GMT, Kelvin@!!!
<ch******************@yahoo.com.hk.hk> wrote:
hi:
in C, we can use strtok() to tokenize a char*
but i can't find any similar member function of string that can tokenize
a
string
so how so i tokenize a string in C++?


you may want to try boost::tokenizer an relatives.

http://www.boost.org/libs/tokenizer/index.html
Jul 23 '05 #3
On Thu, 24 Feb 2005 06:24:31 GMT, "Kelvin@!!!"
<ch******************@yahoo.com.hk.hk> wrote:
hi:
in C, we can use strtok() to tokenize a char*
but i can't find any similar member function of string that can tokenize a
string
so how so i tokenize a string in C++?
do it the C way?

thanks


There is a sample chapter from Accelerated C++ on the web at
http://www.awprofessional.com/articl...le.asp?p=25333

The chapter has a function called split() which does what you seem to
want, it takes a string and returns a vector of all the individual
words:

// true if the argument is whitespace, false otherwise
bool space(char c) { return isspace(c); }

// false if the argument is whitespace, true otherwise
bool not_space(char c) { return !isspace(c); }

vector<string> split(const string& str) {
typedef string::const_iterator iter;
vector<string> ret;
iter i = str.begin();
while (i != str.end()) {
// ignore leading blanks
i = find_if(i, str.end(), not_space);
// find end of next word
iter j = find_if(i, str.end(), space);
// copy the characters in [i, j)
if (i != str.end()) ret.push_back(string(i, j));
i = j;
}
return ret;
}

There is a detailed explanation of the functino in the text.

rossum

--

The ultimate truth is that there is no Ultimate Truth
Jul 23 '05 #4
rossum wrote:
// true if the argument is whitespace, false otherwise
bool space(char c) { return isspace(c); }

// false if the argument is whitespace, true otherwise
bool not_space(char c) { return !isspace(c); }

vector<string> split(const string& str) {
typedef string::const_iterator iter;
vector<string> ret;
iter i = str.begin();
while (i != str.end()) {
// ignore leading blanks
i = find_if(i, str.end(), not_space);
// find end of next word
iter j = find_if(i, str.end(), space);
// copy the characters in [i, j)
if (i != str.end()) ret.push_back(string(i, j));
i = j;
}
return ret;
}


This would be better if it was templatized by an insertion iterator
rather than returning a vector by value. Something along the lines of
(untested)

template <typename InsertIter>
int
tokenize(const std::string& buf,
const std::string& delims,
InsertIter it)
{
std::string::size_type sp; // start position
std::string::size_type ep; // end position
int numTokens = 0;

do {
sp = buf.find_first_not_of(delims, sp);
ep = buf.find_first_of(delims, sp);
if (sp != ep) {
if (ep == buf.npos) {
ep = buf.length();
}
*it++ = buf.substr(sp, ep - sp);
++numTokens;
sp = buf.find_first_not_of(delims, ep + 1);
}
} while (sp != buf.npos);

if (sp != buf.npos) {
*it++ = buf.substr(sp, buf.length() - sp);
++numTokens;
}

return numTokens;
}

called as

std::deque<std::string> tokens;
int numTokens = tokenize(buf, delims, std::back_inserter(tokens));

/david

Jul 23 '05 #5

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

16 posts views Thread by qwweeeit | last post: by
9 posts views Thread by Lans | last post: by
2 posts views Thread by James | last post: by
5 posts views Thread by Lam | last post: by
20 posts views Thread by bubunia2000@yahoo.co.in | last post: by
1 post views Thread by Tim | last post: by
2 posts views Thread by askalottaqs | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.