470,815 Members | 1,327 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 470,815 developers. It's quick & easy.

Help with Tokenizer Please

I use a code snippet called tokenize that works quite well. But I need
to modify it so it returns an empty string if two commas are found
together: ie. string test="1,2,,4,5"

I would like to have

tokens[0] = "1"
tokens[1] = "2"
tokens[3] = ""
tokens[4] = "4"
tokens[5] = "5"

This would allow me to find missing tokens. The following is the code I
use.
Can anyone show me the modification I would need?
-----------------

void Tokenize(const string& str,
vector<string>& tokens,
const string& delimiters = " ,\t\n")
{
string::size_type lastPos = str.find_first_not_of(delimiters, 0);
// Skip delimiters at beginning.
string::size_type pos = str.find_first_of(delimiters, lastPos);
// Find first "non-delimiter".
while (string::npos != pos || string::npos != lastPos)
{
tokens.push_back(str.substr(lastPos, pos - lastPos));
// Found a token, add it to the vector.
lastPos = str.find_first_not_of(delimiters, pos);
// Skip delimiters. Note the "not_of"
pos = str.find_first_of(delimiters, lastPos);
// Find next "non-delimiter"
}
}

--------------

Thanks in advance,
Grant

Feb 26 '06 #1
1 1472
electrixnow wrote:
I use a code snippet called tokenize that works quite well. But I need
to modify it so it returns an empty string if two commas are found
together: ie. string test="1,2,,4,5"

I would like to have

tokens[0] = "1"
tokens[1] = "2"
tokens[3] = ""
tokens[4] = "4"
tokens[5] = "5"

This would allow me to find missing tokens. The following is the code
I use.
Can anyone show me the modification I would need?
-----------------

void Tokenize(const string& str,
vector<string>& tokens,
const string& delimiters = " ,\t\n")
{
string::size_type lastPos = str.find_first_not_of(delimiters, 0);
// Skip delimiters at beginning.
string::size_type pos = str.find_first_of(delimiters, lastPos);
// Find first "non-delimiter".
while (string::npos != pos || string::npos != lastPos)
{
tokens.push_back(str.substr(lastPos, pos - lastPos));
// Found a token, add it to the vector.
lastPos = str.find_first_not_of(delimiters, pos);
// Skip delimiters. Note the "not_of"
I think you shouldn't skip delimiters here. You just need to make
'lastPos' to be the same as 'pos'. Replace the statement above with

lastPos = pos;
pos = str.find_first_of(delimiters, lastPos);
// Find next "non-delimiter"
}
}

--------------


I didn't check it. It's just a hunch.

V
--
Please remove capital As from my address when replying by mail
Feb 26 '06 #2

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

4 posts views Thread by Java Guy | last post: by
10 posts views Thread by Lorenzo J. Lucchini | last post: by
4 posts views Thread by mca | last post: by
1 post views Thread by satan | last post: by
11 posts views Thread by cybervigilante | last post: by
1 post views Thread by Karl Kobata | last post: by
reply views Thread by mihailmihai484 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.