473,856 Members | 1,751 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

C++ Tokenize vectors strings, no values between delimiters

m6s
55 New Member
1. After hours of researching, I used these snippets :
Expand|Select|Wrap|Line Numbers
  1. void Object::TokenizeLines(const string& str, vector<string>& tokens, const string& delimiters)
  2. // Skip delimiters at beginning.
  3.     string::size_type lastPos = str.find_first_not_of(delimiters, 0);
  4.     // Find first "non-delimiter".
  5.     string::size_type pos     = str.find_first_of(delimiters, lastPos);
  6.  
  7.     while (string::npos != pos || string::npos != lastPos) {
  8.         // Found a token, add it to the vector.
  9.         tokens.push_back( CheckWord( str, lastPos, pos - lastPos));
  10.         // Skip delimiters.  Note the "not_of"
  11.         lastPos = str.find_first_not_of(delimiters, pos);
  12.         // Find next "non-delimiter"
  13.         pos = str.find_first_of(delimiters, lastPos);
  14.     }
  15. }
  16.  
for tokenizing large lines.
with a function call like this : string.Tokenize (str, tokens, "\n");
and because in each line I had things like : abc,de,,,f,g,,h
the previous lines were faulty. So I found this one :
Expand|Select|Wrap|Line Numbers
  1. void TokenizeWithComma(const string& str, vector<string>& tokens){
  2.     const char* first = str.c_str();
  3.     const char* last = str.c_str() + strlen(str.c_str());
  4.     while (first != last) {
  5.         const char* next = find(first, last, ',');
  6.         tokens.push_back(string(first, next - first));
  7.         first = min(next + 1, last);
  8.     }
  9. }
  10.  
and I use it after the TokenizeLines to tokenise the words by passing a string and a vector. Both worked.

After spenting too many hours today for that, I have my homework done, but I am not sure what I did here...
Can someone (not novice likeme) give me more detailed view?

Also :
2. Why, the first function don't want to work with strings like a,bc,,d ?
3. I tryed also this for each word ( which are in vector ) :
Expand|Select|Wrap|Line Numbers
  1. for (w_iter = token_lines.begin(); w_iter != token_lines.end(); w_iter++) {
  2. string ff = (*w_iter);
  3. string::size_type loc = ff.find( "abc", 0 );
  4. if( loc != string::npos ) { cout << "Found Omega at " << loc << endl;}
  5. else {cout << "Didn't find Omega" << endl;}
  6.  
But the code while is a working with a normal string ff("abc,ccc,cc" ), seems not to work with the declaration I had. Is the iterator's fault?
This drove me nuts, and made me in order to find patterns, devide a string with substr and have also cases for not making illegal measurements in the substr function ( which is of string again ).

I know it might be borring topic for most, I appreciate your help...
Thank you
Feb 9 '08 #1
6 4259
weaknessforcats
9,208 Recognized Expert Moderator Expert
But the code while is a working with a normal string ff("abc,ccc,cc" ),
string ff("abc,ccc,cc" ) creates a strring object.

Your function has a string& argument so you can use that string object as a argument.

This ("abc,ccc,cc ") is a C-string. A C-string cannot be used a string& because it's not a string object.

Finally, I like your last solution. You should never be use the string::c_str() method unless your function absolutely requires a C-string. Considerinf that the C string library is deprecated in C++, there should little call for this.
Feb 9 '08 #2
m6s
55 New Member
string ff("abc,ccc,cc" ) creates a strring object.

Your function has a string& argument so you can use that string object as a argument.

This ("abc,ccc,cc ") is a C-string. A C-string cannot be used a string& because it's not a string object.

Finally, I like your last solution. You should never be use the string::c_str() method unless your function absolutely requires a C-string. Considerinf that the C string library is deprecated in C++, there should little call for this.
Thank you for your answer, and the good word :-)
I had tried even string temp = *iter, but that didn't work also. I assume by this way it should load a string object and tokenize it, right?
because temp was then passed to the function.But didn't work either.
Finally, how could I make the TokenizeCommas without a c_str().
Build the whole string as char arrays? Or is any other choice in order to stick as much as it can close to C++?
Feb 11 '08 #3
weaknessforcats
9,208 Recognized Expert Moderator Expert
Why can't you use find() to locate the next comma??
Feb 11 '08 #4
m6s
55 New Member
:-) I Don't understand me either!!!
I have a 64bit, can that be the problem?
Just go through yourself if you like it and find it interesting...
I know that didn't result.

In other words, can you make an iterator from words ( if from a file even better) which will be like a,b,c,,,d,f,g ....so on... (did you notice the three commas-->2 spaces?

And then use with just C++ code not C, I mean the first Tokenize function.
What's your result? Oh, assign each iterator to a string and then tokenize it.
For me it was disaster...
Feb 11 '08 #5
weaknessforcats
9,208 Recognized Expert Moderator Expert
In other words, can you make an iterator from words ( if from a file even better) which will be like a,b,c,,,d,f,g ....so on... (did you notice the three commas-->2 spaces?
It's not the string that's your problem. Its your parsing logic.

There is an article in the C/C++ HowTos on the State Design Pattern and inside that article is how tro constrcuct a tokenizer to break a string into individual words. Complete with code. You might read that article.
Feb 12 '08 #6
m6s
55 New Member
Ok, thank you for your close support on this, I am going to check this article too...
Feb 13 '08 #7

Sign in to post your reply or Sign up for a free account.

Similar topics

9
18436
by: Lans | last post by:
I have a string that I need to tokenize but I need to use a string token see example i am trying the following but strtok only uses characters as delimiters and I need to seperate bu a certain word char *mystring "Jane and Peter and Tom and Cindy" char *delim = " and "; char *token; token = strtok(mystring, delim);
6
3988
by: Ram Laxman | last post by:
Hi all, How can I tokenize the integers using strtok. For example: If I have some thing like: "ram":"laxman":"deepak" then I can safely use strtok.But if I have something like below: 10:20:50:79 How can I use strtok to tokenize based on delimiter(In this case --> : )? 10
2
31386
by: James | last post by:
Hi, I am looking for a stringtokenizer class/method in C#, but can't find one. The similar classes in Java and C++ are StringTokenizer and CStringT::tokenize respectively. I need to keep a current position within the string and change the delimiters dynamically when going throught the string. Does anyone know a stringtokenizer in c#? Thanks a lot,
5
5882
by: kurt sune | last post by:
The code: Dim aLine As String = "cat" & vbNewLine & "dog" & vbNewLine & "fox" & vbNewLine Dim csvColumns1 As String() = aLine.Split(vbNewLine, vbCr, vbLf) Dim csvColumns2 As String() = Microsoft.VisualBasic.Strings.Split(aLine, vbNewLine, -1, CompareMethod.Binary)
20
17261
by: bubunia2000 | last post by:
Hi all, I heard that strtok is not thread safe. So I want to write a sample program which will tokenize string without using strtok. Can I get a sample source code for the same. For exp: 0.0.0.0--->I want to tokenize the string using delimiter as as dot. Regards
5
3130
by: Dennis | last post by:
I know this is probably a very overworked issue but thought I'd share the code below to convert words in a text string to capitalize the first letter of the word using an array of word delimiters. Hope it not too simplistic for posting on thie newsgroup: Private Overloads Function CapWords(ByVal textstring As String, ByRef Delimiters() As Char) As String If textstring Is Nothing OrElse textstring.Length <= 0 Then Return Nothing If...
1
2281
by: Tim | last post by:
I ran into a problem with a script i was playing with to check code indents and need some direction. It seems to depend on if tabsize is set to 4 in editor and spaces and tabs indents are mixed on consecutive lines. Works fine when editors tabsize was 8 regardless if indents are mixed. Below are how the 3 test files are laid out, the sample code and output I get. Any help on how to detect this correctly would be appreciated.
4
2084
by: Christian Christmann | last post by:
Hi, what is the best approach to tokenize a "const char*"? The strings look like "sometext 12345". I need to read the number. The first sequence of characters (here "sometext") is not important. Also the number of white spaces is not known. And hints?
4
4399
by: Caudata | last post by:
I am by no means an experienced c++ programmer, but I am trying to use a vector of vectors because it is convenient to store some strings while parsing a text file. I am having trouble with the nested for loop recovery of the stored data and properly dereferencing the data. Here is a bit of test code: #include <iostream> #include <string> #include <vector> using namespace std; struct s { string strA;
0
9908
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
11052
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
10777
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10379
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
7930
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
7090
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5759
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
2
4172
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
3198
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.