473,397 Members | 2,028 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,397 software developers and data experts.

Tokenizer

Does anyone know how I may tokenize a string into words - while ignoring
any whitespace between the words ?

Jul 21 '06 #1
2 1560
* Bit byte:
Does anyone know how I may tokenize a string into words - while ignoring
any whitespace between the words ?
std::istringstream stream( someString );
std::string firstWord;

if( stream >firstWord )
{
std::cout << "The first word is " << firstWord << ".\n";
}

--
A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
Jul 21 '06 #2


Alf P. Steinbach wrote:
* Bit byte:
>Does anyone know how I may tokenize a string into words - while
ignoring any whitespace between the words ?


std::istringstream stream( someString );
std::string firstWord;

if( stream >firstWord )
{
std::cout << "The first word is " << firstWord << ".\n";
}
Thanks

Jul 21 '06 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Angus Mackay | last post by:
I remember python having a generic tokenizer in the library. all I want is to set a list of token seperators and then read tokens out of a stream, the token seperators should be returned as...
5
by: Knackeback | last post by:
task: - read/parse CSV file code snippet: string key,line; typedef tokenizer<char_separator<char> > tokenizer; tokenizer tok(string(""), sep); while ( getline(f, line) ){ ++lineNo;...
4
by: Java Guy | last post by:
This must be a classical topic -- C++ stgring tokenizer. I just switched from C to C++ ( in Unix ). It turns out that there is no existing C++ string tokenizer. Searching on the Web, I found...
10
by: Alex | last post by:
I'm looking for a fast way to split a string into individual tokens. The typical format of the string is token1|token2|token3|...|tokenN| and the number of tokens varies (which is why i use a...
10
by: Lorenzo J. Lucchini | last post by:
Do you see any counter-indication to a token extractor in the following form? typedef int token; token ExtractToken(const char** String, char* Symbol); If it finds a valid token at the start...
0
by: Arthur | last post by:
Given a "linemess.py" file with inconsistent line ending: line 1 \r \r\n line \n tokenized as per: import tokenize f=open('linemess.py','r')
1
by: xlar54 | last post by:
Hey guys, Im writing a simple language tokenizer and Im at that point of "10 ways to do it, which is the best"... All I need it to do, is given a string, look inside it for keywords, and replace...
18
by: Robbie Hatley | last post by:
A couple of days ago I dedecided to force myself to really learn exactly what "strtok" does, and how to use it. I figured I'd just look it up in some book and that would be that. I figured...
1
by: JamesHoward | last post by:
I have searched the board and noticed that there isn't really any sort of good implementation of a string tokenizer that will tokenize based on a custom set of tokens and return both the tokens and...
1
by: Karl Kobata | last post by:
Hi Fredrik, This is exactly what I need. Thank you. I would like to do one additional function. I am not using the tokenizer to parse python code. It happens to work very well for my...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.