472,952 Members | 2,813 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,952 software developers and data experts.

Good String Tokenizer

I have searched the board and noticed that there isn't really any sort
of good implementation of a string tokenizer that will tokenize based
on a custom set of tokens and return both the tokens and the parts
between the tokens.

For example, if I have the string:

"Hello, World! How are you?"

And my splitting points are comma, and exclamation point then I would
expect to get back.

["Hello", ",", " World", "!", " How are you?"]

Does anyone know of a tokenizer that will allow for this sort of use?

Thanks in advance,
Jim Howard

Jul 24 '07 #1
1 3399
JamesHoward <Ja************@gmail.comwrote:
I have searched the board
what board? I don't see any boards here.
And my splitting points are comma, and exclamation point then I would
expect to get back.

["Hello", ",", " World", "!", " How are you?"]

Does anyone know of a tokenizer that will allow for this sort of use?
>>import re
re.split("([!,])", "Hello, World! How are you?")
['Hello', ',', ' World', '!', ' How are you?']
Jul 24 '07 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

by: blrmaani | last post by:
Here is what I want: string s1 = "This is a list of string"; list<string> s2 = s1.some_method(); Now, I should be able to traverse list s2 and get each member ( which is of type 'string' ). ...
by: Java Guy | last post by:
This must be a classical topic -- C++ stgring tokenizer. I just switched from C to C++ ( in Unix ). It turns out that there is no existing C++ string tokenizer. Searching on the Web, I found...
by: Generic Usenet Account | last post by:
Is it that I am blurry eyed, or is it indeed that the C++ string class has no tokenizer method defined? I have defined my own functions , but I would prefer to use the standard functions, if...
by: John Fly | last post by:
I've used standard functions like find_first_of and find_first_not_of for some time. I was about to implement a quick string tokenizer similar to the example below when I ran across an old thread...
by: Andre | last post by:
Hi, Does anyone know whether the ECMA, or an other standard document, specifies a maximum for the value that can be pass to the setTimeOut() function in Javascript? Andre
by: Felix85 | last post by:
I am trying to make a command interpreter for a mud that i am working on the problem i am having right now is that i cannot convert the string into a char array. This is the error I am getting...
by: Bilal | last post by:
Hello, I'm trying to perform some string manipulations in my stylesheet and have gotten stuck on the issue below so hopefully can elicit some useful hints. Namely, the problem is that I need to...
by: nnguyec | last post by:
Hi, I'm trying to write a small code for an assignment which the void String Tokenizer will get a line input, and take out those delimiters from the original string. Then pass each string without the...
by: xetulul | last post by:
my problem is that im unable to match a user input word to a word in a string. the strings are in a file and then placed in nodes. i have to go through each token in each node to find the 'word'....
by: lllomh | last post by:
Define the method first this.state = { buttonBackgroundColor: 'green', isBlinking: false, // A new status is added to identify whether the button is blinking or not } autoStart=()=>{
by: DJRhino | last post by:
Was curious if anyone else was having this same issue or not.... I was just Up/Down graded to windows 11 and now my access combo boxes are not acting right. With win 10 I could start typing...
by: Aliciasmith | last post by:
In an age dominated by smartphones, having a mobile app for your business is no longer an option; it's a necessity. Whether you're a startup or an established enterprise, finding the right mobile app...
by: tracyyun | last post by:
Hello everyone, I have a question and would like some advice on network connectivity. I have one computer connected to my router via WiFi, but I have two other computers that I want to be able to...
by: giovanniandrean | last post by:
The energy model is structured as follows and uses excel sheets to give input data: 1-Utility.py contains all the functions needed to calculate the variables and other minor things (mentions...
by: NeoPa | last post by:
Hello everyone. I find myself stuck trying to find the VBA way to get Access to create a PDF of the currently-selected (and open) object (Form or Report). I know it can be done by selecting :...
by: Teri B | last post by:
Hi, I have created a sub-form Roles. In my course form the user selects the roles assigned to the course. 0ne-to-many. One course many roles. Then I created a report based on the Course form and...
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 1 Nov 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM) Please note that the UK and Europe revert to winter time on...
by: nia12 | last post by:
Hi there, I am very new to Access so apologies if any of this is obvious/not clear. I am creating a data collection tool for health care employees to complete. It consists of a number of...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.