473,320 Members | 1,949 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Tokenizer question

Hello,

I will start working on a project where we have to parse a string and
store the words and the number of times each word appears.

I know you can use the Split function to split a string into words
and then I'd have to savethe words and keep a track of how many times
each one appears.

Does .NET already have a way to do this?

Thank you,

Burak
Jul 21 '05 #1
4 1297
One way to do this would be to Split the String. Now it is in an array.
Then you could sort the array (Array.Sort (arrayName) and loop through
counting the number of times consecutive elements are the same.

Something like this:
Dim Words() as String = Split(TextString)
Dim Count(Words.Length -1) as Integer
Dim CountIndex as Integer = 0
Dim PrevWord as String

PrevWord = Words(0)
For i as Integer = 1 to Words.Length - 1
If Words(i) <> PrevWord Then
CountIndex +=1
End If
Count(CountIndex) += 1
PrevWord = Word(i)
Next

Hope that helps,
--
Marc Butenko
mb******@bresnan.net
"Burak" <bu*********@xpandcorp.com> wrote in message
news:b1**************************@posting.google.c om...
Hello,

I will start working on a project where we have to parse a string and
store the words and the number of times each word appears.

I know you can use the Split function to split a string into words
and then I'd have to savethe words and keep a track of how many times
each one appears.

Does .NET already have a way to do this?

Thank you,

Burak

Jul 21 '05 #2
One way to do this would be to Split the String. Now it is in an array.
Then you could sort the array (Array.Sort (arrayName) and loop through
counting the number of times consecutive elements are the same.

Something like this:
Dim Words() as String = Split(TextString)
Dim Count(Words.Length -1) as Integer
Dim CountIndex as Integer = 0
Dim PrevWord as String

PrevWord = Words(0)
For i as Integer = 1 to Words.Length - 1
If Words(i) <> PrevWord Then
CountIndex +=1
End If
Count(CountIndex) += 1
PrevWord = Word(i)
Next

Hope that helps,
--
Marc Butenko
mb******@bresnan.net
"Burak" <bu*********@xpandcorp.com> wrote in message
news:b1**************************@posting.google.c om...
Hello,

I will start working on a project where we have to parse a string and
store the words and the number of times each word appears.

I know you can use the Split function to split a string into words
and then I'd have to savethe words and keep a track of how many times
each one appears.

Does .NET already have a way to do this?

Thank you,

Burak

Jul 21 '05 #3
Hi Burak,

this I have answered in the vb language newsgroup.

Please send next time one message to more newsgroups.
"Crosspost" and not a lot to every newsgroup "Multipost"

Cor
Jul 21 '05 #4
Hi Burak,

this I have answered in the vb language newsgroup.

Please send next time one message to more newsgroups.
"Crosspost" and not a lot to every newsgroup "Multipost"

Cor
Jul 21 '05 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Angus Mackay | last post by:
I remember python having a generic tokenizer in the library. all I want is to set a list of token seperators and then read tokens out of a stream, the token seperators should be returned as...
5
by: Knackeback | last post by:
task: - read/parse CSV file code snippet: string key,line; typedef tokenizer<char_separator<char> > tokenizer; tokenizer tok(string(""), sep); while ( getline(f, line) ){ ++lineNo;...
4
by: Java Guy | last post by:
This must be a classical topic -- C++ stgring tokenizer. I just switched from C to C++ ( in Unix ). It turns out that there is no existing C++ string tokenizer. Searching on the Web, I found...
10
by: Alex | last post by:
I'm looking for a fast way to split a string into individual tokens. The typical format of the string is token1|token2|token3|...|tokenN| and the number of tokens varies (which is why i use a...
10
by: Lorenzo J. Lucchini | last post by:
Do you see any counter-indication to a token extractor in the following form? typedef int token; token ExtractToken(const char** String, char* Symbol); If it finds a valid token at the start...
0
by: Arthur | last post by:
Given a "linemess.py" file with inconsistent line ending: line 1 \r \r\n line \n tokenized as per: import tokenize f=open('linemess.py','r')
1
by: xlar54 | last post by:
Hey guys, Im writing a simple language tokenizer and Im at that point of "10 ways to do it, which is the best"... All I need it to do, is given a string, look inside it for keywords, and replace...
18
by: Robbie Hatley | last post by:
A couple of days ago I dedecided to force myself to really learn exactly what "strtok" does, and how to use it. I figured I'd just look it up in some book and that would be that. I figured...
1
by: Karl Kobata | last post by:
Hi Fredrik, This is exactly what I need. Thank you. I would like to do one additional function. I am not using the tokenizer to parse python code. It happens to work very well for my...
2
by: ravindra singh | last post by:
i am confused about how to use the Stream Tokenizer class in java, our teacher gave us a program in which v were to use the above class. but i m not getting it. so please help me about this. ...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.