473,385 Members | 1,757 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,385 software developers and data experts.

Smart text parsing

Hi,

I got a text with about 1 million words where I want to count words and put
them sorted to a list
like " list = [(most-common-word,1001),(2nd-word,986), ...] "

I think there are at about 10% (about 100.000) different words in the text.

I am wondering if you can give me something faster than my approach:
My first straightforward approach was:
----
s = "Hello this is my 1 million word text".split()

s2 = s.split()
dict = {}
for i in s2: # the loop needs 10s
if dict.has_key(i):
dict[i] += 1
else:
dict[i] = 1
list = dict.items()
# this is slow:
list.sort(lambda x,y: 2*(x[1] < y[1])-1)
----
That works, but i wonder if there is a faster, more elegant way to do this
....

Thanks for you interest,
Mathias Mamsch
Jul 18 '05 #1
2 2187
s = "Hello this is my 1 million word text".split()

s2 = s.split()
dict = {}
for i in s2: # the loop needs 10s
if dict.has_key(i):
dict[i] += 1
else:
dict[i] = 1 list = dict.items()
# this is slow:
list.sort(lambda x,y: 2*(x[1] < y[1])-1)

list = zip(dict.values(), dict.keys())
list.sort()

Should be faster due to not using the sort function argument.

- Josiah
Jul 18 '05 #2
Mathias Mamsch wrote:
I got a text with about 1 million words where I want to count words and put
them sorted to a list
like " list = [(most-common-word,1001),(2nd-word,986), ...] "

I think there are at about 10% (about 100.000) different words in the text.

I am wondering if you can give me something faster than my approach:
My first straightforward approach was:
----
s = "Hello this is my 1 million word text".split()

s2 = s.split()
dict = {}
for i in s2: # the loop needs 10s
if dict.has_key(i):
dict[i] += 1
else:
dict[i] = 1
list = dict.items()
# this is slow:
list.sort(lambda x,y: 2*(x[1] < y[1])-1)
----


Passing a comparison function to sort slows things down a lot. Try something
like this instead:

parts = "Hello this is my 1 million word text".split()
for part in parts:
if d.has_key(part):
d[part] += 1
else:
d[part] = 1

lst = d.items()
lst = [(t[1], t[0]) for t in lst] # (frequency, string)
lst.sort() # sort as usual
lst.reverse() # reverse, so highest numbers are first

HTH,

--
Hans (ha**@zephyrfalcon.org)
http://zephyrfalcon.org/

Jul 18 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

10
by: RodBillett | last post by:
Using SmartNavigation. Windows2003 Server Framework 1.0 (3705) IIS6 and I have IIS configured to utilize the 1.0 framework. Any Ideas why I would be getting the following jscript error within...
1
by: Hal | last post by:
My most sincere gratitude to anyone who can help me work around this! I have work that needs to be done in javascript on the client whenever a page is unloaded. To this end, I subscribe to...
7
by: Lucas Tam | last post by:
Hi all, Does anyone know of a GOOD example on parsing text with text qualifiers? I am hoping to parse text with variable length delimiters/qualifiers. Also, qualified text could run onto...
13
by: Chris Lieb | last post by:
I am trying to write a regex that will parse BBcode into HTML using JavaScript. Everything was going smoothly using the string class replace() operator with regex's until I got to the list tag....
2
by: JaythePCguy | last post by:
Hi, I am trying to write a text parser to group all nonprintable and control characters, spaces and space delimited words in different groups using Regex class. Using a parsing of...
3
by: toton | last post by:
Hi, I have some ascii files, which are having some formatted text. I want to read some section only from the total file. For that what I am doing is indexing the sections (denoted by .START in...
1
by: dli | last post by:
Hello, I am new to this community, and hope that you will be able to help me out with this. I am trying to load a .csporj file with: my $myXML = XML::Smart->new( $BuildFile,...
5
by: Noozer | last post by:
I'm looking for a "smart folder" program to run on my Windows XP machine. I'm not having any luck finding it and think the logic behind the program is pretty simple, but I'm not sure how I'd...
69
by: Bill Reid | last post by:
This is how I handle a check that the last character of a text file is a newline: /* checks if newline is last character of text file */ unsigned check_text_file_newline_termination(FILE...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: aa123db | last post by:
Variable and constants Use var or let for variables and const fror constants. Var foo ='bar'; Let foo ='bar';const baz ='bar'; Functions function $name$ ($parameters$) { } ...
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.