By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
438,427 Members | 1,378 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 438,427 IT Pros & Developers. It's quick & easy.

Smart text parsing

P: n/a
Hi,

I got a text with about 1 million words where I want to count words and put
them sorted to a list
like " list = [(most-common-word,1001),(2nd-word,986), ...] "

I think there are at about 10% (about 100.000) different words in the text.

I am wondering if you can give me something faster than my approach:
My first straightforward approach was:
----
s = "Hello this is my 1 million word text".split()

s2 = s.split()
dict = {}
for i in s2: # the loop needs 10s
if dict.has_key(i):
dict[i] += 1
else:
dict[i] = 1
list = dict.items()
# this is slow:
list.sort(lambda x,y: 2*(x[1] < y[1])-1)
----
That works, but i wonder if there is a faster, more elegant way to do this
....

Thanks for you interest,
Mathias Mamsch
Jul 18 '05 #1
Share this Question
Share on Google+
2 Replies


P: n/a
s = "Hello this is my 1 million word text".split()

s2 = s.split()
dict = {}
for i in s2: # the loop needs 10s
if dict.has_key(i):
dict[i] += 1
else:
dict[i] = 1 list = dict.items()
# this is slow:
list.sort(lambda x,y: 2*(x[1] < y[1])-1)

list = zip(dict.values(), dict.keys())
list.sort()

Should be faster due to not using the sort function argument.

- Josiah
Jul 18 '05 #2

P: n/a
Mathias Mamsch wrote:
I got a text with about 1 million words where I want to count words and put
them sorted to a list
like " list = [(most-common-word,1001),(2nd-word,986), ...] "

I think there are at about 10% (about 100.000) different words in the text.

I am wondering if you can give me something faster than my approach:
My first straightforward approach was:
----
s = "Hello this is my 1 million word text".split()

s2 = s.split()
dict = {}
for i in s2: # the loop needs 10s
if dict.has_key(i):
dict[i] += 1
else:
dict[i] = 1
list = dict.items()
# this is slow:
list.sort(lambda x,y: 2*(x[1] < y[1])-1)
----


Passing a comparison function to sort slows things down a lot. Try something
like this instead:

parts = "Hello this is my 1 million word text".split()
for part in parts:
if d.has_key(part):
d[part] += 1
else:
d[part] = 1

lst = d.items()
lst = [(t[1], t[0]) for t in lst] # (frequency, string)
lst.sort() # sort as usual
lst.reverse() # reverse, so highest numbers are first

HTH,

--
Hans (ha**@zephyrfalcon.org)
http://zephyrfalcon.org/

Jul 18 '05 #3

This discussion thread is closed

Replies have been disabled for this discussion.