471,305 Members | 1,386 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,305 software developers and data experts.

python/regex question... hope someone can help

I have a list of strings. These strings are previously selected
bigrams with underscores between them ('and_the', 'nothing_given', and
so on). I need to write a regex that will read another text string
that this list was derived from and replace selections in this text
string with those from my list. So in my text string, '... and the...
' becomes ' ... and_the...'. I can't figure out how to manipulate

re.sub(r'([a-z]*) ([a-z]*)', r'(????)', textstring)

Any suggestions?

Thank you if you can help!
Dec 9 '07 #1
4 1366
On Dec 9, 6:13 pm, charonzen <your.mas...@gmail.comwrote:
I have a list of strings. These strings are previously selected
bigrams with underscores between them ('and_the', 'nothing_given', and
so on). I need to write a regex that will read another text string
that this list was derived from and replace selections in this text
string with those from my list. So in my text string, '... and the...
' becomes ' ... and_the...'. I can't figure out how to manipulate

re.sub(r'([a-z]*) ([a-z]*)', r'(????)', textstring)

Any suggestions?
The usual suggestion is: Don't bother with regexes when simple string
methods will do the job.
>>def ch_replace(alist, text):
.... for bigram in alist:
.... original = bigram.replace('_', ' ')
.... text = text.replace(original, bigram)
.... return text
....
>>print ch_replace(
.... ['quick_brown', 'lazy_dogs', 'brown_fox'],
.... 'The quick brown fox jumped over the lazy dogs.'
.... )
The quick_brown_fox jumped over the lazy_dogs.
>>print ch_replace(['red_herring'], 'He prepared herring fillets.')
He prepared_herring fillets.
>>>
Another suggestion is to ensure that the job specification is not
overly simplified. How did you parse the text into "words" in the
prior exercise that produced the list of bigrams? Won't you need to
use the same parsing method in the current exercise of tagging the
bigrams with an underscore?

Cheers,
John
Dec 9 '07 #2
On Dec 9, 6:13 pm, charonzen <your.mas...@gmail.comwrote:

The following *may* come close to doing what your revised spec
requires:

import re
def ch_replace2(alist, text):
for bigram in alist:
pattern = r'\b' + bigram.replace('_', ' ') + r'\b'
text = re.sub(pattern, bigram, text)
return text

Cheers,
John
Dec 9 '07 #3
Another suggestion is to ensure that the job specification is not
overly simplified. How did you parse the text into "words" in the
prior exercise that produced the list of bigrams? Won't you need to
use the same parsing method in the current exercise of tagging the
bigrams with an underscore?

Cheers,
John
Thank you John, that definitely puts things in perspective! I'm very
new to both Python and text parsing, and I often feel that I can't see
the forest for the trees. If you're asking, I'm working on a project
that utilizes Church's mutual information score. I tokenize my text,
split it into a list, derive some unigram and bigram dictionaries, and
then calculate a pmi dictionary based on x,y from the bigrams and
unigrams. The bigrams that pass my threshold then get put into my
list of x_y strings, and you know the rest. By modifying the original
text file, I can view 'x_y', z pairs as x,y and iterate it until I
have some collocations that are worth playing with. So I think that
covers the question the same parsing method. I'm sure there are more
pythonic ways to do it, but I'm on deadline :)

Thanks again!

Brandon
Dec 9 '07 #4
En Sun, 09 Dec 2007 16:45:53 -0300, charonzen <yo*********@gmail.com>
escribió:
>[John Machin] Another suggestion is to ensure that the job
specification is not
overly simplified. How did you parse the text into "words" in the
prior exercise that produced the list of bigrams? Won't you need to
use the same parsing method in the current exercise of tagging the
bigrams with an underscore?

Thank you John, that definitely puts things in perspective! I'm very
new to both Python and text parsing, and I often feel that I can't see
the forest for the trees. If you're asking, I'm working on a project
that utilizes Church's mutual information score. I tokenize my text,
split it into a list, derive some unigram and bigram dictionaries, and
then calculate a pmi dictionary based on x,y from the bigrams and
unigrams. The bigrams that pass my threshold then get put into my
list of x_y strings, and you know the rest. By modifying the original
text file, I can view 'x_y', z pairs as x,y and iterate it until I
have some collocations that are worth playing with. So I think that
covers the question the same parsing method. I'm sure there are more
pythonic ways to do it, but I'm on deadline :)
Looks like you should work with the list of tokens, collapsing consecutive
elements, not with the original text. Should be easier, and faster because
you don't regenerate the text and tokenize it again and again.

--
Gabriel Genellina

Dec 10 '07 #5

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

14 posts views Thread by 2mc | last post: by
31 posts views Thread by surfunbear | last post: by
17 posts views Thread by clintonG | last post: by
8 posts views Thread by vbmark | last post: by
5 posts views Thread by sb | last post: by
6 posts views Thread by Johny | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.