Regular Expression Blues

[Iasthaai]

Hello everyone, I'm new to Python and also new to this forum and I have a question regarding regular expressions, so here goes:

I am attempting to read in a text and do some pattern matching before I store words in a dictionary. I basically want to read in the words and remove any numbers, punctuation, or other special characters from the beginning and end of the word. However, I do wish to keep all of the above in the word if the character is embedded between an A-Z or a-z character.

I have tried many different regular expressions and I will list a few below:

import re
regword = re.compile("[A-Z]+ (\S* [A-Z]+)*, re.I)

This one seems to work when I just use the following script:

for line in open('text.txt', 'r'):
    for word in regword.finditer(line):
        print (word.group(0)
 

However, when I go to put these words into a dictionary, the first letter of each word is getting chopped off! Here is the following code that I used, attempting to place these words into a dictionary...

>>> def manip_word(file):
       wordlist = re_word.findall(open(file).read())
       wordfreq = [wordlist.count(p) for p in wordlist]
       dictionary = dict(zip(wordlist, wordfreq))
       aux = [(key, dictionary[key]) for key in dictionary]
       aux.sort()
       for a in aux: print a
 

After using this code, I realized that file seemed to be a key word, and thus I changed file in both the argument for the function definition and the open argument to file1. This did seem to affect my output, but in an even worse way...it only listed words beginning with ' and -. IE 't like in haven't and -expression as in sub-expression.

I also attempted to explicitly state the characters I would accept between a-z characters...but to no avail.

 
import re
regword = re.compile("[A-Z]+([\.,\?,...]*[A-Z]+)*, re.I)
 
 

Does anyone have any suggestions? The fact that my first attempt worked when I simply listed the words out seems to point to the way I'm storing in a dictionary as incorrect.

Thanks for reading and any possible help! :-)

[bvdet]

Hello everyone, I'm new to Python and also new to this forum and I have a question regarding regular expressions, so here goes:

I am attempting to read in a text and do some pattern matching before I store words in a dictionary. I basically want to read in the words and remove any numbers, punctuation, or other special characters from the beginning and end of the word. However, I do wish to keep all of the above in the word if the character is embedded between an A-Z or a-z character.

I have tried many different regular expressions and I will list a few below:

import re
regword = re.compile("[A-Z]+ (\S* [A-Z]+)*, re.I)

This one seems to work when I just use the following script:

for line in open('text.txt', 'r'):
    for word in regword.finditer(line):
        print (word.group(0)
 

However, when I go to put these words into a dictionary, the first letter of each word is getting chopped off! Here is the following code that I used, attempting to place these words into a dictionary...

>>> def manip_word(file):
       wordlist = re_word.findall(open(file).read())
       wordfreq = [wordlist.count(p) for p in wordlist]
       dictionary = dict(zip(wordlist, wordfreq))
       aux = [(key, dictionary[key]) for key in dictionary]
       aux.sort()
       for a in aux: print a
 

After using this code, I realized that file seemed to be a key word, and thus I changed file in both the argument for the function definition and the open argument to file1. This did seem to affect my output, but in an even worse way...it only listed words beginning with ' and -. IE 't like in haven't and -expression as in sub-expression.

I also attempted to explicitly state the characters I would accept between a-z characters...but to no avail.

 
import re
regword = re.compile("[A-Z]+([\.,\?,...]*[A-Z]+)*, re.I)
 
 

Does anyone have any suggestions? The fact that my first attempt worked when I simply listed the words out seems to point to the way I'm storing in a dictionary as incorrect.

Thanks for reading and any possible help! :-)

Do you have to use 're'?

import re
def manip_word(file):
    patt = re.compile(r'[a-zA-Z]')
    wordlist = open(fn).read().strip().split()
    wordlist1 = []
    for w in wordlist:
        if patt.search(w):
            wordlist1.append(w.lower().strip(".,:?!()[]/\\\n\"\'"))
    wordfreq = [wordlist1.count(p) for p in wordlist]
    dictionary = dict(zip(wordlist1, wordfreq))
    aux = [(key, dictionary[key]) for key in dictionary]
    aux.sort()
    for a in aux: print a
    return aux

[bartonc]

Regex drive me nuts. Hours of fiddling with them to get them to work just right.
Here is a site that might help with the regex part. Please post back here for help with the python part of the problem.

[ghostdog74]

assuming input file sample is like this

this is a 123$@#test$#@000 string
with mulitple 3453line400##@*&

In this case, i expect to find 'test' and 'line', if i interpreted your requirements correctly.
I tested with this

>>> import re
>>> pat = re.compile(r"[^a-zA-Z \n]+(.*?)[^a-zA-Z \n]+", re.I|re.M)
>>> data = open("file3").read()
>>> pat.findall(data)
['test', 'line']
 

[Iasthaai]

Hey everyone! Thanks for all the responses, they were all very helpful. It turns out that I solved this one shortly after posting (that always happens to me!) and I did use a regular expression, mostly the same as my first one (it turns out that the + flag on the first [A-Z] was unnecessary for my purposes):

import re
regword = re.compile("[A-Z](\S*[A-Z]+)*", re.I)
 

It turns out that the above solves all of my problems for defining the regular expression. My problem was in the way I was USING the regular expression, and I have still to understand why fully. Anyway, here is the code I used that definitely works!

def word_manip(file1):
    wordlist = []
    for line in open(file1, 'r'):
        for word in regword.finditer(line): 
            lowercase = (word.group(0)).lower()  
            wordlist.append(lowercase)
    wordfreq = [wordlist.count(p) for p in wordlist]
    dictionary = dict(zip(wordlist, wordfreq))
    aux = [(key, dictionary[key]) for key in dictionary]
    aux.sort()
    for a in aux: print a
 

Earlier, I was using regword.findall(line) which returns a list. I didn't use the for loop and it was storing in the dictionary, except it stored all the words with the first letter chopped off of the front. I guess there must be a function to tell the dictionary where to start each word and I didn't specify.

So that's that. Thanks again!

[bartonc]

I'm glad that you have kept us up to date. Thanks for that.
Welcome to TheScripts.com. I hope that you keep posting.