Module RE, Have a couple questions

[dasacc]

(1) How do I perform a search for "word" and have it return every line
that this instance is found?

(2) How do I perform a search for "word" and "wordtwo" at the same time
to return every line these instances are found so that the order in
which these lines are in are left intact.

If there's another standard module more suited for this let me know,
and no I dont want to use sed :)

Thanks

[Marc Huffnagle]

Dasacc

There is a better (faster/easier) way to do it than using the re module,
the find method of the string class.

da****@gmail.com wrote:

(1) How do I perform a search for "word" and have it return every line
that this instance is found?
[line for line in document if line.find('a') != -1]

(2) How do I perform a search for "word" and "wordtwo" at the same time
to return every line these instances are found so that the order in
which these lines are in are left intact.
[line for line in document if (line.find('word') != -1 \
and line.find('wordtwo'))]

If there's another standard module more suited for this let me know,
and no I dont want to use sed :)

Thanks

[Marc Huffnagle]

Oops, made a mistake.

Marc Huffnagle wrote:

Dasacc

There is a better (faster/easier) way to do it than using the re module,
the find method of the string class.

da****@gmail.com wrote:

(1) How do I perform a search for "word" and have it return every line
that this instance is found?

[line for line in document if line.find('a') != -1]

(2) How do I perform a search for "word" and "wordtwo" at the same time
to return every line these instances are found so that the order in
which these lines are in are left intact.

[line for line in document if (line.find('word') != -1 \
and line.find('wordtwo'))]

This should have been:

[line for line in document if (line.find('word') != -1 \
and line.find('wordtwo') != -1)]

If there's another standard module more suited for this let me know,
and no I dont want to use sed :)

Thanks

[Francis Girard]

Le mardi 1 Mars 2005 16:52, Marc Huffnagle a écrit*:

[line for line in document if (line.find('word') != -1 \
********and line.find('wordtwo') != -1)]

Hi,

Using re might be faster than scanning the same line twice :

=== begin snap
## rewords.py

import re
import sys

def iWordsMatch(lines, word, word2):
reWordOneTwo = re.compile(r".*(%s|%s).*" % (word,word2))
return (line for line in lines if reWordOneTwo.match(line))

for line in iWordsMatch(open("rewords.py"), "re", "return"):
sys.stdout.write(line)
=== end snap

Furthermore, using list comprehension generator (2.4 only I think) and file
iterator, you can scan files as big as you want with very little memory
usage.

Regards,

Francis Girard

[Marc Huffnagle]

Francis Girard wrote:

Le mardi 1 Mars 2005 16:52, Marc Huffnagle a écrit :

[line for line in document if (line.find('word') != -1 \
and line.find('wordtwo') != -1)]

Hi,

Using re might be faster than scanning the same line twice :

My understanding of the second question was that he wanted to find lines
which contained both words but, looking at it again, it could go either
way. If he wants to find lines that contain both of the words, in any
order, then I don't think that it can be done without scanning the line
twice (regex or not).

To the OP: What kind of data are you testing? Could you try both of
these solutions on your sample data and let us know which runs faster?

=== begin snap
## rewords.py

import re
import sys

def iWordsMatch(lines, word, word2):
reWordOneTwo = re.compile(r".*(%s|%s).*" % (word,word2))
return (line for line in lines if reWordOneTwo.match(line))

for line in iWordsMatch(open("rewords.py"), "re", "return"):
sys.stdout.write(line)
=== end snap

Furthermore, using list comprehension generator (2.4 only I think) and file
iterator, you can scan files as big as you want with very little memory
usage.

Regards,

Francis Girard

[Francis Girard]

Le mardi 1 Mars 2005 21:38, Marc Huffnagle a écrit*:

My understanding of the second question was that he wanted to find lines
which contained both words but, looking at it again, it could go either
way. *If he wants to find lines that contain both of the words, in any
order, then I don't think that it can be done without scanning the line
twice (regex or not).

I don't know if it is really faster but here's a version that finds both words
on the same line. My understanding is that re needs to parse the line only
once. This might count on very large inputs.

=== Begin SNAP
## rewords.py

import re
import sys

def iWordsMatch(lines, word, word2):
reWordOneTwo = re.compile(r".*((%s.*%s)|(%s.*%s)).*" %
(word,word2,word2,word))
return (line for line in lines if reWordOneTwo.match(line))

for line in iWordsMatch(open("rewords.py"), "re", "return"):
sys.stdout.write(line)
=== End SNAP

Regards,

Francis Girard

[dasacc]

I'm still very new to python (my 2nd day atm) but this is what I come
up with.

First note, I wasn't clear (I reread what I wrote) about my 'word'
'wordtwo' problem. Both words do Not need to be on the same line. But
rather say there was

Line 4: This is a line
Line 5: Yet another one
Line 6: its a line

I'd like to perform a search for 'line' and 'one' and get the above
returned in the same order. They don't both necessarily need to be
together on the same line.

Now I don't know this stuff very well but I dont think the code

[line for line in document if (line.find('word') != -1 \
and line.find('wordtwo') != -1)]

would do this as it answers the question in how you thought I asked.

Also I can't seem to implement your code to work with more than a
single letter. For example if I do this from the python shell

var1 = "this is a test\nand another test"
[line for line in var1 if line.find('t') != -1]

then I get 6 results of 't' which is correct, But if i try

var1 = "this is a test\nand another test"
[line for line in var1 if line.find('test') != -1]

I get no returned results for test. I tried doing this after I couldn't
retrieve any data larger than a single letter from my source file.

As per Francis Girard's suggestion, it seems to work as long as the
data is in a file.

if I change open("rewords.py") to my object containing the data

for line in iWordsMatch(data, "microsoft", "windows"):

I get no results. But if I instead first write this data to file

open('output', 'w').write(data)

and then run the suggestions as

for line in iWordsMatch(open("output"), "re", "return"):

it works fine for all practical purposes.

I'd still like to know how to use your suggestion beyond a single
character. Am I goofing something up with: ?

var1 = "this is a test\nand another test"
[line for line in var1 if line.find('test') != -1] []

[Francis Girard]

Hi,

This might even be faster since using re.search, we don't need to parse the
whole line.

Regards,

Francis Girard
=== BEGIN SNAP
## rewords.py

import re
import sys

def iWordsMatch(lines, word, word2):
reWordOneTwo = re.compile(r"((%s.*%s)|(%s.*%s))" %
(word,word2,word2,word))
return (line for line in lines if reWordOneTwo.search(line))

for line in iWordsMatch(open("rewords.py"), "re", "return"):
sys.stdout.write(line)
=== END SNAP


Le mardi 1 Mars 2005 21:57, Francis Girard a écrit*:

Le mardi 1 Mars 2005 21:38, Marc Huffnagle a écrit*:

My understanding of the second question was that he wanted to find lines
which contained both words but, looking at it again, it could go either
way. *If he wants to find lines that contain both of the words, in any
order, then I don't think that it can be done without scanning the line
twice (regex or not).

I don't know if it is really faster but here's a version that finds both
words on the same line. My understanding is that re needs to parse the line
only once. This might count on very large inputs.

=== Begin SNAP
## rewords.py

import re
import sys

def iWordsMatch(lines, word, word2):
reWordOneTwo = re.compile(r".*((%s.*%s)|(%s.*%s)).*" %
(word,word2,word2,word))
return (line for line in lines if reWordOneTwo.match(line))

for line in iWordsMatch(open("rewords.py"), "re", "return"):
sys.stdout.write(line)
=== End SNAP

Regards,

Francis Girard

[Francis Girard]

Hi,

Le mardi 1 Mars 2005 22:15, da****@gmail.com a écrit*:

Now I don't know this stuff very well but I dont think the code

[line for line in document if (line.find('word') != -1 \
* * * * and line.find('wordtwo') != -1)]
would do this as it answers the question in how you thought I asked.

Just use "or" instead of "and" and you'll get what you need.
var1 = "this is a test\nand another test"
[line for line in var1 if line.find('t') != -1]
You are scanning the letters in the string.
You first need to split your input into lines, in order to scan over strings
in a list of strings. Use :

var1 = "this is a test\nand another test"
[line for line in var1.splitlines() if line.find('t') != -1]
for line in iWordsMatch(data, "microsoft", "windows")

Same as above. Use:

for line in iWordsMatch(data.splitlines(), "microsoft", "windows")

Why Microsoft and Windows ? I am very pleased to see that Microsoft Windowsis
now used instead of foo bar.

Regards

[dasacc]

Why Microsoft and Windows ?

B/c it was actually in the data I was trying to parse (though not
something I was needing to parse), I obscured everything except my test
search terms *shrugs*

I saw something on this group about 'to many "or's"' so I figured it
was an option. Thanks for the .splitlines() pointer, it fixed my woe's
I was experiencing as stated earlier.