searching for words in a file

6,596

Expert 4TB

If I have a large number of files how would I search for a particular word in all of the files then have the system print out all the lines in those files that have the word that I am looking for.

What would have to be different if I wanted to do another program that would search for two words of my choice and then only print out the lines in those files that contain both those words.

1) Use the glob module to look through directory trees for the files. Open each file and read it into a buffer. Close the file. Use the re module to create a Regular Expression object. Print the (file name?) that glob just gave you (or the buffer if you meant "print the file") if your Regular Expression gets a match.

2) build your Regular Expression with variables.

Jun 19 '07 #2

I am not sure what you mean by the glob module or anything? What I am trying to do is to write a script that would search 60 files for a certain word and then print out all the lines that have that word, then I want to also in a different program write a script that given two words, would search for those words then print out the lines in those files that contain both those words.

Jun 19 '07 #3

What I have are about 70 different files in a folder called "files" In that folder each file if opened in a program like word contains a list of sentences. So what I need to do is to be able to type in a search word and then have the program return all the sentences out of all the files that contain that word.

Jun 19 '07 #4

bvdet

2,851

Expert Mod 2GB

What I have are about 70 different files in a folder called "files" In that folder each file if opened in a program like word contains a list of sentences. So what I need to do is to be able to type in a search word and then have the program return all the sentences out of all the files that contain that word.

Compile a list of file names in memory. The os module can be used for that. Create a for loop to iterate on the list of file names:

Expand|Select|Wrap|Line Numbers

for file_name in fileList:

Open and iterate on each file:

Expand|Select|Wrap|Line Numbers

for line in file_object:

Use the 'in' operator to test if the word is contained in line.

Expand|Select|Wrap|Line Numbers

 if word in line:

    print line

For multiple keywords, a regex solution can be implemented.

Expand|Select|Wrap|Line Numbers

 import re

keyList = ['word1', 'word2']

patt = re.compile('|'.join(keyList), re.IGNORECASE)

for fn in fileList:

    f = open(fn)

    for line in f:

        if patt.search(line.lower()):

            print line

    f.close()

Jun 20 '07 #5

All of the files I need to search through are in a folder called 'files' on my c: drive so how would I tell it to look in the folder and then look through each of those text files and search for a word that I tell it to

Jun 21 '07 #6

bvdet

2,851

Expert Mod 2GB

All of the files I need to search through are in a folder called 'files' on my c: drive so how would I tell it to look in the folder and then look through each of those text files and search for a word that I tell it to

This returns a list of entries in 'dir_name':

Expand|Select|Wrap|Line Numbers

 >>> import os

>>> dir_name = r'C:\files'

>>> entryList = os.listdir(dir_name)

Subdirectories are included in the list. Iterate on the list to search for your keywords.

Jun 21 '07 #7

6,596

Expert 4TB

All of the files I need to search through are in a folder called 'files' on my c: drive so how would I tell it to look in the folder and then look through each of those text files and search for a word that I tell it to

Hi texas22. I'm glad to see that bvdet has provided the os module call needed for your purposes.

I'd like to explain what I see going on in this thread:
1) You have provided lots of words describing your goal; That's good.
2) I have replied with lots of words (instead of code) describing a solution; That's due to #1, above
3) bvdet has provide the bare minimum to get you going; That's the way things work around here. We help (not do it for you).
4) You have not shown any attempt (by posting code) to make this work yourself; That's not good.

So give it an honest try. You'll see the level of participation from members go up!

Jun 21 '07 #8

I'm kind of in Texas22's stage here. You all know what you are talking about but it's all Greek to me. You assume we would know what a glob module is when I don't have the concept of what needs to happen. How can we try it? I don't mean to offend and I really appreciate your help but could you simplify your explanation for us python toddlers?

Hi texas22. I'm glad to see that bvdet has provided the os module call needed for your purposes.

I'd like to explain what I see going on in this thread:
1) You have provided lots of words describing your goal; That's good.
2) I have replied with lots of words (instead of code) describing a solution; That's due to #1, above
3) bvdet has provide the bare minimum to get you going; That's the way things work around here. We help (not do it for you).
4) You have not shown any attempt (by posting code) to make this work yourself; That's not good.

So give it an honest try. You'll see the level of participation from members go up!

Jun 21 '07 #9

Thanks, for all the input and yes you are right I am trying to use the help that I am given so I will start posting my code in order to become more involved. My big problem is just getting started and then trying to figure out what each step means by seeing it so anyways here is what I got so far. Being new to programming and especially no prior knowledge whatsoever on python this is a big learning curve for me. The comments I put next to each line is what I am assuming each line does so if you could clarify for me.

(code)
import os

dir_name = r'C:\books' #This tells the program where to look#

entryList = os.listdir(dir_name) # not sure what exactly this does#

for file_name in fileList: #not sure what this does but when I run the program says this is not defined#

for line in file_object: #what does this line exactly do#

if word in line:

print line

So guess my question is, is there a place in the program that will prompt the user for the word they want to search for. What does it take to prompt the user for such a thing cause as far as I can tell there doesn't seem to be a place that does that. Right now I guess all this will do is pretty much tell the program where to look for the files but it won't yet conduct the search. Am I seeing it right or am I way off base.
Thanks for your help

Jun 21 '07 #10

I have the following code. It prints out the list of files inside the folder but doesn't seem to be searching the files.

#!C:\PYTHON25\PYTHON.EXE

import os
dir_name= r'C:\python25\books\books\books'
entryList=os.listdir(dir_name)
print entryList
searchWord= "Moses"
for file_name in entryList:
for line in file_name:
if searchWord in line:
print line

Am I even close? Do I have to open and close each file?

Thanks for your help.

Jun 21 '07 #11

ilikepython

844

Expert 512MB

Thanks, for all the input and yes you are right I am trying to use the help that I am given so I will start posting my code in order to become more involved. My big problem is just getting started and then trying to figure out what each step means by seeing it so anyways here is what I got so far. Being new to programming and especially no prior knowledge whatsoever on python this is a big learning curve for me. The comments I put next to each line is what I am assuming each line does so if you could clarify for me.

(code)
import os

dir_name = r'C:\books' #This tells the program where to look#

entryList = os.listdir(dir_name) # not sure what exactly this does#

for file_name in fileList: #not sure what this does but when I run the program says this is not defined#

for line in file_object: #what does this line exactly do#

if word in line:

print line

So guess my question is, is there a place in the program that will prompt the user for the word they want to search for. What does it take to prompt the user for such a thing cause as far as I can tell there doesn't seem to be a place that does that. Right now I guess all this will do is pretty much tell the program where to look for the files but it won't yet conduct the search. Am I seeing it right or am I way off base.
Thanks for your help

Expand|Select|Wrap|Line Numbers

entryList = os.listdir(dir_name)

That line lists all the files and folders in dir_name.

In the next code you have your variable names mixed up a bit:

Expand|Select|Wrap|Line Numbers

 
for file_name in fileList:          # should be entryList instead of fileList

    for line in file_object:         # should be "for line in file(file_name).readlines():"

        if word in line:

            print line

So the corrected code would look like this:

Expand|Select|Wrap|Line Numbers

 
dir_name = r"C:\Books"

entryList = os.listdirs(dir_name)
 
for file_name in entryList:                           # iterate over files in entryList

    for line in file(file_name).readlines():        # iterate over lines in each file 

        if word in line:

            print line

If you want to prompt the user for a word to search for you use raw_input().
The place where to put it depends on how often do you want to change the word to search for. If it's going to stay the whole program you can just add this line at the top of your file:

Expand|Select|Wrap|Line Numbers

word = raw_input("Enter a word to search for: ")

Jun 21 '07 #12

ilikepython

844

Expert 512MB

I have the following code. It prints out the list of files inside the folder but doesn't seem to be searching the files.

#!C:\PYTHON25\PYTHON.EXE

import os
dir_name= r'C:\python25\books\books\books'
entryList=os.listdir(dir_name)
print entryList
searchWord= "Moses"
for file_name in entryList:
for line in file_name:
if searchWord in line:
print line

Am I even close? Do I have to open and close each file?

Thanks for your help.

You are iterating over the actual string of the file name. You want to iterate over the file object. Please see my previous reply and use python code tags ([code=python ][/code ]).

Jun 21 '07 #13

[code=python ]

import os

dir_name = r"C:\books"

word = raw_input("Enter a word to search for: ")

entryList = os.listdirs(dir_name)

for file_name in entryList:

for line in file(file_name).readlines():

if word in line:

print line

[/code ]

This is what I have so far when I run it I am getting the error
"AttributeError: 'module' object has no attribute 'listdirs'

What does this mean and am I on the right track?

Jun 21 '07 #14

sorry guys should not have troubled you with that one just had to take the 's' off of the line
[code = python]

entrylist=os.listdir(dir_name) #no 's' on the end of list dir

The error I'm getting now is:
"for line in file(file_name).readlines():

IO Error: [Errno2] No such file or directory: '1ch'

The '1ch' is the name of one of the files contained in the folder books

Jun 21 '07 #15

bvdet

2,851

Expert Mod 2GB

Expand|Select|Wrap|Line Numbers

import os

dir_name = r"C:\books"

word = raw_input("Enter a word to search for: ")

entryList = os.listdirs(dir_name)

for file_name in entryList:

    for line in file(file_name).readlines():

        if word in line:

            print line

This is what I have so far when I run it I am getting the error
"AttributeError: 'module' object has no attribute 'listdirs'

What does this mean and am I on the right track?

You are on the correct path. You have an extra space in your closing code tag. It is 'os.listdir' instead of 'os.listdirs'. You need to exclude subdirectories from the file list. The full path and file name must be used to open the files unless the directory is on the Python path. Try this:

Expand|Select|Wrap|Line Numbers

 import os

dir_name = r'C:\my_directory'

word = raw_input("Enter a word to search for: ")

entryList = [os.path.join(dir_name, fn) for fn in os.listdir(dir_name) if os.path.isfile(os.path.join(dir_name, fn))]
 
for file_name in entryList:

    for line in file(file_name).readlines():

        if word in line:

            print line

Jun 21 '07 #16

I change the second line below to read the lines but I am getting the error message below.

Expand|Select|Wrap|Line Numbers

 
for file_name in entryList:

    for line in file(file_name).readlines():

        if searchWord in line:

            print line

Traceback (most recent call last):
File "C:/Python25/search.py", line 10, in <module>
for line in file(file_name).readlines():
IOError: [Errno 2] No such file or directory: '1ch'

I double checked the file address and it is printing the file name out so I think the address is ok. I also opened the file in VIM and it seems ok. What am I doing wrong?

Jun 21 '07 #17

ghostdog74

511

Expert 256MB

I change the second line below to read the lines but I am getting the error message below.

Expand|Select|Wrap|Line Numbers

for file_name in entryList:

    for line in file(file_name).readlines():

        if searchWord in line:

            print line

Traceback (most recent call last):
File "C:/Python25/search.py", line 10, in <module>
for line in file(file_name).readlines():
IOError: [Errno 2] No such file or directory: '1ch'

I double checked the file address and it is printing the file name out so I think the address is ok. I also opened the file in VIM and it seems ok. What am I doing wrong?

check the file names in those directories, whether there are spaces in them.

Jun 22 '07 #18

check the file names in those directories, whether there are spaces in them.

None of the files have a space in the name.

Jun 22 '07 #19

bvdet

2,851

Expert Mod 2GB

None of the files have a space in the name.

Try this:

Expand|Select|Wrap|Line Numbers

open(os.path.join(dir_name, file_name)).readlines()

Jun 22 '07 #20

Try this:

Expand|Select|Wrap|Line Numbers

open(os.path.join(dir_name, file_name)).readlines()

I am not sure where to place that so I put it before the loop and got an error that file_name wasn't defined. I placed it inside the loop and it didn't make a difference.

Jun 22 '07 #21

bvdet

2,851

Expert Mod 2GB

I am not sure where to place that so I put it before the loop and got an error that file_name wasn't defined. I placed it inside the loop and it didn't make a difference.

Expand|Select|Wrap|Line Numbers

 import os

dir_name = 'your_directory'

word = raw_input("Enter a word to search for: ")

entryList = [os.path.join(dir_name, fn) for fn in os.listdir(dir_name) if os.path.isfile(os.path.join(dir_name, fn))]
 
for file_name in entryList:

    for line in file(file_name).readlines():

        if word in line:

            print line

Expand|Select|Wrap|Line Numbers

 import os

dir_name = r'C:\SDS2_7.0\macro\New Versions'

word = raw_input("Enter a word to search for: ")

entryList = []

for fn in os.listdir(dir_name):

    fn = os.path.join(dir_name, fn)

    if os.path.isfile(fn):

        entryList.append(fn)
 
# See above for the rest of the code

Jun 22 '07 #22

Thanks for all the help guys

Jun 22 '07 #23

That worked, now I need to work on more than one search word. Thanks!

Expand|Select|Wrap|Line Numbers

import os

dir_name = 'your_directory'

word = raw_input("Enter a word to search for: ")

entryList = [os.path.join(dir_name, fn) for fn in os.listdir(dir_name) if os.path.isfile(os.path.join(dir_name, fn))]

for file_name in entryList:

    for line in file(file_name).readlines():

        if word in line:

            print line

OR

Expand|Select|Wrap|Line Numbers

import os

dir_name = r'C:\SDS2_7.0\macro\New Versions'

word = raw_input("Enter a word to search for: ")

entryList = []

for fn in os.listdir(dir_name):

    fn = os.path.join(dir_name, fn)

    if os.path.isfile(fn):

        entryList.append(fn)

# See above for the rest of the code

Jun 22 '07 #24

bvdet

2,851

Expert Mod 2GB

That worked, now I need to work on more than one search word. Thanks!

See post #5 this thread.

Jun 22 '07 #25

6,596

Expert 4TB

I'm kind of in Texas22's stage here. You all know what you are talking about but it's all Greek to me. You assume we would know what a glob module is when I don't have the concept of what needs to happen. How can we try it? I don't mean to offend and I really appreciate your help but could you simplify your explanation for us python toddlers?

As you are new to the Python Forum here on TheScripts (and I really hate to be the "enforcer"), I have allowed your posts to remain in this tread even though (and I hope you can see it, too) it makes quite a mess when there are two people asking questions in the same thread. I'm anxious for this question to be resolved so that you two "toddlers" can start new threads of your own.

Please feel free to hit that "start discussion" button any ol' time. And, please, don't jump in on someone else's thread in the future.

Thanks.

By the way, we actually have some Posting Guidelines which may help you get to know some of the rules and policies better.

Jun 23 '07 #26

Sorry, I thought this line in the guidelines and thought I should use Texas22's rather than start an identical one.

"If you wish to post a question do not post it in a discussion created by someone else unless it is about exactly the same problem."

Thanks

As you are new to the Python Forum here on TheScripts (and I really hate to be the "enforcer"), I have allowed your posts to remain in this tread even though (and I hope you can see it, too) it makes quite a mess when there are two people asking questions in the same thread. I'm anxious for this question to be resolved so that you two "toddlers" can start new threads of your own.

Please feel free to hit that "start discussion" button any ol' time. And, please, don't jump in on someone else's thread in the future.

Thanks.

By the way, we actually have some Posting Guidelines which may help you get to know some of the rules and policies better.

Jun 25 '07 #27