newb question: file searching

[jaysherby]

I'm new at Python and I need a little advice. Part of the script I'm
trying to write needs to be aware of all the files of a certain
extension in the script's path and all sub-directories. Can someone
set me on the right path to what modules and calls to use to do that?
You'd think that it would be a fairly simple proposition, but I can't
find examples anywhere. Thanks.

[jaysherby]

Here's my code:

def getFileList():
import os
imageList = []
for dirpath, dirnames, filenames in os.walk(os.getc wd()):
for filename in filenames:
for dirname in dirnames:
if not dirname.startsw ith('.'):
if filename.lower( ).endswith('.jp g') and not
filename.starts with('.'):
imageList.appen d(os.path.join( dirpath, filename))
return imageList

I've adapted it around all the much appreciated suggestions. However,
I'm running into two very peculiar logical errors. First, I'm getting
repeated entries. That's no good. One image, one entry in the list.
The other is that if I run the script from my Desktop folder, it won't
find any files, and I make sure to have lots of jpegs in the Desktop
folder for the test. Can anyone figure this out?

ja*******@gmail .com wrote:

I'm new at Python and I need a little advice. Part of the script I'm
trying to write needs to be aware of all the files of a certain
extension in the script's path and all sub-directories. Can someone
set me on the right path to what modules and calls to use to do that?
You'd think that it would be a fairly simple proposition, but I can't
find examples anywhere. Thanks.

[jaysherby]

Something's really not reliable in my logic. I say this because if I
change the extension to .png then a file in a hidden directory (one the
starts with '.') shows up! The most frustrating part is that there are
..jpg files in the very same directory that don't show up when it
searches for jpegs.

I tried os.walk('.') and it works, so I'll be using that instead.

ja*******@gmail .com wrote:

Here's my code:

def getFileList():
import os
imageList = []
for dirpath, dirnames, filenames in os.walk(os.getc wd()):
for filename in filenames:
for dirname in dirnames:
if not dirname.startsw ith('.'):
if filename.lower( ).endswith('.jp g') and not
filename.starts with('.'):
imageList.appen d(os.path.join( dirpath, filename))
return imageList

I've adapted it around all the much appreciated suggestions. However,
I'm running into two very peculiar logical errors. First, I'm getting
repeated entries. That's no good. One image, one entry in the list.
The other is that if I run the script from my Desktop folder, it won't
find any files, and I make sure to have lots of jpegs in the Desktop
folder for the test. Can anyone figure this out?

ja*******@gmail .com wrote:

I'm new at Python and I need a little advice. Part of the script I'm
trying to write needs to be aware of all the files of a certain
extension in the script's path and all sub-directories. Can someone
set me on the right path to what modules and calls to use to do that?
You'd think that it would be a fairly simple proposition, but I can't
find examples anywhere. Thanks.

[Gabriel Genellina]

At Tuesday 8/8/2006 21:11, ja*******@gmail .com wrote:

>Here's my code:

def getFileList():
import os
imageList = []
for dirpath, dirnames, filenames in os.walk(os.getc wd()):
for filename in filenames:
for dirname in dirnames:
if not dirname.startsw ith('.'):
if
filename.lower( ).endswith('.jp g') and not
filename.start swith('.'):

imageList.appe nd(os.path.join (dirpath, filename))
return imageList

I've adapted it around all the much appreciated suggestions. However,
I'm running into two very peculiar logical errors. First, I'm getting
repeated entries. That's no good. One image, one entry in the list.

That's because of the double iteration. dirnames and filenames are
two distinct, complementary, lists. (If a directory entry is a
directory it goes into dirnames; if it's a file it goes into
filenames). So you have to process them one after another.

>def getFileList():
import os
imageList = []
for dirpath, dirnames, filenames in os.walk(os.getc wd()):
for filename in filenames:
if filename.lower( ).endswith('.jp g') and
not filename.starts with('.'):

imageList.appe nd(os.path.join (dirpath, filename))
for i in reversed(range( len(dirnames))) :
if dirnames[i].startswith('.' ): del dirnames[i]
return imageList

reversed() because you need to modify dirnames in-place, so it's
better to process the list backwards.

Gabriel Genellina
Softlab SRL

_______________ _______________ _______________ _____
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas

[jaysherby]

I've narrowed down the problem. All the problems start when I try to
eliminate the hidden files and directories. Is there a better way to
do this?
ja*******@gmail .com wrote:

I'm new at Python and I need a little advice. Part of the script I'm
trying to write needs to be aware of all the files of a certain
extension in the script's path and all sub-directories. Can someone
set me on the right path to what modules and calls to use to do that?
You'd think that it would be a fairly simple proposition, but I can't
find examples anywhere. Thanks.

[jaysherby]

That worked perfectly. Thank you. That was exactly what I was looking
for. However, can you explain to me what the following code actually
does?

reversed(range( len(dirnames)))
Gabriel Genellina wrote:

At Tuesday 8/8/2006 21:11, ja*******@gmail .com wrote:

Here's my code:

def getFileList():
import os
imageList = []
for dirpath, dirnames, filenames in os.walk(os.getc wd()):
for filename in filenames:
for dirname in dirnames:
if not dirname.startsw ith('.'):
if
filename.lower( ).endswith('.jp g') and not
filename.starts with('.'):

imageList.appen d(os.path.join( dirpath, filename))
return imageList

I've adapted it around all the much appreciated suggestions. However,
I'm running into two very peculiar logical errors. First, I'm getting
repeated entries. That's no good. One image, one entry in the list.

That's because of the double iteration. dirnames and filenames are
two distinct, complementary, lists. (If a directory entry is a
directory it goes into dirnames; if it's a file it goes into
filenames). So you have to process them one after another.

def getFileList():
import os
imageList = []
for dirpath, dirnames, filenames in os.walk(os.getc wd()):
for filename in filenames:
if filename.lower( ).endswith('.jp g') and
not filename.starts with('.'):

imageList.appen d(os.path.join( dirpath, filename))
for i in reversed(range( len(dirnames))) :
if dirnames[i].startswith('.' ): del dirnames[i]
return imageList

reversed() because you need to modify dirnames in-place, so it's
better to process the list backwards.

Gabriel Genellina
Softlab SRL

_______________ _______________ _______________ _____
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas

[Justin Azoff]

ja*******@gmail .com wrote:

I've narrowed down the problem. All the problems start when I try to
eliminate the hidden files and directories. Is there a better way to
do this?

Well you almost have it, but your problem is that you are trying to do
too many things in one function. (I bet I am starting to sound like a
broken record :-)) The four distinct things you are doing are:

* getting a list of all files in a tree
* combining a files directory with its name to give the full path
* ignoring hidden directories
* matching files based on their extension

If you split up each of those things into their own function you will
end up with smaller easier to test pieces, and separate, reusable
functions.

The core function would be basically what you already have:

def get_files(direc tory, include_hidden= False):
"""Return an expanded list of files for a directory tree
optionally not ignoring hidden directories"""
for path, dirs, files in os.walk(directo ry):
for fn in files:
full = os.path.join(pa th, fn)
yield full

if not include_hidden:
remove_hidden(d irs)

and remove_hidden is a short, but tricky function since the directory
list needs to be edited in place:

def remove_hidden(d irlist):
"""For a list containing directory names, remove
any that start with a dot"""

dirlist[:] = [d for d in dirlist if not d.startswith('. ')]

at this point, you can play with get_files on it's own, and test
whether or not the include_hidden parameter works as expected.

For the final step, I'd use an approach that pulls out the extension
itself, and checks to see if it is in a list(or better, a set) of
allowed filenames. globbing (*.foo) works as well, but if you are only
ever matching on the extension, I believe this will work better.

def get_files_by_ex t(directory, ext_list, include_hidden= False):
"""Return an expanded list of files for a directory tree
where the file ends with one of the extensions in ext_list"""
ext_list = set(ext_list)

for fn in get_files(direc tory, include_hidden) :
_, ext = os.path.splitex t(fn)
ext=ext[1:] #remove dot
if ext.lower() in ext_list:
yield fn

notice at this point we still haven't said anything about images! The
task of finding files by extension is pretty generic, so it shouldn't
be concerned about the actual extensions.

once that works, you can simply do

def get_images(dire ctory, include_hidden= False):
image_exts = ('jpg','jpeg',' gif','png','bmp ')
return get_files_by_ex t(directory, image_exts, include_hidden)

Hope this helps :-)

--
- Justin

[jaysherby]

I do appreciate the advice, but I've got a 12 line function that does
all of that. And it works! I just wish I understood a particular line
of it.

def getFileList(*ex tensions):
import os
imageList = []
for dirpath, dirnames, files in os.walk('.'):
for filename in files:
name, ext = os.path.splitex t(filename)
if ext.lower() in extensions and not filename.starts with('.'):
imageList.appen d(os.path.join( dirpath, filename))
for dirname in reversed(range( len(dirnames))) :
if dirnames[dirname].startswith('.' ):
del dirnames[dirname]

return imageList

print getFileList('.j pg', '.gif', '.png')

The line I don't understand is:
reversed(range( len(dirnames)))
Justin Azoff wrote:

ja*******@gmail .com wrote:

I've narrowed down the problem. All the problems start when I try to
eliminate the hidden files and directories. Is there a better way to
do this?

Well you almost have it, but your problem is that you are trying to do
too many things in one function. (I bet I am starting to sound like a
broken record :-)) The four distinct things you are doing are:

* getting a list of all files in a tree
* combining a files directory with its name to give the full path
* ignoring hidden directories
* matching files based on their extension

If you split up each of those things into their own function you will
end up with smaller easier to test pieces, and separate, reusable
functions.

The core function would be basically what you already have:

def get_files(direc tory, include_hidden= False):
"""Return an expanded list of files for a directory tree
optionally not ignoring hidden directories"""
for path, dirs, files in os.walk(directo ry):
for fn in files:
full = os.path.join(pa th, fn)
yield full

if not include_hidden:
remove_hidden(d irs)

and remove_hidden is a short, but tricky function since the directory
list needs to be edited in place:

def remove_hidden(d irlist):
"""For a list containing directory names, remove
any that start with a dot"""

dirlist[:] = [d for d in dirlist if not d.startswith('. ')]

at this point, you can play with get_files on it's own, and test
whether or not the include_hidden parameter works as expected.

For the final step, I'd use an approach that pulls out the extension
itself, and checks to see if it is in a list(or better, a set) of
allowed filenames. globbing (*.foo) works as well, but if you are only
ever matching on the extension, I believe this will work better.

def get_files_by_ex t(directory, ext_list, include_hidden= False):
"""Return an expanded list of files for a directory tree
where the file ends with one of the extensions in ext_list"""
ext_list = set(ext_list)

for fn in get_files(direc tory, include_hidden) :
_, ext = os.path.splitex t(fn)
ext=ext[1:] #remove dot
if ext.lower() in ext_list:
yield fn

notice at this point we still haven't said anything about images! The
task of finding files by extension is pretty generic, so it shouldn't
be concerned about the actual extensions.

once that works, you can simply do

def get_images(dire ctory, include_hidden= False):
image_exts = ('jpg','jpeg',' gif','png','bmp ')
return get_files_by_ex t(directory, image_exts, include_hidden)

Hope this helps :-)

--
- Justin

[Justin Azoff]

ja*******@gmail .com wrote:

I do appreciate the advice, but I've got a 12 line function that does
all of that. And it works! I just wish I understood a particular line
of it.

You miss the point. The functions I posted, up until get_files_by_ex t
which is the equivalent of your getFileList, total 17 actual lines.
The 5 extra lines give 3 extra features. Maybe in a while when you
need to do a similar file search you will realize why my way is better.

[snip]

The line I don't understand is:
reversed(range( len(dirnames)))

This is why I wrote and documented a separate remove_hidden function,
it can be tricky. If you broke it up into multiple lines, and added
print statements it would be clear what it does.

l = len(dirnames) # l is the number of elements in dirnames, e.g. 6
r = range(l) # r contains the numbers 0,1,2,3,4,5
rv = reversed(r) # rv contains the numbers 5,4,3,2,1,0

The problem arises from how to remove elements in a list as you are
going through it. If you delete element 0, element 1 then becomes
element 0, and funny things happen. That particular solution is
relatively simple, it just deletes elements from the end instead. That
complicated expression arises because python doesn't have "normal" for
loops. The version of remove_hidden I wrote is simpler, but relies on
the even more obscure lst[:] construct for re-assigning a list. Both
of them accomplish the same thing though, so if you wanted, you should
be able to replace those 3 lines with just

dirnames[:] = [d for d in dirnames if not d.startswith('. ')]
--
- Justin

[John Machin]

ja*******@gmail .com wrote:

I do appreciate the advice, but I've got a 12 line function that does
all of that. And it works! I just wish I understood a particular line
of it.

def getFileList(*ex tensions):
import os
imageList = []
for dirpath, dirnames, files in os.walk('.'):
for filename in files:
name, ext = os.path.splitex t(filename)
if ext.lower() in extensions and not filename.starts with('.'):
imageList.appen d(os.path.join( dirpath, filename))
for dirname in reversed(range( len(dirnames))) :
if dirnames[dirname].startswith('.' ):
del dirnames[dirname]

return imageList

print getFileList('.j pg', '.gif', '.png')

The line I don't understand is:
reversed(range( len(dirnames)))

For a start, change "dirname" to "dirindex" (without changing
"dirnames"! ) in that line and the next two lines -- this may help your
understanding.

The purpose of that loop is to eliminate from dirnames any entries
which start with ".". This needs to be done in-situ -- concocting a new
list and binding the name "dirnames" to it won't work.
The safest understandable way to delete entries from a list while
iterating over it is to do it backwards.

Doing it forwards doesn't always work; example:

#>>dirnames = ['foo', 'bar', 'zot']
#>>for x in range(len(dirna mes)):
.... if dirnames[x] == 'bar':
.... del dirnames[x]
....
Traceback (most recent call last):
File "<stdin>", line 2, in ?
IndexError: list index out of range

HTH,
John

[jaysherby]

I'm sorry. I didn't mean to offend you. I never thought your way was
inferior.
Justin Azoff wrote:

ja*******@gmail .com wrote:

I do appreciate the advice, but I've got a 12 line function that does
all of that. And it works! I just wish I understood a particular line
of it.

You miss the point. The functions I posted, up until get_files_by_ex t
which is the equivalent of your getFileList, total 17 actual lines.
The 5 extra lines give 3 extra features. Maybe in a while when you
need to do a similar file search you will realize why my way is better.

[snip]

The line I don't understand is:
reversed(range( len(dirnames)))

This is why I wrote and documented a separate remove_hidden function,
it can be tricky. If you broke it up into multiple lines, and added
print statements it would be clear what it does.

l = len(dirnames) # l is the number of elements in dirnames, e.g. 6
r = range(l) # r contains the numbers 0,1,2,3,4,5
rv = reversed(r) # rv contains the numbers 5,4,3,2,1,0

The problem arises from how to remove elements in a list as you are
going through it. If you delete element 0, element 1 then becomes
element 0, and funny things happen. That particular solution is
relatively simple, it just deletes elements from the end instead. That
complicated expression arises because python doesn't have "normal" for
loops. The version of remove_hidden I wrote is simpler, but relies on
the even more obscure lst[:] construct for re-assigning a list. Both
of them accomplish the same thing though, so if you wanted, you should
be able to replace those 3 lines with just

dirnames[:] = [d for d in dirnames if not d.startswith('. ')]
--
- Justin