My function to recursively list all files in directory tree..

[smoothoperator12]

Hi all,


I have spent hours trying to figure out where I have went wrong with my code for my recursive function to list all the files in a directory, and all of the files in all
of its subdirectories.

This is the code:

 
import os
import sys
import dircache
import stat
import string
 
 
 
 
def dirrecur(currsub):
 
 
        thispath = os.getcwd()
 
        print thispath
        print "\n"
 
 
        if(len(currsub) == 0):   #if len = 0, no subdirectories were found.
 
                os.chdir('..')   #go back up one step.
 
        else:
                dlist = []
 
                for z in range(0,len(currsub)):
 
 
                        os.chdir(currsub[z])
 
                        print currsub[z]
                        print "\n"
 
                        direntries = dircache.opendir('.') #open this directory
 
                        for j in range(0,len(direntries)):
 
                                statresult = os.stat(direntries[j])
 
                                if (stat.S_ISDIR(statresult.st_mode) != 0):
 
 
                                        dlist.append(direntries[j])     #dlist is built here.
                                        print direntries[j]
 
                                else:
 
                                        print direntries[j]
 
                         dirrecur(dlist)   #recur again with full list of subdirectories.
 
 
 
#MAIN
 
 
abspath = os.getcwd()
 
direntries = dircache.opendir(abspath)
 
dirlist = []
 
 
 
for i in range(0,len(direntries)):
 
 
        statresult = os.stat(direntries[i])
 
        if (stat.S_ISDIR(statresult.st_mode) != 0): #if it's a directory file.
 
                dirlist.append(direntries[i])
                print direntries[i]                 #print it out too.
 
        else:
 
                print direntries[i]
 
 
 
dirrecur(dirlist)
 
 
 

It goes down to level 3 of the directory tree, but at one point I get the
error 2, the no such file or directory message. I have checked the
file permissions of the directory that it gets stuck on, and I have no idea why
it is gettintg stuck on that directory and not one of the directories before it.

I would be extremely grateful if someone could lend another set of eyes to my
code and see where I'm going wrong at, and why it goes to level 3 so to speak
of the tree and then fails.

I'm stumped!


Thanks so much in advance. Any and all input would be greatly appreciated!

[bartonc]

Perhaps you are trying too hard. os.walk() will do the work for you:

walk( top[, topdown=True [, onerror=None]])

walk() generates the file names in a directory tree, by walking the tree either top down or bottom up. For each directory in the tree rooted at directory top (including top itself), it yields a 3-tuple (dirpath, dirnames, filenames).
dirpath is a string, the path to the directory. dirnames is a list of the names of the subdirectories in dirpath (excluding '.' and '..'). filenames is a list of the names of the non-directory files in dirpath. Note that the names in the lists contain no path components. To get a full path (which begins with top) to a file or directory in dirpath, do os.path.join(dirpath, name).

If optional argument topdown is true or not specified, the triple for a directory is generated before the triples for any of its subdirectories (directories are generated top down). If topdown is false, the triple for a directory is generated after the triples for all of its subdirectories (directories are generated bottom up).

When topdown is true, the caller can modify the dirnames list in-place (perhaps using del or slice assignment), and walk() will only recurse into the subdirectories whose names remain in dirnames; this can be used to prune the search, impose a specific order of visiting, or even to inform walk() about directories the caller creates or renames before it resumes walk() again. Modifying dirnames when topdown is false is ineffective, because in bottom-up mode the directories in dirnames are generated before dirpath itself is generated.

By default errors from the os.listdir() call are ignored. If optional argument onerror is specified, it should be a function; it will be called with one argument, an OSError instance. It can report the error to continue with the walk, or raise the exception to abort the walk. Note that the filename is available as the filename attribute of the exception object.

There are others as well.

[bartonc]

Perhaps you are trying too hard. os.walk() will do the work for you:There are others as well.

Unless you really want the experience of writing a directory walker; in which case, I'll have a look at what you got after work.

[smoothoperator12]

hi bartonc,


Thanks a million for taking a look!!


I would definitely like to learn how to write my own directory walker, and
it is nice to know that if I learn how to use os.walk() that I can get the output
I want, however I really would like to know where my code is going haywire at,
and I am really confused why it goes down to the third level of the directory
tree and then decides to crap out.


Thanks again so much.

[bvdet]

 
import os
import sys
import dircache
import stat
import string
 
 
 
 
def dirrecur(currsub):
 
 
        thispath = os.getcwd()
 
        print thispath
        print "\n"
 
 
        if(len(currsub) == 0):   #if len = 0, no subdirectories were found.
 
                os.chdir('..')   #go back up one step.
 
        else:
                dlist = []
 
                for z in range(0,len(currsub)):
 
 
                        os.chdir(currsub[z])
 
                        print currsub[z]
                        print "\n"
 
                        direntries = dircache.opendir('.') #open this directory
 
                        for j in range(0,len(direntries)):
 
                                statresult = os.stat(direntries[j])
 
                                if (stat.S_ISDIR(statresult.st_mode) != 0):
 
 
                                        dlist.append(direntries[j])     #dlist is built here.
                                        print direntries[j]
 
                                else:
 
                                        print direntries[j]
 
                         dirrecur(dlist)   #recur again with full list of subdirectories.
 
 
 
#MAIN
 
 
abspath = os.getcwd()
 
direntries = dircache.opendir(abspath)
 
dirlist = []
 
 
 
for i in range(0,len(direntries)):
 
 
        statresult = os.stat(direntries[i])
 
        if (stat.S_ISDIR(statresult.st_mode) != 0): #if it's a directory file.
 
                dirlist.append(direntries[i])
                print direntries[i]                 #print it out too.
 
        else:
 
                print direntries[i]
 
 
 
dirrecur(dirlist)
 
 
 

I think it would be simpler to use os.listdir(). Check out the thread HERE.

<Mod EDIT: Nice recovery below, BV. Admin is still trying to work the bugs out of code tags inside quotes. The weird thing is that it doesn't always happen. I've removed the quote here, rather than the code tags.>

[bvdet]

Hi all,


I have spent hours trying to figure out where I have went wrong with my code for my recursive function to list all the files in a directory, and all of the files in all
of its subdirectories.





It goes down to level 3 of the directory tree, but at one point I get the
error 2, the no such file or directory message. I have checked the
file permissions of the directory that it gets stuck on, and I have no idea why
it is gettintg stuck on that directory and not one of the directories before it.

I would be extremely grateful if someone could lend another set of eyes to my
code and see where I'm going wrong at, and why it goes to level 3 so to speak
of the tree and then fails.

I'm stumped!


Thanks so much in advance. Any and all input would be greatly appreciated!

I think it would be simpler to use os.listdir(). Check out the thread HERE.
BV

[smoothoperator12]

If it helps any what I am looking for in my output,

I want the output to resemble the UNIX program ls -R as closely as possible.


I am running solaris 2.10.


I notice that when I use os.walk(), that each directory is showing files that
even ls -a won't show on those directories.

Are those files placed there by the system administrator?

[bartonc]

If it helps any what I am looking for in my output,

I want the output to resemble the UNIX program ls -R as closely as possible.


I am running solaris 2.10.


I notice that when I use os.walk(), that each directory is showing files that
even ls -a won't show on those directories.

Are those files placed there by the system administrator?

I'm afraid that you are dealing with a bunch of Windows users, here.
Not going to be much help on your output format, but, with regard to that, I do see a need to pass the recursion depth into the next iteration. Perhaps starting with a very simple, clean example of a recursive directory walker would help. I really like the simplicity of copytree() from the shutil module:

 
def copytree(src, dst, symlinks=False):
    """Recursively copy a directory tree using copy2().
 
    The destination directory must not already exist.
    If exception(s) occur, an Error is raised with a list of reasons.
 
    If the optional symlinks flag is true, symbolic links in the
    source tree result in symbolic links in the destination tree; if
    it is false, the contents of the files pointed to by symbolic
    links are copied.
 
    XXX Consider this example code rather than the ultimate tool.
 
    """
    names = os.listdir(src)
    os.mkdir(dst)
    errors = []
    for name in names:
        srcname = os.path.join(src, name)
        dstname = os.path.join(dst, name)
        try:
            if symlinks and os.path.islink(srcname):
                linkto = os.readlink(srcname)
                os.symlink(linkto, dstname)
            elif os.path.isdir(srcname):
                copytree(srcname, dstname, symlinks)
            else:
                copy2(srcname, dstname)
            # XXX What about devices, sockets etc.?
        except (IOError, os.error), why:
            errors.append((srcname, dstname, why))
        # catch the Error from the recursive copytree so that we can
        # continue with other files
        except Error, err:
            errors.extend(err.args[0])
    if errors:
        raise Error, errors

So, what's that? Maybe 3 lines to get the name sorted out and 3 more to decide where or not to do the recursion.

[ghostdog74]

If it helps any what I am looking for in my output,

I want the output to resemble the UNIX program ls -R as closely as possible.


I am running solaris 2.10.


I notice that when I use os.walk(), that each directory is showing files that
even ls -a won't show on those directories.

Are those files placed there by the system administrator?

what does not show in ls -a that is shown in os.walk? here's a little walker for you

from os import listdir, sep
from os.path import isdir
def skywalker(dir):
    for file in listdir(dir):
        path = dir + sep + file        
        if isdir(path):
            print "path: ",path
            skywalker(path)
        else:
            print '->' + file
skywalker("/home/test")
 

[bvdet]

I have been using this function:

import os
 
def dirEntries(dir_name, subdir, *args):
    '''Return a list of file names found in directory 'dir_name'
    If 'subdir' is True, recursively access subdirectories under 'dir_name'.
    Additional arguments, if any, are file extensions to match filenames. Matched
        file names are added to the list.
    If there are no additional arguments, all files found in the directory are
        added to the list.
    Example usage: fileList = dir_list(r'H:\TEMP', False, 'txt', 'py')
        Only files with 'txt' and 'py' extensions will be added to the list.
    Example usage: fileList = dir_list(r'H:\TEMP', True)
        All files and all the files in subdirectories under H:\TEMP will be added
        to the list.
    '''
    fileList = []
    for file in os.listdir(dir_name):
        dirfile = os.path.join(dir_name, file)
        if os.path.isfile(dirfile):
            if len(args) == 0:
                fileList.append(dirfile)
            else:
                if os.path.splitext(dirfile)[1][1:] in args:
                    fileList.append(dirfile)
        # recursively access file names in subdirectories
        elif os.path.isdir(dirfile) and subdir:
            print "Accessing directory:", dirfile
            fileList += dirEntries(dirfile, subdir, *args)
    return fileList