469,327 Members | 1,207 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,327 developers. It's quick & easy.

My function to recursively list all files in directory tree..

Hi all,


I have spent hours trying to figure out where I have went wrong with my code for my recursive function to list all the files in a directory, and all of the files in all
of its subdirectories.

This is the code:


Expand|Select|Wrap|Line Numbers
  1.  
  2. import os
  3. import sys
  4. import dircache
  5. import stat
  6. import string
  7.  
  8.  
  9.  
  10.  
  11. def dirrecur(currsub):
  12.  
  13.  
  14.         thispath = os.getcwd()
  15.  
  16.         print thispath
  17.         print "\n"
  18.  
  19.  
  20.         if(len(currsub) == 0):   #if len = 0, no subdirectories were found.
  21.  
  22.                 os.chdir('..')   #go back up one step.
  23.  
  24.         else:
  25.                 dlist = []
  26.  
  27.                 for z in range(0,len(currsub)):
  28.  
  29.  
  30.                         os.chdir(currsub[z])
  31.  
  32.                         print currsub[z]
  33.                         print "\n"
  34.  
  35.                         direntries = dircache.opendir('.') #open this directory
  36.  
  37.                         for j in range(0,len(direntries)):
  38.  
  39.                                 statresult = os.stat(direntries[j])
  40.  
  41.                                 if (stat.S_ISDIR(statresult.st_mode) != 0):
  42.  
  43.  
  44.                                         dlist.append(direntries[j])     #dlist is built here.
  45.                                         print direntries[j]
  46.  
  47.                                 else:
  48.  
  49.                                         print direntries[j]
  50.  
  51.                          dirrecur(dlist)   #recur again with full list of subdirectories.
  52.  
  53.  
  54.  
  55. #MAIN
  56.  
  57.  
  58. abspath = os.getcwd()
  59.  
  60. direntries = dircache.opendir(abspath)
  61.  
  62. dirlist = []
  63.  
  64.  
  65.  
  66. for i in range(0,len(direntries)):
  67.  
  68.  
  69.         statresult = os.stat(direntries[i])
  70.  
  71.         if (stat.S_ISDIR(statresult.st_mode) != 0): #if it's a directory file.
  72.  
  73.                 dirlist.append(direntries[i])
  74.                 print direntries[i]                 #print it out too.
  75.  
  76.         else:
  77.  
  78.                 print direntries[i]
  79.  
  80.  
  81.  
  82. dirrecur(dirlist)
  83.  
  84.  
  85.  


It goes down to level 3 of the directory tree, but at one point I get the
error 2, the no such file or directory message. I have checked the
file permissions of the directory that it gets stuck on, and I have no idea why
it is gettintg stuck on that directory and not one of the directories before it.

I would be extremely grateful if someone could lend another set of eyes to my
code and see where I'm going wrong at, and why it goes to level 3 so to speak
of the tree and then fails.

I'm stumped!


Thanks so much in advance. Any and all input would be greatly appreciated!
Jul 16 '07 #1
9 19318
bartonc
6,596 Expert 4TB
Perhaps you are trying too hard. os.walk() will do the work for you:
walk( top[, topdown=True [, onerror=None]])

walk() generates the file names in a directory tree, by walking the tree either top down or bottom up. For each directory in the tree rooted at directory top (including top itself), it yields a 3-tuple (dirpath, dirnames, filenames).
dirpath is a string, the path to the directory. dirnames is a list of the names of the subdirectories in dirpath (excluding '.' and '..'). filenames is a list of the names of the non-directory files in dirpath. Note that the names in the lists contain no path components. To get a full path (which begins with top) to a file or directory in dirpath, do os.path.join(dirpath, name).

If optional argument topdown is true or not specified, the triple for a directory is generated before the triples for any of its subdirectories (directories are generated top down). If topdown is false, the triple for a directory is generated after the triples for all of its subdirectories (directories are generated bottom up).

When topdown is true, the caller can modify the dirnames list in-place (perhaps using del or slice assignment), and walk() will only recurse into the subdirectories whose names remain in dirnames; this can be used to prune the search, impose a specific order of visiting, or even to inform walk() about directories the caller creates or renames before it resumes walk() again. Modifying dirnames when topdown is false is ineffective, because in bottom-up mode the directories in dirnames are generated before dirpath itself is generated.

By default errors from the os.listdir() call are ignored. If optional argument onerror is specified, it should be a function; it will be called with one argument, an OSError instance. It can report the error to continue with the walk, or raise the exception to abort the walk. Note that the filename is available as the filename attribute of the exception object.
There are others as well.
Jul 16 '07 #2
bartonc
6,596 Expert 4TB
Perhaps you are trying too hard. os.walk() will do the work for you:There are others as well.
Unless you really want the experience of writing a directory walker; in which case, I'll have a look at what you got after work.
Jul 16 '07 #3
hi bartonc,


Thanks a million for taking a look!!


I would definitely like to learn how to write my own directory walker, and
it is nice to know that if I learn how to use os.walk() that I can get the output
I want, however I really would like to know where my code is going haywire at,
and I am really confused why it goes down to the third level of the directory
tree and then decides to crap out.


Thanks again so much.
Jul 16 '07 #4
bvdet
2,851 Expert Mod 2GB
Expand|Select|Wrap|Line Numbers
  1.  
  2. import os
  3. import sys
  4. import dircache
  5. import stat
  6. import string
  7.  
  8.  
  9.  
  10.  
  11. def dirrecur(currsub):
  12.  
  13.  
  14.         thispath = os.getcwd()
  15.  
  16.         print thispath
  17.         print "\n"
  18.  
  19.  
  20.         if(len(currsub) == 0):   #if len = 0, no subdirectories were found.
  21.  
  22.                 os.chdir('..')   #go back up one step.
  23.  
  24.         else:
  25.                 dlist = []
  26.  
  27.                 for z in range(0,len(currsub)):
  28.  
  29.  
  30.                         os.chdir(currsub[z])
  31.  
  32.                         print currsub[z]
  33.                         print "\n"
  34.  
  35.                         direntries = dircache.opendir('.') #open this directory
  36.  
  37.                         for j in range(0,len(direntries)):
  38.  
  39.                                 statresult = os.stat(direntries[j])
  40.  
  41.                                 if (stat.S_ISDIR(statresult.st_mode) != 0):
  42.  
  43.  
  44.                                         dlist.append(direntries[j])     #dlist is built here.
  45.                                         print direntries[j]
  46.  
  47.                                 else:
  48.  
  49.                                         print direntries[j]
  50.  
  51.                          dirrecur(dlist)   #recur again with full list of subdirectories.
  52.  
  53.  
  54.  
  55. #MAIN
  56.  
  57.  
  58. abspath = os.getcwd()
  59.  
  60. direntries = dircache.opendir(abspath)
  61.  
  62. dirlist = []
  63.  
  64.  
  65.  
  66. for i in range(0,len(direntries)):
  67.  
  68.  
  69.         statresult = os.stat(direntries[i])
  70.  
  71.         if (stat.S_ISDIR(statresult.st_mode) != 0): #if it's a directory file.
  72.  
  73.                 dirlist.append(direntries[i])
  74.                 print direntries[i]                 #print it out too.
  75.  
  76.         else:
  77.  
  78.                 print direntries[i]
  79.  
  80.  
  81.  
  82. dirrecur(dirlist)
  83.  
  84.  
  85.  
I think it would be simpler to use os.listdir(). Check out the thread HERE.

<Mod EDIT: Nice recovery below, BV. Admin is still trying to work the bugs out of code tags inside quotes. The weird thing is that it doesn't always happen. I've removed the quote here, rather than the code tags.>
Jul 16 '07 #5
bvdet
2,851 Expert Mod 2GB
Hi all,


I have spent hours trying to figure out where I have went wrong with my code for my recursive function to list all the files in a directory, and all of the files in all
of its subdirectories.





It goes down to level 3 of the directory tree, but at one point I get the
error 2, the no such file or directory message. I have checked the
file permissions of the directory that it gets stuck on, and I have no idea why
it is gettintg stuck on that directory and not one of the directories before it.

I would be extremely grateful if someone could lend another set of eyes to my
code and see where I'm going wrong at, and why it goes to level 3 so to speak
of the tree and then fails.

I'm stumped!


Thanks so much in advance. Any and all input would be greatly appreciated!
I think it would be simpler to use os.listdir(). Check out the thread HERE.
BV
Jul 16 '07 #6
If it helps any what I am looking for in my output,

I want the output to resemble the UNIX program ls -R as closely as possible.


I am running solaris 2.10.


I notice that when I use os.walk(), that each directory is showing files that
even ls -a won't show on those directories.

Are those files placed there by the system administrator?
Jul 16 '07 #7
bartonc
6,596 Expert 4TB
If it helps any what I am looking for in my output,

I want the output to resemble the UNIX program ls -R as closely as possible.


I am running solaris 2.10.


I notice that when I use os.walk(), that each directory is showing files that
even ls -a won't show on those directories.

Are those files placed there by the system administrator?
I'm afraid that you are dealing with a bunch of Windows users, here.
Not going to be much help on your output format, but, with regard to that, I do see a need to pass the recursion depth into the next iteration. Perhaps starting with a very simple, clean example of a recursive directory walker would help. I really like the simplicity of copytree() from the shutil module:
Expand|Select|Wrap|Line Numbers
  1.  
  2. def copytree(src, dst, symlinks=False):
  3.     """Recursively copy a directory tree using copy2().
  4.  
  5.     The destination directory must not already exist.
  6.     If exception(s) occur, an Error is raised with a list of reasons.
  7.  
  8.     If the optional symlinks flag is true, symbolic links in the
  9.     source tree result in symbolic links in the destination tree; if
  10.     it is false, the contents of the files pointed to by symbolic
  11.     links are copied.
  12.  
  13.     XXX Consider this example code rather than the ultimate tool.
  14.  
  15.     """
  16.     names = os.listdir(src)
  17.     os.mkdir(dst)
  18.     errors = []
  19.     for name in names:
  20.         srcname = os.path.join(src, name)
  21.         dstname = os.path.join(dst, name)
  22.         try:
  23.             if symlinks and os.path.islink(srcname):
  24.                 linkto = os.readlink(srcname)
  25.                 os.symlink(linkto, dstname)
  26.             elif os.path.isdir(srcname):
  27.                 copytree(srcname, dstname, symlinks)
  28.             else:
  29.                 copy2(srcname, dstname)
  30.             # XXX What about devices, sockets etc.?
  31.         except (IOError, os.error), why:
  32.             errors.append((srcname, dstname, why))
  33.         # catch the Error from the recursive copytree so that we can
  34.         # continue with other files
  35.         except Error, err:
  36.             errors.extend(err.args[0])
  37.     if errors:
  38.         raise Error, errors
So, what's that? Maybe 3 lines to get the name sorted out and 3 more to decide where or not to do the recursion.
Jul 17 '07 #8
ghostdog74
511 Expert 256MB
If it helps any what I am looking for in my output,

I want the output to resemble the UNIX program ls -R as closely as possible.


I am running solaris 2.10.


I notice that when I use os.walk(), that each directory is showing files that
even ls -a won't show on those directories.

Are those files placed there by the system administrator?
what does not show in ls -a that is shown in os.walk? here's a little walker for you
Expand|Select|Wrap|Line Numbers
  1. from os import listdir, sep
  2. from os.path import isdir
  3. def skywalker(dir):
  4.     for file in listdir(dir):
  5.         path = dir + sep + file        
  6.         if isdir(path):
  7.             print "path: ",path
  8.             skywalker(path)
  9.         else:
  10.             print '->' + file
  11. skywalker("/home/test")
  12.  
Jul 17 '07 #9
bvdet
2,851 Expert Mod 2GB
I have been using this function:
Expand|Select|Wrap|Line Numbers
  1. import os
  2.  
  3. def dirEntries(dir_name, subdir, *args):
  4.     '''Return a list of file names found in directory 'dir_name'
  5.     If 'subdir' is True, recursively access subdirectories under 'dir_name'.
  6.     Additional arguments, if any, are file extensions to match filenames. Matched
  7.         file names are added to the list.
  8.     If there are no additional arguments, all files found in the directory are
  9.         added to the list.
  10.     Example usage: fileList = dir_list(r'H:\TEMP', False, 'txt', 'py')
  11.         Only files with 'txt' and 'py' extensions will be added to the list.
  12.     Example usage: fileList = dir_list(r'H:\TEMP', True)
  13.         All files and all the files in subdirectories under H:\TEMP will be added
  14.         to the list.
  15.     '''
  16.     fileList = []
  17.     for file in os.listdir(dir_name):
  18.         dirfile = os.path.join(dir_name, file)
  19.         if os.path.isfile(dirfile):
  20.             if len(args) == 0:
  21.                 fileList.append(dirfile)
  22.             else:
  23.                 if os.path.splitext(dirfile)[1][1:] in args:
  24.                     fileList.append(dirfile)
  25.         # recursively access file names in subdirectories
  26.         elif os.path.isdir(dirfile) and subdir:
  27.             print "Accessing directory:", dirfile
  28.             fileList += dirEntries(dirfile, subdir, *args)
  29.     return fileList
Jul 17 '07 #10

Post your reply

Sign in to post your reply or Sign up for a free account.

Similar topics

4 posts views Thread by Ruby Tuesday | last post: by
9 posts views Thread by Penn Markham | last post: by
1 post views Thread by Antonio Lopez Arredondo | last post: by
1 post views Thread by Fatemeh | last post: by
3 posts views Thread by dibblm | last post: by
3 posts views Thread by Chris | last post: by
1 post views Thread by CARIGAR | last post: by
reply views Thread by zhoujie | last post: by
reply views Thread by suresh191 | last post: by
reply views Thread by listenups61195 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.