By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
439,930 Members | 2,018 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 439,930 IT Pros & Developers. It's quick & easy.

Is there an alternative to os.walk?

P: n/a
Hi all,
I have a question about traversing file systems, and could use some
help. Because of directories with many files in them, os.walk appears
to be rather slow. I`m thinking there is a potential for speed-up since
I don`t need os.walk to report filenames of all the files in every
directory it visits. Is there some clever way to use os.walk or another
tool that would provide functionality like os.walk except for the
listing of the filenames?

Oct 4 '06 #1
Share this Question
Share on Google+
6 Replies


P: n/a
Bruce wrote:
Hi all,
I have a question about traversing file systems, and could use some
help. Because of directories with many files in them, os.walk appears
to be rather slow.
Provide more info/code. I suspect it is not os.walk itself that is slow,
but rather the code that processes its result...
I`m thinking there is a potential for speed-up since
I don`t need os.walk to report filenames of all the files in every
directory it visits. Is there some clever way to use os.walk or another
tool that would provide functionality like os.walk except for the
listing of the filenames?
You may want to take a look at os.path.walk then.

--Irmen
Oct 4 '06 #2

P: n/a
Bruce wrote:
Hi all,
I have a question about traversing file systems, and could use some
help. Because of directories with many files in them, os.walk appears
to be rather slow. I`m thinking there is a potential for speed-up since
I don`t need os.walk to report filenames of all the files in every
directory it visits. Is there some clever way to use os.walk or another
tool that would provide functionality like os.walk except for the
listing of the filenames?
You might want to check out the path module [1] (not os.path). The
following is from the docs:
The method path.walk() returns an iterator which steps recursively
through a whole directory tree. path.walkdirs() and path.walkfiles()
are the same, but they yield only the directories and only the files,
respectively.
Oh, and you can thank Paul Bissex for pointing me to path [2].

[1]: http://www.jorendorff.com/articles/python/path/
[2]: http://e-scribe.com/news/289

Oct 4 '06 #3

P: n/a
waylan wrote:
Bruce wrote:
Hi all,
I have a question about traversing file systems, and could use some
help. Because of directories with many files in them, os.walk appears
to be rather slow. I`m thinking there is a potential for speed-up since
I don`t need os.walk to report filenames of all the files in every
directory it visits. Is there some clever way to use os.walk or another
tool that would provide functionality like os.walk except for the
listing of the filenames?

You might want to check out the path module [1] (not os.path). The
following is from the docs:
The method path.walk() returns an iterator which steps recursively
through a whole directory tree. path.walkdirs() and path.walkfiles()
are the same, but they yield only the directories and only the files,
respectively.

Oh, and you can thank Paul Bissex for pointing me to path [2].
[1]: http://www.jorendorff.com/articles/python/path/
[2]: http://e-scribe.com/news/289
A little late but.. thanks for the replies, was very useful. Here`s
what I do in this case:

def search(a_dir):
valid_dirs = []
walker = os.walk(a_dir)
while 1:
try:
dirpath, dirnames, filenames = walker.next()
except StopIteration:
break
if dirtest(dirpath,filenames):
valid_dirs.append(dirpath)
return valid_dirs

def dirtest(a_dir):
testfiles = ['a','b','c']
for f in testfiles:
if not os.path.exists(os.path.join(a_dir,f)):
return 0
return 1

I think you`re right - it`s not os.walk that makes this slow, it`s the
dirtest method that takes so much more time when there are many files
in a directory. Also, thanks for pointing me to the path module, was
interesting.

Oct 7 '06 #4

P: n/a
"Bruce" <ep****@gmail.comwrote:
>
A little late but.. thanks for the replies, was very useful. Here`s
what I do in this case:

def search(a_dir):
valid_dirs = []
walker = os.walk(a_dir)
while 1:
try:
dirpath, dirnames, filenames = walker.next()
except StopIteration:
break
if dirtest(dirpath,filenames):
valid_dirs.append(dirpath)
return valid_dirs

def dirtest(a_dir):
testfiles = ['a','b','c']
for f in testfiles:
if not os.path.exists(os.path.join(a_dir,f)):
return 0
return 1

I think you`re right - it`s not os.walk that makes this slow, it`s the
dirtest method that takes so much more time when there are many files
in a directory. Also, thanks for pointing me to the path module, was
interesting.
Umm, may I point out that you don't NEED the "os.path.exists" call, because
you are already being HANDED a list of all the filenames in that directory?
You could "dirtest" with this much faster routinee:

def dirtest(a_dir,filenames):
for f in ['a','b','c']:
if not f in filenames:
return 0
return 1
--
- Tim Roberts, ti**@probo.com
Providenza & Boekelheide, Inc.
Oct 8 '06 #5

P: n/a
On 10/8/06, Tim Roberts <ti**@probo.comwrote:
Umm, may I point out that you don't NEED the "os.path.exists" call, because
you are already being HANDED a list of all the filenames in that directory?
You could "dirtest" with this much faster routinee:

def dirtest(a_dir,filenames):
for f in ['a','b','c']:
if not f in filenames:
return 0
return 1
Or False / True for sufficiently new versions of Python. :)

-- Theerasak
Oct 8 '06 #6

P: n/a
Ant
The idiomatic way of doing the tree traversal is:

def search(a_dir):
valid_dirs = []
for dirpath, dirnames, filenames in os.walk(a_dir):
if dirtest(filenames):
valid_dirs.append(dirpath)
return valid_dirs

Also since you are given a list of filenames in the directory, then why
not just check the list of those files for your test files:

def dirtest(filenames):
testfiles = ['a','b','c']
for f in testfiles:
if not f in filenames:
return False
return False

You'd have to test this to see if it made a difference in performance,
but it makes for more readable code

Oct 8 '06 #7

This discussion thread is closed

Replies have been disabled for this discussion.