Walk thru each subdirectory from a top directory

[silverburgh.meryl]

i am trying to use python to walk thru each subdirectory from a top
directory. Here is my script:

savedPagesDirectory = "/home/meryl/saved_pages/data"

dir=open(savedPagesDirectory, 'r')

for file in dir:
if (isdir(file)):
# get the full path of the file
fileName = savedPagesDirectory + file + 'index.html'
print fileName

$ ./scripts/regressionTest.py
Traceback (most recent call last):
File "./scripts/regressionTest.py", line 12, in ?
dir=open(savedPagesDirectory, 'r')
IOError: [Errno 21] Is a directory

But I get the above error:

Can you please tell me what did I do wrong?

Thank you.

[Diez B. Roggisch]

si***************@gmail.com schrieb:

i am trying to use python to walk thru each subdirectory from a top
directory. Here is my script:

savedPagesDirectory = "/home/meryl/saved_pages/data"

dir=open(savedPagesDirectory, 'r')

for file in dir:
if (isdir(file)):
# get the full path of the file
fileName = savedPagesDirectory + file + 'index.html'
print fileName

$ ./scripts/regressionTest.py
Traceback (most recent call last):
File "./scripts/regressionTest.py", line 12, in ?
dir=open(savedPagesDirectory, 'r')
IOError: [Errno 21] Is a directory

But I get the above error:

Can you please tell me what did I do wrong?

You can't open a directory.

Use the function os.walk to do what you want.

Diez

[Adam]

On Feb 26, 9:28 pm, [e-mail] wrote:

i am trying to use python to walk thru each subdirectory from a top
directory. Here is my script:

savedPagesDirectory = "/home/meryl/saved_pages/data"

dir=open(savedPagesDirectory, 'r')

for file in dir:
if (isdir(file)):
# get the full path of the file
fileName = savedPagesDirectory + file + 'index.html'
print fileName

$ ./scripts/regressionTest.py
Traceback (most recent call last):
File "./scripts/regressionTest.py", line 12, in ?
dir=open(savedPagesDirectory, 'r')
IOError: [Errno 21] Is a directory

But I get the above error:

Can you please tell me what did I do wrong?

Thank you.

>From Alan Gaulds Tut.

>>for t in os.walk('Root'):

.... print t
....
('Root', ['D1', 'D2', 'D3'], ['FA.txt', 'FB.txt'])
('Root/D1', ['D1-1'], ['FC.txt'])
('Root/D1/D1-1', [], ['FF.txt'])
('Root/D2', [], ['FD.txt'])
('Root/D3', ['D3-1'], ['FE.txt'])
('Root/D3/D3-1', [], ['target.txt'])

>>>

This bit below is from one of my first programs as I'm currently
learning. It is designed to go form the root down and return the full
paths of everything it finds into a list. (I then check the reults for
files paths that exceed a certain length - but you don't need to know
that.)
def findallfiles(self, base):
self.results = []
for root,dirs,files in os.walk(base):
os.chdir(root)
self.scan = glob.glob("*")
for r in self.scan:
if root[-1] == "\\":
self.results.append(root + r)
else:
self.results.append(root + "\\" + r)
return self.results

[Adam]

Whoops, the first bit of my reply ended up in the quoted text. See
above.

Adam

[Scott David Daniels]

Adam wrote:

On Feb 26, 9:28 pm, [e-mail] wrote:

>i am trying to use python to walk thru each subdirectory from a top
directory. Here is my script: ....

This bit below is from one of my first programs as I'm currently
learning. It is designed to go form the root down and return the full
paths of everything it finds into a list. (I then check the reults for
files paths that exceed a certain length - but you don't need to know
that.)
def findallfiles(self, base):
self.results = []
for root,dirs,files in os.walk(base):
os.chdir(root)

^^^ Mistake here, don't change directories during os.walk ^^^

self.scan = glob.glob("*")
for r in self.scan:
if root[-1] == "\\":
self.results.append(root + r)
else:
self.results.append(root + "\\" + r)
return self.results

def produce_all_files(base):
for root, dirs, files in os.walk(base):
for r in files:
yield os.path.join(root, r)
### possibly also (but I'd only go for files)
#for r in dirs:
# yield os.path.join(root, r)

def findallfiles(base):
return list(produce_all_files(base))

--
--Scott David Daniels
sc***********@acm.org

[Steven D'Aprano]

On Tue, 27 Feb 2007 20:31:43 -0800, Scott David Daniels wrote:

> def findallfiles(self, base):
self.results = []
for root,dirs,files in os.walk(base):
os.chdir(root)

^^^ Mistake here, don't change directories during os.walk ^^^

I came across this problem some time ago. I had to walk a directory tree,
calling an external program on each file. Unfortunately, that external
program wrote directly to the current working directory, which caused all
sorts of havoc. This is how I dealt with it:
def unbin(where):
"""Walk through a directory tree, calling macunpack to extract the
contents of MacBinary files.
"""
def _unbin(data, dirname, files):
for oldname in files:
fullname = os.path.normpath(os.path.join(dirname, oldname))
if os.path.isfile(fullname):
# Dammit, macunpack writes directly to the current
# working directory. Changing the cwd breaks the file
# tree walker, so we have to remember the current
# directory, change it to where we want to be, then
# change it back.
wd = os.getcwd()
os.chdir(dirname)
result = os.system('macunpack -f "%s"' % oldname)
if result == 0:
# Unpacking worked, so delete the original.
os.remove(oldname)
os.chdir(wd) # sigh...

os.path.walk(where, _unbin, None)
Is there another (better) way of dealing with this sort of situation?
--
Steven D'Aprano

[Steven D'Aprano]

On Tue, 27 Feb 2007 06:22:51 +0000, Dennis Lee Bieber wrote:

[snip 350-odd file names]

Just a sample of the start of a very long listing...

Thanks for sharing. What's with the nude ferret?

--
Steven D'Aprano

[Peter Otten]

Steven D'Aprano wrote:

On Tue, 27 Feb 2007 20:31:43 -0800, Scott David Daniels wrote:

>> def findallfiles(self, base):
self.results = []
for root,dirs,files in os.walk(base):
os.chdir(root)

^^^ Mistake here, don't change directories during os.walk ^^^

I came across this problem some time ago. I had to walk a directory tree,
calling an external program on each file. Unfortunately, that external
program wrote directly to the current working directory, which caused all
sorts of havoc. This is how I dealt with it:
def unbin(where):
"""Walk through a directory tree, calling macunpack to extract the
contents of MacBinary files.
"""
def _unbin(data, dirname, files):
for oldname in files:
fullname = os.path.normpath(os.path.join(dirname, oldname))
if os.path.isfile(fullname):
# Dammit, macunpack writes directly to the current
# working directory. Changing the cwd breaks the file
# tree walker, so we have to remember the current
# directory, change it to where we want to be, then
# change it back.
wd = os.getcwd()
os.chdir(dirname)
result = os.system('macunpack -f "%s"' % oldname)
if result == 0:
# Unpacking worked, so delete the original.
os.remove(oldname)
os.chdir(wd) # sigh...

os.path.walk(where, _unbin, None)
Is there another (better) way of dealing with this sort of situation?

Does the problem occur if you pass an absolute path to
os.walk()/os.path.walk()?

Peter

[Steven D'Aprano]

On Wed, 28 Feb 2007 08:38:41 +0100, Peter Otten wrote:

Does the problem occur if you pass an absolute path to
os.walk()/os.path.walk()?

Well, arguably the problem is that macunpack insists on writing to the
current working directory.

Or the problem is that os.walk sets the working directory to the initial
argument when you start, and then keeps it until it finishes. (Arguably
that's the right behaviour.)

You can play around with this to check:

def walker(where):
"""Walk through a directory tree, printing the current working directory."
def _walk(data, dirname, files):
print "dirname = '%s'; wd = '%s'" % (dirname, os.getcwd())
os.path.walk(where, _walk, None)
For those times when os.walk's behaviour doesn't mesh well with that of
the external program you are calling (like macunpack) is there an
alternative to:

- save the cwd;
- change directories;
- call the program;
- return to the saved directory

?

--
Steven D'Aprano

[Peter Otten]

Steven D'Aprano wrote:

For those times when os.walk's behaviour doesn't mesh well with that of
the external program you are calling (like macunpack) is there an
alternative to:

- save the cwd;
- change directories;
- call the program;
- return to the saved directory

?

os.walk() creates paths from the root you provide and the files/direcotories
it finds via os.listdir(), and uses that to determine. If that root is an
absolute path any such paths are absolute, too, and undoing the directory
change is not necessary.

cwd = os.getcwd()
for path, dirs, files in os.walk(os.path.abspath(root)):
os.chdir(path)
for file in files:
invoke external program
os.chdir(cwd)

That said, undoing the change immediately is better style, and probably a
good usecase for a with statement like

with current_working_directory(path):
invoke external program

Last minute idea:

for path, dirs, files in os.walk(root):
for file in files:
os.system("cd '%s' && macunpack '%s'" % (path, file))

Peter