Sorting directory contents

[Wolfgang Draxinger]

H folks,

I got, hmm not really a problem, more a question of elegance:

In a current project I have to read in some files in a given
directory in chronological order, so that I can concatenate the
contents in those files into a new one (it's XML and I have to
concatenate some subelements, about 4 levels below the root
element). It all works, but somehow I got the feeling, that my
solution is not as elegant as it could be:

src_file_paths = dict()
for fname in os.listdir(sourcedir):
fpath = sourcedir+os.sep+fname
if not match_fname_pattern(fname): continue
src_file_paths[os.stat(fpath).st_mtime] = fpath
for ftime in src_file_paths.keys().sort():
read_and_concatenate(src_file_paths[ftime])

of course listdir and sorting could be done in a separate
function, but I wonder if there was a more elegant approach.

Wolfgang Draxinger
--
E-Mail address works, Jabber: he******@jabber.org, ICQ: 134682867

[Jussi Salmela]

Wolfgang Draxinger kirjoitti:

H folks,

I got, hmm not really a problem, more a question of elegance:

In a current project I have to read in some files in a given
directory in chronological order, so that I can concatenate the
contents in those files into a new one (it's XML and I have to
concatenate some subelements, about 4 levels below the root
element). It all works, but somehow I got the feeling, that my
solution is not as elegant as it could be:

src_file_paths = dict()
for fname in os.listdir(sourcedir):
fpath = sourcedir+os.sep+fname
if not match_fname_pattern(fname): continue
src_file_paths[os.stat(fpath).st_mtime] = fpath
for ftime in src_file_paths.keys().sort():
read_and_concatenate(src_file_paths[ftime])

of course listdir and sorting could be done in a separate
function, but I wonder if there was a more elegant approach.

Wolfgang Draxinger

I'm not claiming the following to be more elegant, but I would do it
like this (not tested!):

src_file_paths = dict()
prefix = sourcedir + os.sep
for fname in os.listdir(sourcedir):
if match_fname_pattern(fname):
fpath = prefix + fname
src_file_paths[os.stat(fpath).st_mtime] = fpath
for ftime in src_file_paths.keys().sort():
read_and_concatenate(src_file_paths[ftime])
Cheers,
Jussi

[Wolfgang Draxinger]

Jussi Salmela wrote:

I'm not claiming the following to be more elegant, but I would
do it like this (not tested!):

src_file_paths = dict()
prefix = sourcedir + os.sep
for fname in os.listdir(sourcedir):
if match_fname_pattern(fname):
fpath = prefix + fname
src_file_paths[os.stat(fpath).st_mtime] = fpath
for ftime in src_file_paths.keys().sort():
read_and_concatenate(src_file_paths[ftime])

Well, both versions, mine and yours won't work as it was written
down, as they neglegt the fact, that different files can have
the same st_mtime and that <listtype>.sort() doesn't return a
sorted list.

However this code works (tested) and behaves just like listdir,
only that it sorts files chronologically, then alphabetically.

def listdir_chrono(dirpath):
import os
files_dict = dict()
for fname in os.listdir(dirpath):
mtime = os.stat(dirpath+os.sep+fname).st_mtime
if not mtime in files_dict:
files_dict[mtime] = list()
files_dict[mtime].append(fname)

mtimes = files_dict.keys()
mtimes.sort()
filenames = list()
for mtime in mtimes:
fnames = files_dict[mtime]
fnames.sort()
for fname in fnames:
filenames.append(fname)
return filenames

Wolfgang Draxinger
--
E-Mail address works, Jabber: he******@jabber.org, ICQ: 134682867

[Larry Bates]

Wolfgang Draxinger wrote:

Jussi Salmela wrote:

>I'm not claiming the following to be more elegant, but I would
do it like this (not tested!):

src_file_paths = dict()
prefix = sourcedir + os.sep
for fname in os.listdir(sourcedir):
if match_fname_pattern(fname):
fpath = prefix + fname
src_file_paths[os.stat(fpath).st_mtime] = fpath
for ftime in src_file_paths.keys().sort():
read_and_concatenate(src_file_paths[ftime])

Well, both versions, mine and yours won't work as it was written
down, as they neglegt the fact, that different files can have
the same st_mtime and that <listtype>.sort() doesn't return a
sorted list.

However this code works (tested) and behaves just like listdir,
only that it sorts files chronologically, then alphabetically.

def listdir_chrono(dirpath):
import os
files_dict = dict()
for fname in os.listdir(dirpath):
mtime = os.stat(dirpath+os.sep+fname).st_mtime
if not mtime in files_dict:
files_dict[mtime] = list()
files_dict[mtime].append(fname)

mtimes = files_dict.keys()
mtimes.sort()
filenames = list()
for mtime in mtimes:
fnames = files_dict[mtime]
fnames.sort()
for fname in fnames:
filenames.append(fname)
return filenames

Wolfgang Draxinger

Four suggestions:

1) You might want to use os.path.join(dirpath, fname) instead of
dirpath+os.sep+fname.

2) You may be able to use glob.glob(<pattern>) to filter the files
more easily.

3) You didn't handle the possibility that there is s subdirectory
in the current directory. You need to check to make sure it is
a file you are processing as os.listdir() returns files AND
directories.

4) If you just put a tuple containing (mtime, filename) in a list
each time through the loop you can just sort that list at the
end it will be sorted by mtime and then alphabetically.

Example (not tested):

def listdir_chrono(dirpath):
import os
#
# Get a list of full pathnames for all the files in dirpath
# and exclude all the subdirectories. Note: This might be
# able to be replaced by glob.glob() to simplify. I would then
# add a second optional parameter: mask="" that would allow me
# to pass in a mask.
#
# List comprehensions are our friend when we are processing
# lists of things.
#
files=[os.path.join(dirpath, x) for x in os.listdir(dirpath)
if not os.path.isdir(os.path.join(dirpath, x)]

#
# Get a list of tuples that contain (mtime, filename) that
# I can sort.
#
flist=[(os.stat(x).st_mtime, x) for x in files]

#
# Sort them. Sort will sort on mtime, then on filename
#
flist.sort()
#
# Extract a list of the filenames only and return it
#
return [x[1] for x in flist]
#
# or if you only want the basenames of the files
#
#return [os.path.basename(x[1]) for x in flist]

-Larry Bates

[Paul Rubin]

Wolfgang Draxinger <wd********@darkstargames.dewrites:

src_file_paths = dict()
for fname in os.listdir(sourcedir):
fpath = sourcedir+os.sep+fname
if not match_fname_pattern(fname): continue
src_file_paths[os.stat(fpath).st_mtime] = fpath
for ftime in src_file_paths.keys().sort():
read_and_concatenate(src_file_paths[ftime])

Note you have to used sorted() and not .sort() to get back a value
that you can iterate through.

Untested:

from itertools import ifilter

goodfiles = ifilter(match_fname_pattern,
(sourcedir+os.sep+fname for \
fname in os.listdir(sourcedir))

for f,t in sorted((fname,os.stat(f).st_mtime) for fname in goodfiles,
key=lambda (fname,ftime): ftime):
read_and_concatenate(f)

If you're a lambda-phobe you can use operator.itemgetter(1) instead of
the lambda. Obviously you don't need the separate goodfiles variable
but things get a bit deeply nested without it.

[Jussi Salmela]

Wolfgang Draxinger kirjoitti:

Jussi Salmela wrote:

>I'm not claiming the following to be more elegant, but I would
do it like this (not tested!):

src_file_paths = dict()
prefix = sourcedir + os.sep
for fname in os.listdir(sourcedir):
if match_fname_pattern(fname):
fpath = prefix + fname
src_file_paths[os.stat(fpath).st_mtime] = fpath
for ftime in src_file_paths.keys().sort():
read_and_concatenate(src_file_paths[ftime])

Well, both versions, mine and yours won't work as it was written
down, as they neglegt the fact, that different files can have
the same st_mtime and that <listtype>.sort() doesn't return a
sorted list.

However this code works (tested) and behaves just like listdir,
only that it sorts files chronologically, then alphabetically.

def listdir_chrono(dirpath):
import os
files_dict = dict()
for fname in os.listdir(dirpath):
mtime = os.stat(dirpath+os.sep+fname).st_mtime
if not mtime in files_dict:
files_dict[mtime] = list()
files_dict[mtime].append(fname)

mtimes = files_dict.keys()
mtimes.sort()
filenames = list()
for mtime in mtimes:
fnames = files_dict[mtime]
fnames.sort()
for fname in fnames:
filenames.append(fname)
return filenames

Wolfgang Draxinger

More elegant or not ... I did it MY WAYYYY!!! (and tested this time
really carefully;)):

#-------------------------------
def listdir_chrono_2(dirpath):
import os
files_dict = {}
prefix = dirpath + os.sep
for fname in os.listdir(dirpath):
mtime = os.stat(prefix + fname).st_mtime
files_dict.setdefault(mtime, []).append(fname)

mtimes = sorted(files_dict.keys())
filenames = []
for mtime in mtimes:
filenames += sorted(files_dict[mtime])
return filenames

firstLst = listdir_chrono('.')
secondLst = listdir_chrono_2('.')
if firstLst == secondLst: print 'OK'
else: print 'ERROR!!!'
#-------------------------------

I keep taking the "dirpath + os.sep" part out of the loop because it is
a loop invariant and doesn't have to be inside the loop.

Cheer

[Rob Wolfe]

Wolfgang Draxinger wrote:

However this code works (tested) and behaves just like listdir,
only that it sorts files chronologically, then alphabetically.

def listdir_chrono(dirpath):
import os
files_dict = dict()
for fname in os.listdir(dirpath):
mtime = os.stat(dirpath+os.sep+fname).st_mtime
if not mtime in files_dict:
files_dict[mtime] = list()
files_dict[mtime].append(fname)

mtimes = files_dict.keys()
mtimes.sort()
filenames = list()
for mtime in mtimes:
fnames = files_dict[mtime]
fnames.sort()
for fname in fnames:
filenames.append(fname)
return filenames

Using the builtin functions `sorted`, `filter` and the `setdefault`
method of dictionary could a little shorten your code:

def listdir_chrono(dirpath):
import os
files_dict = {}
for fname in filter(os.path.isfile, os.listdir(dirpath)):
mtime = os.stat(os.path.join(dirpath, fname)).st_mtime
files_dict.setdefault(mtime, []).append(fname)

filenames = []
for mtime in sorted(files_dict):
for fname in sorted(files_dict[mtime]):
filenames.append(fname)
return filenames

--
HTH,
Rob

[Wolfgang Draxinger]

Larry Bates wrote:

3) You didn't handle the possibility that there is s
subdirectory
in the current directory. You need to check to make sure it
is a file you are processing as os.listdir() returns files
AND directories.

Well, the directory the files are in is not supposed to have any
subdirectories.

4) If you just put a tuple containing (mtime, filename) in a
list
each time through the loop you can just sort that list at
the end it will be sorted by mtime and then alphabetically.

Ah, of course. Hmm, seems I was short of caffeine when I hacked
my code :-P

def listdir_chrono(dirpath):
import os
#
# Get a list of full pathnames for all the files in dirpath
# and exclude all the subdirectories. Note: This might be
# able to be replaced by glob.glob() to simplify. I would
# then add a second optional parameter: mask="" that would
# allow me to pass in a mask.
#
# List comprehensions are our friend when we are processing
# lists of things.
#
files=[os.path.join(dirpath, x) for x in
os.listdir(dirpath)
if not os.path.isdir(os.path.join(dirpath, x)]

#
# Get a list of tuples that contain (mtime, filename) that
# I can sort.
#
flist=[(os.stat(x).st_mtime, x) for x in files]

#
# Sort them. Sort will sort on mtime, then on filename
#
flist.sort()
#
# Extract a list of the filenames only and return it
#
return [x[1] for x in flist]
#
# or if you only want the basenames of the files
#
#return [os.path.basename(x[1]) for x in flist]

Now, THAT is elegant.

Wolfgang Draxinger
--
E-Mail address works, Jabber: he******@jabber.org, ICQ: 134682867

[Peter Otten]

Wolfgang Draxinger wrote:

I got, hmm not really a problem, more a question of elegance:

In a current project I have to read in some files in a given
directory in chronological order, so that I can concatenate the
contents in those files into a new one (it's XML and I have to
concatenate some subelements, about 4 levels below the root
element). It all works, but somehow I got the feeling, that my
solution is not as elegant as it could be:

src_file_paths = dict()
for fname in os.listdir(sourcedir):
fpath = sourcedir+os.sep+fname
if not match_fname_pattern(fname): continue
src_file_paths[os.stat(fpath).st_mtime] = fpath
for ftime in src_file_paths.keys().sort():
read_and_concatenate(src_file_paths[ftime])

of course listdir and sorting could be done in a separate
function, but I wonder if there was a more elegant approach.

If glob.glob() is good enough to replace your custom match_fname_pattern()
you can save a few steps:

pattern = os.path.join(sourcedir, "*.xml")
files = glob.glob(pattern)
files.sort(key=os.path.getmtime)
for fn in files:
read_and_concatenate(fn)

Peter

[Jussi Salmela]

Larry Bates kirjoitti:

Wolfgang Draxinger wrote:

>Jussi Salmela wrote:

>>I'm not claiming the following to be more elegant, but I would
do it like this (not tested!):

src_file_paths = dict()
prefix = sourcedir + os.sep
for fname in os.listdir(sourcedir):
if match_fname_pattern(fname):
fpath = prefix + fname
src_file_paths[os.stat(fpath).st_mtime] = fpath
for ftime in src_file_paths.keys().sort():
read_and_concatenate(src_file_paths[ftime])

Well, both versions, mine and yours won't work as it was written
down, as they neglegt the fact, that different files can have
the same st_mtime and that <listtype>.sort() doesn't return a
sorted list.

However this code works (tested) and behaves just like listdir,
only that it sorts files chronologically, then alphabetically.

def listdir_chrono(dirpath):
import os
files_dict = dict()
for fname in os.listdir(dirpath):
mtime = os.stat(dirpath+os.sep+fname).st_mtime
if not mtime in files_dict:
files_dict[mtime] = list()
files_dict[mtime].append(fname)

mtimes = files_dict.keys()
mtimes.sort()
filenames = list()
for mtime in mtimes:
fnames = files_dict[mtime]
fnames.sort()
for fname in fnames:
filenames.append(fname)
return filenames

Wolfgang Draxinger

Four suggestions:

1) You might want to use os.path.join(dirpath, fname) instead of
dirpath+os.sep+fname.

2) You may be able to use glob.glob(<pattern>) to filter the files
more easily.

3) You didn't handle the possibility that there is s subdirectory
in the current directory. You need to check to make sure it is
a file you are processing as os.listdir() returns files AND
directories.

4) If you just put a tuple containing (mtime, filename) in a list
each time through the loop you can just sort that list at the
end it will be sorted by mtime and then alphabetically.

Example (not tested):

def listdir_chrono(dirpath):
import os
#
# Get a list of full pathnames for all the files in dirpath
# and exclude all the subdirectories. Note: This might be
# able to be replaced by glob.glob() to simplify. I would then
# add a second optional parameter: mask="" that would allow me
# to pass in a mask.
#
# List comprehensions are our friend when we are processing
# lists of things.
#
files=[os.path.join(dirpath, x) for x in os.listdir(dirpath)
if not os.path.isdir(os.path.join(dirpath, x)]

#
# Get a list of tuples that contain (mtime, filename) that
# I can sort.
#
flist=[(os.stat(x).st_mtime, x) for x in files]

#
# Sort them. Sort will sort on mtime, then on filename
#
flist.sort()
#
# Extract a list of the filenames only and return it
#
return [x[1] for x in flist]
#
# or if you only want the basenames of the files
#
#return [os.path.basename(x[1]) for x in flist]

-Larry Bates

And as in Peter Ottens glob.glob variation, this shortens considerably
by using sort with key instead of a separate list flist:

files.sort(key=lambda x:(os.stat(x).st_mtime, x))

Cheers,
Jussi