472,378 Members | 1,487 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,378 software developers and data experts.

PEP ? os.listdir enhancement

Hi,
I noticed that when I use os.listdir I need to work with absolute paths
90% of times.
While I can use a for cycle, I'd prefere to use a list comprehension,
but it becomes too long.

I propose to add an 'abs' keyword which would make os.listdir return the
absolute path of files instead of a relative path.
This would bring only advantages, I think.

I show two examples

from os import listdir
from os.path import isdir,join,getsize,isfile

### e.g. 1 part 1 - getting a list of directories ###
dirs=[]
for i in os.listdir(path):
tmp_path=os.path.join(path,i)
if os.path.isdir(tmp_path):
dirs.append(tmp_path)

### e.g. 1 part 2 ###
dirs=[join(path,x) for x in listdir(path) if isdir(join(path,x))]

here the list comprehension is still clear, but only because we have
direct references to join and friends. moreover whe need to use join twice
for each directory.

### e.g. 2 part 1 - getting a list of (file,size) tuples ###
path_size=[]
for i in os.listdir(path):
tmp_path=os.path.join(path,i)
if os.path.isfile(tmp_path):
path_size.append((tmp_path,getsize(tmp_path))

### e.g. 2 part 2 ###
dirs=[(join(path,x),getsize(join(path,x)) for x in listdir(path) if
isfile(join(path,x))]
now list comprehension is way too long, and in the worst case we must use
join 3 times for each iteration.

adding an 'abs' keyword to os.listdir would give benefits both to for
cycle and list comprehensions.
for cycle would lose the tmp_path assignment and list comprehensions ...
### e.g. 1 part 2 bis ###
dirs=[x for x in listdir(path,abs=True) if isdir(x)]

here we gain clearness and speed.

### e.g. 2 part 2 bis ###
dirs=[(x,getsize(x)) for x in listdir(path,abs=True) if isfile(x)]

and here we gain clearness, speed and a truely _usable_ list comprehension

What do you think about this ?

Thanks for reading,
Riccardo

--
Riccardo Galli
Sideralis Programs
http://www.sideralis.net
Jul 19 '05 #1
15 2478
Why not just define the function yourself? Not every 3-line function
needs to be built in.

def listdir_joined(path):
return [os.path.join(path, entry) for entry in os.listdir(path)]

dirs = [x for x in listdir_joined(path) if os.path.isdir(x)]

path_size = [(x, getsize(x)) for x in listdir_joined(path) if os.path.isfile(x)]
Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQFCuZFYJd01MZaTXX0RAqz2AKCozk1iJjrTwFRv4up5Nd eVXqUNUQCeKniG
Q4Sl42Sp6dMnMFL0u607gwI=
=QEy2
-----END PGP SIGNATURE-----

Jul 19 '05 #2
Riccardo Galli wrote:
I noticed that when I use os.listdir I need to work with absolute paths
90% of times.
While I can use a for cycle, I'd prefere to use a list comprehension,
but it becomes too long.

### e.g. 1 part 1 - getting a list of directories ###
dirs=[]
for i in os.listdir(path):
tmp_path=os.path.join(path,i)
if os.path.isdir(tmp_path):
dirs.append(tmp_path)

### e.g. 1 part 2 ###
dirs=[join(path,x) for x in listdir(path) if isdir(join(path,x))]


Using Jason Orendorff's "path" module, all this code basically collapses
down to this beauty (with your variable "path" renamed to myPath to
avoid a name collision):

from path import path
dirs = path(myPath).abspath().dirs()

-Peter
Jul 19 '05 #3
Peter Hansen wrote:
Using Jason Orendorff's "path" module, all this code basically collapses
down to this beauty (with your variable "path" renamed to myPath to
avoid a name collision):


This has to be the non-stdlib library I use the most. It's a great module.
--
Michael Hoffman
Jul 19 '05 #4
On Wed, 22 Jun 2005 11:27:06 -0500, Jeff Epler wrote:
Why not just define the function yourself? Not every 3-line function
needs to be built in.


Of course I can code such a function, and I agree with the second
sentence, but I think that obtaining absolutes path is a task so commonly
needed that adding a keyword to an existing function would give a plus to
the library without adding complexity (speaking of number of modules).

Usually when you use os.listdir do you end using os.path.join to obtain
absolutes path? I'm interested to know if the task is so common as I
think, or if I'm wrong.

Thank you,
Riccardo
--
Riccardo Galli
Sideralis Programs
http://www.sideralis.net
Jul 19 '05 #5
What's wrong with

(os.path.join(d, x) for x in os.listdir(d))

It's short, and easier to understand then some obscure option ;)

Andreas

On Thu, Jun 23, 2005 at 11:05:57AM +0200, Riccardo Galli wrote:
On Wed, 22 Jun 2005 11:27:06 -0500, Jeff Epler wrote:
Why not just define the function yourself? Not every 3-line function
needs to be built in.


Of course I can code such a function, and I agree with the second
sentence, but I think that obtaining absolutes path is a task so commonly
needed that adding a keyword to an existing function would give a plus to
the library without adding complexity (speaking of number of modules).

Usually when you use os.listdir do you end using os.path.join to obtain
absolutes path? I'm interested to know if the task is so common as I
think, or if I'm wrong.

Thank you,
Riccardo
--
Riccardo Galli
Sideralis Programs
http://www.sideralis.net
--
http://mail.python.org/mailman/listinfo/python-list

Jul 19 '05 #6
On 6/22/05, Riccardo Galli <riccardo_cut1@cut2_sideralis.net> wrote:
I propose to add an 'abs' keyword which would make os.listdir return the
absolute path of files instead of a relative path.


What about os.listdir(dir='relative/path', abs=True)? Should listdir
call abspath on results? Should we add another keyword rel? Would it
complicate listdir unnecessarily?

- kv
Jul 19 '05 #7
On Thu, 23 Jun 2005 11:34:02 +0200, Andreas Kostyrka wrote:
What's wrong with

(os.path.join(d, x) for x in os.listdir(d))

It's short, and easier to understand then some obscure option ;)

Andreas


how does it help in using list comprehension, as the ones in the first
post?
--
Riccardo Galli
Sideralis Programs
http://www.sideralis.net
Jul 19 '05 #8
On Thu, 23 Jun 2005 12:56:08 +0300, Konstantin Veretennicov wrote:
On 6/22/05, Riccardo Galli <riccardo_cut1@cut2_sideralis.net> wrote:
I propose to add an 'abs' keyword which would make os.listdir return the
absolute path of files instead of a relative path.


What about os.listdir(dir='relative/path', abs=True)? Should listdir call
abspath on results? Should we add another keyword rel? Would it complicate
listdir unnecessarily?

- kv


keyword dir not exists (don't know if you added as example or not) and
abs set to true would return abspath on result. What else could it do ?
--
Riccardo Galli
Sideralis Programs
http://www.sideralis.net
Jul 19 '05 #9
Riccardo Galli wrote:
On Thu, 23 Jun 2005 11:34:02 +0200, Andreas Kostyrka wrote:

What's wrong with

(os.path.join(d, x) for x in os.listdir(d))

It's short, and easier to understand then some obscure option ;)

Andreas

how does it help in using list comprehension, as the ones in the first
post?


You can nest list comprehension
[ e
for e in (os.path.join(d, x) for x in os.listdir(d))
if os.path.isdir (e)]

You might also want to look at module itertools, which has better
support for transforming and filtering in multiple steps.

Daniel
Jul 19 '05 #10
Riccardo Galli wrote:
On Thu, 23 Jun 2005 12:56:08 +0300, Konstantin Veretennicov wrote:
What about os.listdir(dir='relative/path', abs=True)? Should listdir call
abspath on results? Should we add another keyword rel? Would it complicate
listdir unnecessarily?


keyword dir not exists (don't know if you added as example or not) and
abs set to true would return abspath on result. What else could it do ?


He probably meant that a 'join' option would be more natural than an
'abs' option. After all, your examples use os.path.join to create a
valid path that can be used as the argument to other module os
functions. Whether the results are absolute or relative should depend on
the initial argument to os.listdir.

Daniel
Jul 19 '05 #11
Konstantin Veretennicov wrote:
On 6/22/05, Riccardo Galli <riccardo_cut1@cut2_sideralis.net> wrote:
I propose to add an 'abs' keyword which would make os.listdir return the
absolute path of files instead of a relative path.


What about os.listdir(dir='relative/path', abs=True)? Should listdir
call abspath on results? Should we add another keyword rel? Would it
complicate listdir unnecessarily?

- kv


I'm in favour of Riccardo's suggestion, but I think he's got the name of the
keyword wrong. The signature should be

listdir(path, with_path=False)

and cater absolute and relative paths alike.

The need for such an enhancement is not particularly pressing, as
workarounds abound. Here's another one:
glob.glob("/usr/lib/games/../games/*")

['/usr/lib/games/../games/schwarzerpeter']
Peter

Jul 19 '05 #12
On Thu, 23 Jun 2005 13:25:06 +0200, Daniel Dittmar wrote:
He probably meant that a 'join' option would be more natural than an 'abs'
option. After all, your examples use os.path.join to create a valid path
that can be used as the argument to other module os functions. Whether the
results are absolute or relative should depend on the initial argument to
os.listdir.

Daniel


I got the point, and you're right. I didn't though about that and 'abs' as
keyword becomes nonsense. Needing a more general kewyword, as pointed out
by Peter Otten
--
Riccardo Galli
Sideralis Programs
http://www.sideralis.net
Jul 19 '05 #13
Am Wed, 22 Jun 2005 17:57:14 +0200 schrieb Riccardo Galli:
Hi,
I noticed that when I use os.listdir I need to work with absolute paths
90% of times.
While I can use a for cycle, I'd prefere to use a list comprehension,
but it becomes too long.


Hi,

I like it. But as you noticed, too, "join" would be better than "abs".

Example:

# mylistdir.py
import os
import sys

def mylistdir(dir, join=False):
for file in os.listdir(dir):
yield os.path.join(dir, file)
print list(mylistdir(sys.argv[1]))
Thomas

--
Thomas GŁttler, http://www.thomas-guettler.de/
Jul 19 '05 #14
Hi All--

Thomas Guettler wrote:

I like it. But as you noticed, too, "join" would be better than "abs".

Example:

# mylistdir.py
import os
import sys

def mylistdir(dir, join=False):
for file in os.listdir(dir):
yield os.path.join(dir, file)

print list(mylistdir(sys.argv[1]))


Mmmm, how about:

# mylistdir.py
import os, os.path
import sys

def mylistdir(dir, join=False):
for file in os.listdir(dir):
if join:
yield join(dir, file)
else:
yield file

print list(mylistdir(sys.argv[1]))

or

print list(mylistdir(sys.argv[1],os.path.join))

That way I could def my own join and call it as

print list(mylistdir(sys.argv[1],myjoin))

(Note that in your version the join argument isn't used at all.)

Metta,
Ivan
----------------------------------------------
Ivan Van Laningham
God N Locomotive Works
http://www.andi-holmes.com/
http://www.foretec.com/python/worksh...oceedings.html
Army Signal Corps: Cu Chi, Class of '70
Author: Teach Yourself Python in 24 Hours
Jul 19 '05 #15
On Thu, 23 Jun 2005 09:21:55 -0600, Ivan Van Laningham wrote:
Mmmm, how about:

# mylistdir.py
import os, os.path
import sys

def mylistdir(dir, join=False):
for file in os.listdir(dir):
if join:
yield join(dir, file)
else:
yield file

print list(mylistdir(sys.argv[1]))

or

print list(mylistdir(sys.argv[1],os.path.join))

That way I could def my own join and call it as

print list(mylistdir(sys.argv[1],myjoin))


I think that the implementation of listdir is done in C, so the
functionality would be added in C too.
by the way, I think 'join' (cute keyword) should be a boolean and not a
callable: the obvious way is that we join path using os.path.join, and not
some sophisticated function. With a boolean keyword code is smaller and if
we want to use our special join function we can do it simply.

e.g
def func(dir,join=False):
return (join and join(dir,x) or x for x in os.listdir(dir))

os.listdir is actually supposed not to be a generator, like you suggested.
Are there known future changes ?

Bye,
Riccardo

--
Riccardo Galli
Sideralis Programs
http://www.sideralis.net
Jul 19 '05 #16

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

11
by: Jason Kratz | last post by:
OK. I've search on google groups and around the web for this and I haven't found an answer. I'm a Python newbie and have what I assume is a basic question. os.listdir takes a pathname as an arg...
8
by: Hannu Kankaanp?? | last post by:
This may be a bug or simply a strange result of undefined behaviour, but this is what I get with Python 2.3.2 on Windows XP: >>> import os >>> os.listdir('') >>> os.listdir(u'')
0
by: Ishwor | last post by:
hi check your seperator variable in the os module. :) for example >>> import os >>> os.sep '\\' Now what you do is :- >> os.listdir("D:" + os.sep + "any_other_folder_name" + os.sep); :)
1
by: Doran_Dermot | last post by:
Hi All, I'm currently using "os.listdir" to obtain the contents of some slow Windows shares. I think I've seen another way of doing this using the win32 library but I can't find the example...
7
by: Kenneth Pronovici | last post by:
I have some confusion regarding the relationship between locale, os.listdir() and unicode pathnames. I'm running Python 2.3.5 on a Debian system. If it matters, all of the files I'm dealing with...
1
by: kai | last post by:
Hello, I use dircache.listdir(myDir) in my module repeatedly. On OS WIN 2000 listdir() will re-read the directory structure! But on AIX, listdir() will not re-read the directory structure (see...
6
by: Stef Mientki | last post by:
hello, I want to find all files with the extension "*.txt". From the examples in "Learning Python, Lutz and Asher" and from the website I see examples where you also may specify a wildcard...
3
by: vedrandekovic | last post by:
Hello Here is my simple listdir example: Here is my error: WindowsError: The system cannot find the path specified: 'l/ *.*'
0
by: scriptmann | last post by:
Hi, I'm trying to use os.listdir() to list directories with simplified chinese filenames. However, while I can see the filenames correctly in windows explorer, I am getting ? in the filename...
2
by: Kemmylinns12 | last post by:
Blockchain technology has emerged as a transformative force in the business world, offering unprecedented opportunities for innovation and efficiency. While initially associated with cryptocurrencies...
0
by: Naresh1 | last post by:
What is WebLogic Admin Training? WebLogic Admin Training is a specialized program designed to equip individuals with the skills and knowledge required to effectively administer and manage Oracle...
0
Oralloy
by: Oralloy | last post by:
Hello Folks, I am trying to hook up a CPU which I designed using SystemC to I/O pins on an FPGA. My problem (spelled failure) is with the synthesis of my design into a bitstream, not the C++...
0
by: Carina712 | last post by:
Setting background colors for Excel documents can help to improve the visual appeal of the document and make it easier to read and understand. Background colors can be used to highlight important...
0
BLUEPANDA
by: BLUEPANDA | last post by:
At BluePanda Dev, we're passionate about building high-quality software and sharing our knowledge with the community. That's why we've created a SaaS starter kit that's not only easy to use but also...
0
by: Rahul1995seven | last post by:
Introduction: In the realm of programming languages, Python has emerged as a powerhouse. With its simplicity, versatility, and robustness, Python has gained popularity among beginners and experts...
2
by: Ricardo de Mila | last post by:
Dear people, good afternoon... I have a form in msAccess with lots of controls and a specific routine must be triggered if the mouse_down event happens in any control. Than I need to discover what...
1
by: Johno34 | last post by:
I have this click event on my form. It speaks to a Datasheet Subform Private Sub Command260_Click() Dim r As DAO.Recordset Set r = Form_frmABCD.Form.RecordsetClone r.MoveFirst Do If...
0
by: F22F35 | last post by:
I am a newbie to Access (most programming for that matter). I need help in creating an Access database that keeps the history of each user in a database. For example, a user might have lesson 1 sent...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.