PEP on path module for standard library

[Michael Hoffman]

Many of you are familiar with Jason Orendorff's path module
<http://www.jorendorff. com/articles/python/path/>, which is frequently
recommended here on c.l.p. I submitted an RFE to add it to the Python
standard library, and Reinhold Birkenfeld started a discussion on it in
python-dev
<http://mail.python.org/pipermail/python-dev/2005-June/054438.html>.

The upshot of the discussion was that many python-dev'ers wanted path
added to the stdlib, but Guido was not convinced and said it must have a
PEP. So Reinhold and I are going to work on one. Reinhold has already
made some changes to the module to fit the python-dev discussion and put
it in CPython CVS at nondist/sandbox/path.

For the PEP, do any of you have arguments for or against including path?
Code samples that are much easier or more difficult with this class
would also be most helpful.

I use path in more of my modules and scripts than any other third-party
module, and I know it will be very helpful when I no longer have to
worry about deploying it.

Thanks in advance,
--
Michael Hoffman

[Stefan Rank]

on 22.07.2005 16:14 George Sakkis said the following:

"Duncan Booth" <du**********@i nvalid.invalid> wrote:

Personally I think the concept of a specific path type is a good one, but
subclassing string just cries out to me as the wrong thing to do. In other
words, to me a path represents something in a filesystem, the fact that it
has one, or indeed several string representations does not mean that the
path itself is simply a more specific type of string.

[snip] practicality-vs-purity decision. You're right, conceptually a path
HAS_A string description, not IS_A string, so from a pure OO point of
view, it should not inherit string.

Java has `File` which mixes the concepts "an object in the filesystem"
and "a structured locator for such objects (in a hierarchical fs) that
might or might not correspond to an object that is actually there".

`file` and `path` separate that. I think this is very reasonable.

(It would be nice to get `path`(s) easily from a `file`, at the moment
there is only file.name if I'm not mistaken).

And a `path`, to me, actually IS_A string (unicode string) that happens
to have special behaviour (such as os dependent quirks like a
pathseparator that automatically get translated, comparable to '\n' used
internally in strings translated to '\n'|'\r\n')

stefan

[Michael Hoffman]

George Sakkis wrote:

Havind said that, I think
the choice between subclassing or not is going to be a
practicality-vs-purity decision. You're right, conceptually a path
HAS_A string description, not IS_A string, so from a pure OO point of
view, it should not inherit string. OTOH, people in favor of the
subclassing point out the convenience for many (or most) common cases.

It would be an entirely different matter if we were designing a language
from scratch. But we have to deal with an existing codebase that expects
strings.

Here's some code I just wrote seconds ago to construct a path for a scp
upload:

"""
DST_DIRPATH = path("host:~/destination")
RSS_EXT = "rss"

dst_filenamebas e = os.extsep.join([postcode.lower( ), RSS_EXT])
dst_filepath = DST_DIRPATH.joi npath(dst_filen amebase)
"""

With the current path implementation, this Just Works. If I were using
something that parsed and understood paths, the scp/rcp convention of
host:filename would either cause an error or have to be programmed in
separately. The current implementation is much more flexible.

What are the practical advantages and conveniences of *not* subclassing
from basestring?
--
Michael Hoffman

[George Sakkis]

"Duncan Booth" <du**********@i nvalid.invalid> wrote:

Personally I think the concept of a specific path type is a good one, but
subclassing string just cries out to me as the wrong thing to do. In other
words, to me a path represents something in a filesystem, the fact that it
has one, or indeed several string representations does not mean that the
path itself is simply a more specific type of string.

You should need an explicit call to convert a path to a string and that
forces you when passing the path to something that requires a string to
think whether you wanted the string relative, absolute, UNC, uri etc.

First off, I find this is a relatively small detail overall, that is, regardless of whether path
subclasses string or not, its addition in the standard library will be a step forward. Havind said
that, I think the choice between subclassing or not is going to be a practicality-vs-purity
decision. You're right, conceptually a path HAS_A string description, not IS_A string, so from a
pure OO point of view, it should not inherit string. OTOH, people in favor of the subclassing point
out the convenience for many (or most) common cases. It's a tradeoff, so arguments for both cases
should be discussed.

George

[Duncan Booth]

Peter Hansen wrote:

Duncan, are you another formerly non-user of path who has this opinion,
or have you already attempted to use path extensively in your code?
I'm a currently non-user of path who would probably use it if it were in
the standard library but so far have been satisfied to use os.path.
I'm not saying I dismiss the opinions of those who haven't actually
tried working with a string-based path object, but it's worth
considering that you might adopt a different opinion after using it for
a while.
I fully accept that. My point is simply that as a non-user, it sounds to me
as though subclassing string is the wrong approach. I would have expected a
path object to be a sequence of path elements rather than a sequence of
characters. This is basically just a gut feeling though, so I'm perfectly
happy to be told that I'm wrong.

BTW, does it matter at all in practical use that the base class of path
varies between str and unicode depending on the platform?

John Roth wrote: You have to start somewhere. One of the lessons that's beginning
to seep into people's minds is that getting something that works
out there is almost always preferable to (over) design by committee.

Dead right, but once it goes into the standard library it has to pretty
well stop evolving, so it needs to be right, or as close as possible before
that happens.

[Daniel Dittmar]

Duncan Booth wrote:

I would have expected a
path object to be a sequence of path elements rather than a sequence of
characters.

Maybe it's nitpicking, but I don't think that a path object should be a
'sequence of path elements' in an iterator context.

This means that

for element in pathobject:

has no intuitive meaning for me, so it shouldn't be allowed.

Daniel

[John Roth]

"Duncan Booth" <du**********@i nvalid.invalid> wrote in message
news:Xn******** *************** **@127.0.0.1...

John Roth wrote:

You have to start somewhere. One of the lessons that's beginning
to seep into people's minds is that getting something that works
out there is almost always preferable to (over) design by committee.
Dead right, but once it goes into the standard library it has to pretty
well stop evolving, so it needs to be right, or as close as possible
before
that happens.

It has to stop evolving in incompatible directions, at least. Although
there is a precident with the process functions, classes, module,
whatever it is. It's up to five versions now, isn't it?

AFAICT, from a very broad brush perspective, there is really
only one substantive issue: how to handle multiple path-like
"things". URLs have been mentioned in this thread, different
file systems and a possible in-memory file system have been
mentioned in other threads.

So whatever gets out there first shouldn't preempt the ability
to eventually fit into a wider structure without substantial
and incompatible modifications.

John Roth

[John Roth]

"Daniel Dittmar" <da************ @sap.corp> wrote in message
news:db******** **@news.sap-ag.de...

Duncan Booth wrote:

I would have expected a path object to be a sequence of path elements
rather than a sequence of characters.
Maybe it's nitpicking, but I don't think that a path object should be a
'sequence of path elements' in an iterator context.

This means that

for element in pathobject:

has no intuitive meaning for me, so it shouldn't be allowed.

However, a path as a sequence of characters has even less
meaning - I can't think of a use, while I have an application
where traversing a path as a sequence of path elements makes
perfect sense: I need to descend the directory structure, directory
by directory, looking for specific files and types.

John Roth
Daniel

[Duncan Booth]

Michael Hoffman wrote:

Here's some code I just wrote seconds ago to construct a path for a
scp upload:

"""
DST_DIRPATH = path("host:~/destination")
RSS_EXT = "rss"

dst_filenamebas e = os.extsep.join([postcode.lower( ), RSS_EXT])
dst_filepath = DST_DIRPATH.joi npath(dst_filen amebase)
"""

With the current path implementation, this Just Works.
It isn't at all obvious to me that it works:

import os
from path import path
postcode = "AA2 9ZZ"
DST_DIRPATH = path("host:~/destination")
RSS_EXT = "rss"

dst_filenamebas e = os.extsep.join([postcode.lower( ), RSS_EXT])
dst_filepath = DST_DIRPATH.joi npath(dst_filen amebase)
print dst_filepath host:~/destination\aa2 9zz.rss

If I were using
something that parsed and understood paths, the scp/rcp convention of
host:filename would either cause an error or have to be programmed in
separately. The current implementation is much more flexible.
You still have to program your scp path separately from your filesystem
path in order to handle the different conventions for path separator
characters and maybe also escaping special characters in the path (I don't
use scp much so I don't know if this is required).
What are the practical advantages and conveniences of *not*
subclassing from basestring?

Simplification of the api: not having methods such as center, expandtabs
and zfill.

Not having the base class change from str to unicode depending on which
system you run your code?

Fewer undetected bugs (explicit is better than implicit)?

Perhaps none of these matter in practice. As I said elsewhere I haven't
used path for anything real, so I'm still finding surprises such as why
this doesn't do what I expect:
p = path('a/b')
q = path('c/d')
p+q

path(u'a/bc/d')

If path didn't subclass string then either this would have been
implemented, and probably would Do The Right Thing, or it wouldn't be
implemented so I'd quickly realise I needed to do something else. Instead
it does something suprising.

[Andrew Dalke]

Duncan Booth wrote:

Personally I think the concept of a specific path type is a good one, but
subclassing string just cries out to me as the wrong thing to do.
I disagree. I've tried using a class which wasn't derived from
a basestring and kept running into places where it didn't work well.
For example, "open" and "mkdir" take strings as input. There is no
automatic coercion.

class Spam: .... def __getattr__(sel f, name):
.... print "Want", repr(name)
.... raise AttributeError, name
.... open(Spam()) Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: coercing to Unicode: need string or buffer, instance found import os
os.mkdir(Spam() ) Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: coercing to Unicode: need string or buffer, instance found

The solutions to this are:
1) make the path object be derived from str or unicode. Doing
this does not conflict with any OO design practice (eg, Liskov
substitution).

2) develop a new "I represent a filename" protocol, probably done
via adapt().

I've considered the second of these but I think it's a more
complicated solution and it won't fit well with existing APIs
which do things like
if isinstance(inpu t, basestring):
input = open(input, "rU")
for line in input:
print line

I showed several places in the stdlib and in 3rd party packages
where this is used.

In other words, to me a path represents something in a filesystem,
Being picky - or something that could be in a filesystem.
the fact that it
has one, or indeed several string representations does not mean that the
path itself is simply a more specific type of string.
I didn't follow this.
You should need an explicit call to convert a path to a string and that
forces you when passing the path to something that requires a string to
think whether you wanted the string relative, absolute, UNC, uri etc.
You are broadening the definition of a file path to include URIs?
That's making life more complicated. Eg, the rules for joining
file paths may be different than the rules for joining URIs.
Consider if I have a file named "mail:da***@exa mple.com" and I
join that with "file://home/dalke/badfiles/".

Additionally, the actions done on URIs are different than on file
paths. What should os.listdir("htt p://www.python.org/") do?

As I mentioned, I tried some classes which emulated file
paths. One was something like

class TempDir:
"""removes the directory when the refcount goes to 0"""
def __init__(self):
self.filename = ... use a function from the tempfile module
def __del__(self):
if os.path.exists( self.filename):
shutil.rmtree(s elf.filename)
def __str__(self):
return self.filename

I could do

dirname = TempDir()

but then instead of

os.mkdir(dirnam e)
tmpfile = os.path.join(di rname, "blah.txt")

I needed to write it as

os.mkdir(str(di rname))
tmpfile = os.path.join(st r(dirname), "blah.txt") )

or have two variables, one which could delete the
directory and the other for the name. I didn't think
that was good design.
If I had derived from str/unicode then things would
have been cleaner.

Please note, btw, that some filesystems are unicode
based and others are not. As I recall, one nice thing
about the path module is that it chooses the appropriate
base class at import time. My "str()" example above
does not and would fail on a Unicode filesystem aware
Python build.
It may even be that we need a hierarchy of path
classes: URLs need similar but not identical manipulations
to file paths, so if we want to address the failings
of os.path perhaps we should also look at the failings
of urlparse at the same time.

I've found that hierarchies are rarely useful compared
to the number of times they are proposed and used. One
of the joys to me of Python is its deemphasis of class
hierarchies.

I think the same is true here. File paths and URIs are
sufficiently different that there are only a few bits
of commonality between them. Consider 'split' which
for files creates (dirname, filename) while for urls
it creates (scheme, netloc, path, query, fragment)

Andrew
da***@dalkescie ntific.com

[Andrew Dalke]

George Sakkis wrote:

You're right, conceptually a path
HAS_A string description, not IS_A string, so from a pure OO point of
view, it should not inherit string.

How did you decide it's "has-a" vs. "is-a"?

All C calls use a "char *" for filenames and paths,
meaning the C model file for the filesystem says
paths are strings.

Paths as strings fit the Liskov substitution principle
in that any path object can be used any time a
string is used (eg, "loading from " + filename)

Good information hiding suggests that a better API
is one that requires less knowledge. I haven't
seen an example of how deriving from (unicode)
string makes things more complicated than not doing so.

Andrew
da***@dalkescie ntific.com