PEP on path module for standard library

[Michael Hoffman]

Many of you are familiar with Jason Orendorff's path module
<http://www.jorendorff. com/articles/python/path/>, which is frequently
recommended here on c.l.p. I submitted an RFE to add it to the Python
standard library, and Reinhold Birkenfeld started a discussion on it in
python-dev
<http://mail.python.org/pipermail/python-dev/2005-June/054438.html>.

The upshot of the discussion was that many python-dev'ers wanted path
added to the stdlib, but Guido was not convinced and said it must have a
PEP. So Reinhold and I are going to work on one. Reinhold has already
made some changes to the module to fit the python-dev discussion and put
it in CPython CVS at nondist/sandbox/path.

For the PEP, do any of you have arguments for or against including path?
Code samples that are much easier or more difficult with this class
would also be most helpful.

I use path in more of my modules and scripts than any other third-party
module, and I know it will be very helpful when I no longer have to
worry about deploying it.

Thanks in advance,
--
Michael Hoffman

[Peter Hansen]

Michael Hoffman wrote:

For the PEP, do any of you have arguments for or against including path?
Code samples that are much easier or more difficult with this class
would also be most helpful.

I believe the strongest argument for "path" can be made for how it
integrates functionality which, although closely related conceptually,
is currently distributed across a half dozen or more different modules
in the standard library. Especially for newbies (I can only imagine, at
this stage) it would make working with files much easier in a many ways.

"Easier" or "more difficult" is a subjective thing, of course, but one
can't argue with the fact that path can sometimes do through a single
object what would otherwise require several imports and a bunch of calls
into things like open(), os.path, grep, and shutil.

Examples showing effective uses of path that simplify those cases would
probably merit the label "easier" even in Guido's mind, though
unfortunately that's not certain. "Easier" in some minds might simply
translate to "many lines less code", and while path can sometimes do
that, aside from the ease of splitting and joining stuff without
multiple calls to os.path.this-and-that, it really doesn't often reduce
code size _that_ much, in my experience. (Postings to c.l.p showing a
50% reduction in code size for contrived examples notwithstanding .)

A related thoughts: since paths are objects, they have attributes or
properties, and having things like ".basename" and ".parent" readily
available without having to do obscure things like
os.path.split(s omepath)[0] makes things much easier to read (therefore
more maintainable). In fact, I'd propose that as another strong
argument in path's favour: it makes code much more readable, even if not
"easier" to write.

Hmm... does "easier" or "more difficult" apply to the writing of the
code or the reading of it? I find it self-evident that code written
using "path" is much easier to read, not necessarily much easier to
write (for non-newbies).

I'd summarize this by saying that the integration of "path" in the
stdlib would make it easier for newbies to write code (that might not be
obvious to a non-newbie... shall we ask some to help?), and easier for
everyone to read code (self-evident, no?), and if that's not a
sufficient condition for inclusion I don't know what is.

-Peter

[Fernando Perez]

Peter Hansen wrote:

Michael Hoffman wrote:

For the PEP, do any of you have arguments for or against including path?
Code samples that are much easier or more difficult with this class
would also be most helpful.

I believe the strongest argument for "path" can be made for how it
integrates functionality which, although closely related conceptually,
is currently distributed across a half dozen or more different modules
in the standard library. Especially for newbies (I can only imagine, at
this stage) it would make working with files much easier in a many ways.

+10

One of the few things that annoys me about the stdlib is what one could call
performing 'shell-scripting-like' tasks, and precisely because of the problem
you point out. A number of conceptually related and common tasks are
scattered all over, and every time I need to write this kind of code, I find
myself paging over the docs for multiple modules, with no real intuition as to
where I could even guess where to find things. This is very unusual for
python, where in most cases things are so well organized, that blind guessing
tends to work remarkably well.

Personally I like the path module _a lot_, though I'm sure a thorough once-over
from c.l.py and python-dev, via a PEP, can only make it better and smooth out
hidden rough edges and corner cases. But I'll be very happy if it does go
into the stdlib in the future.

Just my .02.

Best,

f

[mkent]

I really love Jason's 'path' module. Sadly, I've encountered a serious
problem with using it. When you try to 'freeze' an application module,
and Jason's 'path' module is present in any of the directories that are
looked at by freeze's module finder (your app doesn't have to import
it), freeze goes into an infinite loop of imports, eventually getting a
'maximum recursion depth' exception. This seems to be related to
freeze getting confused between 'os.path' and Jason's 'path'.

I encountered this using Jason's latest 'path' module and Python 2.3.2.
I was able to solve it for my use by renaming path.py to newpath.py
and using 'from newpath import path' in my modules.

I've just notified Jason about this. I presume a solution like mine
will be used, and look forward to seeing Jason's module in stdlib.

[Reinhold Birkenfeld]

FYI: I modified the path module a bit so that it fits many of the suggestions
from python-dev, and put the result in the Python CVS tree under
nondist/sandbox/path.

Most prominent change is that it doesn't inherit from str/unicode anymore.
I found this distinction important, because as a str subclass the Path object
has many methods that don't make sense for it.

Peter Hansen wrote:

Michael Hoffman wrote:

For the PEP, do any of you have arguments for or against including path?
Code samples that are much easier or more difficult with this class
would also be most helpful.
I believe the strongest argument for "path" can be made for how it
integrates functionality which, although closely related conceptually,
is currently distributed across a half dozen or more different modules
in the standard library. Especially for newbies (I can only imagine, at
this stage) it would make working with files much easier in a many ways.

"Easier" or "more difficult" is a subjective thing, of course, but one
can't argue with the fact that path can sometimes do through a single
object what would otherwise require several imports and a bunch of calls
into things like open(), os.path, grep, and shutil.

Correct.
Examples showing effective uses of path that simplify those cases would
probably merit the label "easier" even in Guido's mind, though
unfortunately that's not certain. "Easier" in some minds might simply
translate to "many lines less code", and while path can sometimes do
that, aside from the ease of splitting and joining stuff without
multiple calls to os.path.this-and-that, it really doesn't often reduce
code size _that_ much, in my experience. (Postings to c.l.p showing a
50% reduction in code size for contrived examples notwithstanding .)
Well, these examples are the ones we'd like to see here. So, people:
If you posted examples to c.l.py in the past, please try to collect
them here!
A related thoughts: since paths are objects, they have attributes or
properties, and having things like ".basename" and ".parent" readily
available without having to do obscure things like
os.path.split(s omepath)[0] makes things much easier to read (therefore
more maintainable). In fact, I'd propose that as another strong
argument in path's favour: it makes code much more readable, even if not
"easier" to write.

Hmm... does "easier" or "more difficult" apply to the writing of the
code or the reading of it? I find it self-evident that code written
using "path" is much easier to read, not necessarily much easier to
write (for non-newbies).
And it is much more "Pythonic" in my eyes. Though that word may be inaccurate
when it comes from someone else that Guido, I feel that endless chains of
'[os.path.join(os .path.join(z, "a"), x) for x in os.path.listdir (os.path.join(z , "a") if os.path.isfile( os.path.join(.. ..
are not qualified as being Pythonic.
I'd summarize this by saying that the integration of "path" in the
stdlib would make it easier for newbies to write code (that might not be
obvious to a non-newbie... shall we ask some to help?), and easier for
everyone to read code (self-evident, no?), and if that's not a
sufficient condition for inclusion I don't know what is.

Reinhold

[Terry Reedy]

"Reinhold Birkenfeld" <re************ ************@wo lke7.net> wrote in
message news:3k******** ****@individual .net...

Most prominent change is that it doesn't inherit from str/unicode
anymore.
I found this distinction important, because as a str subclass the Path
object
has many methods that don't make sense for it.

While I am - on adding more to learn, I am + on collecting scattered
filesystem functions into methods of coherent classes for eventually
replacement of the former. And I understand that this will mean a period
of duplication. But it is important to get the replacement right.

My only knowledge of the path module is what has been posted. However, it
seems to me that a path is conceptually a sequence of strings, rather than
the single joined-string representation thereof. If so, then of course it
is not and should not be a subclass of single strings. But that Path was
so defined makes me wonder, in my ignorance, whether the current
implementation is the best we can do for the future.

One advantage, for instance, of a split-up list implementation is that a
set of paths with a common prefix could be represented by replacing the
last string with a set of strings.

A more OO-friendly OS than the dominant ones today would accept a path as a
list (sequence) instead of requiring that the list be joined (by an
artifactual character) just to be split back into a list again.

My thoughts anyway.

Terry J. Reedy

[Michael Hoffman]

mk***@webmd.net wrote:

I really love Jason's 'path' module. Sadly, I've encountered a serious
problem with using it. When you try to 'freeze' an application module,
and Jason's 'path' module is present in any of the directories that are
looked at by freeze's module finder (your app doesn't have to import
it), freeze goes into an infinite loop of imports, eventually getting a
'maximum recursion depth' exception. This seems to be related to
freeze getting confused between 'os.path' and Jason's 'path'.

I encountered this using Jason's latest 'path' module and Python 2.3.2.
I was able to solve it for my use by renaming path.py to newpath.py
and using 'from newpath import path' in my modules.

I've just notified Jason about this. I presume a solution like mine
will be used, and look forward to seeing Jason's module in stdlib.

This sounds like a bug in "freeze" rather than something that should be
worked around in the standard library. Although there have already been
people opposed to naming it path because the duplication with os.path
might confuse humans.
--
Michael Hoffman

[Michael Hoffman]

Reinhold Birkenfeld wrote:

FYI: I modified the path module a bit so that it fits many of the suggestions
from python-dev, and put the result in the Python CVS tree under
nondist/sandbox/path.

Most prominent change is that it doesn't inherit from str/unicode anymore.
I found this distinction important, because as a str subclass the Path object
has many methods that don't make sense for it.

Having path descend from str/unicode is extremely useful since I can
then pass a path object to any function someone else wrote without
having to worry about whether they were checking for basestring. I think
there is a widely used pattern of accepting either a basestring[1] or a
file-like object as a function argument, and using isinstance() to
figure out which it is.

What do you gain from removing these methods? A smaller dir()?

[1] Probably str in actuality.
--
Michael Hoffman

[Reinhold Birkenfeld]

Michael Hoffman wrote:

Reinhold Birkenfeld wrote:

FYI: I modified the path module a bit so that it fits many of the suggestions
from python-dev, and put the result in the Python CVS tree under
nondist/sandbox/path.

Most prominent change is that it doesn't inherit from str/unicode anymore.
I found this distinction important, because as a str subclass the Path object
has many methods that don't make sense for it.
Having path descend from str/unicode is extremely useful since I can
then pass a path object to any function someone else wrote without
having to worry about whether they were checking for basestring. I think
there is a widely used pattern of accepting either a basestring[1] or a
file-like object as a function argument, and using isinstance() to
figure out which it is.

Where do you see that pattern? IIRC it's not in the stdlib.
What do you gain from removing these methods? A smaller dir()?

It made sense to me at the time I changed this, although at the moment
I can't exactly recall the reasons.

Probably as Terry said: a path is both a list and a string.

Reinhold

[Reinhold Birkenfeld]

Michael Hoffman wrote:

mk***@webmd.net wrote:

I really love Jason's 'path' module. Sadly, I've encountered a serious
problem with using it. When you try to 'freeze' an application module,
and Jason's 'path' module is present in any of the directories that are
looked at by freeze's module finder (your app doesn't have to import
it), freeze goes into an infinite loop of imports, eventually getting a
'maximum recursion depth' exception. This seems to be related to
freeze getting confused between 'os.path' and Jason's 'path'.

I encountered this using Jason's latest 'path' module and Python 2.3.2.
I was able to solve it for my use by renaming path.py to newpath.py
and using 'from newpath import path' in my modules.

I've just notified Jason about this. I presume a solution like mine
will be used, and look forward to seeing Jason's module in stdlib.

This sounds like a bug in "freeze" rather than something that should be
worked around in the standard library. Although there have already been
people opposed to naming it path because the duplication with os.path
might confuse humans.

As the most likely placement will be a class named "Path" inside the "os.path"
module, that bug with freeze won't apply to the "stdlib version" of Path.

Reinhold