Reinvent no more forever

[Ben Finney]

Howdy all,

On dirtSimple.org[0], PJE wrote:

"Why is Python "blessed" with so much reinvention? Because it's
often cheaper to rewrite than to reuse. Python code is easy to
write, but hard to depend on. You pretty much have to:

1. limit yourself to platforms with a suitable packaging
system,
2. bundle all your dependencies into your distribution, or
3. make your users do all the work themselves

Ouch. No wonder rewriting looks easier."

He then goes on to talk about his own solution: PyPI, setuptools,
EasyInstall, and so on for better dependency control. He concludes:

"But for right now, I plan to start by putting my dependencies
where my mouth is. I will bundle no more forever, for I am tired
of keeping them up-to-date with their upstream maintainers. I
shall rend my monolithic packages into smaller projects, so that
people can make use of cool parts of PEAK without having to
install anything but the part they want. I will find out if this
stuff really works. From where the setuptools now stand, I will
bundle no more forever."

Fine sentiments. What does this mean for PyPI though? How should those
of us who also want to "reinvent no more forever" proceed? How do we
deal with the rampant proliferation of a zillion implementations of
some standard idiom in PyPI?

The case that led me to this quandary is simple: I'm developing an
application that could become simple, and I'm on the lookout for
pieces that might be valuable outside this application. The first one
to catch my eye is an Enum implementation.

I now have a setuptools package of my Enum implementation. Should I
submit it to PyPI?

Advantages:

- I can declare a dependency on that package for all my other
packages that need such functionality, instead of bundling it
every time.

- Others can benefit from my code, instead of yet again including an
Enum implementation (home-grown, or picked from a cookbook) by a
simple dependency declaration.

- The code hopefully gets improved and generalised and all the other
benefits that accrue to software with many users.

Disadvantages:

- Proliferation. What's the protocol when[1] someone else puts an
(incompatible, differently-specified) Enum implementation into
PyPI?

- Naming. How many ways can a package providing an Enum be named?
I'd prefer mine to be named "Enum" or "enum", but why should mine
be the one that claims that name?

- It's just a pretty simple type, with unit tests. Does this really
justify a PyPI package?

This is common to any repository of code. But the "automatic
dependency" problem means that all those packages, and many more
outside that repository, need to know how those problems are resolved.

Operating system distributors have policies (to greater or lesser
degrees of strictness) to ensure this kind of quality control. My
understanding of PyPI is that there's no such policy.

I'd love to follow the mantra PJE espouses, but if it's good for one
person it's probably good for countless others. How do we deal with
that? What actions can we take in advance to prevent problems in
future?
[0] <URL:http://dirtsimple.org/2005/07/reinvent-no-more-forever.html>

[1] Of course, someone already has. I prefer mine to theirs, hence the
question.

--
\ "I planted some bird seed. A bird came up. Now I don't know |
`\ what to feed it." -- Steven Wright |
_o__) |
Ben Finney

[Fuzzyman]

Ben Finney wrote:

Howdy all,

Hello,
On dirtSimple.org[0], PJE wrote:

"Why is Python "blessed" with so much reinvention? Because it's
often cheaper to rewrite than to reuse. Python code is easy to
write, but hard to depend on. You pretty much have to:

1. limit yourself to platforms with a suitable packaging
system,
2. bundle all your dependencies into your distribution, or

For pure python modules I don't find this to be a big problem.
3. make your users do all the work themselves

Ouch. No wonder rewriting looks easier."
[snip..]
I now have a setuptools package of my Enum implementation. Should I
submit it to PyPI?

Advantages:

- I can declare a dependency on that package for all my other
packages that need such functionality, instead of bundling it
every time.

- Others can benefit from my code, instead of yet again including an
Enum implementation (home-grown, or picked from a cookbook) by a
simple dependency declaration.

- The code hopefully gets improved and generalised and all the other
benefits that accrue to software with many users.

These are all very compelling reasons.
Disadvantages:

- Proliferation. What's the protocol when[1] someone else puts an
(incompatible, differently-specified) Enum implementation into
PyPI?

That shouldn't be any more of a problem than the current situation. In
fact as the setuptools system becomes more established then it should
be easier for people to discover that an existing package does what
they want - rather than creating their own.
- Naming. How many ways can a package providing an Enum be named?
I'd prefer mine to be named "Enum" or "enum", but why should mine
be the one that claims that name?

I think "first come - first served" is the only possible way this can
work.
- It's just a pretty simple type, with unit tests. Does this really
justify a PyPI package?

If it solves a common problem in an elegant way, then it's better
shared. :-)
This is common to any repository of code. But the "automatic
dependency" problem means that all those packages, and many more
outside that repository, need to know how those problems are resolved.

Operating system distributors have policies (to greater or lesser
degrees of strictness) to ensure this kind of quality control. My
understanding of PyPI is that there's no such policy.

I'd love to follow the mantra PJE espouses, but if it's good for one
person it's probably good for countless others. How do we deal with
that? What actions can we take in advance to prevent problems in
future?

Isn't setuptools *the system* that addresses these issues ? It provides
a way to track and auto-install dependencies - so long as you have the
right kind of netowrk connection [1] and the packager of the
dependencies provides them in a way compatible with setuptools.

All the best,

Fuzzy
http://www.voidspace.org.uk/python/index.shtml

[1] setuptools uses python code to access the internet and install
dependencies. Python code can't fetch https URLs through a proxy and
can't use a SOCKS proxy at all. That means with some kinds of internet
conenctions (e.g. behind company firewalls) it doesn't work. Building
optional support for pycurl into setuptools could possibly resolve
this.

[0] <URL:http://dirtsimple.org/2005/07/reinvent-no-more-forever.html>

[1] Of course, someone already has. I prefer mine to theirs, hence the
question.

--
\ "I planted some bird seed. A bird came up. Now I don't know |
`\ what to feed it." -- Steven Wright |
_o__) |
Ben Finney

[Fuzzyman]

Ben Finney wrote:

Howdy all,

Hello,
On dirtSimple.org[0], PJE wrote:

"Why is Python "blessed" with so much reinvention? Because it's
often cheaper to rewrite than to reuse. Python code is easy to
write, but hard to depend on. You pretty much have to:

1. limit yourself to platforms with a suitable packaging
system,
2. bundle all your dependencies into your distribution, or

For pure python modules I don't find this to be a big problem.
3. make your users do all the work themselves

Ouch. No wonder rewriting looks easier."
[snip..]
I now have a setuptools package of my Enum implementation. Should I
submit it to PyPI?

Advantages:

- I can declare a dependency on that package for all my other
packages that need such functionality, instead of bundling it
every time.

- Others can benefit from my code, instead of yet again including an
Enum implementation (home-grown, or picked from a cookbook) by a
simple dependency declaration.

- The code hopefully gets improved and generalised and all the other
benefits that accrue to software with many users.

These are all very compelling reasons.
Disadvantages:

- Proliferation. What's the protocol when[1] someone else puts an
(incompatible, differently-specified) Enum implementation into
PyPI?

That shouldn't be any more of a problem than the current situation. In
fact as the setuptools system becomes more established then it should
be easier for people to discover that an existing package does what
they want - rather than creating their own.
- Naming. How many ways can a package providing an Enum be named?
I'd prefer mine to be named "Enum" or "enum", but why should mine
be the one that claims that name?

I think "first come - first served" is the only possible way this can
work.
- It's just a pretty simple type, with unit tests. Does this really
justify a PyPI package?

If it solves a common problem in an elegant way, then it's better
shared. :-)
This is common to any repository of code. But the "automatic
dependency" problem means that all those packages, and many more
outside that repository, need to know how those problems are resolved.

Operating system distributors have policies (to greater or lesser
degrees of strictness) to ensure this kind of quality control. My
understanding of PyPI is that there's no such policy.

I'd love to follow the mantra PJE espouses, but if it's good for one
person it's probably good for countless others. How do we deal with
that? What actions can we take in advance to prevent problems in
future?

Isn't setuptools *the system* that addresses these issues ? It provides
a way to track and auto-install dependencies - so long as you have the
right kind of netowrk connection [1] and the packager of the
dependencies provides them in a way compatible with setuptools.

All the best,

Fuzzy
http://www.voidspace.org.uk/python/index.shtml

[1] setuptools uses python code to access the internet and install
dependencies. Python code can't fetch https URLs through a proxy and
can't use a SOCKS proxy at all. That means with some kinds of internet
conenctions (e.g. behind company firewalls) it doesn't work. Building
optional support for pycurl into setuptools could possibly resolve
this.

[0] <URL:http://dirtsimple.org/2005/07/reinvent-no-more-forever.html>

[1] Of course, someone already has. I prefer mine to theirs, hence the
question.

--
\ "I planted some bird seed. A bird came up. Now I don't know |
`\ what to feed it." -- Steven Wright |
_o__) |
Ben Finney

[Ben Finney]

Fuzzyman <fu******@gmail.com> wrote:

Ben Finney wrote:

On dirtSimple.org[0], PJE wrote:

[...] Python code is easy to write, but hard to depend on. You
pretty much have to:

1. limit yourself to platforms with a suitable packaging
system,
2. bundle all your dependencies into your distribution, or

For pure python modules I don't find this to be a big problem.

The problem isn't whether it's Python or not; the problem is the
complexity of the code base. A simple application that depends on
infrastructure from five different non-standard packages becomes a
right mess to distribute, is stuck with code that is peripheral to its
own purpose, and can't benefit from new versions of those packages.

[distributing a simple package via PyPI]

Disadvantages:

- Proliferation. What's the protocol when[1] someone else puts
an (incompatible, differently-specified) Enum implementation
into PyPI?

That shouldn't be any more of a problem than the current situation.
In fact as the setuptools system becomes more established then it
should be easier for people to discover that an existing package
does what they want - rather than creating their own.

I hold up all existing public source code repositories as evidence
against your prediction. Reinventing the wheel is distressingly
common, and a dependency tool isn't going to stop that.

- Naming. How many ways can a package providing an Enum be
named? I'd prefer mine to be named "Enum" or "enum", but why
should mine be the one that claims that name?

I think "first come - first served" is the only possible way this
can work.

Perhaps so.

- It's just a pretty simple type, with unit tests. Does this
really justify a PyPI package?

If it solves a common problem in an elegant way, then it's better
shared. :-)

Sure. But the question was, is PyPI the place to do so?

This is common to any repository of code. But the "automatic
dependency" problem means that all those packages, and many more
outside that repository, need to know how those problems are
resolved.

Operating system distributors have policies (to greater or lesser
degrees of strictness) to ensure this kind of quality control. My
understanding of PyPI is that there's no such policy.

Isn't setuptools *the system* that addresses these issues ? It
provides a way to track and auto-install dependencies [...]

No, setuptools is only the answer for the packages themselves; the
problems I describe are inherent to the package repository.

The setuptools dependency model rests on the reliability of the
package repository. Not only in terms of availability but also in
terms of the quality and functionality of the package set.

--
\ "I went to the museum where they had all the heads and arms |
`\ from the statues that are in all the other museums." -- Steven |
_o__) Wright |
Ben Finney

[Ben Finney]

Fuzzyman <fu******@gmail.com> wrote:

Ben Finney wrote:

On dirtSimple.org[0], PJE wrote:

[...] Python code is easy to write, but hard to depend on. You
pretty much have to:

1. limit yourself to platforms with a suitable packaging
system,
2. bundle all your dependencies into your distribution, or

For pure python modules I don't find this to be a big problem.

The problem isn't whether it's Python or not; the problem is the
complexity of the code base. A simple application that depends on
infrastructure from five different non-standard packages becomes a
right mess to distribute, is stuck with code that is peripheral to its
own purpose, and can't benefit from new versions of those packages.

[distributing a simple package via PyPI]

Disadvantages:

- Proliferation. What's the protocol when[1] someone else puts
an (incompatible, differently-specified) Enum implementation
into PyPI?

That shouldn't be any more of a problem than the current situation.
In fact as the setuptools system becomes more established then it
should be easier for people to discover that an existing package
does what they want - rather than creating their own.

I hold up all existing public source code repositories as evidence
against your prediction. Reinventing the wheel is distressingly
common, and a dependency tool isn't going to stop that.

- Naming. How many ways can a package providing an Enum be
named? I'd prefer mine to be named "Enum" or "enum", but why
should mine be the one that claims that name?

I think "first come - first served" is the only possible way this
can work.

Perhaps so.

- It's just a pretty simple type, with unit tests. Does this
really justify a PyPI package?

If it solves a common problem in an elegant way, then it's better
shared. :-)

Sure. But the question was, is PyPI the place to do so?

This is common to any repository of code. But the "automatic
dependency" problem means that all those packages, and many more
outside that repository, need to know how those problems are
resolved.

Operating system distributors have policies (to greater or lesser
degrees of strictness) to ensure this kind of quality control. My
understanding of PyPI is that there's no such policy.

Isn't setuptools *the system* that addresses these issues ? It
provides a way to track and auto-install dependencies [...]

No, setuptools is only the answer for the packages themselves; the
problems I describe are inherent to the package repository.

The setuptools dependency model rests on the reliability of the
package repository. Not only in terms of availability but also in
terms of the quality and functionality of the package set.

--
\ "I went to the museum where they had all the heads and arms |
`\ from the statues that are in all the other museums." -- Steven |
_o__) Wright |
Ben Finney

[Phillip J. Eby]

Ben Finney wrote:

- Proliferation. What's the protocol when[1] someone else puts an
(incompatible, differently-specified) Enum implementation into
PyPI?
Either one of the two will be judged better, and the other will wither
away, or else each will be better for different circumstances, and
different people will use them. Either way, not a problem.

- Naming. How many ways can a package providing an Enum be named?
I'd prefer mine to be named "Enum" or "enum", but why should mine
be the one that claims that name?
If you feel badly about it, just call it BFEnum, or SuperEnum, or make
up some other "branding" to use for your packages. Otherwise, take the
rightful spoils of the pioneer by staking your claim to the name. :)

- It's just a pretty simple type, with unit tests. Does this really
justify a PyPI package?
Yes. Excellent documentation would be a plus, of course, and perhaps
one of those screencasts that seem to be all the rage nowadays. ;)

I'd love to follow the mantra PJE espouses, but if it's good for one
person it's probably good for countless others. How do we deal with
that? What actions can we take in advance to prevent problems in
future?
It's simple, really. Ridicule and scorn are quite effective behavior
modification techniques for a community to employ in furthering its
operational goals. So, when people step out of line, we'll just make
fun of them until they conform. :)

[1] Of course, someone already has. I prefer mine to theirs, hence the
question.

Okay, so call yours "SuperEnum" or "PowerEnum" or "UltraEnum" or
"BetterEnum", "Enum-O-Matic", "Symbolitron"...

or just think about *why* yours is better, for *whom* it's better, and
*when*, and then give it a name that emphasizes one or more of those
things. Even though "all the good domain names are taken", there still
seem to be an infinity of names remaining. That's also true for PyPI.

[Phillip J. Eby]

Ben Finney wrote:

- Proliferation. What's the protocol when[1] someone else puts an
(incompatible, differently-specified) Enum implementation into
PyPI?
Either one of the two will be judged better, and the other will wither
away, or else each will be better for different circumstances, and
different people will use them. Either way, not a problem.

- Naming. How many ways can a package providing an Enum be named?
I'd prefer mine to be named "Enum" or "enum", but why should mine
be the one that claims that name?
If you feel badly about it, just call it BFEnum, or SuperEnum, or make
up some other "branding" to use for your packages. Otherwise, take the
rightful spoils of the pioneer by staking your claim to the name. :)

- It's just a pretty simple type, with unit tests. Does this really
justify a PyPI package?
Yes. Excellent documentation would be a plus, of course, and perhaps
one of those screencasts that seem to be all the rage nowadays. ;)

I'd love to follow the mantra PJE espouses, but if it's good for one
person it's probably good for countless others. How do we deal with
that? What actions can we take in advance to prevent problems in
future?
It's simple, really. Ridicule and scorn are quite effective behavior
modification techniques for a community to employ in furthering its
operational goals. So, when people step out of line, we'll just make
fun of them until they conform. :)

[1] Of course, someone already has. I prefer mine to theirs, hence the
question.

Okay, so call yours "SuperEnum" or "PowerEnum" or "UltraEnum" or
"BetterEnum", "Enum-O-Matic", "Symbolitron"...

or just think about *why* yours is better, for *whom* it's better, and
*when*, and then give it a name that emphasizes one or more of those
things. Even though "all the good domain names are taken", there still
seem to be an infinity of names remaining. That's also true for PyPI.

[Ben Finney]

Phillip J. Eby <pj*@telecommunity.com> wrote:

Ben Finney wrote:

- It's just a pretty simple type, with unit tests. Does this
really justify a PyPI package?

Yes.

Thanks for the brief, but supportive discussion from everyone. I've
now packaged and uploaded my simple module. (No prizes for discovering
it; I don't think it warrants discussion in this thread.)

I guess I wasn't putting much faith in social measures for solving
problems after they occur, and imagined some kind of oversight or
moderation or suchlike. In retrospect, that sounds like a terrible
idea, and social measures are the only thing that *can* properly deal
with these issues.

--
\ "Nothing in life is so exhilarating as to be shot at without |
`\ result." -- Winston Churchill |
_o__) |
Ben Finney

[Ben Finney]

Phillip J. Eby <pj*@telecommunity.com> wrote:

Ben Finney wrote:

- It's just a pretty simple type, with unit tests. Does this
really justify a PyPI package?

Yes.

Thanks for the brief, but supportive discussion from everyone. I've
now packaged and uploaded my simple module. (No prizes for discovering
it; I don't think it warrants discussion in this thread.)

I guess I wasn't putting much faith in social measures for solving
problems after they occur, and imagined some kind of oversight or
moderation or suchlike. In retrospect, that sounds like a terrible
idea, and social measures are the only thing that *can* properly deal
with these issues.

--
\ "Nothing in life is so exhilarating as to be shot at without |
`\ result." -- Winston Churchill |
_o__) |
Ben Finney

[Dave Hansen]

On 17 Nov 2005 18:06:55 -0800 in comp.lang.python, "Phillip J. Eby"
<pj*@telecommunity.com> wrote:

[...]

Okay, so call yours "SuperEnum" or "PowerEnum" or "UltraEnum" or
"BetterEnum", "Enum-O-Matic", "Symbolitron"...

ITYM "EnumOMatic" or "Enum_O_Matic". "Enum-O-Matic" is a syntax
error. Or worse...

Regards,
-=Dave

--
Change is inevitable, progress is not.

[Dave Hansen]

On 17 Nov 2005 18:06:55 -0800 in comp.lang.python, "Phillip J. Eby"
<pj*@telecommunity.com> wrote:

[...]

Okay, so call yours "SuperEnum" or "PowerEnum" or "UltraEnum" or
"BetterEnum", "Enum-O-Matic", "Symbolitron"...

ITYM "EnumOMatic" or "Enum_O_Matic". "Enum-O-Matic" is a syntax
error. Or worse...

Regards,
-=Dave

--
Change is inevitable, progress is not.

[Phillip J. Eby]

Dave Hansen wrote:

On 17 Nov 2005 18:06:55 -0800 in comp.lang.python, "Phillip J. Eby"
<pj*@telecommunity.com> wrote:

[...]

Okay, so call yours "SuperEnum" or "PowerEnum" or "UltraEnum" or
"BetterEnum", "Enum-O-Matic", "Symbolitron"...

ITYM "EnumOMatic" or "Enum_O_Matic". "Enum-O-Matic" is a syntax
error. Or worse...

Not in a PyPI project name, it isn't. Project names aren't limited to
being Python identifiers.

[Phillip J. Eby]

Dave Hansen wrote:

On 17 Nov 2005 18:06:55 -0800 in comp.lang.python, "Phillip J. Eby"
<pj*@telecommunity.com> wrote:

[...]

Okay, so call yours "SuperEnum" or "PowerEnum" or "UltraEnum" or
"BetterEnum", "Enum-O-Matic", "Symbolitron"...

ITYM "EnumOMatic" or "Enum_O_Matic". "Enum-O-Matic" is a syntax
error. Or worse...

Not in a PyPI project name, it isn't. Project names aren't limited to
being Python identifiers.

[ismaelgfk]

Ben Finney wrote:

Fine sentiments. What does this mean for PyPI though? How should those
of us who also want to "reinvent no more forever" proceed? How do we
deal with the rampant proliferation of a zillion implementations of
some standard idiom in PyPI?

How about some kind of "mega util" package? One big package with all
those recurring reinventions. If it gets popular enough, I'm sure it
could stop so much reinvention.
Perhaps even pack all the similar-but-different implementations in one
package (and then have
from wheel import enumA
from wheel import enumB , etc.. )

Actually, "The wheel" is quite a good name for such a package :-P

I'm thinking that a Wiki would be quite suitable for development, to
incite more contributions (rather than only CVS).

What do you think?

Ismael

[ismaelgfk]

Ben Finney wrote:

Fine sentiments. What does this mean for PyPI though? How should those
of us who also want to "reinvent no more forever" proceed? How do we
deal with the rampant proliferation of a zillion implementations of
some standard idiom in PyPI?

How about some kind of "mega util" package? One big package with all
those recurring reinventions. If it gets popular enough, I'm sure it
could stop so much reinvention.
Perhaps even pack all the similar-but-different implementations in one
package (and then have
from wheel import enumA
from wheel import enumB , etc.. )

Actually, "The wheel" is quite a good name for such a package :-P

I'm thinking that a Wiki would be quite suitable for development, to
incite more contributions (rather than only CVS).

What do you think?

Ismael

[Ben Finney]

is*******@gmail.com wrote:

Ben Finney wrote:

How do we deal with the rampant proliferation of a zillion
implementations of some standard idiom in PyPI?
How about some kind of "mega util" package? One big package with all
those recurring reinventions. If it gets popular enough, I'm sure it
could stop so much reinvention.

No, you'd just ensure there would be several, competing, incompatible
"mega util" packages. Plus, you'd have the additional problem of
depending on a big, low-coherence package for one or two idioms within
it.
I'm thinking that a Wiki would be quite suitable for development, to
incite more contributions (rather than only CVS).

<URL:http://wiki.python.org/moin/>

--
\ "As the most participatory form of mass speech yet developed, |
`\ the Internet deserves the highest protection from governmental |
_o__) intrusion." -- U.S. District Court Judge Dalzell |
Ben Finney

[Ben Finney]

is*******@gmail.com wrote:

Ben Finney wrote:

How do we deal with the rampant proliferation of a zillion
implementations of some standard idiom in PyPI?
How about some kind of "mega util" package? One big package with all
those recurring reinventions. If it gets popular enough, I'm sure it
could stop so much reinvention.

No, you'd just ensure there would be several, competing, incompatible
"mega util" packages. Plus, you'd have the additional problem of
depending on a big, low-coherence package for one or two idioms within
it.
I'm thinking that a Wiki would be quite suitable for development, to
incite more contributions (rather than only CVS).

<URL:http://wiki.python.org/moin/>

--
\ "As the most participatory form of mass speech yet developed, |
`\ the Internet deserves the highest protection from governmental |
_o__) intrusion." -- U.S. District Court Judge Dalzell |
Ben Finney