473,850 Members | 2,189 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Features for a Python package manager?

Hello c.l.py,

what features would you expect of a Python package manager, similar to
CPAN or rubygems?

I am currently planning to write such a thing, at first privately for
myself, and if it's proving useful, I think about releasing it.

I plan to model it after gentoo's portage, with less features of course.
Would this be acceptable?

feel-free-to-comment-ly yours,
Georg
Jul 18 '05 #1
8 2076
Georg Brandl wrote:
Hello c.l.py,

what features would you expect of a Python package manager, similar to
CPAN or rubygems?

I am currently planning to write such a thing, at first privately for
myself, and if it's proving useful, I think about releasing it.
I've contemplated such a beast myself, but have never got further than a cursory
look at "what work has already been done that I should take into account?"

Howevever, it sounds like that info would be useful to you, so here's what I've got:

PEP 314 (Metadata for Python Packages v 1.1) details the properties of the
metadata that the Python repository relies on.

Don't worry about PEP 241 - it has been superceded by PEP 314.

PEP 301 describes PyPI (Python Package Index) - the central index of available
packages. A running version of it can be found at http://www.python.org/pypi

PEP 243 is probably worth looking at, although I believe it is basically
superceded by PEP 301.

PEP 262 describes a proposed (local system) installation database. An
implementation already exists in the nondist section of CPython's CVS
repository, but further progress is currently stalled due to lack of a champion.

It would also be good to be aware of how distutils currently works (even if
that's just by reading its documentation). Doing your own trawl through PEP 0
wouldn't hurt, either (it's possible I missed something relevant). The archives
of python-dev and the distutils SIG may also provide good info.

If you're serious about doing a good job (i.e. to a "possibly included with the
standard Python distribution some day" standard), it may be worth posting a
question on python-dev about it. Just say that you're contemplating writing such
a tool for your own use, and are interested in any ideas which aren't currently
documented in the places I mentioned above. You'll find several of the authors
of the above material frequent python-dev, and will hopefully have some good
pointers regarding what needs to be done.
I plan to model it after gentoo's portage, with less features of course.
Would this be acceptable?


I don't know enough about Portage to answer that question. I do know any package
manager which made it into the standard distribution would need to work for at
least the big three platforms (Windows/Mac/*nix) :)

Cheers,
Nick.

--
Nick Coghlan | nc******@email. com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net
Jul 18 '05 #2
Nick Coghlan <nc******@iinet .net.au> writes:
I don't know enough about Portage to answer that question. I do know
any package manager which made it into the standard distribution would
need to work for at least the big three platforms (Windows/Mac/*nix) :)


Being written in python - and hopefully integrated into Distutils -
why shouldn't it work on any platform that Python worked on?

<mike
--
Mike Meyer <mw*@mired.or g> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
Jul 18 '05 #3
Mike Meyer wrote:
Nick Coghlan <nc******@iinet .net.au> writes:

I don't know enough about Portage to answer that question. I do know
any package manager which made it into the standard distribution would
need to work for at least the big three platforms (Windows/Mac/*nix) :)

Being written in python - and hopefully integrated into Distutils -
why shouldn't it work on any platform that Python worked on?


Assumptions about directory structures, and the like. IIRC, Portage was
written for the Gentoo project, so it could assume that it was
installing stuff to a Gentoo system. Package management systems have a
distressing habit of infecting their *architecture* with these
assumptions. It makes adapting them difficult.

Also, Portage needs to execute subprocesses, something which is
notoriously platform dependent. 2.4's subprocess module should probably
be used here, but I don't think Portage does, yet. OTOH, it's Gentoo, so
it wouldn't surprise me, either. :-)

--
Robert Kern
rk***@ucsd.edu

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter
Jul 18 '05 #4
Robert Kern wrote:
Mike Meyer wrote:
Nick Coghlan <nc******@iinet .net.au> writes:

I don't know enough about Portage to answer that question. I do know
any package manager which made it into the standard distribution would
need to work for at least the big three platforms (Windows/Mac/*nix) :)

Being written in python - and hopefully integrated into Distutils -
why shouldn't it work on any platform that Python worked on?


Assumptions about directory structures, and the like.


That is already taken care by the distutils. The various PEPs already
describe a simple method how to store package metadata, and I will try
to follow these standards as close as possible. Of course, the PyPI
would have to be adapted for that.
IIRC, Portage was
written for the Gentoo project, so it could assume that it was
installing stuff to a Gentoo system. Package management systems have a
distressing habit of infecting their *architecture* with these
assumptions. It makes adapting them difficult. Also, Portage needs to execute subprocesses, something which is
notoriously platform dependent. 2.4's subprocess module should probably
be used here, but I don't think Portage does, yet. OTOH, it's Gentoo, so
it wouldn't surprise me, either. :-)


That's right, but I would just draw the line at Python's standard
library. What is possible with it, is done, what not, is left out.

regards,
Georg
Jul 18 '05 #5
Georg Brandl wrote:
Robert Kern wrote:
Mike Meyer wrote:
Nick Coghlan <nc******@iinet .net.au> writes:

I don't know enough about Portage to answer that question. I do know
any package manager which made it into the standard distribution would
need to work for at least the big three platforms (Windows/Mac/*nix) :)
Being written in python - and hopefully integrated into Distutils -
why shouldn't it work on any platform that Python worked on?
Assumptions about directory structures, and the like.

That is already taken care by the distutils.


More or less. I'm in the middle of packaging up ~40 Python packages for
the Mac[1]. For a "standard" packaging mechanism, distutils allows for
some bloody idiosyncratic ways to say "put these files there". This is a
hard problem, and it's not solved entirely by distutils.

I don't think anyone has satisfactorily solved the problem of
distributing data with libraries. Well, okay, the *distribution* isn't
the problem. Having the library be able to locate that data on all
platforms is difficult. Some packages will more-or-less hardcode
*nix-type paths which may be inappropriate even on some *nix-type
platforms (yes, PyX, I'm looking at you :-)). A general package system
like Portage has the freedom of being able to dictate these things. A
Python package manager does not. You can establish a standard for each
of the Big Three platforms, but it may not do you much good if the
libraries don't know about it.

CPAN is a closer analogue in this regard and would probably be a better
tool to study and copy from than Portage. I don't know much about it,
but how it responds to these issues will probably more instructive than
how Portage does.

You also have problems with distributing non-Python dependencies like
libraries. You can sometimes punt and require the user to have already
installed said libraries and headers to compile against. This will
probably only work for source distributions on *nixes. You can probably
get away with appropriately placed DLLs on Windows. Macs may be trickier
(we Mac users are fussy :-)).

Exercise: VTK with its Python and TCL libraries (why TCL? It's necessary
to use VTK with Tkinter, which the premier VTK-Python application,
MayaVi, uses) should package up cleanly for all 3 main platforms.
External libraries *and* data! It's a joy!

Some other "problem packages"[2] to practice on: PyX, ReportLab, PIL,
Scipy, matplotlib.

[1] http://www.scipy.org/wikis/featurerequests/MacEnthon
[2] In the sense that they are, to some extent, not painless to package
up in a very neat way on multiple platforms for one reason or another.
These are all very fine packages; packaging is a hard problem; and I'm
grateful for how well they *do* package up so far.
The various PEPs already
describe a simple method how to store package metadata, and I will try
to follow these standards as close as possible. Of course, the PyPI
would have to be adapted for that.


Fair enough. What parts of Portage do you intend to steal, then?

Please don't read any of this as discouragement. Python package
management desperately needs a champion, and I'm very glad to see
someone tackling these issues. I for one fully intend to use as much of
what you produce as I can to make MacEnthon fully upgradable.

--
Robert Kern
rk***@ucsd.edu

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter
Jul 18 '05 #6
Robert Kern wrote:
Georg Brandl wrote:
Robert Kern wrote:
Mike Meyer wrote:

Nick Coghlan <nc******@iinet .net.au> writes:

>I don't know enough about Portage to answer that question. I do know
>any package manager which made it into the standard distribution would
>need to work for at least the big three platforms (Windows/Mac/*nix) :)
Being written in python - and hopefully integrated into Distutils -
why shouldn't it work on any platform that Python worked on?

Assumption s about directory structures, and the like.

That is already taken care by the distutils.


More or less. I'm in the middle of packaging up ~40 Python packages for
the Mac[1]. For a "standard" packaging mechanism, distutils allows for
some bloody idiosyncratic ways to say "put these files there". This is a
hard problem, and it's not solved entirely by distutils.

I don't think anyone has satisfactorily solved the problem of
distributing data with libraries. Well, okay, the *distribution* isn't
the problem. Having the library be able to locate that data on all
platforms is difficult. Some packages will more-or-less hardcode
*nix-type paths which may be inappropriate even on some *nix-type
platforms (yes, PyX, I'm looking at you :-)). A general package system
like Portage has the freedom of being able to dictate these things. A
Python package manager does not. You can establish a standard for each
of the Big Three platforms, but it may not do you much good if the
libraries don't know about it.

CPAN is a closer analogue in this regard and would probably be a better
tool to study and copy from than Portage. I don't know much about it,
but how it responds to these issues will probably more instructive than
how Portage does.


CPAN, as I understand it, is a mirror system for packages, which
downloads packages and manages those. Correct me if I'm wrong, but the
actual installation is done by each package's Makefile.PL script and not
by CPAN itself.

This is also Portage's philosopy. It only issues those commands needed
to build and install the package (ideally "tar xzf package; cd package;
make install"). Of course one can do additional steps such as applying
patches, but where the files go etc. is the author's responsibility, not
the package system's.

[Problems with dependencies]

I am aware of these problems - however I feel like the package author
has to be the one to address them. There must be a cooperation between
author and packager to clear these issues.

About non-Python-dependencies: The plan is to add a "Non-Py-Requires:"
(or so) field to the metadata that is displayed to the user, and it will
be his responsibility to install these libraries/headers/files first.
The various PEPs already
describe a simple method how to store package metadata, and I will try
to follow these standards as close as possible. Of course, the PyPI
would have to be adapted for that.


Fair enough. What parts of Portage do you intend to steal, then?


Right, that's my weak point. I do not overly like the PEPs' way, but
integrating with distutils is the way to go.

regards,
Georg
Jul 18 '05 #7
On Sat, 25 Dec 2004 11:37:42 +0100, Georg Brandl <g.******@gmx.n et>
wrote:
what features would you expect of a Python package manager, similar to
CPAN or rubygems?


IMVHO it would be nice if it had a feature for "upload package/module
I have just developed" - maybe PyPi would fill up faster if it would.

--
It's a man's life in a Python Programming Association.
Jul 18 '05 #8
Bulba! <bu***@bulba.co m> writes:
On Sat, 25 Dec 2004 11:37:42 +0100, Georg Brandl <g.******@gmx.n et>
wrote:
what features would you expect of a Python package manager, similar to
CPAN or rubygems?


IMVHO it would be nice if it had a feature for "upload package/module
I have just developed" - maybe PyPi would fill up faster if it would.


Distutils already has a feature to do that for you.

<mike
--
Mike Meyer <mw*@mired.or g> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
Jul 18 '05 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
1910
by: Rodrigo Benenson | last post by:
Hi, Python is a great language. Python has great crossplatform support. Python has a great community. Python has excelent tools for other languages binding. Python has many good thing, and has some low points. To my eyes Python has two very bad things: 1- Tkinter is great for programmers but ugly to everyone. 2- Python software deployement can be a disaster.
29
2475
by: Stephen Ferg | last post by:
I am a very satisfied user of Python and have been for number of years. I would never willing use another language. I wish all good things for Python, and that moves me to express some thoughts about Python's future prospects. I submit that the future expansion of Python usage is constrained by Python's lack of a CPAN-like facility, and I submit that without a CPyAN Python will never even get close to achieving the degree of...
5
6989
by: Martyn Quick | last post by:
On my desk here at work I have a Mac G4 running Mac OS X v10.2.8. When I go into a terminal and type "python" up comes a nice python interface and all seems great. However when I type "import Tkinter" I'm greeted by the following error. >>> import Tkinter Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/lib/python2.2/lib-tk/Tkinter.py", line 35, in ?
217
9286
by: gyromagnetic | last post by:
The following url points to an article written by Damian Conway entitled "Ten Essential Development Practices": http://www.perl.com/pub/a/2005/07/14/bestpractices.html Althought the article has Perl as a focus, I thought that some of the general points made might be of interest to the Python community. It would certainly be interesting to put together an analogous version of this article that centers on Python. Best Regards,
9
2260
by: seberino | last post by:
I have been using distuils for a while and was wondering when Python Eggs (new project) is better? So basically Python Eggs precompiles and compresses binaries for you so you just have to load it to run your app? Chris
9
3766
by: Jack | last post by:
Installing a Python package is easy, most of time just "Setup.py install" However, setup.py doesn't seem to support an uninstall command. If I want to delete a package that I do not use any more, should I just manually delete the corresponding sub directory under Lib\site-packages?
2
2740
by: yinglcs | last post by:
Hi, Red hat 4 comes with python 2.3, and I am trying to upgrade to python 2.4. So I download and compile the source of python2.4. But as I run it I get the following error, can you please tell me how to fix it? # /root/src/Python-2.4.4/python ./nov/scripts/stressTestServlet.py ./ nov/scripts/stressTestInputFile 10 127.0.0.1
8
3284
by: Fett | last post by:
I am trying to find a wrapper to do linear programming within python. I am using an ubuntu machine and I have apt-get'd lp_solve, which works just fine. If someone knows of a wrapper that will work with that that'd be great. I also heard that scipy has a wrapper, however, I can't find any documentation on it, nor can I seem to find it with dir(). If anyone knows where there is good documentation on this I would love to use that (the...
20
3984
by: timotoole | last post by:
Hi all, On a (sun) webserver that I use, there is python 2.5.1 installed. I'd like to use sqlite3 with this, however sqlite3 is not installed on the webserver. If I were able to compile sqlite using a sun machine (I normally use linux machines) and place this in my lunix home account would I be able to use python and sqlite? Any thoughts? I know its a bit of a stretch ...
0
9895
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
11011
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10666
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
9503
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
7899
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
7071
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5735
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5929
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4546
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.