473,836 Members | 1,538 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Eggs, VirtualEnv, and Apt - best practices?

Hello all,

Our development group at work seems to be heading towards adopting
python as one of our standard "systems languages" for internal
application development (yeah!). One of the issues that's come up is
the problem with apt (deb packages) vs eggs, vs virtual environments.
We're probably gonna end up using Pylons or TurboGears for web-based
apps, and I've recommended virtualenv, but one of the other developers
has had some "inconsistencie s" when mixing systems with python installed
from apt (all our servers are debian or ubuntu based) vs when installed
under virtualenv.

I have basically recommended that we only install the python base (core
language) from apt, and that everything else should be installed into
virtual environments. But I wanted to check to see how other enterprises
are handling this issue? Are you building python from scratch, or using
specific sets of .deb packages, or some other process.

Any insight into the best way to have a consistent, repeatable,
controllable development and production environment would be much
appreciated.

Suggestions on build/rollout tools (like zc.buildout, Paver, etc) would
also be appreciated.

Thanks!!!

-Scott
Sep 25 '08 #1
12 3946
Scott Sharkey wrote:
Any insight into the best way to have a consistent, repeatable,
controllable development and production environment would be much
appreciated.
you have just described OS package building ;)

I can't speak for everybody, but supporting multiple platforms (PHP, Perl,
Python, Java) we found that the only way to stay consistent is to use OS
native packaging tools (in your case apt and .deb ) and if you're missing
something - roll your own package. After a while you accumulate plenty of
templates to chose from when you need yet-another-library not available
upstream in your preferred package format. Remember that some python tools
might depend on non-python packages, so the only way to make sure all that
is consistent across environment - use unified package management.

Sorry, not specific pointers though as we're redhat shop and debs are not
our everyday business.
Sep 25 '08 #2
Dmitry S. Makovey wrote:
you have just described OS package building ;)

I can't speak for everybody, but supporting multiple platforms (PHP, Perl,
Python, Java) we found that the only way to stay consistent is to use OS
native packaging tools (in your case apt and .deb ) and if you're missing
something - roll your own package. After a while you accumulate plenty of
templates to chose from when you need yet-another-library not available
upstream in your preferred package format. Remember that some python tools
might depend on non-python packages, so the only way to make sure all that
is consistent across environment - use unified package management.
you're speaking for lots of organizations, at least.

rpm/debs from supplier's repository
subversion (or equivalent) -locally built rpm/debs
+ organization's favourite deployment tools
---------------------------------
deployed application

</F>

Sep 25 '08 #3
Scott Sharkey schrieb:
Hello all,

Our development group at work seems to be heading towards adopting
python as one of our standard "systems languages" for internal
application development (yeah!). One of the issues that's come up is
the problem with apt (deb packages) vs eggs, vs virtual environments.
We're probably gonna end up using Pylons or TurboGears for web-based
apps, and I've recommended virtualenv, but one of the other developers
has had some "inconsistencie s" when mixing systems with python installed
from apt (all our servers are debian or ubuntu based) vs when installed
under virtualenv.
I have basically recommended that we only install the python base (core
language) from apt, and that everything else should be installed into
virtual environments. But I wanted to check to see how other enterprises
are handling this issue? Are you building python from scratch, or using
specific sets of .deb packages, or some other process.

Any insight into the best way to have a consistent, repeatable,
controllable development and production environment would be much
appreciated.
This is the exact way we are deploying our software. You can even use
the virtualenv --no-site-packages option to completely isolate the VE
from the underlying system site-packages.

I would recommend that all you install into the system python is
virtualenv, and maybe some uncritical C-modules such as psycopg2.

Currently there is much going on regarding setuptools. A fork,
"Distribute " has been announced, and "pyinstall" by Ian Bicking, an
easy_install replacement that deals with some of it's ancestors
shortcomings.

Then people (shameless plug warning: including me) are working on
"eggbasket" , a PYPI-clone that allows to have a local repository of eggs
so that you don't fall prey to old versions not longer available on PYPI.

Eggbasket will feature "easterbunn y", a tool to publish a virtualenv as
whole to the eggbasket and also keep track of the precise version set
uploaded. Through a specific url on eggbasket you can then limit the
contents of eggbasket to that exact version set - which helps dealing
with subtle (or not so subtle) version conflicts.

I personally can say that I'm really thrilled by the prospects of all
these developments. And as much bad rap as setuptools had here and
elsewhere, sometimes rightfully so - it certainly does a lot of stuff
right, and pushing the whole stack of tools to manage software
dependencies in Python to the next level is of great value.

Diez
Sep 25 '08 #4
Scott Sharkey <ss******@linux unlimited.comwr ote:
B Our development group at work seems to be heading towards adopting
python as one of our standard "systems languages" for internal
application development (yeah!). One of the issues that's come up is
the problem with apt (deb packages) vs eggs, vs virtual environments.
We're probably gonna end up using Pylons or TurboGears for web-based
apps, and I've recommended virtualenv, but one of the other developers
has had some "inconsistencie s" when mixing systems with python installed
from apt (all our servers are debian or ubuntu based) vs when installed
under virtualenv.

I have basically recommended that we only install the python base (core
language) from apt, and that everything else should be installed into
virtual environments. But I wanted to check to see how other enterprises
are handling this issue? Are you building python from scratch, or using
specific sets of .deb packages, or some other process.

Any insight into the best way to have a consistent, repeatable,
controllable development and production environment would be much
appreciated.
I'll admit to not knowing what you mean by virtual environment...

In our debian systems we use python from apt and all modules from apt.

If there is a module we can't find then we build it into a .deb using
setup.py to build an rpm and converting to a .deb.

The app is then tested with "etch" or whatever.

If easy_install could build debs that would be really helpful!
Suggestions on build/rollout tools (like zc.buildout, Paver, etc) would
also be appreciated.
Use setup.py to build into .debs is what we do.

--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick
Sep 25 '08 #5
Dmitry S. Makovey schrieb:
Scott Sharkey wrote:
>Any insight into the best way to have a consistent, repeatable,
controllable development and production environment would be much
appreciated.

you have just described OS package building ;)

I can't speak for everybody, but supporting multiple platforms (PHP, Perl,
Python, Java) we found that the only way to stay consistent is to use OS
native packaging tools (in your case apt and .deb ) and if you're missing
something - roll your own package. After a while you accumulate plenty of
templates to chose from when you need yet-another-library not available
upstream in your preferred package format. Remember that some python tools
might depend on non-python packages, so the only way to make sure all that
is consistent across environment - use unified package management.
That this is a desirable goal can't be argued against. Yet two big
hurdles make it often impractical to be dogmatic about that:

- different OS. I for one don't know about a package management tool
for windows. And while our servers use Linux (and I as developer as
well), all the rest of our people use windows. No use telling them to
apt-get instal python-imaging.

- keeping track of recent developments. In the Python webframework
world for example (which the OP seems to be working with), things move
fast. Or extremly slow, regarding releases. Take Django - until 2 month
ago, there hasn't been a stable release for *years*. Virtually everybody
was working with trunk. And given the rather strict packaging policies
of debian and consorts, you'd be cut off of recent developments as well
as of bugfixes.

Diez
Sep 25 '08 #6
Nick Craig-Wood schrieb:
Scott Sharkey <ss******@linux unlimited.comwr ote:
B Our development group at work seems to be heading towards adopting
> python as one of our standard "systems languages" for internal
application development (yeah!). One of the issues that's come up is
the problem with apt (deb packages) vs eggs, vs virtual environments.
We're probably gonna end up using Pylons or TurboGears for web-based
apps, and I've recommended virtualenv, but one of the other developers
has had some "inconsistencie s" when mixing systems with python installed
from apt (all our servers are debian or ubuntu based) vs when installed
under virtualenv.

I have basically recommended that we only install the python base (core
language) from apt, and that everything else should be installed into
virtual environments. But I wanted to check to see how other enterprises
are handling this issue? Are you building python from scratch, or using
specific sets of .deb packages, or some other process.

Any insight into the best way to have a consistent, repeatable,
controllable development and production environment would be much
appreciated.

I'll admit to not knowing what you mean by virtual environment...
virtualenv is a simple tool to create isolated python environments where
you can install whatever packages you like without interfering with
other installations, even allowing to install conflicting package
versions (per VE of course they must be without conflicts).

Diez
Sep 25 '08 #7
Diez B. Roggisch wrote:
Dmitry S. Makovey schrieb:
>Scott Sharkey wrote:
>>Any insight into the best way to have a consistent, repeatable,
controllabl e development and production environment would be much
appreciated .

you have just described OS package building ;)
Except that we do need multiple different environments on one server,
and also have issues where our servers may be Windows.
>I can't speak for everybody, but supporting multiple platforms (PHP,
Perl,
>Python, Java) we found that the only way to stay consistent is to use OS
native packaging tools (in your case apt and .deb ) and if you're
missing
>something - roll your own package. After a while you accumulate
plenty of
>templates to chose from when you need yet-another-library not available
upstream in your preferred package format. Remember that some python
tools
>might depend on non-python packages, so the only way to make sure
all that
>is consistent across environment - use unified package management.

That this is a desirable goal can't be argued against. Yet two big
hurdles make it often impractical to be dogmatic about that:
>
- different OS. I for one don't know about a package management tool
for windows. And while our servers use Linux (and I as developer as
well), all the rest of our people use windows. No use telling them to
apt-get instal python-imaging.

Exactly!
- keeping track of recent developments. In the Python webframework
world for example (which the OP seems to be working with), things move
fast. Or extremly slow, regarding releases. Take Django - until 2 month
ago, there hasn't been a stable release for *years*. Virtually everybody
was working with trunk. And given the rather strict packaging policies
of debian and consorts, you'd be cut off of recent developments as well
as of bugfixes.

Very much the case. Most of debian's packages for python are woefully
out of date, it seems. And then we're at the whim of the os provider as
to when updates happen, rather than being controlled by our staff.

I am very interested in the eggbasket project - that's something that's
been needed for a while. And I'm aware of the setuptools fork, and the
discussion on the distutils sig mailing list.

Thanks.
-Scott

Sep 25 '08 #8
Diez B. Roggisch wrote:
- different OS. I for one don't know about a package management tool
for windows. And while our servers use Linux (and I as developer as
well), all the rest of our people use windows. No use telling them to
apt-get instal python-imaging.
that is a very valid point, but it seemed that Scott has homogeneous
environment: Debian/Ubuntu so my post was relative to the original request.
I agree that when you throw Windows/MacOS into the mix things
become "interestin g". But then it's better when your developers develop on
server/platform they are going to be using, using same stack they going to
face in production etc. It all depends on requirements and current
practices in company.
- keeping track of recent developments. In the Python webframework
world for example (which the OP seems to be working with), things move
fast. Or extremly slow, regarding releases. Take Django - until 2 month
ago, there hasn't been a stable release for *years*. Virtually everybody
was working with trunk. And given the rather strict packaging policies
of debian and consorts, you'd be cut off of recent developments as well
as of bugfixes.
that definitely becomes tricky however not impossible to track. You do need
a common snapshot for all developers to use anyway - so why not just
package it up?

Note: I do agree that depending on environment/development
practices/policies/etc my statement might become invalid or useless.
However when you're dealing with homogeneous environment or you require
development and testing to be done on your servers running targeted
application stack - things become much easier to manage :)

Sep 25 '08 #9
that is a very valid point, but it seemed that Scott has homogeneous
environment: Debian/Ubuntu so my post was relative to the original request.
I agree that when you throw Windows/MacOS into the mix things
become "interestin g". But then it's better when your developers develop on
server/platform they are going to be using, using same stack they going to
face in production etc. It all depends on requirements and current
practices in company.
Well, you certainly want a desktop-orientied Linux for users, so you
chose ubuntu - but then on the server you go with a more stable debian
system. Even though the both have the same technical and even package
management-base, they are still incompatible wrt to package versions for
python.

And other constraints such as Photoshop not being available for Linux
can complicate things further.
> - keeping track of recent developments. In the Python webframework
world for example (which the OP seems to be working with), things move
fast. Or extremly slow, regarding releases. Take Django - until 2 month
ago, there hasn't been a stable release for *years*. Virtually everybody
was working with trunk. And given the rather strict packaging policies
of debian and consorts, you'd be cut off of recent developments as well
as of bugfixes.

that definitely becomes tricky however not impossible to track. You do need
a common snapshot for all developers to use anyway - so why not just
package it up?
I do, but based on Python eggs. They are platform independent (at
ultimo, you can use the source distribution, albeit that sux for windows
most of the time), and as I explained in my other post - things are
moving in the right direction.

Don't get me wrong - I love .deb-based systems. But if using them for my
development means that I have to essentially create a full zoo of
various packages *nobody else* uses - I rather stick with what's working
for me.

Diez
Sep 25 '08 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

136
9482
by: Matt Kruse | last post by:
http://www.JavascriptToolbox.com/bestpractices/ I started writing this up as a guide for some people who were looking for general tips on how to do things the 'right way' with Javascript. Their code was littered with document.all and eval, for example, and I wanted to create a practical list of best practices that they could easily put to use. The above URL is version 1.0 (draft) that resulted. IMO, it is not a replacement for the FAQ,...
10
3490
by: jojobar | last post by:
Hello, I am trying to use vs.net 2005 to migrate a project originally in vs.net 2003. I started with creation of a "web site", and then created folders for each component of the site. I read somewhere that each folder under the "web site" is compiled in separate assembly. I however, did not find that the "web site" creation in vs.net 2005 created any AssemblyInfo.cs file.
9
2260
by: seberino | last post by:
I have been using distuils for a while and was wondering when Python Eggs (new project) is better? So basically Python Eggs precompiles and compresses binaries for you so you just have to load it to run your app? Chris
1
2155
by: Srijit Kumar Bhadra | last post by:
I have browsed the following links 1) http://peak.telecommunity.com/DevCenter/EasyInstall 2) When Python *Eggs* better than Python *distutils*?? What's Eggs? (http://tinyurl.com/m8dyd) But I am still not clear what to do with an .egg file. For example, if I have a basic Python installation (http://www.python.org/ftp/python/2.4.3/python-2.4.3.msi) what should I do with a typical .egg file (e.g....
0
1316
by: Jeff Rush | last post by:
Eggs are important to Python and will become more so over the next few years, if understood and embraced by the community. They are a key competitive feature - I've been asked so many times what is the Python equivalent to CPAN and finally we're developing an answer. At PyCon 2007 we need a solid set of talks about eggs, broken down into just using them, how to build and distribute them, and then an advanced talk on some of the more...
2
2194
by: nisimura | last post by:
Hi, I'm trying to use MySQL Python module on Cygwin. Since there is no binary package, I compiled MySQL C client library and Python module (http://sourceforge.net/projects/mysql-python) manually. But when I was trying to load MySQLdb module, following errors happened. Traceback (most recent call last):
4
1188
by: Damjan | last post by:
There are several attempts to allow python to work with per user (or even per session) 'site-packages' like virtualpython / workingenv / virtualenv. But they all have their own shortcomings and quirks. My question is, shoudn't it be enough to set PYTHONPATH and everything automagically to work then? Is there some work done on this for python 3.0 or 2.6 perhaps?
1
1210
by: oj | last post by:
Hi all! As is about to become apparent, I really don't know what I'm doing when it comes to using eggs. I'm writing some software that is going to be deployed on a machine as a number of eggs. Which is all well and good. These eggs all end up depending on each other; modules in egg A want to import modules in egg B etc.
5
1561
by: jryanfallon | last post by:
I am trying to write a program to sort eggs, extra eggs are discarded the program must have a pop up box for enter the eggs box type extra large - 12 dozen (144 eggs) large - 6 dozen (72 eggs) medium - 3 dozen (36 eggs) small - 1 dozen (12 eggs) example output picture box
0
9677
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10560
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
9389
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
7798
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
6986
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5662
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5838
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4468
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
4033
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.