Location of bytecode files (pyc)

[Rim]

Hi,

How can I control the location of the bytecode files?

The reason I ask is I want to place the bytecode in a directory
different than source code for backup reasons. Our backup system is
directory based and I only want to have source code backed up, not
object files or pyc files.

Thanks,
-Rim

[Martin v. Löwis]

ri*******@yahoo .com (Rim) writes:

How can I control the location of the bytecode files?

You currently can't. See PEP 304, though, at

http://www.python.org/peps/pep-0304.html

Comments on the PEP are encouraged: If you won't comment now on
whether the proposed change would solve your problem, it might be that
you find something useless got implemented later.

Regards,
Martin

[Kerim]

"Rim" <ri*******@yaho o.com> wrote in message
news:6f******** *************** ***@posting.goo gle.com...

Hi,

How can I control the location of the bytecode files?

The reason I ask is I want to place the bytecode in a directory
different than source code for backup reasons. Our backup system is
directory based and I only want to have source code backed up, not
object files or pyc files.
If you're on Windows say:

del /S *.pyc

to remove all .pyc's from cwd and below.

Thanks,
-Rim

[Bengt Richter]

On 03 Sep 2003 07:12:07 +0200, ma****@v.loewis .de (Martin v. =?iso-8859-15?q?L=F6wis?=) wrote:

ri*******@yaho o.com (Rim) writes:

How can I control the location of the bytecode files?

You currently can't. See PEP 304, though, at

http://www.python.org/peps/pep-0304.html

Comments on the PEP are encouraged: If you won't comment now on
whether the proposed change would solve your problem, it might be that
you find something useless got implemented later.

Personally, I am a minimalist when it comes to environment variables.
IMO that name space has the same problems as any global namespace,
and since a single default set of user's environment variables tends to
presented to most programs s/he runs from a command window, the name space
usage tends towards a hodgepodge and/or wrapping apps in setup scripts (which
can work fine, but I still don't like it as a standard way to go).

IMO the os.environ name space should be treated like a root directory name space,
and not have application data per se in it (with reluctant exceptions where it is used wholesale
as in CGI). Rather, IMO, and only if necessary in the first place, it should be used to specify
location or search path info for config data, not *be* config data.

And a user-set environment variable should not be able to cause a bypass of root/admin-defined
config info where the distinction is necessary.

(The PYTHONBYTECODEB ASE variable does refer to a directory, but now that makes two variables,
counting PYTHONPATH, and where will it end?)

Provision for admin/root level config data separate from user preference and session state type
config data should be made as necessary and desirable, but secondary/user config data search
should be controllable by the primary/root/admin config data (which e.g. could say to ignore
or use user-controlled/attempted-control environment variables etc.).

This would seem to imply a standard place to look for root/admin level config data, not directed
by env variable. E.g., a read-only file in the same directory as the python interpreter executable,
with, say, .conf or .cfg appended to the name. *That* file can then specify how/whether to look
for user config stuff etc., or specify password access to some features, etc. etc., if we wind up
doing restricted exec stuff.

A user config file overriding any other *user* config info could be specified by command line option,
e.g., -cfg myConfigFile.co nf, and whether this or other command line options were allowed could be
(and should be able to be when control is necessary) specified in the root/admin config file.

.... just my .02USD

Regards,
Bengt Richter

[Rim]

> If you're on Windows say:

del /S *.pyc

to remove all .pyc's from cwd and below.

Yes, and in linux 'find . -name "*.pyc" -exec rm -rf {} \;', but that
is not the point. I did not ask about how to remove pyc files, I asked
about placing them far from source code, in another directory, so they
would not be seen by the backup software.

The backup software runs continuously day, night, weekends, all the
time. Sweeping the pyc's under the carpet each time the backup is
about to be saving my files is impossible. That is why we store all
gcc object files and executables we produce on non-backed up
filesystems by prefixing target filenames in makefiles with a variable
like $OBJECT_DIR and $EXEC_DIR.

We save hundreds of Gigs of disk space this way.

PEP 304 appears to give me what I need.

As Bengt suggest, adding yet another env var to control the operation
of an application polutes the name space for all applications, but
each app only needs to look at the variables it is concerned with.
PYTHONBYTECODEB ASE is a cleaner approach that has less overhead than a
configuration file. Also, env variables are directly usable in
makefiles, which makes them very attractive and easy to use for
managing where compiler output goes.

Say you work on multiple projects simultaneously with different
locations for the byte code. The configuration files approach becomes
an administrative burden with conditionals based on the project name
in your config file.

The typical way we handle project specific configurations is through
environment variables that get set in the startup files. Depending on
a PROJECT environment variable, the PYTHONBYTECODEB ASE can easily be
set to different directories without creating additional .cfg files.

Regards,
-Rim

[Peter Hansen]

Rim wrote:

If you're on Windows say:

del /S *.pyc

to remove all .pyc's from cwd and below.

Yes, and in linux 'find . -name "*.pyc" -exec rm -rf {} \;', but that
is not the point. I did not ask about how to remove pyc files, I asked
about placing them far from source code, in another directory, so they
would not be seen by the backup software.

The backup software runs continuously day, night, weekends, all the
time. Sweeping the pyc's under the carpet each time the backup is
about to be saving my files is impossible. That is why we store all
gcc object files and executables we produce on non-backed up
filesystems by prefixing target filenames in makefiles with a variable
like $OBJECT_DIR and $EXEC_DIR.

You have a fancy backup system like this, involving hundreds of Gigs,
and it can't apply a simple filter to avoid .pyc and .obj files?!

-Peter

[Skip Montanaro]

Bengt,

Thanks for your feedback...

MvL> http://www.python.org/peps/pep-0304.html

MvL> Comments on the PEP are encouraged: ...

Bengt> Personally, I am a minimalist when it comes to environment
Bengt> variables. IMO that name space has the same problems as any
Bengt> global namespace, and since a single default set of user's
Bengt> environment variables tends to presented to most programs s/he
Bengt> runs from a command window, the name space usage tends towards a
Bengt> hodgepodge and/or wrapping apps in setup scripts (which can work
Bengt> fine, but I still don't like it as a standard way to go).

You can set sys.bytecodebas e explicitly, though you might have trouble
setting it early enough to push all your .pyc files where you want. Note
that the patch associated with PEP 304 is written in C (import.c and
sysmodule.c are affected).

Bengt> Provision for admin/root level config data separate from user
Bengt> preference and session state type config data should be made as
Bengt> necessary and desirable, but secondary/user config data search
Bengt> should be controllable by the primary/root/admin config data
Bengt> (which e.g. could say to ignore or use
Bengt> user-controlled/attempted-control environment variables etc.).

Do you want Python to locate and load a config file at startup? By the time
a config file parser is loaded and sys.bytecodebas e set, lots of .pycs may
well have already been generated in the wrong place. This shouldn't happen
with the patch I created.

Bengt> This would seem to imply a standard place to look for root/admin
Bengt> level config data, not directed by env variable. E.g., a
Bengt> read-only file in the same directory as the python interpreter
Bengt> executable, with, say, .conf or .cfg appended to the name. *That*
Bengt> file can then specify how/whether to look for user config stuff
Bengt> etc., or specify password access to some features, etc. etc., if
Bengt> we wind up doing restricted exec stuff.

Like I said, lots of other stuff will likely have been imported by the time
you realize your .pyc files belong "over there" instead of where they're
being written, or were you volunteering to write a C version of ConfigParser
which is statically linked into the interpreter? ;-)

Skip

[Skip Montanaro]

Rim> As Bengt suggest, adding yet another env var to control the
Rim> operation of an application polutes the name space for all
Rim> applications, but each app only needs to look at the variables it
Rim> is concerned with. PYTHONBYTECODEB ASE is a cleaner approach that
Rim> has less overhead than a configuration file. Also, env variables
Rim> are directly usable in makefiles, which makes them very attractive
Rim> and easy to use for managing where compiler output goes.

You can also set sys.bytecodebas e in your application code instead of
relying on PYTHONBYTECODEB ASE. That may not be helpful when doing a big pyc
compile, as when running compileall.py. In that case, you can simply set
PYTHONBYTECODE base for that specific command. Still, in your environment,
it sounds like you'd want to set it in the /etc/profile (or similar). It
could still be set to a user-specific directory, just one that's on a
filesystem which is not backed up.

The intention was that the most common usage would be for a person to
execute something like

PYTHONBYTECODEB ASE=/tmp/$USER/python ; export PYTHONBYTECODEB ASE

in their .profile (or equivalent for non-sh-style shells or platforms).
Clearly, other usage is possible however.

Skip

[Rim]

> The intention was that the most common usage would be for a person to

execute something like

PYTHONBYTECODEB ASE=/tmp/$USER/python ; export PYTHONBYTECODEB ASE

This is exactly how I intent to use it.

Regards,
-Rim