Help: Python2.3 & Python2.4 on RHEL4 x86_64

[Christopher Taylor]

RHEL comes with Python2.3 installed. A program I need to install
requires Python2.4

So I got Python2.4 from source and compiled it up. I configured it
with --prefix=/usr --exec-prefix=/usr and --enable-unicode=ucs4 . I
then make'd it and then make altinstall so that it didn't overwrite
the /usr/bin/Python link to /usr/bin/Python2.3 .

Well, for some reason, the arch dependent files did NOT get placed
properly in /usr/lib64/Python2.4, they instead went to
/usr/lib/Python2.4.

Also, when I tried to load pysqlite:
$ Python2.4

>>from pysqlite2 import test

I get the following traceback:

Traceback (most recent call last):
File "setup.py", line 24, in ?
import glob, os, re, sys
File "/usr/lib64/python2.3/glob.py", line 4, in ?
import fnmatch
File "/usr/lib64/python2.3/fnmatch.py", line 13, in ?
import re
File "/usr/lib64/python2.3/re.py", line 5, in ?
from sre import *
File "/usr/lib64/python2.3/sre.py", line 97, in ?
import sre_compile
File "/usr/lib64/python2.3/sre_compile.py", line 17, in ?
assert _sre.MAGIC == MAGIC, "SRE module mismatch"
AssertionError: SRE module mismatch

This basically means to me that Python2.4 is loading gloab.py from
/usr/lib64/Python2.3 insead of /usr/lib/Python2.4 (even thought I
wanted to install the related files in /usr/lib64/Python2.4)

Can someome please help!

Respectfully,
Christopher Taylor

[Martin v. Löwis]

Christopher Taylor schrieb:

This basically means to me that Python2.4 is loading gloab.py from
/usr/lib64/Python2.3 insead of /usr/lib/Python2.4 (even thought I
wanted to install the related files in /usr/lib64/Python2.4)

Can someome please help!

Can you please report what sys.path is? Do you have PYTHONPATH
set, by any chance? Python shouldn't normally look into the library
of a totally unrelated installation.

Regards,
Martin

[Martin v. Löwis]

Christopher Taylor schrieb:

This basically means to me that Python2.4 is loading gloab.py from
/usr/lib64/Python2.3 insead of /usr/lib/Python2.4 (even thought I
wanted to install the related files in /usr/lib64/Python2.4)

Can someome please help!

Can you please report what sys.path is? Do you have PYTHONPATH
set, by any chance? Python shouldn't normally look into the library
of a totally unrelated installation.

Regards,
Martin

[Christopher Taylor]

PYTHONPATH was the problem.

I had /usr/lib64/python2.3 included and that's why it was breaking. I
took it out and it works fine now. Unfortunately, it still puts the
lib files in /usr/lib instead of /usr/lib64. I'm assuming all I need
to do is set libdir accordingly and the files will get put in
/usr/lib64. How do I then build the library files for 32bits?

Respectfully,
Christopher Taylor

[Martin v. Löwis]

Christopher Taylor schrieb:

I had /usr/lib64/python2.3 included and that's why it was breaking. I
took it out and it works fine now. Unfortunately, it still puts the
lib files in /usr/lib instead of /usr/lib64. I'm assuming all I need
to do is set libdir accordingly and the files will get put in
/usr/lib64. How do I then build the library files for 32bits?

Can you explain what "then" is? If you set libdir so that all files go
into /usr/lib64, why do want *then* that those library files are built
for 32bits? Shouldn't they be built for 64bits if they go to /usr/lib64?

In any case, you usually select the target architecture by either
selecting the right compiler, or the right compiler flags. Just set
CC to a 32-bit-compiler, and Python should be built as 32-bit binaries.

Regards,
Martin

[Christopher Taylor]

Ok, so if I need to build the 32bit binaries, I just need to make sure
I pass the right argument through to the compiler then? (I'm trying
to build 32 bit and 64 bit binaries on the same system, but I'll wait
untill I get just the 64bit stuff built first before I tackle that)

Also, I've fooled around with passing --libdir=/usr/lib64 to the
configure script and for whatever reason, the Makefile isn't correctly
written. It always ends up copying the lib files to /usr/lib.

So I thought I'd manually edit the Makefile by editing the makefile on
lines 83 and 87 so they are now:
83: LIBDIR= $(exec_prefix)/lib64
87: SCRIPTDIR= $(prefix)/lib64

then:
make clean
make
make test
make altinstall
cd ..
python2.4

I get the following error message:
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
'import site' failed; use -v for traceback

so then I edited /etc/profile and added
export PYTHONHOME="/usr"

then restarted and I got ride of the "Could not find platform
in/dependent libraries" errors but I still get the " 'import site'
failed " error.

I did a python2.4 -v and below is the output:
# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
'import site' failed; traceback:
ImportError: No module named site
*** Now I'm lost ***

Just for sanity sakes I undid the changes to lines 83 & 87 of the make
file, did a "make clean", "make", "make test", "make altinstall",
"python2.4" followed by some goofing around and everything works fine.
So I'm guessing my manual edits of the makefile didn't go well. Any
idea what else I need to change?

Respectfully,
Christopher Taylor

[Martin v. Löwis]

Christopher Taylor schrieb:

Also, I've fooled around with passing --libdir=/usr/lib64 to the
configure script and for whatever reason, the Makefile isn't correctly
written. It always ends up copying the lib files to /usr/lib.

Ah, I see. The reason is pretty simple: Makefile.pre.in has

LIBDIR= $(exec_prefix)/lib

so it seems that LIBDIR isn't configurable.

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
'import site' failed; use -v for traceback

Just for sanity sakes I undid the changes to lines 83 & 87 of the make
file, did a "make clean", "make", "make test", "make altinstall",
"python2.4" followed by some goofing around and everything works fine.
So I'm guessing my manual edits of the makefile didn't go well. Any
idea what else I need to change?

You will have to debug this (can't do right now); see Modules/getpath.c,
and the definition of PYTHONPATH. It seems this isn't really supported.

Regards,
Martin

[Martin v. Löwis]

Christopher Taylor schrieb:

Also, I've fooled around with passing --libdir=/usr/lib64 to the
configure script and for whatever reason, the Makefile isn't correctly
written. It always ends up copying the lib files to /usr/lib.

Ah, I see. The reason is pretty simple: Makefile.pre.in has

LIBDIR= $(exec_prefix)/lib

so it seems that LIBDIR isn't configurable.

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
'import site' failed; use -v for traceback

Just for sanity sakes I undid the changes to lines 83 & 87 of the make
file, did a "make clean", "make", "make test", "make altinstall",
"python2.4" followed by some goofing around and everything works fine.
So I'm guessing my manual edits of the makefile didn't go well. Any
idea what else I need to change?

You will have to debug this (can't do right now); see Modules/getpath.c,
and the definition of PYTHONPATH. It seems this isn't really supported.

Regards,
Martin

[Christopher Taylor]

Ah, I see. The reason is pretty simple: Makefile.pre.in has

>
LIBDIR= $(exec_prefix)/lib

so it seems that LIBDIR isn't configurable.

So you would agree that this is a bug? I'll post on Python-Dev for
further advice. I'd like to fix this for the x86_64 community.

You will have to debug this (can't do right now); see Modules/getpath.c,
and the definition of PYTHONPATH. It seems this isn't really supported.

I'll ask on Python-Dev for more details on how to fix this.

Respectfully,
Christopher Taylor

[Martin v. Löwis]

Christopher Taylor schrieb:

>Ah, I see. The reason is pretty simple: Makefile.pre.in has

LIBDIR= $(exec_prefix)/lib

so it seems that LIBDIR isn't configurable.

So you would agree that this is a bug?

I personally think this is a bug in AMD64-Linux. Libraries should
be stored in /usr/lib, binaries in /usr/bin, etc. If they need
simultaneous installation of 32-bit binaries for compatibility,
they should store them in architecture-specific directories.

I'll post on Python-Dev for
further advice. I'd like to fix this for the x86_64 community.

The x86_64 community has been using Python for a while, and
apparently has solved this problem already. You should try
to find out how they did it.

Regards,
Martin

[Martin v. Löwis]

Christopher Taylor schrieb:

>Ah, I see. The reason is pretty simple: Makefile.pre.in has

LIBDIR= $(exec_prefix)/lib

so it seems that LIBDIR isn't configurable.

So you would agree that this is a bug?

I personally think this is a bug in AMD64-Linux. Libraries should
be stored in /usr/lib, binaries in /usr/bin, etc. If they need
simultaneous installation of 32-bit binaries for compatibility,
they should store them in architecture-specific directories.

I'll post on Python-Dev for
further advice. I'd like to fix this for the x86_64 community.

The x86_64 community has been using Python for a while, and
apparently has solved this problem already. You should try
to find out how they did it.

Regards,
Martin

[Christopher Taylor]

I personally think this is a bug in AMD64-Linux. Libraries should

be stored in /usr/lib, binaries in /usr/bin, etc. If they need
simultaneous installation of 32-bit binaries for compatibility,
they should store them in architecture-specific directories.

I disagree. From what I see, the error, as far as python is
considered, is not being able to specify the location where libs are
put, despite the fact that the --LIBDIR= option is listed. It just
happens to manifest itself in AMD64/EM64T Linux, specifically RH linux
where 64bit libs are put in /usr/lib64 and 32bit libs in /usr/lib.

The x86_64 community has been using Python for a while, and
apparently has solved this problem already. You should try
to find out how they did it.

Where might I go to look on how to get this working?

Respectfully,
Christopher Taylor

P.S. I posted on python-dev and I haven't seen my post show up yet,
nor any responces .... is that list moderated?

[Martin v. Löwis]

Christopher Taylor schrieb:

I disagree. From what I see, the error, as far as python is
considered, is not being able to specify the location where libs are
put, despite the fact that the --LIBDIR= option is listed. It just
happens to manifest itself in AMD64/EM64T Linux, specifically RH linux
where 64bit libs are put in /usr/lib64 and 32bit libs in /usr/lib.

Ok. One solution might be to remove the libdir option, then. Python
attempts to be "movable", i.e. the libraries are found relative to
the executable (via dirname(sys.executable)+"../lib/..."). If
libdir is supported, this approach must be given up - or libdir must
be given up.

>The x86_64 community has been using Python for a while, and
apparently has solved this problem already. You should try
to find out how they did it.

Where might I go to look on how to get this working?

The Linux distributions already provide Python binaries (I believe
Redhat does, too). You could study what they do to achieve that.

P.S. I posted on python-dev and I haven't seen my post show up yet,
nor any responces .... is that list moderated?

No, it should show up automatically.

Regards,
Martin

[Christopher Taylor]

Ok. One solution might be to remove the libdir option, then. Python

attempts to be "movable", i.e. the libraries are found relative to
the executable (via dirname(sys.executable)+"../lib/..."). If
libdir is supported, this approach must be given up - or libdir must
be given up.

Well, honestly, seeings how LIBDIR doesn't do anything ... I can see
the argument for removing it.
However, I would like to propose a more flexible solution: Perhaps a
../configure parameter called LIBLOC or something like that so that
LIBDIR = $(EXEC_PREFIX)/$(LIBLOC). Of course, LIBLOC should default
to "lib". This way the flexibility you mentioned above can be
maintained and the 64bit community (which will eventually be the
entire community) can move forward with minimal hassle.

The Linux distributions already provide Python binaries (I believe
Redhat does, too). You could study what they do to achieve that.

Yes, this is true ... but they do not package the most up-to-date
version .. which I need.

Respectfully,
Christopher Taylor

[Martin v. Löwis]

Christopher Taylor schrieb:

>The Linux distributions already provide Python binaries (I believe
Redhat does, too). You could study what they do to achieve that.

Yes, this is true ... but they do not package the most up-to-date
version .. which I need.

Sure. However, I still think they have solved the problem already -
if just for older versions. The build procedure hasn't changed that
much, so the solution might be applicable to the version you want
as well.

Regards,
Martin

[Christopher Taylor]

Does anyone have any comments on my proposed solution?

However, I would like to propose a more flexible solution: Perhaps a
./configure parameter called LIBLOC or something like that so that
LIBDIR = $(EXEC_PREFIX)/$(LIBLOC). Of course, LIBLOC should default
to "lib". This way the flexibility you mentioned above can be
maintained and the 64bit community (which will eventually be the
entire community) can move forward with minimal hassle.

Respectfully,
Christopher Taylor