Building things with setup.py

[James Stroud]

Hello All,

This is annoying. I am trying to build scipy right now but every .so
file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
to the ld flags.

Main Question: When building with "setup.py", where can I include this
information so I don't have to link every .so file by hand?

Also, why would this not be caught by the developers of scipy and numpy?
What factors go into needing to specify libraries and to properly link
..so files manually? Certainly the enthought didn't forget to test the
build process, so why would my machine be special in this regard?

I'm running Linux FC4 on a dual intel p4 (~3.2 GHz) with sundry
libraries in non-standard places. Is this non-standard part what's
killing me?

James

--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/

[James Stroud]

James Stroud wrote:

Hello All,

This is annoying. I am trying to build scipy right now but every .so
file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
to the ld flags.

Main Question: When building with "setup.py", where can I include this
information so I don't have to link every .so file by hand?

Also, why would this not be caught by the developers of scipy and numpy?
What factors go into needing to specify libraries and to properly link
.so files manually? Certainly the enthought didn't forget to test the
build process, so why would my machine be special in this regard?

I'm running Linux FC4 on a dual intel p4 (~3.2 GHz) with sundry
libraries in non-standard places. Is this non-standard part what's
killing me?

James

Well I added those arguments to $LDFLAGS, but it seemed like a pretty
miserable hack, especially the "-shared" part.

I'm still curious how to make a 3rd party setup.py work correctly.

James

--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/

[Robert Kern]

James Stroud wrote:

Hello All,

This is annoying. I am trying to build scipy right now but every .so
file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
to the ld flags.

Main Question: When building with "setup.py", where can I include this
information so I don't have to link every .so file by hand?

If it's actually necessary,

python setup.py build_ext -lmylib

Also, why would this not be caught by the developers of scipy and numpy?

Because we've never seen it before. There's no reason that you should have to
add those libraries manually. Can you post your build log (the one that fails)
to scipy-dev?

http://www.scipy.org/Mailing_Lists

What factors go into needing to specify libraries and to properly link
.so files manually? Certainly the enthought didn't forget to test the
build process, so why would my machine be special in this regard?

I'm running Linux FC4 on a dual intel p4 (~3.2 GHz) with sundry
libraries in non-standard places. Is this non-standard part what's
killing me?

Possibly. Won't know more until I see the build log.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Robert Kern]

James Stroud wrote:

Hello All,

This is annoying. I am trying to build scipy right now but every .so
file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
to the ld flags.

I'm running Linux FC4 on a dual intel p4 (~3.2 GHz) with sundry
libraries in non-standard places. Is this non-standard part what's
killing me?

It's possible that distutils is not finding the information about how the python
interpreter was built. Did you build your own python(1)? Is the information in
$PREFIX/lib/python2.4/config/ accurate?

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[James Stroud]

Robert Kern wrote:

James Stroud wrote:

>Hello All,

This is annoying. I am trying to build scipy right now but every .so
file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl
-shared" to the ld flags.

>I'm running Linux FC4 on a dual intel p4 (~3.2 GHz) with sundry
libraries in non-standard places. Is this non-standard part what's
killing me?

It's possible that distutils is not finding the information about how
the python interpreter was built. Did you build your own python(1)? Is
the information in $PREFIX/lib/python2.4/config/ accurate?

I did build my own python 2.5, yesterday, requiring me to rebuild all
extensions. Everything I do is compiled by hand as joe-user. I'm in a
situation where I can't do RPM (and I don't have root on my work machine
(theoretically ;-)) so, to be a good joe-user, everything I add goes
into the prefix:

$HOME/Programs

This is the listing from $HOME/Programs/lib/python2.5/config:

euler 6% ls
total 4092
8 config.c 12 install-sh* 44 Makefile 8 python.o
8 Setup.config
8 config.c.in 3960 libpython2.5.a 12 makesetup* 24 Setup
8 Setup.local

The build process, by the way, required my copying libpython2.5.a to
$HOME/Programs/lib.

The text files Setup.config and Setup.local do not seem to have terribly
specific information in them. Which file in particular should I inspect?

I did not capture output from the build and I could not find a file with
the word "log" in it that appears to be a build log. The jist of the
problem is that first it can't find symbols from libpython2.5, then from
libthread, etc. Then, it complains about no "MAIN__" when linking the
..so files with g77 and no "main" with gcc (which is curious), so I must
include the -shared flag, after including -llibrary type flags for all
of the libraries it doesn't know about.

If its necessary, I can run setup.py build again and send the output to
a file and post that to scipy-dev if you think it might be helpful.

James

--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/

[Robert Kern]

James Stroud wrote:

I did build my own python 2.5, yesterday, requiring me to rebuild all
extensions.

Do other extensions build correctly? If so, it's beginning to look like a
problem in numpy.distutils .

Everything I do is compiled by hand as joe-user. I'm in a
situation where I can't do RPM (and I don't have root on my work machine
(theoretically ;-)) so, to be a good joe-user, everything I add goes
into the prefix:

$HOME/Programs

This is the listing from $HOME/Programs/lib/python2.5/config:

euler 6% ls
total 4092
8 config.c 12 install-sh* 44 Makefile 8 python.o
8 Setup.config
8 config.c.in 3960 libpython2.5.a 12 makesetup* 24 Setup
8 Setup.local

The build process, by the way, required my copying libpython2.5.a to
$HOME/Programs/lib.

Hmm. That doesn't quite sound right, but it's been a while since I compiled the
interpreter from source.

The text files Setup.config and Setup.local do not seem to have terribly
specific information in them. Which file in particular should I inspect?

Makefile has most of that information. You can verify that distutils is finding
it like so:

>>from distutils import sysconfig
sysconfig.get _makefile_filen ame()

'/Library/Frameworks/Python.framewor k/Versions/2.4/lib/python2.4/config/Makefile'

>>d = sysconfig.parse _makefile(_)
import pprint
pprint.pprint (d)

{'AR': 'ar',
...
}

I did not capture output from the build and I could not find a file with
the word "log" in it that appears to be a build log. The jist of the
problem is that first it can't find symbols from libpython2.5, then from
libthread, etc. Then, it complains about no "MAIN__" when linking the
.so files with g77 and no "main" with gcc (which is curious), so I must
include the -shared flag, after including -llibrary type flags for all
of the libraries it doesn't know about.

What versions of gcc and g77 are you using?

If its necessary, I can run setup.py build again and send the output to
a file and post that to scipy-dev if you think it might be helpful.

Yes, that is what I intended.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Robert Kern]

James Stroud wrote:

Well I added those arguments to $LDFLAGS, but it seemed like a pretty
miserable hack, especially the "-shared" part.

Hmmm. Did you have $LDFLAGS set to anything (or more likely, nothing) before?
I.e. did you have something like this?

export LDFLAGS=""
python setup.py build

If such environment variables are present, I think that numpy.distutils may
clobber the values that are inferred from the configuration.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Martin v. Löwis]

James Stroud schrieb:

This is annoying. I am trying to build scipy right now but every .so
file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
to the ld flags.

That shouldn't be necessary. Linking without this should work just fine.

That way, since you made libpython2.5.a a static library, you link an
entire Python interpreter into each and every extension module. This
should not be done.

What happens if you omit these flags?

Regards,
Martin

[James Stroud]

Robert Kern wrote:

James Stroud wrote:

>I did build my own python 2.5, yesterday, requiring me to rebuild all
extensions.

Do other extensions build correctly? If so, it's beginning to look like
a problem in numpy.distutils .

It seems that every thing has built without incident (numarray, numeric,
mxtexttools, pil, reportlab, just to name a few), with the exception,
strangely enough, of python2.5 itself. Python required hiding previous
python installations.

>Everything I do is compiled by hand as joe-user. I'm in a situation
where I can't do RPM (and I don't have root on my work machine
(theoretical ly ;-)) so, to be a good joe-user, everything I add goes
into the prefix:

$HOME/Programs

This is the listing from $HOME/Programs/lib/python2.5/config:

euler 6% ls
total 4092
8 config.c 12 install-sh* 44 Makefile 8 python.o
8 Setup.config
8 config.c.in 3960 libpython2.5.a 12 makesetup* 24 Setup
8 Setup.local

The build process, by the way, required my copying libpython2.5.a to
$HOME/Programs/lib.

Hmm. That doesn't quite sound right, but it's been a while since I
compiled the interpreter from source.

Sorry. To clarify, making libpython2.5a available in a $LD_LIBRARY_PAT H
was necessary to build numpy and scipy.

>The text files Setup.config and Setup.local do not seem to have
terribly specific information in them. Which file in particular should
I inspect?

Makefile has most of that information. You can verify that distutils is
finding it like so:

>>from distutils import sysconfig
>>sysconfig.get _makefile_filen ame()

'/Library/Frameworks/Python.framewor k/Versions/2.4/lib/python2.4/config/Makefile'

>>d = sysconfig.parse _makefile(_)
>>import pprint
>>pprint.pprint (d)

{'AR': 'ar',
...
}

This output from the check is here:
http://www.jamesstroud.com/build-sci...util-check.txt

I'm not sure if this is relevant to what you see in the check, but I
installed scipy_distutils AFTER installing scipy.

>I did not capture output from the build and I could not find a file
with the word "log" in it that appears to be a build log. The jist of
the problem is that first it can't find symbols from libpython2.5,
then from libthread, etc. Then, it complains about no "MAIN__" when
linking the .so files with g77 and no "main" with gcc (which is
curious), so I must include the -shared flag, after including
-llibrary type flags for all of the libraries it doesn't know about.

What versions of gcc and g77 are you using?

euler 2% g77 -v
Reading specs from
/auto_nfs/data10/users/jstroud/Programs/bin/../lib/gcc/i686-pc-linux-gnu/3.4.2/specs
Configured with: ./configure --prefix=/data1/users/jstroud/Programs :
(reconfigured) ./configure --prefix=/data1/users/jstroud/Programs
Thread model: posix
gcc version 3.4.2
euler 3% gcc -v
Reading specs from
/auto_nfs/data10/users/jstroud/Programs/bin/../lib/gcc/i686-pc-linux-gnu/3.4.2/specs
Configured with: ./configure --prefix=/data1/users/jstroud/Programs :
(reconfigured) ./configure --prefix=/data1/users/jstroud/Programs
Thread model: posix
gcc version 3.4.2

>If its necessary, I can run setup.py build again and send the output
to a file and post that to scipy-dev if you think it might be helpful.

Yes, that is what I intended.

This is posted here:
http://www.jamesstroud.com/build-sci...y2.5-euler.txt

For symmetry, similar output for numpy is posted here:
http://www.jamesstroud.com/build-sci...y2.5-euler.txt

Please let me know if I can be of any further help with these issues.

James

[James Stroud]

Martin v. Löwis wrote:

James Stroud schrieb:

>This is annoying. I am trying to build scipy right now but every .so
file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
to the ld flags.

That shouldn't be necessary. Linking without this should work just fine.

Unfortunately, this was not my experience.

That way, since you made libpython2.5.a a static library, you link an
entire Python interpreter into each and every extension module. This
should not be done.

Actually, a straight configure, make, install of the python2.5 source
produced this library.

What happens if you omit these flags?

Please see my last message to Robert Kern.

James