By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
443,750 Members | 1,367 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 443,750 IT Pros & Developers. It's quick & easy.

Building things with setup.py

P: n/a
Hello All,

This is annoying. I am trying to build scipy right now but every .so
file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
to the ld flags.

Main Question: When building with "setup.py", where can I include this
information so I don't have to link every .so file by hand?

Also, why would this not be caught by the developers of scipy and numpy?
What factors go into needing to specify libraries and to properly link
..so files manually? Certainly the enthought didn't forget to test the
build process, so why would my machine be special in this regard?

I'm running Linux FC4 on a dual intel p4 (~3.2 GHz) with sundry
libraries in non-standard places. Is this non-standard part what's
killing me?

James

--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/
Sep 22 '06 #1
Share this Question
Share on Google+
22 Replies


P: n/a
James Stroud wrote:
Hello All,

This is annoying. I am trying to build scipy right now but every .so
file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
to the ld flags.

Main Question: When building with "setup.py", where can I include this
information so I don't have to link every .so file by hand?

Also, why would this not be caught by the developers of scipy and numpy?
What factors go into needing to specify libraries and to properly link
.so files manually? Certainly the enthought didn't forget to test the
build process, so why would my machine be special in this regard?

I'm running Linux FC4 on a dual intel p4 (~3.2 GHz) with sundry
libraries in non-standard places. Is this non-standard part what's
killing me?

James
Well I added those arguments to $LDFLAGS, but it seemed like a pretty
miserable hack, especially the "-shared" part.

I'm still curious how to make a 3rd party setup.py work correctly.

James

--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/
Sep 22 '06 #2

P: n/a
James Stroud wrote:
Hello All,

This is annoying. I am trying to build scipy right now but every .so
file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
to the ld flags.

Main Question: When building with "setup.py", where can I include this
information so I don't have to link every .so file by hand?
If it's actually necessary,

python setup.py build_ext -lmylib
Also, why would this not be caught by the developers of scipy and numpy?
Because we've never seen it before. There's no reason that you should have to
add those libraries manually. Can you post your build log (the one that fails)
to scipy-dev?

http://www.scipy.org/Mailing_Lists
What factors go into needing to specify libraries and to properly link
.so files manually? Certainly the enthought didn't forget to test the
build process, so why would my machine be special in this regard?

I'm running Linux FC4 on a dual intel p4 (~3.2 GHz) with sundry
libraries in non-standard places. Is this non-standard part what's
killing me?
Possibly. Won't know more until I see the build log.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

Sep 22 '06 #3

P: n/a
James Stroud wrote:
Hello All,

This is annoying. I am trying to build scipy right now but every .so
file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
to the ld flags.
I'm running Linux FC4 on a dual intel p4 (~3.2 GHz) with sundry
libraries in non-standard places. Is this non-standard part what's
killing me?
It's possible that distutils is not finding the information about how the python
interpreter was built. Did you build your own python(1)? Is the information in
$PREFIX/lib/python2.4/config/ accurate?

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

Sep 22 '06 #4

P: n/a
Robert Kern wrote:
James Stroud wrote:
>Hello All,

This is annoying. I am trying to build scipy right now but every .so
file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl
-shared" to the ld flags.

>I'm running Linux FC4 on a dual intel p4 (~3.2 GHz) with sundry
libraries in non-standard places. Is this non-standard part what's
killing me?


It's possible that distutils is not finding the information about how
the python interpreter was built. Did you build your own python(1)? Is
the information in $PREFIX/lib/python2.4/config/ accurate?
I did build my own python 2.5, yesterday, requiring me to rebuild all
extensions. Everything I do is compiled by hand as joe-user. I'm in a
situation where I can't do RPM (and I don't have root on my work machine
(theoretically ;-)) so, to be a good joe-user, everything I add goes
into the prefix:

$HOME/Programs

This is the listing from $HOME/Programs/lib/python2.5/config:

euler 6% ls
total 4092
8 config.c 12 install-sh* 44 Makefile 8 python.o
8 Setup.config
8 config.c.in 3960 libpython2.5.a 12 makesetup* 24 Setup
8 Setup.local

The build process, by the way, required my copying libpython2.5.a to
$HOME/Programs/lib.

The text files Setup.config and Setup.local do not seem to have terribly
specific information in them. Which file in particular should I inspect?

I did not capture output from the build and I could not find a file with
the word "log" in it that appears to be a build log. The jist of the
problem is that first it can't find symbols from libpython2.5, then from
libthread, etc. Then, it complains about no "MAIN__" when linking the
..so files with g77 and no "main" with gcc (which is curious), so I must
include the -shared flag, after including -llibrary type flags for all
of the libraries it doesn't know about.

If its necessary, I can run setup.py build again and send the output to
a file and post that to scipy-dev if you think it might be helpful.

James

--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/
Sep 22 '06 #5

P: n/a
James Stroud wrote:
I did build my own python 2.5, yesterday, requiring me to rebuild all
extensions.
Do other extensions build correctly? If so, it's beginning to look like a
problem in numpy.distutils .
Everything I do is compiled by hand as joe-user. I'm in a
situation where I can't do RPM (and I don't have root on my work machine
(theoretically ;-)) so, to be a good joe-user, everything I add goes
into the prefix:

$HOME/Programs

This is the listing from $HOME/Programs/lib/python2.5/config:

euler 6% ls
total 4092
8 config.c 12 install-sh* 44 Makefile 8 python.o
8 Setup.config
8 config.c.in 3960 libpython2.5.a 12 makesetup* 24 Setup
8 Setup.local

The build process, by the way, required my copying libpython2.5.a to
$HOME/Programs/lib.
Hmm. That doesn't quite sound right, but it's been a while since I compiled the
interpreter from source.
The text files Setup.config and Setup.local do not seem to have terribly
specific information in them. Which file in particular should I inspect?
Makefile has most of that information. You can verify that distutils is finding
it like so:
>>from distutils import sysconfig
sysconfig.get_makefile_filename()
'/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/config/Makefile'
>>d = sysconfig.parse_makefile(_)
import pprint
pprint.pprint(d)
{'AR': 'ar',
...
}
I did not capture output from the build and I could not find a file with
the word "log" in it that appears to be a build log. The jist of the
problem is that first it can't find symbols from libpython2.5, then from
libthread, etc. Then, it complains about no "MAIN__" when linking the
.so files with g77 and no "main" with gcc (which is curious), so I must
include the -shared flag, after including -llibrary type flags for all
of the libraries it doesn't know about.
What versions of gcc and g77 are you using?
If its necessary, I can run setup.py build again and send the output to
a file and post that to scipy-dev if you think it might be helpful.
Yes, that is what I intended.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

Sep 22 '06 #6

P: n/a
James Stroud wrote:
Well I added those arguments to $LDFLAGS, but it seemed like a pretty
miserable hack, especially the "-shared" part.
Hmmm. Did you have $LDFLAGS set to anything (or more likely, nothing) before?
I.e. did you have something like this?

export LDFLAGS=""
python setup.py build

If such environment variables are present, I think that numpy.distutils may
clobber the values that are inferred from the configuration.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

Sep 22 '06 #7

P: n/a
James Stroud schrieb:
This is annoying. I am trying to build scipy right now but every .so
file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
to the ld flags.
That shouldn't be necessary. Linking without this should work just fine.

That way, since you made libpython2.5.a a static library, you link an
entire Python interpreter into each and every extension module. This
should not be done.

What happens if you omit these flags?

Regards,
Martin
Sep 22 '06 #8

P: n/a
Robert Kern wrote:
James Stroud wrote:
>I did build my own python 2.5, yesterday, requiring me to rebuild all
extensions.

Do other extensions build correctly? If so, it's beginning to look like
a problem in numpy.distutils .
It seems that every thing has built without incident (numarray, numeric,
mxtexttools, pil, reportlab, just to name a few), with the exception,
strangely enough, of python2.5 itself. Python required hiding previous
python installations.
>Everything I do is compiled by hand as joe-user. I'm in a situation
where I can't do RPM (and I don't have root on my work machine
(theoretically ;-)) so, to be a good joe-user, everything I add goes
into the prefix:

$HOME/Programs

This is the listing from $HOME/Programs/lib/python2.5/config:

euler 6% ls
total 4092
8 config.c 12 install-sh* 44 Makefile 8 python.o
8 Setup.config
8 config.c.in 3960 libpython2.5.a 12 makesetup* 24 Setup
8 Setup.local

The build process, by the way, required my copying libpython2.5.a to
$HOME/Programs/lib.

Hmm. That doesn't quite sound right, but it's been a while since I
compiled the interpreter from source.
Sorry. To clarify, making libpython2.5a available in a $LD_LIBRARY_PATH
was necessary to build numpy and scipy.
>The text files Setup.config and Setup.local do not seem to have
terribly specific information in them. Which file in particular should
I inspect?

Makefile has most of that information. You can verify that distutils is
finding it like so:
>>from distutils import sysconfig
>>sysconfig.get_makefile_filename()
'/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/config/Makefile'
>>d = sysconfig.parse_makefile(_)
>>import pprint
>>pprint.pprint(d)
{'AR': 'ar',
...
}
This output from the check is here:
http://www.jamesstroud.com/build-sci...util-check.txt

I'm not sure if this is relevant to what you see in the check, but I
installed scipy_distutils AFTER installing scipy.
>I did not capture output from the build and I could not find a file
with the word "log" in it that appears to be a build log. The jist of
the problem is that first it can't find symbols from libpython2.5,
then from libthread, etc. Then, it complains about no "MAIN__" when
linking the .so files with g77 and no "main" with gcc (which is
curious), so I must include the -shared flag, after including
-llibrary type flags for all of the libraries it doesn't know about.

What versions of gcc and g77 are you using?
euler 2% g77 -v
Reading specs from
/auto_nfs/data10/users/jstroud/Programs/bin/../lib/gcc/i686-pc-linux-gnu/3.4.2/specs
Configured with: ./configure --prefix=/data1/users/jstroud/Programs :
(reconfigured) ./configure --prefix=/data1/users/jstroud/Programs
Thread model: posix
gcc version 3.4.2
euler 3% gcc -v
Reading specs from
/auto_nfs/data10/users/jstroud/Programs/bin/../lib/gcc/i686-pc-linux-gnu/3.4.2/specs
Configured with: ./configure --prefix=/data1/users/jstroud/Programs :
(reconfigured) ./configure --prefix=/data1/users/jstroud/Programs
Thread model: posix
gcc version 3.4.2
>If its necessary, I can run setup.py build again and send the output
to a file and post that to scipy-dev if you think it might be helpful.

Yes, that is what I intended.
This is posted here:
http://www.jamesstroud.com/build-sci...y2.5-euler.txt

For symmetry, similar output for numpy is posted here:
http://www.jamesstroud.com/build-sci...y2.5-euler.txt

Please let me know if I can be of any further help with these issues.

James
Sep 22 '06 #9

P: n/a
Martin v. Lwis wrote:
James Stroud schrieb:
>This is annoying. I am trying to build scipy right now but every .so
file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
to the ld flags.

That shouldn't be necessary. Linking without this should work just fine.
Unfortunately, this was not my experience.
That way, since you made libpython2.5.a a static library, you link an
entire Python interpreter into each and every extension module. This
should not be done.
Actually, a straight configure, make, install of the python2.5 source
produced this library.
What happens if you omit these flags?
Please see my last message to Robert Kern.

James
Sep 22 '06 #10

P: n/a
Robert Kern wrote:
James Stroud wrote:
>I did build my own python 2.5, yesterday, requiring me to rebuild all
extensions.

Do other extensions build correctly? If so, it's beginning to look like
a problem in numpy.distutils .
Actually, I just found that MySQLdb had a similar problem with zlib, but
instead of bonking during linking, it complained about the compress
symbol when trying to import into python. The fix was adding "-lz" for
the linking step, after which it imported just fine.

James
Sep 22 '06 #11

P: n/a
James Stroud wrote:
Robert Kern wrote:
>James Stroud wrote:
>>The build process, by the way, required my copying libpython2.5.a to
$HOME/Programs/lib.
Hmm. That doesn't quite sound right, but it's been a while since I
compiled the interpreter from source.

Sorry. To clarify, making libpython2.5a available in a $LD_LIBRARY_PATH
was necessary to build numpy and scipy.
I don't see how that could have affected anything. Static libraries aren't
looked up in $LD_LIBRARY_PATH. Certainly not for build-time linking.
I'm not sure if this is relevant to what you see in the check, but I
installed scipy_distutils AFTER installing scipy.
scipy_distutils is no longer used. All of its functionality has been folded into
numpy.

What do your environment variables look like when you build?

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

Sep 22 '06 #12

P: n/a
Robert Kern wrote:
>Sorry. To clarify, making libpython2.5a available in a
$LD_LIBRARY_PATH was necessary to build numpy and scipy.

I don't see how that could have affected anything. Static libraries
aren't looked up in $LD_LIBRARY_PATH. Certainly not for build-time linking.
OK--new thing learned for the day. <rant>I try things until a build
works. I consider everything I do until acquiring a working build as
necessary. Going back to see exactly what I did to make things work is
not a viable option given that every new program I want requires my
building it and all of its dependencies--a consequence of the [adjective
deleted] beuracratic constraints (with regard to the machine on my
desktop) that have evolved here. These constraints are no doubt a
byproduct of the "Institute" part of my signature below, associated
paranoia, and over-worked sys-admins.</rant>

The upside is that my entire environment turns out to be roughly
transportable to other Linux machines on the network via an NFS home
directory--more or less independent of the configuration of said other
machines.
What do your environment variables look like when you build?
euler 2% echo $CPPFLAGS
-I/data10/users/jstroud/Programs/include
-I/data10/users/jstroud/Programs/qt/include -I/usr/include
-I/data10/users/jstroud/Programs/include/freetype2/freetype
euler 3% echo $LD_LIBRARY_PATH
/data10/users/jstroud/Programs/lib:/data10/users/jstroud/Programs/qt/lib:/usr/lib:/usr/X11R6/lib/:/usr/local/lib:/usr/lib/gcc-lib/i386-redhat-linux/3.2.3:/data10/users/jstroud/Programs/ccp4-5.0.2/lib
euler 4% echo $LDFLAGS
-L/data10/users/jstroud/Programs/lib
-L/usr/lib/gcc-lib/i386-redhat-linux/3.2.3

Here is how I customize my ld.so.cache:

euler 9% cat ldconf
rm ld.so.cache
/sbin/ldconfig -v -f ./ld.so.conf -C ./ld.so.cache
echo source this

euler 10% cat ld.so.conf
/data10/users/jstroud/Programs/lib
/data10/users/jstroud/Programs/qt/lib
/data10/users/jstroud/Programs/newlib
/usr/lib
/usr/local/lib
/usr/X11R6/lib
/usr/lib/qt3/lib
/usr/lib/wine
/usr/include/pgsql

Again, see the above rant for an explanation about any needless paths or
flags--though advice and knowledge is welcome.

James

--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/
Sep 22 '06 #13

P: n/a
Robert Kern wrote:
James Stroud wrote:
>Well I added those arguments to $LDFLAGS, but it seemed like a pretty
miserable hack, especially the "-shared" part.


Hmmm. Did you have $LDFLAGS set to anything (or more likely, nothing)
before? I.e. did you have something like this?

export LDFLAGS=""
python setup.py build

If such environment variables are present, I think that numpy.distutils
may clobber the values that are inferred from the configuration.
Didn't see this message last night.

No, my $LDFLAGS were set. I haven't tried unsetenv'ing them (I use tcsh
as a consequence of my anti-bash religious upbringing).

James

--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/
Sep 22 '06 #14

P: n/a
James Stroud schrieb:
>What happens if you omit these flags?

Please see my last message to Robert Kern.
If you don't want me to help you, that's fine, then I won't.

I couldn't find an answer to this question in any of your
messages in this thread.

Still, it *should* work out of the box. If it doesn't, it
means there is an error somewhere. If you don't want to
help finding the error, that's fine, too (but please say
so, because that will constrain the solution space).

Regards,
Martin
Sep 22 '06 #15

P: n/a
Martin v. Lwis wrote:
James Stroud schrieb:
>>>What happens if you omit these flags?

Please see my last message to Robert Kern.


If you don't want me to help you, that's fine, then I won't.

I couldn't find an answer to this question in any of your
messages in this thread.
Sorry, I didn't mean to sound dismissive. Perhaps I misunderstood what
"these" actually referred to. The results of omitting these flags
specifically:

"-lpython2.5 -lpthread -lm -lutil -ldl -shared"

Are posted here for both numpy and scipy:

http://www.jamesstroud.com/build-sci...y2.5-euler.txt
http://www.jamesstroud.com/build-sci...y2.5-euler.txt

I have not tried unsetenv'ing $LDFLAGS entirely, an option which did not
occur to me given their necessity for just about everything else. If you
think that this is a worthwhile experiment, I will run it.

Still, it *should* work out of the box. If it doesn't, it
means there is an error somewhere. If you don't want to
help finding the error, that's fine, too (but please say
so, because that will constrain the solution space).
I think I would like to know how to avoid or correct these sort of
issues in the future, which seem to be limited, for me at least, to
scipy and numpy, with the possible exception of MySQLdb and its
dependency on zlib. Ideally, I would like to understand exactly what
causes these errors in addition to knowing what I can do to correct them
in the future.
James
--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/
Sep 22 '06 #16

P: n/a
James Stroud wrote:
Robert Kern wrote:
>>Sorry. To clarify, making libpython2.5a available in a
$LD_LIBRARY_PATH was necessary to build numpy and scipy.
I don't see how that could have affected anything. Static libraries
aren't looked up in $LD_LIBRARY_PATH. Certainly not for build-time linking.

OK--new thing learned for the day. <rant>I try things until a build
works. I consider everything I do until acquiring a working build as
necessary. Going back to see exactly what I did to make things work is
not a viable option given that every new program I want requires my
building it and all of its dependencies--a consequence of the [adjective
deleted] beuracratic constraints (with regard to the machine on my
desktop) that have evolved here. These constraints are no doubt a
byproduct of the "Institute" part of my signature below, associated
paranoia, and over-worked sys-admins.</rant>
:-) I've always thought that a Kubuntu LiveCD would be a useful path around much
of that nonsense, but I've fortunately not been forced to attempt it.
>What do your environment variables look like when you build?

euler 2% echo $CPPFLAGS
-I/data10/users/jstroud/Programs/include
-I/data10/users/jstroud/Programs/qt/include -I/usr/include
-I/data10/users/jstroud/Programs/include/freetype2/freetype
euler 3% echo $LD_LIBRARY_PATH
/data10/users/jstroud/Programs/lib:/data10/users/jstroud/Programs/qt/lib:/usr/lib:/usr/X11R6/lib/:/usr/local/lib:/usr/lib/gcc-lib/i386-redhat-linux/3.2.3:/data10/users/jstroud/Programs/ccp4-5.0.2/lib
euler 4% echo $LDFLAGS
-L/data10/users/jstroud/Programs/lib
-L/usr/lib/gcc-lib/i386-redhat-linux/3.2.3
Okay, this is possibly part of the problem. numpy.distutils handles FORTRAN code
differently than other extension modules; this is why pure C extension modules
both inside numpy and elsewehere were linking fine. Try unsetenving $CPPFLAGS
and $LDFLAGS and adding those arguments to build_ext, if necessary.

Looking on line 516 of numpy/fcompiler/__init__.py (why there's a class
definition in __init__.py, I'll never know), it does look like the linker flags
which are carefully constructed by numpy.distutils are indeed overridden by the
user's setting of $LDFLAGS.

The fact that this appears to be incompatible with how distutils does it for
everything else is a bug, IMO, although it *is* a useful feature if you really
do need to completely control the linker flags. That's probably what the
implementor was thinking. Although, sadly, not documenting. He should be back
from his honeymoon, now, so I'll raise the issue with him.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

Sep 22 '06 #17

P: n/a
James Stroud <js*****@mbi.ucla.eduwrites:
<rant>I try things until a build works. I consider everything I do
until acquiring a working build as necessary. Going back to see
exactly what I did to make things work is not a viable option
Before trying such "I don't know what I need to do, but I'm going to
want to know after I do it" procedures, I usually start 'script' in
the shell session and then start all my experimenting. ('script' is a
tool from BSD available on most GNU/Linux distributions as well.)

That way, the entire session, fumbling dead-ends and all, is available
for later examination and trimming back to a "what I did to get it
working" document.

--
\ "I have yet to see any problem, however complicated, which, |
`\ when you looked at it in the right way, did not become still |
_o__) more complicated." -- Paul Anderson |
Ben Finney

Sep 23 '06 #18

P: n/a
James Stroud schrieb:
I think I would like to know how to avoid or correct these sort of
issues in the future, which seem to be limited, for me at least, to
scipy and numpy, with the possible exception of MySQLdb and its
dependency on zlib. Ideally, I would like to understand exactly what
causes these errors in addition to knowing what I can do to correct them
in the future.
Let's take a specific failure, namely the line

/auto_nfs/data10/users/jstroud/Programs/bin/g77
-L/data10/users/jstroud/Programs/lib
-L/usr/lib/gcc-lib/i386-redhat-linux/3.2.3
build/temp.linux-i686-2.5/numpy/core/blasdot/_dotblas.o
-L/data10/users/jstroud/Programs/lib -lblas -lg2c -o
build/lib.linux-i686-2.5/numpy/core/_dotblas.so

This gives errors like

build/temp.linux-i686-2.5/numpy/core/blasdot/_dotblas.o(.text+0x758):numpy/core/blasdot/_dotblas.c:226:
undefined reference to `PyTuple_New'

That's not too surprising: this line tries to link the input
*as an executable program*, despite calling the output
_dotblas.so. In an executable program, all symbols need to
be defined; that's why it it complains about so many missing
symbols (including MAIN__ - which should never be missing
in a library). Even though adding python2.5.a to the linker
link makes these symbols appear, the result still won't
work, as you can't use an executable program file as if
it were a shared library.

Now, compare this to a succeeding build of a C extension
module,

gcc -pthread -shared -L/data10/users/jstroud/Programs/lib
-L/usr/lib/gcc-lib/i386-redhat-linux/3.2.3
-I/data10/users/jstroud/Programs/include
-I/data10/users/jstroud/Programs/qt/include -I/usr/include
-I/data10/users/jstroud/Programs/include/freetype2/freetype
build/temp.linux-i686-2.5/build/src.linux-i686-2.5/numpy/core/src/umathmodule.o
-lm -o build/lib.linux-i686-2.5/numpy/core/umath.so

Notice that this passes -shared to the compiler, requesting
construction of a shared library. This is the problem with
the g77 linker line; try invoking

/auto_nfs/data10/users/jstroud/Programs/bin/g77 -shared
-L/data10/users/jstroud/Programs/lib
-L/usr/lib/gcc-lib/i386-redhat-linux/3.2.3
build/temp.linux-i686-2.5/numpy/core/blasdot/_dotblas.o
-L/data10/users/jstroud/Programs/lib -lblas -lg2c -o
build/lib.linux-i686-2.5/numpy/core/_dotblas.so

manually (whether or not -pthread is also necessary
or supported for g77, I don't know). This should at least
make the complaints about missing symbols go away; you
should then see whether the resulting module can be
imported in Python.

If that solves the problem, the question is why the
-shared option isn't passed automatically; your setting
LDFLAGS might indeed be a cause.

Regards,
Martin
Sep 23 '06 #19

P: n/a
Ben Finney wrote:
James Stroud <js*****@mbi.ucla.eduwrites:
><rant>I try things until a build works. I consider everything I do
until acquiring a working build as necessary. Going back to see
exactly what I did to make things work is not a viable option

Before trying such "I don't know what I need to do, but I'm going to
want to know after I do it" procedures, I usually start 'script' in
the shell session and then start all my experimenting. ('script' is a
tool from BSD available on most GNU/Linux distributions as well.)

That way, the entire session, fumbling dead-ends and all, is available
for later examination and trimming back to a "what I did to get it
working" document.
Beautiful suggestion. I've never even heard of that before. With OS X
Terminal.app, I can buffer "unlimited lines" on my 6 year old ibook I
use at home and on the road. But at work, on my brand new dual pentium,
I've found myself limited to the thousand or so lines that konsole gives me.

James
Sep 23 '06 #20

P: n/a
Martin v. Lwis wrote:
James Stroud schrieb:
>I think I would like to know how to avoid or correct these sort of
issues in the future, which seem to be limited, for me at least, to
scipy and numpy, with the possible exception of MySQLdb and its
dependency on zlib. Ideally, I would like to understand exactly what
causes these errors in addition to knowing what I can do to correct them
in the future.

Let's take a specific failure, namely the line

/auto_nfs/data10/users/jstroud/Programs/bin/g77
-L/data10/users/jstroud/Programs/lib
-L/usr/lib/gcc-lib/i386-redhat-linux/3.2.3
build/temp.linux-i686-2.5/numpy/core/blasdot/_dotblas.o
-L/data10/users/jstroud/Programs/lib -lblas -lg2c -o
build/lib.linux-i686-2.5/numpy/core/_dotblas.so

This gives errors like

build/temp.linux-i686-2.5/numpy/core/blasdot/_dotblas.o(.text+0x758):numpy/core/blasdot/_dotblas.c:226:
undefined reference to `PyTuple_New'

That's not too surprising: this line tries to link the input
*as an executable program*, despite calling the output
_dotblas.so. In an executable program, all symbols need to
be defined; that's why it it complains about so many missing
symbols (including MAIN__ - which should never be missing
in a library). Even though adding python2.5.a to the linker
link makes these symbols appear, the result still won't
work, as you can't use an executable program file as if
it were a shared library.

Now, compare this to a succeeding build of a C extension
module,

gcc -pthread -shared -L/data10/users/jstroud/Programs/lib
-L/usr/lib/gcc-lib/i386-redhat-linux/3.2.3
-I/data10/users/jstroud/Programs/include
-I/data10/users/jstroud/Programs/qt/include -I/usr/include
-I/data10/users/jstroud/Programs/include/freetype2/freetype
build/temp.linux-i686-2.5/build/src.linux-i686-2.5/numpy/core/src/umathmodule.o
-lm -o build/lib.linux-i686-2.5/numpy/core/umath.so

Notice that this passes -shared to the compiler, requesting
construction of a shared library. This is the problem with
the g77 linker line; try invoking

/auto_nfs/data10/users/jstroud/Programs/bin/g77 -shared
-L/data10/users/jstroud/Programs/lib
-L/usr/lib/gcc-lib/i386-redhat-linux/3.2.3
build/temp.linux-i686-2.5/numpy/core/blasdot/_dotblas.o
-L/data10/users/jstroud/Programs/lib -lblas -lg2c -o
build/lib.linux-i686-2.5/numpy/core/_dotblas.so

manually (whether or not -pthread is also necessary
or supported for g77, I don't know). This should at least
make the complaints about missing symbols go away; you
should then see whether the resulting module can be
imported in Python.

If that solves the problem, the question is why the
-shared option isn't passed automatically; your setting
LDFLAGS might indeed be a cause.

Regards,
Martin
As per your and Robert Kern's suggestions, unsetenv'ing $LDFLAGS and
$CPPFLAGS indeed did the trick for numpy, scipy, and mysqldb. What a
tough lesson in distutils!

<COMMENTARY>Though great for self development, I'm not so sure such
lessons should be necessary to build these tools.</COMMENTARY>

James

James
Sep 23 '06 #21

P: n/a
Robert Kern wrote:
Okay, this is possibly part of the problem. numpy.distutils handles
FORTRAN code differently than other extension modules; this is why pure
C extension modules both inside numpy and elsewehere were linking fine.
Try unsetenving $CPPFLAGS and $LDFLAGS and adding those arguments to
build_ext, if necessary.

Looking on line 516 of numpy/fcompiler/__init__.py (why there's a class
definition in __init__.py, I'll never know), it does look like the
linker flags which are carefully constructed by numpy.distutils are
indeed overridden by the user's setting of $LDFLAGS.
I unset $LDFLAGS and $CPPFLAGS (not trying one at a time), and this did
the trick. Everything built without a hitch.
James
Sep 23 '06 #22

P: n/a
James Stroud schrieb:
<COMMENTARY>Though great for self development, I'm not so sure such
lessons should be necessary to build these tools.</COMMENTARY>
Yes. The lesson you should take from this is: don't try to be smarter
than the authors of the software. It should build out of the box, if
you follow the build instructions literally. Deviate to the tiniest
from the instructions, and you are on your own.

If it doesn't work out of the box even though you followed the
instructions, don't try to fix it. You didn't write it, fixing
it will be a huge effort. Instead, get help from somebody, and/or
report a bug.

Always strive for the simplest possible setup. If you find that you
have to customize something, and the instructions did not tell you
to do so, question this customization. It is likely wrong or
unnecessary.

Regards,
Martin
Sep 23 '06 #23

This discussion thread is closed

Replies have been disabled for this discussion.