473,414 Members | 1,954 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,414 software developers and data experts.

LinearAlgebra incredibly slow for eigenvalue problems

Hello,

I need to calculate the eigenvectors and eigenvalues for a 3600 X 3600
covariance matrix.

The LinearAlgebra package in Python is incredibly slow to perform the
above calculations (about 1.5 hours). This in spite of the fact that
I have installed Numeric with the full ATLAS and LAPACK libraries.

Also note that my computer has dual Pentium IV (3.1 GHz) processors
with 2Gb ram.

Every Web discussion I have seen about such issues indicates that
one can expect huge speed ups if one compiles and installs Numeric
linked against the ATLAS and LAPACK libraries.

Even more perplexing is that the same calculation takes a mere 7 min
in Matlab V6.5. Matlab uses both ATLAS and LAPACK.

Moreover, the above calculation takes the same amount of time for
Numeric to complete with --and-- without ATLAS and PACK. I am certain
that I have done the install correctly.

Can anyone provide some insight?
Thanks in advance for your help.
Daran

Jul 18 '05 #1
9 2561
"drife" <da*******@yahoo.com> writes:
Hello,

I need to calculate the eigenvectors and eigenvalues for a 3600 X 3600
covariance matrix.

The LinearAlgebra package in Python is incredibly slow to perform the
above calculations (about 1.5 hours). This in spite of the fact that
I have installed Numeric with the full ATLAS and LAPACK libraries.

Also note that my computer has dual Pentium IV (3.1 GHz) processors
with 2Gb ram.

Every Web discussion I have seen about such issues indicates that
one can expect huge speed ups if one compiles and installs Numeric
linked against the ATLAS and LAPACK libraries.
Are you *sure* that Numeric is linked against these?
Even more perplexing is that the same calculation takes a mere 7 min
in Matlab V6.5. Matlab uses both ATLAS and LAPACK.

Moreover, the above calculation takes the same amount of time for
Numeric to complete with --and-- without ATLAS and PACK. I am certain
that I have done the install correctly.


This is good evidence that Numeric *isn't* linked to them.

If you're on a Linux system, you can check with ldd:
cookedm@arbutus$ ldd /usr/lib/python2.3/site-packages/Numeric/lapack_lite.so
liblapack.so.3 => /usr/lib/atlas/liblapack.so.3 (0x0000002a95677000)
libblas.so.3 => /usr/lib/atlas/libblas.so.3 (0x0000002a95e55000)
libg2c.so.0 => /usr/lib/libg2c.so.0 (0x0000002a96721000)
libpthread.so.0 => /lib/libpthread.so.0 (0x0000002a96842000)
libc.so.6 => /lib/libc.so.6 (0x0000002a96957000)
libm.so.6 => /lib/libm.so.6 (0x0000002a96b96000)
/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)

You can see that lapack and blas (the Atlas versions) are linked to
the lapack_lite.so.

To install Numeric using Lapack:
- remove the build/ directory in your Numeric sources, so you don't
any old binaries
- edit setup.py and follow the comments on using Lapack (you need to
comment out a few lines, and set some directories)
Also set use_dotblas to 1.
- do the 'python setup.py build', 'python setup.py install' dance.

--
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke
|cookedm(at)physics(dot)mcmaster(dot)ca
Jul 18 '05 #2
Hi David,

I performed the above check, and sure enough, Numeric
is --not-- linked to the ATLAS libraries.

I followed each of your steps outlined above, and Numeric
still is not linking to the ATLAS libraries.

My setup.py file is attached below.

Thanks ,

Daran

--#!/usr/bin/env python
# To use:
# python setup.py install
# or:
# python setup.py bdist_rpm (you'll end up with RPMs in dist)
#
import os, sys, string, re
from glob import glob
if not hasattr(sys, 'version_info') or sys.version_info <
(2,0,0,'alpha',0):
raise SystemExit, "Python 2.0 or later required to build Numeric."
import distutils
from distutils.core import setup, Extension

# Get all version numbers
execfile(os.path.join('Lib','numeric_version.py'))
numeric_version = version

execfile(os.path.join('Packages', 'MA', 'Lib', 'MA_version.py'))
MA_version = version

headers = glob (os.path.join ("Include","Numeric","*.h"))
extra_compile_args = [] # You could put "-O4" etc. here.
mathlibs = ['m']
define_macros = [('HAVE_INVERSE_HYPERBOLIC',None)]
undef_macros = []
# You might need to add a case here for your system
if sys.platform in ['win32']:
mathlibs = []
define_macros = []
undef_macros = ['HAVE_INVERSE_HYPERBOLIC']
elif sys.platform in ['mac', 'beos5']:
mathlibs = []

# delete all but the first one in this list if using your own
LAPACK/BLAS
sourcelist = [os.path.join('Src', 'lapack_litemodule.c')]
# set these to use your own BLAS;

library_dirs_list = ['/d2/lib/atlas']
libraries_list = ['lapack', 'ptcblas', 'ptf77blas', 'atlas', 'g2c']

# set to true (1), if you also want BLAS optimized
matrixmultiply/dot/innerproduct
use_dotblas = 1
include_dirs = ['/d2/include']
# You may need to set this to find cblas.h
# e.g. on UNIX using ATLAS this should be
['/usr/include/atlas']
extra_link_args = []

# for MacOS X to link against vecLib if present
VECLIB_PATH = '/System/Library/Frameworks/vecLib.framework'
if os.path.exists(VECLIB_PATH):
extra_link_args = ['-framework', 'vecLib']
include_dirs = [os.path.join(VECLIB_PATH, 'Headers')]

# The packages are split in this way to allow future optional inclusion
# Numeric package
packages = ['']
package_dir = {'': 'Lib'}
include_dirs.append('Include')
ext_modules = [
Extension('_numpy',
[os.path.join('Src', '_numpymodule.c'),
os.path.join('Src', 'arrayobject.c'),
os.path.join('Src', 'ufuncobject.c')],
extra_compile_args = extra_compile_args),
Extension('multiarray',
[os.path.join('Src', 'multiarraymodule.c')],
extra_compile_args = extra_compile_args),
Extension('umath',
[os.path.join('Src', 'umathmodule.c')],
libraries = mathlibs,
define_macros = define_macros,
undef_macros = undef_macros,
extra_compile_args = extra_compile_args),
Extension('arrayfns',
[os.path.join('Src', 'arrayfnsmodule.c')],
extra_compile_args = extra_compile_args),
Extension('ranlib',
[os.path.join('Src', 'ranlibmodule.c'),
os.path.join('Src', 'ranlib.c'),
os.path.join('Src', 'com.c'),
os.path.join('Src', 'linpack.c')],
extra_compile_args = extra_compile_args),
Extension('lapack_lite', sourcelist,
library_dirs = library_dirs_list,
libraries = libraries_list,
extra_link_args = extra_link_args,
extra_compile_args = extra_compile_args)
]

# add FFT package (optional)
packages.append('FFT')
package_dir['FFT'] = os.path.join('Packages','FFT','Lib')
include_dirs.append(os.path.join('Packages','FFT', 'Include'))
ext_modules.append(Extension('FFT.fftpack',
[os.path.join('Packages','FFT','Src',
'fftpackmodule.c'),
os.path.join('Packages', 'FFT', 'Src',
'fftpack.c')],
extra_compile_args = extra_compile_args))

# add MA package (optional)
packages.append('MA')
package_dir['MA'] = os.path.join('Packages', 'MA', 'Lib')

# add RNG package (optional)
packages.append('RNG')
packages.append('RNG')
package_dir['RNG'] = os.path.join('Packages', 'RNG', 'Lib')
include_dirs.append(os.path.join('Packages', 'RNG', 'Include'))
ext_modules.append(Extension('RNG.RNG',
[os.path.join('Packages', 'RNG', 'Src',
'RNGmodule.c'),
os.path.join('Packages', 'RNG', 'Src',
'ranf.c'),
os.path.join('Packages', 'RNG', 'Src',
'pmath_rng.c')],
extra_compile_args = extra_compile_args))
# add dotblas package (optional)
if use_dotblas:
packages.append('dotblas')
package_dir['dotblas'] = os.path.join('Packages', 'dotblas',
'dotblas')
ext_modules.append(Extension('_dotblas',
[os.path.join('Packages', 'dotblas',
'dotblas', '_dotblas.c')],
library_dirs = library_dirs_list,
libraries = libraries_list,

extra_compile_args=extra_compile_args))


long_description = """
Numerical Extension to Python with subpackages.

The authors and maintainers of the subpackages are:

FFTPACK-3.1
maintainer = "Numerical Python Developers"
maintainer_email = "nu**************@lists.sourceforge.net"
description = "Fast Fourier Transforms"
url = "http://numpy.sourceforge.net"

MA-%s
author = "Paul F. Dubois"
description = "Masked Array facility"
maintainer = "Paul F. Dubois"
maintainer_email = "du****@users.sf.net"
url = "http://sourceforge.net/projects/numpy"

RNG-3.1
author = "Lee Busby, Paul F. Dubois, Fred Fritsch"
maintainer = "Paul F. Dubois"
maintainer_email = "du****@users.sf.net"
description = "Cray-like Random number package."
""" % (MA_version, )

# Oops, another bug in Distutils!?
# Write rpm_build.sh pointing to this python
rpm_build_text="""env CFLAGS="$RPM_OPT_FLAGS" %setup.py build\n""" %
sys.executable
rpm_script = open("rpm_build.sh", "w")
rpm_script.write(rpm_build_text)
rpm_script.close()

# Write rpm_install.sh pointing to this python
rpm_install_text=sys.executable +""" setup.py install
--root=$RPM_BUILD_ROOT

cat >INSTALLED_FILES <<EOF
%doc Demo
EOF
find $RPM_BUILD_ROOT -type f | sed -e "s|$RPM_BUILD_ROOT||g"
INSTALLED_FILES


"""
rpm_script = open("rpm_install.sh", "w")
rpm_script.write(rpm_install_text)
rpm_script.close()

setup (name = "Numeric",
version = numeric_version,
maintainer = "Numerical Python Developers",
maintainer_email = "nu**************@lists.sourceforge.net",
description = "Numerical Extension to Python",
long_description = long_description,
url = "http://numpy.sourceforge.net",

# distutils allows you to fix or fudge anything :-)
extra_path = 'Numeric',
packages = packages,
package_dir = package_dir,
headers = headers,
include_dirs = include_dirs,
ext_modules = ext_modules
)

print 'MA Version', MA_version
print 'Numeric Version', numeric_version

Jul 18 '05 #3
>>>>> "drife" == drife <da*******@yahoo.com> writes:

drife> Hi David, I performed the above check, and sure enough,
drife> Numeric is --not-- linked to the ATLAS libraries.

drife> I followed each of your steps outlined above, and Numeric
drife> still is not linking to the ATLAS libraries.

drife> My setup.py file is attached below.

Are you sure that you

1) 'rm -rf build' before rebuilding

2) are using the python / Numeric that you think you are, eg, might
you have one python in /usr/bin and another in /usr/local. root
paths can differ from user paths which may effect which python is
used at install time versus run time

Jul 18 '05 #4
Hi John,

I do have more than one version of Python laying around.

To do the build and install I am typing:

/d2/python/bin/python setup.by build > &! build.out
/d2/python/bin/python setup.by install > &! install.out
Should I be doing something different?

Daran

Jul 18 '05 #5
"drife" <da*******@yahoo.com> writes:
Hi David,

I performed the above check, and sure enough, Numeric
is --not-- linked to the ATLAS libraries.

I followed each of your steps outlined above, and Numeric
still is not linking to the ATLAS libraries.

My setup.py file is attached below. # delete all but the first one in this list if using your own LAPACK/BLAS
sourcelist = [os.path.join('Src', 'lapack_litemodule.c')]
# set these to use your own BLAS;

library_dirs_list = ['/d2/lib/atlas']
libraries_list = ['lapack', 'ptcblas', 'ptf77blas', 'atlas', 'g2c']

# set to true (1), if you also want BLAS optimized matrixmultiply/dot/innerproduct
use_dotblas = 1
include_dirs = ['/d2/include']


This all look right (assuming you've got the right stuff in /d2).

When it compiles, does it look like it's actually doing the linking?
After doing python setup.py build, you can run ldd on the libraries in
the build directory (something like build/lib.linux-i386-2.3/lapack_lite.so).
If that's linked, then it's not being installed right.

You don't have a previous Numeric installation that's being picked up
instead of the one you're trying to install, do you?

At the interpreter prompt, check that
import Numeric
Numeric.__file__


gives you something you're expecting, and not something else.

--
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke
|cookedm(at)physics(dot)mcmaster(dot)ca
Jul 18 '05 #6
Hi David,

Yes, when Numeric compiles it does look like the linking is being
done properly. I captured the build to a file and see a few lines
similar to:
gcc -pthread -shared build/temp.linux-i686-2.4/Src/lapack_litemodule.o
-L/d2/lib/atlas -llapack -lptcblas -lptf77blas -latlas -lg2c -o
build/lib.linux-i686-2.4/lapack_lite.so

I checked the files in build/lib.linux-i686-2.4 and none have
dependencies on ATLAS.

When I run Python interpreter with:
import Numeric
Numeric.__file__


I get the answer I expect.
I am totall baffled.

Any other ideas?

Thanks for your help,

Daran

Jul 18 '05 #7
David,

One more thing. I checked to see if the SciPy libraries had
dependencies on ATLAS. They do not, however, the eigenvector
calculation is still much faster than Numeric?
This is very strange.

Daran

Jul 18 '05 #8
David,

I noticed that the libraries that ATLAS builds are not shared
objects (e.g., liblapack.a). Should these be shared objects?
I see nothing in the ATLAS documentation about building
things as shared objects.
Wondering if this is why the Numeric install is failing.
Daran

Jul 18 '05 #9
drife wrote:
David,

I noticed that the libraries that ATLAS builds are not shared
objects (e.g., liblapack.a). Should these be shared objects?
I see nothing in the ATLAS documentation about building
things as shared objects.
Wondering if this is why the Numeric install is failing.


No, they should work just fine as static libraries.

--
Robert Kern
rk***@ucsd.edu

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter
Jul 18 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

16
by: mamo74 | last post by:
Hello. I am administering a SQL Server (Enterprise Edition on Windows 2003) from some month and can't understand what is going on in the latest week (when the db grow a lot). The DB is around...
5
by: Carlos Moreno | last post by:
Hi, I'm experiencing some really odd behaviour after upgrading to postgresql 7.4.1 We have three different machines: 1) Dual Athlon, 1GB RAM running RedHat 7.3 with all the updates...
4
by: frizzle | last post by:
Hi group. I have a news management system, with a mySQL backend. I tested it yesterday with 1.000.000+ records, testing my url system. I pulled out records calling them by the url field. It was...
2
LacrosseB0ss
by: LacrosseB0ss | last post by:
Hello All; I have the following code in php. It was working fine up until last week or a bit before then the speed came to a grinding halt. Now when the page tries to load it gets stuck in the...
4
by: thatguyNYC | last post by:
Hey there-- I have a form with two listboxes, each populated by SQL statements that pull from separate tables, like this: lstAvail.RowSource = SELECT Person_Id, LastName, FirstName FROM HREmps...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.