473,670 Members | 2,609 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

LinearAlgebra incredibly slow for eigenvalue problems

Hello,

I need to calculate the eigenvectors and eigenvalues for a 3600 X 3600
covariance matrix.

The LinearAlgebra package in Python is incredibly slow to perform the
above calculations (about 1.5 hours). This in spite of the fact that
I have installed Numeric with the full ATLAS and LAPACK libraries.

Also note that my computer has dual Pentium IV (3.1 GHz) processors
with 2Gb ram.

Every Web discussion I have seen about such issues indicates that
one can expect huge speed ups if one compiles and installs Numeric
linked against the ATLAS and LAPACK libraries.

Even more perplexing is that the same calculation takes a mere 7 min
in Matlab V6.5. Matlab uses both ATLAS and LAPACK.

Moreover, the above calculation takes the same amount of time for
Numeric to complete with --and-- without ATLAS and PACK. I am certain
that I have done the install correctly.

Can anyone provide some insight?
Thanks in advance for your help.
Daran

Jul 18 '05 #1
9 2573
"drife" <da*******@yaho o.com> writes:
Hello,

I need to calculate the eigenvectors and eigenvalues for a 3600 X 3600
covariance matrix.

The LinearAlgebra package in Python is incredibly slow to perform the
above calculations (about 1.5 hours). This in spite of the fact that
I have installed Numeric with the full ATLAS and LAPACK libraries.

Also note that my computer has dual Pentium IV (3.1 GHz) processors
with 2Gb ram.

Every Web discussion I have seen about such issues indicates that
one can expect huge speed ups if one compiles and installs Numeric
linked against the ATLAS and LAPACK libraries.
Are you *sure* that Numeric is linked against these?
Even more perplexing is that the same calculation takes a mere 7 min
in Matlab V6.5. Matlab uses both ATLAS and LAPACK.

Moreover, the above calculation takes the same amount of time for
Numeric to complete with --and-- without ATLAS and PACK. I am certain
that I have done the install correctly.


This is good evidence that Numeric *isn't* linked to them.

If you're on a Linux system, you can check with ldd:
cookedm@arbutus $ ldd /usr/lib/python2.3/site-packages/Numeric/lapack_lite.so
liblapack.so.3 => /usr/lib/atlas/liblapack.so.3 (0x0000002a9567 7000)
libblas.so.3 => /usr/lib/atlas/libblas.so.3 (0x0000002a95e5 5000)
libg2c.so.0 => /usr/lib/libg2c.so.0 (0x0000002a9672 1000)
libpthread.so.0 => /lib/libpthread.so.0 (0x0000002a9684 2000)
libc.so.6 => /lib/libc.so.6 (0x0000002a9695 7000)
libm.so.6 => /lib/libm.so.6 (0x0000002a96b9 6000)
/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x000000552aaa a000)

You can see that lapack and blas (the Atlas versions) are linked to
the lapack_lite.so.

To install Numeric using Lapack:
- remove the build/ directory in your Numeric sources, so you don't
any old binaries
- edit setup.py and follow the comments on using Lapack (you need to
comment out a few lines, and set some directories)
Also set use_dotblas to 1.
- do the 'python setup.py build', 'python setup.py install' dance.

--
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke
|cookedm(at)phy sics(dot)mcmast er(dot)ca
Jul 18 '05 #2
Hi David,

I performed the above check, and sure enough, Numeric
is --not-- linked to the ATLAS libraries.

I followed each of your steps outlined above, and Numeric
still is not linking to the ATLAS libraries.

My setup.py file is attached below.

Thanks ,

Daran

--#!/usr/bin/env python
# To use:
# python setup.py install
# or:
# python setup.py bdist_rpm (you'll end up with RPMs in dist)
#
import os, sys, string, re
from glob import glob
if not hasattr(sys, 'version_info') or sys.version_inf o <
(2,0,0,'alpha', 0):
raise SystemExit, "Python 2.0 or later required to build Numeric."
import distutils
from distutils.core import setup, Extension

# Get all version numbers
execfile(os.pat h.join('Lib','n umeric_version. py'))
numeric_version = version

execfile(os.pat h.join('Package s', 'MA', 'Lib', 'MA_version.py' ))
MA_version = version

headers = glob (os.path.join ("Include","Num eric","*.h"))
extra_compile_a rgs = [] # You could put "-O4" etc. here.
mathlibs = ['m']
define_macros = [('HAVE_INVERSE_ HYPERBOLIC',Non e)]
undef_macros = []
# You might need to add a case here for your system
if sys.platform in ['win32']:
mathlibs = []
define_macros = []
undef_macros = ['HAVE_INVERSE_H YPERBOLIC']
elif sys.platform in ['mac', 'beos5']:
mathlibs = []

# delete all but the first one in this list if using your own
LAPACK/BLAS
sourcelist = [os.path.join('S rc', 'lapack_litemod ule.c')]
# set these to use your own BLAS;

library_dirs_li st = ['/d2/lib/atlas']
libraries_list = ['lapack', 'ptcblas', 'ptf77blas', 'atlas', 'g2c']

# set to true (1), if you also want BLAS optimized
matrixmultiply/dot/innerproduct
use_dotblas = 1
include_dirs = ['/d2/include']
# You may need to set this to find cblas.h
# e.g. on UNIX using ATLAS this should be
['/usr/include/atlas']
extra_link_args = []

# for MacOS X to link against vecLib if present
VECLIB_PATH = '/System/Library/Frameworks/vecLib.framewor k'
if os.path.exists( VECLIB_PATH):
extra_link_args = ['-framework', 'vecLib']
include_dirs = [os.path.join(VE CLIB_PATH, 'Headers')]

# The packages are split in this way to allow future optional inclusion
# Numeric package
packages = ['']
package_dir = {'': 'Lib'}
include_dirs.ap pend('Include')
ext_modules = [
Extension('_num py',
[os.path.join('S rc', '_numpymodule.c '),
os.path.join('S rc', 'arrayobject.c' ),
os.path.join('S rc', 'ufuncobject.c' )],
extra_compile_a rgs = extra_compile_a rgs),
Extension('mult iarray',
[os.path.join('S rc', 'multiarraymodu le.c')],
extra_compile_a rgs = extra_compile_a rgs),
Extension('umat h',
[os.path.join('S rc', 'umathmodule.c' )],
libraries = mathlibs,
define_macros = define_macros,
undef_macros = undef_macros,
extra_compile_a rgs = extra_compile_a rgs),
Extension('arra yfns',
[os.path.join('S rc', 'arrayfnsmodule .c')],
extra_compile_a rgs = extra_compile_a rgs),
Extension('ranl ib',
[os.path.join('S rc', 'ranlibmodule.c '),
os.path.join('S rc', 'ranlib.c'),
os.path.join('S rc', 'com.c'),
os.path.join('S rc', 'linpack.c')],
extra_compile_a rgs = extra_compile_a rgs),
Extension('lapa ck_lite', sourcelist,
library_dirs = library_dirs_li st,
libraries = libraries_list,
extra_link_args = extra_link_args ,
extra_compile_a rgs = extra_compile_a rgs)
]

# add FFT package (optional)
packages.append ('FFT')
package_dir['FFT'] = os.path.join('P ackages','FFT', 'Lib')
include_dirs.ap pend(os.path.jo in('Packages',' FFT','Include') )
ext_modules.app end(Extension(' FFT.fftpack',
[os.path.join('P ackages','FFT', 'Src',
'fftpackmodule. c'),
os.path.join('P ackages', 'FFT', 'Src',
'fftpack.c')],
extra_compile_a rgs = extra_compile_a rgs))

# add MA package (optional)
packages.append ('MA')
package_dir['MA'] = os.path.join('P ackages', 'MA', 'Lib')

# add RNG package (optional)
packages.append ('RNG')
packages.append ('RNG')
package_dir['RNG'] = os.path.join('P ackages', 'RNG', 'Lib')
include_dirs.ap pend(os.path.jo in('Packages', 'RNG', 'Include'))
ext_modules.app end(Extension(' RNG.RNG',
[os.path.join('P ackages', 'RNG', 'Src',
'RNGmodule.c'),
os.path.join('P ackages', 'RNG', 'Src',
'ranf.c'),
os.path.join('P ackages', 'RNG', 'Src',
'pmath_rng.c')],
extra_compile_a rgs = extra_compile_a rgs))
# add dotblas package (optional)
if use_dotblas:
packages.append ('dotblas')
package_dir['dotblas'] = os.path.join('P ackages', 'dotblas',
'dotblas')
ext_modules.app end(Extension(' _dotblas',
[os.path.join('P ackages', 'dotblas',
'dotblas', '_dotblas.c')],
library_dirs = library_dirs_li st,
libraries = libraries_list,

extra_compile_a rgs=extra_compi le_args))


long_descriptio n = """
Numerical Extension to Python with subpackages.

The authors and maintainers of the subpackages are:

FFTPACK-3.1
maintainer = "Numerical Python Developers"
maintainer_emai l = "nu************ **@lists.source forge.net"
description = "Fast Fourier Transforms"
url = "http://numpy.sourcefor ge.net"

MA-%s
author = "Paul F. Dubois"
description = "Masked Array facility"
maintainer = "Paul F. Dubois"
maintainer_emai l = "du****@users.s f.net"
url = "http://sourceforge.net/projects/numpy"

RNG-3.1
author = "Lee Busby, Paul F. Dubois, Fred Fritsch"
maintainer = "Paul F. Dubois"
maintainer_emai l = "du****@users.s f.net"
description = "Cray-like Random number package."
""" % (MA_version, )

# Oops, another bug in Distutils!?
# Write rpm_build.sh pointing to this python
rpm_build_text= """env CFLAGS="$RPM_OP T_FLAGS" %setup.py build\n""" %
sys.executable
rpm_script = open("rpm_build .sh", "w")
rpm_script.writ e(rpm_build_tex t)
rpm_script.clos e()

# Write rpm_install.sh pointing to this python
rpm_install_tex t=sys.executabl e +""" setup.py install
--root=$RPM_BUILD _ROOT

cat >INSTALLED_FILE S <<EOF
%doc Demo
EOF
find $RPM_BUILD_ROOT -type f | sed -e "s|$RPM_BUILD_R OOT||g"
INSTALLED_FIL ES


"""
rpm_script = open("rpm_insta ll.sh", "w")
rpm_script.writ e(rpm_install_t ext)
rpm_script.clos e()

setup (name = "Numeric",
version = numeric_version ,
maintainer = "Numerical Python Developers",
maintainer_emai l = "nu************ **@lists.source forge.net",
description = "Numerical Extension to Python",
long_descriptio n = long_descriptio n,
url = "http://numpy.sourcefor ge.net",

# distutils allows you to fix or fudge anything :-)
extra_path = 'Numeric',
packages = packages,
package_dir = package_dir,
headers = headers,
include_dirs = include_dirs,
ext_modules = ext_modules
)

print 'MA Version', MA_version
print 'Numeric Version', numeric_version

Jul 18 '05 #3
>>>>> "drife" == drife <da*******@yaho o.com> writes:

drife> Hi David, I performed the above check, and sure enough,
drife> Numeric is --not-- linked to the ATLAS libraries.

drife> I followed each of your steps outlined above, and Numeric
drife> still is not linking to the ATLAS libraries.

drife> My setup.py file is attached below.

Are you sure that you

1) 'rm -rf build' before rebuilding

2) are using the python / Numeric that you think you are, eg, might
you have one python in /usr/bin and another in /usr/local. root
paths can differ from user paths which may effect which python is
used at install time versus run time

Jul 18 '05 #4
Hi John,

I do have more than one version of Python laying around.

To do the build and install I am typing:

/d2/python/bin/python setup.by build > &! build.out
/d2/python/bin/python setup.by install > &! install.out
Should I be doing something different?

Daran

Jul 18 '05 #5
"drife" <da*******@yaho o.com> writes:
Hi David,

I performed the above check, and sure enough, Numeric
is --not-- linked to the ATLAS libraries.

I followed each of your steps outlined above, and Numeric
still is not linking to the ATLAS libraries.

My setup.py file is attached below. # delete all but the first one in this list if using your own LAPACK/BLAS
sourcelist = [os.path.join('S rc', 'lapack_litemod ule.c')]
# set these to use your own BLAS;

library_dirs_li st = ['/d2/lib/atlas']
libraries_list = ['lapack', 'ptcblas', 'ptf77blas', 'atlas', 'g2c']

# set to true (1), if you also want BLAS optimized matrixmultiply/dot/innerproduct
use_dotblas = 1
include_dirs = ['/d2/include']


This all look right (assuming you've got the right stuff in /d2).

When it compiles, does it look like it's actually doing the linking?
After doing python setup.py build, you can run ldd on the libraries in
the build directory (something like build/lib.linux-i386-2.3/lapack_lite.so) .
If that's linked, then it's not being installed right.

You don't have a previous Numeric installation that's being picked up
instead of the one you're trying to install, do you?

At the interpreter prompt, check that
import Numeric
Numeric.__file_ _


gives you something you're expecting, and not something else.

--
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke
|cookedm(at)phy sics(dot)mcmast er(dot)ca
Jul 18 '05 #6
Hi David,

Yes, when Numeric compiles it does look like the linking is being
done properly. I captured the build to a file and see a few lines
similar to:
gcc -pthread -shared build/temp.linux-i686-2.4/Src/lapack_litemodu le.o
-L/d2/lib/atlas -llapack -lptcblas -lptf77blas -latlas -lg2c -o
build/lib.linux-i686-2.4/lapack_lite.so

I checked the files in build/lib.linux-i686-2.4 and none have
dependencies on ATLAS.

When I run Python interpreter with:
import Numeric
Numeric.__file_ _


I get the answer I expect.
I am totall baffled.

Any other ideas?

Thanks for your help,

Daran

Jul 18 '05 #7
David,

One more thing. I checked to see if the SciPy libraries had
dependencies on ATLAS. They do not, however, the eigenvector
calculation is still much faster than Numeric?
This is very strange.

Daran

Jul 18 '05 #8
David,

I noticed that the libraries that ATLAS builds are not shared
objects (e.g., liblapack.a). Should these be shared objects?
I see nothing in the ATLAS documentation about building
things as shared objects.
Wondering if this is why the Numeric install is failing.
Daran

Jul 18 '05 #9
drife wrote:
David,

I noticed that the libraries that ATLAS builds are not shared
objects (e.g., liblapack.a). Should these be shared objects?
I see nothing in the ATLAS documentation about building
things as shared objects.
Wondering if this is why the Numeric install is failing.


No, they should work just fine as static libraries.

--
Robert Kern
rk***@ucsd.edu

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter
Jul 18 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

16
21306
by: mamo74 | last post by:
Hello. I am administering a SQL Server (Enterprise Edition on Windows 2003) from some month and can't understand what is going on in the latest week (when the db grow a lot). The DB is around 250G, and has one table with 1 billion rows. It is performing in a decent way, but can't understand why a particolar table has strong performance problem.
5
2468
by: Carlos Moreno | last post by:
Hi, I'm experiencing some really odd behaviour after upgrading to postgresql 7.4.1 We have three different machines: 1) Dual Athlon, 1GB RAM running RedHat 7.3 with all the updates (kernel, glibc, gcc/g++ 2.96-113)
4
2347
by: frizzle | last post by:
Hi group. I have a news management system, with a mySQL backend. I tested it yesterday with 1.000.000+ records, testing my url system. I pulled out records calling them by the url field. It was incredibly fast, but now, when i call 5 records, ordered by date (which were inserted randomly) it's incredibly slow. And i only have 100.000 records left at the moment. My database structure is below, and i son't know how i could get it to
2
1794
LacrosseB0ss
by: LacrosseB0ss | last post by:
Hello All; I have the following code in php. It was working fine up until last week or a bit before then the speed came to a grinding halt. Now when the page tries to load it gets stuck in the loop below. There are ~2500 records in the table (SQL database, not access). Weird thing is, we have asp applications that run off this database also and they are fine. I copied the query into Query Analyzer and it produced all results within 2...
4
5274
by: thatguyNYC | last post by:
Hey there-- I have a form with two listboxes, each populated by SQL statements that pull from separate tables, like this: lstAvail.RowSource = SELECT Person_Id, LastName, FirstName FROM HREmps WHERE Person_Id NOT IN (SELECT Person_Id FROM Employees) ORDER BY LastName, FirstName; lstSel.RowSource = SELECT Person_Id, cLName, cFName FROM Employees ORDER BY cLName, cFName, Person_Id; This shows all employees in the Employees table (in...
0
8386
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
8903
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
8815
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
8592
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
7421
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
4213
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4393
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2802
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2044
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.