I am doing linear algebra with large numarray. It is very efficient, but I
have a small problem due to the size of my data. The dot product of a
10,000x3 double array with a 3x6,250,000 double array will consume 500GB of
memory. I need to break the operations up into managable chunks, so I dont
consume all the available memory and get a segmentation fault.
Its not a problem with numpy, I just need to intelligently slice up one of
my arrays so my routine works within the available system resources. Are
there any utilities that can query how much memory is available?
Thanks,
Darren 5 2320
>>>>> "Darren" == Darren Dale <dd**@cornell.edu> writes:
Darren> Its not a problem with numpy, I just need to intelligently
Darren> slice up one of my arrays so my routine works within the
Darren> available system resources. Are there any utilities that
Darren> can query how much memory is available?
What platform? On Linux at least you can do 'cat /proc/meminfo' to get
more info than you probably want to have...
--
Ville Vainio http://tinyurl.com/2prnb
Darren Dale <dd**@cornell.edu> writes: I am doing linear algebra with large numarray. It is very efficient, but I have a small problem due to the size of my data. The dot product of a 10,000x3 double array with a 3x6,250,000 double array will consume 500GB of memory. I need to break the operations up into managable chunks, so I dont consume all the available memory and get a segmentation fault.
Its not a problem with numpy, I just need to intelligently slice up one of my arrays so my routine works within the available system resources. Are there any utilities that can query how much memory is available?
Not really, it tends to quite operating-system specific.
Instead of saying "What's the largest chunk I can do at a time", how
about "What's the smallest chunk, where bigger chunks won't get me
much?". If you operate on chunks that are on the order of the cache
size of the processor, that's probably sufficient.
Also, if you're using numarray.dot, note that it doesn't use BLAS (yet), so
it's not as efficient as it could be if it used it (through ATLAS, for
instance).
--
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke
|cookedm(at)physics(dot)mcmaster(dot)ca
On Fri, 15 Oct 2004 15:59:12 -0400, Darren Dale wrote: I am doing linear algebra with large numarray. It is very efficient, but I have a small problem due to the size of my data. The dot product of a 10,000x3 double array with a 3x6,250,000 double array" will consume 500GB of memory. I need to break the operations up into managable chunks, so I dont consume all the available memory and get a segmentation fault.
Its not a problem with numpy, I just need to intelligently slice up one of my arrays so my routine works within the available system resources. Are there any utilities that can query how much memory is available?
I don't know what you're doing with that, but you're well into the domain
where you may have to trade running time for memory.
I am not familiar with the terms "10,000x3 double array with a 3x6,250,000
double array" (particularly "double array"), but speaking in general
terms, assuming the dot product is something like the vector dot product I
know, you can wrap your two source arrays in an object that lazily
computes the relevant dot product. Shell:
class LazyDotProduct(object):
def __init__(self, a, b):
self.a = a
self.b = b
def __getitem__(self, index):
return dot_prodect(self.a, self.b, index)
Add an optional cache to getitem if you need it and can afford it.
"dot_product" computes the relevant dot product element.
Just a thought; I may be over-extrapolating from what I know.
On Fri, 15 Oct 2004 23:29:48 +0000, Jeremy Bowers wrote: I am not familiar with the terms "10,000x3 double array with a 3x6,250,000 double array" (particularly "double array"),
Oh, duh, array of "doubles". The specification of dimensions had me
thinking of some sort of array where each cell had 2 elements in it or
something :-)
Now I am pretty sure you can compute it lazily. I am doing linear algebra with large numarray. It is very efficient, but I have a small problem due to the size of my data. The dot product of a 10,000x3 double array with a 3x6,250,000 double array will consume 500GB of memory. I need to break the operations up into managable chunks, so I dont consume all the available memory and get a segmentation fault.
Its not a problem with numpy, I just need to intelligently slice up one of my arrays so my routine works within the available system resources. Are there any utilities that can query how much memory is available?
Unless you are running bigmem patches on linux, or the equivalent in
windows, you are limited to 2 gigs of memory per process.
How much memory do you really have?
- Josiah This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Steven T. Hatton |
last post by:
Some people have suggested the desire for code completion and refined
edit-time error detection are an indication of incompetence on the part of
the programmer who wants such features. ...
|
by: Mikhail Teterin |
last post by:
Hello!
Consider the following simple accessor function:
typedef struct {
int i;
char name;
} MY_TYPE;
const char *
|
by: Kovan Akrei |
last post by:
Hi,
I wonder if it is possible to get hold of avaiable memory (only RAM) on a
machine through .Net class library? I do not want to call windws API.
I would like to use this to decide how many...
|
by: jacob navia |
last post by:
Many compilers check printf for errors, lcc-win32 too. But there are
other functions that would be worth to check, specially memset.
Memset is used mainly to clear a memory zone, receiving a...
|
by: CBFalconer |
last post by:
We often find hidden, and totally unnecessary, assumptions being
made in code. The following leans heavily on one particular
example, which happens to be in C. However similar things can (and...
|
by: Johan Tibell |
last post by:
I've written a piece of code that uses sockets a lot (I know that
sockets aren't portable C, this is not a question about sockets per
se). Much of my code ended up looking like this:
if...
|
by: jacob navia |
last post by:
We hear very often in this discussion group that
bounds checking, or safety tests are too expensive
to be used in C.
Several researchers of UCSD have published an interesting
paper about this...
|
by: George2 |
last post by:
Hello everyone,
I am using Windows Server 2003. I am confused about the available (memory) value under Physical Memory category.
From search there are two meanings,
1. available means free...
|
by: karthikbalaguru |
last post by:
Hi,
Will 'free' return the memory Immediately to the OS ?
Thx in advans,
Karthik Balaguru
|
by: taylorcarr |
last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
|
by: Charles Arthur |
last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
|
by: ryjfgjl |
last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: BarryA |
last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
|
by: nemocccc |
last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers,...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
| |