I am doing linear algebra with large numarray. It is very efficient, but I
have a small problem due to the size of my data. The dot product of a
10,000x3 double array with a 3x6,250,000 double array will consume 500GB of
memory. I need to break the operations up into managable chunks, so I dont
consume all the available memory and get a segmentation fault.
Its not a problem with numpy, I just need to intelligently slice up one of
my arrays so my routine works within the available system resources. Are
there any utilities that can query how much memory is available?
Thanks,
Darren 5 2344
>>>>> "Darren" == Darren Dale <dd**@cornell.e du> writes:
Darren> Its not a problem with numpy, I just need to intelligently
Darren> slice up one of my arrays so my routine works within the
Darren> available system resources. Are there any utilities that
Darren> can query how much memory is available?
What platform? On Linux at least you can do 'cat /proc/meminfo' to get
more info than you probably want to have...
--
Ville Vainio http://tinyurl.com/2prnb
Darren Dale <dd**@cornell.e du> writes: I am doing linear algebra with large numarray. It is very efficient, but I have a small problem due to the size of my data. The dot product of a 10,000x3 double array with a 3x6,250,000 double array will consume 500GB of memory. I need to break the operations up into managable chunks, so I dont consume all the available memory and get a segmentation fault.
Its not a problem with numpy, I just need to intelligently slice up one of my arrays so my routine works within the available system resources. Are there any utilities that can query how much memory is available?
Not really, it tends to quite operating-system specific.
Instead of saying "What's the largest chunk I can do at a time", how
about "What's the smallest chunk, where bigger chunks won't get me
much?". If you operate on chunks that are on the order of the cache
size of the processor, that's probably sufficient.
Also, if you're using numarray.dot, note that it doesn't use BLAS (yet), so
it's not as efficient as it could be if it used it (through ATLAS, for
instance).
--
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke
|cookedm(at)phy sics(dot)mcmast er(dot)ca
On Fri, 15 Oct 2004 15:59:12 -0400, Darren Dale wrote: I am doing linear algebra with large numarray. It is very efficient, but I have a small problem due to the size of my data. The dot product of a 10,000x3 double array with a 3x6,250,000 double array" will consume 500GB of memory. I need to break the operations up into managable chunks, so I dont consume all the available memory and get a segmentation fault.
Its not a problem with numpy, I just need to intelligently slice up one of my arrays so my routine works within the available system resources. Are there any utilities that can query how much memory is available?
I don't know what you're doing with that, but you're well into the domain
where you may have to trade running time for memory.
I am not familiar with the terms "10,000x3 double array with a 3x6,250,000
double array" (particularly "double array"), but speaking in general
terms, assuming the dot product is something like the vector dot product I
know, you can wrap your two source arrays in an object that lazily
computes the relevant dot product. Shell:
class LazyDotProduct( object):
def __init__(self, a, b):
self.a = a
self.b = b
def __getitem__(sel f, index):
return dot_prodect(sel f.a, self.b, index)
Add an optional cache to getitem if you need it and can afford it.
"dot_produc t" computes the relevant dot product element.
Just a thought; I may be over-extrapolating from what I know.
On Fri, 15 Oct 2004 23:29:48 +0000, Jeremy Bowers wrote: I am not familiar with the terms "10,000x3 double array with a 3x6,250,000 double array" (particularly "double array"),
Oh, duh, array of "doubles". The specification of dimensions had me
thinking of some sort of array where each cell had 2 elements in it or
something :-)
Now I am pretty sure you can compute it lazily. I am doing linear algebra with large numarray. It is very efficient, but I have a small problem due to the size of my data. The dot product of a 10,000x3 double array with a 3x6,250,000 double array will consume 500GB of memory. I need to break the operations up into managable chunks, so I dont consume all the available memory and get a segmentation fault.
Its not a problem with numpy, I just need to intelligently slice up one of my arrays so my routine works within the available system resources. Are there any utilities that can query how much memory is available?
Unless you are running bigmem patches on linux, or the equivalent in
windows, you are limited to 2 gigs of memory per process.
How much memory do you really have?
- Josiah This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Steven T. Hatton |
last post by:
Some people have suggested the desire for code completion and refined
edit-time error detection are an indication of incompetence on the part of
the programmer who wants such features. Unfortunately these ad hominem
rhetorts are frequently introduced into purely technical discussions on the
feasibility of supporting such functionality in C++. That usually serves
to divert the discussion from the technical subject to a discussion of the...
|
by: Mikhail Teterin |
last post by:
Hello!
Consider the following simple accessor function:
typedef struct {
int i;
char name;
} MY_TYPE;
const char *
|
by: Kovan Akrei |
last post by:
Hi,
I wonder if it is possible to get hold of avaiable memory (only RAM) on a
machine through .Net class library? I do not want to call windws API.
I would like to use this to decide how many alive threads my program could
have each time the program runs. I use a number of threads in a simulation
tool.
Many thanks in advance.
Best regards
|
by: jacob navia |
last post by:
Many compilers check printf for errors, lcc-win32 too. But there are
other functions that would be worth to check, specially memset.
Memset is used mainly to clear a memory zone, receiving a pointer to
the start, the value (most of the time zero) and the size of the
memory array to clear.
Problems appear when the size given is not the size of the object
given as its first argument. For instance
void fn(void)
|
by: CBFalconer |
last post by:
We often find hidden, and totally unnecessary, assumptions being
made in code. The following leans heavily on one particular
example, which happens to be in C. However similar things can (and
do) occur in any language.
These assumptions are generally made because of familiarity with
the language. As a non-code example, consider the idea that the
faulty code is written by blackguards bent on foulling the
language. The term...
| |
by: Johan Tibell |
last post by:
I've written a piece of code that uses sockets a lot (I know that
sockets aren't portable C, this is not a question about sockets per
se). Much of my code ended up looking like this:
if (function(socket, args) == -1) {
perror("function");
exit(EXIT_FAILURE);
}
I feel that the ifs destroy the readability of my code. Would it be
|
by: jacob navia |
last post by:
We hear very often in this discussion group that
bounds checking, or safety tests are too expensive
to be used in C.
Several researchers of UCSD have published an interesting
paper about this problem.
http://www.jilp.org/vol9/v9paper10.pdf
Specifically, they measured the overhead of a bounds
|
by: George2 |
last post by:
Hello everyone,
I am using Windows Server 2003. I am confused about the available (memory) value under Physical Memory category.
From search there are two meanings,
1. available means free physical memory, not used yet by any application;
2. available means the total size of physical memory user application could use (exclude System Cache and Kernel Memory, which user application can not use), the user application may actually...
|
by: karthikbalaguru |
last post by:
Hi,
Will 'free' return the memory Immediately to the OS ?
Thx in advans,
Karthik Balaguru
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| |
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| |
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |