Memory problem

[Yi Xing]

Hi,

I need to read a large amount of data into a list. So I am trying to
see if I'll have any memory problem. When I do
x=range(2700*27 00*3) I got the following message:

Traceback (most recent call last):
File "<stdin>", line 1, in ?
MemoryError

Any way to get around this problem? I have a machine of 4G memory. The
total number of data points (float) that I need to read is in the order
of 200-300 millions.

Thanks.

[John Machin]

Yi Xing wrote:

On a related question: how do I initialize a list or an array with a
pre-specified number of elements, something like
int p[100] in C? I can do append() for 100 times but this looks silly...

Thanks.

Yi Xing

In the case of an array, you may wish to consider the fromfile()
method.

Cheers,
John

[Martin v. Löwis]

Yi Xing wrote:

Thanks! I just found that that I have no problem with
x=[[10.0]*2560*2560]*500, but x=range(1*2560* 2560*30) doesn't work.

That's no surprise. In the first case, try

x[0][0] = 20.0
print x[1][0]

You have the very same (identical) list of 2560*2560 values in x
500 times.

To create such a structure correctly, do

x = [None] * 500
for i in range(500)
x[i] = [10.0]*2560*2560

In any case, check ulimit(1).

Regards,
Martin

[John Machin]

Yi Xing wrote:

Thanks! I just found that that I have no problem with
x=[[10.0]*2560*2560]*500, but x=range(1*2560* 2560*30) doesn't work.

range(1*2560*25 60*30) is creating a list of 196M *unique* ints.
Assuming 32-bit ints and pointers: that's 4 bytes each for the value, 4
for the type pointer, 4 for the refcount and 4 for the actual list
element (a pointer to the 12-byte object). so that's one chunk of
4x196M = 786MB of contiguous list, plus 196M chunks each whatever size
gets allocated for a request of 12 bytes. Let's guess at 16. So the
total memory you need is 3920M.

Now let's look at [[10.0]*2560*2560]*500.
Firstly that creates a tiny list [10.0]. then you create a list that
contains 2560*2560 = 6.5 M references to that *one* object containing
10.0. That's 26MB. Then you make a list of 500 references to that big
list. This new list costs you 2000 bytes. Total required: about 26.2MB.
The minute you start having non-unique numbers instead of 10.0, this
all falls apart.

In any case, your above comparison is nothing at all to do with the
solution that you need, which as already explained will involve
array.array or numpy.

What you now need to do is answer the questions about your pagefile
etc.

Cheers,
John

[Yi Xing]

I used the array module and loaded all the data into an array.
Everything works fine now.
On Aug 14, 2006, at 4:01 PM, John Machin wrote:

Yi Xing wrote:

>Thanks! I just found that that I have no problem with
x=[[10.0]*2560*2560]*500, but x=range(1*2560* 2560*30) doesn't work.

range(1*2560*25 60*30) is creating a list of 196M *unique* ints.
Assuming 32-bit ints and pointers: that's 4 bytes each for the value, 4
for the type pointer, 4 for the refcount and 4 for the actual list
element (a pointer to the 12-byte object). so that's one chunk of
4x196M = 786MB of contiguous list, plus 196M chunks each whatever size
gets allocated for a request of 12 bytes. Let's guess at 16. So the
total memory you need is 3920M.

Now let's look at [[10.0]*2560*2560]*500.
Firstly that creates a tiny list [10.0]. then you create a list that
contains 2560*2560 = 6.5 M references to that *one* object containing
10.0. That's 26MB. Then you make a list of 500 references to that big
list. This new list costs you 2000 bytes. Total required: about 26.2MB.
The minute you start having non-unique numbers instead of 10.0, this
all falls apart.

In any case, your above comparison is nothing at all to do with the
solution that you need, which as already explained will involve
array.array or numpy.

What you now need to do is answer the questions about your pagefile
etc.

Cheers,
John

--
http://mail.python.org/mailman/listinfo/python-list