More efficient array processing

[John [H2O]]

Hello,

I'm trying to do the following:

datagrid = numpy.zeros(360 ,180,3,73,20)

But I get an error saying that the dimensions are too large? Is there a
memory issue here?

So, my workaround is this:

numpoint = 73

datagrid = numpy.zeros(360 ,180,3,73,1)

for np in range(numpoint) :
datagrid[:,:,:,np,0] = datagrid[:,:,:,np,0] + concgrid[:,:,:,np,0]

But this is SLOW.. what can I do to increase efficiency here? Is there a way
to create the larger array? The program loops through several days actually,
filling the 5th dimension. Eventually I just sum the 5th dimension anyway
(as done in the loop of the workaround).

Thanks!
john
--
Configuration
``````````````` ```````````
Plone 2.5.3-final,
CMF-1.6.4,
Zope (Zope 2.9.7-final, python 2.4.4, linux2),
Five 1.4.1,
Python 2.4.4 (#1, Jul 3 2007, 22:58:17) [GCC 4.1.1 20070105 (Red Hat
4.1.1-51)],
PIL 1.1.6
Mailman 2.1.9
Postfix 2.4.5
Procmail v3.22 2001/09/10
--
View this message in context: http://www.nabble.com/More-efficient...p20136676.html
Sent from the Python - python-list mailing list archive at Nabble.com.

[Marc 'BlackJack' Rintsch]

On Thu, 23 Oct 2008 11:11:32 -0700, John [H2O] wrote:

I'm trying to do the following:

datagrid = numpy.zeros(360 ,180,3,73,20)

But I get an error saying that the dimensions are too large? Is there a
memory issue here?

Let's see:

You have: 360 * 180 * 3 * 73 * 20 * 8 bytes
You want: GiB
* 2.1146536
/ 0.47289069

Do you have a 32 bit system? Then 2Â*GiB is too much for a process.

Ciao,
Marc 'BlackJack' Rintsch

[John [H2O]]

Thanks for the clarification.

What is strange though, is that I have several Fortran programs that create
the exact same array srtucture... wouldn't they be restricted to the 2Gb
limit as well?

Thoughts on a more efficient work around?
Marc 'BlackJack' Rintsch wrote:

On Thu, 23 Oct 2008 11:11:32 -0700, John [H2O] wrote:

>I'm trying to do the following:

datagrid = numpy.zeros(360 ,180,3,73,20)

But I get an error saying that the dimensions are too large? Is there a
memory issue here?

Let's see:

You have: 360 * 180 * 3 * 73 * 20 * 8 bytes
You want: GiB
* 2.1146536
/ 0.47289069

Do you have a 32 bit system? Then 2Â*GiB is too much for a process.

Ciao,
Marc 'BlackJack' Rintsch
--
http://mail.python.org/mailman/listinfo/python-list

--
View this message in context: http://www.nabble.com/More-efficient...p20137263.html
Sent from the Python - python-list mailing list archive at Nabble.com.

[Marc 'BlackJack' Rintsch]

On Thu, 23 Oct 2008 11:44:04 -0700, John [H2O] wrote:

What is strange though, is that I have several Fortran programs that
create the exact same array srtucture... wouldn't they be restricted to
the 2Gb limit as well?

They should be. What about the data type of the elements? Any chance
they are just 4 byte floats in your Fortran code i.e. C floats instead of
C doubles like the default in `numpy`?

Ciao,
Marc 'BlackJack' Rintsch

[John [H2O]]

I'm using zeros with type np.float, is there a way to define the data type to
be 4 byte floats?
Marc 'BlackJack' Rintsch wrote:

>
On Thu, 23 Oct 2008 11:44:04 -0700, John [H2O] wrote:

>What is strange though, is that I have several Fortran programs that
create the exact same array srtucture... wouldn't they be restricted to
the 2Gb limit as well?

They should be. What about the data type of the elements? Any chance
they are just 4 byte floats in your Fortran code i.e. C floats instead of
C doubles like the default in `numpy`?

Ciao,
Marc 'BlackJack' Rintsch
--
http://mail.python.org/mailman/listinfo/python-list

--
View this message in context: http://www.nabble.com/More-efficient...p20139062.html
Sent from the Python - python-list mailing list archive at Nabble.com.

[Marc 'BlackJack' Rintsch]

On Thu, 23 Oct 2008 13:56:22 -0700, John [H2O] wrote:

I'm using zeros with type np.float, is there a way to define the data
type to be 4 byte floats?

Yes:

In [13]: numpy.zeros(5, numpy.float32)
Out[13]: array([ 0., 0., 0., 0., 0.], dtype=float32)

Ciao,
Marc 'BlackJack' Rintsch

[Robert Kern]

John [H2O] wrote:

I'm using zeros with type np.float, is there a way to define the data type to
be 4 byte floats?

np.float32. np.float is not part of the numpy API. It's just Python's builtin
float type which corresponds to C doubles.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Ivan Reborin]

On Thu, 23 Oct 2008 11:44:04 -0700 (PDT), "John [H2O]"
<wa******@gmail .comwrote:

>
Thanks for the clarification.

What is strange though, is that I have several Fortran programs that create
the exact same array srtucture... wouldn't they be restricted to the 2Gb
limit as well?

Depends on lot of things, as Mark has hinted.
But, why are you rewriting fortran subroutines to py ? Expecially for
this kind of array processing.

And, if it's no secret, would you mind telling what do you need an
array of that size for ?
I cannot think of many uses that would require an array of that size
and that many dimensions, which couldn't be rewritten in a more
efficient manner, to several smaller arrays.
--
Ivan

>
Thoughts on a more efficient work around?

[Ivan Reborin]

On Fri, 24 Oct 2008 00:32:11 +0200, Ivan Reborin
<ir******@delet e.this.gmail.co mwrote:

>On Thu, 23 Oct 2008 11:44:04 -0700 (PDT), "John [H2O]"
<wa******@gmai l.comwrote:

>>
Thanks for the clarification.

What is strange though, is that I have several Fortran programs that create
the exact same array srtucture... wouldn't they be restricted to the 2Gb
limit as well?

Depends on lot of things, as Mark has hinted.

*Marc*

Apologies.

--
Ivan

[sturlamolden]

On Oct 23, 8:11*pm, "John [H2O]" <washa...@gmail .comwrote:

datagrid = numpy.zeros(360 ,180,3,73,20)

On a 32 bit system, try this instead:

datagrid = numpy.zeros((36 0,180,3,73,20), dtype=numpy.flo at32)

(if you can use single precision that is.)