By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
439,977 Members | 1,369 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 439,977 IT Pros & Developers. It's quick & easy.

More efficient array processing

P: n/a

Hello,

I'm trying to do the following:

datagrid = numpy.zeros(360,180,3,73,20)

But I get an error saying that the dimensions are too large? Is there a
memory issue here?

So, my workaround is this:

numpoint = 73

datagrid = numpy.zeros(360,180,3,73,1)

for np in range(numpoint):
datagrid[:,:,:,np,0] = datagrid[:,:,:,np,0] + concgrid[:,:,:,np,0]

But this is SLOW.. what can I do to increase efficiency here? Is there a way
to create the larger array? The program loops through several days actually,
filling the 5th dimension. Eventually I just sum the 5th dimension anyway
(as done in the loop of the workaround).

Thanks!
john
--
Configuration
``````````````````````````
Plone 2.5.3-final,
CMF-1.6.4,
Zope (Zope 2.9.7-final, python 2.4.4, linux2),
Five 1.4.1,
Python 2.4.4 (#1, Jul 3 2007, 22:58:17) [GCC 4.1.1 20070105 (Red Hat
4.1.1-51)],
PIL 1.1.6
Mailman 2.1.9
Postfix 2.4.5
Procmail v3.22 2001/09/10
--
View this message in context: http://www.nabble.com/More-efficient...p20136676.html
Sent from the Python - python-list mailing list archive at Nabble.com.

Oct 23 '08 #1
Share this Question
Share on Google+
9 Replies


P: n/a
On Thu, 23 Oct 2008 11:11:32 -0700, John [H2O] wrote:
I'm trying to do the following:

datagrid = numpy.zeros(360,180,3,73,20)

But I get an error saying that the dimensions are too large? Is there a
memory issue here?
Let's see:

You have: 360 * 180 * 3 * 73 * 20 * 8 bytes
You want: GiB
* 2.1146536
/ 0.47289069

Do you have a 32 bit system? Then 2*GiB is too much for a process.

Ciao,
Marc 'BlackJack' Rintsch
Oct 23 '08 #2

P: n/a

Thanks for the clarification.

What is strange though, is that I have several Fortran programs that create
the exact same array srtucture... wouldn't they be restricted to the 2Gb
limit as well?

Thoughts on a more efficient work around?
Marc 'BlackJack' Rintsch wrote:

On Thu, 23 Oct 2008 11:11:32 -0700, John [H2O] wrote:
>I'm trying to do the following:

datagrid = numpy.zeros(360,180,3,73,20)

But I get an error saying that the dimensions are too large? Is there a
memory issue here?
Let's see:

You have: 360 * 180 * 3 * 73 * 20 * 8 bytes
You want: GiB
* 2.1146536
/ 0.47289069

Do you have a 32 bit system? Then 2*GiB is too much for a process.

Ciao,
Marc 'BlackJack' Rintsch
--
http://mail.python.org/mailman/listinfo/python-list
--
View this message in context: http://www.nabble.com/More-efficient...p20137263.html
Sent from the Python - python-list mailing list archive at Nabble.com.

Oct 23 '08 #3

P: n/a
On Thu, 23 Oct 2008 11:44:04 -0700, John [H2O] wrote:
What is strange though, is that I have several Fortran programs that
create the exact same array srtucture... wouldn't they be restricted to
the 2Gb limit as well?
They should be. What about the data type of the elements? Any chance
they are just 4 byte floats in your Fortran code i.e. C floats instead of
C doubles like the default in `numpy`?

Ciao,
Marc 'BlackJack' Rintsch
Oct 23 '08 #4

P: n/a

I'm using zeros with type np.float, is there a way to define the data type to
be 4 byte floats?
Marc 'BlackJack' Rintsch wrote:
>
On Thu, 23 Oct 2008 11:44:04 -0700, John [H2O] wrote:
>What is strange though, is that I have several Fortran programs that
create the exact same array srtucture... wouldn't they be restricted to
the 2Gb limit as well?

They should be. What about the data type of the elements? Any chance
they are just 4 byte floats in your Fortran code i.e. C floats instead of
C doubles like the default in `numpy`?

Ciao,
Marc 'BlackJack' Rintsch
--
http://mail.python.org/mailman/listinfo/python-list

--
View this message in context: http://www.nabble.com/More-efficient...p20139062.html
Sent from the Python - python-list mailing list archive at Nabble.com.

Oct 23 '08 #5

P: n/a
On Thu, 23 Oct 2008 13:56:22 -0700, John [H2O] wrote:
I'm using zeros with type np.float, is there a way to define the data
type to be 4 byte floats?
Yes:

In [13]: numpy.zeros(5, numpy.float32)
Out[13]: array([ 0., 0., 0., 0., 0.], dtype=float32)

Ciao,
Marc 'BlackJack' Rintsch
Oct 23 '08 #6

P: n/a
John [H2O] wrote:
I'm using zeros with type np.float, is there a way to define the data type to
be 4 byte floats?
np.float32. np.float is not part of the numpy API. It's just Python's builtin
float type which corresponds to C doubles.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

Oct 23 '08 #7

P: n/a
On Thu, 23 Oct 2008 11:44:04 -0700 (PDT), "John [H2O]"
<wa******@gmail.comwrote:
>
Thanks for the clarification.

What is strange though, is that I have several Fortran programs that create
the exact same array srtucture... wouldn't they be restricted to the 2Gb
limit as well?
Depends on lot of things, as Mark has hinted.
But, why are you rewriting fortran subroutines to py ? Expecially for
this kind of array processing.

And, if it's no secret, would you mind telling what do you need an
array of that size for ?
I cannot think of many uses that would require an array of that size
and that many dimensions, which couldn't be rewritten in a more
efficient manner, to several smaller arrays.
--
Ivan

>
Thoughts on a more efficient work around?

Oct 23 '08 #8

P: n/a
On Fri, 24 Oct 2008 00:32:11 +0200, Ivan Reborin
<ir******@delete.this.gmail.comwrote:
>On Thu, 23 Oct 2008 11:44:04 -0700 (PDT), "John [H2O]"
<wa******@gmail.comwrote:
>>
Thanks for the clarification.

What is strange though, is that I have several Fortran programs that create
the exact same array srtucture... wouldn't they be restricted to the 2Gb
limit as well?

Depends on lot of things, as Mark has hinted.
*Marc*

Apologies.

--
Ivan
Oct 23 '08 #9

P: n/a
On Oct 23, 8:11*pm, "John [H2O]" <washa...@gmail.comwrote:
datagrid = numpy.zeros(360,180,3,73,20)
On a 32 bit system, try this instead:

datagrid = numpy.zeros((360,180,3,73,20), dtype=numpy.float32)

(if you can use single precision that is.)




Oct 24 '08 #10

This discussion thread is closed

Replies have been disabled for this discussion.