Hi,
I have the following loop that I think can be written to run faster in
Numeric. I am currently using Numeric.
range_va = [60,61,62,63,64,65,66,67,68,69,70,71,72]
main.xsize= 600
main.ysize= 600
#msgva is an (600x600) Numeric array with mutiple occurrences of the
values in range_va
#sat_id is an (600x600) Numeric array with values ranging from -2 to 2
for z in range_va:
count = 0
mbias = 0
for i in range(main.xsize):
for j in range(main.ysize):
if msgva[i,j] == z:
mbias += sat_id[i,j] # take the sum of the
values
count += 1 # count the occurrences
tmp_array[0,index] = round(mbias/count,1) # store the mean
tmp_array[1,index] = z
index += 1
Any help would be greatly appreciated!
Sincerely,
Sheldon 7 1639
Sheldon wrote: Hi,
I have the following loop that I think can be written to run faster in Numeric. I am currently using Numeric. range_va = [60,61,62,63,64,65,66,67,68,69,70,71,72] main.xsize= 600 main.ysize= 600 #msgva is an (600x600) Numeric array with mutiple occurrences of the values in range_va #sat_id is an (600x600) Numeric array with values ranging from -2 to 2 for z in range_va: count = 0 mbias = 0 for i in range(main.xsize): for j in range(main.ysize): if msgva[i,j] == z: mbias += sat_id[i,j] # take the sum of the values count += 1 # count the occurrences tmp_array[0,index] = round(mbias/count,1) # store the mean tmp_array[1,index] = z index += 1
Any help would be greatly appreciated!
I'm not sufficiently sure this isn't a homework problem, so here's a
partial answer.
Your intuition is correct--there's almost always a faster way to do it
when you're cycling through Numeric array indices in a Python for loop.
Numeric and successors exist mostly to get rid of them.
Given z, you can calculate mbias in one line:
mbias = Numeric.sum(
Numeric.ravel(Numeric.where(msgva==z,sat_id,0)))
Here, the Numeric.where function allows you to filter out entries: it
returns an array with the elements of sat_id, but only where the
corresponding element of msvga==z; otherwise the entry is zero.
Numeric.ravel flattens an multidimensional array into one dimension,
and, of course, Numeric.sum adds up all the elements in the array.
How to get count and to work this into your loop are left as an
exercise.
Carl Banks
Hi Carl,
My days as a student is over for the most part. I am learning python on
my own and Numeric is not properly documented so I am learning by doing
and copying from others. I thought about the problem and a solution
another problem given to me earlier using "putmask" is the solution but
there is a bug that I cannot figure out:
************************
index = 0
for z in range_va:
wk = msgva # working arrary
sattmp = sat_id # working array
mask = where(equal(wk,z),1,0) # creating a mask of
valid data pixs
putmask(sattmp,mask==0,-999) # since zero is valid
data, -999 is used instead
rdata =
compress(ravel(not_equal(sattmp,-999)),ravel(sattmp))
if sum(sum(rdata)) == 0:
av = 0
else:
av = average(rdata,axis=None)
tmparray[0,index] = av
tmparray[1,index] = z
index += 1
print tmparray
***********************************
But the tmparray is returning zeros as averages. When I try just one
value for z everything works. I can't see where I going wrong. I am not
using the original arrays, only the copies and when a new z is chosen
then these are recreated.
Care to help out with this?
/Sheldon
Carl Banks wrote: Sheldon wrote: Hi,
I have the following loop that I think can be written to run faster in Numeric. I am currently using Numeric. range_va = [60,61,62,63,64,65,66,67,68,69,70,71,72] main.xsize= 600 main.ysize= 600 #msgva is an (600x600) Numeric array with mutiple occurrences of the values in range_va #sat_id is an (600x600) Numeric array with values ranging from -2 to 2 for z in range_va: count = 0 mbias = 0 for i in range(main.xsize): for j in range(main.ysize): if msgva[i,j] == z: mbias += sat_id[i,j] # take the sum of the values count += 1 # count the occurrences tmp_array[0,index] = round(mbias/count,1) # store the mean tmp_array[1,index] = z index += 1
Any help would be greatly appreciated!
I'm not sufficiently sure this isn't a homework problem, so here's a partial answer.
Your intuition is correct--there's almost always a faster way to do it when you're cycling through Numeric array indices in a Python for loop. Numeric and successors exist mostly to get rid of them.
Given z, you can calculate mbias in one line:
mbias = Numeric.sum( Numeric.ravel(Numeric.where(msgva==z,sat_id,0)))
Here, the Numeric.where function allows you to filter out entries: it returns an array with the elements of sat_id, but only where the corresponding element of msvga==z; otherwise the entry is zero. Numeric.ravel flattens an multidimensional array into one dimension, and, of course, Numeric.sum adds up all the elements in the array.
How to get count and to work this into your loop are left as an exercise.
Carl Banks
Sheldon wrote: Carl Banks wrote: I'm not sufficiently sure this isn't a homework problem, so here's a partial answer.
[snip] My days as a student is over for the most part. I am learning python on my own and Numeric is not properly documented so I am learning by doing and copying from others.
Are you aware of this guide? http://numeric.scipy.org/numpydoc/numdoc.htm
I've seen better documents but it's fairly complete--I wouldn't call it
improper.
I thought about the problem and a solution another problem given to me earlier using "putmask" is the solution but there is a bug that I cannot figure out: ************************ index = 0 for z in range_va: wk = msgva # working arrary sattmp = sat_id # working array
This appears to be doing something you don't expect. In Python all
names are references; assignments don't create new objects but rather
new names for the same object. In the above, wk and msvga are the same
array: any changes in wk also appear in msvga. So wk[1,1] = 4 also
causes msgva[1,1] to be 4.
Numeric arrays even share data when slicing. If you were to take a
slice of wk, for example, a = wk[0:10,0:10], then modifying a would
also show up in wk (and msvga). (However, slicing most other objects
copies rather than shares the data.)
It's completely different from Matlab, which always copies data.
Here's what you should do:
wk = array(msgva) # this creates a new array for wk
sattmp = array(sat_id)
mask = where(equal(wk,z),1,0) # creating a mask of valid data pixs
Note that the "where" is unnecessary here. equal creates an array of
ones and zeros.
putmask(sattmp,mask==0,-999) # since zero is valid data, -999 is used instead
This would overwrite sat_id unless you copy the array as I've shown
above.
rdata = compress(ravel(not_equal(sattmp,-999)),ravel(sattmp))
I believe you could avoid the above putmask above step and just use
ravel(mask) as the first argument of compress here, or even just
equal(wk,z).
"equal(wx,z)", "mask", and "not_equal(sattmp,-999)" are all equal
arrays.
if sum(sum(rdata)) == 0: av = 0
rdata is one-dimensional, so you only need one sum call; but why do
this? average will still work if the sum is zero.
else: av = average(rdata,axis=None)
axis argument isn't necessary here since rdata is one-dimesional.
tmparray[0,index] = av tmparray[1,index] = z index += 1 print tmparray *********************************** But the tmparray is returning zeros as averages. When I try just one value for z everything works. I can't see where I going wrong. I am not using the original arrays, only the copies and when a new z is chosen then these are recreated. Care to help out with this? /Sheldon
Other than the array sharing mistake, it looks like it should work.
Also, note that this group disdains top-posting; replies should go
below the quoted text because in these sorts of discussions it's best
to keep things in conversational order, especially for the sake of
other interested readers.
Good luck.
Carl Banks
Carl Banks wrote: Sheldon wrote:Carl Banks wrote: I'm not sufficiently sure this isn't a homework problem, so here's a partial answer. [snip] My days as a student is over for the most part. I am learning python on my own and Numeric is not properly documented so I am learning by doing and copying from others.
Are you aware of this guide?
http://numeric.scipy.org/numpydoc/numdoc.htm
I've seen better documents but it's fairly complete--I wouldn't call it improper.
I thought about the problem and a solution another problem given to me earlier using "putmask" is the solution but there is a bug that I cannot figure out: ************************ index = 0 for z in range_va: wk = msgva # working arrary sattmp = sat_id # working array
This appears to be doing something you don't expect. In Python all names are references; assignments don't create new objects but rather new names for the same object. In the above, wk and msvga are the same array: any changes in wk also appear in msvga. So wk[1,1] = 4 also causes msgva[1,1] to be 4.
Numeric arrays even share data when slicing. If you were to take a slice of wk, for example, a = wk[0:10,0:10], then modifying a would also show up in wk (and msvga). (However, slicing most other objects copies rather than shares the data.)
It's completely different from Matlab, which always copies data.
Here's what you should do:
wk = array(msgva) # this creates a new array for wk sattmp = array(sat_id)
mask = where(equal(wk,z),1,0) # creating a mask of valid data pixs
Note that the "where" is unnecessary here. equal creates an array of ones and zeros.
putmask(sattmp,mask==0,-999) # since zero is valid data, -999 is used instead
This would overwrite sat_id unless you copy the array as I've shown above.
rdata = compress(ravel(not_equal(sattmp,-999)),ravel(sattmp))
I believe you could avoid the above putmask above step and just use ravel(mask) as the first argument of compress here, or even just equal(wk,z).
"equal(wx,z)", "mask", and "not_equal(sattmp,-999)" are all equal arrays.
if sum(sum(rdata)) == 0: av = 0
rdata is one-dimensional, so you only need one sum call; but why do this? average will still work if the sum is zero.
else: av = average(rdata,axis=None)
axis argument isn't necessary here since rdata is one-dimesional.
tmparray[0,index] = av tmparray[1,index] = z index += 1 print tmparray *********************************** But the tmparray is returning zeros as averages. When I try just one value for z everything works. I can't see where I going wrong. I am not using the original arrays, only the copies and when a new z is chosen then these are recreated. Care to help out with this? /Sheldon
Other than the array sharing mistake, it looks like it should work.
Also, note that this group disdains top-posting; replies should go below the quoted text because in these sorts of discussions it's best to keep things in conversational order, especially for the sake of other interested readers.
Good luck.
Carl Banks
Thanks for the tips about the array and how it is copied. I figured
this out late yesterday and wrote a dirty solution to this problem but
your is more elegant and concise.
Yes, where statement is not needed and I removed it. The replacement is
the following:
average(compress(ravel(equal(wk,z)),ravel(sattmp)) ,axis=None)
This is much more compact and elegant. Thanks for pointing this out.
I don't know why average() returned a divide by zero error and to avoid
this I inserted this if statement. Now it works much better !
Much obliged,
Sheldon
Sheldon wrote: average(compress(ravel(equal(wk,z)),ravel(sattmp)) ,axis=None) This is much more compact and elegant. Thanks for pointing this out. I don't know why average() returned a divide by zero error and to avoid this I inserted this if statement. Now it works much better !
Probably you had a case where the array length was zero, but it
wouldn't happen in the present case unless your input arrays are zero
by zero.
Carl Banks
Carl Banks wrote:
Sheldon wrote:
average(compress(ravel(equal(wk,z)),ravel(sattmp)) ,axis=None)
This is much more compact and elegant. Thanks for pointing this out.
I don't know why average() returned a divide by zero error and to avoid
this I inserted this if statement. Now it works much better !
Probably you had a case where the array length was zero, but it
wouldn't happen in the present case unless your input arrays are zero
by zero.
Carl Banks
Thanks for you help kind Sir!
/Sheldon This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Stan Heckman |
last post by:
Is the following behavior expected, or have I broken my Numeric
installation somehow?
$python
Python 2.2.2 (#1, Mar 21 2003, 23:01:54)
on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import Numeric
>>> Numeric.__version__
'23.0'
|
by: Kyler Laird |
last post by:
Python 2.3.3 (#2, Jan 4 2004, 12:24:16)
on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import Numeric
>>> import MLab
>>> import Scientific.Statistics
>>> MLab.std(Numeric.array(, Numeric.Int0))
0.0
>>> MLab.std(Numeric.array(, Numeric.UnsignedInt8))
181.01933598375618
|
by: Chris Weisiger |
last post by:
I'm trying to install numeric on my MacOS X box using Darwin, with the
eventual goal of satisfying all of PyGame's dependencies so I can
finally start working on my semester project. I would be using MacPython,
except that I can't seem to get its Package Manager to work. Anyway,
when I try to install numeric, I get the following error:
...
|
by: Johannes Nix |Johannes.Nix |
last post by:
Hi,
I have a tricky problem with Numeric. Some time ago, I have generated
a huge and complex data structure, and stored it using the cPickle
module. Now I want to evaluate it quickly again on a workstation
cluster with 64-Bit Opteron CPUs - I have no more than three days to
do this. Compiling Python and running Numeric has been no problem...
|
by: Gezeala 'Eyah' Bacu\361o II |
last post by:
hey guys..need your help on this..
i have a plpgsql function where in i compute numeric values for my php scripts..
my problem is my function just won't round some numbers properly..
what i want it to do is like this.
example:
| |
by: BBFrost |
last post by:
I'm receiving decimal values from database queries and placing them on a
report page.
The users want to see the following ....
Db Value Display Value
123.3400 123.34
123.0000 123
i.e. I want to trim trailing zeros and (decimal point if no decimal values
|
by: M.A. Oude Kotte |
last post by:
Hi All,
I hope this is the correct mailing list for this question. But neither
postgresql.org nor google could help me out on this subject.
I did find one disturbing topic on the mailing list archives
(http://archives.postgresql.org/pgsql-admin/2000-05/msg00032.php), but
since it was quite old I'm posting my question anyway.
I'm writing...
|
by: Dimitrios Charitatos |
last post by:
Hello,
I suspect that there is a quite straight forward answer to this, but I can't
find it... I want to import an image and extract a matrix (or array) from it
with elements showing the RGB value of each pixel. But I want to be able to do
this with all types of image formats. It was suggested that a function such as
'B =...
|
by: ronysk |
last post by:
Hi,
I am posting here to seek for help on type conversion between Python (Numeric Python) and C.
Attachment A is a math function written in C, which is called by a Python program. I had studied SWIG and Python/C API a bit. I was able to pass numeric array (Numeric Python) into the C function and access/change these arrays.
Problem I...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language...
| |
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it. ...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...
| |