P: n/a

Turning algs for old NumPy modules into numpy code I suffer from this:
Upon further processing of returns of numpy calculations, lots of data in an apps object tree will become elementary numpy types.
First there is some inefficiency in calculations. And then you get data inflation and questionable dependencies  e.g. with pickle,ZODB,mpi's ... :
>>l=array((1.,0)) l.prod()
0.0
>>cPickle.dumps(_)
"cnumpy.core.multiarray\nscalar\np1\n(cnumpy\ndtyp e\np2\n(S'f8'\nI0\nI1\ntRp3\n(I2\nS'<'\nNNNI1\nI1\ntbS'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'\n tRp4\n."
>>cPickle.dumps(0.0)
'F0\n.'
>>l=array((1,0)) l.prod()
0
>>cPickle.dumps(_)
"cnumpy.core.multiarray\nscalar\np1\n(cnumpy\ndtyp e\np2\n(S'i4'\nI0\nI1\ntRp3\n(I2\nS'<'\nNNNI1\nI1\ntbS'\\x00\\x00\\x00\\x00'\ntRp4\n."
>>cPickle.dumps(0)
'I0\n.'
>>type(l.prod())
<type 'numpy.int32'>
To avoid this you'd need a type cast in Python code everywhere you get scalars from numpy into a python variable. Error prone task. Or check/rerender your whole object tree.
Wouldn't it be much better if numpy would return Python scalars for float64 (maybe even for float32) and int32, int64 ... where possible? (as numarray and Numeric did)
I suppose numpy knows internally very quickly how to cast.
Or is there maybe a configsetting to turn numpy this way?
Robert  
Share this Question
P: n/a

robert wrote:
Turning algs for old NumPy modules into numpy code I suffer from this:
Upon further processing of returns of numpy calculations, lots of data in an apps object tree will become elementary numpy types.
First there is some inefficiency in calculations. And then you get data inflation and questionable dependencies  e.g. with pickle,ZODB,mpi's ... :
>>>l=array((1.,0)) l.prod()
0.0
>>>cPickle.dumps(_)
"cnumpy.core.multiarray\nscalar\np1\n(cnumpy\ndtyp e\np2\n(S'f8'\nI0\nI1\ntRp3\n(I2\nS'<'\nNNNI1\nI1\ntbS'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'\n tRp4\n."
>>>cPickle.dumps(0.0)
'F0\n.'
>>>l=array((1,0)) l.prod()
0
>>>cPickle.dumps(_)
"cnumpy.core.multiarray\nscalar\np1\n(cnumpy\ndtyp e\np2\n(S'i4'\nI0\nI1\ntRp3\n(I2\nS'<'\nNNNI1\nI1\ntbS'\\x00\\x00\\x00\\x00'\ntRp4\n."
>>>cPickle.dumps(0)
'I0\n.'
>>>type(l.prod())
<type 'numpy.int32'>
To avoid this you'd need a type cast in Python code everywhere you get scalars from numpy into a python variable. Error prone task. Or check/rerender your whole object tree.
Wouldn't it be much better if numpy would return Python scalars for float64 (maybe even for float32) and int32, int64 ... where possible? (as numarray and Numeric did)
I suppose numpy knows internally very quickly how to cast.
The short answer is no, it would not be better. There are some trade
offs involved here, but overall, always returning numpy scalars is a
significant improvement over returning Python scalars some of the time.
Which is why numpy does it that way now; it was a conscious choice, it
didn't just happen. Please search the archives of numpydiscussion for
previous discussions of this and if that is not enlightening enough
please ask at on the numpydiscussion list (the address of which just
changed and I don't have it handy, but I'm sure you can find it).
For your particular issue, you might try tweaking pickle to convert
int64 objects to int objects. Assuming of course that you have enough of
these to matter, otherwise, I suggest just leaving things alone.
tim  
P: n/a

Tim Hochberg wrote:
robert wrote:
>To avoid this you'd need a type cast in Python code everywhere you get scalars from numpy into a python variable. Error prone task. Or check/rerender your whole object tree. Wouldn't it be much better if numpy would return Python scalars for float64 (maybe even for float32) and int32, int64 ... where possible? (as numarray and Numeric did) I suppose numpy knows internally very quickly how to cast.
The short answer is no, it would not be better. There are some trade
offs involved here, but overall, always returning numpy scalars is a
significant improvement over returning Python scalars some of the time.
Which is why numpy does it that way now; it was a conscious choice, it
didn't just happen. Please search the archives of numpydiscussion for
previous discussions of this and if that is not enlightening enough
please ask at on the numpydiscussion list (the address of which just
changed and I don't have it handy, but I'm sure you can find it).
Didn't find the relevant reasoning within time. Yet guess the reason is isolatedmodulecentric.
All further computations in python are much slower and I cannot even see a speed increase when (rare case) puting a numpyic scalar back into a numpy array:
>>a=array([1.,0,0,0,0]) f=1.0 fn=a[0] type(fn)
<type 'numpy.float64'>
>>timeit.Timer("f+f",glbls=globals()).timeit(10000 )
0.0048265910890909324
>>timeit.Timer("f+f",glbls=globals()).timeit(10000 0)
0.045992158221226376
>>timeit.Timer("fn+fn",glbls=globals()).timeit(100 000)
0.14901307289054877
>>timeit.Timer("a[1]=f",glbls=globals()).timeit(100000)
0.060825607723899111
>>timeit.Timer("a[1]=fn",glbls=globals()).timeit(100000)
0.059519575812004177
>>timeit.Timer("x=a[0]",glbls=globals()).timeit(100000)
0.12302317752676117
>>timeit.Timer("x=float(a[0])",glbls=globals()).timeit(100000)
0.31556273213496411
creation of numpy scalar objects seems not be cheap/advantagous anyway:
>>oa=array([1.0,1.0,1.0,1.0,1],numpy.object) oa
array([1.0, 1.0, 1.0, 1.0, 1], dtype=object)
>>timeit.Timer("x=a[0]",glbls=globals()).timeit(100000)
0.12025438987348025
>>timeit.Timer("x=oa[0]",glbls=globals()).timeit(100000)
0.050609225474090636
>>timeit.Timer("a+a",glbls=globals()).timeit(10000 0)
1.3081539692893784
>>timeit.Timer("oa+oa",glbls=globals()).timeit(100 000)
1.5201345422392478
For your particular issue, you might try tweaking pickle to convert
int64 objects to int objects. Assuming of course that you have enough of
these to matter, otherwise, I suggest just leaving things alone.
( int64've not had so far don't know whats with python L's )
the main problem is with hundreds of allday normal floats (now numpy.float64) and ints (numpy.int32) variables.
Speed issues, memory consumption... And a pickled tree cannot be read by an app which has not numpy available. and the pickles are very big.
I still really wonder how all this observations and the things which I can imagine so far can sum up to an overall advantage for letting numpy.float64 & numpy.int32 scalars out by default  and also possibly not for numpy.float32 which has somewhat importance in practice ?
Letting out nan and inf.. objects and offering an explicit type case is of course ok.
Robert  
P: n/a

robert wrote:
Didn't find the relevant reasoning within time. Yet guess the reason is isolatedmodulecentric.
I gave you a brief rundown on this list already. http://mail.python.org/pipermail/pyt...er/411145.html
And I'll note again that a fuller discussion is given in Chapter 2 of the _Guide
to NumPy_. http://numpy.scipy.org/numpybooksample.pdf
And yet again, the best place for numpy questions is the numpy mailing list, not
here. Here, you will get maybe one or two people responding to you, usually me,
and I'm a cranky SOB. There you will get much nicer people answering your
questions and more fully. http://www.scipy.org/Mailing_Lists

Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
 Umberto Eco  
P: n/a

Robert Kern wrote:
robert wrote:
>Didn't find the relevant reasoning within time. Yet guess the reason is isolatedmodulecentric.
I gave you a brief rundown on this list already.
http://mail.python.org/pipermail/pyt...er/411145.html
think I took this into account already for this discussion.
>>array([1.,.0,4,4,],float32)
array([ 1., 0., 4., 4.], dtype=float32)
>>a=_ a+3
array([ 4., 3., 7., 7.], dtype=float32)
>>a+3.0
array([ 4., 3., 7., 7.], dtype=float32)
>>3+a
array([ 4., 3., 7., 7.], dtype=float32)
>>3.0+a
array([ 4., 3., 7., 7.], dtype=float32)
>>3.0*a
array([ 3., 0., 12., 12.], dtype=float32)
numpy does anyway not force the precision upwards for float32 x pythonfloat. ( what is good so).
There remains the argument, that (float64,int32) scalars coming out should  by default  support the array interface.
How many people are there to expect and use this? I'd have never noticed it, if it wouldn't have been mentioned here. Have never seen such code nor seen somewhere or felt myself such a requirement. Thats very special an maybe can be turned on by a global option  if there is more than a handful of users for that.
I still think this is overdesign and that it brings much much more disadvantages than advantages to let these beasts out by default into a general purpose language like python. The target area for numpy output is much bigger than that e.g. for matlab script someone uses in a rush to create a paper. Maybe for users who want to make an altmatlabonly box out of python there could be a global switch somewhere or a enforcing versions of the data types ...
Seeing the speed impact and pickleproblems now everywere on postcomputations upon numpy out, its a critical decision to migrate code to numpy. Almost a killer. Even if I spread float()casts everywhere, this cost a lot of speed, makes code ugly etc., and its an holey sieve.
I think I'll stay as a voice to vote heavily against that scheme of numpy scalar types. 11 on a scale from 0 to 10 :)
And yet again, the best place for numpy questions is the numpy mailing list, not
here. Here, you will get maybe one or two people responding to you, usually me,
and I'm a cranky SOB. There you will get much nicer people answering your
questions and more fully.
http://www.scipy.org/Mailing_Lists
Maybe once I take the hurdle to use this. Access and searching on such lists is somewhat proprietry. Numerics is a major field in Python land. There are also lots of cross relations to other libs and techniques. Maybe there could be a nntpcomfortable comp.lang.python.numeric for users  and also a comp.lang.python.net, comp.lang.python.ui. I think that would greatly strentghen Pythons "marketing" in the numerics domain. main clp's posting frequency is too high anyway meanwhile.
Robert  
P: n/a

robert wrote:
There remains the argument, that (float64,int32) scalars coming out should  by default  support the array interface.
How many people are there to expect and use this? I'd have never noticed it, if it wouldn't have been mentioned here. Have never seen such code nor seen somewhere or felt myself such a requirement. Thats very special an maybe can be turned on by a global option  if there is more than a handful of users for that.
It derived from our experience building scipy. Writing a library of functions
that work on scalars, vectors and higherdimensional arrays requires either a
certain amount of generic behavior in its types or a lot of hairy code. We went
for the former. "Global options" affecting the behavior of types don't fit very
well in a library.
I still think this is overdesign and that it brings much much more disadvantages than advantages to let these beasts out by default into a general purpose language like python.
It's a judgement call. You judged differently than we have. <shrug>
I think I'll stay as a voice to vote heavily against that scheme of numpy scalar types. 11 on a scale from 0 to 10 :)
Vote all you like; no one's taking a poll at this time.
[I wrote:]
>And yet again, the best place for numpy questions is the numpy mailing list, not here. Here, you will get maybe one or two people responding to you, usually me, and I'm a cranky SOB. There you will get much nicer people answering your questions and more fully.
http://www.scipy.org/Mailing_Lists
Maybe once I take the hurdle to use this. Access and searching on such lists is somewhat proprietry. Numerics is a major field in Python land. There are also lots of cross relations to other libs and techniques. Maybe there could be a nntpcomfortable comp.lang.python.numeric for users  and also a comp.lang.python.net, comp.lang.python.ui. I think that would greatly strentghen Pythons "marketing" in the numerics domain. main clp's posting frequency is too high anyway meanwhile.
When you have to put "numpy" in your subject lines because you're asking
questions about how and why numpy does this one particular thing, it's time to
think about posting to the appropriate list. If you need NNTP, use GMane. http://dir.gmane.org/gmane.comp.python.numeric.general
Here's the root of the problem: many of the people you want to talk to aren't
here. They don't read comp.lang.python; mostly it's just me, and I'm getting
crankier by the character.

Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
 Umberto Eco   This discussion thread is closed Replies have been disabled for this discussion.   Question stats  viewed: 4288
 replies: 5
 date asked: Nov 17 '06
