473,695 Members | 2,502 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

switching to numpy and failing, a user story

After using numeric for almost ten years, I decided to attempt to
switch a large codebase (python and C++) to using numpy. Here's are
some comments about how that went.

- The code to automatically switch python stuff over just kind of
works. But it was a 90% solution, I could do the rest by hand. Of
course, the problem is that then the code is still using the old
numeric API, so it's not a long term solution. Unfortunately, to switch
to the numpy API one needs documentation, which is a problem; see
below.

- Well, ok, the automatic switching code doesn't really work all that
well... my uses of RandomArray still work, but they generate different
numbers. The underlying random-number generator must have changed. I'm
sure that it's "better" now, but it's different. This is a major pain
for my regression tests that rely on seeding the random number
generator and getting particular results. But that's ok, I can update
the regressions for the new RNG.

- My extension modules just won't build because the new numpy stuff
lives in a different location from where Numeric used to live. I
probably could fix this, but without the documentation I can't figure
out how to do that. I'd also need to figure out how to port my code to
use the new numpy API instead of the compatibility layer, but I can't
do that without docs either.

- I guess I should just buy the documentation. I don't like this idea,
because I think it's counter-productive to the project to have payware
docs (would Python be successful if you had to buy the documentation? I
don't think so), but that's the way this project goes. I'm doubly
unhappy about it because they payment system is using Paypal and I
don't like Paypal at all, but I guess that's just the way it goes. Oh,
wait, I *can't* buy the docs because I'm not in the US and the payment
page requires a US address. I give up; I guess NumPy is only for people
living in the US.

I guess I'll come back to NumPy in 2010, when the docs are available.

-greg

Oct 4 '06 #1
15 2519
gr**********@gm ail.com wrote:
After using numeric for almost ten years, I decided to attempt to
switch a large codebase (python and C++) to using numpy. Here's are
some comments about how that went.

- The code to automatically switch python stuff over just kind of
works. But it was a 90% solution, I could do the rest by hand. Of
course, the problem is that then the code is still using the old
numeric API, so it's not a long term solution. Unfortunately, to switch
to the numpy API one needs documentation, which is a problem; see
below.
Actually, it's not too hard to do so with the files that come with the source
and utilizing the mailing list. I've converted quite a lot of code without
reference to _The Guide to NumPy_.
- Well, ok, the automatic switching code doesn't really work all that
well... my uses of RandomArray still work, but they generate different
numbers. The underlying random-number generator must have changed. I'm
sure that it's "better" now, but it's different. This is a major pain
for my regression tests that rely on seeding the random number
generator and getting particular results. But that's ok, I can update
the regressions for the new RNG.
Sorry, but the old PRNG code had an non-open source license that prohibited
commercial use, and it had to be replaced.
- My extension modules just won't build because the new numpy stuff
lives in a different location from where Numeric used to live. I
probably could fix this, but without the documentation I can't figure
out how to do that.
That is documented in the files that come with the source. Or you could have
asked us on the numpy mailing list. In short, if you use numpy.distutils ,
everything would have been taken care of for you; otherwise, numpy.get_inclu de()
will give you the location of the headers.
- I guess I should just buy the documentation. I don't like this idea,
because I think it's counter-productive to the project to have payware
docs
<shrugOtherwi se the docs wouldn't have been written at all. And most likely,
numpy wouldn't exist, either.
(would Python be successful if you had to buy the documentation? I
don't think so), but that's the way this project goes. I'm doubly
unhappy about it because they payment system is using Paypal and I
don't like Paypal at all, but I guess that's just the way it goes. Oh,
wait, I *can't* buy the docs because I'm not in the US and the payment
page requires a US address. I give up; I guess NumPy is only for people
living in the US.
Or you could have emailed Travis, and he would have worked around the issue.

I'm sorry that you had such problems, but if you had let us know about them
earlier, we could have helped you out.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

Oct 4 '06 #2
gr**********@gm ail.com wrote:
After using numeric for almost ten years, I decided to attempt to
switch a large codebase (python and C++) to using numpy. Here's are
some comments about how that went.

- The code to automatically switch python stuff over just kind of
works. But it was a 90% solution, I could do the rest by hand. Of
course, the problem is that then the code is still using the old
numeric API, so it's not a long term solution. Unfortunately, to switch
to the numpy API one needs documentation, which is a problem; see
below.
I'm glad to hear of your experiences (good and bad). We need feedback
exactly from users like you in order to improve things. Yep, the code
converter is a 80% solution, but it does work. Improvements are always
welcome.
>
- Well, ok, the automatic switching code doesn't really work all that
well... my uses of RandomArray still work, but they generate different
numbers. The underlying random-number generator must have changed. I'm
sure that it's "better" now, but it's different. This is a major pain
for my regression tests that rely on seeding the random number
generator and getting particular results. But that's ok, I can update
the regressions for the new RNG.
You definitely can't expect the same random number generator. The new
one (thanks to Robert Kern) is quite good.

- My extension modules just won't build because the new numpy stuff
lives in a different location from where Numeric used to live.
This is an easy one and is documented in lots of places on the new
http://www.scipy.org site. Plus, people are always willing to help out
if you just ask. The numpy-discussion list is quite active. Don't be
shy.

- I guess I should just buy the documentation. I don't like this idea,
because I think it's counter-productive to the project to have payware
docs (would Python be successful if you had to buy the documentation? I
don't think so), but that's the way this project goes.
It's probably better to call it "complete documentation." Normal
open-source documentation is available from http://www.scipy.org. There
are lots of people who have helped it. I had to do something to at
least pretend to justify the time NumPy took me to the people who care
about how I spend my time (including my family). This was the best I
could come up with.

I'm doubly
unhappy about it because they payment system is using Paypal and I
don't like Paypal at all, but I guess that's just the way it goes. Oh,
wait, I *can't* buy the docs because I'm not in the US and the payment
page requires a US address.
Is that really true? A lot of people not living in the U.S. have used
Paypal successfully. When difficulties arise, communicating with me
your difficulty is usually productive.

I give up; I guess NumPy is only for people living in the US.

Definitely not true. People in Singapore, Japan, Ghana, South Africa,
France, Germany, New Zealand, Australia, and many other countries are
using NumPy successfully. Gratefully, a few have contributed by buying
the book, but a lot more have downloaded and are successfully using it.

I'm sorry about your experience, you definitely don't *have* to buy the
book to use NumPy. Just like you don't *have* to buy any Python book to
use Python. The amount of documentation for NumPy is growing and I
expect that trend to continue. There is a lot of information in the
source file.
>
I guess I'll come back to NumPy in 2010, when the docs are available.
Or just ask on the mailing lists, use the numpy.oldnumeri c interface
(the differences are all documented in the first few pages of my book
which is available for free now).

Thanks,

-Travis

Oct 4 '06 #3

Travis E. Oliphant wrote:
Definitely not true. People in Singapore, Japan, Ghana, South Africa,
France, Germany, New Zealand, Australia, and many other countries are
using NumPy successfully. Gratefully, a few have contributed by buying
the book, but a lot more have downloaded and are successfully using it.
>From the PayPal site for purchasing your book: "Select Payment Type:
Don't have a PayPal account? You don't need an account. Pay securely
using your credit card." I bought the book from Norway using my credit
card, and received the pdf file soon afterwards. Obviously the book can
be bought without a PayPal account or a us billing address.

NumPy is the most versatile array type I have worked with, including
those of Matlab and Fortran 95. In particular, the explosive memory use
of Matlab is avoided. Keep up the good work!

Oct 4 '06 #4

sturlamolden wrote:
Travis E. Oliphant wrote:
Definitely not true. People in Singapore, Japan, Ghana, South Africa,
France, Germany, New Zealand, Australia, and many other countries are
using NumPy successfully. Gratefully, a few have contributed by buying
the book, but a lot more have downloaded and are successfully using it.
From the PayPal site for purchasing your book: "Select Payment Type:
Don't have a PayPal account? You don't need an account. Pay securely
using your credit card." I bought the book from Norway using my credit
card, and received the pdf file soon afterwards. Obviously the book can
be bought without a PayPal account or a us billing address.
ok, my apologies on this one. I was wrong about the paypal option. I
missed the spot (right at the top, where I should have seen it) to
change the country. There is indeed a way to buy the book if you don't
live in the US.

-greg

Oct 4 '06 #5

Travis E. Oliphant wrote:
gr**********@gm ail.com wrote:
- I guess I should just buy the documentation. I don't like this idea,
because I think it's counter-productive to the project to have payware
docs (would Python be successful if you had to buy the documentation? I
don't think so), but that's the way this project goes.

It's probably better to call it "complete documentation." Normal
open-source documentation is available from http://www.scipy.org. There
are lots of people who have helped it. I had to do something to at
least pretend to justify the time NumPy took me to the people who care
about how I spend my time (including my family). This was the best I
could come up with.
Given the quality of python's (free) documentation and how good it's
been for a very long time, it's bit ironic to be using the phrase
"normal open-source documentation" on this mailing list. Numeric
python, which numpy aspires to be a replacement for, has perfectly
reasonable documentation. It wasn't perfect, but it told you pretty
much everything you needed to know to get started, use the system, and
build extension modules. I guess this set my expectations for NumPy.
Or just ask on the mailing lists, use the numpy.oldnumeri c interface
(the differences are all documented in the first few pages of my book
which is available for free now).
"Ask on the mailing lists" is viable for the occasional question or
detail, but it's not really an efficient way to get started with a
system. At least not for me. But that's fine, I have something that
works (numeric), and I can do what I need to do there.

-greg

Oct 4 '06 #6
gr**********@gm ail.com wrote:
Travis E. Oliphant wrote:
Given the quality of python's (free) documentation and how good it's
been for a very long time, it's bit ironic to be using the phrase
"normal open-source documentation" on this mailing list. Numeric
python, which numpy aspires to be a replacement for, has perfectly
reasonable documentation.
And it is still perfectly useful. Only a couple of details have
changed. The overall description is still useful.

It wasn't perfect, but it told you pretty
much everything you needed to know to get started, use the system, and
build extension modules. I guess this set my expectations for NumPy.
This documentation was written largely due to funding from a national
laboratory. I didn't have those resources. If somebody wanted to step
up to the plate and make me an offer, the NumPy docs could be free as
well. So far, people have been content to buy it a piece at a time.
>
"Ask on the mailing lists" is viable for the occasional question or
detail, but it's not really an efficient way to get started with a
system. At least not for me. But that's fine, I have something that
works (numeric), and I can do what I need to do there.

Absolutely, that's the advantage of open source. If the world moves a
head you don't *have* to. It's entirely your choice. There is no lock-in.
-Travis

Oct 4 '06 #7

gr**********@gm ail.com wrote:
Given the quality of python's (free) documentation and how good it's
been for a very long time, it's bit ironic to be using the phrase
"normal open-source documentation" on this mailing list. Numeric
python, which numpy aspires to be a replacement for, has perfectly
reasonable documentation. It wasn't perfect, but it told you pretty
much everything you needed to know to get started, use the system, and
build extension modules. I guess this set my expectations for NumPy.

NumPy is perhaps the most well thought array object known to man. That
includes those of Matlab, Fortran 95 and C++ libraries (e.g. Blitz++).
I don't think we should be modest about the quality of NumPy. NumPy
allows us to do serious number crunching in a well designed language -
Python. Scientists pay thosands of dollars for software that are not
par with NumPy, and we get NumPy for free.

Those involved in the development of NumPy must receive some
compensation. Financial support to NumPy also ensure that the
developmentcan continue. I for one does not want to see NumPy as
abandonware in the near future. Unfortunately, getting scientists to
make volounteer financial contributions to an open-source project has
proven difficult. A modest charge for the documentation is a fair way
of doing things. I can not be reimbursed by my employer for making a
donation to NumPy, so I would have to take that out of my own pocket. I
can be reimbursed for buying a copy of the documentation, however.

Oct 6 '06 #8
sturlamolden wrote:
Those involved in the development of NumPy must receive some
compensation. Financial support to NumPy also ensure that the
developmentcan continue. I for one does not want to see NumPy as
Then charge for NumPy ... or write a book *besides* the documentation.
One in which you make good use of NumPy and demonstrate the actual
problem solving process.

Charging for docs is just shooting yourself in the foot.

Plus that so called documention seems very unwieldy and unattractive,
long winded text, you can't search it, google won't index it -people
won't find what they are looking for.

I.

Oct 6 '06 #9
Istvan Albert wrote:
sturlamolden wrote:
>Those involved in the development of NumPy must receive some
compensation . Financial support to NumPy also ensure that the
developmentc an continue. I for one does not want to see NumPy as

Then charge for NumPy ... or write a book *besides* ....
Charging for docs is just shooting yourself in the foot.
You overlook the fact that this pricing model means that a developer
can produce some software, and freely deliver that software (with all
needed libraries) to a customer at a price that needn't be so high as
to make each of the programmer's customers pay as if they are using
the full library.

--Scott David Daniels
sc***********@a cm.org
Oct 6 '06 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

22
1999
by: J | last post by:
Hi I hope the title of this message indicates my question. I am looking for basic array functionality in Python and it turns out that there are all these packages which are somehow related. Some are allegedly discontinued but still seem to get updated. Could we start a discussion about which package will or may or should survive ?
20
2577
by: mclaugb | last post by:
Has anyone recompiled the Scientific Computing package using NumPy instead of Numeric? I need a least squares algorithm and a Newton Rhaphson algorithm which is contained in Numeric but all the documentation out there says that Numeric is crap and all code should be using NumPy. Thanks, Bryan
1
1523
by: drife | last post by:
Hello, I use the Python Numeric package extensively, and had been an avid user of the "old" scipy. In my view, both pieces of software are truly first rate, and have greatly improved my productivity in the area of scientific analysis. Thus, I was excited to make the transition to the new scipy core (numpy). Unfortunately, I am experiencing a problem that I cannot sort out. I am running Python 2.4.2 on a Debian box (V3.1), using
13
3917
by: Gary Wessle | last post by:
Hi I am trying to install NumPy in my debian/testing linux 2.6.15-1-686. with no numpy for debian/testing, I am left alone, since the experimental version available by debian will result in a dependency nightmares, so after unpacking the downloaded file "numpy-0.9.6.tar.gz"
10
2226
by: Bryan | last post by:
hi, what is the difference among numeric, numpy and numarray? i'm going to start using matplotlib soon and i'm not sure which one i should use. this page says, "Numarray is a re-implementation of an older Python array module called Numeric" http://www.stsci.edu/resources/software_hardware/numarray
2
3966
by: Chris Smith | last post by:
Howdy, I'm a college student and for one of we are writing programs to numerically compute the parameters of antenna arrays. I decided to use Python to code up my programs. Up to now I haven't had a problem, however we have a problem set where we are creating a large matrix and finding it's inverse to solve the problem. To invert the matrix I've tried using numpy.numarray.linear_algebra.inverse and...
18
22390
by: robert | last post by:
Is there a ready made function in numpy/scipy to compute the correlation y=mx+o of an X and Y fast: m, m-err, o, o-err, r-coef,r-coef-err ? Or a formula to to compute the 3 error ranges? -robert PS: numpy.corrcoef computes only the bare coeff:
5
4855
by: robert | last post by:
Turning algs for old NumPy modules into numpy code I suffer from this: Upon further processing of returns of numpy calculations, lots of data in an apps object tree will become elementary numpy types. First there is some inefficiency in calculations. And then you get data inflation and questionable dependencies - e.g. with pickle,ZODB,mpi's ... : 0.0...
6
2614
by: John [H2O] | last post by:
I have a script: from numpy import float OutD= v= OutD.append(]) On linux: Python 2.5.1 (r251:54863, Mar 7 2008, 04:10:12)
0
8636
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
8996
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
8853
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8831
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7667
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6494
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
1
3011
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2277
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
1978
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.