Python and Memory

[Ben Fairbank]

I have to pick a language to commit to for general purpose
scientific/statistical/utility/database programming at my office and
have pretty much narrowed it down to R or Python. Problem: none of
the various Python books I have looked into has had much to say about
memory. I will be operating on some big arrays, probably with Numpy;
if I run out of space and upgrade a Win 2000 or Win XP pro machine
from 256 Meg to 500Meg or even 1G will Python automatically recognize
and take advantage of the increase? Where are questionss such as this
discussed in the documentation?

Thanks for any suggestions,

Ben Fairbank

[Oren Tirosh]

On Thu, Oct 02, 2003 at 04:59:15AM +0000, Ben Fairbank wrote:

I have to pick a language to commit to for general purpose
scientific/statistical/utility/database programming at my office and
have pretty much narrowed it down to R or Python. Problem: none of
the various Python books I have looked into has had much to say about
memory. I will be operating on some big arrays, probably with Numpy;
Python isn't terribly memory-efficient. Every int, for example, takes all
the overhead of an object plus any allocator overheads. But if the bulk
of the data is going to be numpy arrays this isn't a problem. An array
object has a small one-time overhead and then each item takes exactly its
native size in bytes. Make sure you use in-place operations as much as
possible on arrays so you don't force numpy to allocate large temporary
arrays.
if I run out of space and upgrade a Win 2000 or Win XP pro machine
from 256 Meg to 500Meg or even 1G will Python automatically recognize
and take advantage of the increase? Where are questionss such as this
discussed in the documentation?

Python can immediately use any memory expansion. It also runs on 64 bit
platforms if you ever need more than 4GB of data.

Oren

[Michael Hudson]

ba*@texas.antispam.net (Ben Fairbank) writes:

I have to pick a language to commit to for general purpose
scientific/statistical/utility/database programming at my office and
have pretty much narrowed it down to R or Python. Problem: none of
the various Python books I have looked into has had much to say about
memory. I will be operating on some big arrays, probably with Numpy;
Hmm. Python is not *amazingly* memory efficient. But if you're
dealing with relatively few Numpy arrays, the memory consumption will
probably not be that much more than any other language you use
(unlike, say, using a list of floats).
if I run out of space and upgrade a Win 2000 or Win XP pro machine
from 256 Meg to 500Meg or even 1G will Python automatically recognize
and take advantage of the increase?
This is more a question about the OS than Python. I would be amazed
if Python couldn't use the extra memory.
Where are questionss such as this discussed in the documentation?

I don't think they are, really. The numpy manual might have a few
words to say on the subject.

Cheers,
mwh

--
Famous remarks are very seldom quoted correctly.
-- Simeon Strunsky

[Alex Martelli]

Ben Fairbank wrote:

I have to pick a language to commit to for general purpose
scientific/statistical/utility/database programming at my office and
have pretty much narrowed it down to R or Python. Problem: none of
the various Python books I have looked into has had much to say about
memory. I will be operating on some big arrays, probably with Numpy;
if I run out of space and upgrade a Win 2000 or Win XP pro machine
from 256 Meg to 500Meg or even 1G will Python automatically recognize
and take advantage of the increase? Where are questionss such as this
discussed in the documentation?

Python gets its information about memory (and allocates its memory)
from the operating system. This is not discussed in the documentation
because it's more or less taken for granted -- that's what one should
assume of any app or language unless other behavior is documented. If
your Win/2K, Win/XP, or whatever, is able to see your new memory and
make it available to apps (and I see no reason why not), Python, like
any other normal app, should be able to use it just fine.
Alex

[Cameron Laird]

In article <3f***************@news.houston.sbcglobal.net>,
Ben Fairbank <ba*@texas.antispam.net> wrote:

I have to pick a language to commit to for general purpose
scientific/statistical/utility/database programming at my office and
have pretty much narrowed it down to R or Python. Problem: none of
the various Python books I have looked into has had much to say about
memory. I will be operating on some big arrays, probably with Numpy;
if I run out of space and upgrade a Win 2000 or Win XP pro machine
from 256 Meg to 500Meg or even 1G will Python automatically recognize
and take advantage of the increase? Where are questionss such as this
discussed in the documentation?

[Scott David Daniels]

Ben Fairbank wrote:

I have to pick a language to commit to for general purpose
scientific/statistical/utility/database programming at my office and
have pretty much narrowed it down to R or Python.

In a previous life, I went for R and Python and eventually
stayed in python (with numeric, of course) because, in my case,
moving data over to R and then operating there was always more
expensive than simply doing the calculations in python. These
days I'd check if numarray did the trick. If you want to play
in this world, check out the enthought packaging of python:

http://www.enthought.com/python/

They've got most of the goodies in there that you'd be likely
to want to run out and add to your python in the first months.

-Scott David Daniels
Sc***********@Acm.Org

[Cameron Laird]

In article <vn************@corp.supernews.com>, I offered:

In article <3f***************@news.houston.sbcglobal.net>,
Ben Fairbank <ba*@texas.antispam.net> wrote:

I have to pick a language to commit to for general purpose
scientific/statistical/utility/database programming at my office and
have pretty much narrowed it down to R or Python. Problem: none of

.
.
.
Tough choice. As your research apparently has already
disclosed, both R and Python are quite capable in the
role you're contemplating. The differences between
them are likely to be the deep sort that are difficult
to determine beforehand. Are you aware that, at least
within limits, you don't *have* to choose? <URL:
http://www.togaware.com/linux/survivor/Python_R.html >?

I apparently misled some readers. I want to emphasize: both
R and Python are wonderful for the sort of scientific program-
ming under consideration here. Anyone who has narrowed down
his choices to these two has already done all the hard work.
It's reasonably safe to choose either, without regrets; there
is no need in general to learn both.

<URL: http://phaseit.net/claird/comp.progr...e_science.html >
provides reading of related interest.
--

Cameron Laird <Ca*****@Lairds.com>
Business: http://www.Phaseit.net
Personal: http://phaseit.net/claird/home.html

[Fernando Perez]

Ben Fairbank wrote:

I have to pick a language to commit to for general purpose
scientific/statistical/utility/database programming at my office and
have pretty much narrowed it down to R or Python. Problem: none of
the various Python books I have looked into has had much to say about
memory. I will be operating on some big arrays, probably with Numpy;
if I run out of space and upgrade a Win 2000 or Win XP pro machine
from 256 Meg to 500Meg or even 1G will Python automatically recognize
and take advantage of the increase? Where are questionss such as this
discussed in the documentation?

Thanks for any suggestions,

For large arrays, look into numarray (next-gen Numeric). That's where most of
the effort of numarray has gone: being a _lot_ smarter than Numeric in memory
management for large arrays. It's also cleaner and nicer than numeric in many
ways, and already mature enough that many people are using it for production
work. Its only big stumbling block right now is that small array performance
is pretty poor, and this one will take a bit of time to fix.

I'd personally stick to python (with the scipy libraries, as was already
suggested), and would add the RPy interface for those cases when you need a
routine which R provides, which you need NOW, and which would be more work to
write in python.

Cheers,

f.