473,748 Members | 2,471 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Python and Memory

I have to pick a language to commit to for general purpose
scientific/statistical/utility/database programming at my office and
have pretty much narrowed it down to R or Python. Problem: none of
the various Python books I have looked into has had much to say about
memory. I will be operating on some big arrays, probably with Numpy;
if I run out of space and upgrade a Win 2000 or Win XP pro machine
from 256 Meg to 500Meg or even 1G will Python automatically recognize
and take advantage of the increase? Where are questionss such as this
discussed in the documentation?

Thanks for any suggestions,

Ben Fairbank
Jul 18 '05 #1
7 5845
On Thu, Oct 02, 2003 at 04:59:15AM +0000, Ben Fairbank wrote:
I have to pick a language to commit to for general purpose
scientific/statistical/utility/database programming at my office and
have pretty much narrowed it down to R or Python. Problem: none of
the various Python books I have looked into has had much to say about
memory. I will be operating on some big arrays, probably with Numpy;
Python isn't terribly memory-efficient. Every int, for example, takes all
the overhead of an object plus any allocator overheads. But if the bulk
of the data is going to be numpy arrays this isn't a problem. An array
object has a small one-time overhead and then each item takes exactly its
native size in bytes. Make sure you use in-place operations as much as
possible on arrays so you don't force numpy to allocate large temporary
arrays.
if I run out of space and upgrade a Win 2000 or Win XP pro machine
from 256 Meg to 500Meg or even 1G will Python automatically recognize
and take advantage of the increase? Where are questionss such as this
discussed in the documentation?


Python can immediately use any memory expansion. It also runs on 64 bit
platforms if you ever need more than 4GB of data.

Oren

Jul 18 '05 #2
ba*@texas.antis pam.net (Ben Fairbank) writes:
I have to pick a language to commit to for general purpose
scientific/statistical/utility/database programming at my office and
have pretty much narrowed it down to R or Python. Problem: none of
the various Python books I have looked into has had much to say about
memory. I will be operating on some big arrays, probably with Numpy;
Hmm. Python is not *amazingly* memory efficient. But if you're
dealing with relatively few Numpy arrays, the memory consumption will
probably not be that much more than any other language you use
(unlike, say, using a list of floats).
if I run out of space and upgrade a Win 2000 or Win XP pro machine
from 256 Meg to 500Meg or even 1G will Python automatically recognize
and take advantage of the increase?
This is more a question about the OS than Python. I would be amazed
if Python couldn't use the extra memory.
Where are questionss such as this discussed in the documentation?


I don't think they are, really. The numpy manual might have a few
words to say on the subject.

Cheers,
mwh

--
Famous remarks are very seldom quoted correctly.
-- Simeon Strunsky
Jul 18 '05 #3
Ben Fairbank wrote:
I have to pick a language to commit to for general purpose
scientific/statistical/utility/database programming at my office and
have pretty much narrowed it down to R or Python. Problem: none of
the various Python books I have looked into has had much to say about
memory. I will be operating on some big arrays, probably with Numpy;
if I run out of space and upgrade a Win 2000 or Win XP pro machine
from 256 Meg to 500Meg or even 1G will Python automatically recognize
and take advantage of the increase? Where are questionss such as this
discussed in the documentation?


Python gets its information about memory (and allocates its memory)
from the operating system. This is not discussed in the documentation
because it's more or less taken for granted -- that's what one should
assume of any app or language unless other behavior is documented. If
your Win/2K, Win/XP, or whatever, is able to see your new memory and
make it available to apps (and I see no reason why not), Python, like
any other normal app, should be able to use it just fine.
Alex

Jul 18 '05 #4
In article <3f************ ***@news.housto n.sbcglobal.net >,
Ben Fairbank <ba*@texas.anti spam.net> wrote:
I have to pick a language to commit to for general purpose
scientific/statistical/utility/database programming at my office and
have pretty much narrowed it down to R or Python. Problem: none of
the various Python books I have looked into has had much to say about
memory. I will be operating on some big arrays, probably with Numpy;
if I run out of space and upgrade a Win 2000 or Win XP pro machine
from 256 Meg to 500Meg or even 1G will Python automatically recognize
and take advantage of the increase? Where are questionss such as this
discussed in the documentation?

Jul 18 '05 #5
Ben Fairbank wrote:
I have to pick a language to commit to for general purpose
scientific/statistical/utility/database programming at my office and
have pretty much narrowed it down to R or Python.

In a previous life, I went for R and Python and eventually
stayed in python (with numeric, of course) because, in my case,
moving data over to R and then operating there was always more
expensive than simply doing the calculations in python. These
days I'd check if numarray did the trick. If you want to play
in this world, check out the enthought packaging of python:

http://www.enthought.com/python/

They've got most of the goodies in there that you'd be likely
to want to run out and add to your python in the first months.

-Scott David Daniels
Sc***********@A cm.Org

Jul 18 '05 #6
In article <vn************ @corp.supernews .com>, I offered:
In article <3f************ ***@news.housto n.sbcglobal.net >,
Ben Fairbank <ba*@texas.anti spam.net> wrote:
I have to pick a language to commit to for general purpose
scientific/statistical/utility/database programming at my office and
have pretty much narrowed it down to R or Python. Problem: none of

.
.
.
Tough choice. As your research apparently has already
disclosed, both R and Python are quite capable in the
role you're contemplating. The differences between
them are likely to be the deep sort that are difficult
to determine beforehand. Are you aware that, at least
within limits, you don't *have* to choose? <URL:
http://www.togaware.com/linux/survivor/Python_R.html >?


I apparently misled some readers. I want to emphasize: both
R and Python are wonderful for the sort of scientific program-
ming under consideration here. Anyone who has narrowed down
his choices to these two has already done all the hard work.
It's reasonably safe to choose either, without regrets; there
is no need in general to learn both.

<URL: http://phaseit.net/claird/comp.progr...e_science.html >
provides reading of related interest.
--

Cameron Laird <Ca*****@Lairds .com>
Business: http://www.Phaseit.net
Personal: http://phaseit.net/claird/home.html
Jul 18 '05 #7
Ben Fairbank wrote:
I have to pick a language to commit to for general purpose
scientific/statistical/utility/database programming at my office and
have pretty much narrowed it down to R or Python. Problem: none of
the various Python books I have looked into has had much to say about
memory. I will be operating on some big arrays, probably with Numpy;
if I run out of space and upgrade a Win 2000 or Win XP pro machine
from 256 Meg to 500Meg or even 1G will Python automatically recognize
and take advantage of the increase? Where are questionss such as this
discussed in the documentation?

Thanks for any suggestions,


For large arrays, look into numarray (next-gen Numeric). That's where most of
the effort of numarray has gone: being a _lot_ smarter than Numeric in memory
management for large arrays. It's also cleaner and nicer than numeric in many
ways, and already mature enough that many people are using it for production
work. Its only big stumbling block right now is that small array performance
is pretty poor, and this one will take a bit of time to fix.

I'd personally stick to python (with the scipy libraries, as was already
suggested), and would add the RPy interface for those cases when you need a
routine which R provides, which you need NOW, and which would be more work to
write in python.

Cheers,

f.
Jul 18 '05 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
3046
by: lebo | last post by:
Hi I'm trying to understand how Python handles memory usage and dynamic object loading and unloading. Problem to solve? Build a very low memory footprint (non-GUI) Python application for Unix/Windows. Why use Python? End user flexibility. So far it seems that (on Windows): (1) memory increases when you import <module>. Expected behaviour. (2) memory does not release when you del <module>. Why not? Is Python
3
2602
by: Saravanan | last post by:
Hello, Im running Python Application as a Windows Service (using windows extensions). But, sporadically the application crashes (crash in Python23.dll) and this stops the service. This problem cann't be reproduced easily in my system and the call stack generated by the application is given below. Please note that the call stack generation is taken from crash dump file.
0
1597
by: Robby Dermody | last post by:
Hey guys (thus begins a book of a post :), I'm in the process of writing a commercial VoIP call monitoring and recording application suite in python and pyrex. Basically, this software sits in a VoIP callcenter-type environment (complete with agent phones and VoIP servers), sniffs voice data off of the network, and allows users to listen into calls. It can record calls as well. The project is about a year and 3 months in the making and...
25
5767
by: abhinav | last post by:
Hello guys, I am a novice in python.I have to implement a full fledged mail server ..But i am not able to choose the language.Should i go for C(socket API) or python for this project? What are the advantages of one over the other in implementing this server.which language will be easier? What are the performance issues?In what language are mail servers generally written?
20
9358
by: mariano.difelice | last post by:
Hi, I've a big memory problem with my application. First, an example: If I write: a = range(500*1024) I see that python process allocate approximately 80Mb of memory.
17
8481
by: frederic.pica | last post by:
Greets, I've some troubles getting my memory freed by python, how can I force it to release the memory ? I've tried del and gc.collect() with no success. Here is a code sample, parsing an XML file under linux python 2.4 (same problem with windows 2.5, tried with the first example) : #Python interpreter memory usage : 1.1 Mb private, 1.4 Mb shared #Using http://www.pixelbeat.org/scripts/ps_mem.py to get memory information
5
3345
by: vishnu | last post by:
Hi there, I am embedding python 2.5 on embedded system running on RTOS where I had strict memory constraints. As python is a huge malloc intensive application, I observed huge memory fragmentation in my system which is leading to out of memory after running few scripts. So I decided to re-initialise the python with out restarting the whole python. I tried to use Py_Finalise() after completion of each script , then call Py_Initialise as...
206
8357
by: WaterWalk | last post by:
I've just read an article "Building Robust System" by Gerald Jay Sussman. The article is here: http://swiss.csail.mit.edu/classes/symbolic/spring07/readings/robust-systems.pdf In it there is a footprint which says: "Indeed, one often hears arguments against building exibility into an engineered sys- tem. For example, in the philosophy of the computer language Python it is claimed: \There should be one|and preferably only one|obvious...
0
8984
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
8823
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9530
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9363
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
9238
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
8237
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6793
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
4593
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4864
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.