473,320 Members | 1,863 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Memory limit to dict?

I was wondering whether certain data structures in Python, e.g. dict,
might have limits as to the amount of memory they're allowed to take up.
Is there any documentation on that?

Why am I asking? I'm reading 3.6 GB worth of BLAST output files into a
nested dictionary structure (dict within dict ...). Looks something like
this:

{ GenomeID:
{ ProteinID:
{ GenomeID:
{ ProteinID, Score, PercentValue, EValue } } } }

Now, the thing is: Even on a machine with 16 GB RAM, the program
terminates with a MemoryError, obviously long before the machine's RAM
is used up.

I've no idea how far the Windows task manager's resource monitor can be
trusted -- probably not as far as I could throw a heavy-set St Bernard
--, but it seems to stop roughly when that monitor records a swap file
size of 2.2 GB.

Barring any revamping of the code itself, which I will have to do
anyway, is there anything so far that would indicate a problem inherent
to Python?

(I can post the relevant code too, of course, if that would help.)

TIA!

--
Peter
Apr 11 '06 #1
3 3770
Peter Beattie <Pe***********@web.de> writes:

<snip>
I've no idea how far the Windows task manager's resource monitor can be
trusted -- probably not as far as I could throw a heavy-set St Bernard
--, but it seems to stop roughly when that monitor records a swap file
size of 2.2 GB.


Not being a windows expert at all, but I would assume with 32 bit
windows each process in the system can have an address space of ~2
gigs. In linux the process address space is split in half, bottom 2
gigs for OS mappings, top for the process, so it looks like you might
just be hitting the maximum allowed address space mapping.

You should partition your data into hierarchial modules and let python
do the swapping for you...although you have 16 gigs (I have to put a
holy crap after that!) you will always run into process limits, at
least until true 64 bit os's are in vouge.

--
burton samograd kruhft .at. gmail
kruhft.blogspot.com www.myspace.com/kruhft metashell.blogspot.com
Apr 11 '06 #2
Em Ter, 2006-04-11 Ã*s 19:45 +0200, Peter Beattie escreveu:
I was wondering whether certain data structures in Python, e.g. dict,
might have limits as to the amount of memory they're allowed to take up.
Is there any documentation on that?

Why am I asking? I'm reading 3.6 GB worth of BLAST output files into a
nested dictionary structure (dict within dict ...). Looks something like
this:

{ GenomeID:
{ ProteinID:
{ GenomeID:
{ ProteinID, Score, PercentValue, EValue } } } }


I don't have the answer to your question and I'll make a new one: isn't
the overhead (performance and memory) of creating dicts too large to be
used in this scale?

I'm just speculating, but I *think* that using lists and objects may be
better.

My 2 cents,

--
Felipe.

Apr 11 '06 #3
An alternative is to use ZODB. For example, you could use the BTree
class for the outermost layers of the nested dict, and a regular dict
for the innermost layer. If broken up properly, you can store
apparently unlimited amount of data with reasonable performance.

Just remember not to iterate over the entire collection of objects
without aborting the transaction regularly.

Apr 11 '06 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
by: Benjamin Scott | last post by:
Hello. I attempted to build a compound dictionary: len(Lst)=1000 len(nuerLst)=250 len(nuestLst)=500 Dict={}
4
by: Thomas Rast | last post by:
Hello everyone My scenario is somewhat involved, so if you're in a hurry, here's an abstract: I have a daemon that needs about 80MB of RAM to build its internal data structures, but can pack...
9
by: Chris S. | last post by:
Is it possible to determine how much memory is allocated by an arbitrary Python object? There doesn't seem to be anything in the docs about this, but considering that Python manages memory...
5
by: Ian Bicking | last post by:
I got a puzzler for y'all. I want to allow the editing of functions in-place. I won't go into the reason (it's for HTConsole -- http://blog.ianbicking.org/introducing-htconsole.html), except that...
3
by: sadanjan | last post by:
Hi , Appreciate if someone can clarify if database Share Memory Limit (2 GB ) in Unix 32 bit boxes is the top limit for all the databases put together in a database or is it for each of the...
17
by: frederic.pica | last post by:
Greets, I've some troubles getting my memory freed by python, how can I force it to release the memory ? I've tried del and gc.collect() with no success. Here is a code sample, parsing an XML...
2
by: Jonas Maurus | last post by:
Hello everybody, I'm pondering the following problem: I want to write a Python program that receives messages via SMTP and stores them in a dict or an array. For my purposes it would be...
19
by: jsanshef | last post by:
Hi, after a couple of days of script debugging, I kind of found that some assumptions I was doing about the memory complexity of my classes are not true. I decided to do a simple script to...
0
by: rkmr.em | last post by:
the memory usage of a python app keeps growing in a x86 64 linux continuously, whereas in 32 bit linux this is not the case. Python version in both 32 bit and 64 bit linux - 2.6.24.4-64.fc8 Python...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.