473,378 Members | 1,592 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,378 software developers and data experts.

import data.py using massive amounts of memory

I've been dumping a database in a python code format (for use with
Python on S60 mobile phone actually) and I've noticed that it uses
absolutely tons of memory as compared to how much the data structure
actually needs once it is loaded in memory.

The programs below create a file (z.py) with a data structure in which
looks like this

-- z.py ----------------------------------------------------
z = {
0 : (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19),
1 : (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20),
2 : (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21),
[snip]
998 : (998, 999, 1000, 1001, 1002, ..., 1012, 1013, 1014, 1015, 1016, 1017),
999 : (999, 1000, 1001, 1002, 1003, ..., 1013, 1014, 1015, 1016, 1017, 1018),
}
------------------------------------------------------------

Under python2.2-python2.4 "import z" uses 8 MB, whereas loading a
pickled dump of the file only takes 450kB. This has been improved in
python2.5 so it only takes 2.2 MB.

$ python2.5 memory_usage.py
Memory used to import is 2284 kB
Total size of repr(z) is 105215
Memory used to unpickle is 424 kB
Total size of repr(z) is 105215

$ python2.4 memory_usage.py
Memory used to import is 8360 kB
Total size of repr(z) is 105215
Memory used to unpickle is 456 kB
Total size of repr(z) is 105215

$ python2.3 memory_usage.py
Memory used to import is 8436 kB
Total size of repr(z) is 105215
Memory used to unpickle is 456 kB
Total size of repr(z) is 105215

$ python2.2 memory_usage.py
Memory used to import is 8568 kB
Total size of repr(z) is 105215
Memory used to unpickle is 392 kB
Total size of repr(z) is 105215

$ python2.1 memory_usage.py
Memory used to import is 10756 kB
Total size of repr(z) is 105215
Memory used to unpickle is 384 kB
Total size of repr(z) is 105215

Why does it take so much memory? Is it some consequence of the way
the datastructure is parsed?

Note that once it has made the .pyc file the subsequent runs take even
less memory than the cpickle import.

S60 python is version 2.2.1. It doesn't have pickle unfortunately, but
it does have marshal and the datastructures I need are marshal-able so
that provides a good solution to my actual problem.

Save the two programs below with the names given to demonstrate the
problem. Note that these use some linux-isms to measure the memory
used by the current process which will need to be adapted if you don't
run it on linux!

-- memory_usage.py -----------------------------------------

import os
import sys
import re
from cPickle import dump

def memory():
"""Returns memory used (RSS) in kB"""
status = open("/proc/self/status").read()
match = re.search(r"(?m)^VmRSS:\s+(\d+)", status)
memory = 0
if match:
memory = int(match.group(1))
return memory

def write_file():
"""Write the file to be imported"""
fd = open("z.py", "w")
fd.write("z = {\n")
for i in xrange(1000):
fd.write(" %d : %r,\n" % (i, tuple(range(i,i+20))))
fd.write("}\n")
fd.close()

def main():
write_file()
before = memory()
from z import z
after = memory()
print "Memory used to import is %s kB" % (after-before)
print "Total size of repr(z) is ",len(repr(z))

# Save a pickled copy for later
dump(z, open("z.bin", "wb"))

# Run the next part
os.system("%s memory_usage1.py" % sys.executable)

if __name__ == "__main__":
main()

-- memory_usage1.py ----------------------------------------

from memory_usage import memory
from cPickle import load

before = memory()
z = load(open("z.bin", "rb"))
after = memory()
print "Memory used to unpickle is %s kB" % (after-before)
print "Total size of repr(z) is ",len(repr(z))

------------------------------------------------------------

--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick
Jun 27 '07 #1
1 2989
Note that once it has made the .pyc file the subsequent runs take even
less memory than the cpickle import.
Could that be the compiler compiling?

Without knowing to much details about that process, but from 2.4 to
2.5 the compiler was totally exchanged, <whateverto AST.
That would explain the drop from 8Meg -2.2 Meg.

Harald

Jun 27 '07 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
by: Unforgiven | last post by:
I have an application, where I continuously get new binary data input, in the form of a char*. This data comes from the Windows Multimedia wave input functions, but that's not important. What it...
5
by: Framework fan | last post by:
Hello, If I wrote the next ebay (yes I know, yawn-snore) and I had a database with 5 million auction items in it, what would be a really good strategy to get a search done very quickly? Would...
0
by: Doug R | last post by:
Hello, I have a system that I am writing to automaticly import Credit Transaction data into a SQL Server 2000 Database. I am using a VB.Net application to detect when the file arives and prep...
3
by: Wayne Marsh | last post by:
Hi all. I am working on an audio application which needs reasonably fast access to large amounts of data. For example, the program may load a 120 second stereo sound sample stored at 4bytes per...
5
by: Scott Reynolds | last post by:
Hello! I developed a web application to display results from the database. Now I need to add search function, to search, sort and filter data. My question is, which way is better... 1)...
7
by: =?Utf-8?B?TW9iaWxlTWFu?= | last post by:
Hello everyone: I am looking for everyone's thoughts on moving large amounts (actually, not very large, but large enough that I'm throwing exceptions using the default configurations). We're...
19
by: Zytan | last post by:
I want multiple instances of the same .exe to run and share the same data. I know they all can access the same file at the same time, no problem, but I'd like to have this data in RAM, which they...
0
by: volt9000 | last post by:
I'm using PdfSharp (an open-source PDF manipulation library) to generate a very large PDF ( 1500+ pages.) My program crashes before reaching the end because of the massive amounts of memory being...
0
by: Ben Lee | last post by:
hi folks -- a quick python and sqlite3 performance question. i find that inserting a million rows of in-memory data into an in-memory database via a single executemany() is about 30% slower...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...
0
by: ryjfgjl | last post by:
In our work, we often need to import Excel data into databases (such as MySQL, SQL Server, Oracle) for data analysis and processing. Usually, we use database tools like Navicat or the Excel import...
0
by: taylorcarr | last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.