def windows():
import os
excludes = ['hiberfil.sys', 'ipnathlp.dll', 'helpctr.exe', 'etc',
'etc', 'etc']
size_list = []
for root, dirs, files in os.walk('/'):
total = [x for x in files if x not in excludes]
for t in total:
s = file(os.path.join(root,t))
size = s.read()
size_list.append(size)
s.close()
windows()
The above function crashes with a memory error on Windows XP Pro at the
'size = s.read()' line. Page File usage (normally ~120 MB) will rise to
300+ MB and pythonw.exe will consume about 200 MB of actual ram right
before the crash. The machine has 512 MB of ram and is doing nothing
else while running the script.
I've written the script several ways, all with the same result. I've
noticed that a binary read 'rb' consumes almost twice as much physical
memory and causes the crash to happen quicker, but that's about it.
Initially, I wanted to use Python to open every file on the system (that
could be opened) and read the contents so I'd know the size of the file
and then add all of the reads to a list that I'd sum up. Basically
attempt to add up all bytes on the machine's disk drive.
Any ideas on what I'm doing wrong or suggestions on how to do this
differently?
Thanks,
Bart 3 3550
Bart Nessux wrote: def windows(): import os excludes = ['hiberfil.sys', 'ipnathlp.dll', 'helpctr.exe', 'etc', 'etc', 'etc'] size_list = [] for root, dirs, files in os.walk('/'): total = [x for x in files if x not in excludes] for t in total: s = file(os.path.join(root,t)) size = s.read() size_list.append(size) s.close()
windows()
The above function crashes with a memory error on Windows XP Pro at the 'size = s.read()' line. Page File usage (normally ~120 MB) will rise to 300+ MB and pythonw.exe will consume about 200 MB of actual ram right before the crash. The machine has 512 MB of ram and is doing nothing else while running the script.
I've written the script several ways, all with the same result. I've noticed that a binary read 'rb' consumes almost twice as much physical memory and causes the crash to happen quicker, but that's about it.
Initially, I wanted to use Python to open every file on the system (that could be opened) and read the contents so I'd know the size of the file and then add all of the reads to a list that I'd sum up. Basically attempt to add up all bytes on the machine's disk drive.
Any ideas on what I'm doing wrong or suggestions on how to do this differently?
Your building an array containing the *contents* of all your files.
If you really need to use read(), use "size = len(s.read())", but this
still requires to read and hold a complete file at a time in memory (and
probably chokes when it stumbles over your divx collection ;)
I think using os.stat() should be better...
Am Mittwoch, 2. Juni 2004 15:11 schrieb Bart Nessux: size = s.read()
You read the complete content of the file here. size will not contain the
length of the file, but the complete file data. What you want is either
len(s.read()) (which is sloooooooooow), or have a look at os.path.getsize().
size_list.append(size)
This appends the complete file to the list. And as such should explain the
memory usage you're seeing...
HTH!
Heiko.
On Wed, 02 Jun 2004 09:11:13 -0400, Bart Nessux
<ba*********@hotmail.com> wrote: def windows(): import os excludes = ['hiberfil.sys', 'ipnathlp.dll', 'helpctr.exe', 'etc', 'etc', 'etc'] size_list = [] for root, dirs, files in os.walk('/'): total = [x for x in files if x not in excludes] for t in total: s = file(os.path.join(root,t)) size = s.read() size_list.append(size) s.close()
windows()
Yeah, what the other guys said about os.stat and os.path.getsize.
Also, if you really want to read the actual file into memory, just get
small chunks and add those up.
Like (untested):
numberofbytes = 0
CHUNKSIZE = 4096
for root,dirs, files in os.walk('/'):
for name in files:
if name not in excludes:
f = file(os.path.join(root,name))
while 1:
s = f.read(CHUNKSIZE)
if not s:
f.close()
break
numberofbytes += len(s)
this way you never have more than 4k of data in memory at once.
(well it might be 8k, I dont know enought about the internals to tell
you when the previous 's' is garbage collected.)
<{{{*> This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Srijit Kumar Bhadra |
last post by:
Hello,
Here is some sample code with pywin32 build 203 and ctypes 0.9.6.
Best regards,
/Srijit
File: SharedMemCreate_Mutex_win32all.py
# This application should be used with...
|
by: Attila.Rajmund.Nohl |
last post by:
Hello!
I'm using KAI C++ Compiler (KCC) Version 4.0d on Sparc Solaris 8 with
Sun WorkShop 6 update 2 backend (KCC compiles C++ code to C than
uses the Sun compiler to produce machine...
|
by: Andrzej |
last post by:
Used to read newsgroup for answers, now have to ask for them as well.
I have an application (C#, .NET 1.1) that connects to local db on MSDE
2000 SP3a (using ADO from MDAC 2.71) on one side and...
|
by: Zeng |
last post by:
I finally narrowed down my code to this situation, quite a few (not all) of
my CMyClass objects got hold up after each run of this function via the
simple webpage that shows NumberEd editbox. My...
|
by: Sean Shanny |
last post by:
To all,
Running into an out of memory error on our data warehouse server. This
occurs only with our data from the 'September' section of a large fact
table. The exact same query running over...
|
by: saran |
last post by:
I am having a problem with MySQL consuming a lot of memory and
eventually throwing an Out of Memory error and restarting itself. The
symptoms are that swap usage continues to rise until some...
|
by: tgnelson85 |
last post by:
Hello, C question here (running on Linux, though there should be no platform specific code).
After reading through a few examples, and following one in a book, for linked lists i thought i would...
|
by: kumarmdb2 |
last post by:
Hi guys,
For last few days we are getting out of private memory error. We have a
development environment. We tried to figure out the problem but we
believe that it might be related to the OS...
|
by: George2 |
last post by:
Hello everyone,
Should I delete memory pointed by pointer a if there is bad_alloc when
allocating memory in memory pointed by pointer b? I am not sure
whether there will be memory leak if I do...
|
by: ravishi |
last post by:
Well, this is my first topic at this forum and I'm a newbie on C programming too.
I'm coding a little program and I've used some "dynamic arrays" on it. Compiling and running the program on Linux...
|
by: Charles Arthur |
last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
|
by: ryjfgjl |
last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
|
by: ryjfgjl |
last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: nemocccc |
last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
|
by: Sonnysonu |
last post by:
This is the data of csv file
1 2 3
1 2 3
1 2 3
1 2 3
2 3
2 3
3
the lengths should be different i have to store the data by column-wise with in the specific length.
suppose the i have to...
|
by: Hystou |
last post by:
There are some requirements for setting up RAID:
1. The motherboard and BIOS support RAID configuration.
2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers,...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
| |