Memory leak - malloc/free implementation - GC kicking in late - know bug
- or ?
Using python-2.2.2-26 on RH9 (shrike) x86 -fully patched
The following program slowly eats up more and more memory when run on
large datasets... can anyone tell what the trouble is?
i've run it up to 240000 recsets so far - and it eats about .1% of my
mem pr. 1000 (doesn't really matter how much does it?).
--
Med Venlig Hilsen / Regards
Kim Petersen - Kyborg A/S (Udvikling)
IT - Innovationshuset
Havneparken 2
7100 Vejle
Tlf. +4576408183 || Fax. +4576408188
#!/usr/bin/python
#
# Created: 13:32 10/07-2003 by Kim Petersen <kp@kyborg.dk>
#
# $Id$
from __future__ import generators
import gzip
import re
err1=re.compile("^'ERROR:\s+(.*?)' in '(.*)'\s*$")
def iterator(file):
buffer=[]
while 1:
if not buffer:
buffer=file.readlines(1000)
line=buffer[0]
del buffer[0]
if not line:
raise
yield line
def getrec(lines):
result=[]
while 1:
line=lines.next().rstrip()
if not line: break
result.append(line)
if not result: return None
(error,dataset)=(result[:-1],eval(result[-1]))
error=''.join(error)[16:]
return error,dataset
if __name__ == "__main__":
import sys
lines=iterator(gzip.open("error.txt.gz"))
i=0
while 1:
if (i%1000)==0:
sys.stdout.write("%-10.10d\r" % (i,))
sys.stdout.flush()
rec=getrec(lines)
if not rec: break
(errline,dataset)=rec
if not err1.match(errline):
sys.stdout.write("%s\n" % (errline,))
sys.stdout.write("%-10.10d\r" % (i,))
sys.stdout.flush()
i+=1
sys.stdout.write("%-10.10d\n" % (i,))
sys.stdout.flush()
# Local Variables:
# tab-width: 3
# py-indent-offset: 3
# End: