Memory error due to the huge/huge input file size

[tejsupra]

Hello Everyone,

I need to read a .csv file which has a size of 2.26 GB . And I wrote a
Python script , where I need to read this file. And my Computer has 2
GB RAM Please see the code as follows:

"""
This program has been developed to retrieve all the promoter sequences
for the specified
list of genes in the given cluster

So, this program will act as a substitute to the whole EZRetrieve
system

Input arguments:

1) Cluster.txt or DowRatClust1617 18bwithDummy.tx t
2) TransProCrossRe ferenceAndSeque nces.csv -This is the file that has
all the promoter sequences
3) -2000
4) 500
"""

import time
import csv
import sys
import linecache
import re
from sets import Set
import gc

print time.localtime( )

fileInputHandle r = open(sys.argv[1],"r")
line = fileInputHandle r.readline()

refSeqIDsinTran sPro = []
promoterSequenc esinTransPro = []
reader2 = csv.reader(open (sys.argv[2],"rb"))
reader2_list = []
reader2_list.ex tend(reader2)

for data2 in reader2_list:
refSeqIDsinTran sPro.append(dat a2[3])
for data2 in reader2_list:
promoterSequenc esinTransPro.ap pend(data2[4])

while line:
l = line.rstrip('\n ')
for j in range(1,len(ref SeqIDsinTransPr o)):
found = re.search(l,ref SeqIDsinTransPr o[j])
if found:
"""promoterSequ encesinTransPro[j] """
print l

line = fileInputHandle r.readline()
fileInputHandle r.close()
The error that I got is given as follows:
Traceback (most recent call last):
File "RefSeqsToPromo terSequences.py ", line 31, in <module>
reader2_list.ex tend(reader2)
MemoryError

I understand that the issue is Memory error and it is caused because
of the line reader2_list.ex tend(reader2). Is there any other
alternative method in reading the .csv file line by line?

sincerely,
Suprabhath

[James Mills]

On Tue, Nov 11, 2008 at 7:47 AM, <te******@gmail .comwrote:

refSeqIDsinTran sPro = []
promoterSequenc esinTransPro = []
reader2 = csv.reader(open (sys.argv[2],"rb"))
reader2_list = []
reader2_list.ex tend(reader2)

Without testing, this looks like you're reading the _ENTIRE_
input stream into memory! Try this:

def readCSV(file):

if type(file) == str:
fd = open(file, "rU")
else:
fd = file

sniffer = csv.Sniffer()
dialect = sniffer.sniff(f d.readline())
fd.seek(0)

reader = csv.reader(fd, dialect)
for line in reader:
yield line

for line in readCSV(open("f oo.csv", "r")):
...

--JamesMills

--
--
-- "Problems are solved by method"

[John Machin]

On Nov 11, 8:47*am, tejsu...@gmail. com wrote:

import linecache

Why???

reader2 = csv.reader(open (sys.argv[2],"rb"))
reader2_list = []
reader2_list.ex tend(reader2)

for data2 in reader2_list:
* *refSeqIDsinTra nsPro.append(da ta2[3])
for data2 in reader2_list:
* *promoterSequen cesinTransPro.a ppend(data2[4])

All you need to do is replace the above by:

reader2 = csv.reader(open (sys.argv[2],"rb"))

for data2 in reader2:
refSeqIDsinTran sPro.append(dat a2[3])
promoterSequenc esinTransPro.ap pend(data2[4])

[tejsupra]

On Nov 10, 4:47*pm, tejsu...@gmail. com wrote:

Hello Everyone,

I need to read a .csv file which has a size of 2.26 GB . And I wrote a
Python script , where I need to read this file. And my Computer has 2
GB RAM Please see the code as follows:

"""
This program has been developed to retrieve all the promoter sequences
for the specified
list of genes in the given cluster

So, this program will act as a substitute to the whole EZRetrieve
system

Input arguments:

1) Cluster.txt or DowRatClust1617 18bwithDummy.tx t
2) TransProCrossRe ferenceAndSeque nces.csv -This is the file that has
all the promoter sequences
3) -2000
4) 500
"""

import time
import csv
import sys
import linecache
import re
from sets import Set
import gc

print time.localtime( )

fileInputHandle r = open(sys.argv[1],"r")
line = fileInputHandle r.readline()

refSeqIDsinTran sPro = []
promoterSequenc esinTransPro = []
reader2 = csv.reader(open (sys.argv[2],"rb"))
reader2_list = []
reader2_list.ex tend(reader2)

for data2 in reader2_list:
* *refSeqIDsinTra nsPro.append(da ta2[3])
for data2 in reader2_list:
* *promoterSequen cesinTransPro.a ppend(data2[4])

while line:
* *l = line.rstrip('\n ')
* *for j in range(1,len(ref SeqIDsinTransPr o)):
* * * found = re.search(l,ref SeqIDsinTransPr o[j])
* * * if found:
* * * * *"""promoterSeq uencesinTransPr o[j] *"""
* * * * *print l

* *line = fileInputHandle r.readline()

fileInputHandle r.close()

The error that I got is given as follows:
Traceback (most recent call last):
* File "RefSeqsToPromo terSequences.py ", line 31, in <module>
* * reader2_list.ex tend(reader2)
MemoryError

I understand that the issue is Memory error and it is caused because
of the *line reader2_list.ex tend(reader2). Is there any other
alternative method in reading the .csv file *line by line?

sincerely,
Suprabhath

Thanks a Lot James Mills. It worked