By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
440,235 Members | 1,057 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 440,235 IT Pros & Developers. It's quick & easy.

file reading by record separator (not line by line)

P: n/a
Dear all,
I would like to read a really huge file that looks like this:
name1....
line_11
line_12
line_13
....
>name2 ...
line_21
line_22
....
etc

where line_ij is just a free form text on that line.

how can i read file so that every time i do a "read()" i get exactly
one record
up to the next ">"

many thanks
Lee

May 31 '07 #1
Share this Question
Share on Google+
6 Replies


P: n/a
I wanted to also say that this file is really huge, so I cannot
just do a read() and then split on ">" to get a record
thanks
lee

On May 31, 1:26 pm, Lee Sander <lesa...@gmail.comwrote:
Dear all,
I would like toreada really hugefilethat looks like this:
name1....

line_11
line_12
line_13
...>name2 ...

line_21
line_22
...
etc

where line_ij is just a free form text on that line.

how can ireadfileso that every time i do a "read()" i get exactly
onerecord
up to the next ">"

many thanks
Lee

May 31 '07 #2

P: n/a

something like

name=None
lines=[]
for line in open('yourfilename.txt'):
if line.startwith('>'):
if name!=None:
print 'Here is the record', name
print lines
print
name=line.stripr('\r')
lines=[]
else:
lines.append(line.stripr('\n'))

On 31 mai, 14:39, Lee Sander <lesa...@gmail.comwrote:
I wanted to also say that this file is really huge, so I cannot
just do a read() and then split on ">" to get a record
thanks
lee

On May 31, 1:26 pm, Lee Sander <lesa...@gmail.comwrote:
Dear all,
I would like toreada really hugefilethat looks like this:
name1....
line_11
line_12
line_13
...>name2 ...
line_21
line_22
...
etc
where line_ij is just a free form text on that line.
how can ireadfileso that every time i do a "read()" i get exactly
onerecord
up to the next ">"
many thanks
Lee

May 31 '07 #3

P: n/a
Lee Sander wrote:
I wanted to also say that this file is really huge, so I cannot
just do a read() and then split on ">" to get a record
thanks
lee
Below is the easy solution. To get even better performance, or if '<' is not
always at the start of the line, you would have to implement the buffering
that is done by readline() yourself (see _fileobject in socket.py in the
standard lib for example).

def chunkreader(f):
name = None
lines = []
while True:
line = f.readline()
if not line: break
if line[0] == '>':
if name is not None:
yield name, lines
name = line[1:].rstrip()
lines = []
else:
lines.append(line)
if name is not None:
yield name, lines

if __name__ == '__main__':
from StringIO import StringIO
s = \
"""name1
line1
line2
line3
name2
line 4
line 5
line 6"""
f = StringIO(s)
for name, lines in chunkreader(f):
print '***', name
print ''.join(lines)
$ python test.py
*** name1
line1
line2
line3

*** name2
line 4
line 5
line 6

--

Regards,
Tijs
May 31 '07 #4

P: n/a
aspineux wrote:
>
something like

name=None
lines=[]
for line in open('yourfilename.txt'):
if line.startwith('>'):
if name!=None:
print 'Here is the record', name
print lines
print
name=line.stripr('\r')
lines=[]
else:
lines.append(line.stripr('\n'))
That would miss the last chunk.

--

Regards,
Tijs
May 31 '07 #5

P: n/a
In <11**********************@g4g2000hsf.googlegroups. com>, Lee Sander
wrote:
Dear all,
I would like to read a really huge file that looks like this:
>name1....
line_11
line_12
line_13
...
>>name2 ...
line_21
line_22
...
etc

where line_ij is just a free form text on that line.

how can i read file so that every time i do a "read()" i get exactly
one record
up to the next ">"
There was just recently a thread with a `itertools.groupby()` solution.
Something like this:

from itertools import count, groupby, imap
from operator import itemgetter

def mark_records(lines):
counter = 0
for line in lines:
if line.startswith('>'):
counter += 1
yield (counter, line)
def iter_records(lines):
fst = itemgetter(0)
snd = itemgetter(1)
for dummy, record_lines in groupby(mark_records(lines), fst):
yield imap(snd, record_lines)
def main():
source = """\
name1....
line_11
line_12
line_13
....
name2 ...
line_21
line_22
....""".splitlines()

for record in iter_records(source):
print 'Start of record...'
for line in record:
print ':', line

Ciao,
Marc 'BlackJack' Rintsch
May 31 '07 #6

P: n/a
"Lee Sander" <le..e@gmail.com>wrote:

I wanted to also say that this file is really huge, so I cannot
just do a read() and then split on ">" to get a record
thanks
lee

On May 31, 1:26 pm, Lee Sander <lesa...@gmail.comwrote:
Dear all,
I would like toreada really hugefilethat looks like this:
name1....
line_11
line_12
line_13
...>name2 ...

line_21
line_22
...
etc

where line_ij is just a free form text on that line.

how can ireadfileso that every time i do a "read()" i get exactly
onerecord
up to the next ">"

many thanks
Lee
I would do something like: (not tested):

def get_a_record(f,sep):
ret_rec = ''
while True:
char = f.read(1)
if char == sep:
break
else:
ret_rec += char
return ret_rec

- Hendrik

Jun 1 '07 #7

This discussion thread is closed

Replies have been disabled for this discussion.