streaming a file object through re.finditer

[Erick]

Hello,

I've been looking for a while for an answer, but so far I haven't been
able to turn anything up yet. Basically, what I'd like to do is to use
re.finditer to search a large file (or a file stream), but I haven't
figured out how to get finditer to work without loading the entire file
into memory, or just reading one line at a time (or more complicated
buffering).

For example, say I do this:
cat a b c > blah

Then run this python script:

import re
for m in re.finditer('\w+', buffer(file('blah'))): .... print m.group()
....
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: buffer object expected

Of course, this works fine, but it loads the file completely into
memory (right?): for m in re.finditer('\w+', buffer(file('blah').read())):

.... print m.group()
....
a
b
c

So, is there any way to do this?

Thanks,

-e

[Erick]

Ack, typo. What I meant was this:
cat a b c > blah

import re
for m in re.finditer('\w+', file('blah')):
.... print m.group()
....
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: buffer object expected

Of course, this works fine, but it loads the file completely into
memory (right?): for m in re.finditer('\w+', file('blah').read()):

.... print m.group()
....
a
b
c

[Daniel Bickett]

The following example loads the file into memory only one line at a
time, so it should suit your purposes:

data = file( "important.dat" , "w" )
data.write("this\nis\nimportant\ndata")
data.close()
now read it....
import re
data = file( "important.dat" , "r" )
line = data.readline()
while line: for x in re.finditer( "\w+" , line):
print x.group()
line = data.readline()
this
is
important
data

--
Daniel Bickett
dbickett at gmail.com
http://heureusement.org/

[Erick]

True, but it doesn't work with multiline regular expressions :(

-e

[Erik Johnson]

Is it not possible to wrap your loop below within a loop doing
file.read([size]) (or readline() or readlines([size]),
reading the file a chunk at a time then running your re on a per-chunk
basis?

-ej
"Erick" <id******@gmail.com> wrote in message
news:11*********************@f14g2000cwb.googlegro ups.com...

Ack, typo. What I meant was this:
cat a b c > blah

import re
for m in re.finditer('\w+', file('blah')):
... print m.group()
...
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: buffer object expected

Of course, this works fine, but it loads the file completely into
memory (right?): for m in re.finditer('\w+', file('blah').read()):

... print m.group()
...
a
b
c

[Daniel Bickett]

Erick wrote:

True, but it doesn't work with multiline regular expressions :(

If your intent is for the expression to traverse multiple lines (and
possibly match *across* multiple lines,) then, as far as I know, you
have no choice but to load the whole file into memory.

--
Daniel Bickett
dbickett at gmail.com
http://heureusement.org/

[Steven Bethard]

Erick wrote:

Hello,

I've been looking for a while for an answer, but so far I haven't been
able to turn anything up yet. Basically, what I'd like to do is to use
re.finditer to search a large file (or a file stream), but I haven't
figured out how to get finditer to work without loading the entire file
into memory, or just reading one line at a time (or more complicated
buffering).

Can you use mmap?

http://docs.python.org/lib/module-mmap.html

"You can use mmap objects in most places where strings are expected; for
example, you can use the re module to search through a memory-mapped file."

Seems applicable, and it should keep your memory use down, but I'm not
very experienced with it...

Steve