Scanning a file

[pinkfloydhomer]

I want to scan a file byte for byte for occurences of the the four byte
pattern 0x00000100. I've tried with this:

# start
import sys

numChars = 0
startCode = 0
count = 0

inputFile = sys.stdin

while True:
ch = inputFile.read( 1)
numChars += 1

if len(ch) < 1: break

startCode = ((startCode << 8) & 0xffffffff) | (ord(ch))
if numChars < 4: continue

if startCode == 0x00000100:
count = count + 1

print count
# end

But it is very slow. What is the fastest way to do this? Using some
native call? Using a buffer? Using whatever?

/David

[Gerhard Häring]

pi************@ gmail.com wrote:

I want to scan a file byte for byte [...]
while True:
ch = inputFile.read( 1)
[...] But it is very slow. What is the fastest way to do this? Using some
native call? Using a buffer? Using whatever?

Read in blocks, not byte for byte. I had good experiences with block
sizes like 4096 or 8192.

-- Gerhard

[pinkfloydhomer]

Okay, how do I do this?

Also, if you look at the code, I build a 32-bit unsigned integer from
the bytes I read. And the 32-bit pattern I am looking for can start on
_any_ byte boundary in the file. It would be nice if I could somehow
just scan for that pattern explicitly, without having to build a 32-bit
integer first. If I could tell python "scan this file for the bytes 0,
0, 1, 0 in succession. How many 0, 0, 1, 0 did you find?"

/David

[Paul Rubin]

pi************@ gmail.com writes:

I want to scan a file byte for byte for occurences of the the four byte
pattern 0x00000100. I've tried with this:

use re.search or string.find. The simplest way is just read the whole
file into memory first. If the file is too big, you have to read it in
chunks and include some hair to notice if the four byte pattern straddles
two adjoining chunks.

[pinkfloydhomer]

I'm now down to:

f = open("filename" , "rb")
s = f.read()
sub = "\x00\x00\x01\x 00"
count = s.count(sub)
print count

Which is quite fast. The only problems is that the file might be huge.
I really have no need for reading the entire file into a string as I am
doing here. All I want is to count occurences this substring. Can I
somehow count occurences in a file without reading it into a string
first?

/David

[Björn Lindström]

"pi************ @gmail.com" <pi************ @gmail.com> writes:

f = open("filename" , "rb")
s = f.read()
sub = "\x00\x00\x01\x 00"
count = s.count(sub)
print count

That's a lot of lines. This is a bit off topic, but I just can't stand
unnecessary local variables.

print file("filename" , "rb").read().co unt("\x00\x00\x 01\x00")

--
Björn Lindström <bk**@stp.lingf il.uu.se>
Student of computational linguistics, Uppsala University, Sweden

[Jorge Godoy]

"pi************ @gmail.com" <pi************ @gmail.com> writes:

Which is quite fast. The only problems is that the file might be huge.
I really have no need for reading the entire file into a string as I am
doing here. All I want is to count occurences this substring. Can I
somehow count occurences in a file without reading it into a string
first?

How about iterating through the file? You can read it line by line, two lines
at a time. Pseudocode follows:

line1 = read_line
while line2 = read_line:
line_to_check = ''.join([line1, line2])
check_for_desir ed_string
line1 = line2

With that you always have two lines in the buffer and you can check all of
them for your desired string, no matter what the size of the file is.
Be seeing you,
--
Jorge Godoy <go***@ieee.org >

[Bernhard Herzog]

Jorge Godoy <go***@ieee.org > writes:

How about iterating through the file? You can read it line by line, two lines
at a time. Pseudocode follows:

line1 = read_line
while line2 = read_line:
line_to_check = ''.join([line1, line2])
check_for_desir ed_string
line1 = line2

With that you always have two lines in the buffer and you can check all of
them for your desired string, no matter what the size of the file is.

This will fail if the string to search for is e.g. "\n\n\n\n" and it
actually occcurs in the file.

Bernhard

--
Intevation GmbH http://intevation.de/
Skencil http://skencil.org/
Thuban http://thuban.intevation.org/

[pinkfloydhomer]

First of all, this isn't a text file, it is a binary file. Secondly,
substrings can overlap. In the sequence 0010010 the substring 0010
occurs twice.

/David

[Andrew McCarthy]

On 2005-10-28, pi************@ gmail.com <pi************ @gmail.com> wrote:

I'm now down to:

f = open("filename" , "rb")
s = f.read()
sub = "\x00\x00\x01\x 00"
count = s.count(sub)
print count

Which is quite fast. The only problems is that the file might be huge.
I really have no need for reading the entire file into a string as I am
doing here. All I want is to count occurences this substring. Can I
somehow count occurences in a file without reading it into a string
first?

Yes - use memory mapping (the mmap module). An mmap object is like a
cross between a file and a string, but the data is only read into RAM
when, and for as long as, necessary. An mmap object doesn't have a
count() method, but you can just use find() in a while loop instead.

Andrew