471,349 Members | 1,260 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,349 software developers and data experts.

readlines() reading incorrect number of lines?

Hi,

I'm currently using Python to deal with a fairly large text file (800
MB), which I know has about 85,000 lines of text. I can confirm this
because (1) I built the file myself, and (2) running a basic Java
program to count lines yields a number in that range.

However, when I use Python's various methods -- readline(),
readlines(), or xreadlines() and loop through the lines of the file,
the line program exits at 16,000 lines. No error output or anything --
it seems the end of the loop was reached, and the code was executed
successfully.

I'm baffled and confused, and would be grateful for any advice as to
what I'm doing wrong, or why this may be happening.

Thank you,
Wojciech Gryc
Dec 20 '07 #1
7 3805
On Dec 21, 6:48 am, Wojciech Gryc <wojci...@gmail.comwrote:
Hi,

I'm currently using Python to deal with a fairly large text file (800
MB), which I know has about 85,000 lines of text. I can confirm this
because (1) I built the file myself, and (2) running a basic Java
program to count lines yields a number in that range.

However, when I use Python's various methods -- readline(),
readlines(), or xreadlines() and loop through the lines of the file,
the line program exits at 16,000 lines. No error output or anything --
it seems the end of the loop was reached, and the code was executed
successfully.

I'm baffled and confused, and would be grateful for any advice as to
what I'm doing wrong, or why this may be happening.
What platform, what version of python?

One possibility: you are running this on Windows and the file contains
Ctrl-Z aka chr(26) aka '\x1a'.
Dec 20 '07 #2
Hi,

Python 2.5, on Windows XP. Actually, I think you may be right about
\x1a -- there's a few lines that definitely have some strange
character sequences, so this would make sense... Would you happen to
know how I can actually fix this (e.g. replace the character)? Since
Python doesn't see the rest of the file, I don't even know how to get
to it to fix the problem... Due to the nature of the data I'm working
with, manual editing is also not an option.

Thanks,
Wojciech

On Dec 20, 3:30 pm, John Machin <sjmac...@lexicon.netwrote:
On Dec 21, 6:48 am, Wojciech Gryc <wojci...@gmail.comwrote:
Hi,
I'm currently using Python to deal with a fairly large text file (800
MB), which I know has about 85,000 lines of text. I can confirm this
because (1) I built the file myself, and (2) running a basic Java
program to count lines yields a number in that range.
However, when I use Python's various methods -- readline(),
readlines(), or xreadlines() and loop through the lines of the file,
the line program exits at 16,000 lines. No error output or anything --
it seems the end of the loop was reached, and the code was executed
successfully.
I'm baffled and confused, and would be grateful for any advice as to
what I'm doing wrong, or why this may be happening.

What platform, what version of python?

One possibility: you are running this on Windows and the file contains
Ctrl-Z aka chr(26) aka '\x1a'.
Dec 20 '07 #3
On Dec 21, 7:41 am, Wojciech Gryc <wojci...@gmail.comwrote:
Hi,

Python 2.5, on Windows XP. Actually, I think you may be right about
\x1a -- there's a few lines that definitely have some strange
character sequences, so this would make sense... Would you happen to
know how I can actually fix this (e.g. replace the character)? Since
Python doesn't see the rest of the file, I don't even know how to get
to it to fix the problem... Due to the nature of the data I'm working
with, manual editing is also not an option.
Please don't top-post.

Quick hack to remove all occurrences of '\x1a' (untested):

fin = open('old_file', 'rb') # note b BINARY
fout = open('new_file', 'wb')
blksz = 1024 * 1024
while True:
blk = fin.read(blksz)
if not blk: break
fout.write(blk.replace('\x1a', ''))
fout.close()
fin.close()

You may however want to investigate the "strange character sequences"
that have somehow appeared in your file after you built it
yourself :-)

HTH,
John
Dec 20 '07 #4
[Fixing top-posting.]

On Thu, 20 Dec 2007 12:41:44 -0800, Wojciech Gryc wrote:
On Dec 20, 3:30 pm, John Machin <sjmac...@lexicon.netwrote:
[snip]
However, when I use Python's various methods -- readline(),
readlines(), or xreadlines() and loop through the lines of the file,
the line program exits at 16,000 lines. No error output or anything
-- it seems the end of the loop was reached, and the code was
executed successfully.
....
>One possibility: you are running this on Windows and the file contains
Ctrl-Z aka chr(26) aka '\x1a'.

Hi,

Python 2.5, on Windows XP. Actually, I think you may be right about \x1a
-- there's a few lines that definitely have some strange character
sequences, so this would make sense... Would you happen to know how I
can actually fix this (e.g. replace the character)? Since Python doesn't
see the rest of the file, I don't even know how to get to it to fix the
problem... Due to the nature of the data I'm working with, manual
editing is also not an option.

Thanks,
Wojciech

Open the file in binary mode:

open(filename, 'rb')
and Windows should do no special handling of Ctrl-Z characters.

--
Steven
Dec 20 '07 #5
On Dec 21, 8:13 am, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.auwrote:
[Fixing top-posting.]

On Thu, 20 Dec 2007 12:41:44 -0800, Wojciech Gryc wrote:
On Dec 20, 3:30 pm, John Machin <sjmac...@lexicon.netwrote:
[snip]
However, when I use Python's various methods -- readline(),
readlines(), or xreadlines() and loop through the lines of the file,
the line program exits at 16,000 lines. No error output or anything
-- it seems the end of the loop was reached, and the code was
executed successfully.
...
One possibility: you are running this on Windows and the file contains
Ctrl-Z aka chr(26) aka '\x1a'.
Hi,
Python 2.5, on Windows XP. Actually, I think you may be right about \x1a
-- there's a few lines that definitely have some strange character
sequences, so this would make sense... Would you happen to know how I
can actually fix this (e.g. replace the character)? Since Python doesn't
see the rest of the file, I don't even know how to get to it to fix the
problem... Due to the nature of the data I'm working with, manual
editing is also not an option.
Thanks,
Wojciech

Open the file in binary mode:

open(filename, 'rb')

and Windows should do no special handling of Ctrl-Z characters.

--
Steven
I don't know whether it's a bug or a feature or just a dark corner,
but using mode='rU' does no special handling of Ctrl-Z either.
>>x = 'foo\r\n\x1abar\r\n'
f = open('udcray.txt', 'wb')
f.write(x)
f.close()
open('udcray.txt', 'r').readlines()
['foo\n']
>>open('udcray.txt', 'rU').readlines()
['foo\n', '\x1abar\n']
>>for line in open('udcray.txt', 'rU'):
.... print repr(line)
....
'foo\n'
'\x1abar\n'
>>>
Using 'rU' should make the OP's task of finding the strange character
sequences a bit easier -- he won't have to read a block at a time and
worry about the guff straddling a block boundary.
Dec 20 '07 #6
Something I've occasionally found helpful with problem text files is
to build a histogram of character counts, something like this:
"""
chist.py
print a histogram of character frequencies in a nemed input file
"""

import sys

whitespace = ' \t\n\r\v\f'
lowercase = 'abcdefghijklmnopqrstuvwxyz'
uppercase = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
letters = lowercase + uppercase
ascii_lowercase = lowercase
ascii_uppercase = uppercase
ascii_letters = ascii_lowercase + ascii_uppercase
digits = '0123456789'
hexdigits = digits + 'abcdef' + 'ABCDEF'
octdigits = '01234567'
punctuation = """!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~"""
printable = digits + letters + punctuation

try:
fname = sys.argv[1]
except:
print "usage is chist yourfilename"
sys.exit()

chars = {}

f = open (fname, "rb")
lines = f.readlines()
for line in lines:
for c in line:
try:
chars[ord(c)] += 1
except:
chars[ord(c)] = 1

ords = chars.keys()
ords.sort()

for o in ords:
if chr(o) in printable:
c = chr(o)
else:
c = "UNP"

print "%5d %-5s %10d" % (o, c, chars[o])
print "_" * 50
Gerry

On Dec 20, 5:47 pm, John Machin <sjmac...@lexicon.netwrote:
On Dec 21, 8:13 am, Steven D'Aprano <st...@REMOVE-THIS-

cybersource.com.auwrote:
[Fixing top-posting.]
On Thu, 20 Dec 2007 12:41:44 -0800, Wojciech Gryc wrote:
On Dec 20, 3:30 pm, John Machin <sjmac...@lexicon.netwrote:
[snip]
However, when I use Python's various methods -- readline(),
readlines(), or xreadlines() and loop through the lines of the file,
the line program exits at 16,000 lines. No error output or anything
-- it seems the end of the loop was reached, and the code was
executed successfully.
...
>One possibility: you are running this on Windows and the file contains
>Ctrl-Z aka chr(26) aka '\x1a'.
Hi,
Python 2.5, on Windows XP. Actually, I think you may be right about \x1a
-- there's a few lines that definitely have some strange character
sequences, so this would make sense... Would you happen to know how I
can actually fix this (e.g. replace the character)? Since Python doesn't
see the rest of the file, I don't even know how to get to it to fix the
problem... Due to the nature of the data I'm working with, manual
editing is also not an option.
Thanks,
Wojciech
Open the file in binary mode:
open(filename, 'rb')
and Windows should do no special handling of Ctrl-Z characters.
--
Steven

I don't know whether it's a bug or a feature or just a dark corner,
but using mode='rU' does no special handling of Ctrl-Z either.
>x = 'foo\r\n\x1abar\r\n'
f = open('udcray.txt', 'wb')
f.write(x)
f.close()
open('udcray.txt', 'r').readlines()
['foo\n']
>open('udcray.txt', 'rU').readlines()

['foo\n', '\x1abar\n']>>for line in open('udcray.txt', 'rU'):

... print repr(line)
...
'foo\n'
'\x1abar\n'

Using 'rU' should make the OP's task of finding the strange character
sequences a bit easier -- he won't have to read a block at a time and
worry about the guff straddling a block boundary.
Dec 21 '07 #7
En Fri, 21 Dec 2007 16:42:21 -0300, Gerry <ge**********@gmail.com>
escribi:
whitespace = ' \t\n\r\v\f'
lowercase = 'abcdefghijklmnopqrstuvwxyz'
uppercase = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
letters = lowercase + uppercase
ascii_lowercase = lowercase
ascii_uppercase = uppercase
ascii_letters = ascii_lowercase + ascii_uppercase
digits = '0123456789'
hexdigits = digits + 'abcdef' + 'ABCDEF'
octdigits = '01234567'
punctuation = """!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~"""
printable = digits + letters + punctuation
You do know that most -if not all- of those sets are available as
attributes of the string module, don't you?
You could replace all the lines above with: from string import printable,
as it's the only constant used.

--
Gabriel Genellina

Dec 27 '07 #8

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

2 posts views Thread by Yong Wang | last post: by
3 posts views Thread by Jeremy | last post: by
2 posts views Thread by Alexander Schmidt | last post: by
7 posts views Thread by Nikhil | last post: by
reply views Thread by XIAOLAOHU | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.