Graceful detection of EOF

[MickeyBob]

How does one detect the EOF gracefully? Assuming I have a pickle file
containing an unknown number of objects, how can I read (i.e.,
pickle.load()) until the EOF is encountered without generating an EOF
exception?

Thanks for any assistance.
MickeyBob

[Jeff Epler]

Write a file-like object that can "look ahead" and provide a flag to
check in your unpickling loop, and which implements enough of the file
protocol ("read" and "readline", apparently) to please pickle. The
following worked for me.

class PeekyFile:
def __init__(self, f):
self.f = f
self.peek = ""

def eofnext(self):
if self.peek: return False
try:
self.peek = self.f.read(1)
except EOFError:
return True
return not self.peek

def read(self, n=None):
if n is not None:
n = n - len(self.peek)
result = self.peek + self.f.read(n)
else:
result = self.peek + self.f.read()
self.peek = ""
return result

def readline(self):
result = self.peek + self.f.readline ()
self.peek = ""
return result

import StringIO, pickle
o = StringIO.String IO()
for x in range(5):
pickle.dump(x, o)
i = PeekyFile(Strin gIO.StringIO(o. getvalue()))
while 1:
i.eofnext()
if i.eofnext():
break
print pickle.load(i)
print "at the end"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQFBZZsVJd0 1MZaTXX0RAl0FAJ 9GCBIWmLaS+UbhC gZGR6PlJ94c4QCe Pq/k
x9c7Hokjaj+RpSY ryvEwCJ8=
=sIw8
-----END PGP SIGNATURE-----

[Andrew Dalke]

MickeyBob wrote:

How does one detect the EOF gracefully? Assuming I have a pickle file
containing an unknown number of objects, how can I read (i.e.,
pickle.load()) until the EOF is encountered without generating an EOF
exception?

Why isn't catching the exception graceful?

# UNTESTED CODE

def load_pickle_ite r(infile):
while 1:
try:
yield pickle.load(inf ile)
except EOFError:
break

for obj in load_pickle_ite r(open("mydata. pickle", "rb")):
print obj
This is well in line with the normal Python idiom,
as compared to "look before you leap".

Andrew
da***@dalkescie ntific.com

[Jeremy Jones]

Andrew Dalke wrote:

MickeyBob wrote:

How does one detect the EOF gracefully? Assuming I have a pickle file
containing an unknown number of objects, how can I read (i.e.,
pickle.load()) until the EOF is encountered without generating an EOF
exception?

Why isn't catching the exception graceful?

# UNTESTED CODE

def load_pickle_ite r(infile):
while 1:
try:
yield pickle.load(inf ile)
except EOFError:
break

for obj in load_pickle_ite r(open("mydata. pickle", "rb")):
print obj
This is well in line with the normal Python idiom,
as compared to "look before you leap".

Andrew
da***@dalkescie ntific.com

So, what you're saying is that the Python way, in contradistincti on to
"look before you leap", is "land in it, then wipe it off?" Can we get
that in the Zen of Python? :-)

Seriously, this is beautiful. I understand generators, but haven't
become accustomed to using them yet. That is just beautiful, which _is_
Zen.
Jeremy Jones

[Egbert Bouwman]

A file is too large to fit into memory.
The first line must receive a special treatment, because
it contains information about how to handle the rest of the file.

Of course it is not difficult to test if you are reading the first line
or another one, but it hurts my feelings to do a test which by definition
succeeds at the first record, and never afterwards.
Any suggestions ?
egbert
--
Egbert Bouwman - Keizersgracht 197 II - 1016 DS Amsterdam - 020 6257991
=============== =============== =============== =============== ============

[Peter Otten]

Egbert Bouwman wrote:

A file is too large to fit into memory.
The first line must receive a special treatment, because
it containsÂ*Â*inf ormationÂ*about Â*howÂ*toÂ*hand leÂ*theÂ*restÂ* ofÂ*theÂ*file.

Of course it is not difficult to test if you are reading the first line
or another one, but it hurts my feelings to do a test which by definition
succeeds at the first record, and never afterwards.

lines = iter("abc")
for first in lines: .... print first
.... break
....
a for line in lines:

.... print line
....
b
c

Unless it hurts your feelings to unconditionally break out of a for-loop,
that is.

Peter

[Gerrit]

Peter Otten wrote:

>>> lines = iter("abc")
for first in lines: ... print first
... break
...
a for line in lines: ... print line
...
b
c

Unless it hurts your feelings to unconditionally break out of a for-loop,
that is.

How about:

lines = iter("abc")
first = lines.next()
print first a for line in lines:

.... print line
....
b
c

Would hurt less feeling I presume.

Gerrit.

--
Weather in Twenthe, Netherlands 08/10 11:25:
11.0°C Few clouds mostly cloudy wind 0.9 m/s None (57 m above NAP)
--
In the councils of government, we must guard against the acquisition of
unwarranted influence, whether sought or unsought, by the
military-industrial complex. The potential for the disastrous rise of
misplaced power exists and will persist.
-Dwight David Eisenhower, January 17, 1961

[Alex Martelli]

Jeremy Jones <za******@bells outh.net> wrote:
...

This is well in line with the normal Python idiom,
as compared to "look before you leap".

Andrew
da***@dalkescie ntific.com

So, what you're saying is that the Python way, in contradistincti on to
"look before you leap", is "land in it, then wipe it off?" Can we get
that in the Zen of Python? :-)

The "normal Python idiom" is often called, in honor and memory of
Admiral Grace Murray-Hopper (arguably the most significant woman in the
history of programming languages to this time), "it's Easier to Ask
Forgiveness than Permission" (EAFP, vs the LBYL alternative). This
motto has been attributed to many, but Ms Hopper was undoubtedly the
first one to use it reportedly and in our field.

In the general case, trying to ascertain that an operation will succeed
before attempting the operation has many problems. Often you end up
repeating the same steps between the ascertaining and the actual usage,
which offends the "Once and Only Once" principle as well as slowing
things down. Sometimes you cannot ensure that the ascertaining and the
operating pertain to exactly the same thing -- the world can have
changed in-between, or the code might present subtle differences between
the two cases.

In contrast, if a failed attempt can be guaranteed to not alter
persistent state and only result in an easily catchable exception, EAFP
can better deliver on its name. In terms of your analogy, there's
nothing to "wipe off" -- if the leap "misfires", no damage is done.
Alex

[Alex Martelli]

Egbert Bouwman <eg*********@hc cnet.nl> wrote:

A file is too large to fit into memory.
The first line must receive a special treatment, because
it contains information about how to handle the rest of the file.

Of course it is not difficult to test if you are reading the first line
or another one, but it hurts my feelings to do a test which by definition
succeeds at the first record, and never afterwards.

option 1, the one I would use:

thefile = open('somehugef ile.txt')
first_line = thefile.next()
deal_with_first (first_line)
for line in thefile:
deal_with_other (line)

this requires Python 2.3 or better, so that thefile IS-AN iterator; in
2.2, get an iterator with foo=iter(thefil e) and use .next and for on
that (better still, upgrade!).

option 2, not unreasonable (not repeating the open & calls...):

first_line = thefile.readlin e()
for line in thefile: ...

option 3, a bit cutesy:

for first_line in thefile: break
for line in thefile: ...

(again, in 2.2 you'll need some foo=iter(thefil e)).
I'm sure there are others, but 3 is at least 2 too many already,
so...;-)
Alex

[Peter Otten]

Gerrit wrote:

first = lines.next()
[as opposed to 'for first in lines: break']

Would hurt less feeling I presume.

iter("").next()

Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration

I feel a little uneasy with that ...unless I'm sure I want to deal with the
StopIteration elsewhere.
Looking at it from another angle, the initial for-loop ist just a peculiar
way to deal with an empty iterable. So the best (i. e. clear, robust and
general) approach is probably

items = iter(...)
try:
first = items.next()
except StopIteration:
# deal with empty iterator, e. g.:
raise ValueError("nee d at least one item")
else:
# process remaining data

part of which is indeed your suggestion.

Peter