Canonical way of dealing with null-separated lines?

[Douglas Alan]

Is there a canonical way of iterating over the lines of a file that
are null-separated rather than newline-separated? Sure, I can
implement my own iterator using read() and split(), etc., but
considering that using "find -print0" is so common, it seems like
there should be a more cannonical way.

|>oug

[Christopher De Vries]

On Wed, Feb 23, 2005 at 10:54:50PM -0500, Douglas Alan wrote:

Is there a canonical way of iterating over the lines of a file that
are null-separated rather than newline-separated?

I'm not sure if there is a canonical method, but I would recommending using a
generator to get something like this, where 'f' is a file object:

def readnullsep(f):
# Need a place to put potential pieces of a null separated string
# across buffer boundaries
retain = []

while True:
instr = f.read(2048)
if len(instr)==0:
# End of file
break

# Split over nulls
splitstr = instr.split('\0 ')

# Combine with anything left over from previous read
retain.append(s plitstr[0])
splitstr[0] = ''.join(retain)

# Keep last piece for next loop and yield the rest
retain = [splitstr[-1]]
for element in splitstr[:-1]:
yield element

# yield anything left over
yield retain[0]

Chris

[Scott David Daniels]

Douglas Alan wrote:

Is there a canonical way of iterating over the lines of a file that
are null-separated rather than newline-separated? Sure, I can
implement my own iterator using read() and split(), etc., but
considering that using "find -print0" is so common, it seems like
there should be a more cannonical way.

You could start with this code and add '\0' as a line terminator:

http://members.dsl-only.net/~daniels/ilines.html
--Scott David Daniels
Sc***********@A cm.Org

[Douglas Alan]

Christopher De Vries <de*****@idolst arastronomer.co m> writes:

I'm not sure if there is a canonical method, but I would
recommending using a generator to get something like this, where 'f'
is a file object:

Thanks for the generator. It returns an extra blank line at the end
when used with "find -print0", which is probably not ideal, and is
also not how the normal file line iterator behaves. But don't worry
-- I can fix it.

In any case, as a suggestion to the whomever it is that arranges for
stuff to be put into the standard library, there should be something
like this there, so everyone doesn't have to reinvent the wheel (even
if it's an easy wheel to reinvent) for something that any sysadmin
(and many other users) would want to do on practically a daily basis.

|>oug

[Scott David Daniels]

Douglas Alan wrote:
....

In any case, as a suggestion to the whomever it is that arranges for
stuff to be put into the standard library, there should be something
like this there, so everyone doesn't have to reinvent the wheel (even
if it's an easy wheel to reinvent) for something that any sysadmin
(and many other users) would want to do on practically a daily basis.

The general model is that you produce a module, and if it gains a
audience to a stable interface, inclusion might be considered. I'd
suggest you put up a recipe at ActiveState.

--Scott David Daniels
Sc***********@A cm.Org

[Christopher De Vries]

On Thu, Feb 24, 2005 at 02:03:52PM -0500, Douglas Alan wrote:

Thanks for the generator. It returns an extra blank line at the end
when used with "find -print0", which is probably not ideal, and is
also not how the normal file line iterator behaves. But don't worry
-- I can fix it.

Sorry... I forgot to try it with a null terminated string. I guess it further
illustrates the power of writing good test cases. Something like this would
help:

# yield anything left over
if retain[0]:
yield retain[0]

The other modification would be an option to ignore multiple nulls in a row,
rather than returning empty strings, which could be done in a similar way.

Chris

[John Machin]

On Thu, 24 Feb 2005 11:53:32 -0500, Christopher De Vries
<de*****@idolst arastronomer.co m> wrote:

On Wed, Feb 23, 2005 at 10:54:50PM -0500, Douglas Alan wrote:

Is there a canonical way of iterating over the lines of a file that
are null-separated rather than newline-separated?
I'm not sure if there is a canonical method, but I would recommending using a
generator to get something like this, where 'f' is a file object:

def readnullsep(f):
# Need a place to put potential pieces of a null separated string
# across buffer boundaries
retain = []

while True:
instr = f.read(2048)
if len(instr)==0:
# End of file
break

# Split over nulls
splitstr = instr.split('\0 ')

# Combine with anything left over from previous read
retain.append(s plitstr[0])
splitstr[0] = ''.join(retain)

# Keep last piece for next loop and yield the rest
retain = [splitstr[-1]]
for element in splitstr[:-1]:

(1) Inefficient (copies all but the last element of splitstr)
yield element

# yield anything left over
yield retain[0]

(2) Dies when the input file is empty.

(3) As noted by the OP, can return a spurious empty line at the end.

Try this:

!def readweird(f, line_end='\0', bufsiz=8192):
! retain = ''
! while True:
! instr = f.read(bufsiz)
! if not instr:
! # End of file
! break
! splitstr = instr.split(lin e_end)
! if splitstr[-1]:
! # last piece not terminated
! if retain:
! splitstr[0] = retain + splitstr[0]
! retain = splitstr.pop()
! else:
! if retain:
! splitstr[0] = retain + splitstr[0]
! retain = ''
! del splitstr[-1]
! for element in splitstr:
! yield element
! if retain:
! yield retain

Cheers,
John

[John Machin]

On Thu, 24 Feb 2005 14:51:07 -0500, Christopher De Vries
<de*****@idolst arastronomer.co m> wrote:

The other modification would be an option to ignore multiple nulls in a row,
rather than returning empty strings, which could be done in a similar way.

Why not leave this to the caller? Efficiency?? Filtering out empty
lines is the least of your worries.

Try giving the callers options to do things they *can't* do
themselves, like a different line-terminator or a buffer size > 2048
[which could well enhance efficiency] or < 10 [which definitely
enhances testing]

[Christopher De Vries]

On Fri, Feb 25, 2005 at 07:56:49AM +1100, John Machin wrote:

Try this:
!def readweird(f, line_end='\0', bufsiz=8192):
! retain = ''
! while True:
! instr = f.read(bufsiz)
! if not instr:
! # End of file
! break
! splitstr = instr.split(lin e_end)
! if splitstr[-1]:
! # last piece not terminated
! if retain:
! splitstr[0] = retain + splitstr[0]
! retain = splitstr.pop()
! else:
! if retain:
! splitstr[0] = retain + splitstr[0]
! retain = ''
! del splitstr[-1]
! for element in splitstr:
! yield element
! if retain:
! yield retain

I think this is a definite improvement... especially putting the buffer size
and line terminators as optional arguments, and handling empty files. I think,
however that the if splitstr[-1]: ... else: ... clauses aren't necessary, so I
would probably reduce it to this:

!def readweird(f, line_end='\0', bufsiz=8192):
! retain = ''
! while True:
! instr = f.read(bufsiz)
! if not instr:
! # End of file
! break
! splitstr = instr.split(lin e_end)
! if retain:
! splitstr[0] = retain + splitstr[0]
! retain = splitstr.pop()
! for element in splitstr:
! yield element
! if retain:
! yield retain

Popping off that last member and then iterating over the rest of the list as
you suggested is so much more efficient, and it looks a lot better.

Chris

[John Machin]

On Thu, 24 Feb 2005 16:51:22 -0500, Christopher De Vries
<de*****@idolst arastronomer.co m> wrote:

[snip]

I think this is a definite improvement... especially putting the buffer size
and line terminators as optional arguments, and handling empty files. I think,
however that the if splitstr[-1]: ... else: ... clauses aren't necessary,
Indeed. Any efficiency gain would be negated by the if test and it's
only once per buffer-full anyway. I left all that stuff in to show
that I had actually analyzed the four cases i.e. it wasn't arrived at
by lucky accident.
so I
would probably reduce it to this:
[snip]
Popping off that last member and then iterating over the rest of the list as
you suggested is so much more efficient, and it looks a lot better.

Yeah. If it looks like a warthog, it is a warthog. The converse is of
course not true; examples of elegant insufficiency abound.

Cheers,
John