simultaneous reading and writing a textfile

[Marco Herrn]

Hi,

I have a text file with some lines in it.
Now I want to iterate over this file and exchange some lines with some
others. I tried this approach:

try:
myfile= file('myfile', 'r+')

while 1:
line= myfile.readline()
if not line: break

l= line.strip().split()
if len(l) == 1:
hostname= l[0]
myfile.write(hostname+' '+mac+'\n')
break

finally:
if not myfile is None:
myfile.close()
This should inspect the file and find the first line, that can't be
split into two parts (that means, which has only one word in it).
This line should be exchanged with a line that contains some more
info.

Unfortunately (or how better python programmers than I am would say,
"of course") this doesn't work. The line is exchanged, but also some
more lines.

Now how can I achieve, what I want? Really exchange one line with
another, regardless of their length. Is this possible? If this is not
possible, then what would be the best approach to do this?

I do not want to read the whole file, exchange the line in memory and
then write the whole file. This would be quite slow with large files.

Regard1s
Marco

[Larry Bates]

Marco Herrn wrote:

Hi,

I have a text file with some lines in it.
Now I want to iterate over this file and exchange some lines with some
others. I tried this approach:

try:
myfile= file('myfile', 'r+')

while 1:
line= myfile.readline()
if not line: break

l= line.strip().split()
if len(l) == 1:
hostname= l[0]
myfile.write(hostname+' '+mac+'\n')
break

finally:
if not myfile is None:
myfile.close()
This should inspect the file and find the first line, that can't be
split into two parts (that means, which has only one word in it).
This line should be exchanged with a line that contains some more
info.

Unfortunately (or how better python programmers than I am would say,
"of course") this doesn't work. The line is exchanged, but also some
more lines.

Now how can I achieve, what I want? Really exchange one line with
another, regardless of their length. Is this possible? If this is not
possible, then what would be the best approach to do this?

I do not want to read the whole file, exchange the line in memory and
then write the whole file. This would be quite slow with large files.

Regard1s
Marco

The only way that in-place writes will work is if you adopt a fixed
line length for the file and use read(bytes) instead of readline()
method. You could create file with fixed length lines and write them
in-place without disturbing lines that follow. Take a look at seek(),
write() methods.

The alternative is to create a different file (read/write
the entire file)each time you want to make a change. Unless the file
is REALLY long, this will be extremely fast.

If you have a LOT of data you want to work with this way and it is
changing a lot, you need to use a database not a text file for data
storage.

-Larry

[Peter Otten]

Marco Herrn wrote:

I have a text file with some lines in it.
Now I want to iterate over this file and exchange some lines with some
others. I tried this approach: This should inspect the file and find the first line, that can't be
split into two parts (that means, which has only one word in it).
This line should be exchanged with a line that contains some more
info.

Unfortunately (or how better python programmers than I am would say,
"of course") this doesn't work. The line is exchanged, but also some
more lines.

Now how can I achieve, what I want? Really exchange one line with
another, regardless of their length. Is this possible? If this is not
possible, then what would be the best approach to do this?
A file is exposed as a sequence of bytes. You can only exchange a string of
bytes with the same number of (different) bytes. Inserting or removing
bytes means you have to rewrite the file from the insertion/deletion point
onwards. Usually you don't bother and just rewrite the whole file.
I do not want to read the whole file, exchange the line in memory and
then write the whole file. This would be quite slow with large files.

Here's a simple approach that can deal with large files:

# does not work on windows
BLOCKSIZE = 2**20
infile = open(filename)
os.unlink(filename) # rename if you want to play it safe
outfile = open(filename, "w")
lines = iter(infile.readline, "")
# copy one line at a time
for line in lines:
parts = line.split(None, 1)
if len(parts) == 1:
outfile.write("%s %s\n" % (parts[0], mac))
break
outfile.write(line)
# after the insertion point proceed with larger steps
for block in iter(lambda: infile.read(BLOCKSIZE), ""):
outfile.write(block)
infile.close()
outfile.close()

When this becomes too slow you should consider a database.

Peter

[Marco Herrn]

On 2006-05-16, Peter Otten <__*******@web.de> wrote:

Now how can I achieve, what I want? Really exchange one line with
another, regardless of their length. Is this possible? If this is not
possible, then what would be the best approach to do this?

A file is exposed as a sequence of bytes. You can only exchange a string of
bytes with the same number of (different) bytes. Inserting or removing
bytes means you have to rewrite the file from the insertion/deletion point
onwards. Usually you don't bother and just rewrite the whole file.

Thanks to you both. I will use that. I do not think, that the files
will be that large, that a database is useful. The main advantage of
the file is, that it is directly human readible.

Regards
Marco