By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
455,548 Members | 1,487 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 455,548 IT Pros & Developers. It's quick & easy.

Simple file writing techiques ...

P: n/a
Hello,

I've often found that I am writing little scripts at the interpretor to
read a text file, perform some conversion, and then write the converted
data back out to a file. I normally accomplish the above task by
reading the lines of the entire file into a list, preforming some
function to that list and then writing the list back out. I want to
write a generic function/module that will free me from repeatedly
typing the same thing, perhaps convertFile(filename, covertFunc) and I
was wondering what would be a more optimal solution for writing the
data,

fout = open('somefile','w')
for line in convertedData:
fout.write("%s\n" % line)
fout.close()

-- or --

fout = open('somefile','w')
fout.write("%s" % '\n'.join(convertedData))
fout.close()

.... or maybe some hybrid of the two which writes chunks of the
convertedData list out in one shot ...

An issue that I'm probably most concerned with is scalabitiy, what if
the file was huge, like some sort of log file. As well, I know from
'import this' that there should only be one obvious way to do something
.... however to me it's not so obvious :(

Any suggestions,

Colin

Jul 19 '06 #1
Share this Question
Share on Google+
6 Replies


P: n/a
cdecarlo wrote:
>
fout = open('somefile','w')
for line in convertedData:
fout.write("%s\n" % line)
fout.close()

-- or --

fout = open('somefile','w')
fout.write("%s" % '\n'.join(convertedData))
fout.close()
I'd go for something like...

fout = open('somefile','w')
fout.writelines( line+"\n" for line in convertedData )
fout.close()

Although your first solution should perform about the same.

If you have 2.5, you may prefer this...

with open('somefile','w') as fout:
fout.writelines( line+"\n" for line in convertedData )

... or maybe some hybrid of the two which writes chunks of the
convertedData list out in one shot ...
The OS should buffer it for you.
Will McGugan

Jul 19 '06 #2

P: n/a
Ant
fout = open('somefile','w')
for line in convertedData:
fout.write("%s\n" % line)
fout.close()

-- or --

fout = open('somefile','w')
fout.write("%s" % '\n'.join(convertedData))
fout.close()
I shouldn't think it matters too much which of these you use - time
them and see what happens.
An issue that I'm probably most concerned with is scalabitiy, what if
the file was huge, like some sort of log file.
Sucking in the entire file into a list won't scale well, as a huge log
file could quickly eat all of your available memory. You'd be better
off processing each line as you go, and writing it to a temp file,
renaming the temp file once you have finished. Something like:

in_f = "access.log"
out_f = "access.log.tmp"

infile = open(in_f)
outfile = open(out_f)

for line in infile:
outfile.write(process(line))

infile.close()
outfile.close()

os.remove(in_f)
os.rename(out_f, in_f)

(Not tested, but you get the idea...)

Jul 19 '06 #3

P: n/a
Ant
Whoops:
outfile = open(out_f)
outfile = open(out_f, 'w')

may be better ;-)

Jul 19 '06 #4

P: n/a
>>>>cdecarlo <cd******@gmail.comwrites:
fout = open('somefile','w')
for line in convertedData:
fout.write("%s\n" % line)
fout.close()
-- or --
fout = open('somefile','w')
fout.write("%s" % '\n'.join(convertedData))
fout.close()
... or maybe some hybrid of the two which writes chunks of the
convertedData list out in one shot ...
The second option would be definitely faster.
An issue that I'm probably most concerned with is scalabitiy, what if
the file was huge, like some sort of log file.
Considering that you've already read in the whole file into a list, it's too
late to worry about scalability when writing out :-). Have you considered
the fileinput module?

Ganesan

--
Ganesan Rajagopal

Jul 19 '06 #5

P: n/a
See also this recent discussion on techniques for making sure you get
the file closed, even if there is an exception.

http://tinyurl.com/fryrv

rd

Jul 19 '06 #6

P: n/a
cdecarlo wrote:
Hello,

I've often found that I am writing little scripts at the interpretor to
read a text file, perform some conversion, and then write the converted
data back out to a file. I normally accomplish the above task by
<snip>
>
Any suggestions,

Colin
You should check out the fileinput module.

HTH,
~Simon

Jul 19 '06 #7

This discussion thread is closed

Replies have been disabled for this discussion.