Hello,
I've often found that I am writing little scripts at the interpretor to
read a text file, perform some conversion, and then write the converted
data back out to a file. I normally accomplish the above task by
reading the lines of the entire file into a list, preforming some
function to that list and then writing the list back out. I want to
write a generic function/module that will free me from repeatedly
typing the same thing, perhaps convertFile(fil ename, covertFunc) and I
was wondering what would be a more optimal solution for writing the
data,
fout = open('somefile' ,'w')
for line in convertedData:
fout.write("%s\ n" % line)
fout.close()
-- or --
fout = open('somefile' ,'w')
fout.write("%s" % '\n'.join(conve rtedData))
fout.close()
.... or maybe some hybrid of the two which writes chunks of the
convertedData list out in one shot ...
An issue that I'm probably most concerned with is scalabitiy, what if
the file was huge, like some sort of log file. As well, I know from
'import this' that there should only be one obvious way to do something
.... however to me it's not so obvious :(
Any suggestions,
Colin 6 1292
cdecarlo wrote:
>
fout = open('somefile' ,'w')
for line in convertedData:
fout.write("%s\ n" % line)
fout.close()
-- or --
fout = open('somefile' ,'w')
fout.write("%s" % '\n'.join(conve rtedData))
fout.close()
I'd go for something like...
fout = open('somefile' ,'w')
fout.writelines ( line+"\n" for line in convertedData )
fout.close()
Although your first solution should perform about the same.
If you have 2.5, you may prefer this...
with open('somefile' ,'w') as fout:
fout.writelines ( line+"\n" for line in convertedData )
... or maybe some hybrid of the two which writes chunks of the
convertedData list out in one shot ...
The OS should buffer it for you.
Will McGugan
fout = open('somefile' ,'w')
for line in convertedData:
fout.write("%s\ n" % line)
fout.close()
-- or --
fout = open('somefile' ,'w')
fout.write("%s" % '\n'.join(conve rtedData))
fout.close()
I shouldn't think it matters too much which of these you use - time
them and see what happens.
An issue that I'm probably most concerned with is scalabitiy, what if
the file was huge, like some sort of log file.
Sucking in the entire file into a list won't scale well, as a huge log
file could quickly eat all of your available memory. You'd be better
off processing each line as you go, and writing it to a temp file,
renaming the temp file once you have finished. Something like:
in_f = "access.log "
out_f = "access.log.tmp "
infile = open(in_f)
outfile = open(out_f)
for line in infile:
outfile.write(p rocess(line))
infile.close()
outfile.close()
os.remove(in_f)
os.rename(out_f , in_f)
(Not tested, but you get the idea...)
Whoops:
outfile = open(out_f)
outfile = open(out_f, 'w')
may be better ;-)
>>>>cdecarlo <cd******@gmail .comwrites:
fout = open('somefile' ,'w')
for line in convertedData:
fout.write("%s\ n" % line)
fout.close()
-- or --
fout = open('somefile' ,'w')
fout.write("%s" % '\n'.join(conve rtedData))
fout.close()
... or maybe some hybrid of the two which writes chunks of the
convertedData list out in one shot ...
The second option would be definitely faster.
An issue that I'm probably most concerned with is scalabitiy, what if
the file was huge, like some sort of log file.
Considering that you've already read in the whole file into a list, it's too
late to worry about scalability when writing out :-). Have you considered
the fileinput module?
Ganesan
--
Ganesan Rajagopal
See also this recent discussion on techniques for making sure you get
the file closed, even if there is an exception. http://tinyurl.com/fryrv
rd
cdecarlo wrote:
Hello,
I've often found that I am writing little scripts at the interpretor to
read a text file, perform some conversion, and then write the converted
data back out to a file. I normally accomplish the above task by
<snip>
>
Any suggestions,
Colin
You should check out the fileinput module.
HTH,
~Simon This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Rob Somers |
last post by:
Hey all
I am writing a program to keep track of expenses and so on - it is not
a school project, I am learning C as a hobby - At any rate, I am new
to structs and reading and writing to files, two aspects which I want
to incorporate into my program eventually. That aside, my most
pressing problem right now is how to get rid of the newline in the
input when I use fgets(). Now I have looked around on the net, not so
much in this group...
|
by: Greg |
last post by:
I am continuing writing my simple file system and I have run across
the following problem.
I am writing a superblock to a certain frame in the file as follows:
(&sb is the address of a superblock struct with size < FRAME_SZ * 4)
#define FRAME_SZ 512
int frame = 3 ;
|
by: Paul Kuebler |
last post by:
??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????...
|
by: serge calderara |
last post by:
Dear all,
I need to build a web application which will contains articles (long or short)
I was wondering on what is the correct way to retrive those article on web
page.
In orther words, when there is such information to be displayed are they
coming from imported files, database ?
Where and how this type of information is stored ?
What is the way to retrieve such information in order to display it in page ?
|
by: Tomás |
last post by:
Up until lately I've been writing all mad kinds of code for accomplishing
things, but lately I've decided to lean more toward the whole readability,
etc. side of things.
I have a command line application, which I intend to be used like so:
unicon.exe -8to32 source.txt target.txt
There are three command line arguments. The first is something like:
| |
by: true911m |
last post by:
This is a simple walkthrough to get PyInstaller up and running.
I decided to give PI a try, because it claims to be more selective about what it bundles into its executable files by default, and it also integrates UPX (Ultimate Packer for eXecutables) into the build process, if you have it installed. It also claims functionality on linux, as a bonus (I didn't test this). I highly recommend the UPX options, and will cover how to get it...
|
by: its me Rock |
last post by:
Hi All,
How to load bulk of records from one file to another file.Is there any memory allocation techiques available.Need suggestion in that....
Eg:
just reading the data from the file and changing some value and storing into another file .. It's happening for record by record .. But i want to insert bulk of record at the same time.
any response appreciatable..
|
by: theronnightstar |
last post by:
I am having a problem with a small bit of code. I can't figure out why
it won't work. Here:
//code:
#include <iostream>
#include <fstream>
#include <ctype.h>
#include <string>
|
by: jerry |
last post by:
i have written a simple phonebook program,i'll show you some of the
codes,the program's head file is member.h . i suppose the head file
works well.so i don't post it. here's the clips of main function
which i think has problem
// this a simple program about phonebook,it can add items,del
items,find items and
#include "member.h"
#include <fstream>
#include <vector>
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
| |
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |