By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
440,621 Members | 1,101 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 440,621 IT Pros & Developers. It's quick & easy.

Formatting text files

P: 1
Im starting new project where I have to take data from one text file and toss it into another is a specified format ( i have a 27 page document outlining that format). I am new to python and was wondering if anyone had some tips on usefull ways to parse text files using python?
Jan 5 '07 #1
Share this Question
Share on Google+
5 Replies


bvdet
Expert Mod 2.5K+
P: 2,851
Im starting new project where I have to take data from one text file and toss it into another is a specified format ( i have a 27 page document outlining that format). I am new to python and was wondering if anyone had some tips on usefull ways to parse text files using python?
I'm no expert on parsing data, but regular expressions (module re) is a powerful tool.
Jan 5 '07 #2

bartonc
Expert 5K+
P: 6,596
Im starting new project where I have to take data from one text file and toss it into another is a specified format ( i have a 27 page document outlining that format). I am new to python and was wondering if anyone had some tips on usefull ways to parse text files using python?
Python has many powerful and easy to use tools for dealing with text. So many, in fact, that picking the tools is sometimes the hard part. After getting a grasp on the language syntax and structures, parsing text is fairly easy to implement.
Jan 6 '07 #3

dshimer
Expert 100+
P: 136
In my mind it also depends a great deal on the structure of the text. I do this task constantly and because of the setting I am in I deal primarily with two types of data files.

One is text or numeric data that is fairly structured, for example each line may contain a variety of numbers or strings delimited by a character or white space. For example to describe a point in space, a file may have hundreds of lines with
Name,X,Y,Z,R1,R2,R3,Desc
Where Name is the name of a point, x,y,z, are geographic coordinates and the R's are some kind of real numbers, and Desc is a text description. When working with these kinds of files I find it easiest to take the data from the input file and create a list of lists then just loop over the lists outputing the data in the new format. I usually grab the whole file using readlines() then run it through a function I call lines2lists which works on data separated by whitespace but could easily be modified to add a delimiter variable.

Expand|Select|Wrap|Line Numbers
  1. def lines2lists(AListOfDataLines):
  2.     '''
  3.     Function readlines returns an entire file with each line as a string in a
  4.     list of data.  This function will convert each string into a list of words,
  5.     then return a list of lists.
  6.     Example:            lines2lists(['first line','the second line','or sentences'])
  7.     Would return:       [['first','line'],['the','second','line'],['or','sentences]]
  8.     '''
  9.     DataList=[]
  10.     for Line in AListOfDataLines:
  11.         DataList.append(Line.split())
  12.     return DataList
  13.  
I usually combine it with readlines(), for example
Expand|Select|Wrap|Line Numbers
  1. Data=utilitymodule.lines2lists(AnOpenFile.readlines())
Where Data is the resulting list in which the example above would look something like the following.
[[Name,X,Y,Z,R1,R2,R3,Desc],[Name,X,Y,Z,R1,R2,R3,Desc],[Name,X,Y,Z,R1,R2,R3,Desc]]

The for each element in Data I format and output the elements of each coordinate (in this case).

The other type of file I work with a lot is one in which there may be multiple elements of a particular dataset but they are on different lines of the file. Using the example above it might look like.
Name1 aString
Name2 aString
X1 aNumber
X2 aNumber
Y1 aNumber
Y2 aNumber
and so on....

In this case I still read in the lines, build lists from them, then use something like the count() function to test for a value in a particular list, if the test value exists, I grab the other member of the list which is the actual data then append it to the actual list of data. In pseudocode it would be something like

If InputString.count(test value like name1) is true then datastring.append( the value associated with name1) then when all the input strings have been parsed. for each element of datastring, format and output the values.

There are probably easier ways to do this, but most of the time I may only need to write a program every couple of weeks, may only have 5 minutes notice and need to have it done very quickly. Since there is so much power in lists I tend to stick with the functions I know and love and can hack together quickly.
Jan 9 '07 #5

bvdet
Expert Mod 2.5K+
P: 2,851
dshimer,
Whatever works for you in a efficient manner IS the easiest way. Here are two things that I learned on this forum:
Expand|Select|Wrap|Line Numbers
  1. f = open(dlg1.import_file, "r")
  2. # Files can be used as iterators. Internally, for calls file next() method.
  3. for item in f:
  4.     ....do stuff....
Expand|Select|Wrap|Line Numbers
  1. # If a sub-string is in a string, evaluate True
  2. if "subject_text" in item.lower():
  3.     pt = re.split('[:;,]', item)
Maybe you can use these sometime.
Jan 9 '07 #6

Post your reply

Sign in to post your reply or Sign up for a free account.