editing a text file

[bluemountain]

Hi there,

Iam new to python forms and programming too

I had a text file where i need to extract few words of data from the header(which is of 3 lines) and search for the keyword TEXT1, TEXT2, TEXT3in entire file(file consisting of 150 lines) that is related to a particular id and get the sum of the, WRITES of TEXT1, TEXT2, TEXT3

ex: input file will be as below


Windows 2000 text text text text text text text
343434 text text text text text text code textplayback
text text text text text start check text..

text text text text text 343434, text ID: 504181, File: text1, Write_0=24
text text text text text 343434, text ID: 504182, File: text1, Write_0=20
text text text text text 343434, text ID: 504183, File: text1, Write_0=21
text text text text text 343434, text ID: 504181, File: text2, Write_0=29
text text text text text 343434, text ID: 504182, File: text2, Write_0=20
text text text text text 343434, text ID: 504183, File: text2, Write_0=23
text text text text text 343434, text ID: 504181, File: text3, Write_0=21
text text text text text 343434, text ID: 504182, File: text3, Write_0=24
text text text text text 343434, text ID: 504183, File: text3, Write_0=23

output should be like:

343434 textplayback, start check-->this is from the header
text ID: 504181: sum of (24+21+29) ie 27----> this is result of entire file search

Please help me in this regard
Thanks in advance for your help

[bluemountain]

Hi there,

i need help in getting the needed contents of the text file

how can i modify the text file, by filtering out the unwanted text

ex: input file will be as below


Windows 2000 text text text text text text text
343434 text text text text text text code textplayback
text text text text text start check text..

text text text text text 343434, text ID: 504181, File: text1, Write_0=24
text text text text text 343434, text ID: 504182, File: text1, Write_0=20
text text text text text 343434, text ID: 504183, File: text1, Write_0=21
text text text text text 343434, text ID: 504181, File: text2, Write_0=29
text text text text text 343434, text ID: 504182, File: text2, Write_0=20
text text text text text 343434, text ID: 504183, File: text2, Write_0=23
text text text text text 343434, text ID: 504181, File: text3, Write_0=21
text text text text text 343434, text ID: 504182, File: text3, Write_0=24
text text text text text 343434, text ID: 504183, File: text3, Write_0=23

output should be like:

i need to get 1st and 2nd line avoiding the third like in the header

and also sum of the write times for the text id504181 and file text1 from the entire file

How can i search for them. Please help me in this regard
Thanks so much

[dshimer]

I haven't had a lot of time to think through the entire process, but it seems like some basic concepts would be helpful. Here is a session I just ran through and commented that shows how some of the values could be accessed. You will find with python that there are generally dozens of different ways to accomplish a given and there is always a more compact way (unless you get it from a couple of the real pros here). These are just ideas.

>>> f=open('/tmp/test.txt','r')
# Open the file in text, read mode
 
>>> data=f.readlines()
# I really like reading the whole file into a list in which each line is an item in the list.
 
>>> print data[2]
text text text text text start check text..
# Each list item can be treated individually, processed, printed out, or skipped.
 
>>> data[8]
'text text text text text 343434, text ID: 504182, File: text2, Write_0=20\n'
# This is just one of the data lines randomly selected and printed out.
 
>>> data[8].find('504182')
42
If you just need to find a piece of text then this will return it's position in the string or -1 if not found. Note that not finding it does not return a conditional FALSE of 0 because 0 is a valid position.
 
>>> data[8].split(',')
['text text text text text 343434', ' text ID: 504182', ' File: text2', ' Write_0=20\n']
# If individual fields need to be tested you can split on the delimiter (comma in this case).
 
>>> data[8].split(',')[3]
' Write_0=20\n'
# So pulling out an idividual field just requires it's index value.
 
>>> data[8].split(',')[3].split('=')
[' Write_0', '20\n']
# If I want to break that field into it's components I can split it again based on the fields delimeter (here the =)
 
>>> data[8].split(',')[3].split('=')[1]
'20\n'
# So by specifying the index value of the actual data I can get that number. (but it's still a string)
 
>>> int(data[8].split(',')[3].split('=')[1])
20
# Which can be solved by the int() method.
 
>>> data[8].split(',')[1].split(':')[1]
' 504182'
# Using the same idea to pull out a key value.

Hi there,

i need help in getting the needed contents of the text file

how can i modify the text file, by filtering out the unwanted text

ex: input file will be as below


Windows 2000 text text text text text text text
343434 text text text text text text code textplayback
text text text text text start check text..

text text text text text 343434, text ID: 504181, File: text1, Write_0=24
text text text text text 343434, text ID: 504182, File: text1, Write_0=20
text text text text text 343434, text ID: 504183, File: text1, Write_0=21
text text text text text 343434, text ID: 504181, File: text2, Write_0=29
text text text text text 343434, text ID: 504182, File: text2, Write_0=20
text text text text text 343434, text ID: 504183, File: text2, Write_0=23
text text text text text 343434, text ID: 504181, File: text3, Write_0=21
text text text text text 343434, text ID: 504182, File: text3, Write_0=24
text text text text text 343434, text ID: 504183, File: text3, Write_0=23

output should be like:

i need to get 1st and 2nd line avoiding the third like in the header

and also sum of the write times for the text id504181 and file text1 from the entire file

How can i search for them. Please help me in this regard
Thanks so much

[bvdet]

bluemountain,

Hopefully dshimer's suggestions will enable you to find a solution. The following makes use of the re module.

import re
 
def parse_data(fn, textID, hdrkeys):
    lineList = [item.strip() for item in open(fn).readlines()]
    pattH = re.compile(r'%s' % ('|'.join(hdrkeys)))
    # check first three lines for header key words
    hdrList = pattH.findall(''.join(lineList[:3]))
    # create pattern to extract values
    pattD = re.compile(r'=(\d+)')
    valueList = []
    for item in lineList[3:]:
        # if textID is in list item, extract the value
        if textID in item:
            valueList.append(pattD.search(item).group(1))
    # Create output string by joining formatted data
    return '\n'.join([','.join(hdrList), 'textID: %s: sum of (%s) = %s' % \
                      (textID, ','.join(valueList), \
                       sum([int(i) for i in valueList]))
                      ])
 
fn = r'H:\TEMP\temsys\textID.txt'    
hdrkeys = ['343434', 'textplayback', 'start check']
textID = '504181'
print parse_data(fn, textID, hdrkeys)

Output:

>>> 343434,textplayback,start check
textID: 504181: sum of (24,29,21) = 74
>>> 

[bluemountain]

>>> import re
>>> def parse_data(fn, TagID, hdrkeys):
... lineList = [item.strip() for item in open(fn).readlines()]
... pattH = re.compile(r'%s' % ('|'.join(hdrkeys)))
... hdrList = pattH.findall(''.join(lineList[:4]))
... pattD = re.compile(r'=(\d+)')
... valueList = []
... for item in lineList[4:]:
... if TagID in item:
... valueList.append(pattD.search(item).group(1))
... return '\n'.join([','.join(hdrList), 'TagID: %s: sum of (%s) = %s' % \
... (TagID, ','.join(valueList), \
... sum([int(i) for i in valueList]))
... ])
... fn = r'C:\log.txt'
File "<stdin>", line 14
fn = r'C:\log.txt'
^
SyntaxError: invalid syntax

At line 14 it is saying invalid syntax highlighting "fn"

i tried as below
... fn = r'H:\TEMP\temsys\LOG.TXT
File "<stdin>", line 14
fn = r'H:\TEMP\temsys\Log.txt
^
SyntaxError: invalid syntax
>>>

I have a question, where should i specify the input file name
Iam unable to figure out, please correct me where i went wrong

[bluemountain]

And also Thank you so much for your help BVDET

[bvdet]

>>> import re
>>> def parse_data(fn, TagID, hdrkeys):
... lineList = [item.strip() for item in open(fn).readlines()]
... pattH = re.compile(r'%s' % ('|'.join(hdrkeys)))
... hdrList = pattH.findall(''.join(lineList[:4]))
... pattD = re.compile(r'=(\d+)')
... valueList = []
... for item in lineList[4:]:
... if TagID in item:
... valueList.append(pattD.search(item).group(1))
... return '\n'.join([','.join(hdrList), 'TagID: %s: sum of (%s) = %s' % \
... (TagID, ','.join(valueList), \
... sum([int(i) for i in valueList]))
... ])
... fn = r'C:\log.txt'
File "<stdin>", line 14
fn = r'C:\log.txt'
^
SyntaxError: invalid syntax

At line 14 it is saying invalid syntax highlighting "fn"

i tried as below
... fn = r'H:\TEMP\temsys\LOG.TXT
File "<stdin>", line 14
fn = r'H:\TEMP\temsys\Log.txt
^
SyntaxError: invalid syntax
>>>

I have a question, where should i specify the input file name
Iam unable to figure out, please correct me where i went wrong

Please use code tags around your code. There may be an indentation problem or missing parentheses. The following code executed at the DOS command prompt as shown below:

import re
 
def parse_data(fn, textID, hdrkeys):
    lineList = [item.strip() for item in open(fn).readlines()]
    pattH = re.compile(r'%s' % ('|'.join(hdrkeys)))
    # check first three lines for header key words
    hdrList = pattH.findall(''.join(lineList[:3]))
    # create pattern to extract values
    pattD = re.compile(r'=(\d+)')
    valueList = []
    for item in lineList[3:]:
        # if textID is in list item, extract the value
        if textID in item:
            valueList.append(pattD.search(item).group(1))
    # Create output string by joining formatted data
    return '\n'.join([','.join(hdrList), 'textID: %s: sum of (%s) = %s' % \
                      (textID, ','.join(valueList), \
                       sum([int(i) for i in valueList]))
                      ])
 
if __name__ == '__main__':
    fn = r'H:\TEMP\temsys\textID.txt'    
    hdrkeys = ['343434', 'textplayback', 'start check']
    textID = '504181'
    print parse_data(fn, textID, hdrkeys)

C:\Python23>python file_textID.py
343434,textplayback,start check
textID: 504181: sum of (24,29,21) = 74

It also works in Pythonwin.

[bluemountain]

Firstly Thanks for the fast info you had provided me.
This time i ran the whole test in pyinteractive shell instead of running it in command prompt. Its pretty easy running from there, no need of checking for indentation all the time.

The code you provided is working, but having a problem retrieving the sum
it is ginving me sum=0

import re

def parse_data(fn, TagID, hdrkeys):
lineList = [item.strip() for item in open(fn).readlines()]
pattH = re.compile(r'%s' % ('|'.join(hdrkeys)))
# check first three lines for header key words
hdrList = pattH.findall(''.join(lineList[:3]))
# create pattern to extract values
pattD = re.compile(r'=(\d+)')
valueList = []
for item in lineList[3:]:
# if textID is in list item, extract the value
if TagID in item:
valueList.append(pattD.search(item).group(1))
# Create output string by joining formatted data
return '\n'.join([','.join(hdrList), 'TagID: %s: sum of (%s) = %s' % \
(TagID, ','.join(valueList), \
sum([int(i) for i in valueList]))
])

if __name__ == '__main__':
fn = r'c:\log1.txt'
hdrkeys = ['53301', 'RS485', 'Write']
TagID = '504181'
print parse_data(fn, TagID, hdrkeys)

I dont have any knowledge of copying and pasting it along with numbers.

but the above is the code i tried to run and the result is like
IDLE 1.1.4
>>>

TagID: 504181: sum of () = 0
>>>

Please correct me

Thanks for all the help you were extending

[bvdet]

Firstly Thanks for the fast info you had provided me.
This time i ran the whole test in pyinteractive shell instead of running it in command prompt. Its pretty easy running from there, no need of checking for indentation all the time.

The code you provided is working, but having a problem retrieving the sum
it is ginving me sum=0

import re
 
def parse_data(fn, TagID, hdrkeys):
    lineList = [item.strip() for item in open(fn).readlines()]
    pattH = re.compile(r'%s' % ('|'.join(hdrkeys)))
    # check first three lines for header key words
    hdrList = pattH.findall(''.join(lineList[:3]))
    # create pattern to extract values
    pattD = re.compile(r'=(\d+)')
    valueList = []
    for item in lineList[3:]:
        # if textID is in list item, extract the value
        if TagID in item:
            valueList.append(pattD.search(item).group(1))
    # Create output string by joining formatted data
    return '\n'.join([','.join(hdrList), 'TagID: %s: sum of (%s) = %s' % \
                      (TagID, ','.join(valueList), \
                       sum([int(i) for i in valueList]))
                      ])
 
if __name__ == '__main__':
    fn = r'c:\log1.txt'    
    hdrkeys = ['53301', 'RS485', 'Write']
    TagID = '504181'
    print parse_data(fn, TagID, hdrkeys)

I dont have any knowledge of copying and pasting it along with numbers.

but the above is the code i tried to run and the result is like
IDLE 1.1.4
>>>

TagID: 504181: sum of () = 0
>>>

Please correct me

Thanks for all the help you were extending

I added code tags for you above. Now isn't that better? There is nothing in valueList to sum. Maybe the TagID you are using is not in the file.

[bluemountain]

After adding the tag codes , it is looking great. Please let me know how to add them and coming to code.

As you said, there is nothing in the value list. I provided the value list that need to be searched, it is throwing an error msg saying the data i provided in the value list is not defined. I tried defining it at the starting of the script, but din fetch me anythin, still error. Please help me

Code:
import re
int = MIN1, MIN2, MIN3

def parse_data(fn, TagID, hdrkeys):
lineList = [item.strip() for item in open(fn).readlines()]
pattH = re.compile(r'%s' % ('|'.join(hdrkeys)))
# check first three lines for header key words
hdrList = pattH.findall(''.join(lineList[:3]))
# create pattern to extract values
pattD = re.compile(r'=(\d+)')
valueList = []
for item in lineList[3:]:
# if textID is in list item, extract the value
if TagID in item:
valueList.append(pattD.search(item).group(1))
# Create output string by joining formatted data
return '\n'.join([','.join(hdrList), 'TagID: %s: sum of (%s) = %s' % \
(TagID, ','.join(MIN1,MIN2,MIN3), \
sum([int(i) for i in valueList]))
])

if __name__ == '__main__':
fn = r'c:\log1.txt'
hdrkeys = ['53301', 'RS485', 'Write']
TagID = '504181'
print parse_data(fn, TagID, hdrkeys)

Error:

Traceback (most recent call last):
File "C:\Python24\file_tagid.py", line 2, in -toplevel-
int = MIN1, MIN2, MIN3
NameError: name 'MIN1_13_5' is not defined
>>>

Iam pretty sure the tag is id there in the text file
Thanks

[bvdet]

After adding the tag codes , it is looking great. Please let me know how to add them and coming to code.

As you said, there is nothing in the value list. I provided the value list that need to be searched, it is throwing an error msg saying the data i provided in the value list is not defined. I tried defining it at the starting of the script, but din fetch me anythin, still error. Please help me

Code:
import re
int = MIN1, MIN2, MIN3

def parse_data(fn, TagID, hdrkeys):
lineList = [item.strip() for item in open(fn).readlines()]
pattH = re.compile(r'%s' % ('|'.join(hdrkeys)))
# check first three lines for header key words
hdrList = pattH.findall(''.join(lineList[:3]))
# create pattern to extract values
pattD = re.compile(r'=(\d+)')
valueList = []
for item in lineList[3:]:
# if textID is in list item, extract the value
if TagID in item:
valueList.append(pattD.search(item).group(1))
# Create output string by joining formatted data
return '\n'.join([','.join(hdrList), 'TagID: %s: sum of (%s) = %s' % \
(TagID, ','.join(MIN1,MIN2,MIN3), \
sum([int(i) for i in valueList]))
])

if __name__ == '__main__':
fn = r'c:\log1.txt'
hdrkeys = ['53301', 'RS485', 'Write']
TagID = '504181'
print parse_data(fn, TagID, hdrkeys)

Error:

Traceback (most recent call last):
File "C:\Python24\file_tagid.py", line 2, in -toplevel-
int = MIN1, MIN2, MIN3
NameError: name 'MIN1_13_5' is not defined
>>>

Iam pretty sure the tag is id there in the text file
Thanks

See reply guidelines for code tags. It looks like your problem happens with the assignment of int (int is a built-in function, and you should not use int as a variable name) to variables MIN1, MIN2, MIN3. Maybe you should post part of your data file.