How to remove the duplicate lines retaining first occurences

[helloR]

Let's say a input text file "input_msg.txt" file ( file size is 70,000 kb ) contains following records..

Jan 1 02:32:40 hello welcome to python world
Jan 1 02:32:40 hello welcome to python world
Mar 31 23:31:55 learn python
Mar 31 23:31:55 learn python be smart
Mar 31 23:31:56 python is good scripting language
Jan 1 00:00:01 hello welcome to python world
Jan 1 00:00:02 hello welcome to python world
Mar 31 23:31:55 learn python
Mar 31 23:31:56 python is good scripting language

The expected output file ( Let's say outputfile.txt ) should contain below records...

Jan 1 02:32:40 hello welcome to python world
Jan 1 02:32:40 hello welcome to python world
Mar 31 23:31:55 learn python
Mar 31 23:31:55 learn python be smart
Mar 31 23:31:56 python is good scripting language
Jan 1 00:00:01 hello welcome to python world
Jan 1 00:00:02 hello welcome to python world

Note: I need all the records (including duplicate) which are starting with "Jan 1" and also I don't need Duplicate records not starting with "Jan 1"

I have tried the following program where all the duplicate records are getting deleted.

def remove_Duplicate_Lines(inputfile, outputfile):  
   with open(inputfile) as fin, open(outputfile, 'w') as out:
      lines = (line.rstrip() for line in fin)
      unique_lines = OrderedDict.fromkeys( (line for line in lines if line) )
      out.writelines("\n".join(unique_lines.iterkeys()))
 return 0

Oputput of my program are below:

Jan 1 02:32:40 hello welcome to python world
Mar 31 23:31:55 learn python
Mar 31 23:31:55 learn python be smart
Mar 31 23:31:56 python is good scripting language
Jan 1 00:00:01 hello welcome to python world

Your help would be appreciated!!!

[bvdet]

Use a for loop and conditionally append to a list.

data = """Jan 1 02:32:40 hello welcome to python world
Jan 1 02:32:40 hello welcome to python world
Mar 31 23:31:55 learn python
Mar 31 23:31:55 learn python be smart
Mar 31 23:31:56 python is good scripting language
Jan 1 00:00:01 hello welcome to python world
Jan 1 00:00:02 hello welcome to python world
Mar 31 23:31:55 learn python
Mar 31 23:31:56 python is good scripting language"""
 
output = []
 
for line in data.split("\n"):
    if line.startswith("Jan 1"):
        output.append(line)
    elif line not in output:
        output.append(line)
 
print "\n".join(output)

The output:

>>> Jan 1 02:32:40 hello welcome to python world
Jan 1 02:32:40 hello welcome to python world
Mar 31 23:31:55 learn python
Mar 31 23:31:55 learn python be smart
Mar 31 23:31:56 python is good scripting language
Jan 1 00:00:01 hello welcome to python world
Jan 1 00:00:02 hello welcome to python world
>>> 

[helloR]

@bvdet: Thanks you very much!! I have already been tried this solution but here the problem is if the input message file size is more then it takes more time....

Below is the program which I have tried:

inputFile = open("in.txt", "r")
log = []
for line in inputFile:
    if line in log and line[0:5] != "Jan 1":
        pass
    else:
        log.append(line)
inputFile.close()
outFile = open("out.txt", "w")
for item in log:
    outFile.write(item)
outFile.close()
 

Note: I have tried with input file size as ~70000 kb and it takes ~9 minutes to complete the execution.

Pls let me know if we can do it some elegant way.....

[bvdet]

Try writing to the file one time.

outFile.write("\n".join(log))

[helloR]

@bvet: You mean something like below:

inputFile = open("in.txt", "r")
outFile = open("out.txt", "w")
log = []
for line in inputFile:
   if line in log and line[0:5] != "Jan 1":
      pass
   else:
      log.append(line)
   outFile.write("\n".join(log))
inputFile.close()
outFile.close()
 

Pls correct me if i am wrong.

[bvdet]

No, write to the file outside of the for loop:

outFile.write("\n".join(log))
inputFile.close()
outFile.close()

[helloR]

@bvdet: Thank you for your help!!! This is working but still there is performance issue whenever input file size is more....

Could you please take a look into below code...

def remove_Duplicate_Lines(inputfile, outputfile):
   with open(inputfile) as fin, open(outputfile, 'w') as out:
      lines = (line.rstrip() for line in fin)
      unique_lines = OrderedDict.fromkeys( (line for line in lines if line) )
      out.writelines("\n".join(unique_lines.iterkeys()))
   return 0
 

[bvdet]

You are iterating over the lines in the file twice. Try eliminating one of them. It is possible OrderedDict may be slower than a for loop. I don't know one way or the other. You can use module timeit to check different methods.