remove duplicates from file

[topala1984]

hello all

excuse my english, I`m a beginer in java and I want to remove duplicate cities + rearrenging file`s lines elements from one 150 mb file, in my file I have that:

"68586240","685 86367","IRVING"
"68586368","685 86431","DENTON"
"68586432","685 86495","DENTON"

and I want to become

"68586240","685 86367","IRVING"
"68586368","685 86495","DENTON"

i mean

1 2 x
3 4 x
5 6 y

become

1 4 x
5 6 y

I have allready done the program but i`m stuck . can anyone help me?

 
import java.util.regex.*;
import java.io.*;
 
    public class test {
 
        private static final String REGEX = ",";
        private static Pattern pattern;
 
         public static void main(String[] argv) {
 
             pattern = Pattern.compile(REGEX);
 
            try{
                FileInputStream fstream = new FileInputStream("test.txt");
                DataInputStream in = new DataInputStream(fstream);
                BufferedReader br = new BufferedReader(new InputStreamReader(in));
                String strLine;
 
                int i = 1;
 
 
                BufferedWriter make_file = new BufferedWriter(new FileWriter("file_temp.txt", true));
                String make_test = "\"TEST\",\"TEST\",\"TEST\"";
                make_file.write(make_test);
                make_file.close();
 
 
                BufferedWriter out = new BufferedWriter(new FileWriter("good.txt", true));                
 
                while ((strLine = br.readLine()) != null)   {
 
                    //explode line                    
                    String[] items = pattern.split(strLine);
 
                    // read + memory content from temp file    
                    FileReader input = new FileReader("file_temp.txt");
                    BufferedReader bufRead = new BufferedReader(input);
                       String line = ""; 
                    String temp = "";    
                    line = bufRead.readLine();
                    while (line != null){
                        temp = temp+line;
                        line = bufRead.readLine();                
                    }
                    bufRead.close();
 
                    // explode line from temp file
                    String[] items_temp = pattern.split(temp);
 
                    //delete temp file
                    new    File("file_temp.txt").delete();
 
                    BufferedWriter out_temp = new BufferedWriter(new FileWriter("file_temp.txt", true));
                    out_temp.write(strLine);
                    out_temp.close();
 
                    String x1 = items_temp[2];
                    String x2 = items[2];
 
                    if (x1.equalsIgnoreCase(x2)) {
                        // ??????????
                    } 
                    else {
                        out.write(items[0]+","+items_temp[1]+","+items[2]+"\n");
                        /* 
                         input   output  correct 
                         1 2 x   1 2 x   1 4 x
                         3 4 x   5 6 y   5 6 y
                         5 6 y
                        */
 
                    }
 
                    System.out.println(i+"-"+strLine);            
                    i++;
 
                }
                 out.close();
                in.close();
            }
            catch (Exception e){
                System.err.println("Error: " + e.getMessage());
            }    
 
    }
 
}
 
 

tnx

[dmjpro]

Sorry for less time in my hand to look after your code throughly :-)
Have a look at my code ....

BufferedReader in = new BufferedReader(new FileReader("src_file"));
FileWriter out = new FileWriter("tgt_file");
String line;
HashSet hs = new HashSet();
String op = "";
while(!(line=in.readLine())!=null){
StringTokenizer s = new StringTokenizer(line,",");
String city;
while(s.hasMoreTokes()) city = s.nextToken();
op += hs.add(city) ? (line+System.getProperty("line.separator")) : "";
//here it ignores duplicate items
}
out.write(op);
in.close();
op.close();
 

Enjoy this code.
Good Luck :-)

Kind regards,
Dmjpro.

[r035198x]

Sorry for less time in my hand to look after your code throughly :-)
Have a look at my code ....

BufferedReader in = new BufferedReader(new FileReader("src_file"));
FileWriter out = new FileWriter("tgt_file");
String line;
HashSet hs = new HashSet();
String op = "";
while(!(line=in.readLine())!=null){
StringTokenizer s = new StringTokenizer(line,",");
String city;
while(s.hasMoreTokes()) city = s.nextToken();
op += hs.add(city) ? (line+System.getProperty("line.separator")) : "";
//here it ignores duplicate items
}
out.write(op);
in.close();
op.close();
 

Enjoy this code.
Good Luck :-)

Kind regards,
Dmjpro.

@OP Do not use InputStreams to read text files. Use FileReader or Scanner
@dmjpro Do not use StringTokenizer

[JosAH]

@OP: what should be done with the following file contents:

1 2 x
3 4 x
5 6 x
 

or with this?

1 2 x
3 4 y
5 6 x
 

I think the problem description should be less ambiguous and more complete
to start with before we can start thinking of a solution.

kind regards,

Jos

[topala1984]

more info

input :
1 2 x
3 4 x
5 6 y

output :
1 2 x
5 6 y

correct:
1 4 x
5 6 y

compare the third element of line 1 with the third element of line 2.
if equal join first element from line 1 with second element from line 2 and the third comun element in ... all in new file
if not equal put line 2 in new file

compare the third element of line 2 with the third element of line 3.
if equal join first element from line 2 with second element from line 3 and the third comun element ... all in new file
if not equal put line 3 in new file
............... ..............

compare the third element of line line n-1 with the third element of line n.
if equal join first element from line n-1 with second element from line n and the third comun element ... all in new file
if not equal put line n in new file

tnx all

[dmjpro]

more info

input :
1 2 x
3 4 x
5 6 y

output :
1 2 x
5 6 y

correct:
1 4 x
5 6 y

compare the third element of line 1 with the third element of line 2.
if equal join first element from line 1 with second element from line 2 and the third comun element in ... all in new file
if not equal put line 2 in new file

compare the third element of line 2 with the third element of line 3.
if equal join first element from line 2 with second element from line 3 and the third comun element ... all in new file
if not equal put line 3 in new file
............... ..............

compare the third element of line line n-1 with the third element of line n.
if equal join first element from line n-1 with second element from line n and the third comun element ... all in new file
if not equal put line n in new file

tnx all

Did you try my code :-)

Debasis Jana

[JosAH]

Did you try my code :-)

Debasis Jana

I noticed that you're spoonfeeding quite a bit of (incorrect) code lately; please don't do that.

Jos