469,270 Members | 1,011 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,270 developers. It's quick & easy.

how to read large CSV file

228 100+
hi everyone
i'm trying to read data from CSV file file is about 1MB (or bigger)
i figured out how to read it quickly from the disc but it takes 3000 milliseconds to change it into arrays so my question is
is there a faster way of doing it?
thats my class:
Expand|Select|Wrap|Line Numbers
  1. public class ReadWrite {
  2.  
  3.     private static String filename = "myFile.csv";
  4.     private static File file;
  5.     private static Scanner in;
  6.     private static String str="";
  7.     private int[] time =     new int[100000];
  8.     private float[] value =  new float[100000];
  9.     private String [] date = new String[100000];
  10.     private String[] update =new String[100000];
  11.  
  12.     public static int count = 0;
  13.     public boolean hasNext = true;
  14.  
  15.     static private String str2="";
  16.  
  17.     /** Creates a new instance of ReadWrite */
  18.     public ReadWrite() throws FileNotFoundException, IOException {
  19.         file = new File(filename);
  20.         fileReader = new FileReader(file);
  21.  
  22.         readFile();
  23.  
  24.         in = new Scanner(this.str);
  25.         in.useDelimiter("(\t|\n)");
  26.  
  27.     }
  28. // thats where the problems are this method is to slow
  29. // slower then PHP :/  
  30. //(in php i would use explode("\n",str) and explode("\t",str)) 
  31.     public void read(){
  32.         while(in.hasNextInt()){
  33.             time[count] = in.nextInt() ;
  34.             value[count] = in.nextFloat();
  35.             date[count] = in.next();
  36.             update[count] = in.next();
  37.             count++;
  38.         }
  39.     }
  40.  ------------------------------------end of problems -----------------------------
  41.     public String toString(int i){
  42.         return time[i] + " " +value[i] +" "+ date[i];
  43.     }
  44.  
  45.     FileReader fileReader;
  46.     void readFile() throws IOException{
  47.         while(hasNext){
  48.             char [] c = new char[4096];
  49.             if(fileReader.read(c) > -1 ){
  50.                 str += String.valueOf(c);
  51.             }else{
  52.                 hasNext = false;                
  53.             }
  54.         }
  55.     }
  56.  
  57.  
  58. }
  59.  
Feb 3 '08 #1
4 8841
BigDaddyLH
1,216 Expert 1GB
I don't know about the speed of code, but first copying the entire file into a string is a whacky thing to do. Why not just read from the file?
Expand|Select|Wrap|Line Numbers
  1. in = new Scanner(new File(filename));
Also, your use of static fields needs to be rethought, but on another day.

Finally, why are you loading the entire file into memory at once? Can you avoid this?
Feb 4 '08 #2
jx2
228 100+
I don't know about the speed of code, but first copying the entire file into a string is a whacky thing to do. Why not just read from the file?
Expand|Select|Wrap|Line Numbers
  1. in = new Scanner(new File(filename));
Also, your use of static fields needs to be rethought, but on another day.

Finally, why are you loading the entire file into memory at once? Can you avoid this?
actualy i used scanner but it was the slowest one
perhaps i 'm missing something importent
Expand|Select|Wrap|Line Numbers
  1.  str = in.nextLine(); 
is there a better way to read whole file in ( i mean faster way)
Feb 5 '08 #3
BigDaddyLH
1,216 Expert 1GB
There are libraries for working with CSV files. Google for them.
Feb 5 '08 #4
rather than use str += String.valueOf(c) try using a StringBuffer and use the append.

something like:
sb.append(String.valueOf(c))
Feb 20 '08 #5

Post your reply

Sign in to post your reply or Sign up for a free account.

Similar topics

11 posts views Thread by Sebastian Krause | last post: by
75 posts views Thread by Greg McIntyre | last post: by
1 post views Thread by David Arden Stevensonn | last post: by
6 posts views Thread by Rolf Schroedter | last post: by
35 posts views Thread by RyanS09 | last post: by
6 posts views Thread by comp.lang.php | last post: by
3 posts views Thread by =?Utf-8?B?ZGF2aWQ=?= | last post: by
2 posts views Thread by Kevin Ar18 | last post: by
reply views Thread by zhoujie | last post: by
reply views Thread by suresh191 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.