By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
454,650 Members | 1,388 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 454,650 IT Pros & Developers. It's quick & easy.

which api/method to use for concatenating multiple large files using least RAM?

P: 1
Hi

I have multiple files (between 50-100) that basically contain simple text. Each file ranges in size from few MB to 50MB.

I have to concatenate all these files to produce one single file that contains the content of all the files-- and I have to do this in a machine that has quite less RAM-- I can use max 512MB RAM for the JVM.

How do I do this? Which API/method do I use to accomplish concatenation of multiple files while using least values?
Jan 4 '13 #1
Share this Question
Share on Google+
2 Replies


P: 3
try this
Expand|Select|Wrap|Line Numbers
  1. public void concatFile() {
  2.         PrintWriter pw = new PrintWriter(new FileOutputStream("C:/temp/out/out.txt"));
  3.         File file = new File("C:/temp/");
  4.         File[] files = file.listFiles();
  5.         for (int i = 0; i < files.length; i++) {
  6.                 BufferedReader br = new BufferedReader(new FileReader(files[i].getPath()));
  7.                 String line = br.readLine();
  8.                 while (line != null) {
  9.                         pw.println(line);
  10.                         line = br.readLine();
  11.                 }
  12.                 br.close();
  13.         }
  14.         pw.close();
  15. }
  16.  
simply read line by line, each file
Jan 4 '13 #2

100+
P: 185
You may leverage the operating system capabilities to implement this task. Windows copy command can do this utlra fast.
For example, if you have 4 files to concatenate (T1.txt... T4.txt) into one file T5.txt, the following command can make it for you.

Expand|Select|Wrap|Line Numbers
  1. copy /B T1.txt+T2.txt+T3.txt+T4.txt T5.txt
Linux version would be:
Expand|Select|Wrap|Line Numbers
  1. cat T1.txt T2.txt T3.txt T4.txt > T5.txt
I'm expecting you to know how to execute OS commands from Java.

Good luck.
Jan 5 '13 #3

Post your reply

Sign in to post your reply or Sign up for a free account.