473,382 Members | 1,084 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,382 software developers and data experts.

How do you read large text files in Java >20MB

N002213F
I have a Java text reader program that can easily read small text, but seems fails to read larger files >20MB. It seems to truncate the contents it reads.

I there a way of making sure all the data is read.

Thanks in advance.
Oct 16 '08 #1
6 9140
r035198x
13,262 8TB
If you are sure there are no inefficiencies with the code, try using the classes in the nio package. They try to speed up things by using the OS itself.
Oct 16 '08 #2
Some sample code, hope its helps;


Expand|Select|Wrap|Line Numbers
  1. int startPos = "2008-10-15 13:00:00,640 ".length();
  2. BufferedReader br = new BufferedReader(new FileReader("c:\\temp\\test-prices_bulk.log"));
  3. PrintWriter pw = new PrintWriter("c:\\temp\\test-prices_NASDAQ.log");
  4. String line = null;
  5. while ((line = br.readLine()) != null) {
  6.     int endPos = line.indexOf("|");
  7.     if(endPos == -1 || endPos < startPos){
  8.         System.out.println("| not found or occurs earlier that expected.");
  9.         continue;
  10.     }
  11.  
  12.     String contractCode = line.substring(startPos, endPos);
  13.     if(contractCode.indexOf("NASDAQ") != -1){
  14.         pw.println(line);
  15.     }
  16. }
  17. pw.close();
  18.  
Oct 16 '08 #3
Found the problem, but please promise you won't yell at me. I hadn't closed the PrintWriter.

The maximum file size a Java program can read is dependent on the maximum memory you allocate to it when its starts. ~20MB is fairly small considering the default is 64MB
Oct 16 '08 #4
r035198x
13,262 8TB
Found the problem, but please promise you won't yell at me. I hadn't closed the PrintWriter.

The maximum file size a Java program can read is dependent on the maximum memory you allocate to it when its starts. ~20MB is fairly small considering the default is 64MB
<Yelling>
But your line 17 ...
</Yelling>
Oct 16 '08 #5
@WideEyed
Surprisingly i added it on the notepad when i was assembling the sample code, there is a lot that occurrs in between and didn't want to include it.

@Aha
I guess i should just stick to the notepad.
Oct 16 '08 #6
@I_Know_these_guys_from_somewhere
This mistake reminds me of "No Problem"!!

that is the effect of copy and paste sipatha! But nice going.
Oct 17 '08 #7

Sign in to post your reply or Sign up for a free account.

Similar topics

6
by: Matt Liverance | last post by:
I REALLY dont want to switch to oracle :( but I cant get these tables working any faster. I've got 2 dedicated servers, each with a slave, all run 32gig 15k rpm raid 5 on u320 perc raid...
4
by: Michi | last post by:
I was wondering what the best solution is for making large numbers of TEXT (or BLOB?) fields searchable. For example, if I have a forum, what is the best way to be able to search for specific...
3
by: Chris | last post by:
When you have to read a big file (5-30MB) and throw the data into the database, ofcourse some logics inbetween (doesn't matter) which of the ADO methods is recommended. 1. read line by line and do...
11
by: Ron Vecchi | last post by:
I've used System.Web.Mail before but have never had the need to send attchemnets through it...until now. A client of mine would like a form on the website to allow a user to type up a message and...
1
by: bongoo | last post by:
I'm reading a 20Mb xml file in vb.net using .net framework 1.1 But when I'm opening that file for the second time, memory consuption goes wild... Do I need to take newer .net framework, or to...
23
by: ShaneO | last post by:
Hello, I wish to extract embedded string data from a file using a Binary Read method. The following code sample is used in VB.NET and similar code is used in VB6 - (Assume variable...
3
by: toton | last post by:
Hi, I have some ascii files, which are having some formatted text. I want to read some section only from the total file. For that what I am doing is indexing the sections (denoted by .START in...
10
by: =?Utf-8?B?RGFu?= | last post by:
I'm using a pure ASP upload script to handle file uploading. We run IIS6 and I'm aware of the AspMaxRequestEntityAllowed variable in the metabase. I upped this to allow the file size maximum and it...
1
by: Mohamed Yousef | last post by:
Hello , let's say , I'm moving large files through network between devices what is the fastest way to do this ? what i came up with :- 1) using regular file operations with an in memory limit...
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
by: ryjfgjl | last post by:
In our work, we often need to import Excel data into databases (such as MySQL, SQL Server, Oracle) for data analysis and processing. Usually, we use database tools like Navicat or the Excel import...
0
by: taylorcarr | last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
0
by: aa123db | last post by:
Variable and constants Use var or let for variables and const fror constants. Var foo ='bar'; Let foo ='bar';const baz ='bar'; Functions function $name$ ($parameters$) { } ...
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.