473,388 Members | 924 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,388 software developers and data experts.

ResultSet limit

9
Hi,

This is vinod.

i have tried to read a 10 lacks records with column count 14 from the database at a single time. But my application shows OUT OF MEMORY error.

Any help?

my id : <id removed per posting guidelines>

Thanks,
Vinod
Aug 16 '07 #1
15 8712
JosAH
11,448 Expert 8TB
Hi,

This is vinod.

i have tried to read a 10 lacks records with column count 14 from the database at a single time. But my application shows OUT OF MEMORY error.

Any help?

my id : <id removed as per posting guidelines>

Thanks,
Vinod
How much is 'a 10 lacks'?

kind regards,

Jos
Aug 16 '07 #2
pvinod
9
10,00,000


Thanks,
vinod
Aug 16 '07 #3
JosAH
11,448 Expert 8TB
10,00,000


Thanks,
vinod
So that's one million records with 14 columns each? It depends a bit on the length
of the columns but I can see that you get an OutOfMemoryError ... there are a
few -X flags that increase the JVM's heap size but I think that's just postponing
the execution. What do you want to load that many records in memory at once?

kind regards,

Jos
Aug 16 '07 #4
pvinod
9
So that's one million records with 14 columns each? It depends a bit on the length
of the columns but I can see that you get an OutOfMemoryError ... there are a
few -X flags that increase the JVM's heap size but I think that's just postponing
the execution. What do you want to load that many records in memory at once?

kind regards,

Jos
Thanks for ur response Jos.

I need to import data from one database to another database. Please give me some way to avoid this Outof Memory error.

Regards,
Vinod
Aug 16 '07 #5
JosAH
11,448 Expert 8TB
Thanks for ur response Jos.

I need to import data from one database to another database. Please give me some way to avoid this Outof Memory error.

Regards,
Vinod
You might try to specify the Statement.setFetchSize() on your Statement

kind regards,

Jos
Aug 16 '07 #6
pvinod
9
You might try to specify the Statement.setFetchSize() on your Statement

kind regards,

Jos

I have tried this setFetchSize() method. But I got error message(Invalid fetch size).


Thanks,
Vinod
Aug 16 '07 #7
JosAH
11,448 Expert 8TB
I have tried this setFetchSize() method. But I got error message(Invalid fetch size).

Thanks,
Vinod
So what happens if you experiment a bit with different fetch sizes?

kind regards,

Jos
Aug 16 '07 #8
pvinod
9
So what happens if you experiment a bit with different fetch sizes?

kind regards,

Jos
Hi,
I got the following error.

rs.setFetchSize(50);

java.sql.SQLException: Invalid Fetch Size


Thanks,
Vinod
Aug 16 '07 #9
JosAH
11,448 Expert 8TB
Hi,
I got the following error.

rs.setFetchSize(50);

java.sql.SQLException: Invalid Fetch Size


Thanks,
Vinod
Can you show us a bit of relevant code please? What database are you using?

kind regards,

Jos
Aug 17 '07 #10
praveen2gupta
201 100+
Hi
The ResultSet is having a limit of 1000 results. You can not access unlimited number of records from a singlr sql query. So you are getting errors
Aug 17 '07 #11
JosAH
11,448 Expert 8TB
The ResultSet is having a limit of 1000 results.
That all depends on the database vendor and its JDBC driver.

kind regards,

Jos
Aug 17 '07 #12
pvinod
9
That all depends on the database vendor and its JDBC driver.

kind regards,

Jos

Hi,

I have tried to read 3,00,000 records. I was able to do this.

Please advise. Is any other way to perform this.

Thanks,
Vinod
Aug 17 '07 #13
JosAH
11,448 Expert 8TB
Hi,

I have tried to read 3,00,000 records. I was able to do this.

Please advise. Is any other way to perform this.

Thanks,
Vinod
I don't know if you followed an other identical thread but my observations are:

1) users are not willing to scroll through so much data. They want it in reasonable
chunks through which they can navigate.
2) you can't collect that much database because of memory constraints.

So my tip is to 'page' through all the data. Depending on your database and its
SQL dialect you might check the 'offset/limit' statement parts or the 'numrows'
statement part or whatever your database supports.

A few buttons for navigational purposes and just a bit of data in the Session
object can do the rest. Don't try to retrieve and show all the data in one piece.

kind regards,

Jos
Aug 17 '07 #14
pvinod
9
I don't know if you followed an other identical thread but my observations are:

1) users are not willing to scroll through so much data. They want it in reasonable
chunks through which they can navigate.
2) you can't collect that much database because of memory constraints.

So my tip is to 'page' through all the data. Depending on your database and its
SQL dialect you might check the 'offset/limit' statement parts or the 'numrows'
statement part or whatever your database supports.

A few buttons for navigational purposes and just a bit of data in the Session
object can do the rest. Don't try to retrieve and show all the data in one piece.

kind regards,

Jos

thanks Jos. Let me try

vinod
Aug 17 '07 #15
Please use this this will help you in storing the bulk data without memory dependency



Statement stmt = con.createStatement(java.sql.ResultSet.TYPE_FORWAR D_ONLY,java.sql.ResultSet.CONCUR_READ_ONLY);
stmt.setFetchSize(Integer.MIN_VALUE);
Mar 16 '15 #16

Sign in to post your reply or Sign up for a free account.

Similar topics

6
by: Steve | last post by:
Hi, I have a web application where there are a number of columns a user can sort on (price, item etc). The database itself is HUGE (around 6 million rows) and even with all the indexing it slows...
3
by: Rainer Collet | last post by:
hi! i tested several php database abstraction layers (db, mdb(2), creole, adodb, etc), but i always missed one really important feature: i need a method for a limited select which gives me the...
0
by: RoyalScorpion | last post by:
hi guys, i get an updatable resultset from a connection then used it to insert a new row but the result set doesn't chane after insertion, i mean the no of rows before insertion is the same after...
0
by: Karam Chand | last post by:
Greetings I have a table with the following table structure - Field Type Collation Null Key Default Extra ------- ------------- ----------------- ------ ------ ...
0
by: Karam Chand | last post by:
Greetings I have a table with the following table structure - Field Type Collation Null Key Default Extra ------- ------------- ----------------- ------ ------ ...
4
by: _link98 | last post by:
Problem: java ResultSet cursor from SQL/PL stored-procedure is FORWARD_ONLY. Is it possible to have ResultSet cursors from SQL/PL procedures to scroll forward and backwards? Perhaps I am missing...
3
by: natG | last post by:
Hi; In db2 8.2, how do I limit a query resultset to x rows? Thanks, nat
7
by: Garth Wells | last post by:
I use a filtered SELECT to populate the SQLDataReader (rdr) with a filename and a blob (pdf). I then use File.WriteAllBytes to write each pdf to disk. ----------------------------------------...
0
Sl1ver
by: Sl1ver | last post by:
That is my code, if i put a updateQuery statement in there, then it updates my database, but when i query the database then it doesn't populate the resultset public ResultSet User(String user) ...
0
by: taylorcarr | last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.