473,394 Members | 1,696 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,394 software developers and data experts.

Large Data handle with Vector!

dmjpro
2,476 2GB
Actually when I trying to put 50000+ data into Vector then it gets out of memory.
How should I handle with it efficiently without change the Java Heap Space?


Debasis Jana
Mar 29 '08 #1
5 2072
JosAH
11,448 Expert 8TB
Actually when I trying to put 50000+ data into Vector then it gets out of memory.
How should I handle with it efficiently without change the Java Heap Space?


Debasis Jana
50K elements isn't that much if each element isn't that big. If you do want to
change the heap space start "java -X" for help on the X options. (one of them
changes the maximum heap space) but I doubt whether you really need it;
I smell that the OOME is caused by something else.

kind regards,

Jos
Mar 29 '08 #2
dmjpro
2,476 2GB
50K elements isn't that much if each element isn't that big. If you do want to
change the heap space start "java -X" for help on the X options. (one of them
changes the maximum heap space) but I doubt whether you really need it;
I smell that the OOME is caused by something else.

kind regards,

Jos

Vector.addElement gets this Error.
Sorry Josh no of Elements i mentioned wrong.
The actuall data is 1404648.
Now what I am doing is copying the the table values into the Vetor from a ResultSet.
That's why I getting this error.
Without changing the java heap space could not I get the fine tuning over Vector?
One more thing Josh .... Then how ResultSet APIs handle with this large amount of Data?

Debasis Jana
Mar 29 '08 #3
JosAH
11,448 Expert 8TB
Vector.addElement gets this Error.
Sorry Josh no of Elements i mentioned wrong.
The actuall data is 1404648.
Now what I am doing is copying the the table values into the Vetor from a ResultSet.
That's why I getting this error.
Without changing the java heap space could not I get the fine tuning over Vector?
One more thing Josh .... Then how ResultSet APIs handle with this large amount of Data?

Debasis Jana
Ah, ok: you're talking abourt +- 30 times as much data now. It depends on the
size of the individual objects, do your math. You could use the flag -Xmx:1G
if you want but why are you loading an entire table in memory? What's the use
of it? Your dbms can give you the wanted records; don't fetch just all of them,
that's what databases are for. btw don't forget to close your ResultSet after
you're done with it.

kind regards,

Jos
Mar 29 '08 #4
dmjpro
2,476 2GB
Ah, ok: you're talking abourt +- 30 times as much data now. It depends on the
size of the individual objects, do your math. You could use the flag -Xmx:1G
if you want but why are you loading an entire table in memory? What's the use
of it? Your dbms can give you the wanted records; don't fetch just all of them,
that's what databases are for. btw don't forget to close your ResultSet after
you're done with it.

kind regards,

Jos

Josh I able to do that.
I used ResultSet instead of using Vector.
And then all the MetaData populated from there insert into a target table.

Debasis Jana.
Mar 29 '08 #5
chaarmann
785 Expert 512MB
Actually when I trying to put 50000+ data into Vector then it gets out of memory.
How should I handle with it efficiently without change the Java Heap Space?


Debasis Jana
If you want to store integers for example, you should not use Vector() and then add elements by adding new Integer() objects to it. It wastes a lot of time for inboxing-outboxing and memory space because of storing whole objects.
The best way for this is to use arrays of primitive types, like int[] array. This is much faster and uses up significant less memory space. You already know the number of elements to store, so initializing the array to the correct size should be no problem. There is no need to have a "dynamically growable" array, such as Vector().

You can also use compression techniques if you need to store Strings. Just use String.intern() for example before storing it into your Vector (or better String[] array). Usually the strings that you get back from your resultset are all using up memory, even if they are the same. By using String.intern(), you will store each different String only once and for each additional string which is the same you store only the reference instead of the string itself. This takes much less memory.

If you still have problems, then you can use a compression technique like Winzip is using.
Mar 29 '08 #6

Sign in to post your reply or Sign up for a free account.

Similar topics

0
by: Man | last post by:
Hello, I have a problem with SOAPpy and large data, ex: ********** SOAP server*********** def SOAPservice(self): objListReturn = for i in range(2000): objListReturn.append()
0
by: Roland Hall | last post by:
I'm looking for information on working with large data files using FSO, XML. I have a program which creates a large CSV file, over 7mb. It's a rate table of freight shipping costs. There are...
3
by: chandra.somesh | last post by:
Hi I am having problem indexing large amount of textual data(in range of 200,000 to 1 million). I tried using map container but the compiler just hanged upon entering the data. On bit of...
0
by: 詹亚军 | last post by:
can activex control export large data to container by event's para ? I develop a activex control,and want to export large data to container(IE),web page can use javascript to capture the...
16
by: a | last post by:
Would anybody please tell me how to handle array data of int 2000 * 2000? Even make file passes by using a supercomputer, it seems calloc fails and segmentation fault results during a trial run.
1
by: =?Utf-8?B?SHVzYW0=?= | last post by:
Hi EveryBody: I have the following code to download the data from my web application: the code is: Dim rawan As New List(Of String) ' for File Name Dim rawan1 As New List(Of Byte()) ' for...
7
by: FeelLikeANut | last post by:
I'm using the example on the following page for selecting and fetching a large data object. <http://publib.boulder.ibm.com/infocenter/db2luw/...
6
by: Patrick Sullivan | last post by:
Hello. I will be using some large data sets ("points" from 2 to 12 variables) and would like to use one class for each point rather than a list or dictionary. I imagine this is terribly...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.