I am pulling some data from a source via ODBC and placing the information
in a DataSet. The first pull is very large but once that is complete I plan
to do nightly pulls to get any new data that gets put in the (remote) table.
I can't seem to get past that initial (big) pull of data, as I get OutOfMemory
exceptions. I took a look back at when this occurs and it seems to happen upon
stuffing the DataSet with either the 2097153 record or the 4194305 record.
My question is, is there a physical limitation to a Dataset as these numbers
are fairly consistent numbers? BTW, I have upgraded the server to have 4GB
of memory and have two swap files with 4MB's each. I suspect this would be
enough memory? Perhaps the swap files aren't in play/use when appending
data to a DataSet?
So the key question is if I can do anything to allow the DataSet to store more
memory? I have probably done as much as I can to give as much memory to the
OS. (I am using V2.0.50727.42 of .NET)....
2  7870 
Peter S. wrote:
I am pulling some data from a source via ODBC and placing the
information
in a DataSet. The first pull is very large but once that is complete
I plan to do nightly pulls to get any new data that gets put in the
(remote) table.
I can't seem to get past that initial (big) pull of data, as I get
OutOfMemory exceptions. I took a look back at when this occurs and it
seems to happen upon stuffing the DataSet with either the 2097153
record or the 4194305 record.
My question is, is there a physical limitation to a Dataset as these
numbers are fairly consistent numbers? BTW, I have upgraded the
server to have 4GB
of memory and have two swap files with 4MB's each. I suspect this
would be enough memory? Perhaps the swap files aren't in play/use
when appending
data to a DataSet?
So the key question is if I can do anything to allow the DataSet to
store more memory? I have probably done as much as I can to give as
much memory to the OS. (I am using V2.0.50727.42 of .NET)....
AFIAK, there's no specific limit to DataSet per-se - you're likely facing
the overall limit of the 2Gb user address space available to programs under
Windows.
A couple of things that should help:
1. Change the boot.ini of the server to include the /3Gb switch and mark
your executable as Large Address Aware. This decreases the amount of memory
available to the kernel, so you might not want to leave it like this, but it
could get you past this first hurdle.
2. Move to a 64-bit machine.
3. Change your design to pull the data over in smaller chunks.
-cd
Excellent suggestions, thanks! My latest tack is to bail out when I get
the OutOfmemory exception and squirrel away what I got up to that
point. However I certainly want to minimize this and will try your
suggestions!
Thanks!
"Carl Daniel [VC++ MVP]" wrote:
Peter S. wrote:
I am pulling some data from a source via ODBC and placing the
information
in a DataSet. The first pull is very large but once that is complete
I plan to do nightly pulls to get any new data that gets put in the
(remote) table.
I can't seem to get past that initial (big) pull of data, as I get
OutOfMemory exceptions. I took a look back at when this occurs and it
seems to happen upon stuffing the DataSet with either the 2097153
record or the 4194305 record.
My question is, is there a physical limitation to a Dataset as these
numbers are fairly consistent numbers? BTW, I have upgraded the
server to have 4GB
of memory and have two swap files with 4MB's each. I suspect this
would be enough memory? Perhaps the swap files aren't in play/use
when appending
data to a DataSet?
So the key question is if I can do anything to allow the DataSet to
store more memory? I have probably done as much as I can to give as
much memory to the OS. (I am using V2.0.50727.42 of .NET)....
AFIAK, there's no specific limit to DataSet per-se - you're likely facing
the overall limit of the 2Gb user address space available to programs under
Windows.
A couple of things that should help:
1. Change the boot.ini of the server to include the /3Gb switch and mark
your executable as Large Address Aware. This decreases the amount of memory
available to the kernel, so you might not want to leave it like this, but it
could get you past this first hurdle.
2. Move to a 64-bit machine.
3. Change your design to pull the data over in smaller chunks.
-cd
This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics | |
by: Bill C. |
last post by:
Hi,
I've got a simple console app that just reads an XML file into a DataSet
then prints out a description of each table in the DataSet, including
column names and row values for each column. I'm getting some strange
results depending the input XML file I use. I was wondering if somebody
could help me understand what is going on or point me to a good reference.
The code for my program looks like this:
|
by: Mike Schilling |
last post by:
Instances of SystemOutOfMemoryException do not contain a stack trace. Easy
test to verify this:
class OOM
{
public static void Main() {
try {
Object arr = new Object;
} catch (Exception ex) {
dumpEx(ex);
|
by: J |
last post by:
I'm experimenting with strongly typed datasets, and when the .Fill method is
called, the RowChanging fires for each row. This causes a problem when the
existing data does not adhere to the rules outlined in the RowChanging
method. Is there an easy way to not have the RowChanging event occur when
the dataset is being filled?
Example code is
daTitleAuthor.Fill(dsAuthors1);
|
by: Krzysztof Karnicki |
last post by:
I have such a problem… I have create my custom DataGridColumn inheriting from
System.Windows.Forms.DataGridColumnStyle on using it on DataGrid, to show
rows painted by me self. Because dates taken from database are very large,
and filling DataSet bounded to this DataGrid takes some time, a decided to
put fill logic into new thread. And so method running in this thread clears
only this DataSet bounded to DataGrid and fills it once more...
|
by: Tobias Matzat |
last post by:
Hello!
I'm writing an application that immitates ie behavior to save
html-pages. Well fecht the files in the webpage and change the paths in
the href/src attributes. Well, I#m doing this with the help of the RegEx
class. If the page contains a lot of files to fetch, I'm getting an
System.OutOfMemory Exception when changing something with the
RegEx.Replace(...). Taking a look in my taskmanager shows me that all
off my system memory is...
| | |
by: Elena |
last post by:
I am trying to fill a DataSet using the code at the bottom
of this message. At the Fill line, I get an error:
An unhandled exception of
type 'System.Data.OleDb.OleDbException' occurred in
system.data.dll
I use this same code to fill DataSets, with no error, in
another section of the same form.
I have no idea what is causing this error.
|
by: gillcleeren |
last post by:
I have a datagrid that has to be filled in a WinForms application. The
data comes from an Oracle database.
The normal procedure would be filling a dataset, but I have the problem
that there are over 100.000 records in the tables, so this has very low
performance when executing.
I then tried using an OracleDataReader, but for some reason, I cannot
bind my reader to the datagrid, I get the following error:
"Complex DataBinding accepts...
|
by: Derek Vincent |
last post by:
Why am I having problems filling my dataset when I select fields of type
varchar(350) from interbase? I can fill the dataset when I select every
other field in the table but I receive and error message that says "object
reference not set to an instance of an object" when I return the varchar
field." The detailed error message is below.
Server Error in '/BIC_WEB_TEST' Application....
|
by: JJA |
last post by:
I am confused about all the various methods of handling XML in ASP.Net
2.0. I want to be able to stream XML to some clientside Javascript code
(snippet below):
var urlRequest = "../GetLendersAsXML.aspx"
var request = GXmlHttp.create();
request.open("GET", urlRequest, true);
request.onreadystatechange = function() {
if (request.readyState == 4) {
var xmlDoc = request.responseXML;
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
| | |
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
| | |
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
| |
