Hello,
We running VB.NET application which gets massive amounts of data from SQL
Server, loads data into DataTables, then re-arranges data into tabular
structure and outputs it to a flat file. This is just the nature of
requirements for this application. Business analyst here want certain data
to be "re-horizontalize" and served in a single flat file.
We ran this app many times before on data from different quarters, that
worked without any problems. However, now they want to run this program
against data for the whole last year. We keep getting System.OutOfMem ory
Exception. I realize that it's a huge amount of data that has to be
processed in a dataset..., but is there any quick fixes for this kind of
error, system tweaks (besides more memory). We really don't have time to
redesign this application to handle data any differently at this time. They
need this file yesterday!!! Please help.
By the way; the machine that it's running on has 2gb of RAM,
5  3539 
Just a shot in the dark but be sure you are not running out of disk space.
I remember getting that exception and all that the problem was I was out of
disk space. Probably not it, but wanted to mention it.
Chris
"LP" <lp@a.com> wrote in message
news:um******** ******@TK2MSFTN GP11.phx.gbl... Hello,
We running VB.NET application which gets massive amounts of data from SQL
Server, loads data into DataTables, then re-arranges data into tabular
structure and outputs it to a flat file. This is just the nature of
requirements for this application. Business analyst here want certain data
to be "re-horizontalize" and served in a single flat file.
We ran this app many times before on data from different quarters, that
worked without any problems. However, now they want to run this program
against data for the whole last year. We keep getting System.OutOfMem ory
Exception. I realize that it's a huge amount of data that has to be
processed in a dataset..., but is there any quick fixes for this kind of
error, system tweaks (besides more memory). We really don't have time to
redesign this application to handle data any differently at this time.
They
need this file yesterday!!! Please help.
By the way; the machine that it's running on has 2gb of RAM,
Consider creating a view in SQL Server that gets the data in the correct
format and use a Process object to fire BCP to pull from this view. You will
have to assemble a proper BCP line, but the advantages are as follows:
1. Reduced load on SQL Server
2. No memory overhead issues
3. Much faster than anything you can write in VB.NET
If you are comfortable with running a batch script, you can simply put the
BCP line in a .bat file and run it. Less control, less overhead.
The problem with your application is most likely the 2 GB virtual memory
allocation per application in Windows. You can registry tweak this, but
realize that it could have adverse effects on programs other than yours. BCP
is a better option on a variety of fronts and quite easy to set up from the
command line (using SQL Server books online).
---
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA
*************** ************
Think Outside the Box!
*************** ************
"LP" wrote: Hello,
We running VB.NET application which gets massive amounts of data from SQL
Server, loads data into DataTables, then re-arranges data into tabular
structure and outputs it to a flat file. This is just the nature of
requirements for this application. Business analyst here want certain data
to be "re-horizontalize" and served in a single flat file.
We ran this app many times before on data from different quarters, that
worked without any problems. However, now they want to run this program
against data for the whole last year. We keep getting System.OutOfMem ory
Exception. I realize that it's a huge amount of data that has to be
processed in a dataset..., but is there any quick fixes for this kind of
error, system tweaks (besides more memory). We really don't have time to
redesign this application to handle data any differently at this time. They
need this file yesterday!!! Please help.
By the way; the machine that it's running on has 2gb of RAM,
I just checked over 13gb of disk space. It's powefull Dell Precision machice
with 2gb of RAM, Intel Xeon processor 3.5ghz (i think) used for batch
processing.
"Chris, Master of All Things Insignificant" <chris@No_Spam_ Please.com> wrote
in message news:Ok******** ******@TK2MSFTN GP10.phx.gbl... Just a shot in the dark but be sure you are not running out of disk space.
I remember getting that exception and all that the problem was I was out
of disk space. Probably not it, but wanted to mention it.
Chris
"LP" <lp@a.com> wrote in message
news:um******** ******@TK2MSFTN GP11.phx.gbl... Hello,
We running VB.NET application which gets massive amounts of data from
SQL Server, loads data into DataTables, then re-arranges data into tabular
structure and outputs it to a flat file. This is just the nature of
requirements for this application. Business analyst here want certain
data to be "re-horizontalize" and served in a single flat file.
We ran this app many times before on data from different quarters, that
worked without any problems. However, now they want to run this program
against data for the whole last year. We keep getting System.OutOfMem ory
Exception. I realize that it's a huge amount of data that has to be
processed in a dataset..., but is there any quick fixes for this kind of
error, system tweaks (besides more memory). We really don't have time to
redesign this application to handle data any differently at this time.
They
need this file yesterday!!! Please help.
By the way; the machine that it's running on has 2gb of RAM,
Could you do it in a loop, processing smaller sets of data broken up into
months, and just keep appending to the text file?
"LP" <lp@a.com> wrote in message
news:um******** ******@TK2MSFTN GP11.phx.gbl... Hello,
We running VB.NET application which gets massive amounts of data from SQL
Server, loads data into DataTables, then re-arranges data into tabular
structure and outputs it to a flat file. This is just the nature of
requirements for this application. Business analyst here want certain data
to be "re-horizontalize" and served in a single flat file.
We ran this app many times before on data from different quarters, that
worked without any problems. However, now they want to run this program
against data for the whole last year. We keep getting System.OutOfMem ory
Exception. I realize that it's a huge amount of data that has to be
processed in a dataset..., but is there any quick fixes for this kind of
error, system tweaks (besides more memory). We really don't have time to
redesign this application to handle data any differently at this time.
They need this file yesterday!!! Please help.
By the way; the machine that it's running on has 2gb of RAM,
Could you do it in a loop, processing smaller sets of data broken up into
months, and just keep appending to the text file?
"LP" <lp@a.com> wrote in message
news:um******** ******@TK2MSFTN GP11.phx.gbl... Hello,
We running VB.NET application which gets massive amounts of data from SQL
Server, loads data into DataTables, then re-arranges data into tabular
structure and outputs it to a flat file. This is just the nature of
requirements for this application. Business analyst here want certain data
to be "re-horizontalize" and served in a single flat file.
We ran this app many times before on data from different quarters, that
worked without any problems. However, now they want to run this program
against data for the whole last year. We keep getting System.OutOfMem ory
Exception. I realize that it's a huge amount of data that has to be
processed in a dataset..., but is there any quick fixes for this kind of
error, system tweaks (besides more memory). We really don't have time to
redesign this application to handle data any differently at this time.
They need this file yesterday!!! Please help.
By the way; the machine that it's running on has 2gb of RAM,
This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics | |
by: Mike Schilling |
last post by:
Instances of SystemOutOfMemoryException do not contain a stack trace. Easy
test to verify this:
class OOM
{
public static void Main() {
try {
Object arr = new Object;
} catch (Exception ex) {
dumpEx(ex);
|
by: Tobias Matzat |
last post by:
Hello!
I'm writing an application that immitates ie behavior to save
html-pages. Well fecht the files in the webpage and change the paths in
the href/src attributes. Well, I#m doing this with the help of the RegEx
class. If the page contains a lot of files to fetch, I'm getting an
System.OutOfMemory Exception when changing something with the
RegEx.Replace(...). Taking a look in my taskmanager shows me that all
off my system memory is...
|
by: vMike |
last post by:
If a website is running on a shared server and a system.outofmemory
exception is thrown, is the error because of my website application or could
it be because of anyone's website application. If it is anyone, I guess
there isn't anything I can do, but if it is only my application, how can I
track it down.
Mike
|
by: Ricky Chan |
last post by:
Windows Server 2003 with 2G Ram ,IIS6
-> I have enabled /3gb switch in boot.ini !
-> Machine.config set to 80% memory Limit
However, It still throw outofmemory exception when
loading large amount of data.
|
by: Daniel |
last post by:
Exception of type System.OutOfMemoryException was thrown.
this error is occuring when on arbitrary threads in my .net windows service.
There is 2 gigs of memory on the machine and this .net windows service only
uses 50 megabytes. What could be causing this? how to fix?
| | |
by: Peter S. |
last post by:
I am pulling some data from a source via ODBC and placing the information
in a DataSet. The first pull is very large but once that is complete I plan
to do nightly pulls to get any new data that gets put in the (remote) table.
I can't seem to get past that initial (big) pull of data, as I get OutOfMemory
exceptions. I took a look back at when this occurs and it seems to happen upon
stuffing the DataSet with either the 2097153 record or...
|
by: Nemisis |
last post by:
Hi everyone,
Can someone please tell me why my code hits an "out of memory
exception" on the below code?
All the code does is load some documents from a SQL database and loop
through a data reader to create a thumbnail image for each image in my
dataReader.
The error only occurs when i have more then 3 images in my DataReader.
|
by: Mantorok |
last post by:
Hi all
I'm populating a DataTable with around 1,500,000 rows, the table contains 4
columns, 3 string columns and a decimal column.
However, I keep getting OutOfMemory exceptions when my app starts to reach
around 700MB memory (this is a console app by the way).
So, question is, why does the DT eat so much memory and how can I avoid
these OutOfMemory exceptions? I "need" all of these rows available because
|
by: =?Utf-8?B?TWlrZQ==?= |
last post by:
Hi Guys,
I have a real serious problem that stoped me doing any progress in my project.
in one of my webpages I have a wizard of more then 13 pages and in every
page some Ajax controls, textboxes, dropdownlists, about 4 panels, where
every page in my wizard have almost the same controls; so all what I had to
do is copy the controls from a wizard page then paste it to next one and just
change the controls names and it was going all fine...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| | |
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| | |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |
