Hi all
I'm populating a DataTable with around 1,500,000 rows, the table contains 4
columns, 3 string columns and a decimal column.
However, I keep getting OutOfMemory exceptions when my app starts to reach
around 700MB memory (this is a console app by the way).
So, question is, why does the DT eat so much memory and how can I avoid
these OutOfMemory exceptions? I "need" all of these rows available because
they will be referenced throughout the lifecycle of my app.
Thanks
Kev 8 13362
Hello Mantorok,
Hi all
I'm populating a DataTable with around 1,500,000 rows, the table
contains 4 columns, 3 string columns and a decimal column.
However, I keep getting OutOfMemory exceptions when my app starts to
reach around 700MB memory (this is a console app by the way).
So, question is, why does the DT eat so much memory and how can I
avoid these OutOfMemory exceptions? I "need" all of these rows
available because they will be referenced throughout the lifecycle of
my app.
Thanks
Kev
DataTable contains much more than just teh values. And it is made up from
a lot of different classes that also remain in memory (DataColumn, DataRows,
etc). Why not use a simpler construct.
public class MyImportantThin g
{
private string _string1;
private string _string2;
private string _string3;
private decimal _decimal1;
public MuImportantThin g(string s1, string s2, string s3, decimal d1)
{
_string1 = s1;
...
...
_decimal1 = d1;
}
// Properties here
}
And then instantiate a List<MyImportan tThingthis should be much faster.
If you need find/select capabilities you can add these easily if you create
your own list type:
public class MyImportantThin gCollection: List<MyImportan tThing>
{
public MyImportantThin g FindByString1(s tring s1)
{
foreach (MyImportantThi ng mip in this)
{
if (string.Equals( mip.String1, s1)
{
return mip;
}
}
}
}
If you need fast access through one of the properties of you class, you could
always decide to use a Disctionary<>
public class MyImportantThin gDictionary: Dictionary<MyIm portantThing>
{
}
This will have a much lower memory footprint.
I would also consider a completely different scenario... Don't load a list
of 1500000 items at all and get the item you need from a database when you
need them. That insures you only load the rows you absolutely need. You could
even load the item when it's actually being used, instead of pre-loading
them all. But I guess you have your reasons.
Jesse
Hi Jesse
Thanks for your response, it has prompted me to re-think why I need all
these records, and because of that I've just realised I don't need all of
the records because I could be filtering them at the DB anyway.
Anyhow, the solution regarding custom classes wouldn't apply to my
situation, the DataTables are being returned from 7 web services that we
have in 7 of our remote data sources (the WS takes an SQL statement and
returns the results).
But thanks anyway, I'm going to refactor my code now and should have a much
more efficient solution.
Thanks
Kev
"Jesse Houwing" <Je***********@ nospam-sogeti.nlwrote in message
news:33******** *************** **@news.microso ft.com...
Hello Mantorok,
>Hi all
I'm populating a DataTable with around 1,500,000 rows, the table contains 4 columns, 3 string columns and a decimal column.
However, I keep getting OutOfMemory exceptions when my app starts to reach around 700MB memory (this is a console app by the way).
So, question is, why does the DT eat so much memory and how can I avoid these OutOfMemory exceptions? I "need" all of these rows available because they will be referenced throughout the lifecycle of my app.
Thanks Kev
DataTable contains much more than just teh values. And it is made up from
a lot of different classes that also remain in memory (DataColumn,
DataRows, etc). Why not use a simpler construct.
public class MyImportantThin g
{
private string _string1;
private string _string2;
private string _string3;
private decimal _decimal1;
public MuImportantThin g(string s1, string s2, string s3, decimal d1)
{
_string1 = s1;
...
...
_decimal1 = d1;
}
// Properties here
}
And then instantiate a List<MyImportan tThingthis should be much faster.
If you need find/select capabilities you can add these easily if you
create your own list type:
public class MyImportantThin gCollection: List<MyImportan tThing>
{
public MyImportantThin g FindByString1(s tring s1)
{
foreach (MyImportantThi ng mip in this)
{
if (string.Equals( mip.String1, s1)
{
return mip;
}
}
}
}
If you need fast access through one of the properties of you class, you
could always decide to use a Disctionary<>
public class MyImportantThin gDictionary: Dictionary<MyIm portantThing>
{
}
This will have a much lower memory footprint.
I would also consider a completely different scenario... Don't load a list
of 1500000 items at all and get the item you need from a database when you
need them. That insures you only load the rows you absolutely need. You
could even load the item when it's actually being used, instead of
pre-loading them all. But I guess you have your reasons.
Jesse
Hello Mantorok,
Hi Jesse
Thanks for your response, it has prompted me to re-think why I need
all these records, and because of that I've just realised I don't need
all of the records because I could be filtering them at the DB anyway.
Anyhow, the solution regarding custom classes wouldn't apply to my
situation, the DataTables are being returned from 7 web services that
we have in 7 of our remote data sources (the WS takes an SQL statement
and returns the results).
But thanks anyway, I'm going to refactor my code now and should have a
much more efficient solution.
Thanks
Kev
You're welcome
You could still put the contents of teh datatables in a custom class. You
could then throw away the datatables.
Jesse
"Jesse Houwing" <Je***********@ nospam-sogeti.nlwrote in message
news:33******** *************** **@news.microso ft.com...
>Hello Mantorok,
>>Hi all
I'm populating a DataTable with around 1,500,000 rows, the table contains 4 columns, 3 string columns and a decimal column.
However, I keep getting OutOfMemory exceptions when my app starts to reach around 700MB memory (this is a console app by the way).
So, question is, why does the DT eat so much memory and how can I avoid these OutOfMemory exceptions? I "need" all of these rows available because they will be referenced throughout the lifecycle of my app.
Thanks Kev
DataTable contains much more than just teh values. And it is made up from a lot of different classes that also remain in memory (DataColumn, DataRows, etc). Why not use a simpler construct.
public class MyImportantThin g { private string _string1; private string _string2; private string _string3; private decimal _decimal1; public MuImportantThin g(string s1, string s2, string s3, decimal d1) { _string1 = s1; ... ... _decimal1 = d1; } // Properties here } And then instantiate a List<MyImportan tThingthis should be much faster. If you need find/select capabilities you can add these easily if you create your own list type:
public class MyImportantThin gCollection: List<MyImportan tThing> { public MyImportantThin g FindByString1(s tring s1) { foreach (MyImportantThi ng mip in this) { if (string.Equals( mip.String1, s1) { return mip; } } } } If you need fast access through one of the properties of you class, you could always decide to use a Disctionary<>
public class MyImportantThin gDictionary: Dictionary<MyIm portantThing> { } This will have a much lower memory footprint. I would also consider a completely different scenario... Don't load a list of 1500000 items at all and get the item you need from a database when you need them. That insures you only load the rows you absolutely need. You could even load the item when it's actually being used, instead of pre-loading them all. But I guess you have your reasons. Jesse
You're welcome
>
You could still put the contents of teh datatables in a custom class.
You could then throw away the datatables.
True, however I have a function that queries all 7 sources for me and stuffs
all the results into 1 fat table, and this is where it runs out of memory.
If I created custom classes I would have to pretty much re-write this function
for this 1 purpose, which isn't favourable.
It was sloppy of me not to restrict the data-set coming back, I should know
better but I never thought about it for this particular task.
Kev
Hello Mantorok,
>You're welcome
You could still put the contents of teh datatables in a custom class. You could then throw away the datatables.
True, however I have a function that queries all 7 sources for me and
stuffs
all the results into 1 fat table, and this is where it runs out of
memory.
If I created custom classes I would have to pretty much re-write this
function
for this 1 purpose, which isn't favourable.
It was sloppy of me not to restrict the data-set coming back, I should
know better but I never thought about it for this particular task.
Kev
We all make mistakes ;). Good luck!
Jesse
HI,
In any case 1.5M of rows is TOO much for anything useful.
You better keep them in a DB.
Any operation will take a LONG time in a table with 1.5M rows.
"Mantorok" <no**@none.comw rote in message
news:f8******** **@newsfeed.th. ifl.net...
Hi all
I'm populating a DataTable with around 1,500,000 rows, the table contains
4 columns, 3 string columns and a decimal column.
However, I keep getting OutOfMemory exceptions when my app starts to reach
around 700MB memory (this is a console app by the way).
So, question is, why does the DT eat so much memory and how can I avoid
these OutOfMemory exceptions? I "need" all of these rows available because
they will be referenced throughout the lifecycle of my app.
Thanks
Kev
HI,
"Mantorok" <no**@none.comw rote in message
news:f8******** **@newsfeed.th. ifl.net...
Hi Jesse
Thanks for your response, it has prompted me to re-think why I need all
these records, and because of that I've just realised I don't need all of
the records because I could be filtering them at the DB anyway.
Anyhow, the solution regarding custom classes wouldn't apply to my
situation, the DataTables are being returned from 7 web services that we
have in 7 of our remote data sources (the WS takes an SQL statement and
returns the results).
But thanks anyway, I'm going to refactor my code now and should have a
much more efficient solution.
How ofter you run this?
I would suggest you to insert all those results in a local SQL DB and then
make a query to the local data.
HI,
>
In any case 1.5M of rows is TOO much for anything useful.
You better keep them in a DB.
Any operation will take a LONG time in a table with 1.5M rows.
Yep - I think my brain was on vacation the day I wrote this app ;)
I was going about it the completely wrong way, all this time I could've reduced
the result set and I didn't even think to, anyway it's all ok now.
Thanks
Kev This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: vMike |
last post by:
If a website is running on a shared server and a system.outofmemory
exception is thrown, is the error because of my website application or could
it be because of anyone's website application. If it is anyone, I guess
there isn't anything I can do, but if it is only my application, how can I
track it down.
Mike
|
by: Ricky Chan |
last post by:
Windows Server 2003 with 2G Ram ,IIS6
-> I have enabled /3gb switch in boot.ini !
-> Machine.config set to 80% memory Limit
However, It still throw outofmemory exception when
loading large amount of data.
|
by: Ricky Chan |
last post by:
In the production environment, it always occurs and the worker process
did not recycle automatically. Therefore, it make the system service break
to client.
In development environment, we write a program to loop sth and force
outofmemory throw. however, when the w3wp exceed physical mem,
the process recycle automatically, no outofmemory throw.
Any idea?
thank you
|
by: Joe Ross |
last post by:
(Apologies in advance if there is a better forum for asking advice on this
topic).
Our ASP.NET application occasionally starts spitting out OutOfMemory
exceptions. When this happens, the memory usage for that IIS worker process
is over 1GB. I understand in this sceneario that the virtual memory pool
can become fragmented and produce this type of exception.
This time, however, I was able to do an ADPlus dump while it was in this...
|
by: LP |
last post by:
Hello,
We running VB.NET application which gets massive amounts of data from SQL
Server, loads data into DataTables, then re-arranges data into tabular
structure and outputs it to a flat file. This is just the nature of
requirements for this application. Business analyst here want certain data
to be "re-horizontalize" and served in a single flat file.
We ran this app many times before on data from different quarters, that
worked without...
| |
by: Peter S. |
last post by:
I am pulling some data from a source via ODBC and placing the information
in a DataSet. The first pull is very large but once that is complete I plan
to do nightly pulls to get any new data that gets put in the (remote) table.
I can't seem to get past that initial (big) pull of data, as I get OutOfMemory
exceptions. I took a look back at when this occurs and it seems to happen upon
stuffing the DataSet with either the 2097153 record or...
|
by: Nemisis |
last post by:
Hi everyone,
Can someone please tell me why my code hits an "out of memory
exception" on the below code?
All the code does is load some documents from a SQL database and loop
through a data reader to create a thumbnail image for each image in my
dataReader.
The error only occurs when i have more then 3 images in my DataReader.
|
by: =?Utf-8?B?TWlrZQ==?= |
last post by:
Hi Guys,
I have a real serious problem that stoped me doing any progress in my project.
in one of my webpages I have a wizard of more then 13 pages and in every
page some Ajax controls, textboxes, dropdownlists, about 4 panels, where
every page in my wizard have almost the same controls; so all what I had to
do is copy the controls from a wizard page then paste it to next one and just
change the controls names and it was going all fine...
|
by: njuneardave |
last post by:
I am using a listview to hold a bunch of binary data (approx 25 rows
and 700 cols). I have a compare function that compares each row's bit
at a certain column. i then color the background of each cell as a
result (green for equal value, red for unequal value). This colors
each of the 700 columns either green or red.
now, the problem: when i scroll horizontally to look at all of the
columns, i will periodically run into a horrible...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
| |
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| |
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |