By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
429,471 Members | 722 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 429,471 IT Pros & Developers. It's quick & easy.

OutOfMemoryException on DataTables

P: n/a
Hi,
In my application I have text(flat) file as input and I have to generate
an XML file. The maximum input text file size can be 900MB and gererated xml
may result 2+ GB.

Based on the first column value from the text file, the row will be moved to
any of those 23 DataTable which are created onfly.

For eg.
01;- data for Table 1-
02;- data for Table 2-
03;- data for Table 3-
04;- data for Table 4-
04;- data for Table 4-
04;- data for Table 4-
04;- data for Table 4-
04;- data for Table 4-

Based on the child relation, I have to create XML node, after stripping the
parent-child relation column.

I am using XmlTextWriter to write the xml data, which are fetched from
tables based on select with different condition.

The server uses 2GB RAM memory

When the application process a 500MB file, it throws OutOfMemoryException.
In the task manager, the utilised memory when the system throws
OutOfMemoryException is 2.15 GB.

But in my desktop, for the surprise, the system has 512 MB and when I load
several application, the process memory reached 1GB without any
OutOfMemoryException issues.

Can you one help me in this regard.

Thanks in advance

With Regards
venkatg
Mar 5 '06 #1
Share this Question
Share on Google+
13 Replies


P: n/a
Venkatachalam <Ve***********@discussions.microsoft.com> wrote:
In my application I have text(flat) file as input and I have to generate
an XML file. The maximum input text file size can be 900MB and gererated xml
may result 2+ GB.
Do you really need to hold all of that in memory at the same time? It
sounds like it would be much more likely to work if you could process
the file writing out data as you went, even if you need to process the
input file a few times.

<snip>
When the application process a 500MB file, it throws OutOfMemoryException.
In the task manager, the utilised memory when the system throws
OutOfMemoryException is 2.15 GB.
I believe that without some extra support (which I don't know the
details of) a single process can't go above 2GB.
But in my desktop, for the surprise, the system has 512 MB and when I load
several application, the process memory reached 1GB without any
OutOfMemoryException issues.

Can you one help me in this regard.


That's your desktop swapping memory out to disk. Note that when this
happens, anything which tries to use that virtual memory will slow down
hugely.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Mar 5 '06 #2

P: n/a
Probably your home computer has virtual memory enabled, and its size is
allocated to something more than 512 MB, but your server computer doesn't.
Anyway, when you process a large quantity of data, you should not use
DataSet or DataTable or sorta things like that, because those data objects
will preload all the data into memory. you should use some cursor based
forward only Reader classes to read and process the data on demand.

Sheva
"Venkatachalam" <Ve***********@discussions.microsoft.com> wrote in message
news:81**********************************@microsof t.com...
Hi,
In my application I have text(flat) file as input and I have to generate
an XML file. The maximum input text file size can be 900MB and gererated
xml
may result 2+ GB.

Based on the first column value from the text file, the row will be moved
to
any of those 23 DataTable which are created onfly.

For eg.
01;- data for Table 1-
02;- data for Table 2-
03;- data for Table 3-
04;- data for Table 4-
04;- data for Table 4-
04;- data for Table 4-
04;- data for Table 4-
04;- data for Table 4-

Based on the child relation, I have to create XML node, after stripping
the
parent-child relation column.

I am using XmlTextWriter to write the xml data, which are fetched from
tables based on select with different condition.

The server uses 2GB RAM memory

When the application process a 500MB file, it throws OutOfMemoryException.
In the task manager, the utilised memory when the system throws
OutOfMemoryException is 2.15 GB.

But in my desktop, for the surprise, the system has 512 MB and when I load
several application, the process memory reached 1GB without any
OutOfMemoryException issues.

Can you one help me in this regard.

Thanks in advance

With Regards
venkatg

Mar 5 '06 #3

P: n/a
All the data has to be in the momory, as there are very complex relation
involved among the tables.

If I split the file to read, then the performance will go ver very worst.

And also the server has 4 Intel Xeon CPU 3.00GHz processor.

With regards
Venkat

"Jon Skeet [C# MVP]" wrote:
Venkatachalam <Ve***********@discussions.microsoft.com> wrote:
In my application I have text(flat) file as input and I have to generate
an XML file. The maximum input text file size can be 900MB and gererated xml
may result 2+ GB.


Do you really need to hold all of that in memory at the same time? It
sounds like it would be much more likely to work if you could process
the file writing out data as you went, even if you need to process the
input file a few times.

<snip>
When the application process a 500MB file, it throws OutOfMemoryException.
In the task manager, the utilised memory when the system throws
OutOfMemoryException is 2.15 GB.


I believe that without some extra support (which I don't know the
details of) a single process can't go above 2GB.
But in my desktop, for the surprise, the system has 512 MB and when I load
several application, the process memory reached 1GB without any
OutOfMemoryException issues.

Can you one help me in this regard.


That's your desktop swapping memory out to disk. Note that when this
happens, anything which tries to use that virtual memory will slow down
hugely.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

Mar 5 '06 #4

P: n/a
Venkatachalam <Ve***********@discussions.microsoft.com> wrote:
All the data has to be in the momory, as there are very complex relation
involved among the tables.
Well, even so I'd consider trying to work out a way of dealing with it
without having all of it in memory at a time.
If I split the file to read, then the performance will go ver very worst.

And also the server has 4 Intel Xeon CPU 3.00GHz processor.


The speed of the processors (nor the fact that it's got four of them)
isn't terribly relevant. If you cause the server to start swapping,
performance will be *awful*.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Mar 5 '06 #5

P: n/a
"Very worse", compared to what? your program can never run to an end on a 32
bit system where you only have access to 2GB of user space per process.
The free user space on a 32Bit system after the CLR, framework code and data
has been loaded is less than 1.8GB, so you memory requirements exceed
largely the available memory. Note that your data requirements will double
when converting the text file 'byte' oriented data into XML 'string'
oriented data, add to that the XML overhead and you will understand that you
simply can't hold all this data in memory on a 32 bit system.
So your options are:
- Split the input file into smaller portions, or
- Store the input data in a relational database, or
- Move to a 64 bit OS and add another 2GB of RAM.

Willy.

"Venkatachalam" <Ve***********@discussions.microsoft.com> wrote in message
news:EB**********************************@microsof t.com...
| All the data has to be in the momory, as there are very complex relation
| involved among the tables.
|
| If I split the file to read, then the performance will go ver very worst.
|
| And also the server has 4 Intel Xeon CPU 3.00GHz processor.
|
| With regards
| Venkat
|
| "Jon Skeet [C# MVP]" wrote:
|
| > Venkatachalam <Ve***********@discussions.microsoft.com> wrote:
| > > In my application I have text(flat) file as input and I have to
generate
| > > an XML file. The maximum input text file size can be 900MB and
gererated xml
| > > may result 2+ GB.
| >
| > Do you really need to hold all of that in memory at the same time? It
| > sounds like it would be much more likely to work if you could process
| > the file writing out data as you went, even if you need to process the
| > input file a few times.
| >
| > <snip>
| >
| > > When the application process a 500MB file, it throws
OutOfMemoryException.
| > > In the task manager, the utilised memory when the system throws
| > > OutOfMemoryException is 2.15 GB.
| >
| > I believe that without some extra support (which I don't know the
| > details of) a single process can't go above 2GB.
| >
| > > But in my desktop, for the surprise, the system has 512 MB and when I
load
| > > several application, the process memory reached 1GB without any
| > > OutOfMemoryException issues.
| > >
| > > Can you one help me in this regard.
| >
| > That's your desktop swapping memory out to disk. Note that when this
| > happens, anything which tries to use that virtual memory will slow down
| > hugely.
| >
| > --
| > Jon Skeet - <sk***@pobox.com>
| > http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
| > If replying to the group, please do not mail me too
| >
Mar 5 '06 #6

P: n/a
Lee
Venkatachalam enlightened me by writing:
All the data has to be in the momory, as there are very complex
relation involved among the tables.

If I split the file to read, then the performance will go ver very
worst.

And also the server has 4 Intel Xeon CPU 3.00GHz processor.

With regards
Venkat


Venkat, couldn't you use an embedded database to do this operation?
Maybe FirebirdSQL Embedded or something? This way you could still keep
your entity relatioships intact?
--
Warm Regards,
Lee

"Upon further investigation it appears that your software is missing
just one thing. It definitely needs more cow bell..."
Mar 5 '06 #7

P: n/a
Venkatachalam,

Did you look at the virtual memory settings of both computers?

Just my first thought,

Cor

"Venkatachalam" <Ve***********@discussions.microsoft.com> schreef in bericht
news:81**********************************@microsof t.com...
Hi,
In my application I have text(flat) file as input and I have to generate
an XML file. The maximum input text file size can be 900MB and gererated
xml
may result 2+ GB.

Based on the first column value from the text file, the row will be moved
to
any of those 23 DataTable which are created onfly.

For eg.
01;- data for Table 1-
02;- data for Table 2-
03;- data for Table 3-
04;- data for Table 4-
04;- data for Table 4-
04;- data for Table 4-
04;- data for Table 4-
04;- data for Table 4-

Based on the child relation, I have to create XML node, after stripping
the
parent-child relation column.

I am using XmlTextWriter to write the xml data, which are fetched from
tables based on select with different condition.

The server uses 2GB RAM memory

When the application process a 500MB file, it throws OutOfMemoryException.
In the task manager, the utilised memory when the system throws
OutOfMemoryException is 2.15 GB.

But in my desktop, for the surprise, the system has 512 MB and when I load
several application, the process memory reached 1GB without any
OutOfMemoryException issues.

Can you one help me in this regard.

Thanks in advance

With Regards
venkatg

Mar 6 '06 #8

P: n/a
VM is a non issue here the OP has 2GB of physical memory, so the VM is at
least 2GB and probably > 4GB. The problem is that the max. user address
space is 2GB (or 3GB with 4GT Ram tuning) on 32 bit windows.
Willy.

"Cor Ligthert [MVP]" <no************@planet.nl> wrote in message
news:e%***************@TK2MSFTNGP14.phx.gbl...
| Venkatachalam,
|
| Did you look at the virtual memory settings of both computers?
|
| Just my first thought,
|
| Cor
|
| "Venkatachalam" <Ve***********@discussions.microsoft.com> schreef in
bericht
| news:81**********************************@microsof t.com...
| > Hi,
| > In my application I have text(flat) file as input and I have to
generate
| > an XML file. The maximum input text file size can be 900MB and gererated
| > xml
| > may result 2+ GB.
| >
| > Based on the first column value from the text file, the row will be
moved
| > to
| > any of those 23 DataTable which are created onfly.
| >
| > For eg.
| > 01;- data for Table 1-
| > 02;- data for Table 2-
| > 03;- data for Table 3-
| > 04;- data for Table 4-
| > 04;- data for Table 4-
| > 04;- data for Table 4-
| > 04;- data for Table 4-
| > 04;- data for Table 4-
| >
| > Based on the child relation, I have to create XML node, after stripping
| > the
| > parent-child relation column.
| >
| > I am using XmlTextWriter to write the xml data, which are fetched from
| > tables based on select with different condition.
| >
| > The server uses 2GB RAM memory
| >
| > When the application process a 500MB file, it throws
OutOfMemoryException.
| > In the task manager, the utilised memory when the system throws
| > OutOfMemoryException is 2.15 GB.
| >
| > But in my desktop, for the surprise, the system has 512 MB and when I
load
| > several application, the process memory reached 1GB without any
| > OutOfMemoryException issues.
| >
| > Can you one help me in this regard.
| >
| > Thanks in advance
| >
| > With Regards
| > venkatg
|
|
Mar 6 '06 #9

P: n/a
Willy,

The difference between the laptop and the other computer was triggering me
in this.
What other reason can there been than just a setting of something like this.
But in my desktop, for the surprise, the system has 512 MB and when I load
several application, the process memory reached 1GB without any
OutOfMemoryException issues.
I am not sure if he is doing the same application, probably not,

However as I wrote, it was only a thought,

Cor

"Willy Denoyette [MVP]" <wi*************@telenet.be> schreef in bericht
news:O7*************@TK2MSFTNGP15.phx.gbl... VM is a non issue here the OP has 2GB of physical memory, so the VM is at
least 2GB and probably > 4GB. The problem is that the max. user address
space is 2GB (or 3GB with 4GT Ram tuning) on 32 bit windows.
Willy.

"Cor Ligthert [MVP]" <no************@planet.nl> wrote in message
news:e%***************@TK2MSFTNGP14.phx.gbl...
| Venkatachalam,
|
| Did you look at the virtual memory settings of both computers?
|
| Just my first thought,
|
| Cor
|
| "Venkatachalam" <Ve***********@discussions.microsoft.com> schreef in
bericht
| news:81**********************************@microsof t.com...
| > Hi,
| > In my application I have text(flat) file as input and I have to
generate
| > an XML file. The maximum input text file size can be 900MB and
gererated
| > xml
| > may result 2+ GB.
| >
| > Based on the first column value from the text file, the row will be
moved
| > to
| > any of those 23 DataTable which are created onfly.
| >
| > For eg.
| > 01;- data for Table 1-
| > 02;- data for Table 2-
| > 03;- data for Table 3-
| > 04;- data for Table 4-
| > 04;- data for Table 4-
| > 04;- data for Table 4-
| > 04;- data for Table 4-
| > 04;- data for Table 4-
| >
| > Based on the child relation, I have to create XML node, after
stripping
| > the
| > parent-child relation column.
| >
| > I am using XmlTextWriter to write the xml data, which are fetched from
| > tables based on select with different condition.
| >
| > The server uses 2GB RAM memory
| >
| > When the application process a 500MB file, it throws
OutOfMemoryException.
| > In the task manager, the utilised memory when the system throws
| > OutOfMemoryException is 2.15 GB.
| >
| > But in my desktop, for the surprise, the system has 512 MB and when I
load
| > several application, the process memory reached 1GB without any
| > OutOfMemoryException issues.
| >
| > Can you one help me in this regard.
| >
| > Thanks in advance
| >
| > With Regards
| > venkatg
|
|

Mar 6 '06 #10

P: n/a
The BIG difference is:

One single application can allocate at most 2GB of VAS.
5 applications can allocate up to 2GB VAS each (provided that you have
sufficient RAM and/or that the swap file is large enough), that means a
total of 10GB allocated space.

Willy.

"Cor Ligthert [MVP]" <no************@planet.nl> wrote in message
news:Os**************@TK2MSFTNGP10.phx.gbl...
| Willy,
|
| The difference between the laptop and the other computer was triggering me
| in this.
| What other reason can there been than just a setting of something like
this.
|
| >But in my desktop, for the surprise, the system has 512 MB and when I
load
| >several application, the process memory reached 1GB without any
| >OutOfMemoryException issues.
|
| I am not sure if he is doing the same application, probably not,
|
| However as I wrote, it was only a thought,
|
| Cor
|
| "Willy Denoyette [MVP]" <wi*************@telenet.be> schreef in bericht
| news:O7*************@TK2MSFTNGP15.phx.gbl...
| > VM is a non issue here the OP has 2GB of physical memory, so the VM is
at
| > least 2GB and probably > 4GB. The problem is that the max. user address
| > space is 2GB (or 3GB with 4GT Ram tuning) on 32 bit windows.
| >
| >
| > Willy.
| >
| > "Cor Ligthert [MVP]" <no************@planet.nl> wrote in message
| > news:e%***************@TK2MSFTNGP14.phx.gbl...
| > | Venkatachalam,
| > |
| > | Did you look at the virtual memory settings of both computers?
| > |
| > | Just my first thought,
| > |
| > | Cor
| > |
| > | "Venkatachalam" <Ve***********@discussions.microsoft.com> schreef in
| > bericht
| > | news:81**********************************@microsof t.com...
| > | > Hi,
| > | > In my application I have text(flat) file as input and I have to
| > generate
| > | > an XML file. The maximum input text file size can be 900MB and
| > gererated
| > | > xml
| > | > may result 2+ GB.
| > | >
| > | > Based on the first column value from the text file, the row will be
| > moved
| > | > to
| > | > any of those 23 DataTable which are created onfly.
| > | >
| > | > For eg.
| > | > 01;- data for Table 1-
| > | > 02;- data for Table 2-
| > | > 03;- data for Table 3-
| > | > 04;- data for Table 4-
| > | > 04;- data for Table 4-
| > | > 04;- data for Table 4-
| > | > 04;- data for Table 4-
| > | > 04;- data for Table 4-
| > | >
| > | > Based on the child relation, I have to create XML node, after
| > stripping
| > | > the
| > | > parent-child relation column.
| > | >
| > | > I am using XmlTextWriter to write the xml data, which are fetched
from
| > | > tables based on select with different condition.
| > | >
| > | > The server uses 2GB RAM memory
| > | >
| > | > When the application process a 500MB file, it throws
| > OutOfMemoryException.
| > | > In the task manager, the utilised memory when the system throws
| > | > OutOfMemoryException is 2.15 GB.
| > | >
| > | > But in my desktop, for the surprise, the system has 512 MB and when
I
| > load
| > | > several application, the process memory reached 1GB without any
| > | > OutOfMemoryException issues.
| > | >
| > | > Can you one help me in this regard.
| > | >
| > | > Thanks in advance
| > | >
| > | > With Regards
| > | > venkatg
| > |
| > |
| >
| >
|
|
Mar 6 '06 #11

P: n/a
Hello, Willy!

WDM> One single application can allocate at most 2GB of VAS.
WDM> 5 applications can allocate up to 2GB VAS each (provided that you have
WDM> sufficient RAM and/or that the swap file is large enough), that means
WDM> a total of 10GB allocated space.

IMO AWE can be used to overcome limitations mentioned above.

( http://msdn.microsoft.com/library/de...extensions.asp )


--
Regards, Vadym Stetsyak
www: http://vadmyst.blogspot.com
Mar 7 '06 #12

P: n/a
Hello, Sheva!

S> Probably your home computer has virtual memory enabled, and its size is
S> allocated to something more than 512 MB, but your server computer
S> doesn't.

That is not correct, take a look at Willy's posts about virtual memory...

S> Anyway, when you process a large quantity of data, you should not use
S> DataSet or DataTable or sorta things like that, because those data
S> objects will preload all the data into memory. you should use some
S> cursor based forward only Reader classes to read and process the data on
S> demand.

...or do the processing on the db side and then retrieve and display the results in the application
--
Regards, Vadym Stetsyak
www: http://vadmyst.blogspot.com
Mar 7 '06 #13

P: n/a

"Vadym Stetsyak" <va*****@ukr.net> wrote in message
news:OY**************@TK2MSFTNGP11.phx.gbl...
| Hello, Willy!
|
| WDM> One single application can allocate at most 2GB of VAS.
| WDM> 5 applications can allocate up to 2GB VAS each (provided that you
have
| WDM> sufficient RAM and/or that the swap file is large enough), that means
| WDM> a total of 10GB allocated space.
|
| IMO AWE can be used to overcome limitations mentioned above.
|
| (
http://msdn.microsoft.com/library/de...extensions.asp )
|
|
| --
| Regards, Vadym Stetsyak
| www: http://vadmyst.blogspot.com

No, not really, the CLR does not care about AWE so managed applications do
not take advantage of this, services like SQL Server use this, but this is
only done to compensate for the limited addressable memory range on a 32 bit
OS, it's really of little use now that 64 bit windows is available.

Willy.
Mar 7 '06 #14

This discussion thread is closed

Replies have been disabled for this discussion.