Hi,
In my application I have text(flat) file as input and I have to generate
an XML file. The maximum input text file size can be 900MB and gererated xml
may result 2+ GB.
Based on the first column value from the text file, the row will be moved to
any of those 23 DataTable which are created onfly.
For eg.
01;- data for Table 1-
02;- data for Table 2-
03;- data for Table 3-
04;- data for Table 4-
04;- data for Table 4-
04;- data for Table 4-
04;- data for Table 4-
04;- data for Table 4-
Based on the child relation, I have to create XML node, after stripping the
parent-child relation column.
I am using XmlTextWriter to write the xml data, which are fetched from
tables based on select with different condition.
The server uses 2GB RAM memory
When the application process a 500MB file, it throws OutOfMemoryExce ption.
In the task manager, the utilised memory when the system throws
OutOfMemoryExce ption is 2.15 GB.
But in my desktop, for the surprise, the system has 512 MB and when I load
several application, the process memory reached 1GB without any
OutOfMemoryExce ption issues.
Can you one help me in this regard.
Thanks in advance
With Regards
venkatg 13 12958
Venkatachalam <Ve***********@ discussions.mic rosoft.com> wrote: In my application I have text(flat) file as input and I have to generate an XML file. The maximum input text file size can be 900MB and gererated xml may result 2+ GB.
Do you really need to hold all of that in memory at the same time? It
sounds like it would be much more likely to work if you could process
the file writing out data as you went, even if you need to process the
input file a few times.
<snip>
When the application process a 500MB file, it throws OutOfMemoryExce ption. In the task manager, the utilised memory when the system throws OutOfMemoryExce ption is 2.15 GB.
I believe that without some extra support (which I don't know the
details of) a single process can't go above 2GB.
But in my desktop, for the surprise, the system has 512 MB and when I load several application, the process memory reached 1GB without any OutOfMemoryExce ption issues.
Can you one help me in this regard.
That's your desktop swapping memory out to disk. Note that when this
happens, anything which tries to use that virtual memory will slow down
hugely.
--
Jon Skeet - <sk***@pobox.co m> http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Probably your home computer has virtual memory enabled, and its size is
allocated to something more than 512 MB, but your server computer doesn't.
Anyway, when you process a large quantity of data, you should not use
DataSet or DataTable or sorta things like that, because those data objects
will preload all the data into memory. you should use some cursor based
forward only Reader classes to read and process the data on demand.
Sheva
"Venkatacha lam" <Ve***********@ discussions.mic rosoft.com> wrote in message
news:81******** *************** ***********@mic rosoft.com... Hi, In my application I have text(flat) file as input and I have to generate an XML file. The maximum input text file size can be 900MB and gererated xml may result 2+ GB.
Based on the first column value from the text file, the row will be moved to any of those 23 DataTable which are created onfly.
For eg. 01;- data for Table 1- 02;- data for Table 2- 03;- data for Table 3- 04;- data for Table 4- 04;- data for Table 4- 04;- data for Table 4- 04;- data for Table 4- 04;- data for Table 4-
Based on the child relation, I have to create XML node, after stripping the parent-child relation column.
I am using XmlTextWriter to write the xml data, which are fetched from tables based on select with different condition.
The server uses 2GB RAM memory
When the application process a 500MB file, it throws OutOfMemoryExce ption. In the task manager, the utilised memory when the system throws OutOfMemoryExce ption is 2.15 GB.
But in my desktop, for the surprise, the system has 512 MB and when I load several application, the process memory reached 1GB without any OutOfMemoryExce ption issues.
Can you one help me in this regard.
Thanks in advance
With Regards venkatg
All the data has to be in the momory, as there are very complex relation
involved among the tables.
If I split the file to read, then the performance will go ver very worst.
And also the server has 4 Intel Xeon CPU 3.00GHz processor.
With regards
Venkat
"Jon Skeet [C# MVP]" wrote: Venkatachalam <Ve***********@ discussions.mic rosoft.com> wrote: In my application I have text(flat) file as input and I have to generate an XML file. The maximum input text file size can be 900MB and gererated xml may result 2+ GB.
Do you really need to hold all of that in memory at the same time? It sounds like it would be much more likely to work if you could process the file writing out data as you went, even if you need to process the input file a few times.
<snip>
When the application process a 500MB file, it throws OutOfMemoryExce ption. In the task manager, the utilised memory when the system throws OutOfMemoryExce ption is 2.15 GB.
I believe that without some extra support (which I don't know the details of) a single process can't go above 2GB.
But in my desktop, for the surprise, the system has 512 MB and when I load several application, the process memory reached 1GB without any OutOfMemoryExce ption issues.
Can you one help me in this regard.
That's your desktop swapping memory out to disk. Note that when this happens, anything which tries to use that virtual memory will slow down hugely.
-- Jon Skeet - <sk***@pobox.co m> http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet If replying to the group, please do not mail me too
Venkatachalam <Ve***********@ discussions.mic rosoft.com> wrote: All the data has to be in the momory, as there are very complex relation involved among the tables.
Well, even so I'd consider trying to work out a way of dealing with it
without having all of it in memory at a time.
If I split the file to read, then the performance will go ver very worst.
And also the server has 4 Intel Xeon CPU 3.00GHz processor.
The speed of the processors (nor the fact that it's got four of them)
isn't terribly relevant. If you cause the server to start swapping,
performance will be *awful*.
--
Jon Skeet - <sk***@pobox.co m> http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
"Very worse", compared to what? your program can never run to an end on a 32
bit system where you only have access to 2GB of user space per process.
The free user space on a 32Bit system after the CLR, framework code and data
has been loaded is less than 1.8GB, so you memory requirements exceed
largely the available memory. Note that your data requirements will double
when converting the text file 'byte' oriented data into XML 'string'
oriented data, add to that the XML overhead and you will understand that you
simply can't hold all this data in memory on a 32 bit system.
So your options are:
- Split the input file into smaller portions, or
- Store the input data in a relational database, or
- Move to a 64 bit OS and add another 2GB of RAM.
Willy.
"Venkatacha lam" <Ve***********@ discussions.mic rosoft.com> wrote in message
news:EB******** *************** ***********@mic rosoft.com...
| All the data has to be in the momory, as there are very complex relation
| involved among the tables.
|
| If I split the file to read, then the performance will go ver very worst.
|
| And also the server has 4 Intel Xeon CPU 3.00GHz processor.
|
| With regards
| Venkat
|
| "Jon Skeet [C# MVP]" wrote:
|
| > Venkatachalam <Ve***********@ discussions.mic rosoft.com> wrote:
| > > In my application I have text(flat) file as input and I have to
generate
| > > an XML file. The maximum input text file size can be 900MB and
gererated xml
| > > may result 2+ GB.
| >
| > Do you really need to hold all of that in memory at the same time? It
| > sounds like it would be much more likely to work if you could process
| > the file writing out data as you went, even if you need to process the
| > input file a few times.
| >
| > <snip>
| >
| > > When the application process a 500MB file, it throws
OutOfMemoryExce ption.
| > > In the task manager, the utilised memory when the system throws
| > > OutOfMemoryExce ption is 2.15 GB.
| >
| > I believe that without some extra support (which I don't know the
| > details of) a single process can't go above 2GB.
| >
| > > But in my desktop, for the surprise, the system has 512 MB and when I
load
| > > several application, the process memory reached 1GB without any
| > > OutOfMemoryExce ption issues.
| > >
| > > Can you one help me in this regard.
| >
| > That's your desktop swapping memory out to disk. Note that when this
| > happens, anything which tries to use that virtual memory will slow down
| > hugely.
| >
| > --
| > Jon Skeet - <sk***@pobox.co m>
| > http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
| > If replying to the group, please do not mail me too
| >
Venkatachalam enlightened me by writing: All the data has to be in the momory, as there are very complex relation involved among the tables.
If I split the file to read, then the performance will go ver very worst.
And also the server has 4 Intel Xeon CPU 3.00GHz processor.
With regards Venkat
Venkat, couldn't you use an embedded database to do this operation?
Maybe FirebirdSQL Embedded or something? This way you could still keep
your entity relatioships intact?
--
Warm Regards,
Lee
"Upon further investigation it appears that your software is missing
just one thing. It definitely needs more cow bell..."
Venkatachalam,
Did you look at the virtual memory settings of both computers?
Just my first thought,
Cor
"Venkatacha lam" <Ve***********@ discussions.mic rosoft.com> schreef in bericht
news:81******** *************** ***********@mic rosoft.com... Hi, In my application I have text(flat) file as input and I have to generate an XML file. The maximum input text file size can be 900MB and gererated xml may result 2+ GB.
Based on the first column value from the text file, the row will be moved to any of those 23 DataTable which are created onfly.
For eg. 01;- data for Table 1- 02;- data for Table 2- 03;- data for Table 3- 04;- data for Table 4- 04;- data for Table 4- 04;- data for Table 4- 04;- data for Table 4- 04;- data for Table 4-
Based on the child relation, I have to create XML node, after stripping the parent-child relation column.
I am using XmlTextWriter to write the xml data, which are fetched from tables based on select with different condition.
The server uses 2GB RAM memory
When the application process a 500MB file, it throws OutOfMemoryExce ption. In the task manager, the utilised memory when the system throws OutOfMemoryExce ption is 2.15 GB.
But in my desktop, for the surprise, the system has 512 MB and when I load several application, the process memory reached 1GB without any OutOfMemoryExce ption issues.
Can you one help me in this regard.
Thanks in advance
With Regards venkatg
VM is a non issue here the OP has 2GB of physical memory, so the VM is at
least 2GB and probably > 4GB. The problem is that the max. user address
space is 2GB (or 3GB with 4GT Ram tuning) on 32 bit windows.
Willy.
"Cor Ligthert [MVP]" <no************ @planet.nl> wrote in message
news:e%******** *******@TK2MSFT NGP14.phx.gbl.. .
| Venkatachalam,
|
| Did you look at the virtual memory settings of both computers?
|
| Just my first thought,
|
| Cor
|
| "Venkatacha lam" <Ve***********@ discussions.mic rosoft.com> schreef in
bericht
| news:81******** *************** ***********@mic rosoft.com...
| > Hi,
| > In my application I have text(flat) file as input and I have to
generate
| > an XML file. The maximum input text file size can be 900MB and gererated
| > xml
| > may result 2+ GB.
| >
| > Based on the first column value from the text file, the row will be
moved
| > to
| > any of those 23 DataTable which are created onfly.
| >
| > For eg.
| > 01;- data for Table 1-
| > 02;- data for Table 2-
| > 03;- data for Table 3-
| > 04;- data for Table 4-
| > 04;- data for Table 4-
| > 04;- data for Table 4-
| > 04;- data for Table 4-
| > 04;- data for Table 4-
| >
| > Based on the child relation, I have to create XML node, after stripping
| > the
| > parent-child relation column.
| >
| > I am using XmlTextWriter to write the xml data, which are fetched from
| > tables based on select with different condition.
| >
| > The server uses 2GB RAM memory
| >
| > When the application process a 500MB file, it throws
OutOfMemoryExce ption.
| > In the task manager, the utilised memory when the system throws
| > OutOfMemoryExce ption is 2.15 GB.
| >
| > But in my desktop, for the surprise, the system has 512 MB and when I
load
| > several application, the process memory reached 1GB without any
| > OutOfMemoryExce ption issues.
| >
| > Can you one help me in this regard.
| >
| > Thanks in advance
| >
| > With Regards
| > venkatg
|
|
Willy,
The difference between the laptop and the other computer was triggering me
in this.
What other reason can there been than just a setting of something like this. But in my desktop, for the surprise, the system has 512 MB and when I load several application, the process memory reached 1GB without any OutOfMemoryExc eption issues.
I am not sure if he is doing the same application, probably not,
However as I wrote, it was only a thought,
Cor
"Willy Denoyette [MVP]" <wi************ *@telenet.be> schreef in bericht
news:O7******** *****@TK2MSFTNG P15.phx.gbl... VM is a non issue here the OP has 2GB of physical memory, so the VM is at least 2GB and probably > 4GB. The problem is that the max. user address space is 2GB (or 3GB with 4GT Ram tuning) on 32 bit windows.
Willy.
"Cor Ligthert [MVP]" <no************ @planet.nl> wrote in message news:e%******** *******@TK2MSFT NGP14.phx.gbl.. . | Venkatachalam, | | Did you look at the virtual memory settings of both computers? | | Just my first thought, | | Cor | | "Venkatacha lam" <Ve***********@ discussions.mic rosoft.com> schreef in bericht | news:81******** *************** ***********@mic rosoft.com... | > Hi, | > In my application I have text(flat) file as input and I have to generate | > an XML file. The maximum input text file size can be 900MB and gererated | > xml | > may result 2+ GB. | > | > Based on the first column value from the text file, the row will be moved | > to | > any of those 23 DataTable which are created onfly. | > | > For eg. | > 01;- data for Table 1- | > 02;- data for Table 2- | > 03;- data for Table 3- | > 04;- data for Table 4- | > 04;- data for Table 4- | > 04;- data for Table 4- | > 04;- data for Table 4- | > 04;- data for Table 4- | > | > Based on the child relation, I have to create XML node, after stripping | > the | > parent-child relation column. | > | > I am using XmlTextWriter to write the xml data, which are fetched from | > tables based on select with different condition. | > | > The server uses 2GB RAM memory | > | > When the application process a 500MB file, it throws OutOfMemoryExce ption. | > In the task manager, the utilised memory when the system throws | > OutOfMemoryExce ption is 2.15 GB. | > | > But in my desktop, for the surprise, the system has 512 MB and when I load | > several application, the process memory reached 1GB without any | > OutOfMemoryExce ption issues. | > | > Can you one help me in this regard. | > | > Thanks in advance | > | > With Regards | > venkatg | |
This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Peter Aberline |
last post by:
Hi all,
We have written a C # .NET application and we're
encountering memory problems in the form of
System.OutOfMemoryException.
Our application creates many thousands of objects in a
temporary in memory store, which uses a lot of runtime
memory. Not an ideal situation, but an acceptable short
term solution given our project constraints.
|
by: Ryan Seghers |
last post by:
I've got a program that has no user interface (like a service but not
actually a Windows Service yet) in which I'd like to handle
OutOfMemoryExceptions. I'd at least like to log the failure before exiting,
if possible. I understand that it probably takes some memory to continue
operating, even just to write a message to an open file, but if the
allocation that triggered the OutOfMemoryException was large then there
probably really is a...
|
by: Per Bergland |
last post by:
After many woes, I finally managed to get a stack dump of my System
Service (written in C#) that insists on crashing when launched at system
boot time (see below on how to get this dump - I couldn't find any info
on how to do this).
Here's the stack trace from cordbg:
Unhandled exception generated: (0x04719c94)
<System.Runtime.Remoting.RemotingException>
_className=<null>
_exceptionMethod=<null>
|
by: Michael |
last post by:
I have a problem with catching the OutOfMemoryException in a managed C+
application. When creating small objects on the managed heap neithe
the catch handler for OutOfMemoryException nor the UnhandledExceptio
handler is called in an out of memory situation. Instead the applicatio
writes 'Fatal out of memory' to the console and exits
Is this a bug in the runtime or am I missing something
Here is the example code
#include "stdafx.h...
|
by: Ripul Handa |
last post by:
Hi
We are running IIS 5.0 cluster with cisco local director.
We are running a website on 2 webservers and I have been
observing that from past few days we have are getting
this error message of and on
Error Messag
Remote IP:66.122.242.6
Host:216.211.212.2
| |
by: SMG - Idealake |
last post by:
Hi all,
I am getting following error on my error, what could be the reason?
Exception of type System.OutOfMemoryException was thrown.
Description: An unhandled exception occurred during the execution of the
current web request. Please review the stack trace for more information
about the error and where it originated in the code.
Exception Details: System.OutOfMemoryException: Exception of type
System.OutOfMemoryException was thrown.
|
by: Dave |
last post by:
We just started getting this error message in our application today (stack
trace below). From the OutOfMemoryException, I'm guessing it could be a
memory leak. I'm making sure I'm closing all my connections in the finally
block but I'm not sure what I should be doing.
As far as the "Unable to serialize the session state" error, this app has
been running for days and this is the first I've seen this one as well.
Any thoughts would be...
|
by: Ashkan Daie |
last post by:
Hi All,
When trying to install a performance counter via InstallUtil I get the
following exception:
Creating performance counter category Enterprise Library Caching.
An exception occurred during the Install phase.
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException'
was
|
by: =?Utf-8?B?UGlnZ3k=?= |
last post by:
Hi to all,
I am getting this System.OutOfMemoryException calling the
Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize(<stream>,<Obj>) method.
The type of <streamis IO.MemoryStream
=====Exception:
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException'
was
thrown.
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| |
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
| |
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |