473,566 Members | 2,784 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

OutOfMemoryExce ption When Loading Huge file into MemoryStream

Hello,

I am the sample FPSEPublish
(http://blog.baeke.info/blog/_archive.../3/393158.html) code to upload a
document to Sharepoint (WSS). This works perfectly for samll documents.

Problem:

When I attempt to upload a huge document (300Megabayte) on a PC with 3.6Gig
RAM I am getting a OutOfMemoryExce ption when the program attempts to read the
file into a MemoryStream.

The problem has nothing to do with Sharepoint as it is happening prior to
the FP RPC upload atempt.

Questions: Is there a limit on the amount of memory that is assigned to the
GarbageCollecte d heap that is being reached? Using the Sysinternals
ProcessExplorer program I can see the amount of memory being consumed by the
process as follows:

Total Reserved Bytes: 897,544,192

I have tried to understand how the CLR is allocating memory to no avail.

Any help/pointer is greatly appreciated

Sample Code:

+++++++++++++++ +++++++++++++++ +++++
public void PutDocument(str ing uri, string fileName, string metaInfo)
{
//GC.Collect();
//GC.WaitForPendi ngFinalizers();
//GC.Collect();
Uri myUri = new Uri(uri);
string webUrl, fileUrl;
UrlToWebUrl(uri , out webUrl, out fileUrl);

if (null == metaInfo)
metaInfo = "";

if (!File.Exists(f ileName))
throw new Exception("Coul d not find file" + fileName);

string postBody = String.Format(
"method=put+doc ument&service_n ame=&document=[document_name={ 0};meta_info=[{1}]]&put_option=ove rwrite&comment= &keep_checked_o ut=false\n",
HttpUtility.Url Encode(fileUrl) ,
metaInfo);

ASCIIEncoding encoding = new ASCIIEncoding() ;
MemoryStream stream = new MemoryStream();
stream.Write(en coding.GetBytes (postBody), 0, postBody.Length );

FileStream fs = File.OpenRead(f ileName);
byte[] b = new byte[4096];
while (fs.Read(b, 0, b.Length) > 0)
{
//GC.Collect();
//GC.WaitForPendi ngFinalizers();
//GC.Collect();

try
{
stream.Write(b, 0, b.Length); //FAILS TO WRITE TO STREAM AFTER STREAM
SIZE REACHES APPROX 270Meg
//System.Diagnost ics.Debug.Write Line(stream.Len gth.ToString()) ;
}
catch(Exception ex)
{
System.Diagnost ics.Trace.Write Line(ex.Message );
}
}
fs.Close();

SendRequest(myU ri.GetLeftPart( UriPartial.Auth ority) + webUrl +
"/_vti_bin/_vti_aut/author.dll", stream.GetBuffe r(), stream.Length);
stream.Close();
}
+++++++++++++++ +++++++++++++++ +++++
Jan 9 '06 #1
5 26236


"Naamat" <si****@newsgro up.nospam> wrote in message
news:EB******** *************** ***********@mic rosoft.com...
| Hello,
|
| I am the sample FPSEPublish
| (http://blog.baeke.info/blog/_archive.../3/393158.html) code to
upload a
| document to Sharepoint (WSS). This works perfectly for samll documents.
|
| Problem:
|
| When I attempt to upload a huge document (300Megabayte) on a PC with
3.6Gig
| RAM I am getting a OutOfMemoryExce ption when the program attempts to read
the
| file into a MemoryStream.
|
| The problem has nothing to do with Sharepoint as it is happening prior to
| the FP RPC upload atempt.
|
| Questions: Is there a limit on the amount of memory that is assigned to
the
| GarbageCollecte d heap that is being reached? Using the Sysinternals
| ProcessExplorer program I can see the amount of memory being consumed by
the
| process as follows:
|
| Total Reserved Bytes: 897,544,192
|
| I have tried to understand how the CLR is allocating memory to no avail.
|
| Any help/pointer is greatly appreciated
|
| Sample Code:
|
| +++++++++++++++ +++++++++++++++ +++++
| public void PutDocument(str ing uri, string fileName, string metaInfo)
| {
| //GC.Collect();
| //GC.WaitForPendi ngFinalizers();
| //GC.Collect();
|
|
| Uri myUri = new Uri(uri);
| string webUrl, fileUrl;
| UrlToWebUrl(uri , out webUrl, out fileUrl);
|
| if (null == metaInfo)
| metaInfo = "";
|
| if (!File.Exists(f ileName))
| throw new Exception("Coul d not find file" + fileName);
|
| string postBody = String.Format(
|
"method=put+doc ument&service_n ame=&document=[document_name={ 0};meta_info=[{1}]]&put_option=ove rwrite&comment= &keep_checked_o ut=false\n",
| HttpUtility.Url Encode(fileUrl) ,
| metaInfo);
|
| ASCIIEncoding encoding = new ASCIIEncoding() ;
| MemoryStream stream = new MemoryStream();
| stream.Write(en coding.GetBytes (postBody), 0, postBody.Length );
|
| FileStream fs = File.OpenRead(f ileName);
| byte[] b = new byte[4096];
| while (fs.Read(b, 0, b.Length) > 0)
| {
| //GC.Collect();
| //GC.WaitForPendi ngFinalizers();
| //GC.Collect();
|
| try
| {
| stream.Write(b, 0, b.Length); //FAILS TO WRITE TO STREAM AFTER STREAM
| SIZE REACHES APPROX 270Meg
| //System.Diagnost ics.Debug.Write Line(stream.Len gth.ToString()) ;
| }
| catch(Exception ex)
| {
| System.Diagnost ics.Trace.Write Line(ex.Message );
| }
| }
| fs.Close();
|
| SendRequest(myU ri.GetLeftPart( UriPartial.Auth ority) + webUrl +
| "/_vti_bin/_vti_aut/author.dll", stream.GetBuffe r(), stream.Length);
| stream.Close();
| }
|
|
| +++++++++++++++ +++++++++++++++ +++++

|
| Total Reserved Bytes: 897,544,192

This doesn't mean a lot, but is this the reserved bytes before you attempted
the load?
Anyway, the exception means that you don't have a free contigious area of
270MB available in your process 2GB Virtual address space.
Also keep in mind that strings in .NET are really System.Char arrays, where
a char is 16 bit. So if your doc is ASCII encoded, it will take twice the
size of the file in memory.

Willy.

Jan 9 '06 #2
Willy Denoyette [MVP] <wi************ *@telenet.be> wrote:

<snip>
Also keep in mind that strings in .NET are really System.Char arrays, where
a char is 16 bit. So if your doc is ASCII encoded, it will take twice the
size of the file in memory.


Where is the file being treated as a string? The ASCII encoding is only
used for the URL, as far as I can see.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Jan 9 '06 #3
Naamat <si****@newsgro up.nospam> wrote:
I am the sample FPSEPublish
(http://blog.baeke.info/blog/_archive.../3/393158.html) code to upload a
document to Sharepoint (WSS). This works perfectly for samll documents.


No it doesn't, unfortunately.

Aside from the memory problems you're getting, the code is actually
wrong, and stylistically flawed too (I'm looking at the same code you
pointed at).

1) The code is always trying to read 4K at a time, and then *always*
writing 4K whether it's read 1 byte of 4K. I suggest you have a look at
http://www.pobox.com/~skeet/csharp/readbinary.html

2) It should also have a "using" statement around the FileStream (and
preferrably the MemoryStream too, for consistency) to make sure it
always gets disposed of when it's finished with, whether or not there
was an exception. (This is a fault in other methods too.)

3) Not an error, so much as bad style: It's using the
if (constant==vari able) style which is a hangover from the bad-old days
of C, where if you wrote if (variable=const ant) you wouldn't get a
compiler error. The idiom leads to more reliable coding in C, but at
the expense of a bit of readability.

4) In SendRequest(str ing uri, byte[] postBody, long postLength),
there's a lot of pointless looping around, writing 4K at a time, with
no indication as to why he doesn't just write the whole lot in one go.
Now, as to why you're running out of memory... I suspect the problem is
with how the MemoryStream is growing. You could try creating the
MemoryStream with enough space for the initial piece of text and
everything you need from the post body's data (using Stream.Length).
That would avoid MemoryStream having to copy everything each time it
resizes.

However, a better solution would be to refactor the code to avoid
building up the MemoryStream in the first place. There's no reason why
there shouldn't be a version of SendRequest which takes a stream to
write stream to the request object. Now, whether that actually streams
it to the client or whether it buffers it all up in memory anyway is
another matter, but it would be saving at least one 300M+ buffer!

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Jan 9 '06 #4

"Jon Skeet [C# MVP]" <sk***@pobox.co m> wrote in message
news:MP******** *************** *@msnews.micros oft.com...
| Willy Denoyette [MVP] <wi************ *@telenet.be> wrote:
|
| <snip>
|
| > Also keep in mind that strings in .NET are really System.Char arrays,
where
| > a char is 16 bit. So if your doc is ASCII encoded, it will take twice
the
| > size of the file in memory.
|
| Where is the file being treated as a string? The ASCII encoding is only
| used for the URL, as far as I can see.
|
|
Not in this piece of code, but I guess that's not all there is running. It
was just a warning, some may forget that when they threat the stream a
strings they take twice the size.

Willy.

Jan 10 '06 #5
Hi Simeon,

I just wanted to check how things are going. If there is any question,
please feel free to join the community and we are here to support you at
your convenience. Thanks for your understanding!

Best Regards,

Terry Fei[MSFT]
Microsoft Community Support
Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)
Jan 13 '06 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

24
2722
by: jrefactors | last post by:
I have an upload file operation in the web application. UploadForm.jsp is the form, and UploadAction.jsp is the form processing. The web server is Websphere. //UploadForm.jsp <FORM NAME="InputForm" ACTION="UploadAction.jsp" METHOD="POST" enctype=multipart/form-data> <input type="file" name="fileName"> //etc ...
2
1710
by: Helen | last post by:
Guys and gals, I am having a problem with an IFRAME that is embedded into an ASPX page (ASP.NET application). The IFRAME is originally empty (src=''). When a user selects a date from the drop-down list box above the IFRAME, the IFRAME loads the content of a file on the local machine (c:\bds\blahblahblah.txt). The application that...
2
2510
by: Jon | last post by:
Hi, I am testing the scenario to migrate from MSMXL4 to .NET2.0 system.xml. I experienced xsl file loading problem whenever the xsl file have a user defined script in it. The error message is as follow: "Objects of type 'Script2' do not have such a member" The above error only happens when I set the XsltSettings.TrustedXslt ans use the...
2
2175
by: hazz | last post by:
With this code, the config file(listed below) opens initially into the datagrid display with only a plus sign '+'. On clicking that, it becomes expanded to; - appsettings add When I click on appsettings, it reverts back to '+' On clicking that, - appsettings add is displayed Finally, when I click on appsettings this time, the contents...
5
2221
by: Bill Q | last post by:
Hello, this may not be the correct group for the post although I am using c# to follow one of the code4fun directx tutorial. Anyway the author is loading a texture file using the following TextureLoader.FromFile ( _device, @"..\..\..\Resources\Left.tga" I am not sure what the ..\..\..\ means when loading the file. I know the directory...
4
5789
by: Daniel | last post by:
is there some per-process-limit on memory in .net processes? is there any way to increase it? i keep getting System.OutOfMemoryException when my box has 8 gigs of unused memory.
3
2366
by: powwow | last post by:
I am running out of memory (OutOfMemoryException) when I do something like this: Hashtable h = new Hashtable(10000000); It doesn't seem to use all of the RAM on the machine. I have 4GB of RAM installed. I have a 64 bit OS (Windows Vista). I have 64 bit CPU. (Intel Quad Core). So no 32-bit restriction argument applies here.
2
1286
by: joe | last post by:
I am loading a text file to a variable with XMLHttpRequest() There seems to be some sort of timing issue since loadXML (source code below) returns the contents of the file on seconds try. In Firefox I get an empty string. No errors are reported. If I look at the code with Firefox debugger (Venkman) everything works fine. I should probably...
1
2321
by: daonho | last post by:
Hi Everyone, I have encountered this problem that I am not able to figure out. Please drop me some lines if you have any idea how to solve this issue. I have a global.asax file in my web site. It works fine as I wanted when loading aspx file; however, I have some static page in the same directory such as html and htm page. These page also load...
0
7666
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7584
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
7888
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
8108
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
0
7951
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
0
6260
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
1
5484
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
5213
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
1
1201
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.