473,413 Members | 1,679 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,413 software developers and data experts.

OutOfMemoryException When Loading Huge file into MemoryStream

Hello,

I am the sample FPSEPublish
(http://blog.baeke.info/blog/_archive.../3/393158.html) code to upload a
document to Sharepoint (WSS). This works perfectly for samll documents.

Problem:

When I attempt to upload a huge document (300Megabayte) on a PC with 3.6Gig
RAM I am getting a OutOfMemoryException when the program attempts to read the
file into a MemoryStream.

The problem has nothing to do with Sharepoint as it is happening prior to
the FP RPC upload atempt.

Questions: Is there a limit on the amount of memory that is assigned to the
GarbageCollected heap that is being reached? Using the Sysinternals
ProcessExplorer program I can see the amount of memory being consumed by the
process as follows:

Total Reserved Bytes: 897,544,192

I have tried to understand how the CLR is allocating memory to no avail.

Any help/pointer is greatly appreciated

Sample Code:

+++++++++++++++++++++++++++++++++++
public void PutDocument(string uri, string fileName, string metaInfo)
{
//GC.Collect();
//GC.WaitForPendingFinalizers();
//GC.Collect();
Uri myUri = new Uri(uri);
string webUrl, fileUrl;
UrlToWebUrl(uri, out webUrl, out fileUrl);

if (null == metaInfo)
metaInfo = "";

if (!File.Exists(fileName))
throw new Exception("Could not find file" + fileName);

string postBody = String.Format(
"method=put+document&service_name=&document=[document_name={0};meta_info=[{1}]]&put_option=overwrite&comment=&keep_checked_out=fa lse\n",
HttpUtility.UrlEncode(fileUrl),
metaInfo);

ASCIIEncoding encoding = new ASCIIEncoding();
MemoryStream stream = new MemoryStream();
stream.Write(encoding.GetBytes(postBody), 0, postBody.Length);

FileStream fs = File.OpenRead(fileName);
byte[] b = new byte[4096];
while (fs.Read(b, 0, b.Length) > 0)
{
//GC.Collect();
//GC.WaitForPendingFinalizers();
//GC.Collect();

try
{
stream.Write(b, 0, b.Length); //FAILS TO WRITE TO STREAM AFTER STREAM
SIZE REACHES APPROX 270Meg
//System.Diagnostics.Debug.WriteLine(stream.Length.T oString());
}
catch(Exception ex)
{
System.Diagnostics.Trace.WriteLine(ex.Message);
}
}
fs.Close();

SendRequest(myUri.GetLeftPart(UriPartial.Authority ) + webUrl +
"/_vti_bin/_vti_aut/author.dll", stream.GetBuffer(), stream.Length);
stream.Close();
}
+++++++++++++++++++++++++++++++++++
Jan 9 '06 #1
5 26205


"Naamat" <si****@newsgroup.nospam> wrote in message
news:EB**********************************@microsof t.com...
| Hello,
|
| I am the sample FPSEPublish
| (http://blog.baeke.info/blog/_archive.../3/393158.html) code to
upload a
| document to Sharepoint (WSS). This works perfectly for samll documents.
|
| Problem:
|
| When I attempt to upload a huge document (300Megabayte) on a PC with
3.6Gig
| RAM I am getting a OutOfMemoryException when the program attempts to read
the
| file into a MemoryStream.
|
| The problem has nothing to do with Sharepoint as it is happening prior to
| the FP RPC upload atempt.
|
| Questions: Is there a limit on the amount of memory that is assigned to
the
| GarbageCollected heap that is being reached? Using the Sysinternals
| ProcessExplorer program I can see the amount of memory being consumed by
the
| process as follows:
|
| Total Reserved Bytes: 897,544,192
|
| I have tried to understand how the CLR is allocating memory to no avail.
|
| Any help/pointer is greatly appreciated
|
| Sample Code:
|
| +++++++++++++++++++++++++++++++++++
| public void PutDocument(string uri, string fileName, string metaInfo)
| {
| //GC.Collect();
| //GC.WaitForPendingFinalizers();
| //GC.Collect();
|
|
| Uri myUri = new Uri(uri);
| string webUrl, fileUrl;
| UrlToWebUrl(uri, out webUrl, out fileUrl);
|
| if (null == metaInfo)
| metaInfo = "";
|
| if (!File.Exists(fileName))
| throw new Exception("Could not find file" + fileName);
|
| string postBody = String.Format(
|
"method=put+document&service_name=&document=[document_name={0};meta_info=[{1}]]&put_option=overwrite&comment=&keep_checked_out=fa lse\n",
| HttpUtility.UrlEncode(fileUrl),
| metaInfo);
|
| ASCIIEncoding encoding = new ASCIIEncoding();
| MemoryStream stream = new MemoryStream();
| stream.Write(encoding.GetBytes(postBody), 0, postBody.Length);
|
| FileStream fs = File.OpenRead(fileName);
| byte[] b = new byte[4096];
| while (fs.Read(b, 0, b.Length) > 0)
| {
| //GC.Collect();
| //GC.WaitForPendingFinalizers();
| //GC.Collect();
|
| try
| {
| stream.Write(b, 0, b.Length); //FAILS TO WRITE TO STREAM AFTER STREAM
| SIZE REACHES APPROX 270Meg
| //System.Diagnostics.Debug.WriteLine(stream.Length.T oString());
| }
| catch(Exception ex)
| {
| System.Diagnostics.Trace.WriteLine(ex.Message);
| }
| }
| fs.Close();
|
| SendRequest(myUri.GetLeftPart(UriPartial.Authority ) + webUrl +
| "/_vti_bin/_vti_aut/author.dll", stream.GetBuffer(), stream.Length);
| stream.Close();
| }
|
|
| +++++++++++++++++++++++++++++++++++

|
| Total Reserved Bytes: 897,544,192

This doesn't mean a lot, but is this the reserved bytes before you attempted
the load?
Anyway, the exception means that you don't have a free contigious area of
270MB available in your process 2GB Virtual address space.
Also keep in mind that strings in .NET are really System.Char arrays, where
a char is 16 bit. So if your doc is ASCII encoded, it will take twice the
size of the file in memory.

Willy.

Jan 9 '06 #2
Willy Denoyette [MVP] <wi*************@telenet.be> wrote:

<snip>
Also keep in mind that strings in .NET are really System.Char arrays, where
a char is 16 bit. So if your doc is ASCII encoded, it will take twice the
size of the file in memory.


Where is the file being treated as a string? The ASCII encoding is only
used for the URL, as far as I can see.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Jan 9 '06 #3
Naamat <si****@newsgroup.nospam> wrote:
I am the sample FPSEPublish
(http://blog.baeke.info/blog/_archive.../3/393158.html) code to upload a
document to Sharepoint (WSS). This works perfectly for samll documents.


No it doesn't, unfortunately.

Aside from the memory problems you're getting, the code is actually
wrong, and stylistically flawed too (I'm looking at the same code you
pointed at).

1) The code is always trying to read 4K at a time, and then *always*
writing 4K whether it's read 1 byte of 4K. I suggest you have a look at
http://www.pobox.com/~skeet/csharp/readbinary.html

2) It should also have a "using" statement around the FileStream (and
preferrably the MemoryStream too, for consistency) to make sure it
always gets disposed of when it's finished with, whether or not there
was an exception. (This is a fault in other methods too.)

3) Not an error, so much as bad style: It's using the
if (constant==variable) style which is a hangover from the bad-old days
of C, where if you wrote if (variable=constant) you wouldn't get a
compiler error. The idiom leads to more reliable coding in C, but at
the expense of a bit of readability.

4) In SendRequest(string uri, byte[] postBody, long postLength),
there's a lot of pointless looping around, writing 4K at a time, with
no indication as to why he doesn't just write the whole lot in one go.
Now, as to why you're running out of memory... I suspect the problem is
with how the MemoryStream is growing. You could try creating the
MemoryStream with enough space for the initial piece of text and
everything you need from the post body's data (using Stream.Length).
That would avoid MemoryStream having to copy everything each time it
resizes.

However, a better solution would be to refactor the code to avoid
building up the MemoryStream in the first place. There's no reason why
there shouldn't be a version of SendRequest which takes a stream to
write stream to the request object. Now, whether that actually streams
it to the client or whether it buffers it all up in memory anyway is
another matter, but it would be saving at least one 300M+ buffer!

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Jan 9 '06 #4

"Jon Skeet [C# MVP]" <sk***@pobox.com> wrote in message
news:MP************************@msnews.microsoft.c om...
| Willy Denoyette [MVP] <wi*************@telenet.be> wrote:
|
| <snip>
|
| > Also keep in mind that strings in .NET are really System.Char arrays,
where
| > a char is 16 bit. So if your doc is ASCII encoded, it will take twice
the
| > size of the file in memory.
|
| Where is the file being treated as a string? The ASCII encoding is only
| used for the URL, as far as I can see.
|
|
Not in this piece of code, but I guess that's not all there is running. It
was just a warning, some may forget that when they threat the stream a
strings they take twice the size.

Willy.

Jan 10 '06 #5
Hi Simeon,

I just wanted to check how things are going. If there is any question,
please feel free to join the community and we are here to support you at
your convenience. Thanks for your understanding!

Best Regards,

Terry Fei[MSFT]
Microsoft Community Support
Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)
Jan 13 '06 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

24
by: jrefactors | last post by:
I have an upload file operation in the web application. UploadForm.jsp is the form, and UploadAction.jsp is the form processing. The web server is Websphere. //UploadForm.jsp <FORM...
2
by: Helen | last post by:
Guys and gals, I am having a problem with an IFRAME that is embedded into an ASPX page (ASP.NET application). The IFRAME is originally empty (src=''). When a user selects a date from the...
2
by: Jon | last post by:
Hi, I am testing the scenario to migrate from MSMXL4 to .NET2.0 system.xml. I experienced xsl file loading problem whenever the xsl file have a user defined script in it. The error message is as...
2
by: hazz | last post by:
With this code, the config file(listed below) opens initially into the datagrid display with only a plus sign '+'. On clicking that, it becomes expanded to; - appsettings add When I click on...
5
by: Bill Q | last post by:
Hello, this may not be the correct group for the post although I am using c# to follow one of the code4fun directx tutorial. Anyway the author is loading a texture file using the following ...
4
by: Daniel | last post by:
is there some per-process-limit on memory in .net processes? is there any way to increase it? i keep getting System.OutOfMemoryException when my box has 8 gigs of unused memory.
3
by: powwow | last post by:
I am running out of memory (OutOfMemoryException) when I do something like this: Hashtable h = new Hashtable(10000000); It doesn't seem to use all of the RAM on the machine. I have 4GB of...
2
by: joe | last post by:
I am loading a text file to a variable with XMLHttpRequest() There seems to be some sort of timing issue since loadXML (source code below) returns the contents of the file on seconds try. In...
1
by: daonho | last post by:
Hi Everyone, I have encountered this problem that I am not able to figure out. Please drop me some lines if you have any idea how to solve this issue. I have a global.asax file in my web site. It...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.