473,698 Members | 2,156 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

OutOfMemoryExce ption When Loading Huge file into MemoryStream

Hello,

I am the sample FPSEPublish
(http://blog.baeke.info/blog/_archive.../3/393158.html) code to upload a
document to Sharepoint (WSS). This works perfectly for samll documents.

Problem:

When I attempt to upload a huge document (300Megabayte) on a PC with 3.6Gig
RAM I am getting a OutOfMemoryExce ption when the program attempts to read the
file into a MemoryStream.

The problem has nothing to do with Sharepoint as it is happening prior to
the FP RPC upload atempt.

Questions: Is there a limit on the amount of memory that is assigned to the
GarbageCollecte d heap that is being reached? Using the Sysinternals
ProcessExplorer program I can see the amount of memory being consumed by the
process as follows:

Total Reserved Bytes: 897,544,192

I have tried to understand how the CLR is allocating memory to no avail.

Any help/pointer is greatly appreciated

Sample Code:

+++++++++++++++ +++++++++++++++ +++++
public void PutDocument(str ing uri, string fileName, string metaInfo)
{
//GC.Collect();
//GC.WaitForPendi ngFinalizers();
//GC.Collect();
Uri myUri = new Uri(uri);
string webUrl, fileUrl;
UrlToWebUrl(uri , out webUrl, out fileUrl);

if (null == metaInfo)
metaInfo = "";

if (!File.Exists(f ileName))
throw new Exception("Coul d not find file" + fileName);

string postBody = String.Format(
"method=put+doc ument&service_n ame=&document=[document_name={ 0};meta_info=[{1}]]&put_option=ove rwrite&comment= &keep_checked_o ut=false\n",
HttpUtility.Url Encode(fileUrl) ,
metaInfo);

ASCIIEncoding encoding = new ASCIIEncoding() ;
MemoryStream stream = new MemoryStream();
stream.Write(en coding.GetBytes (postBody), 0, postBody.Length );

FileStream fs = File.OpenRead(f ileName);
byte[] b = new byte[4096];
while (fs.Read(b, 0, b.Length) > 0)
{
//GC.Collect();
//GC.WaitForPendi ngFinalizers();
//GC.Collect();

try
{
stream.Write(b, 0, b.Length); //FAILS TO WRITE TO STREAM AFTER STREAM
SIZE REACHES APPROX 270Meg
//System.Diagnost ics.Debug.Write Line(stream.Len gth.ToString()) ;
}
catch(Exception ex)
{
System.Diagnost ics.Trace.Write Line(ex.Message );
}
}
fs.Close();

SendRequest(myU ri.GetLeftPart( UriPartial.Auth ority) + webUrl +
"/_vti_bin/_vti_aut/author.dll", stream.GetBuffe r(), stream.Length);
stream.Close();
}
+++++++++++++++ +++++++++++++++ +++++
Jan 9 '06 #1
5 26272


"Naamat" <si****@newsgro up.nospam> wrote in message
news:EB******** *************** ***********@mic rosoft.com...
| Hello,
|
| I am the sample FPSEPublish
| (http://blog.baeke.info/blog/_archive.../3/393158.html) code to
upload a
| document to Sharepoint (WSS). This works perfectly for samll documents.
|
| Problem:
|
| When I attempt to upload a huge document (300Megabayte) on a PC with
3.6Gig
| RAM I am getting a OutOfMemoryExce ption when the program attempts to read
the
| file into a MemoryStream.
|
| The problem has nothing to do with Sharepoint as it is happening prior to
| the FP RPC upload atempt.
|
| Questions: Is there a limit on the amount of memory that is assigned to
the
| GarbageCollecte d heap that is being reached? Using the Sysinternals
| ProcessExplorer program I can see the amount of memory being consumed by
the
| process as follows:
|
| Total Reserved Bytes: 897,544,192
|
| I have tried to understand how the CLR is allocating memory to no avail.
|
| Any help/pointer is greatly appreciated
|
| Sample Code:
|
| +++++++++++++++ +++++++++++++++ +++++
| public void PutDocument(str ing uri, string fileName, string metaInfo)
| {
| //GC.Collect();
| //GC.WaitForPendi ngFinalizers();
| //GC.Collect();
|
|
| Uri myUri = new Uri(uri);
| string webUrl, fileUrl;
| UrlToWebUrl(uri , out webUrl, out fileUrl);
|
| if (null == metaInfo)
| metaInfo = "";
|
| if (!File.Exists(f ileName))
| throw new Exception("Coul d not find file" + fileName);
|
| string postBody = String.Format(
|
"method=put+doc ument&service_n ame=&document=[document_name={ 0};meta_info=[{1}]]&put_option=ove rwrite&comment= &keep_checked_o ut=false\n",
| HttpUtility.Url Encode(fileUrl) ,
| metaInfo);
|
| ASCIIEncoding encoding = new ASCIIEncoding() ;
| MemoryStream stream = new MemoryStream();
| stream.Write(en coding.GetBytes (postBody), 0, postBody.Length );
|
| FileStream fs = File.OpenRead(f ileName);
| byte[] b = new byte[4096];
| while (fs.Read(b, 0, b.Length) > 0)
| {
| //GC.Collect();
| //GC.WaitForPendi ngFinalizers();
| //GC.Collect();
|
| try
| {
| stream.Write(b, 0, b.Length); //FAILS TO WRITE TO STREAM AFTER STREAM
| SIZE REACHES APPROX 270Meg
| //System.Diagnost ics.Debug.Write Line(stream.Len gth.ToString()) ;
| }
| catch(Exception ex)
| {
| System.Diagnost ics.Trace.Write Line(ex.Message );
| }
| }
| fs.Close();
|
| SendRequest(myU ri.GetLeftPart( UriPartial.Auth ority) + webUrl +
| "/_vti_bin/_vti_aut/author.dll", stream.GetBuffe r(), stream.Length);
| stream.Close();
| }
|
|
| +++++++++++++++ +++++++++++++++ +++++

|
| Total Reserved Bytes: 897,544,192

This doesn't mean a lot, but is this the reserved bytes before you attempted
the load?
Anyway, the exception means that you don't have a free contigious area of
270MB available in your process 2GB Virtual address space.
Also keep in mind that strings in .NET are really System.Char arrays, where
a char is 16 bit. So if your doc is ASCII encoded, it will take twice the
size of the file in memory.

Willy.

Jan 9 '06 #2
Willy Denoyette [MVP] <wi************ *@telenet.be> wrote:

<snip>
Also keep in mind that strings in .NET are really System.Char arrays, where
a char is 16 bit. So if your doc is ASCII encoded, it will take twice the
size of the file in memory.


Where is the file being treated as a string? The ASCII encoding is only
used for the URL, as far as I can see.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Jan 9 '06 #3
Naamat <si****@newsgro up.nospam> wrote:
I am the sample FPSEPublish
(http://blog.baeke.info/blog/_archive.../3/393158.html) code to upload a
document to Sharepoint (WSS). This works perfectly for samll documents.


No it doesn't, unfortunately.

Aside from the memory problems you're getting, the code is actually
wrong, and stylistically flawed too (I'm looking at the same code you
pointed at).

1) The code is always trying to read 4K at a time, and then *always*
writing 4K whether it's read 1 byte of 4K. I suggest you have a look at
http://www.pobox.com/~skeet/csharp/readbinary.html

2) It should also have a "using" statement around the FileStream (and
preferrably the MemoryStream too, for consistency) to make sure it
always gets disposed of when it's finished with, whether or not there
was an exception. (This is a fault in other methods too.)

3) Not an error, so much as bad style: It's using the
if (constant==vari able) style which is a hangover from the bad-old days
of C, where if you wrote if (variable=const ant) you wouldn't get a
compiler error. The idiom leads to more reliable coding in C, but at
the expense of a bit of readability.

4) In SendRequest(str ing uri, byte[] postBody, long postLength),
there's a lot of pointless looping around, writing 4K at a time, with
no indication as to why he doesn't just write the whole lot in one go.
Now, as to why you're running out of memory... I suspect the problem is
with how the MemoryStream is growing. You could try creating the
MemoryStream with enough space for the initial piece of text and
everything you need from the post body's data (using Stream.Length).
That would avoid MemoryStream having to copy everything each time it
resizes.

However, a better solution would be to refactor the code to avoid
building up the MemoryStream in the first place. There's no reason why
there shouldn't be a version of SendRequest which takes a stream to
write stream to the request object. Now, whether that actually streams
it to the client or whether it buffers it all up in memory anyway is
another matter, but it would be saving at least one 300M+ buffer!

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Jan 9 '06 #4

"Jon Skeet [C# MVP]" <sk***@pobox.co m> wrote in message
news:MP******** *************** *@msnews.micros oft.com...
| Willy Denoyette [MVP] <wi************ *@telenet.be> wrote:
|
| <snip>
|
| > Also keep in mind that strings in .NET are really System.Char arrays,
where
| > a char is 16 bit. So if your doc is ASCII encoded, it will take twice
the
| > size of the file in memory.
|
| Where is the file being treated as a string? The ASCII encoding is only
| used for the URL, as far as I can see.
|
|
Not in this piece of code, but I guess that's not all there is running. It
was just a warning, some may forget that when they threat the stream a
strings they take twice the size.

Willy.

Jan 10 '06 #5
Hi Simeon,

I just wanted to check how things are going. If there is any question,
please feel free to join the community and we are here to support you at
your convenience. Thanks for your understanding!

Best Regards,

Terry Fei[MSFT]
Microsoft Community Support
Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)
Jan 13 '06 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

24
2745
by: jrefactors | last post by:
I have an upload file operation in the web application. UploadForm.jsp is the form, and UploadAction.jsp is the form processing. The web server is Websphere. //UploadForm.jsp <FORM NAME="InputForm" ACTION="UploadAction.jsp" METHOD="POST" enctype=multipart/form-data> <input type="file" name="fileName"> //etc ...
2
1717
by: Helen | last post by:
Guys and gals, I am having a problem with an IFRAME that is embedded into an ASPX page (ASP.NET application). The IFRAME is originally empty (src=''). When a user selects a date from the drop-down list box above the IFRAME, the IFRAME loads the content of a file on the local machine (c:\bds\blahblahblah.txt). The application that contains this page is running on a secure socket layer (SSL). When the IFRAME tries to load the file, a...
2
2518
by: Jon | last post by:
Hi, I am testing the scenario to migrate from MSMXL4 to .NET2.0 system.xml. I experienced xsl file loading problem whenever the xsl file have a user defined script in it. The error message is as follow: "Objects of type 'Script2' do not have such a member" The above error only happens when I set the XsltSettings.TrustedXslt ans use the following statement to load xsl file:
2
2178
by: hazz | last post by:
With this code, the config file(listed below) opens initially into the datagrid display with only a plus sign '+'. On clicking that, it becomes expanded to; - appsettings add When I click on appsettings, it reverts back to '+' On clicking that, - appsettings add is displayed Finally, when I click on appsettings this time, the contents of the config
5
2228
by: Bill Q | last post by:
Hello, this may not be the correct group for the post although I am using c# to follow one of the code4fun directx tutorial. Anyway the author is loading a texture file using the following TextureLoader.FromFile ( _device, @"..\..\..\Resources\Left.tga" I am not sure what the ..\..\..\ means when loading the file. I know the directory the file lives in is C:\Documents and Settings\Bill\My Documents\MSDN\BattleTank2005 (C#)\Resources
4
5797
by: Daniel | last post by:
is there some per-process-limit on memory in .net processes? is there any way to increase it? i keep getting System.OutOfMemoryException when my box has 8 gigs of unused memory.
3
2370
by: powwow | last post by:
I am running out of memory (OutOfMemoryException) when I do something like this: Hashtable h = new Hashtable(10000000); It doesn't seem to use all of the RAM on the machine. I have 4GB of RAM installed. I have a 64 bit OS (Windows Vista). I have 64 bit CPU. (Intel Quad Core). So no 32-bit restriction argument applies here.
2
1290
by: joe | last post by:
I am loading a text file to a variable with XMLHttpRequest() There seems to be some sort of timing issue since loadXML (source code below) returns the contents of the file on seconds try. In Firefox I get an empty string. No errors are reported. If I look at the code with Firefox debugger (Venkman) everything works fine. I should probably put a loop somewhere to check when loading the file is finished. var xmlhttp;
1
2339
by: daonho | last post by:
Hi Everyone, I have encountered this problem that I am not able to figure out. Please drop me some lines if you have any idea how to solve this issue. I have a global.asax file in my web site. It works fine as I wanted when loading aspx file; however, I have some static page in the same directory such as html and htm page. These page also load correctly and the global.asax file does fire when executing my .net development environment....
0
8674
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
8603
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9157
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9027
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
8861
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
6518
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
4369
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
3046
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2329
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.