473,326 Members | 2,813 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

System.Net.ScatterGatherBuffers.MemoryChuck allocates large byte[]

Summary: System.Net.ScatterGatherBuffers.MemoryChuck allocates inordinately
large byte[]s when sending large post data.

The following application consumes inordinate quantities of memory. My code
does not explicitly allocate memory in a loop nor does it explicitly allocate
large blocks of memory. Yet, the application’s memory footprint will grow as
large as 370 MB. Rarely will it run to completion; usually, it throws an out
of memory exception or locks up just before sending the packet that would
total 512 MBs.

SciTech .NET Memory Profiler, reports that
System.Net.ScatterGatherBuffers.MemoryChuck is holding onto very large blocks
of memory, specifically byte[]s, which it appears to have allocated in a
doubling fashion. For example, the sizes of these blocks are 226MB, 113MB,
56.5MB, 28MB, 14MB, and so on down. It’s as if
System.Net.ScatterGatherBuffers is allocating a byte of memory for every byte
I send on the socket, but is allocating increasingly large buffers to store
this.

If I send a smaller amount of data, the send completes, but the
System.Net.ScatterGatherBuffers.MemoryChuck byte[]s are not released when the
socket stream is closed, all references to the WebRequest and its stream are
released, and a garbage collection is explicitly requested.

I’ve also tried performing the socket communication on the UI message pump
thread without seeing a different behavior.

I’m running Windows 2000 on a box with 512 MB of memory. I ran this test on
..NET Fx 1.1 SP1. I haven’t tried any other framework versions.

Am I doing something wrong? Is this a known issue? Do you have any
suggestions or explanations?

Thanks in advance for any help you can provide.
using System;
using System.Drawing;
using System.Collections;
using System.ComponentModel;
using System.Windows.Forms;
using System.Data;
using System.Net;
using System.IO;
using System.Threading;

namespace SocketMemoryLeak {
public class Form1 : System.Windows.Forms.Form {
private const int KB = 1024;
private const int MB = KB * KB;
private System.Windows.Forms.Button btnSend;
private System.ComponentModel.Container components = null;
private System.Windows.Forms.Label lblMBsSent;
private System.Windows.Forms.Label lblGCed;
private System.Windows.Forms.Label lblMemoryAllocated;

public Form1() {
InitializeComponent();
}

protected override void Dispose(bool disposing) {
if (disposing) {
if (components != null) {
components.Dispose();
}
}
base.Dispose(disposing);
}

#region Windows Form Designer generated code
private void InitializeComponent() {
this.btnSend = new System.Windows.Forms.Button();
this.lblMBsSent = new System.Windows.Forms.Label();
this.lblGCed = new System.Windows.Forms.Label();
this.lblMemoryAllocated = new System.Windows.Forms.Label();
this.SuspendLayout();
//
// btnSend
//
this.btnSend.Anchor =
((System.Windows.Forms.AnchorStyles)(((System.Wind ows.Forms.AnchorStyles.Top
| System.Windows.Forms.AnchorStyles.Left)
| System.Windows.Forms.AnchorStyles.Right)));
this.btnSend.Location = new System.Drawing.Point(16, 8);
this.btnSend.Name = "btnSend";
this.btnSend.Size = new System.Drawing.Size(184, 23);
this.btnSend.TabIndex = 0;
this.btnSend.Text = "Start";
this.btnSend.Click += new System.EventHandler(this.btnSend_Click);
//
// lblMBsSent
//
this.lblMBsSent.Anchor =
((System.Windows.Forms.AnchorStyles)(((System.Wind ows.Forms.AnchorStyles.Top
| System.Windows.Forms.AnchorStyles.Left)
| System.Windows.Forms.AnchorStyles.Right)));
this.lblMBsSent.Location = new System.Drawing.Point(16, 40);
this.lblMBsSent.Name = "lblMBsSent";
this.lblMBsSent.Size = new System.Drawing.Size(184, 23);
this.lblMBsSent.TabIndex = 1;
this.lblMBsSent.Text = "Sent: 0 MBs";
this.lblMBsSent.TextAlign =
System.Drawing.ContentAlignment.MiddleLeft;
//
// lblGCed
//
this.lblGCed.Anchor =
((System.Windows.Forms.AnchorStyles)(((System.Wind ows.Forms.AnchorStyles.Top
| System.Windows.Forms.AnchorStyles.Left)
| System.Windows.Forms.AnchorStyles.Right)));
this.lblGCed.Location = new System.Drawing.Point(16, 88);
this.lblGCed.Name = "lblGCed";
this.lblGCed.Size = new System.Drawing.Size(184, 23);
this.lblGCed.TabIndex = 2;
this.lblGCed.Text = "No previous garbage collection";
this.lblGCed.TextAlign =
System.Drawing.ContentAlignment.MiddleLeft;
//
// lblMemoryAllocated
//
this.lblMemoryAllocated.Anchor =
((System.Windows.Forms.AnchorStyles)(((System.Wind ows.Forms.AnchorStyles.Top
| System.Windows.Forms.AnchorStyles.Left)
| System.Windows.Forms.AnchorStyles.Right)));
this.lblMemoryAllocated.Location = new System.Drawing.Point(16,
64);
this.lblMemoryAllocated.Name = "lblMemoryAllocated";
this.lblMemoryAllocated.Size = new System.Drawing.Size(184, 23);
this.lblMemoryAllocated.TabIndex = 3;
this.lblMemoryAllocated.Text = "Memory Allocated: 0 MB";
this.lblMemoryAllocated.TextAlign =
System.Drawing.ContentAlignment.MiddleLeft;
//
// Form1
//
this.AutoScaleBaseSize = new System.Drawing.Size(5, 13);
this.ClientSize = new System.Drawing.Size(208, 117);
this.Controls.Add(this.lblMemoryAllocated);
this.Controls.Add(this.lblGCed);
this.Controls.Add(this.lblMBsSent);
this.Controls.Add(this.btnSend);
this.Name = "Form1";
this.Text = "Form1";
this.ResumeLayout(false);

}
#endregion

[STAThread]
static void Main() {
Application.Run(new Form1());
}

private void btnSend_Click(object sender, System.EventArgs e) {
this.btnSend.Enabled = false;
Thread t = new Thread(new ThreadStart(this.Send));
t.IsBackground = true;
t.Start();
}

private void DoGC(int bytesSent) {
this.lblGCed.Text = "Starting GC...";
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
this.lblGCed.Text = "GC'd at " + (bytesSent / MB) + " MB";
}

private void SetSent(int bytesSent) {
this.lblMBsSent.Text = "Sent: " + (bytesSent / MB) + " MB";
this.lblMemoryAllocated.Text = "Memory Allocated: " +
(Environment.WorkingSet / MB) + " MB";
}

private void Send() {
byte[] buffer = new byte[16 * KB];
int totalBytesToSend = 514 * MB;

HttpWebRequest request = (HttpWebRequest)
WebRequest.Create("http://marias/cgi-bin/leif.cfg/php/enduser/fileUpload.php");
request.Method = "POST";
request.ContentLength = totalBytesToSend;
Stream requestStream = request.GetRequestStream();

int bytesSent = 0;
int gcInterval = 64 * MB;
int displayInterval = 1 * MB;
while (bytesSent < totalBytesToSend) {
requestStream.Write(buffer, 0, buffer.Length);
bytesSent += buffer.Length;
if (bytesSent % displayInterval == 0) {
this.SetSent(bytesSent);
}
if (bytesSent % gcInterval == 0) {
this.DoGC(bytesSent);
}
}

requestStream.Flush();
requestStream.Close();
requestStream = null;
request = null;

this.DoGC(bytesSent);
MessageBox.Show("Send thread ending.");
this.btnSend.Enabled = true;
}
}
}
Nov 16 '05 #1
1 3961
this is because httpwebrequest buffers data, as it migt need to repost
it in so9me cases.

you can disable this by setting AllowWriteStreamBuffering=false

-----
feroze

This posting is provided as-is

http://weblogs.asp.net/feroze_daud
lwickland <lw*******@discussions.microsoft.com> wrote in message news:<F5**********************************@microso ft.com>...
Summary: System.Net.ScatterGatherBuffers.MemoryChuck allocates inordinately
large byte[]s when sending large post data.

The following application consumes inordinate quantities of memory. My code
does not explicitly allocate memory in a loop nor does it explicitly allocate
large blocks of memory. Yet, the application’s memory footprint will grow as
large as 370 MB. Rarely will it run to completion; usually, it throws an out
of memory exception or locks up just before sending the packet that would
total 512 MBs.

SciTech .NET Memory Profiler, reports that
System.Net.ScatterGatherBuffers.MemoryChuck is holding onto very large blocks
of memory, specifically byte[]s, which it appears to have allocated in a
doubling fashion. For example, the sizes of these blocks are 226MB, 113MB,
56.5MB, 28MB, 14MB, and so on down. It’s as if
System.Net.ScatterGatherBuffers is allocating a byte of memory for every byte
I send on the socket, but is allocating increasingly large buffers to store
this.

If I send a smaller amount of data, the send completes, but the
System.Net.ScatterGatherBuffers.MemoryChuck byte[]s are not released when the
socket stream is closed, all references to the WebRequest and its stream are
released, and a garbage collection is explicitly requested.

I’ve also tried performing the socket communication on the UI message pump
thread without seeing a different behavior.

I’m running Windows 2000 on a box with 512 MB of memory. I ran this test on
.NET Fx 1.1 SP1. I haven’t tried any other framework versions.

Am I doing something wrong? Is this a known issue? Do you have any
suggestions or explanations?

Thanks in advance for any help you can provide.
using System;
using System.Drawing;
using System.Collections;
using System.ComponentModel;
using System.Windows.Forms;
using System.Data;
using System.Net;
using System.IO;
using System.Threading;

namespace SocketMemoryLeak {
public class Form1 : System.Windows.Forms.Form {
private const int KB = 1024;
private const int MB = KB * KB;
private System.Windows.Forms.Button btnSend;
private System.ComponentModel.Container components = null;
private System.Windows.Forms.Label lblMBsSent;
private System.Windows.Forms.Label lblGCed;
private System.Windows.Forms.Label lblMemoryAllocated;

public Form1() {
InitializeComponent();
}

protected override void Dispose(bool disposing) {
if (disposing) {
if (components != null) {
components.Dispose();
}
}
base.Dispose(disposing);
}

#region Windows Form Designer generated code
private void InitializeComponent() {
this.btnSend = new System.Windows.Forms.Button();
this.lblMBsSent = new System.Windows.Forms.Label();
this.lblGCed = new System.Windows.Forms.Label();
this.lblMemoryAllocated = new System.Windows.Forms.Label();
this.SuspendLayout();
//
// btnSend
//
this.btnSend.Anchor =
((System.Windows.Forms.AnchorStyles)(((System.Wind ows.Forms.AnchorStyles.Top
| System.Windows.Forms.AnchorStyles.Left)
| System.Windows.Forms.AnchorStyles.Right)));
this.btnSend.Location = new System.Drawing.Point(16, 8);
this.btnSend.Name = "btnSend";
this.btnSend.Size = new System.Drawing.Size(184, 23);
this.btnSend.TabIndex = 0;
this.btnSend.Text = "Start";
this.btnSend.Click += new System.EventHandler(this.btnSend_Click);
//
// lblMBsSent
//
this.lblMBsSent.Anchor =
((System.Windows.Forms.AnchorStyles)(((System.Wind ows.Forms.AnchorStyles.Top
| System.Windows.Forms.AnchorStyles.Left)
| System.Windows.Forms.AnchorStyles.Right)));
this.lblMBsSent.Location = new System.Drawing.Point(16, 40);
this.lblMBsSent.Name = "lblMBsSent";
this.lblMBsSent.Size = new System.Drawing.Size(184, 23);
this.lblMBsSent.TabIndex = 1;
this.lblMBsSent.Text = "Sent: 0 MBs";
this.lblMBsSent.TextAlign =
System.Drawing.ContentAlignment.MiddleLeft;
//
// lblGCed
//
this.lblGCed.Anchor =
((System.Windows.Forms.AnchorStyles)(((System.Wind ows.Forms.AnchorStyles.Top
| System.Windows.Forms.AnchorStyles.Left)
| System.Windows.Forms.AnchorStyles.Right)));
this.lblGCed.Location = new System.Drawing.Point(16, 88);
this.lblGCed.Name = "lblGCed";
this.lblGCed.Size = new System.Drawing.Size(184, 23);
this.lblGCed.TabIndex = 2;
this.lblGCed.Text = "No previous garbage collection";
this.lblGCed.TextAlign =
System.Drawing.ContentAlignment.MiddleLeft;
//
// lblMemoryAllocated
//
this.lblMemoryAllocated.Anchor =
((System.Windows.Forms.AnchorStyles)(((System.Wind ows.Forms.AnchorStyles.Top
| System.Windows.Forms.AnchorStyles.Left)
| System.Windows.Forms.AnchorStyles.Right)));
this.lblMemoryAllocated.Location = new System.Drawing.Point(16,
64);
this.lblMemoryAllocated.Name = "lblMemoryAllocated";
this.lblMemoryAllocated.Size = new System.Drawing.Size(184, 23);
this.lblMemoryAllocated.TabIndex = 3;
this.lblMemoryAllocated.Text = "Memory Allocated: 0 MB";
this.lblMemoryAllocated.TextAlign =
System.Drawing.ContentAlignment.MiddleLeft;
//
// Form1
//
this.AutoScaleBaseSize = new System.Drawing.Size(5, 13);
this.ClientSize = new System.Drawing.Size(208, 117);
this.Controls.Add(this.lblMemoryAllocated);
this.Controls.Add(this.lblGCed);
this.Controls.Add(this.lblMBsSent);
this.Controls.Add(this.btnSend);
this.Name = "Form1";
this.Text = "Form1";
this.ResumeLayout(false);

}
#endregion

[STAThread]
static void Main() {
Application.Run(new Form1());
}

private void btnSend_Click(object sender, System.EventArgs e) {
this.btnSend.Enabled = false;
Thread t = new Thread(new ThreadStart(this.Send));
t.IsBackground = true;
t.Start();
}

private void DoGC(int bytesSent) {
this.lblGCed.Text = "Starting GC...";
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
this.lblGCed.Text = "GC'd at " + (bytesSent / MB) + " MB";
}

private void SetSent(int bytesSent) {
this.lblMBsSent.Text = "Sent: " + (bytesSent / MB) + " MB";
this.lblMemoryAllocated.Text = "Memory Allocated: " +
(Environment.WorkingSet / MB) + " MB";
}

private void Send() {
byte[] buffer = new byte[16 * KB];
int totalBytesToSend = 514 * MB;

HttpWebRequest request = (HttpWebRequest)
WebRequest.Create("http://marias/cgi-bin/leif.cfg/php/enduser/fileUpload.php");
request.Method = "POST";
request.ContentLength = totalBytesToSend;
Stream requestStream = request.GetRequestStream();

int bytesSent = 0;
int gcInterval = 64 * MB;
int displayInterval = 1 * MB;
while (bytesSent < totalBytesToSend) {
requestStream.Write(buffer, 0, buffer.Length);
bytesSent += buffer.Length;
if (bytesSent % displayInterval == 0) {
this.SetSent(bytesSent);
}
if (bytesSent % gcInterval == 0) {
this.DoGC(bytesSent);
}
}

requestStream.Flush();
requestStream.Close();
requestStream = null;
request = null;

this.DoGC(bytesSent);
MessageBox.Show("Send thread ending.");
this.btnSend.Enabled = true;
}
}
}

Nov 16 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: zhimin | last post by:
Hi, I'm writing a program to send large file(100m) through dotnet using TCPListener & TCPClient, I'm sending the file with a ask and response loop: 1. Client send a flag 1 to server indicate it...
12
by: Olaf Baeyens | last post by:
I am porting some of my buffer class code for C++ to C#. This C++ class allocates a block of memory using m_pBuffer=new BYTE; But since the class is also used for pointers for funtions that uses...
0
by: NicK chlam via DotNetMonster.com | last post by:
this is the error i get System.Data.OleDb.OleDbException: Syntax error in INSERT INTO statement. at System.Data.Common.DbDataAdapter.Update(DataRow dataRows, DataTableMapping tableMapping) at...
4
by: Daniel | last post by:
Exception of type System.OutOfMemoryException was thrown. this error is occuring when on arbitrary threads in my .net windows service. There is 2 gigs of memory on the machine and this .net...
14
by: ThazKool | last post by:
I want to see if this code works the way it should on a Big-Endian system. Also if anyone has any ideas on how determine this at compile-time so that I use the right decoding or encoding...
7
by: StupidScript | last post by:
>From the manual "Storage Requirements": "ENUM('value1','value2',...) =1 or 2 bytes, depending on the number of enumeration values (65,535 values maximum)" This seems to mean: "a" = 1 byte...
4
by: Jonathan Wood | last post by:
Does anyone know why the documentation for System.Security.Cryptography.MD5.Create() seems to omit completely any description of allowed arguments. I'm trying to convert some C++ code to C# and...
30
by: nano2k | last post by:
Hi I need an efficient method to convert a string object to it's byte equivalent. I know there are LOTS of methods, but they lack in efficiency. All methods allocate new memory to create the byte...
2
by: Aryan | last post by:
Hi, I am using C# with framework 2.0 and creating PDF files on-fly, along with this I am using Windows 2003 Server. I am using Byte to take the data input and then save into pdf format on...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.