471,319 Members | 1,344 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,319 software developers and data experts.

GetRequestStream slows down?

I have a multithreaded application that is posting XML requests to a remote
web site. As more and more calls are made to
HttpWebRequest.GetRequestStream, the length of time increases for each call
from an initial 2-3 seconds to anywhere from 15-30 seconds. What am I doing
that is causing this method call's duration to increase and how do I make it
run faster? Below is some pseudo-code for the method that is called in a
multi-threaded fashion...

- Jim

private void MultithreadedMethod()
{
string url = ConfigurationManager.AppSettings["providerUrl"];

request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "text/xml";

startCount = Environment.TickCount;
using (Stream s = request.GetRequestStream())
{
elapsedCount = Environment.TickCount - startCount;
//the elapsed time increases with each call
EventLog.WriteEntry("request.GetRequestStream() = " + elapsedCount,
EventLogEntryType.Information);

using (StreamWriter writer = new StreamWriter(s))
{
writer.WriteLine(requestXml);
writer.Flush();
writer.Close();
}

s.Close();
}
request.Timeout = requestTimeoutMilliseconds;
response = request.GetResponse();

//handle the response here
}

Jan 19 '06 #1
8 5987
Are these calls concurrent or consecutive. In 1.1 these requests were
made on the thread pool, so you might be running into some concurrency
issues.

Additionally, depending on the size of the page returned, you might also
be running in to GC issues -- so for example, if the content returned is
over 85k (assuming it being put into one big string variable at some
point) that memory has the potential of being put into the large object
heap which is only swept and not compacted .... so you might be running
into some memory issues.

Also, the s.Close is redundant and repetitive ... it's getting called in
IDispose after you exit from your using block.

Also, make sure that you are closing the response object that you get
back from request.GetResponse(); .... not doing so can lead to some
unmitigated nastiness

John

Jim Toth wrote:
I have a multithreaded application that is posting XML requests to a remote
web site. As more and more calls are made to
HttpWebRequest.GetRequestStream, the length of time increases for each call
from an initial 2-3 seconds to anywhere from 15-30 seconds. What am I doing
that is causing this method call's duration to increase and how do I make it
run faster? Below is some pseudo-code for the method that is called in a
multi-threaded fashion...

- Jim

private void MultithreadedMethod()
{
string url = ConfigurationManager.AppSettings["providerUrl"];

request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "text/xml";

startCount = Environment.TickCount;
using (Stream s = request.GetRequestStream())
{
elapsedCount = Environment.TickCount - startCount;
//the elapsed time increases with each call
EventLog.WriteEntry("request.GetRequestStream() = " + elapsedCount,
EventLogEntryType.Information);

using (StreamWriter writer = new StreamWriter(s))
{
writer.WriteLine(requestXml);
writer.Flush();
writer.Close();
}

s.Close();
}
request.Timeout = requestTimeoutMilliseconds;
response = request.GetResponse();

//handle the response here
}

Jan 19 '06 #2
Aside from the other comments, the title of your method "MultithreadedMethod"
is confusing, as it isn't multithreaded at all the way it is written.
If you really have a lot of this going on, you probably want to look at the
asynchronous BeginXXX / EndXXX overloads which use the Threadpool "under the
hood" - or use a custom threadpool.
Peter

--
Co-founder, Eggheadcafe.com developer portal:
http://www.eggheadcafe.com
UnBlog:
http://petesbloggerama.blogspot.com


"Jim Toth" wrote:
I have a multithreaded application that is posting XML requests to a remote
web site. As more and more calls are made to
HttpWebRequest.GetRequestStream, the length of time increases for each call
from an initial 2-3 seconds to anywhere from 15-30 seconds. What am I doing
that is causing this method call's duration to increase and how do I make it
run faster? Below is some pseudo-code for the method that is called in a
multi-threaded fashion...

- Jim

private void MultithreadedMethod()
{
string url = ConfigurationManager.AppSettings["providerUrl"];

request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "text/xml";

startCount = Environment.TickCount;
using (Stream s = request.GetRequestStream())
{
elapsedCount = Environment.TickCount - startCount;
//the elapsed time increases with each call
EventLog.WriteEntry("request.GetRequestStream() = " + elapsedCount,
EventLogEntryType.Information);

using (StreamWriter writer = new StreamWriter(s))
{
writer.WriteLine(requestXml);
writer.Flush();
writer.Close();
}

s.Close();
}
request.Timeout = requestTimeoutMilliseconds;
response = request.GetResponse();

//handle the response here
}

Jan 20 '06 #3
What I meant by "MultithreadedMethod" is that the method is called many times
on many different threads.

I've seen the asynchronous calls for GetRequestStream... will they get me
around potential contention issues mentioned?

"Peter Bromberg [C# MVP]" wrote:
Aside from the other comments, the title of your method "MultithreadedMethod"
is confusing, as it isn't multithreaded at all the way it is written.
If you really have a lot of this going on, you probably want to look at the
asynchronous BeginXXX / EndXXX overloads which use the Threadpool "under the
hood" - or use a custom threadpool.
Peter

--
Co-founder, Eggheadcafe.com developer portal:
http://www.eggheadcafe.com
UnBlog:
http://petesbloggerama.blogspot.com


"Jim Toth" wrote:
I have a multithreaded application that is posting XML requests to a remote
web site. As more and more calls are made to
HttpWebRequest.GetRequestStream, the length of time increases for each call
from an initial 2-3 seconds to anywhere from 15-30 seconds. What am I doing
that is causing this method call's duration to increase and how do I make it
run faster? Below is some pseudo-code for the method that is called in a
multi-threaded fashion...

- Jim

private void MultithreadedMethod()
{
string url = ConfigurationManager.AppSettings["providerUrl"];

request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "text/xml";

startCount = Environment.TickCount;
using (Stream s = request.GetRequestStream())
{
elapsedCount = Environment.TickCount - startCount;
//the elapsed time increases with each call
EventLog.WriteEntry("request.GetRequestStream() = " + elapsedCount,
EventLogEntryType.Information);

using (StreamWriter writer = new StreamWriter(s))
{
writer.WriteLine(requestXml);
writer.Flush();
writer.Close();
}

s.Close();
}
request.Timeout = requestTimeoutMilliseconds;
response = request.GetResponse();

//handle the response here
}

Jan 20 '06 #4
Correct, which is a misnomer. If it is called many different times, you
should probably be using the Threadpool and the asynchronous
IAsyncResult result = request.BeginGetResponse(
new AsyncCallback(UpdateItem),state);

semantics. This will speed up your operation by an order of magnitude since
it offers parallelism (multiple requests in operation at the same time).
Peter

--
Co-founder, Eggheadcafe.com developer portal:
http://www.eggheadcafe.com
UnBlog:
http://petesbloggerama.blogspot.com


"Jim Toth" wrote:
What I meant by "MultithreadedMethod" is that the method is called many times
on many different threads.

I've seen the asynchronous calls for GetRequestStream... will they get me
around potential contention issues mentioned?

"Peter Bromberg [C# MVP]" wrote:
Aside from the other comments, the title of your method "MultithreadedMethod"
is confusing, as it isn't multithreaded at all the way it is written.
If you really have a lot of this going on, you probably want to look at the
asynchronous BeginXXX / EndXXX overloads which use the Threadpool "under the
hood" - or use a custom threadpool.
Peter

--
Co-founder, Eggheadcafe.com developer portal:
http://www.eggheadcafe.com
UnBlog:
http://petesbloggerama.blogspot.com


"Jim Toth" wrote:
I have a multithreaded application that is posting XML requests to a remote
web site. As more and more calls are made to
HttpWebRequest.GetRequestStream, the length of time increases for each call
from an initial 2-3 seconds to anywhere from 15-30 seconds. What am I doing
that is causing this method call's duration to increase and how do I make it
run faster? Below is some pseudo-code for the method that is called in a
multi-threaded fashion...

- Jim

private void MultithreadedMethod()
{
string url = ConfigurationManager.AppSettings["providerUrl"];

request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "text/xml";

startCount = Environment.TickCount;
using (Stream s = request.GetRequestStream())
{
elapsedCount = Environment.TickCount - startCount;
//the elapsed time increases with each call
EventLog.WriteEntry("request.GetRequestStream() = " + elapsedCount,
EventLogEntryType.Information);

using (StreamWriter writer = new StreamWriter(s))
{
writer.WriteLine(requestXml);
writer.Flush();
writer.Close();
}

s.Close();
}
request.Timeout = requestTimeoutMilliseconds;
response = request.GetResponse();

//handle the response here
}

Jan 20 '06 #5
If you are using the 1.1 or earlier version of the Framework, never make
calls on WebClient or WebRequest from within the threadpool -- as the
calls on those objects are also implemented using the threadpool (even
if you are calling them synchronously) ...... the result is a very high
potential for a deadlock.
Peter Bromberg [C# MVP] wrote:
Correct, which is a misnomer. If it is called many different times, you
should probably be using the Threadpool and the asynchronous
IAsyncResult result = request.BeginGetResponse(
new AsyncCallback(UpdateItem),state);

semantics. This will speed up your operation by an order of magnitude since
it offers parallelism (multiple requests in operation at the same time).
Peter

Jan 20 '06 #6
Perhaps I should have stated my suggestion more clearly. What I meant was,
use the asynchronous BeginGetResponse etc. method, which use the .NET
threadpool.
Peter

--
Co-founder, Eggheadcafe.com developer portal:
http://www.eggheadcafe.com
UnBlog:
http://petesbloggerama.blogspot.com


"John Murray" wrote:
If you are using the 1.1 or earlier version of the Framework, never make
calls on WebClient or WebRequest from within the threadpool -- as the
calls on those objects are also implemented using the threadpool (even
if you are calling them synchronously) ...... the result is a very high
potential for a deadlock.
Peter Bromberg [C# MVP] wrote:
Correct, which is a misnomer. If it is called many different times, you
should probably be using the Threadpool and the asynchronous
IAsyncResult result = request.BeginGetResponse(
new AsyncCallback(UpdateItem),state);

semantics. This will speed up your operation by an order of magnitude since
it offers parallelism (multiple requests in operation at the same time).
Peter

Jan 20 '06 #7
It looks like I'm running into some issues with
ServicePointManager.DefaultConnectionLimit, which is defaulted to 2, because
I'm sending data to the same web site URL over and over. I tried bumping
this up to 10, 15, and 20 and had to bump up the number of threads in the
ThreadPool as a result, however I'm seeing only marginal gains in
performance. Is there a configuration setting, either for .NET or Windows
Server 2003, that I should be looking at, or am I missing something else?

"Peter Bromberg [C# MVP]" wrote:
Correct, which is a misnomer. If it is called many different times, you
should probably be using the Threadpool and the asynchronous
IAsyncResult result = request.BeginGetResponse(
new AsyncCallback(UpdateItem),state);

semantics. This will speed up your operation by an order of magnitude since
it offers parallelism (multiple requests in operation at the same time).
Peter

--
Co-founder, Eggheadcafe.com developer portal:
http://www.eggheadcafe.com
UnBlog:
http://petesbloggerama.blogspot.com


"Jim Toth" wrote:
What I meant by "MultithreadedMethod" is that the method is called many times
on many different threads.

I've seen the asynchronous calls for GetRequestStream... will they get me
around potential contention issues mentioned?

"Peter Bromberg [C# MVP]" wrote:
Aside from the other comments, the title of your method "MultithreadedMethod"
is confusing, as it isn't multithreaded at all the way it is written.
If you really have a lot of this going on, you probably want to look at the
asynchronous BeginXXX / EndXXX overloads which use the Threadpool "under the
hood" - or use a custom threadpool.
Peter

--
Co-founder, Eggheadcafe.com developer portal:
http://www.eggheadcafe.com
UnBlog:
http://petesbloggerama.blogspot.com


"Jim Toth" wrote:

> I have a multithreaded application that is posting XML requests to a remote
> web site. As more and more calls are made to
> HttpWebRequest.GetRequestStream, the length of time increases for each call
> from an initial 2-3 seconds to anywhere from 15-30 seconds. What am I doing
> that is causing this method call's duration to increase and how do I make it
> run faster? Below is some pseudo-code for the method that is called in a
> multi-threaded fashion...
>
> - Jim
>
> private void MultithreadedMethod()
> {
> string url = ConfigurationManager.AppSettings["providerUrl"];
>
> request = (HttpWebRequest)WebRequest.Create(url);
> request.Method = "POST";
> request.ContentType = "text/xml";
>
> startCount = Environment.TickCount;
> using (Stream s = request.GetRequestStream())
> {
> elapsedCount = Environment.TickCount - startCount;
> //the elapsed time increases with each call
> EventLog.WriteEntry("request.GetRequestStream() = " + elapsedCount,
> EventLogEntryType.Information);
>
> using (StreamWriter writer = new StreamWriter(s))
> {
> writer.WriteLine(requestXml);
> writer.Flush();
> writer.Close();
> }
>
> s.Close();
> }
>
>
> request.Timeout = requestTimeoutMilliseconds;
> response = request.GetResponse();
>
> //handle the response here
> }
>

Jan 20 '06 #8
make sure you close the Response object along with any streams /
streamreaders each time. Aside from that, I've seen a lot of old newsgroups
threads on upping the DefaultConnectionLimit with mixed results. It could be
at the server too, you know.
Peter

--
Co-founder, Eggheadcafe.com developer portal:
http://www.eggheadcafe.com
UnBlog:
http://petesbloggerama.blogspot.com


"Jim Toth" wrote:
I have a multithreaded application that is posting XML requests to a remote
web site. As more and more calls are made to
HttpWebRequest.GetRequestStream, the length of time increases for each call
from an initial 2-3 seconds to anywhere from 15-30 seconds. What am I doing
that is causing this method call's duration to increase and how do I make it
run faster? Below is some pseudo-code for the method that is called in a
multi-threaded fashion...

- Jim

private void MultithreadedMethod()
{
string url = ConfigurationManager.AppSettings["providerUrl"];

request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "text/xml";

startCount = Environment.TickCount;
using (Stream s = request.GetRequestStream())
{
elapsedCount = Environment.TickCount - startCount;
//the elapsed time increases with each call
EventLog.WriteEntry("request.GetRequestStream() = " + elapsedCount,
EventLogEntryType.Information);

using (StreamWriter writer = new StreamWriter(s))
{
writer.WriteLine(requestXml);
writer.Flush();
writer.Close();
}

s.Close();
}
request.Timeout = requestTimeoutMilliseconds;
response = request.GetResponse();

//handle the response here
}

Jan 20 '06 #9

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

5 posts views Thread by Narine | last post: by
7 posts views Thread by Piet Van Vlierberghe | last post: by
9 posts views Thread by cow3 | last post: by
14 posts views Thread by Paul_Madden via DotNetMonster.com | last post: by
reply views Thread by kimiraikkonen | last post: by
reply views Thread by rosydwin | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.