HttpWebRequest never terminates

[=?Utf-8?B?SmV0dCBKb25lcw==?=]

I'm working on a webdav client which makes multiple webrequests using the
threadpool.

The initial issue was that some requests never returned, though I could
attach to the application and see they were still blocked at
webReq.GetRespo nse(). I finally fired up netmon to see where the connection
seemed to be getting lost, and it seems to be a race condition in the TCP
handling. Since I know next to nothing about TCP, or asynchronous
programming hopefully somone else can shed some more light on the issue.

When I see the issue, two threads are sending requests from the same client
port, and one server response overlaps a client ack in transit. This makes
me think that the client views this as having lost a packet and needing to
retransmit. However, the server response which caused the retransmit is
ignored, causing one of the client requests to wait forever for the server to
send a packet which the client has already recieved.

This is a simplified version of the trace data- client sequence numbers are
10+, servers are 110+, they're not continuous since there are other threads
sending data as well.
packet order (client) || Time || Source -Dest || Ack# || Seq# || Request
type || Verb
4 4.633 C -S 100 10 HTTP MKCOL
5 4.828 S -C 20 100 TCP (ack)
6 4.828 C -S 110 20 HTTP PROPPATCH
7 4.929 S -C 30 110 HTTP 201 - created
8 5.112 C -S 120 30 TCP (ack)
9 5.113 S -C 30 120 HTTP 100 - continue
10 5.313 C -S 130 30 TCP (ack)

So packets 8 and 9 are transmitted at the same time, so the client resends
packet 8 as packet 10. However, packet 9 which should trigger the body of
the 'proppatch' request, is never handled by the client.

Anyway- I realize this is long and rambling, so thanks to anyone who read
this far, and double thanks if you have any suggestions either for work
arounds or additional debugging.

[Rick Strahl [MVP]]

Hi Jett,

Are you sure that call is not working at all or just really slow?

I've had many issues with the defaut proxy settings which are not using
direct acces but rather try to find the proxy server which fails under
ASP.NET in a typical security environment. THis can be very slow and take up
to 30 seconds+ to resolve for the first call. I suspect this will be true
for any HTTP call on a new thread.

I talked about this here a while back:
http://west-wind.com/weblog/posts/3871.aspx

I have a few schedulers as part of my internal WebLog engine that run a fair
number of requests on separate threads and haven't seen any issues with HTTP
trafficking on these outbound calls. It's not super high volume though but
they do run a few threads simultaneously with WebRequest.

Wonder too if you can try using raw threads rather than the threadpool -
maybe that's causing some issues if the threadpool is pressured from request
load... Finally make sure that your code is completely thread safe. Is it
possible your WebRequest instance is getting shared across threads by
accident? This sounds like a possibility given your traffic analyses.

+++ Rick ---

--
Rick Strahl
West Wind Technologies
www.west-wind.com/weblog

"Jett Jones" <Jett Jo***@discussio ns.microsoft.co mwrote in message
news:87******** *************** ***********@mic rosoft.com...

I'm working on a webdav client which makes multiple webrequests using the
threadpool.

The initial issue was that some requests never returned, though I could
attach to the application and see they were still blocked at
webReq.GetRespo nse(). I finally fired up netmon to see where the
connection
seemed to be getting lost, and it seems to be a race condition in the TCP
handling. Since I know next to nothing about TCP, or asynchronous
programming hopefully somone else can shed some more light on the issue.

When I see the issue, two threads are sending requests from the same
client
port, and one server response overlaps a client ack in transit. This
makes
me think that the client views this as having lost a packet and needing to
retransmit. However, the server response which caused the retransmit is
ignored, causing one of the client requests to wait forever for the server
to
send a packet which the client has already recieved.

This is a simplified version of the trace data- client sequence numbers
are
10+, servers are 110+, they're not continuous since there are other
threads
sending data as well.
packet order (client) || Time || Source -Dest || Ack# || Seq# || Request
type || Verb
4 4.633 C -S 100 10 HTTP MKCOL
5 4.828 S -C 20 100 TCP (ack)
6 4.828 C -S 110 20 HTTP PROPPATCH
7 4.929 S -C 30 110 HTTP 201 - created
8 5.112 C -S 120 30 TCP (ack)
9 5.113 S -C 30 120 HTTP 100 - continue
10 5.313 C -S 130 30 TCP (ack)

So packets 8 and 9 are transmitted at the same time, so the client resends
packet 8 as packet 10. However, packet 9 which should trigger the body of
the 'proppatch' request, is never handled by the client.

Anyway- I realize this is long and rambling, so thanks to anyone who read
this far, and double thanks if you have any suggestions either for work
arounds or additional debugging.

[=?Utf-8?B?SmV0dCBKb25lcw==?=]

Thanks for the response rick, though I think I'm not hitting those specific
issues, but let me know if I'm misunderstandin g something. Also I definately
agree that the most likely answer isn't the framework, but I'm also not
finding anything clearly wrong in my code. I'm making all my requests
synchronously, so a request object should never be used outside the thread
which created it. I've left it waiting for responses for more than half an
hour with no success, so I think it's more than the intial proxy setup time.

For my next step I'll try adding additional locks around creating web
requests, on the off chance that that's not synchronized internally.

"Rick Strahl [MVP]" wrote:

Hi Jett,

Are you sure that call is not working at all or just really slow?

I've had many issues with the defaut proxy settings which are not using
direct acces but rather try to find the proxy server which fails under
ASP.NET in a typical security environment. THis can be very slow and take up
to 30 seconds+ to resolve for the first call. I suspect this will be true
for any HTTP call on a new thread.

I talked about this here a while back:
http://west-wind.com/weblog/posts/3871.aspx

I have a few schedulers as part of my internal WebLog engine that run a fair
number of requests on separate threads and haven't seen any issues with HTTP
trafficking on these outbound calls. It's not super high volume though but
they do run a few threads simultaneously with WebRequest.

Wonder too if you can try using raw threads rather than the threadpool -
maybe that's causing some issues if the threadpool is pressured from request
load... Finally make sure that your code is completely thread safe. Is it
possible your WebRequest instance is getting shared across threads by
accident? This sounds like a possibility given your traffic analyses.

+++ Rick ---

--
Rick Strahl
West Wind Technologies
www.west-wind.com/weblog

"Jett Jones" <Jett Jo***@discussio ns.microsoft.co mwrote in message
news:87******** *************** ***********@mic rosoft.com...

I'm working on a webdav client which makes multiple webrequests using the
threadpool.

The initial issue was that some requests never returned, though I could
attach to the application and see they were still blocked at
webReq.GetRespo nse(). I finally fired up netmon to see where the
connection
seemed to be getting lost, and it seems to be a race condition in the TCP
handling. Since I know next to nothing about TCP, or asynchronous
programming hopefully somone else can shed some more light on the issue.

When I see the issue, two threads are sending requests from the same
client
port, and one server response overlaps a client ack in transit. This
makes
me think that the client views this as having lost a packet and needing to
retransmit. However, the server response which caused the retransmit is
ignored, causing one of the client requests to wait forever for the server
to
send a packet which the client has already recieved.

This is a simplified version of the trace data- client sequence numbers
are
10+, servers are 110+, they're not continuous since there are other
threads
sending data as well.
packet order (client) || Time || Source -Dest || Ack# || Seq# || Request
type || Verb
4 4.633 C -S 100 10 HTTP MKCOL
5 4.828 S -C 20 100 TCP (ack)
6 4.828 C -S 110 20 HTTP PROPPATCH
7 4.929 S -C 30 110 HTTP 201 - created
8 5.112 C -S 120 30 TCP (ack)
9 5.113 S -C 30 120 HTTP 100 - continue
10 5.313 C -S 130 30 TCP (ack)

So packets 8 and 9 are transmitted at the same time, so the client resends
packet 8 as packet 10. However, packet 9 which should trigger the body of
the 'proppatch' request, is never handled by the client.

Anyway- I realize this is long and rambling, so thanks to anyone who read
this far, and double thanks if you have any suggestions either for work
arounds or additional debugging.