TCP - Outstanding Begin Receives

[Larry Herbinaux]

It is possible to have an outstanding BeginReceive call when closing a
socket. There isn't a call from the IAsyncResult to cancel this if it hasn't
completed. This can be an issue when you are accepting a large number of
connections in a long running process.

First off, why isn't there a method for cancelling these requests?

Is there a work around that anyone knows about?

My state object that is referenced by the request is large, so I at least
try to cleanup this reference, but there is still an OS object that is queued
for this request that doesn't get freed.

[Vadym Stetsyak]

Async I/O will be canceled if the thread that initiated that I/O will exit.

This means if you're calling BeginReceive the thread that accepts incomming
connections - it will be sufficient to stop that thread.

Also if you will call Socket.Shutdown, the exception will be thrown in
EndReceive(...) or appropriate SocketError returned ( under .NET 2.0 )

--
Vadym Stetsyak aka Vadmyst
http://vadmyst.blogspot.com

"Larry Herbinaux" <La************@discussions.microsoft.com> wrote in
message news:2A**********************************@microsof t.com...

It is possible to have an outstanding BeginReceive call when closing a
socket. There isn't a call from the IAsyncResult to cancel this if it
hasn't
completed. This can be an issue when you are accepting a large number of
connections in a long running process.

First off, why isn't there a method for cancelling these requests?

Is there a work around that anyone knows about?

My state object that is referenced by the request is large, so I at least
try to cleanup this reference, but there is still an OS object that is
queued
for this request that doesn't get freed.

[Larry Herbinaux]

Thanks Vadym,

On your first point, the async request is being performed in a thread pool
thread (e.g. when the AsyncAccept callback is called it is on a thread in the
thread pool and I immediately do a BeginRead within this thread), so this
thread never exits, right? I'm sure that the request is registered in the OS
and when an IOCompletionPort is signaled, the callback runs on a thread in
the thread pool and this process starts all over again until we close the
socket.

Depending on who (client / server) and how the socket is closed, there still
could be an outstanding async receive request. We are using 1.1, so in your
second point I'm not sure it applies. A couple of questions on this:

1) If I do a close on the socket, will this cleanup any outstanding receive
/ send requests?

2) I am holding on to the IAsyncResult for the outstanding receive and send
(in my wrapper socket class), so I could use the EndReceive prior to closing
the socket. The issue here is that if for some reason the remote (client
socket) is not closed, this call will block (at least that's what MSDN
states). I think 99% of the time the client socket will be closed, but I
also have a cleanup process that checks if we haven't had read / write data
from the socket for a certain amount of time, then I have to do a hard close
from the server...would the EndReceive block in this case?

"Vadym Stetsyak" wrote:

Async I/O will be canceled if the thread that initiated that I/O will exit.

This means if you're calling BeginReceive the thread that accepts incomming
connections - it will be sufficient to stop that thread.

Also if you will call Socket.Shutdown, the exception will be thrown in
EndReceive(...) or appropriate SocketError returned ( under .NET 2.0 )

--
Vadym Stetsyak aka Vadmyst
http://vadmyst.blogspot.com

"Larry Herbinaux" <La************@discussions.microsoft.com> wrote in
message news:2A**********************************@microsof t.com...

It is possible to have an outstanding BeginReceive call when closing a
socket. There isn't a call from the IAsyncResult to cancel this if it
hasn't
completed. This can be an issue when you are accepting a large number of
connections in a long running process.

First off, why isn't there a method for cancelling these requests?

Is there a work around that anyone knows about?

My state object that is referenced by the request is large, so I at least
try to cleanup this reference, but there is still an OS object that is
queued
for this request that doesn't get freed.

[Vadym Stetsyak]

> 1) If I do a close on the socket, will this cleanup any outstanding

receive
/ send requests?
Yes
2) I am holding on to the IAsyncResult for the outstanding receive and
send
(in my wrapper socket class), so I could use the EndReceive prior to
closing
the socket. The issue here is that if for some reason the remote (client
socket) is not closed, this call will block (at least that's what MSDN
states). I think 99% of the time the client socket will be closed, but I
also have a cleanup process that checks if we haven't had read / write
data
from the socket for a certain amount of time, then I have to do a hard
close
from the server...would the EndReceive block in this case?
EndReceive will block for timeout, of Socket keepalive option was not
specified.
If server will close the socket EndReceive in the client side will throw an
exception, or
return 0 bytes, indicating graceful connection close ( more probable ).

--
Vadym Stetsyak aka Vadmyst
http://vadmyst.blogspot.com

"Larry Herbinaux" <La************@discussions.microsoft.com> wrote in
message news:71**********************************@microsof t.com... Thanks Vadym,

On your first point, the async request is being performed in a thread pool
thread (e.g. when the AsyncAccept callback is called it is on a thread in
the
thread pool and I immediately do a BeginRead within this thread), so this
thread never exits, right? I'm sure that the request is registered in the
OS
and when an IOCompletionPort is signaled, the callback runs on a thread in
the thread pool and this process starts all over again until we close the
socket.

Depending on who (client / server) and how the socket is closed, there
still
could be an outstanding async receive request. We are using 1.1, so in
your
second point I'm not sure it applies. A couple of questions on this:

1) If I do a close on the socket, will this cleanup any outstanding
receive
/ send requests?

2) I am holding on to the IAsyncResult for the outstanding receive and
send
(in my wrapper socket class), so I could use the EndReceive prior to
closing
the socket. The issue here is that if for some reason the remote (client
socket) is not closed, this call will block (at least that's what MSDN
states). I think 99% of the time the client socket will be closed, but I
also have a cleanup process that checks if we haven't had read / write
data
from the socket for a certain amount of time, then I have to do a hard
close
from the server...would the EndReceive block in this case?

"Vadym Stetsyak" wrote:

Async I/O will be canceled if the thread that initiated that I/O will
exit.

This means if you're calling BeginReceive the thread that accepts
incomming
connections - it will be sufficient to stop that thread.

Also if you will call Socket.Shutdown, the exception will be thrown in
EndReceive(...) or appropriate SocketError returned ( under .NET 2.0 )

--
Vadym Stetsyak aka Vadmyst
http://vadmyst.blogspot.com

"Larry Herbinaux" <La************@discussions.microsoft.com> wrote in
message news:2A**********************************@microsof t.com...

> It is possible to have an outstanding BeginReceive call when closing a
> socket. There isn't a call from the IAsyncResult to cancel this if it
> hasn't
> completed. This can be an issue when you are accepting a large number
> of
> connections in a long running process.
>
> First off, why isn't there a method for cancelling these requests?
>
> Is there a work around that anyone knows about?
>
> My state object that is referenced by the request is large, so I at
> least
> try to cleanup this reference, but there is still an OS object that is
> queued
> for this request that doesn't get freed.

[Larry Herbinaux]

Thanks a lot this really helped. I just want to clarify that I understand
your response.

As for your answer to #1, if this is the case, then I don't have to worry
about the outstanding requests holding on to my state object, this should all
be garbage collected, right? Do you know the name of the structure that the
OS holds for these requests? I could use CLR Profiler to verify this, huh?

As for #2, I didn't know that there is a possibility of an exception being
thrown when closing the socket; I've always seen a count of 0 and do the
appropriate thing. In most cases, I do a shutdown of the send channel and
wait for the other socket to respond in a similar fashion before I shutdown
the receive channel. I don't like Microsoft's recommendation of shutting
down both channels because this causes an exception in most cases to be
thrown in the outstanding asynchronous receive request (previously, I handled
this but logged it and I don't like cluttering the log for this purpose).
The only time I do a shutdown of both and close is if the server recognizes a
connection that has been inactive for a period of time (e.g. physical line
was severed, etc.).

Thanks again for your input.

"Vadym Stetsyak" wrote:

1) If I do a close on the socket, will this cleanup any outstanding
receive
/ send requests?

Yes

2) I am holding on to the IAsyncResult for the outstanding receive and
send
(in my wrapper socket class), so I could use the EndReceive prior to
closing
the socket. The issue here is that if for some reason the remote (client
socket) is not closed, this call will block (at least that's what MSDN
states). I think 99% of the time the client socket will be closed, but I
also have a cleanup process that checks if we haven't had read / write
data
from the socket for a certain amount of time, then I have to do a hard
close
from the server...would the EndReceive block in this case?

EndReceive will block for timeout, of Socket keepalive option was not
specified.
If server will close the socket EndReceive in the client side will throw an
exception, or
return 0 bytes, indicating graceful connection close ( more probable ).

--
Vadym Stetsyak aka Vadmyst
http://vadmyst.blogspot.com

"Larry Herbinaux" <La************@discussions.microsoft.com> wrote in
message news:71**********************************@microsof t.com...

Thanks Vadym,

On your first point, the async request is being performed in a thread pool
thread (e.g. when the AsyncAccept callback is called it is on a thread in
the
thread pool and I immediately do a BeginRead within this thread), so this
thread never exits, right? I'm sure that the request is registered in the
OS
and when an IOCompletionPort is signaled, the callback runs on a thread in
the thread pool and this process starts all over again until we close the
socket.

Depending on who (client / server) and how the socket is closed, there
still
could be an outstanding async receive request. We are using 1.1, so in
your
second point I'm not sure it applies. A couple of questions on this:

1) If I do a close on the socket, will this cleanup any outstanding
receive
/ send requests?

2) I am holding on to the IAsyncResult for the outstanding receive and
send
(in my wrapper socket class), so I could use the EndReceive prior to
closing
the socket. The issue here is that if for some reason the remote (client
socket) is not closed, this call will block (at least that's what MSDN
states). I think 99% of the time the client socket will be closed, but I
also have a cleanup process that checks if we haven't had read / write
data
from the socket for a certain amount of time, then I have to do a hard
close
from the server...would the EndReceive block in this case?

"Vadym Stetsyak" wrote:

Async I/O will be canceled if the thread that initiated that I/O will
exit.

This means if you're calling BeginReceive the thread that accepts
incomming
connections - it will be sufficient to stop that thread.

Also if you will call Socket.Shutdown, the exception will be thrown in
EndReceive(...) or appropriate SocketError returned ( under .NET 2.0 )

--
Vadym Stetsyak aka Vadmyst
http://vadmyst.blogspot.com

"Larry Herbinaux" <La************@discussions.microsoft.com> wrote in
message news:2A**********************************@microsof t.com...
> It is possible to have an outstanding BeginReceive call when closing a
> socket. There isn't a call from the IAsyncResult to cancel this if it
> hasn't
> completed. This can be an issue when you are accepting a large number
> of
> connections in a long running process.
>
> First off, why isn't there a method for cancelling these requests?
>
> Is there a work around that anyone knows about?
>
> My state object that is referenced by the request is large, so I at
> least
> try to cleanup this reference, but there is still an OS object that is
> queued
> for this request that doesn't get freed.

[Vadym Stetsyak]

For #1

If there is no references to your state object then it will be GC-ed.

BeginReceive call is mapped to WSARecv(...). This is Winsock2 function. For
async io it uses
WSAOVERLAPPED structure. This structure holds the state, however, it holds a
pointer so you won't be able to see the object using it.

Yes, you can use CLR Profiler to verify how many state objects are there in
the memory, but for that you do not need any OS structures.

For #2

Yes, returting 0 is graceful shutdown, however world is not perfect, you may
expereience hardware problems ( on the wire, remote host crashed etc )

Using Socket.Shutdown with Both or Receive will cause an exception or socket
error ( in .NET 2.0 ).
From the doc about Shutdown
"Setting how to Receive specifies that subsequent calls to Receive are not
allowed"
You can use this SocketShudown options to control, the way how connection is
being closed ( with or without sending/receiving data after shutdown, but
before close )

--
Vadym Stetsyak aka Vadmyst
http://vadmyst.blogspot.com

"Larry Herbinaux" <La************@discussions.microsoft.com> wrote in
message news:E4**********************************@microsof t.com...

Thanks a lot this really helped. I just want to clarify that I understand
your response.

As for your answer to #1, if this is the case, then I don't have to worry
about the outstanding requests holding on to my state object, this should
all
be garbage collected, right? Do you know the name of the structure that
the
OS holds for these requests? I could use CLR Profiler to verify this,
huh?

As for #2, I didn't know that there is a possibility of an exception being
thrown when closing the socket; I've always seen a count of 0 and do the
appropriate thing. In most cases, I do a shutdown of the send channel and
wait for the other socket to respond in a similar fashion before I
shutdown
the receive channel. I don't like Microsoft's recommendation of shutting
down both channels because this causes an exception in most cases to be
thrown in the outstanding asynchronous receive request (previously, I
handled
this but logged it and I don't like cluttering the log for this purpose).
The only time I do a shutdown of both and close is if the server
recognizes a
connection that has been inactive for a period of time (e.g. physical line
was severed, etc.).

Thanks again for your input.

"Vadym Stetsyak" wrote:

> 1) If I do a close on the socket, will this cleanup any outstanding
> receive
> / send requests?

Yes

> 2) I am holding on to the IAsyncResult for the outstanding receive and
> send
> (in my wrapper socket class), so I could use the EndReceive prior to
> closing
> the socket. The issue here is that if for some reason the remote
> (client
> socket) is not closed, this call will block (at least that's what MSDN
> states). I think 99% of the time the client socket will be closed, but
> I
> also have a cleanup process that checks if we haven't had read / write
> data
> from the socket for a certain amount of time, then I have to do a hard
> close
> from the server...would the EndReceive block in this case?

EndReceive will block for timeout, of Socket keepalive option was not
specified.
If server will close the socket EndReceive in the client side will throw
an
exception, or
return 0 bytes, indicating graceful connection close ( more probable ).

--
Vadym Stetsyak aka Vadmyst
http://vadmyst.blogspot.com

"Larry Herbinaux" <La************@discussions.microsoft.com> wrote in
message news:71**********************************@microsof t.com...

> Thanks Vadym,
>
> On your first point, the async request is being performed in a thread
> pool
> thread (e.g. when the AsyncAccept callback is called it is on a thread
> in
> the
> thread pool and I immediately do a BeginRead within this thread), so
> this
> thread never exits, right? I'm sure that the request is registered in
> the
> OS
> and when an IOCompletionPort is signaled, the callback runs on a thread
> in
> the thread pool and this process starts all over again until we close
> the
> socket.
>
> Depending on who (client / server) and how the socket is closed, there
> still
> could be an outstanding async receive request. We are using 1.1, so in
> your
> second point I'm not sure it applies. A couple of questions on this:
>
> 1) If I do a close on the socket, will this cleanup any outstanding
> receive
> / send requests?
>
> 2) I am holding on to the IAsyncResult for the outstanding receive and
> send
> (in my wrapper socket class), so I could use the EndReceive prior to
> closing
> the socket. The issue here is that if for some reason the remote
> (client
> socket) is not closed, this call will block (at least that's what MSDN
> states). I think 99% of the time the client socket will be closed, but
> I
> also have a cleanup process that checks if we haven't had read / write
> data
> from the socket for a certain amount of time, then I have to do a hard
> close
> from the server...would the EndReceive block in this case?
>
> "Vadym Stetsyak" wrote:
>
>> Async I/O will be canceled if the thread that initiated that I/O will
>> exit.
>>
>> This means if you're calling BeginReceive the thread that accepts
>> incomming
>> connections - it will be sufficient to stop that thread.
>>
>> Also if you will call Socket.Shutdown, the exception will be thrown in
>> EndReceive(...) or appropriate SocketError returned ( under .NET 2.0 )
>>
>> --
>> Vadym Stetsyak aka Vadmyst
>> http://vadmyst.blogspot.com
>>
>> "Larry Herbinaux" <La************@discussions.microsoft.com> wrote in
>> message news:2A**********************************@microsof t.com...
>> > It is possible to have an outstanding BeginReceive call when closing
>> > a
>> > socket. There isn't a call from the IAsyncResult to cancel this if
>> > it
>> > hasn't
>> > completed. This can be an issue when you are accepting a large
>> > number
>> > of
>> > connections in a long running process.
>> >
>> > First off, why isn't there a method for cancelling these requests?
>> >
>> > Is there a work around that anyone knows about?
>> >
>> > My state object that is referenced by the request is large, so I at
>> > least
>> > try to cleanup this reference, but there is still an OS object that
>> > is
>> > queued
>> > for this request that doesn't get freed.
>>
>>
>>

[Larry Herbinaux]

Thanks Vadmyst,

I solved my issue and it actually didn't have anything to do with
Outstanding Begin Receives. I will explain after quickly responding to point
#2.

#2:
I'm glad that .Net 2.0 throws an exception if you Shutdown both or receive.
It's interesting that prior to 2.0, Microsoft recommended shutting down both,
but this does not make any sense because you wouldn't be able to receive the
close handshake from the client. Greath, thanks for that tidbit of
information. We have 2.0, but I am going to wait a bit before migrating to
production environment:->

My main issue is that my objects used during the connection were not being
GC'd. I thought I had modelled each reference in a simple application and it
should that memory should be released. The only other thing that could have
held on to it was a state object (which also held a refernce to my objects)
be held by the OS which would root everything. I was wrong about this
assumption and overlooked something that would have taken me a while to
figure out if it weren't for SciTech's Memory Profiler. What's really cool
about the profiler is that you can do differences between heap snapshots and
also it will show you were an object is rooted. It only took me five mitues.
If you haven't used it before, then I would recommend getting it. I will
explain my structures and then the problem.

TCPServer - This has a hashtable that holds TCPServerClient.

TCPServerClient - Wrapper Socket class for the socket communicating with
client. It holds a reference to TCPServer (for sending data and passing a
reference to itself when application wants to send data) and
IAppProtocolProcessor.

IAppProtocolProcessor - An application programmer creates a class that
implements the interface (OnAccept, OnReceive, OnDisconnect, OnTimeout).
When OnAccept is called, a reference to TCPServerClient is passed and the
implement class stores a refernce to it in order to Send data among other
things. This class also implements a static method based on the
CreateAppProtocol delegate. This delegate is passed into the TCPServer
constructor so that it can create an instance in the AcceptCallback.

So there is a circular reference between TCPServerClient and
IAppProtocolProcessor, the TCPServerClient has a reference to TCPServer, and
the TCPServerClient instance is referenced in the TCPServer hashtable. When
I close a connection, I simply remove the TCPServerClient form the hashtable.
My first worry was that the refrence form TCPServerClient up to TCPServer
was causing the problem because TCPServer is long-standing. This wasn't the
issue at all and I successfully modelled it. My problem was that in the
TCPServerClient, the timer was holding a reference to the
IAppProtocolProcessor's OnTimeout callback. During the connection close, I
stop the timer but do not remove the reference, and since this his an OS
resource it is rooted. It wouldn't have dawned on me to look there; thank
God for Memory Profiler tools (or actually, thank SciTech:->).

"Vadym Stetsyak" wrote:

For #1

If there is no references to your state object then it will be GC-ed.

BeginReceive call is mapped to WSARecv(...). This is Winsock2 function. For
async io it uses
WSAOVERLAPPED structure. This structure holds the state, however, it holds a
pointer so you won't be able to see the object using it.

Yes, you can use CLR Profiler to verify how many state objects are there in
the memory, but for that you do not need any OS structures.

For #2

Yes, returting 0 is graceful shutdown, however world is not perfect, you may
expereience hardware problems ( on the wire, remote host crashed etc )

Using Socket.Shutdown with Both or Receive will cause an exception or socket
error ( in .NET 2.0 ).
From the doc about Shutdown
"Setting how to Receive specifies that subsequent calls to Receive are not
allowed"
You can use this SocketShudown options to control, the way how connection is
being closed ( with or without sending/receiving data after shutdown, but
before close )

--
Vadym Stetsyak aka Vadmyst
http://vadmyst.blogspot.com

"Larry Herbinaux" <La************@discussions.microsoft.com> wrote in
message news:E4**********************************@microsof t.com...

Thanks a lot this really helped. I just want to clarify that I understand
your response.

As for your answer to #1, if this is the case, then I don't have to worry
about the outstanding requests holding on to my state object, this should
all
be garbage collected, right? Do you know the name of the structure that
the
OS holds for these requests? I could use CLR Profiler to verify this,
huh?

As for #2, I didn't know that there is a possibility of an exception being
thrown when closing the socket; I've always seen a count of 0 and do the
appropriate thing. In most cases, I do a shutdown of the send channel and
wait for the other socket to respond in a similar fashion before I
shutdown
the receive channel. I don't like Microsoft's recommendation of shutting
down both channels because this causes an exception in most cases to be
thrown in the outstanding asynchronous receive request (previously, I
handled
this but logged it and I don't like cluttering the log for this purpose).
The only time I do a shutdown of both and close is if the server
recognizes a
connection that has been inactive for a period of time (e.g. physical line
was severed, etc.).

Thanks again for your input.

"Vadym Stetsyak" wrote:

> 1) If I do a close on the socket, will this cleanup any outstanding
> receive
> / send requests?

Yes

> 2) I am holding on to the IAsyncResult for the outstanding receive and
> send
> (in my wrapper socket class), so I could use the EndReceive prior to
> closing
> the socket. The issue here is that if for some reason the remote
> (client
> socket) is not closed, this call will block (at least that's what MSDN
> states). I think 99% of the time the client socket will be closed, but
> I
> also have a cleanup process that checks if we haven't had read / write
> data
> from the socket for a certain amount of time, then I have to do a hard
> close
> from the server...would the EndReceive block in this case?

EndReceive will block for timeout, of Socket keepalive option was not
specified.
If server will close the socket EndReceive in the client side will throw
an
exception, or
return 0 bytes, indicating graceful connection close ( more probable ).

--
Vadym Stetsyak aka Vadmyst
http://vadmyst.blogspot.com

"Larry Herbinaux" <La************@discussions.microsoft.com> wrote in
message news:71**********************************@microsof t.com...
> Thanks Vadym,
>
> On your first point, the async request is being performed in a thread
> pool
> thread (e.g. when the AsyncAccept callback is called it is on a thread
> in
> the
> thread pool and I immediately do a BeginRead within this thread), so
> this
> thread never exits, right? I'm sure that the request is registered in
> the
> OS
> and when an IOCompletionPort is signaled, the callback runs on a thread
> in
> the thread pool and this process starts all over again until we close
> the
> socket.
>
> Depending on who (client / server) and how the socket is closed, there
> still
> could be an outstanding async receive request. We are using 1.1, so in
> your
> second point I'm not sure it applies. A couple of questions on this:
>
> 1) If I do a close on the socket, will this cleanup any outstanding
> receive
> / send requests?
>
> 2) I am holding on to the IAsyncResult for the outstanding receive and
> send
> (in my wrapper socket class), so I could use the EndReceive prior to
> closing
> the socket. The issue here is that if for some reason the remote
> (client
> socket) is not closed, this call will block (at least that's what MSDN
> states). I think 99% of the time the client socket will be closed, but
> I
> also have a cleanup process that checks if we haven't had read / write
> data
> from the socket for a certain amount of time, then I have to do a hard
> close
> from the server...would the EndReceive block in this case?
>
> "Vadym Stetsyak" wrote:
>
>> Async I/O will be canceled if the thread that initiated that I/O will
>> exit.
>>
>> This means if you're calling BeginReceive the thread that accepts
>> incomming
>> connections - it will be sufficient to stop that thread.
>>
>> Also if you will call Socket.Shutdown, the exception will be thrown in
>> EndReceive(...) or appropriate SocketError returned ( under .NET 2.0 )
>>
>> --
>> Vadym Stetsyak aka Vadmyst
>> http://vadmyst.blogspot.com
>>
>> "Larry Herbinaux" <La************@discussions.microsoft.com> wrote in
>> message news:2A**********************************@microsof t.com...
>> > It is possible to have an outstanding BeginReceive call when closing
>> > a
>> > socket. There isn't a call from the IAsyncResult to cancel this if
>> > it
>> > hasn't
>> > completed. This can be an issue when you are accepting a large
>> > number
>> > of
>> > connections in a long running process.
>> >
>> > First off, why isn't there a method for cancelling these requests?
>> >
>> > Is there a work around that anyone knows about?
>> >
>> > My state object that is referenced by the request is large, so I at
>> > least
>> > try to cleanup this reference, but there is still an OS object that
>> > is
>> > queued
>> > for this request that doesn't get freed.
>>
>>
>>

[Vadym Stetsyak]

I'm glad you resolved your issues :8-)

--
Vadym Stetsyak aka Vadmyst
http://vadmyst.blogspot.com