async sockets and threading

[Alexander Gnauck]

Hello,

while using async sockets I ran into a strange problem which occurs only
on some machines. I wrote a small demo application which can be used to
reproduce the problem. You can download it from here:
http://alex.ag-software.de/SocketTest.zip

If you press the "Connect in new Thread" button from the test
application you may be able to cause the System.IO.IOExc eption: Unable
to write data to the transport connection. While the other button which
executes the same code without a new thread works fine.

The async BeginWrite gets executed in the async EndConnect of the
socket. So the socket must be connected while the exception tells me it
isn't.
I've debugged and studied my code for hours but can't find the problem.
Is there anything I am doing wrong?

I would be great if you can run the code and tell me if you can cause
the exception or not, and which OS and Framework version you are running.

Thanks,
Alex

[Peter Duniho]

On Tue, 21 Oct 2008 09:58:40 -0700, Alexander Gnauck
<gn****@ag-software.dewrot e:

Hello,

while using async sockets I ran into a strange problem which occurs only
on some machines. I wrote a small demo application which can be used to
reproduce the problem. You can download it from here:
http://alex.ag-software.de/SocketTest.zip [...]

Please post a concise-but-complete code sample here. Not only should the
code accompany your question for archival purposes (this thread will be
archived by a number of providers, but your code sample won't be if you
only provide it at a separate web server), the fact is that if your code
is concise enough to be a valid example, it's concise enough to be
included in your post. If you think it's too long to post with your
question, then it's too long to be a true concise-but-complete code sample.

I should point out that just because your connect callback has been
called, that doesn't necessarily mean your socket has been connected. It
just means that the operation has been completed. You need to call
EndConnect() and only proceed with writing to the socket if no exception
occurs and you do in fact get back a valid socket.

Based on your description, it _seems_ likely that this is all fine (after
all, to what socket would you be writing if not the one returned by
EndConnect()?), but without the actual code it's not possible to say for
sure.

Pete

[Alexander Gnauck]

Peter Duniho wrote:

Please post a concise-but-complete code sample here.

ok, here is the example:

using System;
using System.IO;
using System.Text;

using System.Threadin g;
using System.Net;
using System.Net.Sock ets;

namespace Test
{
class Program
{
static void Main(string[] args)
{
Thread connectThread = new Thread(new
ParameterizedTh readStart(Conne ctThread));
connectThread.S tart();

Console.ReadLin e();
}

static void ConnectThread(o bject o)
{
Protocol p = new Protocol();
p.Open();
}

public class Protocol
{
private static Socket _socket;
private static NetworkStream _stream;
private const int BUFFERSIZE = 10240;
private static byte[] m_ReadBuffer = new byte[BUFFERSIZE];

public Protocol()
{
}

public void Open()
{
IPHostEntry ipHostInfo =
Dns.GetHostEntr y("www.google.c om");

IPAddress ipAddress = ipHostInfo.Addr essList[0];
IPEndPoint endPoint = new IPEndPoint(ipAd dress, 80);

_socket = new Socket(AddressF amily.InterNetw ork,
SocketType.Stre am, ProtocolType.Tc p);
_socket.BeginCo nnect(endPoint, new
AsyncCallback(E ndConnect), null);
}

private static void EndConnect(IAsy ncResult ar)
{
try
{
_socket.EndConn ect(ar);
_stream = new NetworkStream(_ socket, false);

string header = "GET / HTTP/1.1\r\n";
header += "Accept: */*\r\n";
header += "User-Agent: myAgent\r\n";
header += "Host: www.google.de\r \n\r\n";

Send(Encoding.U TF8.GetBytes(he ader));
}
catch (Exception ex)
{
Console.WriteLi ne(ex.Message);
}
}

public static void Send(byte[] bData)
{
try
{
_stream.BeginWr ite(bData, 0, bData.Length, new
AsyncCallback(E ndSend), null); // <<== exception gets raised here
}
catch (Exception ex)
{
Console.WriteLi ne(ex.Message);
}
}

private static void EndSend(IAsyncR esult ar)
{
_stream.EndWrit e(ar);
}
}
}
}

I should point out that just because your connect callback has been
called, that doesn't necessarily mean your socket has been connected.
It just means that the operation has been completed. You need to call
EndConnect() and only proceed with writing to the socket if no exception
occurs and you do in fact get back a valid socket.

I get no exception in EndConnect. In EndConnect I also create the
network stream because I have to work with streams to start the TLS
security layer on the connection later.

Alex

[Peter Duniho]

On Tue, 21 Oct 2008 23:24:03 -0700, Alexander Gnauck
<gn****@ag-software.dewrot e:

Peter Duniho wrote:

>Please post a concise-but-complete code sample here.

ok, here is the example: [...]

Thanks for the example. With that in hand, it looks to me as though the
problem is related to garbage collection. In particular, it appears that
the JIT compiler figures out that you don't refer to the _socket or
_stream class members once the thread you started exits, and it makes
those members eligible for garbage collection. And of course, when that
happens (well, once the finalizer is run), the connection winds up closed.

I added some synchronization to ensure that the Open() method doesn't
return until the EndSend() callback has been executed, and that reliably
makes the problem go away. That seems to confirm the GC as the culprit.

Now, whether that's _correct_ behavior, I'm not entirely sure. It _seems_
like a .NET bug to me. In particular, those are static members, and I
would have thought all class static members would be considered a root for
the purpose of garbage collection. In fact, one of the authoritative
articles on .NET garbage collection seems to say just that: " all the
global and static object pointers in an application are considered part of
the application's roots" (from
http://msdn.microsoft.com/en-us/magazine/bb985010.aspx).

So, at the very least, for this code example, it seems to me that the GC
is prematurely collecting the objects. If your real-world scenario also
keeps the objects in static members, then the same bug would be affecting
you.

Of course, if in your real-world scenario, these objects wind up
referenced by instance members that are in a class instance that itself
becomes unreachable due to the use of the thread, then _that_ would be
expected behavior. The fix there would be, of course, to keep a reference
to the instance storing the socket and stream instances, so that they
remain reachable and uncollected.

In either case, I think that the static-based version of the behavior is a
bug, and it would be very helpful if you would post a bug report to the
http://connect.microsoft.com/ web site, for the .NET Framework.

Pete

[Ben Voigt [C++ MVP]]

Peter Duniho wrote:

On Tue, 21 Oct 2008 23:24:03 -0700, Alexander Gnauck
<gn****@ag-software.dewrot e:

>Peter Duniho wrote:

>>Please post a concise-but-complete code sample here.

ok, here is the example: [...]

Thanks for the example. With that in hand, it looks to me as though
the problem is related to garbage collection. In particular, it
appears that the JIT compiler figures out that you don't refer to the
_socket or _stream class members once the thread you started exits,
and it makes those members eligible for garbage collection. And of
course, when that happens (well, once the finalizer is run), the
connection winds up closed.
I added some synchronization to ensure that the Open() method doesn't
return until the EndSend() callback has been executed, and that
reliably makes the problem go away. That seems to confirm the GC as
the culprit.

What if you instead saved the IAsyncResult returned by BeginConnect into a
static variable? Perhaps that is what is being collected (though presumably
the same one gets passed to the completion routine)?

>
Now, whether that's _correct_ behavior, I'm not entirely sure. It
_seems_ like a .NET bug to me. In particular, those are static
members, and I would have thought all class static members would be
considered a root for the purpose of garbage collection. In fact,
one of the authoritative articles on .NET garbage collection seems to
say just that: " all the global and static object pointers in an
application are considered part of the application's roots" (from
http://msdn.microsoft.com/en-us/magazine/bb985010.aspx).

So, at the very least, for this code example, it seems to me that the
GC is prematurely collecting the objects. If your real-world
scenario also keeps the objects in static members, then the same bug
would be affecting you.

I think the BCL should be keeping these objects alive completely independent
of the user's references from static members. After all, the only living
reference could be in the State parameter to BeginConnect.

>
Of course, if in your real-world scenario, these objects wind up
referenced by instance members that are in a class instance that
itself becomes unreachable due to the use of the thread, then _that_
would be expected behavior. The fix there would be, of course, to
keep a reference to the instance storing the socket and stream
instances, so that they remain reachable and uncollected.

In that case, presumably the completion routine would be a instance method,
so the AsyncCallback delegate passed to BeginConnect would reference the
instance and (should) keep it alive.

>
In either case, I think that the static-based version of the behavior
is a bug, and it would be very helpful if you would post a bug report
to the http://connect.microsoft.com/ web site, for the .NET Framework.

Pete

[Peter Duniho]

On Wed, 22 Oct 2008 07:31:16 -0700, Ben Voigt [C++ MVP]
<rb*@nospam.nos pamwrote:

What if you instead saved the IAsyncResult returned by BeginConnect into
a
static variable? Perhaps that is what is being collected (though
presumably
the same one gets passed to the completion routine)?

Nope. Not that it should have made a difference anyway, but I went ahead
and tried that and it had no effect.

[...]

>So, at the very least, for this code example, it seems to me that the
GC is prematurely collecting the objects. If your real-world
scenario also keeps the objects in static members, then the same bug
would be affecting you.

I think the BCL should be keeping these objects alive completely
independent
of the user's references from static members. After all, the only living
reference could be in the State parameter to BeginConnect.

I _think_ I agree. But, keep in mind that the framework only has to keep
a reference to the Socket instance as long as the asynchronous operation
hasn't completed. And there's no _requirement_ that the calling code pass
the Socket instance, directly or indirectly, as part of the call to
BeginConnect() (i.e. via the "state" argument).

So the only reference might be the one implicit in the asynchronous
operation itself, and of course once the callback has called EndConnect(),
with no more uses of the IAsyncResult argument to the callback,
technically any data referenced by that object is no longer reachable.

In other words, it's a variation on the "you need to call GC.KeepAlive()"
problem, in that the JIT compiler has detected that the variable is
actually unused past a certain point in the method, and so can be
collected.

Now, in this particular case, that _shouldn't_ apply. Not only is the
reference stored in a static variable, it's continued to be referenced in
the method. First directly by being passed to the NetworkStream
constructor, and then indirectly by the NetworkStream itself.

But note that in either case, we're relying on the static variable. The
"_socket" static variable _ought_ to be sufficient here, but if there's a
bug that causes it not to be, there's no reason to think that bug won't
also affect the "_stream" variable. If the JIT compiler is deciding that
these static variables are unreachable for some reason, then once the
"_stream" variable has been assigned, that's the last use in the method,
and the bug might (appears to) lead to the GC collecting the objects.

>Of course, if in your real-world scenario, these objects wind up
referenced by instance members that are in a class instance that
itself becomes unreachable due to the use of the thread, then _that_
would be expected behavior. The fix there would be, of course, to
keep a reference to the instance storing the socket and stream
instances, so that they remain reachable and uncollected.

In that case, presumably the completion routine would be a instance
method,
so the AsyncCallback delegate passed to BeginConnect would reference the
instance and (should) keep it alive.

That depends. I agree that if the completion routine were an instance
method, the fact that the code would be executing in the instance method
should be sufficient to keep the object from being collected, as long as
instance members continued to be used in the execution path (for example,
calling an instance method named "Send()" and in that method using an
instance member field named "_stream").

But there's no requirement that the callback method be an instance method,
and I suppose one could create code in which there's a bug where the
instance containing the references to the objects winds up unreachable.
How that could would ever be expected to work, I don't know. In that
hypothetical case, without a reference to the instance, it's difficult to
see how the non-instance Send() method would be able to expect to get at
the _stream instance field. I may be writing about a hypothetical
scenario that is a practical impossibility. :)

In any case, it's probably more useful to talk about the case at hand, in
which objects being referenced by static member fields are somehow being
garbage collected, even though they ought not to be. I don't have a good
answer for that, thus my belief that this is a bug in the framework
somewhere (probably the JIT compiler, but I have so little experience
debugging .NET at that level, I can't say for sure).

For me, the bottom line here is that the code that was posted looks like
it _should_ work fine. I am in fact able to get it to work fine by
keeping the Thread alive long enough for the operation to complete, which
suggests that something about the termination of the Thread is causing the
objects to become invalid.

Note that I don't even know for sure that this is a GC issue. That's just
my supposition, based on the behavior. It _looks_ like a GC issue. But
frankly, the lifetime of a Thread shouldn't really be affecting the
lifetime of objects not specifically tied to that Thread anyway. So if
the static variables are being collected, not only is that a bug, but the
fact that it's being done so just because a Thread exited seems also to be
a bug to me.

It's very odd.

Pete

[Alexander Gnauck]

Peter Duniho wrote:

Note that I don't even know for sure that this is a GC issue. That's
just my supposition, based on the behavior. It _looks_ like a GC
issue. But frankly, the lifetime of a Thread shouldn't really be
affecting the lifetime of objects not specifically tied to that Thread
anyway. So if the static variables are being collected, not only is
that a bug, but the fact that it's being done so just because a Thread
exited seems also to be a bug to me.

the problem is not related to the static members. When I wrote this test
case I first had a flat console application with all static variables
,functions and callbacks. Then I moved stuff to the Protocol class and
forgot to change the static modifiers. In my real application they are
not static and the behavior is the same.

It's very odd.

yes it is

Alex

[Peter Duniho]

On Wed, 22 Oct 2008 13:47:19 -0700, Alexander Gnauck
<gn****@ag-software.dewrot e:

[...] In my real application they are not static and the behavior is the
same.

Including the workaround?

[Alexander Gnauck]

Peter Duniho wrote:

>[...] In my real application they are not static and the behavior is
the same.

Including the workaround?

no, the product is a SDK/library. It would to complex to add the
workaround to the code, because the new thread is created in the end
users code and not my code. Normally threads are not required because
the library is based on async sockets and in the most cases only one
instance of the class is needed in an application.

I filed a bug report at Microsoft Connect. But they are unable to cause
the problem with the provided code. So it does work on many machines and
is crashing only on some machines. So the next question is why does the
GC behave different on different machines or operating systems. It could
be also related to the CPU. I am running XP pro on 2 machines and I get
the exception on both machines with .NET 2.0 and 3.5.

Alex

[Peter Duniho]

On Wed, 22 Oct 2008 23:23:38 -0700, Alexander Gnauck <gn****@alfag.d e>
wrote:

Peter Duniho wrote:

>>[...] In my real application they are not static and the behavior is
the same.

Including the workaround?

no, the product is a SDK/library. It would to complex to add the
workaround to the code, because the new thread is created in the end
users code and not my code. Normally threads are not required because
the library is based on async sockets and in the most cases only one
instance of the class is needed in an application.

I filed a bug report at Microsoft Connect. But they are unable to cause
the problem with the provided code. So it does work on many machines and
is crashing only on some machines. So the next question is why does the
GC behave different on different machines or operating systems. It could
be also related to the CPU. I am running XP pro on 2 machines and I get
the exception on both machines with .NET 2.0 and 3.5.

Unfortunately, I have no answers to your very good questions.

What has Microsoft done with the bug report? Are they continuing to try
to reproduce the bug? Or have they given up on it? They have the ability
to test bugs on an extensively broad variety of configurations; given that
the bug is definitely reproducible on _some_ computers, I don't think they
have any excuse for stopping until they've found a repro case and have
figured out what's going on.

It would be helpful if you could post a link to the bug report here. At
the very least, those of us who have been able to reproduce it can add our
vote of confirmation to the bug report.

Pete