Socket Async Send I/O - Resource Leak / Callbacks not getting called?

[dennis.richardson]

Greetings all.

Here's a problem that's been driving me nuts for the last 48 hours.
I'm hoping that someone has come across this before.

I have a C# Application that reads a UDP broadcast (asynchronously ).
Then it repackages these UDP packets and sends them to a subscriber
via TCP.

Now, I can read the UDP stream all day long without the application
ever using more memory. Everything get's GC'ed just fine.

If I connect a subscriber, and start dispatching the incoming packets
via BeginSend() everything goes according to plan.. for a short while.
Suddenly the application memory consumption starts increasing... and
I've profiled the memory with ".NET Memory Profiler".
It seems that there are many instances of something called
"OverlappedData " accumulating, which I think they mean to be:
"NativeOverlapp ed".
These are created, but never collected by the GC.

Yes.. I am calling "EndSend()" on the callback from "BeginSend( )" in
case you are wondering.
But here's the thing... I've implemented a counter that increments
each BeginSend() dispatch and one counter that increments each
EndSend() callback. After a while I start seeing small differences
that do not go away after the GC kicks in, indicating that a few
Begins were naughty and didn't call back!!

I've googled everywhere and chased up any reference on resource leaks
in Async I/O under .NET but found nothing that indicated that
callbacks might get lost.

I'm at the end of my wits here. Hualp!! Has anyone ever encountered
something similar?

Many thanks in advance,

Dennis Richardson

PS: I've tried this under Windows Vista (64-bit) & Windows XP (32-
bit), no change whatsoever. I'm sure I'm doing something wrong... but
what?

[=?Utf-8?B?UGV0ZXIgQnJvbWJlcmcgW0MjIE1WUF0=?=]

Can you post a "short but complete" block of working sample code to
illustrate what you are doing that's causing the issue?
Peter

--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net


"de************ ***@gmx.net" wrote:

Greetings all.

Here's a problem that's been driving me nuts for the last 48 hours.
I'm hoping that someone has come across this before.

I have a C# Application that reads a UDP broadcast (asynchronously ).
Then it repackages these UDP packets and sends them to a subscriber
via TCP.

Now, I can read the UDP stream all day long without the application
ever using more memory. Everything get's GC'ed just fine.

If I connect a subscriber, and start dispatching the incoming packets
via BeginSend() everything goes according to plan.. for a short while.
Suddenly the application memory consumption starts increasing... and
I've profiled the memory with ".NET Memory Profiler".
It seems that there are many instances of something called
"OverlappedData " accumulating, which I think they mean to be:
"NativeOverlapp ed".
These are created, but never collected by the GC.

Yes.. I am calling "EndSend()" on the callback from "BeginSend( )" in
case you are wondering.
But here's the thing... I've implemented a counter that increments
each BeginSend() dispatch and one counter that increments each
EndSend() callback. After a while I start seeing small differences
that do not go away after the GC kicks in, indicating that a few
Begins were naughty and didn't call back!!

I've googled everywhere and chased up any reference on resource leaks
in Async I/O under .NET but found nothing that indicated that
callbacks might get lost.

I'm at the end of my wits here. Hualp!! Has anyone ever encountered
something similar?

Many thanks in advance,

Dennis Richardson

PS: I've tried this under Windows Vista (64-bit) & Windows XP (32-
bit), no change whatsoever. I'm sure I'm doing something wrong... but
what?

[Michael Rubinstein]

Hi Dennis,

I'm at the end of my wits here. Hualp!! Has anyone ever encountered
something similar?

yes, quite similar.

But here's the thing... I've implemented a counter that increments
each BeginSend() dispatch and one counter that increments each
EndSend() callback. After a while I start seeing small differences
that do not go away after the GC kicks in, indicating that a few
Begins were naughty and didn't call back!!

Do you experience data loss like packets sent, but never received by the
client?
Do you pass an 'out Error' object to BeginSend()? It would not change the
outcome, but could provide a clue.

My server applications get data over UDP, process it and pass processed
data to clients via TCP, same thing as you do. The memory use creeps up
until it reaches a certain limit, then GC 'wakes up' and reduces the amount
of memory for application use about 2.5 - 3 times and from that moment kept
at lower level then at program start. It may take a week before it happens.
I was quite nervous about this. My server applications run unattended for
several month. Since they never crash I am more confident by now. Still, I
have serious reservations about Asynchronous Sockets implementation under
..NET 2.0. A call back never called is just half of the potential problem.
The other half is when the call back is called but the result of Endxxx() is
a disposed object. I never experienced it with EndSend(), but with
EndReceive() it is 100% reproducible.

Michael

<de************ ***@gmx.netwrot e in message
news:11******** *************@n 35g2000prd.goog legroups.com...

Greetings all.

Here's a problem that's been driving me nuts for the last 48 hours.
I'm hoping that someone has come across this before.

I have a C# Application that reads a UDP broadcast (asynchronously ).
Then it repackages these UDP packets and sends them to a subscriber
via TCP.

Now, I can read the UDP stream all day long without the application
ever using more memory. Everything get's GC'ed just fine.

If I connect a subscriber, and start dispatching the incoming packets
via BeginSend() everything goes according to plan.. for a short while.
Suddenly the application memory consumption starts increasing... and
I've profiled the memory with ".NET Memory Profiler".
It seems that there are many instances of something called
"OverlappedData " accumulating, which I think they mean to be:
"NativeOverlapp ed".
These are created, but never collected by the GC.

Yes.. I am calling "EndSend()" on the callback from "BeginSend( )" in
case you are wondering.
But here's the thing... I've implemented a counter that increments
each BeginSend() dispatch and one counter that increments each
EndSend() callback. After a while I start seeing small differences
that do not go away after the GC kicks in, indicating that a few
Begins were naughty and didn't call back!!

I've googled everywhere and chased up any reference on resource leaks
in Async I/O under .NET but found nothing that indicated that
callbacks might get lost.

I'm at the end of my wits here. Hualp!! Has anyone ever encountered
something similar?

Many thanks in advance,

Dennis Richardson

PS: I've tried this under Windows Vista (64-bit) & Windows XP (32-
bit), no change whatsoever. I'm sure I'm doing something wrong... but
what?

[=?Utf-8?B?RGVubmlzIFJpY2hhcmRzb24=?=]

"Peter Bromberg [C# MVP]" wrote:

Can you post a "short but complete" block of working sample code to
illustrate what you are doing that's causing the issue?
Peter

Hello Peter,

Thanks for your response (and Michael too).
Not sure that a working example would be short, but here's basically what
I'm doing:

We have a main Async read loop:

public void OnReceive(IAsyn cResult IAR)
{
// Read the data from the socket into the buffer.
// Then dispatch to TCP socket with BeginSend()
tcpSocket.Begin Send(... callback to OnSend);

// Queue next async read
udpSocket.Begin ReceiveFrom(... callback to OnReceive);
}

public void OnSend(IAsyncRe sult)
{
// Do EndSend and be done...
}

Now, I have played around with it some more and discovered the following:

If the next async receive is delayed until the EndSend Callback was called,
then the problem doesn't appear.
(in other words OnReceive ->BeginSend->OnSend->BeginReceive->OnReceive... etc)
So somehow, if there is only one callback active at a time, then there is no
problem. I thought that it might have to do with ThreadPool threads
terminating and hence leaving the callbacks hanging (I saw a tidbit about
that somewhere).

Am I doing something wrong?

Anyway, does anyone have an idea of how to do a rather simple thing like
read UDP socket and push into one or more TCP sockets? Wouldn't one do it
sort of like I described?

I'm happy to provide more details, just don't want to flood the message with
lots of code.

Regards,

Dennis

[=?Utf-8?B?RGVubmlzIFJpY2hhcmRzb24=?=]

"Michael Rubinstein" wrote:

Do you experience data loss like packets sent,
but never received by the client?

No, I'm not even worried about the client side at this stage. The server
crashing is enough of a problem.

The memory use creeps up until it reaches a certain limit,
then GC 'wakes up' and reduces the amount
of memory for application use about 2.5 - 3 times and from that moment kept
at lower level then at program start.

Well, these instances of "OverlappedData " (called so by the ".NET Memory
Profiler", I suspect it refers to isntances of NativeOverlappe d objects) do
not get collected by the GC (no matter how many generations, full-depth GC
does nothing, so somehow there must be a reference to these things somewhere).

[=?Utf-8?B?RGVubmlzIFJpY2hhcmRzb24=?=]

By the way, this guy had the same problem, apparently (didn't get an answer
though):

http://www.thescripts.com/forum/thread236349.html

[Willy Denoyette [MVP]]

<de************ ***@gmx.netwrot e in message
news:11******** *************@n 35g2000prd.goog legroups.com...

Greetings all.

Here's a problem that's been driving me nuts for the last 48 hours.
I'm hoping that someone has come across this before.

I have a C# Application that reads a UDP broadcast (asynchronously ).
Then it repackages these UDP packets and sends them to a subscriber
via TCP.

Now, I can read the UDP stream all day long without the application
ever using more memory. Everything get's GC'ed just fine.

If I connect a subscriber, and start dispatching the incoming packets
via BeginSend() everything goes according to plan.. for a short while.
Suddenly the application memory consumption starts increasing... and
I've profiled the memory with ".NET Memory Profiler".
It seems that there are many instances of something called
"OverlappedData " accumulating, which I think they mean to be:
"NativeOverlapp ed".
These are created, but never collected by the GC.

Yes.. I am calling "EndSend()" on the callback from "BeginSend( )" in
case you are wondering.
But here's the thing... I've implemented a counter that increments
each BeginSend() dispatch and one counter that increments each
EndSend() callback. After a while I start seeing small differences
that do not go away after the GC kicks in, indicating that a few
Begins were naughty and didn't call back!!

That means that the send actually didn't occur , this means that all managed
and unmanaged resources associated with the logical connection (eg. Winsock
buffers) stay allocated which leads to an increased Private bytes
consumption. The GC cannot free this memory, nor should it because the
action did not complete yet. This would also mean that some subscribers
don't receive all of their expected data, right, now the questions are, does
it happen for particular , or arbitrary subscribers, how are they connected
(LAN, WAN, routed switched or both) how do you finally recover from this
(reconnect or restart the server?). Did you fire up a network trace?

Willy.

[=?Utf-8?B?RGVubmlzIFJpY2hhcmRzb24=?=]

"Willy Denoyette [MVP]" wrote:

now the questions are, does
it happen for particular , or arbitrary subscribers, how are they connected
(LAN, WAN, routed switched or both) how do you finally recover from this
(reconnect or restart the server?). Did you fire up a network trace?

Thanks for your response Willy.

There's only one subscriber at the moment (haven't even added more than one)
where this happens. The subscriber seems to be fine, no problems on that end.
The two machines are on the same network segement (1GBit/s).

When you say the "sends didn't happen" wouldn't that imply that nothing
further should go through? In other words, the socket should be clogged?
Because that's not what I'm seeing. Yes, some callbacks don't seem to
complete, but the socket still transmits (the correct data).

It's a bit bizarre why this would happen, really, from a performance
standpoint as we are talking about only 1,000 UDP packets repacked into TCP
per second. So that can't be it?
To re-emphasize, this only happens on sending stuff. Receiving works fine
without any glitches.

Regards,

Dennis

[Michael Rubinstein]

Dennis,

No, I'm not even worried about the client side at this stage. The server
crashing is enough of a problem.

does it (crash)?

from your code:

>>>>>>>>>>>>>>> >

We have a main Async read loop:

public void OnReceive(IAsyn cResult IAR)
{
// Read the data from the socket into the buffer.
// Then dispatch to TCP socket with BeginSend()
tcpSocket.Begin Send(... callback to OnSend);

// Queue next async read
udpSocket.Begin ReceiveFrom(... callback to OnReceive);
}

public void OnSend(IAsyncRe sult)
{
// Do EndSend and be done...
}

>>>>>>>>>>>

I do very similar things. In my code I call udpSocket.Begin ReceiveFrom(...
callback to OnReceive) before tcpSocket.Begin Send(... callback to OnSend).
Any particular reason for placing UDP async read after TCP BeginSend()? My
applications send processed UDP data to multiple TCP clients that connect
and disconnect 'at will'. I am uneasy about a few things in Async Socket
model, but it seems to be working.

Michael

"Dennis Richardson" <De************ **@discussions. microsoft.comwr ote in
message news:53******** *************** ***********@mic rosoft.com...

"Michael Rubinstein" wrote:

>Do you experience data loss like packets sent,
but never received by the client?

No, I'm not even worried about the client side at this stage. The server
crashing is enough of a problem.

>The memory use creeps up until it reaches a certain limit,
then GC 'wakes up' and reduces the amount
of memory for application use about 2.5 - 3 times and from that moment
kept
at lower level then at program start.

Well, these instances of "OverlappedData " (called so by the ".NET Memory
Profiler", I suspect it refers to isntances of NativeOverlappe d objects)
do
not get collected by the GC (no matter how many generations, full-depth GC
does nothing, so somehow there must be a reference to these things
somewhere).

[=?Utf-8?B?RGVubmlzIFJpY2hhcmRzb24=?=]

"Michael Rubinstein" wrote:

I do very similar things. In my code I call udpSocket.Begin ReceiveFrom(...
callback to OnReceive) before tcpSocket.Begin Send(... callback to OnSend).
Any particular reason for placing UDP async read after TCP BeginSend()?

Thanks Michael. The reason I did it in that order (BeginReceive after
BeginSend) is because I wanted to be reasonably sure that not another
"OnReceive" jumped in there and fired off a BeginSend before this one (which
could then be possible (although probably quite unlikely).
I will give this a try, although I have little hope that it'll make a
difference.