HELP! - .NET sequences segmented packets before data available in NetworkStream?

[Ryan]

Hi,

I am confused with how NetworkStream works.

My application needs to handle heavy requests sent through TCP socket
connection. I use NetworkStream.Read method to get the stream data. The
data are sent from the client servers, black boxes for my project team. But
we have an agreement to the client vendor that we exchange only fixed-length
data, say, 200 bytes per data set. A data set is a bombination of data
fields, like 56 bytes of character, followed by 4 bytes of integer, followed
by 2 bytes of characters, and so on.

I notice that the lower layer of TCP splits some packets into smaller
packets, as expected. I assumed before stream data are loaded into
NetworkStream, the smaller packets will be combined together by their
sequence number, so programmers don't need to worry about the segmentation
things of TCP stream socket. However, it seems the smaller packets are put
into the NetworkStream as soon as they are received. This causes disaster
because the only way we retrieve data sets is by the fixed length. If an
incompleted data set is read from NetworkStream, the following data sets
cannot be understood by our application.

Is it true that .NET doesn't sequence the smaller packets before they are
loaded into NetworkStream? Or do i miss something? Losts of sample codes
available on MSND and Google search, but can't find any article discussing
about this.

What could I do to sovle this problem?

Any suggestion or hint is highly appreciated!



Ryan

[Zane Thomas [.NET/C# MVP]]

"Ryan" <ry**@cradle.com.tw> wrote:

Is it true that .NET doesn't sequence the smaller packets before they are
loaded into NetworkStream?

Yes. It is called Network*Stream* because you receive a stream of bytes, there
is no 'record' or 'packet' as you are thinking of them.

In the Free Stuff folder on my website at www.abderaware.com you will find
several examples of socket programming. A couple of them use a class called
TcpPacketizer which takes care of sending and receiving records (packets) over a
socket's natural stream of bytes. You could adapt it to use NetworkStream, or
simply use the Socket class as I did in the sample projects.
--
Abderaware
Fine Components For .NET
Turn on, tune in, download.
zane a@t abderaware.com

[Ryan]

Hi, Zane,

Thank you for your reply.

I have downloaded the codes and read through TcpPacketizer class, it seems
the codes deal with simalar things that networkStream does, in asynchronous
way. But I can't find the lines dealing with fragmented packets. If the
codes have taken that into consideration, could you please show me which
block of the class handles the job related to combining fragmented packets?

If the codes don't mean to deal with that, do you have any suggestion for us
to tackle down the problem?

BTW, interesting site, i showed it to my colleages.

Ryan
"Zane Thomas [.NET/C# MVP]" <za**@abderaware.com> wrote in message
news:3f***************@news.microsoft.com...

"Ryan" <ry**@cradle.com.tw> wrote:

Is it true that .NET doesn't sequence the smaller packets before they are
loaded into NetworkStream?
Yes. It is called Network*Stream* because you receive a stream of bytes,

there is no 'record' or 'packet' as you are thinking of them.

In the Free Stuff folder on my website at www.abderaware.com you will find
several examples of socket programming. A couple of them use a class called TcpPacketizer which takes care of sending and receiving records (packets) over a socket's natural stream of bytes. You could adapt it to use NetworkStream, or simply use the Socket class as I did in the sample projects.
--
Abderaware
Fine Components For .NET
Turn on, tune in, download.
zane a@t abderaware.com

[Zane Thomas [.NET/C# MVP]]

Ryan,

But I can't find the lines dealing with fragmented packets. If the
codes have taken that into consideration, could you please show me which
block of the class handles the job related to combining fragmented packets?

The code can be somewhat confusing because it's all asynchronous. Maybe I
should write the blocking methods too. :-)

Before a 'packet' is written by the sender a four-byte length is written out,
and then the data itself. On the receive side it goes like this:

1. An instance is created and initialized with a socket to use.

2. It invokes Socket.BeginReceive passing OnReceiveLength as the method to
invoke when some data arrives.

3. OnReceiveLength reads upto four bytes (no more than four!), if there are not
yet four bytes goto 2 (basically, the code is inline in OnReceiveLength).

4. Once four bytes have been received the packet length is known and the
BeginReceive is invoked again, this time with OnReceiveMessage as the callback.

5. In OnReceiveMessage as many bytes as possible, upto but not exceeded the
packet length from step 3, are read and saved.

6. If the packet is not yet complete BeginReceive is invoked with
OnReceiveMessage as the callback, again. If the packet is complete the
TcpPacketizer user's callback is invoked and the code (inline again) returns to
step 2.
Blocking this would be:

1. Keep reading until you have four bytes. That's the length.
2. Keep reading until you have received a length's worth of bytes.

That's all.

--
Abderaware
Fine Components For .NET
Turn on, tune in, download.
zane a@t abderaware.com

[Ryan]

Zane,

I can't thank you enough for your patient explanation.

I start to believe what we are dealing with is a little bit different. We
are doing ok with retrieving the stream data of a single packet from
networkstream, since the data sender agrees to send a fixed-length data set.
However, if a data set (or a message package) is splited into two fragmented
packets, and if these two fragmented packets are separated by other
packet(s) in NETWORKSTREAM ( i have no idea if this could happen or not?!)
where we will retrive the stream data, the programmer will have a nightmare
to put them back together.
If this is not very clear to you, drop me a message. I will draw a picture
to explain that and send it to your email account.
Ryan


"Zane Thomas [.NET/C# MVP]" <za**@abderaware.com> wrote in message
news:3f***************@news.microsoft.com...

Ryan,

But I can't find the lines dealing with fragmented packets. If the
codes have taken that into consideration, could you please show me which
block of the class handles the job related to combining fragmented

packets?
The code can be somewhat confusing because it's all asynchronous. Maybe I
should write the blocking methods too. :-)

Before a 'packet' is written by the sender a four-byte length is written out, and then the data itself. On the receive side it goes like this:

1. An instance is created and initialized with a socket to use.

2. It invokes Socket.BeginReceive passing OnReceiveLength as the method to invoke when some data arrives.

3. OnReceiveLength reads upto four bytes (no more than four!), if there are not yet four bytes goto 2 (basically, the code is inline in OnReceiveLength).

4. Once four bytes have been received the packet length is known and the
BeginReceive is invoked again, this time with OnReceiveMessage as the callback.
5. In OnReceiveMessage as many bytes as possible, upto but not exceeded the packet length from step 3, are read and saved.

6. If the packet is not yet complete BeginReceive is invoked with
OnReceiveMessage as the callback, again. If the packet is complete the
TcpPacketizer user's callback is invoked and the code (inline again) returns to step 2.
Blocking this would be:

1. Keep reading until you have four bytes. That's the length.
2. Keep reading until you have received a length's worth of bytes.

That's all.

--
Abderaware
Fine Components For .NET
Turn on, tune in, download.
zane a@t abderaware.com

[Zane Thomas [.NET/C# MVP]]

Ryan,

I start to believe what we are dealing with is a little bit different. We
are doing ok with retrieving the stream data of a single packet from
networkstream, since the data sender agrees to send a fixed-length data set.
However, if a data set (or a message package) is splited into two fragmented
packets, and if these two fragmented packets are separated by other
packet(s) in NETWORKSTREAM ( i have no idea if this could happen or not?!)
where we will retrive the stream data, the programmer will have a nightmare
to put them back together.

Ok, I think I understand the situation. The first thing you should know is that
if the sender sends you two records A and B, it is true that both A and B may
arrive either in one Receive or in multiple Receives ... but no data for B will
be Received until all the data for A has been Received. TCP takes care of that
for you.

What TCP _stream_ sockets do not do is guarantee that either A or B will be
received with one invocation of Receive. You might get all of A, and then B in
three parts, or A in two parts and B in one part. There is no predicting, it
depends upon network and socket stack conditions.

So ... if you're sending Records then you and the sender must agree on what a
record is. In the common internet protocols (smtp for instance) a record is a
string of characters followed by "crlf". But that's not a big help if you're
sending binary data. If you're sending arbitrary binary data that has no
recognizable record-start or record-end delimeters then you must precede the
binary data with a byte count. TcpPacketizer assumes the worst case and sends a
byte count before each record.

You and the data sender must cooperate in order to correctly send and receive
packets.
--
Abderaware
Fine Components For .NET
Turn on, tune in, download.
zane a@t abderaware.com

[Ryan]

Zane,

Now I am totally there.

Thank you so much. I appreciate your assistance.

Ryan
"Zane Thomas [.NET/C# MVP]" <za**@abderaware.com> wrote in message
news:3f***************@news.microsoft.com...

Ryan,

I start to believe what we are dealing with is a little bit different. Weare doing ok with retrieving the stream data of a single packet from
networkstream, since the data sender agrees to send a fixed-length data set.However, if a data set (or a message package) is splited into two fragmentedpackets, and if these two fragmented packets are separated by other
packet(s) in NETWORKSTREAM ( i have no idea if this could happen or not?!)where we will retrive the stream data, the programmer will have a nightmareto put them back together.
Ok, I think I understand the situation. The first thing you should know

is that if the sender sends you two records A and B, it is true that both A and B may arrive either in one Receive or in multiple Receives ... but no data for B will be Received until all the data for A has been Received. TCP takes care of that for you.

What TCP _stream_ sockets do not do is guarantee that either A or B will be received with one invocation of Receive. You might get all of A, and then B in three parts, or A in two parts and B in one part. There is no predicting, it depends upon network and socket stack conditions.

So ... if you're sending Records then you and the sender must agree on what a record is. In the common internet protocols (smtp for instance) a record is a string of characters followed by "crlf". But that's not a big help if you're sending binary data. If you're sending arbitrary binary data that has no
recognizable record-start or record-end delimeters then you must precede the binary data with a byte count. TcpPacketizer assumes the worst case and sends a byte count before each record.

You and the data sender must cooperate in order to correctly send and receive packets.
--
Abderaware
Fine Components For .NET
Turn on, tune in, download.
zane a@t abderaware.com