Getting serilized data size

[John J. Hughes II]

Using the below code I am send multiple sterilized object across an IP port.
This works fine if only one object is received at a time but with packing
sometimes there is more then one object or half an object in the received
data. If I place the data in a memory stream on the received side is there
a way to determine where one ends and the next one start?

Since the deserializer stream seems to move the pointer I am trying to look
at the next couple bytes to determine if there is more data in the stream.
It seems that serialized data starts with 0,1 but I can not confirm this.
Is there maybe a rule?

I have also noticed that the number of bytes recieved is always bigger then
the number of bytes deserizlized, any clue why?

CODE 1
IFormatter f = new BinaryFormatter ();
MemoryStream ms = new MemoryStream();
f.Serialize(ms, cmd);
byte[] b = ms.GetBuffer();
this.socket.Beg inSend(b, 0, b.Length, SocketFlags.Non e, new
AsyncCallback(S end_Callback), this.socket);

CODE 2
IFormatter i = new BinaryFormatter ();
MemoryStream ms = new MemoryStream(o, 0, len);
ms.Position = 0;
while(ms.Positi on < len)
{
long position = ms.Position;
if(!(ms.ReadByt e() == 0 && ms.ReadByte() == 1))
break;
ms.Position = position;

object obj = i.Deserialize(m s);
.....

Regards,
John

[Helge Jensen]

Sahil Malik wrote:

[ Sahil Malik trying to get large-object-heap problems ]

What am I missing? :)

You are just writing to one MemoryStream, which does have O(n)
mem-usage, it's not an amortization problem (like the one in

Below is the test-program that i originally used to check my assumption.
BTW: That was before .net-sp1, so things may have changed.

I have tried running it again today on SP1, and it doesn't seem to be as
bad anymore. The specific problems I had involved a server that had the
SoapFormatter leak as well, so it may have manifested worse on that machine.

The .NET GC fan-crowd is gonna come down on me like a ton of bricks
saying that the memory-profile of this is just a feature of the GC, not
collecting that memory, reusing it and giving it back to the OS if
needed. But my observation at the time was, that .NET processes using
the large-memory-heap could grind the machine to a halt, and even cause
out-of-memory situations by not releasing memory not referenced by any
object.

The program should use roughly 3*bts.Length[1] memory ~ 30Mb (bts,
stream, bts2) but did eat up all available RAM on an 256Mb machine,
making all other programs swap out, and the machine was unable to
recover from that state, since the TaskManager couldn't start so i was
unable to kill the program.

1. It is checked that the serialization is roughly the size of bts.Length.

--
Helge

[Helge Jensen]

Helge Jensen wrote:

mem-usage, it's not an amortization problem (like the one in

ListView, where inserting elements seems to be O(n^2)

--
Helge

[John J. Hughes II]

Helge Jensen

Yes I am serializing them through the same connection and if I implement
handshaking which I have done in the past with a header showing the expected
size it works fine. Normally protocol stuff: send header with size, read
data until size reached, ack data, send next header/data, etc...

I do try avoid using a memory stream when possible if for no other reason it
has more overhead then a byte array but as far as I am aware the only way to
deserialize the data is to create a memory stream. I try to compensate for
the memory issued but setting values to null when done and also manually
disposing of object when possible. Lets face it .NET like to eat memory and
is not always very quick about giving it back.

As I said I have another application that does handshaking with large data
chucks which I compress before sending and have not really noticed a large
problem with memory usage. I have allocated up to 50 meg chuck, compressed
them down to couple megs and send them, normally the program does not go
over 150 meg usage and that is with debugging turned on.

But yes I agree I am going to have to use some kind of protocol and I will
not put it in a memory stream to decode later. I was just hoping when I
started this thread they MS had built something in that I was not seeing and
based on this thread it does not exist.

Regards,
John

[John J. Hughes II]

Helge Jensen

Wow, thanks for the code... it will take a while to digest but what I have
seen looks good :)

I put the link and e-mail address in the project so if I find anything I
will update you.

Regards,
John

[John J. Hughes II]

Sahil Malik

Yes I agree it needs a wrapper.

Regards,
John