Reading big chunk of binary data from socket

[Willy Stevens]

Hello,

In my application I have to read sometimes quite big chunk of binary data.
I have a buffer which default size is 32000 bytes.

But how could I read binary data that exceeds 32000 bytes?
Is allocating more size to buffer and copy data that exceeds 32000 bytes at
tail the only way
or are there any other possibilities?

Cheers W

[Peter Duniho]

Willy Stevens wrote:

Hello,

In my application I have to read sometimes quite big chunk of binary data.
I have a buffer which default size is 32000 bytes.

But how could I read binary data that exceeds 32000 bytes?
Is allocating more size to buffer and copy data that exceeds 32000 bytes at
tail the only way
or are there any other possibilities?

What kind of socket?

If it's a UDP or other message-oriented protocol, you need to provide a
buffer large enough to contain the message. Some protocols or service
providers may support partial messages via the Partial socket flag
(MSG_PARTIAL in Winsock), but I would say it's better to just provide a
large enough buffer rather than relying on that flag being usable.

For TCP or other stream-oriented protocols, you simply read data in
chunks as large as you feel you can reasonably handle. So if your
buffer is only 32K, you just read 32K at a time until you have extracted
all of the data from the stream.

What you do with the data after each read depends on what you're going
to do with it later. For example, you might be reading a JPEG file that
you want to display, so you'd have to allocate a data structure large
enough to hold all of the JPEG data and iteratively copy the data you
read from the Socket into that data structure. As an example, you might
create a MemoryStream, initializing its size to be equal to the size of
data you're going to read, and just write the read data chunks to the
MemoryStream instance. Then you can use the MemoryStream as the
parameter for Image.FromStream() to create the JPEG.

On the other hand, if it's data you want to save to a file, you'd
iteratively read the data from the Socket and write the new data to the
file with each read.

Pete

[Jon Skeet [C# MVP]]

Willy Stevens <[e-mail]:

In my application I have to read sometimes quite big chunk of binary data.
I have a buffer which default size is 32000 bytes.

But how could I read binary data that exceeds 32000 bytes?
Is allocating more size to buffer and copy data that exceeds 32000 bytes at
tail the only way or are there any other possibilities?

Well, you could create a list of byte arrays, reading one byte array at
a time. The easiest way would just be to use a stream abstraction over
the socket, however, and just keep reading until you're finished.

See http://pobox.com/~skeet/csharp/readbinary.html for some caveats.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

[Willy Stevens]

Hello and thanks for both,

>For TCP or other stream-oriented protocols, you simply read data in chunks
as large as you feel you can reasonably handle. So if your buffer is only
32K, you just read 32K at a time until you have extracted all of the data
from the stream.

Yes it's TCP so this approach suites well for me,

Thanks
"Peter Duniho" <Np*********@NnOwSlPiAnMk.comwrote in message
news:13*************@corp.supernews.com...

Willy Stevens wrote:

>Hello,

In my application I have to read sometimes quite big chunk of binary
data.
I have a buffer which default size is 32000 bytes.

But how could I read binary data that exceeds 32000 bytes?
Is allocating more size to buffer and copy data that exceeds 32000 bytes
at tail the only way
or are there any other possibilities?

What kind of socket?

If it's a UDP or other message-oriented protocol, you need to provide a
buffer large enough to contain the message. Some protocols or service
providers may support partial messages via the Partial socket flag
(MSG_PARTIAL in Winsock), but I would say it's better to just provide a
large enough buffer rather than relying on that flag being usable.

For TCP or other stream-oriented protocols, you simply read data in chunks
as large as you feel you can reasonably handle. So if your buffer is only
32K, you just read 32K at a time until you have extracted all of the data
from the stream.

What you do with the data after each read depends on what you're going to
do with it later. For example, you might be reading a JPEG file that you
want to display, so you'd have to allocate a data structure large enough
to hold all of the JPEG data and iteratively copy the data you read from
the Socket into that data structure. As an example, you might create a
MemoryStream, initializing its size to be equal to the size of data you're
going to read, and just write the read data chunks to the MemoryStream
instance. Then you can use the MemoryStream as the parameter for
Image.FromStream() to create the JPEG.

On the other hand, if it's data you want to save to a file, you'd
iteratively read the data from the Socket and write the new data to the
file with each read.

Pete