Send large block of data failed

[zhimin]

Hi,
I'm writing a program to send large file(100m) through dotnet using
TCPListener & TCPClient, I'm sending the file with a ask and response loop:

1. Client send a flag 1 to server indicate it has data send to server.
2. Client send the buffer block size.
3. Client send the actual buffer to the server.
4. Server send a flag 1 to client indicating that the buffer has been
successfully receeived.
5. The next loop until all data of the file has been sent.

When the data block is about 5k, the code works very fine, but when I set
the block size up to about 10k, after some cycles, an exception raised and
the program failed.
The most strange is: the cycles number are different every time(with the
same conditions), such as the first time I failed at the 13th loop, when I'm
runing the same pair of applications the next time, I might failed at the
17th loop with the same file.

Any help would be appreciated.

Thanks

ps: the main code is listed below, the two cs files are attached.

Client
---------------
TcpClient client = new TcpClient(txtAddress.Text, 8000);
client.SendBufferSize = 1024*1024;
client.ReceiveBufferSize = 1024 * 1024;
client.NoDelay = true;

NetworkStream stream = client.GetStream();
byte[] length = null;
byte[] buffer = new byte[1024 *
Int32.Parse(this.numSize.Value.ToString())];
byte[] response = new byte[1];

while(true)
{
// Read data from file
size = fStream.Read(buffer, 0, buffer.Length);
if(size == 0)
{
break;
}
Console.WriteLine("Time " + time + ", " + size);
// Send a flag to indicate binary stream will be sent

Console.WriteLine("Send flag ...");
stream.WriteByte((byte)0);
stream.Flush();
length = this.IntToBytes(size);

Console.WriteLine("Send length ..." + size);
// Send a length message
stream.Write(length, 0, length.Length);

Console.WriteLine("Send buffer ... size = " + size + "(" +
buffer.Length + ")");
// Send data
stream.Write(buffer, 0, size);
stream.Flush();

Console.WriteLine("Read response ...");
// Waite for response
stream.Read(response, 0, 1);
Console.WriteLine("Response from server is: " + response[0]);

time++;
}
stream.WriteByte((byte)1);
stream.Flush();
stream.Close();
client.Close();
fStream.Close();
}
catch(Exception ex)
{
Console.WriteLine("time="+time);
Console.WriteLine(ex.Message);
Console.WriteLine(ex.StackTrace);
}

Server
----------------
int time = 0;
TcpClient client = listener.AcceptTcpClient();
client.SendBufferSize = 1024*1024;
client.ReceiveBufferSize = 1024*1024;
client.NoDelay = true;
NetworkStream stream = client.GetStream();

byte[] flag = new byte[1];
byte[] length = new byte[4];
byte[] body = new byte[1024 * 1024];
byte[] response = new byte[]{(byte)1};

string filename = "D:\\file.dat";

if(File.Exists(filename))
File.Delete(filename);
FileStream fStream = File.OpenWrite("D:\\file.dat");
try
{
while(true)
{
stream.Read(flag, 0, 1);
Console.WriteLine("Read flag ... " + flag[0]);
if(flag[0] == (byte)0)
{
stream.Read(length, 0, 4);
int size = BytesToInt(length);
Console.WriteLine("Read buffer size + " + size);
size = stream.Read(body, 0, size);
Console.WriteLine("Read buffer ...");

Console.Write("Time " + time + ", " + size);
fStream.Write(body, 0, size);
fStream.Flush();
Console.WriteLine(" Send response");
stream.WriteByte(response[0]);
stream.Flush();

time++;
}
else
{
Console.WriteLine("flag = " + flag[0]);
break;
}
}
}
catch(Exception e)
{
Console.WriteLine(e.Message);
Console.WriteLine(e.StackTrace);
}

Console.WriteLine("server time = " + time);
fStream.Close();
stream.Close();
client.Close();

[Sami Vaaraniemi]

It would help if you told us which exception you get.

Nevertheless, it appears to me that the server expects to read all the data
that was sent in one read. This has a good chance of failing because when
you call stream.Read(body, 0, size) it can return anything from 0 to size
bytes (check out the documentation for NetworkStream.Read). To make sure you
get all the data that was sent, you need to call stream.Read in a loop:

int size = BytesToInt(length);
int pos = 0;
while (size > 0)
{
int nread = stream.Read(body, pos, size);
if (nread == 0) break; // connection was closed
size -= nread;
pos += nread;
}

Sami
www.capehill.net

does not work because
"zhimin" <xi************@163.com> wrote in message
news:Oe****************@TK2MSFTNGP10.phx.gbl...

Hi,
I'm writing a program to send large file(100m) through dotnet using
TCPListener & TCPClient, I'm sending the file with a ask and response loop:
1. Client send a flag 1 to server indicate it has data send to server.
2. Client send the buffer block size.
3. Client send the actual buffer to the server.
4. Server send a flag 1 to client indicating that the buffer has been
successfully receeived.
5. The next loop until all data of the file has been sent.

When the data block is about 5k, the code works very fine, but when I set
the block size up to about 10k, after some cycles, an exception raised and
the program failed.
The most strange is: the cycles number are different every time(with the
same conditions), such as the first time I failed at the 13th loop, when I'm runing the same pair of applications the next time, I might failed at the
17th loop with the same file.

Any help would be appreciated.

Thanks

ps: the main code is listed below, the two cs files are attached.

Client
---------------
TcpClient client = new TcpClient(txtAddress.Text, 8000);
client.SendBufferSize = 1024*1024;
client.ReceiveBufferSize = 1024 * 1024;
client.NoDelay = true;

NetworkStream stream = client.GetStream();
byte[] length = null;
byte[] buffer = new byte[1024 *
Int32.Parse(this.numSize.Value.ToString())];
byte[] response = new byte[1];

while(true)
{
// Read data from file
size = fStream.Read(buffer, 0, buffer.Length);
if(size == 0)
{
break;
}
Console.WriteLine("Time " + time + ", " + size);
// Send a flag to indicate binary stream will be sent

Console.WriteLine("Send flag ...");
stream.WriteByte((byte)0);
stream.Flush();
length = this.IntToBytes(size);

Console.WriteLine("Send length ..." + size);
// Send a length message
stream.Write(length, 0, length.Length);

Console.WriteLine("Send buffer ... size = " + size + "(" +
buffer.Length + ")");
// Send data
stream.Write(buffer, 0, size);
stream.Flush();

Console.WriteLine("Read response ...");
// Waite for response
stream.Read(response, 0, 1);
Console.WriteLine("Response from server is: " + response[0]);

time++;
}
stream.WriteByte((byte)1);
stream.Flush();
stream.Close();
client.Close();
fStream.Close();
}
catch(Exception ex)
{
Console.WriteLine("time="+time);
Console.WriteLine(ex.Message);
Console.WriteLine(ex.StackTrace);
}

Server
----------------
int time = 0;
TcpClient client = listener.AcceptTcpClient();
client.SendBufferSize = 1024*1024;
client.ReceiveBufferSize = 1024*1024;
client.NoDelay = true;
NetworkStream stream = client.GetStream();

byte[] flag = new byte[1];
byte[] length = new byte[4];
byte[] body = new byte[1024 * 1024];
byte[] response = new byte[]{(byte)1};

string filename = "D:\\file.dat";

if(File.Exists(filename))
File.Delete(filename);
FileStream fStream = File.OpenWrite("D:\\file.dat");
try
{
while(true)
{
stream.Read(flag, 0, 1);
Console.WriteLine("Read flag ... " + flag[0]);
if(flag[0] == (byte)0)
{
stream.Read(length, 0, 4);
int size = BytesToInt(length);
Console.WriteLine("Read buffer size + " + size);
size = stream.Read(body, 0, size);
Console.WriteLine("Read buffer ...");

Console.Write("Time " + time + ", " + size);
fStream.Write(body, 0, size);
fStream.Flush();
Console.WriteLine(" Send response");
stream.WriteByte(response[0]);
stream.Flush();

time++;
}
else
{
Console.WriteLine("flag = " + flag[0]);
break;
}
}
}
catch(Exception e)
{
Console.WriteLine(e.Message);
Console.WriteLine(e.StackTrace);
}

Console.WriteLine("server time = " + time);
fStream.Close();
stream.Close();
client.Close();

[zhimin]

Thank you for your help.

I had debug my program, the error should be description as the following:

As the program described, in the request and response cycle, the client send
a byte(which value is zero) to the server to indicate this session will send
a file stream. The server recieve this byte flag, if this byte is equals to
zero, receive data, else close this session. Some of the first sessions are
fine, but after a while, the client send a byte flag which value is zero to
the server, while the server recived it and the value is not zero! The
server then close the tcpclient, and the client throw an exception. And more
large the block is, the error will occured more early.

[Sami Vaaraniemi]

"zhimin" <xi************@163.com> wrote in message
news:ON****************@TK2MSFTNGP10.phx.gbl...

Thank you for your help.

I had debug my program, the error should be description as the following:

As the program described, in the request and response cycle, the client send a byte(which value is zero) to the server to indicate this session will send a file stream. The server recieve this byte flag, if this byte is equals to zero, receive data, else close this session. Some of the first sessions are fine, but after a while, the client send a byte flag which value is zero to the server, while the server recived it and the value is not zero! The
server then close the tcpclient, and the client throw an exception. And more large the block is, the error will occured more early.

This sounds very much like you are running into TCP streaming issues.

Here's one possible scenario how it could fail:

- client sends the 0 flag
- server reads flag
- client sends the length of data (assume its 100 bytes)
- server reads length
- client sends 100 bytes
- server calls size = stream.Read(body, 0, size); where size is 100 but
receives only 50 bytes!
=> the remaining 50 bytes then arrive in the receive buffer and the
server is now out of sync
- server loops and tries to read the flag again but now its getting
actually the 1st byte of the 50 bytes that still remain in the receive
buffer. This byte happens to be nonzero so the server closes connection.

To fix this problem, you must call networkStream.Read in a loop whenever you
expect to read more than 1 byte. This is TCP stream programming 101.

This may or may not resolve all problems, but I think you should start by
fixing this issue first.

Sami
www.capehill.net

[zhimin]

Thanks for you help, all trouble fixed!