TCP Socket end of data?

[gilad]

I am trying to write a program that will send an HTTP request and get an
HTTP response. However, I am having trouble getting the full payload
(i.e. the web page) returned from the host. I am using a blocking
socket, so I need to test the end of the data somehow, or I get a "hung"
effect while the Socket.Receive() method waits.

I constructed a loop like the following to test for end-of-data, but I
only get part of the data returned for large pages and the loop exits.
Socket IPsocket = new Socket(AddressFamily.InterNetwork,
SocketType.Stream, ProtocolType.Tcp);

// Socket connection code was here...
// The first read takes place...
// ...then the following is executed:

while ( rBytes > 0)
{

if (!IPsocket.Poll(3000,SelectMode.SelectRead))
{
rBytes = IPsocket.Receive(RecvBytes, RecvBytes.Length, 0);
mStrm.Write(RecvBytes,0,rBytes);
break;
}
rBytes = IPsocket.Receive(RecvBytes, RecvBytes.Length, 0);

// Write to buffer
mStrm.Write(RecvBytes,0,rBytes);
}


Is there different way I should be testing for the end of the data? I've
tried using Socket.Available(), but found that it returns 0 when there
is in fact more data.

Thanks, JA

[David Browne]

"gilad" <gi***@arbingerBADSPAMBOTsys.com> wrote in message
news:%2****************@TK2MSFTNGP11.phx.gbl...

I am trying to write a program that will send an HTTP request and get an
HTTP response. However, I am having trouble getting the full payload (i.e.
the web page) returned from the host. I am using a blocking socket, so I
need to test the end of the data somehow, or I get a "hung" effect while
the Socket.Receive() method waits.

I constructed a loop like the following to test for end-of-data, but I
only get part of the data returned for large pages and the loop exits.
Socket IPsocket = new Socket(AddressFamily.InterNetwork,
SocketType.Stream, ProtocolType.Tcp);

Is there a particular reason you're not using the .NET HTTP classes?

If you want to implement the HTTP protocol using TCP/IP yourself, you'd
better read the spec:

http://www.w3.org/Protocols/rfc2616/rfc2616.html

Pay special attention to the section on the content-length header, the
Connection header, chunked transfer encoding and the 100 continue response
code.
Consider using HTTP 1.0
http://www.w3.org/Protocols/rfc1945/rfc1945

as it's considerably simpler to implement in raw sockets. For instance,
Socket.Recive() in a loop will work in HTTP 1.0 since the server will close
the connection when the response is done.

David

[David Browne]

"gilad" <gi***@arbingerBADSPAMBOTsys.com> wrote in message
news:%2****************@TK2MSFTNGP11.phx.gbl...

I am trying to write a program that will send an HTTP request and get an
HTTP response. However, I am having trouble getting the full payload (i.e.
the web page) returned from the host. I am using a blocking socket, so I
need to test the end of the data somehow, or I get a "hung" effect while
the Socket.Receive() method waits.

I constructed a loop like the following to test for end-of-data, but I
only get part of the data returned for large pages and the loop exits.
Socket IPsocket = new Socket(AddressFamily.InterNetwork,
SocketType.Stream, ProtocolType.Tcp);

Is there a particular reason you're not using the .NET HTTP classes?

If you want to implement the HTTP protocol using TCP/IP yourself, you'd
better read the spec:

http://www.w3.org/Protocols/rfc2616/rfc2616.html

Pay special attention to the section on the content-length header, the
Connection header, chunked transfer encoding and the 100 continue response
code.
Consider using HTTP 1.0
http://www.w3.org/Protocols/rfc1945/rfc1945

as it's considerably simpler to implement in raw sockets. For instance,
Socket.Recive() in a loop will work in HTTP 1.0 since the server will close
the connection when the response is done.

David

[gilad]

David Browne wrote:

Is there a particular reason you're not using the .NET HTTP classes?
I am building a simple proxy server.
If you want to implement the HTTP protocol using TCP/IP yourself, you'd
better read the spec:

http://www.w3.org/Protocols/rfc2616/rfc2616.html

Pay special attention to the section on the content-length header, the
Connection header, chunked transfer encoding and the 100 continue response
code.

I've done this, but the problem that I see is that the Content-Length
and other fields are not always present. For instance, here is what is
returned from "www.wired.com" as the HTTP header.

HTTP/1.1 200 OK
Date: Wed, 31 Aug 2005 18:48:35 GMT
Server: Apache/1.3.31 (Unix) PHP/4.3.9
Cache-Control: max-age=300
Expires: Wed, 31 Aug 2005 18:53:35 GMT
Connection: close
Content-Type: text/html

(From here on there is just HTML content)

I have found that I either loop with Receive until it hangs (no more
data available, and it blocks), or I've got to check the end of the
transmission. As you can see from the above, there is really nothing to
indicate packet length in the HTTP header.

I am reading the packets in buffers with 4096 bytes length.

[gilad]

David Browne wrote:

Is there a particular reason you're not using the .NET HTTP classes?
I am building a simple proxy server.
If you want to implement the HTTP protocol using TCP/IP yourself, you'd
better read the spec:

http://www.w3.org/Protocols/rfc2616/rfc2616.html

Pay special attention to the section on the content-length header, the
Connection header, chunked transfer encoding and the 100 continue response
code.

I've done this, but the problem that I see is that the Content-Length
and other fields are not always present. For instance, here is what is
returned from "www.wired.com" as the HTTP header.

HTTP/1.1 200 OK
Date: Wed, 31 Aug 2005 18:48:35 GMT
Server: Apache/1.3.31 (Unix) PHP/4.3.9
Cache-Control: max-age=300
Expires: Wed, 31 Aug 2005 18:53:35 GMT
Connection: close
Content-Type: text/html

(From here on there is just HTML content)

I have found that I either loop with Receive until it hangs (no more
data available, and it blocks), or I've got to check the end of the
transmission. As you can see from the above, there is really nothing to
indicate packet length in the HTTP header.

I am reading the packets in buffers with 4096 bytes length.

[gilad]

Actually, I may have figured this out, at least partly. If the Poll()
returns "false" (no more data), then I go ahead and try another read. If
the read returns any bytes (rBytes), then I do not break out of the
loop. If it does, then I break out. So far this seems to work. My loop
code now looks like
while ( rBytes > 0)
{

if (!IPsocket.Poll(3000,SelectMode.SelectRead))
{
rBytes = IPsocket.Receive(RecvBytes, RecvBytes.Length, 0);
mStrm.Write(RecvBytes,0,rBytes);
///////// Added the following, and it seems to have helped:
if (rBytes <= 0) break;

}
rBytes = IPsocket.Receive(RecvBytes, RecvBytes.Length, 0);

// Write to buffer
mStrm.Write(RecvBytes,0,rBytes);
}

[gilad]

Actually, I may have figured this out, at least partly. If the Poll()
returns "false" (no more data), then I go ahead and try another read. If
the read returns any bytes (rBytes), then I do not break out of the
loop. If it does, then I break out. So far this seems to work. My loop
code now looks like
while ( rBytes > 0)
{

if (!IPsocket.Poll(3000,SelectMode.SelectRead))
{
rBytes = IPsocket.Receive(RecvBytes, RecvBytes.Length, 0);
mStrm.Write(RecvBytes,0,rBytes);
///////// Added the following, and it seems to have helped:
if (rBytes <= 0) break;

}
rBytes = IPsocket.Receive(RecvBytes, RecvBytes.Length, 0);

// Write to buffer
mStrm.Write(RecvBytes,0,rBytes);
}

[David Browne]

"gilad" <gi***@arbingerBADSPAMBOTsys.com> wrote in message
news:%2****************@TK2MSFTNGP10.phx.gbl...

Actually, I may have figured this out, at least partly. If the Poll()
returns "false" (no more data), then I go ahead and try another read. If
the read returns any bytes (rBytes), then I do not break out of the loop.
If it does, then I break out. So far this seems to work. My loop code now
looks like
while ( rBytes > 0)
{

if (!IPsocket.Poll(3000,SelectMode.SelectRead))
{
rBytes = IPsocket.Receive(RecvBytes, RecvBytes.Length, 0);
mStrm.Write(RecvBytes,0,rBytes);
///////// Added the following, and it seems to have helped:
if (rBytes <= 0) break;

Yes. A zero byte Recieve always means that the server has closed the
connection.

http://msdn.microsoft.com/library/de...eivetopic1.asp
If the remote host shuts down the Socket connection with the Shutdown
method, and all available data has been received, the Receive method will
complete immediately and return zero bytes.

David

[David Browne]

"gilad" <gi***@arbingerBADSPAMBOTsys.com> wrote in message
news:%2****************@TK2MSFTNGP10.phx.gbl...

Actually, I may have figured this out, at least partly. If the Poll()
returns "false" (no more data), then I go ahead and try another read. If
the read returns any bytes (rBytes), then I do not break out of the loop.
If it does, then I break out. So far this seems to work. My loop code now
looks like
while ( rBytes > 0)
{

if (!IPsocket.Poll(3000,SelectMode.SelectRead))
{
rBytes = IPsocket.Receive(RecvBytes, RecvBytes.Length, 0);
mStrm.Write(RecvBytes,0,rBytes);
///////// Added the following, and it seems to have helped:
if (rBytes <= 0) break;

Yes. A zero byte Recieve always means that the server has closed the
connection.

http://msdn.microsoft.com/library/de...eivetopic1.asp
If the remote host shuts down the Socket connection with the Shutdown
method, and all available data has been received, the Receive method will
complete immediately and return zero bytes.

David

[gilad]

David Browne wrote:

Yes. A zero byte Recieve always means that the server has closed the
connection.

I guess what I didn't understand is why, if that's true, the
while ( rBytes > 0 )
test was not sufficient to break out of the loop. Before, when I
observed the loop to hang, it was constructed as follows (without the
Poll() test):
while ( rBytes > 0 )
{

rBytes = IPsocket.Receive(RecvBytes, RecvBytes.Length, 0);
// Write to buffer
mStrm.Write(RecvBytes,0,rBytes);

}

The Poll() test worked, but showed a inconsistencies, so I did something
else and it seems to be working as expected:
while ( rBytes > 0 )
{

rBytes = IPsocket.Receive(RecvBytes, RecvBytes.Length, 0);
// Write to buffer
if (rBytes > 0) mStrm.Write(RecvBytes,0,rBytes);

}
Before, I assumed that even if rBytes was 0, the Write() method would
simply write 0 bytes to the MemoryStream "mStrm". Could this have been
the cause of the hang? At any rate, it seems to have fixed the problem,
and I do not need to use the Poll() code.

Any thoughts?

Thanks, JA