473,402 Members | 2,055 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,402 software developers and data experts.

Best way for reading HTTP data

I've got a problem which relates to reading HTTP data.
I've got the socket connected to a web site and then I send "GET /
HTTP/1.1\n\n" and the page is received after a while but not all of the
page. Should I implement a timer to read the web page? How do I know
when the page is completed if sometimes socket.Available is 0?

The procedure is as follows:
-Socket socket=new
Socket(AddressFamily.InterNetwork,SocketType.Strea m,ProtocolType.Tcp);
-socket.Connect(endpoint);
-byte[] msg=Encoding.UTF8.GetBytes("GET / HTTP/1.1\n\n");
byte[] bytes=new byte[65536];
int i=socket.Send(msg,0,msg.Length,SocketFlags.None);
MessageBox.Show("Sent "+i.ToString()+" bytes. Available:
"+socket.Available.ToString()+" bytes.");
socket.Receive(bytes,0,socket.Available,SocketFlag s.None);
TrafficLogTextBox.Text+=Encoding.UTF8.GetString(by tes);
TrafficLogTextBox.Text+="\r\n";
MessageBox.Show(Encoding.UTF8.GetString(bytes));

How does HTTPWebResponse implements this? Does it use a timer between
non receiving data times? *How do I know when the page is complete?*
Did I made myself clear?

Thanks a lot,
Nuno Magalhaes.

Nov 25 '05 #1
7 2756
Web page can have large size, that is why it is normal situation that it
will be received with the help of several calls to Receive(...).
To handle this you have to parse HTTP protocol specific data. Size of the
response that server will generate is written into content-size http header.

So the algorithm is the following:
- receive the first part of the response, that contains http header, that
will describe the data that will follow;
- receive the amount of data, specified in the retrieved http header

--
Vadym Stetsyak aka Vadmyst

http://vadmyst.blogspot.com
"Nuno Magalhaes" <nu************@hotmail.com> wrote in message
news:11**********************@g44g2000cwa.googlegr oups.com...
I've got a problem which relates to reading HTTP data.
I've got the socket connected to a web site and then I send "GET /
HTTP/1.1\n\n" and the page is received after a while but not all of the
page. Should I implement a timer to read the web page? How do I know
when the page is completed if sometimes socket.Available is 0?

The procedure is as follows:
-Socket socket=new
Socket(AddressFamily.InterNetwork,SocketType.Strea m,ProtocolType.Tcp);
-socket.Connect(endpoint);
-byte[] msg=Encoding.UTF8.GetBytes("GET / HTTP/1.1\n\n");
byte[] bytes=new byte[65536];
int i=socket.Send(msg,0,msg.Length,SocketFlags.None);
MessageBox.Show("Sent "+i.ToString()+" bytes. Available:
"+socket.Available.ToString()+" bytes.");
socket.Receive(bytes,0,socket.Available,SocketFlag s.None);
TrafficLogTextBox.Text+=Encoding.UTF8.GetString(by tes);
TrafficLogTextBox.Text+="\r\n";
MessageBox.Show(Encoding.UTF8.GetString(bytes));

How does HTTPWebResponse implements this? Does it use a timer between
non receiving data times? *How do I know when the page is complete?*
Did I made myself clear?

Thanks a lot,
Nuno Magalhaes.

Nov 25 '05 #2
In most cases I don't have the "Content-Length" field in the HTTP
response header.
Any hints for what I could be doing wrong or what I should be doing.

Thank you Vadym.

Vadym Stetsyak wrote:
Web page can have large size, that is why it is normal situation that it
will be received with the help of several calls to Receive(...).
To handle this you have to parse HTTP protocol specific data. Size of the
response that server will generate is written into content-size http header.

So the algorithm is the following:
- receive the first part of the response, that contains http header, that
will describe the data that will follow;
- receive the amount of data, specified in the retrieved http header

--
Vadym Stetsyak aka Vadmyst

http://vadmyst.blogspot.com
"Nuno Magalhaes" <nu************@hotmail.com> wrote in message
news:11**********************@g44g2000cwa.googlegr oups.com...
I've got a problem which relates to reading HTTP data.
I've got the socket connected to a web site and then I send "GET /
HTTP/1.1\n\n" and the page is received after a while but not all of the
page. Should I implement a timer to read the web page? How do I know
when the page is completed if sometimes socket.Available is 0?

The procedure is as follows:
-Socket socket=new
Socket(AddressFamily.InterNetwork,SocketType.Strea m,ProtocolType.Tcp);
-socket.Connect(endpoint);
-byte[] msg=Encoding.UTF8.GetBytes("GET / HTTP/1.1\n\n");
byte[] bytes=new byte[65536];
int i=socket.Send(msg,0,msg.Length,SocketFlags.None);
MessageBox.Show("Sent "+i.ToString()+" bytes. Available:
"+socket.Available.ToString()+" bytes.");
socket.Receive(bytes,0,socket.Available,SocketFlag s.None);
TrafficLogTextBox.Text+=Encoding.UTF8.GetString(by tes);
TrafficLogTextBox.Text+="\r\n";
MessageBox.Show(Encoding.UTF8.GetString(bytes));

How does HTTPWebResponse implements this? Does it use a timer between
non receiving data times? *How do I know when the page is complete?*
Did I made myself clear?

Thanks a lot,
Nuno Magalhaes.


Nov 27 '05 #3
I think you are taking the incorrect approach here. You should use the
HTTPWebRequest and HTTPWebResponse classes. They are much, much easier
than raw sockets. Here is some sample code you can use to start with:

Sub Main()
Dim objRequest As HttpWebRequest
Dim strRequest As String
Dim objResponse As HttpWebResponse
Dim srResponse As StreamReader
Dim strUrl As String = "http://www.msn.com"
'initialize the request
objRequest = CType(WebRequest.Create(strUrl), HttpWebRequest)
objRequest.Method = "GET"

'get response
objResponse = CType(objRequest.GetResponse, HttpWebResponse)
srResponse = New StreamReader(objResponse.GetResponseStream)
Console.WriteLine(srResponse.ReadToEnd)
srResponse.Close()
Console.ReadLine()
End Sub

Its in VB.NET but you should be able to convert this quite easily. To
answer your question how do I know when the reading is done; as you can
see calling the ReadToEnd() method on the streamreader object handles
this for you.

I hope this helps

Nov 27 '05 #4
Maybe I'm not passing all the parameters to the server also. Do you
know if sending "GET / HTTP/1.1\n\n" is enough to receive the content
length field?

Nov 27 '05 #5
I can't use that higher level functions because I'm measuring
parameters of QoS such as: time to resolve dns, time to connect, time
to receive data, time to display all web page, etc...

Do you know if the "GET / HTTP/1.1" is enough to receive the
"Content-Length: " parameter in the HTTP response header?

Thank you.

Nov 27 '05 #6
Ok. I feld kinda bad posting VB code in a C# newsgroup as I forgot what
group I was in, so I will convert this on the fly for you. Sorry about
that Guys! My bad!

static void main(string[] args)
{
HttpWebRequest objRequest;
string strRequest;
HttpWebResponse objResponse;
StreamReader srResponse;
string strUrl = "http://www.msn.com";
objRequest = ((HttpWebRequest) (WebRequest.Create(strUrl)));
objRequest.Method = "GET";
objResponse = ((HttpWebResponse) (objRequest.GetResponse));
srResponse = new StreamReader(objResponse.GetResponseStream);
Console.WriteLine(srResponse.ReadToEnd);
srResponse.Close();
Console.ReadLine();
}

Nov 27 '05 #7
The GET should do it; but I would use the HEAD as it only retreive the
headers

HEAD / HTTP/1.1 \r\n
Host: localhost (or whatever)\r\n

Note the \r\n instead of \n\n

\r\n = carriage return/line feed

Nov 27 '05 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Mike Verdone | last post by:
Hello all, I'm trying to implement a PHP program that can handle streaming HTTP data through Apache. I need to somehow get access to the incoming data of the HTTP request as it arrives (i.e. I...
4
by: Andy | last post by:
Hello All: I have a field in the database that is an Image. I have no idea how the data is stored in here (Image, compressed, encrypted, plain text, etc). I am trying to write the contents to...
3
by: Adman | last post by:
Hi all. I've done some websearching, and haven't been able to find my question answered, so I thought I'd post. I apologize if this has already been answered. My question seems to be a...
2
by: Ed | last post by:
Hope someone can help me out... I have been tasked to read some image data from an sql database and save the files to flat files. OK, sounds easy as I'v used BLOBs before. But this is an old...
4
by: Epictitus | last post by:
I am trying to write a script which will allow me to read the contents of an HTTP web page into memory, parse the data, and output a calculated value. The HTTP page is on a web enable appliance -...
3
by: psbasha | last post by:
Hi , When ever we read any data from file ,we read as a single line string ,and we convert the respective field data available in that string based on the data type ( say int,float ). ...
3
by: MF AHMED | last post by:
Hello, I am facing problem writing an algorithm/codes. From a C program, I generated 100 data files, now I have to read those files having three columns each. Using every data of each line of all...
0
by: AboShorouq | last post by:
Hi, i'am new to this group, so i don't know how the things going on here. i have a serious problem and any help from you guys is very appreciated. i am trying to read data from db2 v8 on AIX...
2
by: Steven Cheng | last post by:
Hi Gary, As for the setting you mentioned, it is used to custimze the received data size limitation of WCF binding(such as NetTcpBinding or HttpBinding). After you have definite a customized...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
0
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.