HTTP Object and Retrieving HTML Programatically

[=?Utf-8?B?UGF1bA==?=]

I am using ASP.Net 2.0 and VB.Net (although C#is ok also).

I want to create an object/method/function that will take a URL as an input
parameter and then return all of the HTML in that page.

I also want to return the HTTP header information (response object).

Does anyone have an insight as to any code samples or .Net objects I would
use to accomplish this?

TIA

[Alexey Smirnov]

On Oct 25, 8:41 pm, Paul <P...@discussio ns.microsoft.co mwrote:

I am using ASP.Net 2.0 and VB.Net (although C#is ok also).

I want to create an object/method/function that will take a URL as an input
parameter and then return all of the HTML in that page.

I also want to return the HTTP header information (response object).

Does anyone have an insight as to any code samples or .Net objects I would
use to accomplish this?

TIA

Use WebRequest:

Dim PageUrl As String = "http://..."

Dim req As HttpWebRequest = CType(WebReques t.Create(PageUr l),
HttpWebRequest)
Dim enc As Encoding = Encoding.GetEnc oding(1252)
Dim r As HttpWebResponse
Dim s As System.IO.Strea mReader
r = CType(req.GetRe sponse(), HttpWebResponse ) ' This is your response
s = New System.IO.Strea mReader(r.GetRe sponseStream(), enc)

Dim html As String = s.ReadToEnd()

r.Close()
s.Close()

Response.Write( html)

[sloan]

This should help.

You can see how a querystring and a FORM POST works. The FormPost took some
time to figure out, if I recall correctly.

Rework the naming conventions.
I had to hardcode some query string and form post values, naturally you'll
fix those with either a string[] ... or maybe a Dictionary based generic in
2.0.

public class HTTPHelper
{
private int m_ConnectTimeou t;
private Encoding m_enc;
public HTTPHelper(int ConnectionTimeo ut)
{
m_ConnectTimeou t = ConnectionTimeo ut;
}

//This method will write a text file streamed over HTTP in incremental
chunks defined by the buffer size
//This comes in really handy when you have to transfer REALLY big HTML
files and don't want to peg system memory

public void WriteTextFile(s tring Url, string FilePath, long BufferSize )
{

try
{
string queryStringAll = "?empid=123&car id=1001";
Url += queryStringAll;
//create a web request
HttpWebRequest oHttpWebRequest = null;
oHttpWebRequest = (HttpWebRequest ) System.Net.WebR equest.Create(U rl);

//set the connection timeout
oHttpWebRequest .Timeout = m_ConnectTimeou t;
this.postDataTo HttpWebRequest ( oHttpWebRequest , "postkey1" ,
"postvalue1 " );

//create a response object that we can read a stream from
HttpWebResponse oHttpResponse = (HttpWebRespons e)
oHttpWebRequest .GetResponse();
long workingbuffersi ze = 1;

//if we don't get back anything from the response, throw and exception
if (oHttpResponse == null)
{
throw new Exception("Url is missing or invalid.");
}

//Define the encoding type
try
{
//see if the page will give us back an encoding type
if (oHttpResponse. ContentEncoding .Length 0)
m_enc = Encoding.GetEnc oding(oHttpResp onse.ContentEnc oding);
else
m_enc = Encoding.GetEnc oding(1252);
}
catch
{
// *** Invalid encoding passed
m_enc = Encoding.GetEnc oding(1252);
}
//create a stream reader grabbing text we get over HTTP
StreamReader sr = new
StreamReader(oH ttpResponse.Get ResponseStream( ),m_enc);

//set the variable that we will use as a buffer to store characters in
while the file is downloading
char[] DownloadedCharC hunk = new char[BufferSize];

//go ahead and create our streamwriter to write our file
StreamWriter sw = new StreamWriter(Fi lePath,false,m_ enc);

sw.AutoFlush = false;

//when the working buffer size hits 0 then we know that the file has
finished downloading
while (workingbuffers ize 0)
{
//set the working buffer size based on the length of characters we
receive from the stream
//we will also set DownloadedCharC hunk to the set of characters we
recieve from the stream
workingbuffersi ze = sr.Read(Downloa dedCharChunk,0, (int) BufferSize);

if (workingbuffers ize 0)
{
//write DownloadedCharC hunk to the file on disk
sw.Write(Downlo adedCharChunk,0 ,(int) workingbuffersi ze );
}

} // while
sr.Close();
sw.Close();

}
catch(Exception e)
{
throw e;
}

}


private string buildPostString ( string fpiKey , string fpiValue)
{

StringBuilder sb = new StringBuilder() ;
//string postValue = Encode(Request. Form(postKey));
sb.Append( string.Format(" {0}={1}&", fpiKey , fpiValue ));
return sb.ToString();
}

private void postDataToHttpW ebRequest ( HttpWebRequest webRequest , string
key , string value )
{
if (null != key )
{
ASCIIEncoding encoding=new ASCIIEncoding() ;

byte[] data = encoding.GetByt es(this.buildPo stString(key,va lue));

webRequest.Meth od = "POST";
webRequest.Cont entType="applic ation/x-www-form-urlencoded";
//oHttpWebRequest .ContentType = "text/xml";//Does Not Work

webRequest.Cont entLength = data.Length;
Stream newStream=webRe quest.GetReques tStream();
// Send the data.
newStream.Write (data,0,data.Le ngth);
newStream.Close ();
}

}
}


"Paul" <Pa**@discussio ns.microsoft.co mwrote in message
news:75******** *************** ***********@mic rosoft.com...

>I am using ASP.Net 2.0 and VB.Net (although C#is ok also).

I want to create an object/method/function that will take a URL as an
input
parameter and then return all of the HTML in that page.

I also want to return the HTTP header information (response object).

Does anyone have an insight as to any code samples or .Net objects I would
use to accomplish this?

TIA