473,847 Members | 1,499 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

HTTP Object and Retrieving HTML Programatically

I am using ASP.Net 2.0 and VB.Net (although C#is ok also).

I want to create an object/method/function that will take a URL as an input
parameter and then return all of the HTML in that page.

I also want to return the HTTP header information (response object).

Does anyone have an insight as to any code samples or .Net objects I would
use to accomplish this?

Oct 25 '07 #1
2 2937
On Oct 25, 8:41 pm, Paul <P...@discussio ns.microsoft.co mwrote:
I am using ASP.Net 2.0 and VB.Net (although C#is ok also).

I want to create an object/method/function that will take a URL as an input
parameter and then return all of the HTML in that page.

I also want to return the HTTP header information (response object).

Does anyone have an insight as to any code samples or .Net objects I would
use to accomplish this?

Use WebRequest:

Dim PageUrl As String = "http://..."

Dim req As HttpWebRequest = CType(WebReques t.Create(PageUr l),
Dim enc As Encoding = Encoding.GetEnc oding(1252)
Dim r As HttpWebResponse
Dim s As System.IO.Strea mReader
r = CType(req.GetRe sponse(), HttpWebResponse ) ' This is your response
s = New System.IO.Strea mReader(r.GetRe sponseStream(), enc)

Dim html As String = s.ReadToEnd()


Response.Write( html)

Oct 25 '07 #2
This should help.

You can see how a querystring and a FORM POST works. The FormPost took some
time to figure out, if I recall correctly.

Rework the naming conventions.
I had to hardcode some query string and form post values, naturally you'll
fix those with either a string[] ... or maybe a Dictionary based generic in

public class HTTPHelper
private int m_ConnectTimeou t;
private Encoding m_enc;
public HTTPHelper(int ConnectionTimeo ut)
m_ConnectTimeou t = ConnectionTimeo ut;

//This method will write a text file streamed over HTTP in incremental
chunks defined by the buffer size
//This comes in really handy when you have to transfer REALLY big HTML
files and don't want to peg system memory

public void WriteTextFile(s tring Url, string FilePath, long BufferSize )

string queryStringAll = "?empid=123&car id=1001";
Url += queryStringAll;
//create a web request
HttpWebRequest oHttpWebRequest = null;
oHttpWebRequest = (HttpWebRequest ) System.Net.WebR equest.Create(U rl);

//set the connection timeout
oHttpWebRequest .Timeout = m_ConnectTimeou t;
this.postDataTo HttpWebRequest ( oHttpWebRequest , "postkey1" ,
"postvalue1 " );

//create a response object that we can read a stream from
HttpWebResponse oHttpResponse = (HttpWebRespons e)
oHttpWebRequest .GetResponse();
long workingbuffersi ze = 1;

//if we don't get back anything from the response, throw and exception
if (oHttpResponse == null)
throw new Exception("Url is missing or invalid.");

//Define the encoding type
//see if the page will give us back an encoding type
if (oHttpResponse. ContentEncoding .Length 0)
m_enc = Encoding.GetEnc oding(oHttpResp onse.ContentEnc oding);
m_enc = Encoding.GetEnc oding(1252);
// *** Invalid encoding passed
m_enc = Encoding.GetEnc oding(1252);
//create a stream reader grabbing text we get over HTTP
StreamReader sr = new
StreamReader(oH ttpResponse.Get ResponseStream( ),m_enc);

//set the variable that we will use as a buffer to store characters in
while the file is downloading
char[] DownloadedCharC hunk = new char[BufferSize];

//go ahead and create our streamwriter to write our file
StreamWriter sw = new StreamWriter(Fi lePath,false,m_ enc);

sw.AutoFlush = false;

//when the working buffer size hits 0 then we know that the file has
finished downloading
while (workingbuffers ize 0)
//set the working buffer size based on the length of characters we
receive from the stream
//we will also set DownloadedCharC hunk to the set of characters we
recieve from the stream
workingbuffersi ze = sr.Read(Downloa dedCharChunk,0, (int) BufferSize);

if (workingbuffers ize 0)
//write DownloadedCharC hunk to the file on disk
sw.Write(Downlo adedCharChunk,0 ,(int) workingbuffersi ze );

} // while

catch(Exception e)
throw e;


private string buildPostString ( string fpiKey , string fpiValue)

StringBuilder sb = new StringBuilder() ;
//string postValue = Encode(Request. Form(postKey));
sb.Append( string.Format(" {0}={1}&", fpiKey , fpiValue ));
return sb.ToString();

private void postDataToHttpW ebRequest ( HttpWebRequest webRequest , string
key , string value )
if (null != key )
ASCIIEncoding encoding=new ASCIIEncoding() ;

byte[] data = encoding.GetByt es(this.buildPo stString(key,va lue));

webRequest.Meth od = "POST";
webRequest.Cont entType="applic ation/x-www-form-urlencoded";
//oHttpWebRequest .ContentType = "text/xml";//Does Not Work

webRequest.Cont entLength = data.Length;
Stream newStream=webRe quest.GetReques tStream();
// Send the data.
newStream.Write (data,0,data.Le ngth);
newStream.Close ();


"Paul" <Pa**@discussio ns.microsoft.co mwrote in message
news:75******** *************** ***********@mic rosoft.com...
>I am using ASP.Net 2.0 and VB.Net (although C#is ok also).

I want to create an object/method/function that will take a URL as an
parameter and then return all of the HTML in that page.

I also want to return the HTTP header information (response object).

Does anyone have an insight as to any code samples or .Net objects I would
use to accomplish this?


Oct 26 '07 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

by: Brian Peyton | last post by:
I need to somehow merge two separate web programs written in php together (from the user's perspective). This solution won't be perminant, I just need something done quickly. I was thinking of two different approches. One would be to download a processed version (modified so it will create validating html with the page it's being embeded in) of one of the programs with the other, then just use print to embed it in the page. I...
by: Kevin Phifer | last post by:
Ok, before anyone freaks out, I have a solution I need to create that gathers content from maybe different places. Each one can return a <form> in the html, so its the classic can't have more than one runat=server form on a asp.net page. However, I still want developers to be able to use asp.net controls to create some apps that are created on the page. So I need multiple forms on a asp.net page(doesn't everyone). I purchased the...
by: RHPT | last post by:
I am wanting to capture the XML posted by an InfoPath form with .NET, but I cannot figure out how to capture the XML stream sent back by the InfoPath form. With Classic ASP, I could just create an MSXML object then load the XML with a simple objXML.Load(Request). This would give me the XML stream, which I could then manipulate. However, using System.Xml, this is not possible - not that I have found, anyway. The InfoPath documentation...
by: Rick Derthick | last post by:
Hello, I'm adding a logging capability to error handler routines for troubleshooting (using vb.net) and had just recently come across a way to programatically determine the name of the subroutine or function. For example... Private Sub SomeRoutine() Dim strRoutineName
by: Niron kag | last post by:
Hello ! With c# , I want to create a HTML file programatically . How can I do it ? Thank U !
by: shirleylouis | last post by:
Hi Ppl, Could anyone help me out with retrieving URL from Clipboard using C#??
by: vpriya6 | last post by:
Hi guys, I am new to Ajax, xml and javascript. I want to know how can I retrieve data from xml and display in the html page? please help me out. suppose my xml file is customer.xml the code is below: <CUSTOMER>
by: Aaron Gray | last post by:
Due to M$'s stupidity in not making DOMElements first class citizens the following will not work :- function isElement( o) { return o instanceof Element } It works for FF, Opera and Safari.
by: Marco Bizzarri | last post by:
On Mon, Sep 1, 2008 at 1:06 PM, jorma kala <jjkk73@gmail.comwrote: Looking at the code of HTTPConnection, all that goes through the _output message (including, therefore, the putheaders) are appended to the self._buffer list. Regards Marco --
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.