474,044 Members | 3,337 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Read source code of a web page

Hi All,

I use VC .Net 2003 and I want to create a small program which is able to
read the source code of a web site. I have a textbox with the URL, a button
and a multiline textbox for the source code. If the web page is secure, how
can I post my login details?

Any help appreciated.

Regards,
Nicolae
Jul 21 '05 #1
5 2011
"Nicolae Fieraru" <no****@please. com> wrote:
Hi All,

I use VC .Net 2003 and I want to create a small program which is able to
read the source code of a web site. I have a textbox with the URL, a button
and a multiline textbox for the source code. If the web page is secure, how
can I post my login details?

Any help appreciated.

Regards,
Nicolae


Depends on how the server is secured. In some cases

http://username:pa******@www.domain.com/path/filename

may work but in the majority of cases you'll have to use the
WebClient class and set up its Credentials property.

WebClient Class
http://msdn.microsoft.com/library/de...classtopic.asp

WebClient.Crede ntials
http://msdn.microsoft.com/library/de...tialstopic.asp

NetworkCredenti al Class
http://msdn.microsoft.com/library/de...classtopic.asp

NetworkCredenti al Constructor
http://msdn.microsoft.com/library/de...sctortopic.asp
'The real crux of the "software crisis" is that software IS hard.'
Robert C. Martin,
'Designing Object-Oriented C++ Applications Using the Booch Method', p. ix
Jul 21 '05 #2
The advice from UAError is only appropriate if the web site is on YOUR
computer. If you are trying to read the html contents of a page that is
running on someone else's computer, you will need to be able to log in to
that site. I assume you are talking about a situation where you can log in,
but you want to download the HTML to an application.

Most sites do what .Net does, and that is to make a login page available
that uses cookies. In this model, you log in to the server. The server
checks its database or directory for your credentials. If found, the server
issues a token. This is a string of characters to you, but means something
to the server. It provides that token to your site as a cookie.

All subsequent pages on the site require the cookie. Web browsers already
do this. If a cookie comes from a web site, with instructions for how long
the cookie should remain in the browser, the browser will automatically pass
the cookie back to every subsequent request from that web site.

So, your app has to pretend to be a browser. You have to use the GET and
POST methods of the HTTPWebRequest class to ask for the login page and to
provide the credentials. You will get back a collection of cookies. You
need to attach that cookie collection to all subsequent Web Requests against
that site.

So, look to the HTTPWebRequest HTTPWebResponse and CookieCollectio n classes
for further information on how to do what you want.

Note: you said you wanted to read the "source code" of the site. You will
be able to get the HTML, not the source code per se. If you want the C# or
ASP.NET code for a site, you will need direct access to the filesystem it is
on. That will normally require the cooperation of the system administrator.
In that case, my advice is meaningless. (I'm guessing here).

I hope this helps,

--
--- Nick Malik [Microsoft]
MCSD, CFPS, Certified Scrummaster
http://blogs.msdn.com/nickmalik

Disclaimer: Opinions expressed in this forum are my own, and not
representative of my employer.
I do not answer questions on behalf of my employer. I'm just a
programmer helping programmers.
--
"Nicolae Fieraru" <no****@please. com> wrote in message
news:O8******** ******@TK2MSFTN GP09.phx.gbl...
Hi All,

I use VC .Net 2003 and I want to create a small program which is able to
read the source code of a web site. I have a textbox with the URL, a button and a multiline textbox for the source code. If the web page is secure, how can I post my login details?

Any help appreciated.

Regards,
Nicolae

Jul 21 '05 #3
"Nick Malik [Microsoft]" <ni*******@hotm ail.nospam.com>
wrote:
The advice from UAError is only appropriate if the web site is on YOUR
computer.
Surely you mean a computer within your domain, not your own
local computer? When I'm wrong, I'm wrong but the latter
would seem so restrictive that it is useless. The sample

How To Use WebClient Class To Make HTTP Requests
http://support.microsoft.com/default...b;en-us;328820

states that WebClient supports
- Basic authentication
- Integrated Windows authentication

<snip>All subsequent pages on the site require the cookie. Web browsers already
do this. If a cookie comes from a web site, with instructions for how long
the cookie should remain in the browser, the browser will automatically pass
the cookie back to every subsequent request from that web site.
I was under the (now obviously mistaken) impression that
WebClient was like a stripped down browser. I guess I should
have looked for a Cookies or CookieContainer property and
drawn my own conclusions when it wasn't there.
You have to use the GET and POST methods of the HTTPWebRequest
class to ask for the login page and to provide the credentials.


You had me confused here until I found the
HttpWebRequest. Method property, which was impossible to find
under "Public Methods" :)
Thanks for straightening this out.
Jul 21 '05 #4
sorry for not being clear and I certainly didn't mean to offend.

Thanks for taking my remarks in a good spirit.

--
--- Nick Malik [Microsoft]
MCSD, CFPS, Certified Scrummaster
http://blogs.msdn.com/nickmalik

Disclaimer: Opinions expressed in this forum are my own, and not
representative of my employer.
I do not answer questions on behalf of my employer. I'm just a
programmer helping programmers.
--
Jul 21 '05 #5
The advice from UAError is only appropriate if the web site is on YOUR
computer. If you are trying to read the html contents of a page that is
running on someone else's computer, you will need to be able to log in to
that site. I assume you are talking about a situation where you can log
in,
but you want to download the HTML to an application.


That is not true Nick, if you use Internet Explorer as browser than you see
an icon "Edit" which will allow you to change the page content you are on
even with notepad. (The only point is you can only save it on your own
computers)

Cor
Jul 21 '05 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
8004
by: Noozer | last post by:
I am able to place values on forms that load into my webbrowser control, but I'd like to be able to scan the page for data. I know to use the HTMLDocument object. Basically I'm hoping to find an example of a DocumentComplete routine that will: - display a list of all elements (tags?) on the page - number of forms on the page - ID, name, type and value for each field on the form
6
1306
by: Dung Ping | last post by:
A while ago I posted a message saying that saving Javascript code as external file might hide the code better. All replies to it said it would still be very easy for people to look at the source code. I sometimes look at the source code if a page is impressive or interesting, but have never opened any external file. As experiment, I saved a Javascript code of a web page as the external file of it. After several days, I looked at the...
4
1589
by: Kevin Vogler | last post by:
I need to read the value of a hidden field on a remote HTML page and then use that value in a form submit on my local page. How can I read the hidden field programatically? Thanks in advance. Kevin Vogler
5
313
by: Nicolae Fieraru | last post by:
Hi All, I use VC .Net 2003 and I want to create a small program which is able to read the source code of a web site. I have a textbox with the URL, a button and a multiline textbox for the source code. If the web page is secure, how can I post my login details? Any help appreciated. Regards,
17
3483
by: chris.schwalm | last post by:
I would like to first state that I have searched through the archives and found a lot of related material, but am still new enough to javascript that I can't fit all the pieces together yet. So here is my situation. I would like to create a program that grabs the source of a displayed webpage, sends it off to another program, then receive the slightly modified source back from that other program, and then reposts the slightly modified...
6
9714
by: Karthik | last post by:
I am trying to read Http request body values NOT form values The content type is text/xml and its POST I am posting data from Infopath to an asp.net page. In ASP it works by just writing Response.Write(request) the same does not work in ASP.NET I tried using Rquest.BinaryrRead() and Request.InputStream but it gives me text values as space separated (eg Karthik yyyy aaaa) whereas the data i am looking out for is in XML format.
2
4380
by: keithb | last post by:
This is on a remote server. Everything works in the development envirmonment. Any ideas? Thanks, Keith Failed to update database "C:\BTM_ROOT\APP_DATA\ASPNETDB.MDF" because the database is read-only. Description: An unhandled exception occurred during the execution of the
4
4208
by: Kuldeep | last post by:
Hi All, I am trying to read the contents of a page through its URL. My code snippet is as follows: public void mtdGetPageDataHWR() { HttpWebRequest objRequ = (HttpWebRequest)WebRequest.Create("http://www.microsoft.com"); HttpWebResponse objResp = (HttpWebResponse)objRequ.GetResponse();
9
4289
by: =?Utf-8?B?TWlrZQ==?= | last post by:
Hi. Is it programatically possible in VB.NET to read the contents of web.config's <customErrorselement? I looked at using ConfigurationSettings.AppSettings, but that doesn't work. I need to read the value of redirect from the error statusCode 404. My web.config looks like this: <customErrors mode="RemoteOnly" defaultRedirect="Error.aspx"> <error statusCode="404" redirect="Error404.aspx" /> </customErrors>
0
10546
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
10337
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
12140
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
11602
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
11141
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
10310
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
7869
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
6837
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
5416
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.