Web Scraping on Secured Sites that require UserName and Password.

[John West]

I have a Web Site that I want to monitor and gather
information from which requires a secured user id and
password of which I have but don't know how to gather get
by the secured log in to get to the pages I need. I have
gathered information from sites that don't require secured
login by using the following code:

Function readHtmlPage(ByVal url As String) As String
Dim objResponse As WebResponse
Dim objRequest As WebRequest
Dim result As String
objRequest = System.Net.HttpWebRequest.Create(url)
objResponse = objRequest.GetResponse()
Dim sr As New StreamReader
(objResponse.GetResponseStream())
result = sr.ReadToEnd()
sr.Close()
Return result
End Function

But don't know how to get around the sites that require
secured log-in such as https protocol.

Any help or direction would be appreciated.

[Microsoft]

Have you tried the Internet Transfer control? I use that ocassionally to get
FTP files. It might be a VB6/Legacy thing. It does work in .Net, though.
Fortunately, if you own VB.Net, I think you're licensed to use VB6 as well.
You'd just need to find a copy to install.
"John West" <jo***********@hotmail.com> wrote in message
news:09****************************@phx.gbl...

I have a Web Site that I want to monitor and gather
information from which requires a secured user id and
password of which I have but don't know how to gather get
by the secured log in to get to the pages I need. I have
gathered information from sites that don't require secured
login by using the following code:

Function readHtmlPage(ByVal url As String) As String
Dim objResponse As WebResponse
Dim objRequest As WebRequest
Dim result As String
objRequest = System.Net.HttpWebRequest.Create(url)
objResponse = objRequest.GetResponse()
Dim sr As New StreamReader
(objResponse.GetResponseStream())
result = sr.ReadToEnd()
sr.Close()
Return result
End Function

But don't know how to get around the sites that require
secured log-in such as https protocol.

Any help or direction would be appreciated.

[Herfried K. Wagner [MVP]]

* "Microsoft" <tw*****@nospam.valleycomputer.net> scripsit:

Have you tried the Internet Transfer control? I use that ocassionally to get
FTP files. It might be a VB6/Legacy thing. It does work in .Net, though.
Fortunately, if you own VB.Net, I think you're licensed to use VB6 as well.
You'd just need to find a copy to install.

For FTP:

<http://www.freevbcode.com/ShowCode.Asp?ID=4655>
<http://www.allapi.net/downloads/NetFTP.zip>
<http://www.visualbuilder.com/article/viewarticle.asp?id=1234>
<http://www.abderaware.com/> (FTP component)
<http://www.csharphelp.com/archives/archive9.html>
<http://www.codeproject.com/csharp/FTPDriver1.asp>
....

--
Herfried K. Wagner [MVP]
<http://www.mvps.org/dotnet>

[Cor]

Hi Herfried,

Can we do the Pentagon also with your help.
Nobody was answering this thread, but there is always one
.........................
:-((

Cor