By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
429,426 Members | 1,729 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 429,426 IT Pros & Developers. It's quick & easy.

Web Scraping on Secured Sites that require UserName and Password.

P: n/a
I have a Web Site that I want to monitor and gather
information from which requires a secured user id and
password of which I have but don't know how to gather get
by the secured log in to get to the pages I need. I have
gathered information from sites that don't require secured
login by using the following code:

Function readHtmlPage(ByVal url As String) As String
Dim objResponse As WebResponse
Dim objRequest As WebRequest
Dim result As String
objRequest = System.Net.HttpWebRequest.Create(url)
objResponse = objRequest.GetResponse()
Dim sr As New StreamReader
(objResponse.GetResponseStream())
result = sr.ReadToEnd()
sr.Close()
Return result
End Function

But don't know how to get around the sites that require
secured log-in such as https protocol.

Any help or direction would be appreciated.
Nov 20 '05 #1
Share this Question
Share on Google+
3 Replies


P: n/a
Have you tried the Internet Transfer control? I use that ocassionally to get
FTP files. It might be a VB6/Legacy thing. It does work in .Net, though.
Fortunately, if you own VB.Net, I think you're licensed to use VB6 as well.
You'd just need to find a copy to install.
"John West" <jo***********@hotmail.com> wrote in message
news:09****************************@phx.gbl...
I have a Web Site that I want to monitor and gather
information from which requires a secured user id and
password of which I have but don't know how to gather get
by the secured log in to get to the pages I need. I have
gathered information from sites that don't require secured
login by using the following code:

Function readHtmlPage(ByVal url As String) As String
Dim objResponse As WebResponse
Dim objRequest As WebRequest
Dim result As String
objRequest = System.Net.HttpWebRequest.Create(url)
objResponse = objRequest.GetResponse()
Dim sr As New StreamReader
(objResponse.GetResponseStream())
result = sr.ReadToEnd()
sr.Close()
Return result
End Function

But don't know how to get around the sites that require
secured log-in such as https protocol.

Any help or direction would be appreciated.

Nov 20 '05 #2

P: n/a
* "Microsoft" <tw*****@nospam.valleycomputer.net> scripsit:
Have you tried the Internet Transfer control? I use that ocassionally to get
FTP files. It might be a VB6/Legacy thing. It does work in .Net, though.
Fortunately, if you own VB.Net, I think you're licensed to use VB6 as well.
You'd just need to find a copy to install.


For FTP:

<http://www.freevbcode.com/ShowCode.Asp?ID=4655>
<http://www.allapi.net/downloads/NetFTP.zip>
<http://www.visualbuilder.com/article/viewarticle.asp?id=1234>
<http://www.abderaware.com/> (FTP component)
<http://www.csharphelp.com/archives/archive9.html>
<http://www.codeproject.com/csharp/FTPDriver1.asp>
....

--
Herfried K. Wagner [MVP]
<http://www.mvps.org/dotnet>
Nov 20 '05 #3

P: n/a
Cor
Hi Herfried,

Can we do the Pentagon also with your help.
Nobody was answering this thread, but there is always one
.........................
:-((

Cor
Nov 20 '05 #4

This discussion thread is closed

Replies have been disabled for this discussion.