httpWebRequest through Proxy

[retroviz]

Hi there.

I have written a screen scraping application (both web based and windows forms) in vb.net. When testing on a public broadband link it works fine. However it fails at work due to our proxy server.

To authenticate in the asp version, I just added:

<defaultProxy useDefaultCredentials="true">

to my web.config - this works fine.

However, I need to integrate this with an existing windows form app (vb6) so need to get the windows form version working.

I am struggling to achieve the same settings (as per web.config) directly within my code. This is what I am trying to use so far:

Dim mywebRequest As HttpWebRequest = TryCast(WebRequest.Create("http://www.thedomain.com/login.aspx"), HttpWebRequest)
 
mywebRequest.Proxy = New WebProxy("http://myproxyserver")
mywebRequest.Proxy.Credentials = CredentialCache.DefaultCredentials
mywebRequest.Credentials = CredentialCache.DefaultCredentials

As the page I am trying to access requires authentication, I first capture the viewstate of the page. Once I have this I can then simulate a post of my login credentials and use a streamreader to retrieve the contents of the page.

Click here for a C# tutorial.

The thing I dont understand is that the first part of the code appears to work (and passes through the proxy). It is only when i try and close the contents of the response:

mywebRequest.GetResponse().Close()

that I get the following error:

The remote server returned an error: (407) Proxy Authentication Required.

Could someone advise me as to whether I am defining the proxy correctly?

Thanks.

Ben

[retroviz]

Just to let everyone know I got this working in the end. Made a schoolboy error as I had reinitiated the web request but did not add the proxy information for that instance.

If anyone would like a copy of the complete code (to authenticate then screen scrape) then sent me a message

[Frinavale]

Just to let everyone know I got this working in the end. Made a schoolboy error as I had reinitiated the web request but did not add the proxy information for that instance.

If anyone would like a copy of the complete code (to authenticate then screen scrape) then sent me a message

I'm glad you got it working!
Could you please share the solution so others can learn as well.

Thanks a lot!

-Frinny

[retroviz]

Well I am glad to help. For some tutorials on httpwebrequest and persisting viewstate for sites that require authentication I would seriously recommend:

http://odetocode.com/Articles/162.aspx (c# version of my code)

and

http://aspnet.4guysfromrolla.com/art...spx#postadlink (great article covering the basics of the httpwebrequest class by Scott Mitchell.

I do have a C# version of the following code if you require (as originally created in this then needed to write in vb.net). This also works as a windows form application or a asp.net app.

Create a new class called prices.vb. Insert the following code (comments explain it pretty well):

Imports System
Imports System.Net
Imports System.IO
Imports System.Text
Imports System.Web
Imports System.Text.RegularExpressions
Imports Microsoft.VisualBasic
Namespace PageGetter
    Public Class prices
        Public targetURL As String
        Private Function ExtractViewState(ByVal s As String) As String
            Dim viewStateNameDelimiter As String = "__VIEWSTATE"
            Dim valueDelimiter As String = "value="""
 
 
            Dim viewStateNamePosition As Integer = s.IndexOf(viewStateNameDelimiter)
            Dim viewStateValuePosition As Integer = s.IndexOf(valueDelimiter, viewStateNamePosition)
 
            Dim viewStateStartPosition As Integer = viewStateValuePosition + valueDelimiter.Length
            Dim viewStateEndPosition As Integer = s.IndexOf("""", viewStateStartPosition)
 
            Return HttpUtility.UrlEncodeUnicode(s.Substring(viewStateStartPosition, viewStateEndPosition - viewStateStartPosition))
        End Function
 
        Public Function GetPrices(ByVal targetUrl As String) As String
*********************************************************************************
'only need this if you are behind a proxy            
Dim pxy As New WebProxy("http://yourproxyaddress:0000")
            pxy.Credentials = CredentialCache.DefaultCredentials
*********************************************************************************
            ' first, request the login form to get the viewstate value
            Dim mywebRequest As HttpWebRequest = TryCast(WebRequest.Create("http://somedomain.com/TheLogOnCheck.aspx"), HttpWebRequest)
 
            mywebRequest.Proxy = pxy
 
            'Set the timeout to 1 second (or 1,000 milliseconds)
            mywebRequest.Timeout = 1000
 
 
            Try
 
                Dim responseReader As New StreamReader(mywebRequest.GetResponse().GetResponseStream())
                Dim responseData As String = responseReader.ReadToEnd()
 
                responseReader.Close()
 
                ' extract the viewstate value and build out POST data
                Dim viewState As String = ExtractViewState(responseData)
                Dim postData As String = [String].Format("__VIEWSTATE={0}&TUserName={1}&TPassword={2}&_ctl0%3AContent%3AbtnLogon=Logon&__PREVIOUSPAGE=cys_as-zp6tmeXXlc07FggKJUKD96k3RyL8XYHQ-U3I1&__EVENTVALIDATION=%2FwEWAwLV3qjMAQKL2pbeCALf%2B9ffB29vWwhgfdAvHzzk%2F%2BqB%2BKkddRGi", viewState, "[e-mail]", "password")
 
 
                ' have a cookie container ready to receive the forms auth cookie
                Dim cookies As New CookieContainer()
 
                ' now post to the login form
                mywebRequest = TryCast(WebRequest.Create("http://somedomain.com/TheLogOnCheck.aspx"), HttpWebRequest)
 
                mywebRequest.Proxy = pxy 'only if behind proxy (see above)
 
                mywebRequest.Method = "POST"
                mywebRequest.ContentType = "application/x-www-form-urlencoded"
                mywebRequest.CookieContainer = cookies
 
                ' write the form values into the request message
                Dim requestWriter As New StreamWriter(mywebRequest.GetRequestStream())
                requestWriter.Write(postData)
                requestWriter.Close()
 
                ' we don't need the contents of the response, just the cookie it issues
                mywebRequest.GetResponse().Close()
 
                ' now we can send out cookie along with a request for the protected page
                mywebRequest = TryCast(WebRequest.Create(targetUrl), HttpWebRequest)
                mywebRequest.CookieContainer = cookies
                responseReader = New StreamReader(mywebRequest.GetResponse().GetResponseStream())
 
 
                ' and read the response
                responseData = responseReader.ReadToEnd()
                responseReader.Close()
 
                'Here we set up our Regular expression to snatch what's between the tags we want in our html source
                Dim regex As New Regex("<!-- main table -->((.|" & Chr(10) & ")*?)<!-- / main table -->", RegexOptions.IgnoreCase)
 
                'Here we apply our regular expression to our string using the 
                'Match object. 
                Dim oM As Match = regex.Match(responseData)
 
                Return oM.Value
 
            Catch wex As WebException
                'Something went wrong in the HTTP request!  See if it was a timeout problem
                If wex.Status = WebExceptionStatus.Timeout Then
                    Return ("<font color=red>The httpWebRequest has timed out. Please contact the helpdesk</font>")
                Else
                    Return ("<font color=red>FAILED TO CONNECT<br />Status: " & wex.Status & " Message: " & wex.Message & "</font>")
                End If
            End Try
 
 
        End Function
    End Class
End Namespace

In my windows form I then used the following code to extract (scrape) the page contents (puts the retrieved html into a webbrowser control):

    Private Sub myForm_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
        Dim myPrice As New prices
        Dim temp As String = myPrice.GetPrices("http://www.somedomain.org/thepageyouwant.aspx")
        WebBrowser1.DocumentText = temp
    End Sub

If you have an asp.net application you can do exactly the same but on your page_load event:

Dim futuresprice As New prices
        Dim temp As String = futuresprice.GetPrices("http://www.somedomain.org/thepageyouwant.aspx")
Reponse.Write(temp)

Just some points worth noting. The viewstate value that you build your post data with will not necessarily be the same as mine. What you need to do is download a tool called Fiddler (http://www.fiddler2.com/fiddler2). Open it up and log into your target site. Have a look in the session inspector of the page that is used to authenticate the log in (mine was called logoncheck.aspx) and you can see the complete viewstate value. You can then ammend the code to suit the site your requirements. Also I cannot guarantee that his will work for everyone's proxy servers.
Finally credit must be given to Scott Mitchell and Scott Allen as it is their tutorials I used to produce this application.

Hope this was of help.
Ben Foster
<email removed>

[Frinavale]

Wow!
Thank you for providing your solution.

-Frinny