By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
443,313 Members | 1,113 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 443,313 IT Pros & Developers. It's quick & easy.

httpWebRequest through Proxy

P: 6
Hi there.

I have written a screen scraping application (both web based and windows forms) in vb.net. When testing on a public broadband link it works fine. However it fails at work due to our proxy server.

To authenticate in the asp version, I just added:
Expand|Select|Wrap|Line Numbers
  1. <defaultProxy useDefaultCredentials="true">
to my web.config - this works fine.

However, I need to integrate this with an existing windows form app (vb6) so need to get the windows form version working.

I am struggling to achieve the same settings (as per web.config) directly within my code. This is what I am trying to use so far:

Expand|Select|Wrap|Line Numbers
  1. Dim mywebRequest As HttpWebRequest = TryCast(WebRequest.Create("http://www.thedomain.com/login.aspx"), HttpWebRequest)
  2.  
  3. mywebRequest.Proxy = New WebProxy("http://myproxyserver")
  4. mywebRequest.Proxy.Credentials = CredentialCache.DefaultCredentials
  5. mywebRequest.Credentials = CredentialCache.DefaultCredentials
As the page I am trying to access requires authentication, I first capture the viewstate of the page. Once I have this I can then simulate a post of my login credentials and use a streamreader to retrieve the contents of the page.

Click here for a C# tutorial.

The thing I dont understand is that the first part of the code appears to work (and passes through the proxy). It is only when i try and close the contents of the response:

Expand|Select|Wrap|Line Numbers
  1. mywebRequest.GetResponse().Close()
that I get the following error:

The remote server returned an error: (407) Proxy Authentication Required.
Could someone advise me as to whether I am defining the proxy correctly?

Thanks.

Ben
Jun 20 '07 #1
Share this Question
Share on Google+
4 Replies


P: 6
Just to let everyone know I got this working in the end. Made a schoolboy error as I had reinitiated the web request but did not add the proxy information for that instance.

If anyone would like a copy of the complete code (to authenticate then screen scrape) then sent me a message
Jun 20 '07 #2

Frinavale
Expert Mod 5K+
P: 9,731
Just to let everyone know I got this working in the end. Made a schoolboy error as I had reinitiated the web request but did not add the proxy information for that instance.

If anyone would like a copy of the complete code (to authenticate then screen scrape) then sent me a message
I'm glad you got it working!
Could you please share the solution so others can learn as well.

Thanks a lot!

-Frinny
Jun 20 '07 #3

P: 6
Well I am glad to help. For some tutorials on httpwebrequest and persisting viewstate for sites that require authentication I would seriously recommend:

http://odetocode.com/Articles/162.aspx (c# version of my code)

and

http://aspnet.4guysfromrolla.com/art...spx#postadlink (great article covering the basics of the httpwebrequest class by Scott Mitchell.

I do have a C# version of the following code if you require (as originally created in this then needed to write in vb.net). This also works as a windows form application or a asp.net app.

Create a new class called prices.vb. Insert the following code (comments explain it pretty well):

Expand|Select|Wrap|Line Numbers
  1. Imports System
  2. Imports System.Net
  3. Imports System.IO
  4. Imports System.Text
  5. Imports System.Web
  6. Imports System.Text.RegularExpressions
  7. Imports Microsoft.VisualBasic
  8. Namespace PageGetter
  9.     Public Class prices
  10.         Public targetURL As String
  11.         Private Function ExtractViewState(ByVal s As String) As String
  12.             Dim viewStateNameDelimiter As String = "__VIEWSTATE"
  13.             Dim valueDelimiter As String = "value="""
  14.  
  15.  
  16.             Dim viewStateNamePosition As Integer = s.IndexOf(viewStateNameDelimiter)
  17.             Dim viewStateValuePosition As Integer = s.IndexOf(valueDelimiter, viewStateNamePosition)
  18.  
  19.             Dim viewStateStartPosition As Integer = viewStateValuePosition + valueDelimiter.Length
  20.             Dim viewStateEndPosition As Integer = s.IndexOf("""", viewStateStartPosition)
  21.  
  22.             Return HttpUtility.UrlEncodeUnicode(s.Substring(viewStateStartPosition, viewStateEndPosition - viewStateStartPosition))
  23.         End Function
  24.  
  25.         Public Function GetPrices(ByVal targetUrl As String) As String
  26. *********************************************************************************
  27. 'only need this if you are behind a proxy            
  28. Dim pxy As New WebProxy("http://yourproxyaddress:0000")
  29.             pxy.Credentials = CredentialCache.DefaultCredentials
  30. *********************************************************************************
  31.             ' first, request the login form to get the viewstate value
  32.             Dim mywebRequest As HttpWebRequest = TryCast(WebRequest.Create("http://somedomain.com/TheLogOnCheck.aspx"), HttpWebRequest)
  33.  
  34.             mywebRequest.Proxy = pxy
  35.  
  36.             'Set the timeout to 1 second (or 1,000 milliseconds)
  37.             mywebRequest.Timeout = 1000
  38.  
  39.  
  40.             Try
  41.  
  42.                 Dim responseReader As New StreamReader(mywebRequest.GetResponse().GetResponseStream())
  43.                 Dim responseData As String = responseReader.ReadToEnd()
  44.  
  45.                 responseReader.Close()
  46.  
  47.                 ' extract the viewstate value and build out POST data
  48.                 Dim viewState As String = ExtractViewState(responseData)
  49.                 Dim postData As String = [String].Format("__VIEWSTATE={0}&TUserName={1}&TPassword={2}&_ctl0%3AContent%3AbtnLogon=Logon&__PREVIOUSPAGE=cys_as-zp6tmeXXlc07FggKJUKD96k3RyL8XYHQ-U3I1&__EVENTVALIDATION=%2FwEWAwLV3qjMAQKL2pbeCALf%2B9ffB29vWwhgfdAvHzzk%2F%2BqB%2BKkddRGi", viewState, "yourname@domain.co.uk", "password")
  50.  
  51.  
  52.                 ' have a cookie container ready to receive the forms auth cookie
  53.                 Dim cookies As New CookieContainer()
  54.  
  55.                 ' now post to the login form
  56.                 mywebRequest = TryCast(WebRequest.Create("http://somedomain.com/TheLogOnCheck.aspx"), HttpWebRequest)
  57.  
  58.                 mywebRequest.Proxy = pxy 'only if behind proxy (see above)
  59.  
  60.                 mywebRequest.Method = "POST"
  61.                 mywebRequest.ContentType = "application/x-www-form-urlencoded"
  62.                 mywebRequest.CookieContainer = cookies
  63.  
  64.                 ' write the form values into the request message
  65.                 Dim requestWriter As New StreamWriter(mywebRequest.GetRequestStream())
  66.                 requestWriter.Write(postData)
  67.                 requestWriter.Close()
  68.  
  69.                 ' we don't need the contents of the response, just the cookie it issues
  70.                 mywebRequest.GetResponse().Close()
  71.  
  72.                 ' now we can send out cookie along with a request for the protected page
  73.                 mywebRequest = TryCast(WebRequest.Create(targetUrl), HttpWebRequest)
  74.                 mywebRequest.CookieContainer = cookies
  75.                 responseReader = New StreamReader(mywebRequest.GetResponse().GetResponseStream())
  76.  
  77.  
  78.                 ' and read the response
  79.                 responseData = responseReader.ReadToEnd()
  80.                 responseReader.Close()
  81.  
  82.                 'Here we set up our Regular expression to snatch what's between the tags we want in our html source
  83.                 Dim regex As New Regex("<!-- main table -->((.|" & Chr(10) & ")*?)<!-- / main table -->", RegexOptions.IgnoreCase)
  84.  
  85.                 'Here we apply our regular expression to our string using the 
  86.                 'Match object. 
  87.                 Dim oM As Match = regex.Match(responseData)
  88.  
  89.                 Return oM.Value
  90.  
  91.             Catch wex As WebException
  92.                 'Something went wrong in the HTTP request!  See if it was a timeout problem
  93.                 If wex.Status = WebExceptionStatus.Timeout Then
  94.                     Return ("<font color=red>The httpWebRequest has timed out. Please contact the helpdesk</font>")
  95.                 Else
  96.                     Return ("<font color=red>FAILED TO CONNECT<br />Status: " & wex.Status & " Message: " & wex.Message & "</font>")
  97.                 End If
  98.             End Try
  99.  
  100.  
  101.         End Function
  102.     End Class
  103. End Namespace
In my windows form I then used the following code to extract (scrape) the page contents (puts the retrieved html into a webbrowser control):

Expand|Select|Wrap|Line Numbers
  1.     Private Sub myForm_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
  2.         Dim myPrice As New prices
  3.         Dim temp As String = myPrice.GetPrices("http://www.somedomain.org/thepageyouwant.aspx")
  4.         WebBrowser1.DocumentText = temp
  5.     End Sub
If you have an asp.net application you can do exactly the same but on your page_load event:
Expand|Select|Wrap|Line Numbers
  1. Dim futuresprice As New prices
  2.         Dim temp As String = futuresprice.GetPrices("http://www.somedomain.org/thepageyouwant.aspx")
  3. Reponse.Write(temp)
Just some points worth noting. The viewstate value that you build your post data with will not necessarily be the same as mine. What you need to do is download a tool called Fiddler (http://www.fiddler2.com/fiddler2). Open it up and log into your target site. Have a look in the session inspector of the page that is used to authenticate the log in (mine was called logoncheck.aspx) and you can see the complete viewstate value. You can then ammend the code to suit the site your requirements. Also I cannot guarantee that his will work for everyone's proxy servers.
Finally credit must be given to Scott Mitchell and Scott Allen as it is their tutorials I used to produce this application.

Hope this was of help.
Ben Foster
<email removed>
Jun 21 '07 #4

Frinavale
Expert Mod 5K+
P: 9,731
Wow!
Thank you for providing your solution.

-Frinny
Jun 21 '07 #5

Post your reply

Sign in to post your reply or Sign up for a free account.