473,226 Members | 1,396 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,226 software developers and data experts.

httpWebRequest through Proxy

Hi there.

I have written a screen scraping application (both web based and windows forms) in vb.net. When testing on a public broadband link it works fine. However it fails at work due to our proxy server.

To authenticate in the asp version, I just added:
Expand|Select|Wrap|Line Numbers
  1. <defaultProxy useDefaultCredentials="true">
to my web.config - this works fine.

However, I need to integrate this with an existing windows form app (vb6) so need to get the windows form version working.

I am struggling to achieve the same settings (as per web.config) directly within my code. This is what I am trying to use so far:

Expand|Select|Wrap|Line Numbers
  1. Dim mywebRequest As HttpWebRequest = TryCast(WebRequest.Create("http://www.thedomain.com/login.aspx"), HttpWebRequest)
  3. mywebRequest.Proxy = New WebProxy("http://myproxyserver")
  4. mywebRequest.Proxy.Credentials = CredentialCache.DefaultCredentials
  5. mywebRequest.Credentials = CredentialCache.DefaultCredentials
As the page I am trying to access requires authentication, I first capture the viewstate of the page. Once I have this I can then simulate a post of my login credentials and use a streamreader to retrieve the contents of the page.

Click here for a C# tutorial.

The thing I dont understand is that the first part of the code appears to work (and passes through the proxy). It is only when i try and close the contents of the response:

Expand|Select|Wrap|Line Numbers
  1. mywebRequest.GetResponse().Close()
that I get the following error:

The remote server returned an error: (407) Proxy Authentication Required.
Could someone advise me as to whether I am defining the proxy correctly?


Jun 20 '07 #1
4 9542
Just to let everyone know I got this working in the end. Made a schoolboy error as I had reinitiated the web request but did not add the proxy information for that instance.

If anyone would like a copy of the complete code (to authenticate then screen scrape) then sent me a message
Jun 20 '07 #2
9,735 Expert Mod 8TB
Just to let everyone know I got this working in the end. Made a schoolboy error as I had reinitiated the web request but did not add the proxy information for that instance.

If anyone would like a copy of the complete code (to authenticate then screen scrape) then sent me a message
I'm glad you got it working!
Could you please share the solution so others can learn as well.

Thanks a lot!

Jun 20 '07 #3
Well I am glad to help. For some tutorials on httpwebrequest and persisting viewstate for sites that require authentication I would seriously recommend:

http://odetocode.com/Articles/162.aspx (c# version of my code)


http://aspnet.4guysfromrolla.com/art...spx#postadlink (great article covering the basics of the httpwebrequest class by Scott Mitchell.

I do have a C# version of the following code if you require (as originally created in this then needed to write in vb.net). This also works as a windows form application or a asp.net app.

Create a new class called prices.vb. Insert the following code (comments explain it pretty well):

Expand|Select|Wrap|Line Numbers
  1. Imports System
  2. Imports System.Net
  3. Imports System.IO
  4. Imports System.Text
  5. Imports System.Web
  6. Imports System.Text.RegularExpressions
  7. Imports Microsoft.VisualBasic
  8. Namespace PageGetter
  9.     Public Class prices
  10.         Public targetURL As String
  11.         Private Function ExtractViewState(ByVal s As String) As String
  12.             Dim viewStateNameDelimiter As String = "__VIEWSTATE"
  13.             Dim valueDelimiter As String = "value="""
  16.             Dim viewStateNamePosition As Integer = s.IndexOf(viewStateNameDelimiter)
  17.             Dim viewStateValuePosition As Integer = s.IndexOf(valueDelimiter, viewStateNamePosition)
  19.             Dim viewStateStartPosition As Integer = viewStateValuePosition + valueDelimiter.Length
  20.             Dim viewStateEndPosition As Integer = s.IndexOf("""", viewStateStartPosition)
  22.             Return HttpUtility.UrlEncodeUnicode(s.Substring(viewStateStartPosition, viewStateEndPosition - viewStateStartPosition))
  23.         End Function
  25.         Public Function GetPrices(ByVal targetUrl As String) As String
  26. *********************************************************************************
  27. 'only need this if you are behind a proxy            
  28. Dim pxy As New WebProxy("http://yourproxyaddress:0000")
  29.             pxy.Credentials = CredentialCache.DefaultCredentials
  30. *********************************************************************************
  31.             ' first, request the login form to get the viewstate value
  32.             Dim mywebRequest As HttpWebRequest = TryCast(WebRequest.Create("http://somedomain.com/TheLogOnCheck.aspx"), HttpWebRequest)
  34.             mywebRequest.Proxy = pxy
  36.             'Set the timeout to 1 second (or 1,000 milliseconds)
  37.             mywebRequest.Timeout = 1000
  40.             Try
  42.                 Dim responseReader As New StreamReader(mywebRequest.GetResponse().GetResponseStream())
  43.                 Dim responseData As String = responseReader.ReadToEnd()
  45.                 responseReader.Close()
  47.                 ' extract the viewstate value and build out POST data
  48.                 Dim viewState As String = ExtractViewState(responseData)
  49.                 Dim postData As String = [String].Format("__VIEWSTATE={0}&TUserName={1}&TPassword={2}&_ctl0%3AContent%3AbtnLogon=Logon&__PREVIOUSPAGE=cys_as-zp6tmeXXlc07FggKJUKD96k3RyL8XYHQ-U3I1&__EVENTVALIDATION=%2FwEWAwLV3qjMAQKL2pbeCALf%2B9ffB29vWwhgfdAvHzzk%2F%2BqB%2BKkddRGi", viewState, "yourname@domain.co.uk", "password")
  52.                 ' have a cookie container ready to receive the forms auth cookie
  53.                 Dim cookies As New CookieContainer()
  55.                 ' now post to the login form
  56.                 mywebRequest = TryCast(WebRequest.Create("http://somedomain.com/TheLogOnCheck.aspx"), HttpWebRequest)
  58.                 mywebRequest.Proxy = pxy 'only if behind proxy (see above)
  60.                 mywebRequest.Method = "POST"
  61.                 mywebRequest.ContentType = "application/x-www-form-urlencoded"
  62.                 mywebRequest.CookieContainer = cookies
  64.                 ' write the form values into the request message
  65.                 Dim requestWriter As New StreamWriter(mywebRequest.GetRequestStream())
  66.                 requestWriter.Write(postData)
  67.                 requestWriter.Close()
  69.                 ' we don't need the contents of the response, just the cookie it issues
  70.                 mywebRequest.GetResponse().Close()
  72.                 ' now we can send out cookie along with a request for the protected page
  73.                 mywebRequest = TryCast(WebRequest.Create(targetUrl), HttpWebRequest)
  74.                 mywebRequest.CookieContainer = cookies
  75.                 responseReader = New StreamReader(mywebRequest.GetResponse().GetResponseStream())
  78.                 ' and read the response
  79.                 responseData = responseReader.ReadToEnd()
  80.                 responseReader.Close()
  82.                 'Here we set up our Regular expression to snatch what's between the tags we want in our html source
  83.                 Dim regex As New Regex("<!-- main table -->((.|" & Chr(10) & ")*?)<!-- / main table -->", RegexOptions.IgnoreCase)
  85.                 'Here we apply our regular expression to our string using the 
  86.                 'Match object. 
  87.                 Dim oM As Match = regex.Match(responseData)
  89.                 Return oM.Value
  91.             Catch wex As WebException
  92.                 'Something went wrong in the HTTP request!  See if it was a timeout problem
  93.                 If wex.Status = WebExceptionStatus.Timeout Then
  94.                     Return ("<font color=red>The httpWebRequest has timed out. Please contact the helpdesk</font>")
  95.                 Else
  96.                     Return ("<font color=red>FAILED TO CONNECT<br />Status: " & wex.Status & " Message: " & wex.Message & "</font>")
  97.                 End If
  98.             End Try
  101.         End Function
  102.     End Class
  103. End Namespace
In my windows form I then used the following code to extract (scrape) the page contents (puts the retrieved html into a webbrowser control):

Expand|Select|Wrap|Line Numbers
  1.     Private Sub myForm_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
  2.         Dim myPrice As New prices
  3.         Dim temp As String = myPrice.GetPrices("http://www.somedomain.org/thepageyouwant.aspx")
  4.         WebBrowser1.DocumentText = temp
  5.     End Sub
If you have an asp.net application you can do exactly the same but on your page_load event:
Expand|Select|Wrap|Line Numbers
  1. Dim futuresprice As New prices
  2.         Dim temp As String = futuresprice.GetPrices("http://www.somedomain.org/thepageyouwant.aspx")
  3. Reponse.Write(temp)
Just some points worth noting. The viewstate value that you build your post data with will not necessarily be the same as mine. What you need to do is download a tool called Fiddler (http://www.fiddler2.com/fiddler2). Open it up and log into your target site. Have a look in the session inspector of the page that is used to authenticate the log in (mine was called logoncheck.aspx) and you can see the complete viewstate value. You can then ammend the code to suit the site your requirements. Also I cannot guarantee that his will work for everyone's proxy servers.
Finally credit must be given to Scott Mitchell and Scott Allen as it is their tutorials I used to produce this application.

Hope this was of help.
Ben Foster
<email removed>
Jun 21 '07 #4
9,735 Expert Mod 8TB
Thank you for providing your solution.

Jun 21 '07 #5

Sign in to post your reply or Sign up for a free account.

Similar topics

by: thomas peter | last post by:
I am building a precache engine... one that request over 100 pages on an remote server to cache them remotely... can i use the HttpWebRequest and WebResponse classes for this? or must i use the...
by: Steve Richter | last post by:
I have a page that uses simple HTTP GET to do an ISBN lookup via Amazon.com. The page works when I run it from //localhost. But I have moved it to my godaddy.com shared hoster site, and I get...
by: Imran Aziz | last post by:
Hello All, I am using HttpWebRequest to fetch webpages in my ASP.net C# application. The request works fine without the proxy, but on using the code from within a network that uses proxy the...
by: Dave Brown | last post by:
I am attempting to post to a url (https://FakeURL/logon.asp) using the HttpWebRequest class. The response for a succesful post will contain the html for the logon user's default page. We've...
by: Dave Brown | last post by:
I am attempting to post to a url (https://FakeURL/logon.asp) using the HttpWebRequest class. The response for a succesful post will contain the html for the logon user's default page. We've...
by: Tyler | last post by:
I am using httpwebrequest to do a screen scrape. This works great on my development box, but does not on the production box. Here is the code. Dim webRequest As HttpWebRequest =...
by: =?Utf-8?B?U2ltb25EZXY=?= | last post by:
Hi I have a utility class, called MailHandler, that I wrote to read and operate on emails on an Exchange server using WebDAV. The WebDAV SQL statements are sent using an HttpWebRequest object....
by: moo | last post by:
Is there a simple way to get my logon credentials to make my web request work through our proxy server? I tried CredentialCache.DefaultCredentials, but I get nothing back. I can get it to work if I...
by: =?Utf-8?B?TGFycnlLdXBlcm1hbg==?= | last post by:
Our WebDev team seems to have found a problem that exposes a bug in .NET 2.0. This problem can be shown when trying to access a WebService using SSL and through a proxy server after using the...
by: =?Utf-8?B?TGVuc3Rlcg==?= | last post by:
A C# (.NET 2) application which uses the System.Net.HttpWebRequest object to request a resource over HTTPS is failing following the installation of a new proxy server on our internal network with...
by: VivesProcSPL | last post by:
Obviously, one of the original purposes of SQL is to make data query processing easy. The language uses many English-like terms and syntax in an effort to make it easy to learn, particularly for...
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.