473,581 Members | 2,908 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Strange problem with HttpWebRequest

I have a website I am trying to download a file from. This website is
strange in the fact that there is no direct link to the URL of the
file. The way the site works is as follows:

1) Enter in search criteria
2) Click on Download button to go to download page
3) Click on Download button on Download page
4) Server sends 2 bits of data to a script. The data sent does not
include the query data. I'm assuming that the data is stored either in
a server side session variable, or server-side on a temporary database.
5) The script processes and initiates a file download. Inside any of
the web pages, there are NO direct links directly to a file to
download. Once the file is compiled, the webste starts streaming the
data to the standard IE file download dialog. I need to capture this
file that is downloaded into a stream of some sort, then save it out to
a text file. The exported data is a text file.

Here is the code I am trying to use:

<------ CODE -------->
Dim fileURL As New Uri("http://www.somesite.co m")
Dim myFileWebReques t As WebRequest =
WebRequest.Crea te(fileURL)
myFileWebReques t.Method = WebRequestMetho ds.Http.Post
Dim buffer As Byte() =
Encoding.UTF8.G etBytes("type=n ormal&cmd=DOWNL OAD")

myFileWebReques t.ContentType =
"applicatio n/x-www-form-urlencoded"
myFileWebReques t.ContentLength = buffer.Length
myFileWebReques t.Headers.Add(H ttpRequestHeade r.Cookie,
wb.Document.Coo kie)
'myFileWebReque st.Headers.Add( HttpRequestHead er.Referer,
"http://www.somesite.co m/resource/servlet/search")
'myFileWebReque st.Headers.Add( HttpRequestHead er.Accept,
"image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
application/x-shockwave-flash, application/vnd.ms-excel,
application/vnd.ms-powerpoint, application/msword, */*")

myFileWebReques t.Headers.Add(H ttpRequestHeade r.AcceptLanguag e, "en-us")

myFileWebReques t.Headers.Add(H ttpRequestHeade r.AcceptEncodin g, "gzip,
deflate")
'myFileWebReque st.Headers.Add( HttpRequestHead er.UserAgent,
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
1.1.4322; .NET CLR 2.0.50727)")
'myFileWebReque st.Headers.Add( HttpRequestHead er.Host,
"www.somesite.c om")
'myFileWebReque st.Headers.Add( HttpRequestHead er.Connection,
"Keep-Alive")

myFileWebReques t.Headers.Add(H ttpRequestHeade r.CacheControl,
"no-cache")

Dim myFileWebRespon se As WebResponse =
CType(myFileWeb Request.GetResp onse, HttpWebResponse )
Dim receivestream As Stream =
myFileWebRespon se.GetResponseS tream
Dim encode As Encoding =
System.Text.Enc oding.GetEncodi ng("utf-8")
Dim readStream As New StreamReader(re ceivestream, encode)
Dim read(256) As Char
Dim count As Integer = readStream.Read (read, 0, 256)
Dim objWriter As New StreamWriter(sF ileName)

While count 0
Dim str As New String(read, 0, count)

objWriter.Write (str)
count = readStream.Read (read, 0, 256)
End While
readStream.Clos e()
objWriter.Close ()

myFileWebRespon se.Close()

<--------- END CODE --------->

I am not getting ANY errors or exceptions, the program just hangs on
the following line:
Dim myFileWebRespon se As WebResponse =
CType(myFileWeb Request.GetResp onse, HttpWebResponse )

It will sit on this line forever if you let it. The other problem is
that the website denies GET requests. It only allows POST requests.
I've had a number of other programmers look at it, and they are just as
stumped as me. If anyone can help, it would be most appreciated!

Also, if you look in the above code, you will notice a large number of
lines commented out that are designed to set items in the header. If I
uncomment them, then the program hangs and tells me that I am not
assigning it correctly and I need to use the name parameter. If the
other header items that are uncomment work, then shouldn't the
commented out ones work the same?

Thanks for the help in advance,
Wade

Nov 29 '06 #1
1 1889
I have often found the Open Source Ethereal network analysis tool to be
helpful with this sort of thing:

http://www.ethereal.com/

--
HTH,

Kevin Spencer
Microsoft MVP
Logostician
http://unclechutney.blogspot.com

Parabola is a mate of plane.
"Blackstar" <ww******@hbm2. comwrote in message
news:11******** **************@ 16g2000cwy.goog legroups.com...
>I have a website I am trying to download a file from. This website is
strange in the fact that there is no direct link to the URL of the
file. The way the site works is as follows:

1) Enter in search criteria
2) Click on Download button to go to download page
3) Click on Download button on Download page
4) Server sends 2 bits of data to a script. The data sent does not
include the query data. I'm assuming that the data is stored either in
a server side session variable, or server-side on a temporary database.
5) The script processes and initiates a file download. Inside any of
the web pages, there are NO direct links directly to a file to
download. Once the file is compiled, the webste starts streaming the
data to the standard IE file download dialog. I need to capture this
file that is downloaded into a stream of some sort, then save it out to
a text file. The exported data is a text file.

Here is the code I am trying to use:

<------ CODE -------->
Dim fileURL As New Uri("http://www.somesite.co m")
Dim myFileWebReques t As WebRequest =
WebRequest.Crea te(fileURL)
myFileWebReques t.Method = WebRequestMetho ds.Http.Post
Dim buffer As Byte() =
Encoding.UTF8.G etBytes("type=n ormal&cmd=DOWNL OAD")

myFileWebReques t.ContentType =
"applicatio n/x-www-form-urlencoded"
myFileWebReques t.ContentLength = buffer.Length
myFileWebReques t.Headers.Add(H ttpRequestHeade r.Cookie,
wb.Document.Coo kie)
'myFileWebReque st.Headers.Add( HttpRequestHead er.Referer,
"http://www.somesite.co m/resource/servlet/search")
'myFileWebReque st.Headers.Add( HttpRequestHead er.Accept,
"image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
application/x-shockwave-flash, application/vnd.ms-excel,
application/vnd.ms-powerpoint, application/msword, */*")

myFileWebReques t.Headers.Add(H ttpRequestHeade r.AcceptLanguag e, "en-us")

myFileWebReques t.Headers.Add(H ttpRequestHeade r.AcceptEncodin g, "gzip,
deflate")
'myFileWebReque st.Headers.Add( HttpRequestHead er.UserAgent,
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
1.1.4322; .NET CLR 2.0.50727)")
'myFileWebReque st.Headers.Add( HttpRequestHead er.Host,
"www.somesite.c om")
'myFileWebReque st.Headers.Add( HttpRequestHead er.Connection,
"Keep-Alive")

myFileWebReques t.Headers.Add(H ttpRequestHeade r.CacheControl,
"no-cache")

Dim myFileWebRespon se As WebResponse =
CType(myFileWeb Request.GetResp onse, HttpWebResponse )
Dim receivestream As Stream =
myFileWebRespon se.GetResponseS tream
Dim encode As Encoding =
System.Text.Enc oding.GetEncodi ng("utf-8")
Dim readStream As New StreamReader(re ceivestream, encode)
Dim read(256) As Char
Dim count As Integer = readStream.Read (read, 0, 256)
Dim objWriter As New StreamWriter(sF ileName)

While count 0
Dim str As New String(read, 0, count)

objWriter.Write (str)
count = readStream.Read (read, 0, 256)
End While
readStream.Clos e()
objWriter.Close ()

myFileWebRespon se.Close()

<--------- END CODE --------->

I am not getting ANY errors or exceptions, the program just hangs on
the following line:
Dim myFileWebRespon se As WebResponse =
CType(myFileWeb Request.GetResp onse, HttpWebResponse )

It will sit on this line forever if you let it. The other problem is
that the website denies GET requests. It only allows POST requests.
I've had a number of other programmers look at it, and they are just as
stumped as me. If anyone can help, it would be most appreciated!

Also, if you look in the above code, you will notice a large number of
lines commented out that are designed to set items in the header. If I
uncomment them, then the program hangs and tells me that I am not
assigning it correctly and I need to use the name parameter. If the
other header items that are uncomment work, then shouldn't the
commented out ones work the same?

Thanks for the help in advance,
Wade

Nov 30 '06 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
1983
by: kajol | last post by:
Hi everyone I am trying to get the content of any webpage (URL) using XMLHTTP, and it is working fine for me, but suddenly I have got a URL "http://www.bizrate.com/" which is causing a system error and the error is System.Runtime.InteropServices.COMException(0xC00CE56E): System error:- 1072896658
9
8162
by: Mike Cronin via DotNetMonster.com | last post by:
Hi there, Can anyone tell me what level of encryption is used when making an HTTPS POST request through an instance of the System.Net.HttpWebRequest object? Thanks much in advance! Mike Cronin Data On Call - Programmer
2
2261
by: Jonathan Wax | last post by:
Hi, I spent the last week looking for an answer to the following question: How can you upload an xml file to an HTTPS server with a specific certificate. Basically doing the same as this html form in c# code: <form method="post" enctype="multipart/form-data" action="https://www.site.com/upload/"/> <input type="file" name="xml"...
0
896
by: Rajiv Das | last post by:
Environment: WinXP SP2, .Net 2.0, VS 2005 -------------------------------------------------- This is the code chunk if (url.AbsoluteUri.StartsWith("http://")) { HttpWebRequest HttpWReq = (HttpWebRequest)WebRequest.Create(url); HttpWebResponse HttpWResp = null; try { HttpWResp = (HttpWebResponse)HttpWReq.GetResponse();
0
1792
by: jesper.hvid | last post by:
Hi. I've noticed, after moving some of our code to 2.0, that System.Net.WebRequest.Create(System.String) and System.Uri(System.String) no longer behave as they did in 1.1 framework. Example: string emailHttpPath =
2
4098
by: peter | last post by:
Hi, I have very strange situation but first description ;) I have: 1) project in VB.NET, in this f.e. 1 function: Public Function Login(ByVal UserName As String, ByVal UserPassword As String, Optional ByVal ConnectionParamList As String = Nothing) As String
10
2327
by: John Kraft | last post by:
Hello all, I'm experiencing some, imo, strange behavior with the StreamReader object I am using in the code below. Summary is that I am downloading a file from a website and saving it to disk for further parsing. I know, I could use the WebClient and it would be easier, but I don't have the flexibility I want with it. This code appears...
17
2260
by: Terry Olsen | last post by:
Given that variable dt = "3/31/2007", why does it produce the following exception on some machines? It works fine on my PC, but others have sent me this exception information because it threw up on their PC. DLDT.Columns("Date").DataType = GetType(Date) dr.Item("Date") = Date.Parse(dt.Trim).ToShortDateString System.ArgumentException: The...
1
1422
by: tohnsm | last post by:
Hi. I have been trying to create a cookie container. But the cookie I have stored is not correct. Below is the code: request = CType(WebRequest.Create("....."), httpWebRequest) request.CookieContainer = New CookieContainer() myCookie = request.CookieContainer ' store the cookie as global variavble
1
8008
by: Proogeren | last post by:
I have a problem with a httpwebrequest that I am creating. The request in itself looks correct but using fiddler I see that a www-authentication header is sent along as well. The code is pasted below. I do not add any www-authentication header here so I was wondering if anyone knows how to remove it. I have used almost 2 days trying to figure...
0
7882
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7808
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
8157
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
8312
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
0
8181
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
0
6564
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
1
5683
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
5366
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
1
1410
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.