Strange problem with HttpWebRequest

[Blackstar]

I have a website I am trying to download a file from. This website is
strange in the fact that there is no direct link to the URL of the
file. The way the site works is as follows:

1) Enter in search criteria
2) Click on Download button to go to download page
3) Click on Download button on Download page
4) Server sends 2 bits of data to a script. The data sent does not
include the query data. I'm assuming that the data is stored either in
a server side session variable, or server-side on a temporary database.
5) The script processes and initiates a file download. Inside any of
the web pages, there are NO direct links directly to a file to
download. Once the file is compiled, the webste starts streaming the
data to the standard IE file download dialog. I need to capture this
file that is downloaded into a stream of some sort, then save it out to
a text file. The exported data is a text file.

Here is the code I am trying to use:

<------ CODE -------->
Dim fileURL As New Uri("http://www.somesite.com")
Dim myFileWebRequest As WebRequest =
WebRequest.Create(fileURL)
myFileWebRequest.Method = WebRequestMethods.Http.Post
Dim buffer As Byte() =
Encoding.UTF8.GetBytes("type=normal&cmd=DOWNLOAD")

myFileWebRequest.ContentType =
"application/x-www-form-urlencoded"
myFileWebRequest.ContentLength = buffer.Length
myFileWebRequest.Headers.Add(HttpRequestHeader.Coo kie,
wb.Document.Cookie)
'myFileWebRequest.Headers.Add(HttpRequestHeader.Re ferer,
"http://www.somesite.com/resource/servlet/search")
'myFileWebRequest.Headers.Add(HttpRequestHeader.Ac cept,
"image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
application/x-shockwave-flash, application/vnd.ms-excel,
application/vnd.ms-powerpoint, application/msword, */*")

myFileWebRequest.Headers.Add(HttpRequestHeader.Acc eptLanguage, "en-us")

myFileWebRequest.Headers.Add(HttpRequestHeader.Acc eptEncoding, "gzip,
deflate")
'myFileWebRequest.Headers.Add(HttpRequestHeader.Us erAgent,
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
1.1.4322; .NET CLR 2.0.50727)")
'myFileWebRequest.Headers.Add(HttpRequestHeader.Ho st,
"www.somesite.com")
'myFileWebRequest.Headers.Add(HttpRequestHeader.Co nnection,
"Keep-Alive")

myFileWebRequest.Headers.Add(HttpRequestHeader.Cac heControl,
"no-cache")

Dim myFileWebResponse As WebResponse =
CType(myFileWebRequest.GetResponse, HttpWebResponse)
Dim receivestream As Stream =
myFileWebResponse.GetResponseStream
Dim encode As Encoding =
System.Text.Encoding.GetEncoding("utf-8")
Dim readStream As New StreamReader(receivestream, encode)
Dim read(256) As Char
Dim count As Integer = readStream.Read(read, 0, 256)
Dim objWriter As New StreamWriter(sFileName)

While count 0
Dim str As New String(read, 0, count)

objWriter.Write(str)
count = readStream.Read(read, 0, 256)
End While
readStream.Close()
objWriter.Close()

myFileWebResponse.Close()

<--------- END CODE --------->

I am not getting ANY errors or exceptions, the program just hangs on
the following line:
Dim myFileWebResponse As WebResponse =
CType(myFileWebRequest.GetResponse, HttpWebResponse)

It will sit on this line forever if you let it. The other problem is
that the website denies GET requests. It only allows POST requests.
I've had a number of other programmers look at it, and they are just as
stumped as me. If anyone can help, it would be most appreciated!

Also, if you look in the above code, you will notice a large number of
lines commented out that are designed to set items in the header. If I
uncomment them, then the program hangs and tells me that I am not
assigning it correctly and I need to use the name parameter. If the
other header items that are uncomment work, then shouldn't the
commented out ones work the same?

Thanks for the help in advance,
Wade

[Kevin Spencer]

I have often found the Open Source Ethereal network analysis tool to be
helpful with this sort of thing:

http://www.ethereal.com/

--
HTH,

Kevin Spencer
Microsoft MVP
Logostician
http://unclechutney.blogspot.com

Parabola is a mate of plane.
"Blackstar" <ww******@hbm2.comwrote in message
news:11**********************@16g2000cwy.googlegro ups.com...

>I have a website I am trying to download a file from. This website is
strange in the fact that there is no direct link to the URL of the
file. The way the site works is as follows:

1) Enter in search criteria
2) Click on Download button to go to download page
3) Click on Download button on Download page
4) Server sends 2 bits of data to a script. The data sent does not
include the query data. I'm assuming that the data is stored either in
a server side session variable, or server-side on a temporary database.
5) The script processes and initiates a file download. Inside any of
the web pages, there are NO direct links directly to a file to
download. Once the file is compiled, the webste starts streaming the
data to the standard IE file download dialog. I need to capture this
file that is downloaded into a stream of some sort, then save it out to
a text file. The exported data is a text file.

Here is the code I am trying to use:

<------ CODE -------->
Dim fileURL As New Uri("http://www.somesite.com")
Dim myFileWebRequest As WebRequest =
WebRequest.Create(fileURL)
myFileWebRequest.Method = WebRequestMethods.Http.Post
Dim buffer As Byte() =
Encoding.UTF8.GetBytes("type=normal&cmd=DOWNLOAD")

myFileWebRequest.ContentType =
"application/x-www-form-urlencoded"
myFileWebRequest.ContentLength = buffer.Length
myFileWebRequest.Headers.Add(HttpRequestHeader.Coo kie,
wb.Document.Cookie)
'myFileWebRequest.Headers.Add(HttpRequestHeader.Re ferer,
"http://www.somesite.com/resource/servlet/search")
'myFileWebRequest.Headers.Add(HttpRequestHeader.Ac cept,
"image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
application/x-shockwave-flash, application/vnd.ms-excel,
application/vnd.ms-powerpoint, application/msword, */*")

myFileWebRequest.Headers.Add(HttpRequestHeader.Acc eptLanguage, "en-us")

myFileWebRequest.Headers.Add(HttpRequestHeader.Acc eptEncoding, "gzip,
deflate")
'myFileWebRequest.Headers.Add(HttpRequestHeader.Us erAgent,
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
1.1.4322; .NET CLR 2.0.50727)")
'myFileWebRequest.Headers.Add(HttpRequestHeader.Ho st,
"www.somesite.com")
'myFileWebRequest.Headers.Add(HttpRequestHeader.Co nnection,
"Keep-Alive")

myFileWebRequest.Headers.Add(HttpRequestHeader.Cac heControl,
"no-cache")

Dim myFileWebResponse As WebResponse =
CType(myFileWebRequest.GetResponse, HttpWebResponse)
Dim receivestream As Stream =
myFileWebResponse.GetResponseStream
Dim encode As Encoding =
System.Text.Encoding.GetEncoding("utf-8")
Dim readStream As New StreamReader(receivestream, encode)
Dim read(256) As Char
Dim count As Integer = readStream.Read(read, 0, 256)
Dim objWriter As New StreamWriter(sFileName)

While count 0
Dim str As New String(read, 0, count)

objWriter.Write(str)
count = readStream.Read(read, 0, 256)
End While
readStream.Close()
objWriter.Close()

myFileWebResponse.Close()

<--------- END CODE --------->

I am not getting ANY errors or exceptions, the program just hangs on
the following line:
Dim myFileWebResponse As WebResponse =
CType(myFileWebRequest.GetResponse, HttpWebResponse)

It will sit on this line forever if you let it. The other problem is
that the website denies GET requests. It only allows POST requests.
I've had a number of other programmers look at it, and they are just as
stumped as me. If anyone can help, it would be most appreciated!

Also, if you look in the above code, you will notice a large number of
lines commented out that are designed to set items in the header. If I
uncomment them, then the program hangs and tells me that I am not
assigning it correctly and I need to use the name parameter. If the
other header items that are uncomment work, then shouldn't the
commented out ones work the same?

Thanks for the help in advance,
Wade