I have a website I am trying to download a file from. This website is
strange in the fact that there is no direct link to the URL of the
file. The way the site works is as follows:
1) Enter in search criteria
2) Click on Download button to go to download page
3) Click on Download button on Download page
4) Server sends 2 bits of data to a script. The data sent does not
include the query data. I'm assuming that the data is stored either in
a server side session variable, or server-side on a temporary database.
5) The script processes and initiates a file download. Inside any of
the web pages, there are NO direct links directly to a file to
download. Once the file is compiled, the webste starts streaming the
data to the standard IE file download dialog. I need to capture this
file that is downloaded into a stream of some sort, then save it out to
a text file. The exported data is a text file.
Here is the code I am trying to use:
<------ CODE -------->
Dim fileURL As New Uri("http://www.somesite.co m")
Dim myFileWebReques t As WebRequest =
WebRequest.Crea te(fileURL)
myFileWebReques t.Method = WebRequestMetho ds.Http.Post
Dim buffer As Byte() =
Encoding.UTF8.G etBytes("type=n ormal&cmd=DOWNL OAD")
myFileWebReques t.ContentType =
"applicatio n/x-www-form-urlencoded"
myFileWebReques t.ContentLength = buffer.Length
myFileWebReques t.Headers.Add(H ttpRequestHeade r.Cookie,
wb.Document.Coo kie)
'myFileWebReque st.Headers.Add( HttpRequestHead er.Referer,
"http://www.somesite.co m/resource/servlet/search")
'myFileWebReque st.Headers.Add( HttpRequestHead er.Accept,
"image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
application/x-shockwave-flash, application/vnd.ms-excel,
application/vnd.ms-powerpoint, application/msword, */*")
myFileWebReques t.Headers.Add(H ttpRequestHeade r.AcceptLanguag e, "en-us")
myFileWebReques t.Headers.Add(H ttpRequestHeade r.AcceptEncodin g, "gzip,
deflate")
'myFileWebReque st.Headers.Add( HttpRequestHead er.UserAgent,
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
1.1.4322; .NET CLR 2.0.50727)")
'myFileWebReque st.Headers.Add( HttpRequestHead er.Host,
"www.somesite.c om")
'myFileWebReque st.Headers.Add( HttpRequestHead er.Connection,
"Keep-Alive")
myFileWebReques t.Headers.Add(H ttpRequestHeade r.CacheControl,
"no-cache")
Dim myFileWebRespon se As WebResponse =
CType(myFileWeb Request.GetResp onse, HttpWebResponse )
Dim receivestream As Stream =
myFileWebRespon se.GetResponseS tream
Dim encode As Encoding =
System.Text.Enc oding.GetEncodi ng("utf-8")
Dim readStream As New StreamReader(re ceivestream, encode)
Dim read(256) As Char
Dim count As Integer = readStream.Read (read, 0, 256)
Dim objWriter As New StreamWriter(sF ileName)
While count 0
Dim str As New String(read, 0, count)
objWriter.Write (str)
count = readStream.Read (read, 0, 256)
End While
readStream.Clos e()
objWriter.Close ()
myFileWebRespon se.Close()
<--------- END CODE --------->
I am not getting ANY errors or exceptions, the program just hangs on
the following line:
Dim myFileWebRespon se As WebResponse =
CType(myFileWeb Request.GetResp onse, HttpWebResponse )
It will sit on this line forever if you let it. The other problem is
that the website denies GET requests. It only allows POST requests.
I've had a number of other programmers look at it, and they are just as
stumped as me. If anyone can help, it would be most appreciated!
Also, if you look in the above code, you will notice a large number of
lines commented out that are designed to set items in the header. If I
uncomment them, then the program hangs and tells me that I am not
assigning it correctly and I need to use the name parameter. If the
other header items that are uncomment work, then shouldn't the
commented out ones work the same?
Thanks for the help in advance,
Wade 1 1898
I have often found the Open Source Ethereal network analysis tool to be
helpful with this sort of thing: http://www.ethereal.com/
--
HTH,
Kevin Spencer
Microsoft MVP
Logostician http://unclechutney.blogspot.com
Parabola is a mate of plane.
"Blackstar" <ww******@hbm2. comwrote in message
news:11******** **************@ 16g2000cwy.goog legroups.com...
>I have a website I am trying to download a file from. This website is
strange in the fact that there is no direct link to the URL of the
file. The way the site works is as follows:
1) Enter in search criteria
2) Click on Download button to go to download page
3) Click on Download button on Download page
4) Server sends 2 bits of data to a script. The data sent does not
include the query data. I'm assuming that the data is stored either in
a server side session variable, or server-side on a temporary database.
5) The script processes and initiates a file download. Inside any of
the web pages, there are NO direct links directly to a file to
download. Once the file is compiled, the webste starts streaming the
data to the standard IE file download dialog. I need to capture this
file that is downloaded into a stream of some sort, then save it out to
a text file. The exported data is a text file.
Here is the code I am trying to use:
<------ CODE -------->
Dim fileURL As New Uri("http://www.somesite.co m")
Dim myFileWebReques t As WebRequest =
WebRequest.Crea te(fileURL)
myFileWebReques t.Method = WebRequestMetho ds.Http.Post
Dim buffer As Byte() =
Encoding.UTF8.G etBytes("type=n ormal&cmd=DOWNL OAD")
myFileWebReques t.ContentType =
"applicatio n/x-www-form-urlencoded"
myFileWebReques t.ContentLength = buffer.Length
myFileWebReques t.Headers.Add(H ttpRequestHeade r.Cookie,
wb.Document.Coo kie)
'myFileWebReque st.Headers.Add( HttpRequestHead er.Referer,
"http://www.somesite.co m/resource/servlet/search")
'myFileWebReque st.Headers.Add( HttpRequestHead er.Accept,
"image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
application/x-shockwave-flash, application/vnd.ms-excel,
application/vnd.ms-powerpoint, application/msword, */*")
myFileWebReques t.Headers.Add(H ttpRequestHeade r.AcceptLanguag e, "en-us")
myFileWebReques t.Headers.Add(H ttpRequestHeade r.AcceptEncodin g, "gzip,
deflate")
'myFileWebReque st.Headers.Add( HttpRequestHead er.UserAgent,
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
1.1.4322; .NET CLR 2.0.50727)")
'myFileWebReque st.Headers.Add( HttpRequestHead er.Host,
"www.somesite.c om")
'myFileWebReque st.Headers.Add( HttpRequestHead er.Connection,
"Keep-Alive")
myFileWebReques t.Headers.Add(H ttpRequestHeade r.CacheControl,
"no-cache")
Dim myFileWebRespon se As WebResponse =
CType(myFileWeb Request.GetResp onse, HttpWebResponse )
Dim receivestream As Stream =
myFileWebRespon se.GetResponseS tream
Dim encode As Encoding =
System.Text.Enc oding.GetEncodi ng("utf-8")
Dim readStream As New StreamReader(re ceivestream, encode)
Dim read(256) As Char
Dim count As Integer = readStream.Read (read, 0, 256)
Dim objWriter As New StreamWriter(sF ileName)
While count 0
Dim str As New String(read, 0, count)
objWriter.Write (str)
count = readStream.Read (read, 0, 256)
End While
readStream.Clos e()
objWriter.Close ()
myFileWebRespon se.Close()
<--------- END CODE --------->
I am not getting ANY errors or exceptions, the program just hangs on
the following line:
Dim myFileWebRespon se As WebResponse =
CType(myFileWeb Request.GetResp onse, HttpWebResponse )
It will sit on this line forever if you let it. The other problem is
that the website denies GET requests. It only allows POST requests.
I've had a number of other programmers look at it, and they are just as
stumped as me. If anyone can help, it would be most appreciated!
Also, if you look in the above code, you will notice a large number of
lines commented out that are designed to set items in the header. If I
uncomment them, then the program hangs and tells me that I am not
assigning it correctly and I need to use the name parameter. If the
other header items that are uncomment work, then shouldn't the
commented out ones work the same?
Thanks for the help in advance,
Wade This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: kajol |
last post by:
Hi everyone
I am trying to get the content of any webpage (URL) using XMLHTTP, and it
is working fine for me, but suddenly I have got
a URL "http://www.bizrate.com/" which is causing a system error and the
error is
System.Runtime.InteropServices.COMException(0xC00CE56E): System error:-
1072896658
|
by: Mike Cronin via DotNetMonster.com |
last post by:
Hi there,
Can anyone tell me what level of encryption is used when making an HTTPS
POST request through an instance of the System.Net.HttpWebRequest object?
Thanks much in advance!
Mike Cronin
Data On Call - Programmer
|
by: Jonathan Wax |
last post by:
Hi,
I spent the last week looking for an answer to the following question:
How can you upload an xml file to an HTTPS server with a specific
certificate.
Basically doing the same as this html form in c# code:
<form method="post" enctype="multipart/form-data"
action="https://www.site.com/upload/"/>
<input type="file" name="xml" id="xml="><br>
|
by: Rajiv Das |
last post by:
Environment: WinXP SP2, .Net 2.0, VS 2005
--------------------------------------------------
This is the code chunk
if (url.AbsoluteUri.StartsWith("http://"))
{
HttpWebRequest HttpWReq = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse HttpWResp = null;
try
{
HttpWResp = (HttpWebResponse)HttpWReq.GetResponse();
|
by: jesper.hvid |
last post by:
Hi.
I've noticed, after moving some of our code to 2.0, that
System.Net.WebRequest.Create(System.String) and
System.Uri(System.String) no longer behave as they did in 1.1
framework.
Example:
string emailHttpPath =
| |
by: peter |
last post by:
Hi,
I have very strange situation but first description ;)
I have:
1) project in VB.NET, in this f.e. 1 function:
Public Function Login(ByVal UserName As String, ByVal UserPassword As
String, Optional ByVal ConnectionParamList As String = Nothing) As
String
|
by: John Kraft |
last post by:
Hello all,
I'm experiencing some, imo, strange behavior with the StreamReader
object I am using in the code below. Summary is that I am downloading
a file from a website and saving it to disk for further parsing. I
know, I could use the WebClient and it would be easier, but I don't
have the flexibility I want with it.
This code appears to work exactly the way I want unless the user
cancels the the background operation. In that...
|
by: Terry Olsen |
last post by:
Given that variable dt = "3/31/2007", why does it produce the following
exception on some machines? It works fine on my PC, but others have sent me
this exception information because it threw up on their PC.
DLDT.Columns("Date").DataType = GetType(Date)
dr.Item("Date") = Date.Parse(dt.Trim).ToShortDateString
System.ArgumentException: The string was not recognized as a valid DateTime.
Couldn't store <3/31/2007in Date Column. Expected...
|
by: tohnsm |
last post by:
Hi.
I have been trying to create a cookie container. But the cookie I have stored is not correct.
Below is the code:
request = CType(WebRequest.Create("....."), httpWebRequest)
request.CookieContainer = New CookieContainer()
myCookie = request.CookieContainer ' store the cookie as global variavble
|
by: Proogeren |
last post by:
I have a problem with a httpwebrequest that I am creating. The request in itself looks correct but using fiddler I see that a www-authentication header is sent along as well. The code is pasted below. I do not add any www-authentication header here so I was wondering if anyone knows how to remove it.
I have used almost 2 days trying to figure this out so help would be highly appreciated.
CORRECT
No proxy-authenticate header is present
no...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
| |
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| |
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
|
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |