webpage download?

[Alan Zhong]

i wonder how to download a webpage automatically using a C# program
with WebClient class.
i tried it and it's working. but somehow the data it gets is different
than the page shown in microsoft internet explorer.
and i wonder how to get around the login problems. the C# programs
only downloads the login page, do i need to "instruct" the program to
access the cookie information on my window?
thankx

[Joerg Jooss]

Alan Zhong wrote:

i wonder how to download a webpage automatically using a C# program
with WebClient class.
i tried it and it's working. but somehow the data it gets is different
than the page shown in microsoft internet explorer.
and i wonder how to get around the login problems. the C# programs
only downloads the login page, do i need to "instruct" the program to
access the cookie information on my window?

WebClient is not sufficient to implement HTTP conversations, since it
doesn't support cookies. You must use HttpWebRequest and HttpWebResponse
instead.

Cheers,
--
Joerg Jooss
jo*********@gmx.net

[Alan Zhong]

hi everybody, i am still trying to get this work...
here were what i did.
1) use HttpWebRequest to connection the server with username +
password. call getResponse, successfully get the response header
containing the ["Set-Cookie"] data.
2) put ["Set-Cookie"] into the headers of a newly created request with
the website i want to download under the same protected domain.
3) call getResponose again but it still redirects me to the login
page.

what did i do wrong?
the following are the codes i have:
////////////////////////// test12.cs
using System;
using System.IO;
using System.Net;
class test12
{
static void Main(string[] args)
{
String address = "xxx login page with username and password xxx";
String newaddr = "page wanted to download";

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(address);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();

request = (HttpWebRequest) WebRequest.Create(newaddr);
// response headers info:
// Header Name:Server, Value :Sun-ONE-Web-Server/6.1
// Header Name:Date, Value :Thu, 01 Apr 2004 21:12:48 GMT
// Header Name:Content-type, Value :text/html
// Header Name:Set-cookie, Value :RMID=xxxxx; expires=Friday,
// 01-Apr-2005 21:12:48 GMT; path=/; domain=.xxx.com
// Header Name:Cache-control, Value :no-cache
// Header Name:Pragma, Value :no-cache
// Header Name:Transfer-encoding, Value :chunked
request.Headers.Add("Cookie", response.Headers["Set-Cookie"]);

Console.WriteLine("old Address: " + request.Address);
response = (HttpWebResponse) request.GetResponse();
Console.WriteLine("new Address: " + request.Address);
Console.WriteLine("ResponseUri " + response.ResponseUri);

StreamReader sr = new StreamReader(response.GetResponseStream());
String result = sr.ReadToEnd();
sr.Close();

response.Close();

Console.WriteLine(result.Length);
if (result.Length > 1000)
result = result.Substring(0, 1000);
Console.WriteLine(result);
}

[Alan Zhong]

hi everybody, i am still trying to get this work...
here were what i did.
1) use HttpWebRequest to connection the server with username +
password. call getResponse, successfully get the response header
containing the ["Set-Cookie"] data.
2) put ["Set-Cookie"] into the headers of a newly created request with
the website i want to download under the same protected domain.
3) call getResponose again but it still redirects me to the login
page.

what did i do wrong?
the following are the codes i have:
////////////////////////// test12.cs
using System;
using System.IO;
using System.Net;
class test12
{
static void Main(string[] args)
{
String address = "xxx login page with username and password xxx";
String newaddr = "page wanted to download";

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(address);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();

request = (HttpWebRequest) WebRequest.Create(newaddr);
// response headers info:
// Header Name:Server, Value :Sun-ONE-Web-Server/6.1
// Header Name:Date, Value :Thu, 01 Apr 2004 21:12:48 GMT
// Header Name:Content-type, Value :text/html
// Header Name:Set-cookie, Value :RMID=xxxxx; expires=Friday,
// 01-Apr-2005 21:12:48 GMT; path=/; domain=.xxx.com
// Header Name:Cache-control, Value :no-cache
// Header Name:Pragma, Value :no-cache
// Header Name:Transfer-encoding, Value :chunked
request.Headers.Add("Cookie", response.Headers["Set-Cookie"]);

Console.WriteLine("old Address: " + request.Address);
response = (HttpWebResponse) request.GetResponse();
Console.WriteLine("new Address: " + request.Address);
Console.WriteLine("ResponseUri " + response.ResponseUri);

StreamReader sr = new StreamReader(response.GetResponseStream());
String result = sr.ReadToEnd();
sr.Close();

response.Close();

Console.WriteLine(result.Length);
if (result.Length > 1000)
result = result.Substring(0, 1000);
Console.WriteLine(result);
}

[Alan Zhong]

hi everybody, i am still trying to get this work...
here were what i did.
1) use HttpWebRequest to connection the server with username +
password. call getResponse, successfully get the response header
containing the ["Set-Cookie"] data.
2) put ["Set-Cookie"] into the headers of a newly created request with
the website i want to download under the same protected domain.
3) call getResponose again but it still redirects me to the login
page.

what did i do wrong?
the following are the codes i have:
////////////////////////// test12.cs
using System;
using System.IO;
using System.Net;
class test12
{
static void Main(string[] args)
{
String address = "xxx login page with username and password xxx";
String newaddr = "page wanted to download";

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(address);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();

request = (HttpWebRequest) WebRequest.Create(newaddr);
// response headers info:
// Header Name:Server, Value :Sun-ONE-Web-Server/6.1
// Header Name:Date, Value :Thu, 01 Apr 2004 21:12:48 GMT
// Header Name:Content-type, Value :text/html
// Header Name:Set-cookie, Value :RMID=xxxxx; expires=Friday,
// 01-Apr-2005 21:12:48 GMT; path=/; domain=.xxx.com
// Header Name:Cache-control, Value :no-cache
// Header Name:Pragma, Value :no-cache
// Header Name:Transfer-encoding, Value :chunked
request.Headers.Add("Cookie", response.Headers["Set-Cookie"]);

Console.WriteLine("old Address: " + request.Address);
response = (HttpWebResponse) request.GetResponse();
Console.WriteLine("new Address: " + request.Address);
Console.WriteLine("ResponseUri " + response.ResponseUri);

StreamReader sr = new StreamReader(response.GetResponseStream());
String result = sr.ReadToEnd();
sr.Close();

response.Close();

Console.WriteLine(result.Length);
if (result.Length > 1000)
result = result.Substring(0, 1000);
Console.WriteLine(result);
}

[Alan Zhong]

hi everybody, i am still trying to get this work...
here were what i did.
1) use HttpWebRequest to connection the server with username +
password. call getResponse, successfully get the response header
containing the ["Set-Cookie"] data.
2) put ["Set-Cookie"] into the headers of a newly created request with
the website i want to download under the same protected domain.
3) call getResponose again but it still redirects me to the login
page.

what did i do wrong?
the following are the codes i have:
////////////////////////// test12.cs
using System;
using System.IO;
using System.Net;
class test12
{
static void Main(string[] args)
{
String address = "xxx login page with username and password xxx";
String newaddr = "page wanted to download";

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(address);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();

request = (HttpWebRequest) WebRequest.Create(newaddr);
// response headers info:
// Header Name:Server, Value :Sun-ONE-Web-Server/6.1
// Header Name:Date, Value :Thu, 01 Apr 2004 21:12:48 GMT
// Header Name:Content-type, Value :text/html
// Header Name:Set-cookie, Value :RMID=xxxxx; expires=Friday,
// 01-Apr-2005 21:12:48 GMT; path=/; domain=.xxx.com
// Header Name:Cache-control, Value :no-cache
// Header Name:Pragma, Value :no-cache
// Header Name:Transfer-encoding, Value :chunked
request.Headers.Add("Cookie", response.Headers["Set-Cookie"]);

Console.WriteLine("old Address: " + request.Address);
response = (HttpWebResponse) request.GetResponse();
Console.WriteLine("new Address: " + request.Address);
Console.WriteLine("ResponseUri " + response.ResponseUri);

StreamReader sr = new StreamReader(response.GetResponseStream());
String result = sr.ReadToEnd();
sr.Close();

response.Close();

Console.WriteLine(result.Length);
if (result.Length > 1000)
result = result.Substring(0, 1000);
Console.WriteLine(result);
}