allowing http redirects??

[Jason Manfield]

I am trying to crawl the web using the following code snippet.

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Win32)";

....
By default, the req.AllowAutoRedirect is true and
MaximumAutomaticRedirections is 50.
When I try to crawl the following URL.

http://citeseer.ist.psu.edu/rd/55811...6efficient.pdf

I get NameResolutionFailure exception. However, I am able to open this URL
from the browser and it gets redirected to:

http://citeseer.ist.psu.edu/cache/pa...6efficient.pdf
How do I force my C# code to go to the redirected url?

[Joerg Jooss]

Jason Manfield wrote:

I am trying to crawl the web using the following code snippet.

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Win32)";

...
By default, the req.AllowAutoRedirect is true and
MaximumAutomaticRedirections is 50.
When I try to crawl the following URL.

http://citeseer.ist.psu.edu/rd/55811...oad/http://cit
eseer.ist.psu.edu/cache/papers/cs/7145/http:zSzzSzwww.stanford.eduzSzc
lasszSzcs343zSzpszSzpathprof.pdf/ball96efficient.pdf

I get NameResolutionFailure exception. However, I am able to open
this URL from the browser and it gets redirected to:

http://citeseer.ist.psu.edu/cache/pa...zzSzwww.stanfo
rd.eduzSzclasszSzcs343zSzpszSzpathprof.pdf/ball96efficient.pdf
How do I force my C# code to go to the redirected url?

What .NET version do you use? I can get this document woth no problem:

HTTP/1.1 200 OK
Date: Wed, 01 Jun 2005 20:52:37 GMT
Server: Apache/2.0.53 (Unix)
Last-Modified: Wed, 08 Nov 2000 19:17:28 GMT
ETag: "17801be-21d36-c5214200"
Accept-Ranges: bytes
Content-Length: 138550
Connection: close
Content-Type: application/pdf

Cheers,
--
http://www.joergjooss.de
mailto:ne********@joergjooss.de

[Jason Manfield]

Joerg

My .NET version is 2.0.4.

I assume it worked for you from C# code. Did you set any property in
HttpWebRequest to make it work?

My code can open the redirected URL, but not the original url (with rd in it).

Jason

"Joerg Jooss" wrote:

Jason Manfield wrote:

I am trying to crawl the web using the following code snippet.

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Win32)";

...
By default, the req.AllowAutoRedirect is true and
MaximumAutomaticRedirections is 50.
When I try to crawl the following URL.

http://citeseer.ist.psu.edu/rd/55811...oad/http://cit
eseer.ist.psu.edu/cache/papers/cs/7145/http:zSzzSzwww.stanford.eduzSzc
lasszSzcs343zSzpszSzpathprof.pdf/ball96efficient.pdf

I get NameResolutionFailure exception. However, I am able to open
this URL from the browser and it gets redirected to:

http://citeseer.ist.psu.edu/cache/pa...zzSzwww.stanfo
rd.eduzSzclasszSzcs343zSzpszSzpathprof.pdf/ball96efficient.pdf
How do I force my C# code to go to the redirected url?

What .NET version do you use? I can get this document woth no problem:

HTTP/1.1 200 OK
Date: Wed, 01 Jun 2005 20:52:37 GMT
Server: Apache/2.0.53 (Unix)
Last-Modified: Wed, 08 Nov 2000 19:17:28 GMT
ETag: "17801be-21d36-c5214200"
Accept-Ranges: bytes
Content-Length: 138550
Connection: close
Content-Type: application/pdf

Cheers,
--
http://www.joergjooss.de
mailto:ne********@joergjooss.de

[Joerg Jooss]

Jason Manfield wrote:

Joerg

My .NET version is 2.0.4.

I assume it worked for you from C# code. Did you set any property in
HttpWebRequest to make it work?

Nothing special, but I tried it using .NET 1.1 SP1. It doesn't work for
me in .NET 2.0 Beta 2 either.
Cheers,
--
http://www.joergjooss.de
mailto:ne********@joergjooss.de