WebClient - detect response redirect

[Glen]

I'm writing a console utility to download specific files from web sites
based on the command line options. In most cases, I can trap the 404
error when the file isn't available because the operator mistyped the
URL or it's offline for whatever reason. The problem I'm running into
is with certain sites where the admin has set up a redirect to handle
the 404 condition and redirects the request to another page.

In this case, the redirected page gets downloaded and saved which is not
the desired result. This utility is being used in a scheduling process
to download and process specific data files from public web sites and
the files must exist, or return an error to halt processing.

Is there anyway to tell the WebClient or WebRequest objects to not allow
redirected content? Or, is there a property in either of the objects
that reflects the actual URL of the source when it's redirected? Or, is
there an alternative object that can be used for this purpose to
download files from web sites?

Any help would be greatly appreciated.

- Glen

[Adam Clauss]

I don't think there is a way using the WebClient class.

If you are using HttpWebRequest, there is the AllowAutoRedirect property
which may allow you to accomplish what you need.

--
Adam Clauss
ca*****@tamu.edu
"Glen" <Bu****@hotmail.com> wrote in message
news:u4****************@TK2MSFTNGP14.phx.gbl...

I'm writing a console utility to download specific files from web sites
based on the command line options. In most cases, I can trap the 404
error when the file isn't available because the operator mistyped the URL
or it's offline for whatever reason. The problem I'm running into is with
certain sites where the admin has set up a redirect to handle the 404
condition and redirects the request to another page.

In this case, the redirected page gets downloaded and saved which is not
the desired result. This utility is being used in a scheduling process to
download and process specific data files from public web sites and the
files must exist, or return an error to halt processing.

Is there anyway to tell the WebClient or WebRequest objects to not allow
redirected content? Or, is there a property in either of the objects that
reflects the actual URL of the source when it's redirected? Or, is there
an alternative object that can be used for this purpose to download files
from web sites?

Any help would be greatly appreciated.

- Glen

[Cor Ligthert]

Glen,

I know a lot of sites that have an URL only as a start point and than
redirect, how do you handle those when you have your implementation.

Not that I have the answer however maybe is finding it opening a new problem
and is searching for this solution useless..

Just my thought,

Cor

[Glen]

Hi Cor,

Since I'm not building a browser and I'm looking to retrieve very specific
data from the web, I expect that any changes in the remote data source will
be documented and handled by the system operators. Again, this is a critical
task, much like getting files via FTP. I don't expect that when I log into
an FTP site, I'm going to get a different file than the one I requested.

In any case, thank you for your input.

- Glen

"Cor Ligthert" <no************@planet.nl> wrote in message
news:OJ****************@TK2MSFTNGP12.phx.gbl...

Glen,

I know a lot of sites that have an URL only as a start point and than
redirect, how do you handle those when you have your implementation.

Not that I have the answer however maybe is finding it opening a new problem and is searching for this solution useless..

Just my thought,

Cor

[Glen]

Thanks Adam. I'll give that one a look.

- Glen

"Adam Clauss" <ca*****@nospam.tamu.edu> wrote in message
news:OM****************@TK2MSFTNGP10.phx.gbl...

I don't think there is a way using the WebClient class.

If you are using HttpWebRequest, there is the AllowAutoRedirect property
which may allow you to accomplish what you need.

--
Adam Clauss
ca*****@tamu.edu
"Glen" <Bu****@hotmail.com> wrote in message
news:u4****************@TK2MSFTNGP14.phx.gbl...

I'm writing a console utility to download specific files from web sites
based on the command line options. In most cases, I can trap the 404
error when the file isn't available because the operator mistyped the URL or it's offline for whatever reason. The problem I'm running into is with certain sites where the admin has set up a redirect to handle the 404
condition and redirects the request to another page.

In this case, the redirected page gets downloaded and saved which is not
the desired result. This utility is being used in a scheduling process to download and process specific data files from public web sites and the
files must exist, or return an error to halt processing.

Is there anyway to tell the WebClient or WebRequest objects to not allow
redirected content? Or, is there a property in either of the objects that reflects the actual URL of the source when it's redirected? Or, is there an alternative object that can be used for this purpose to download files from web sites?

Any help would be greatly appreciated.

- Glen

[Glen]

Unfortunately, this parameter doesn't appear to work in some cases. The
site in question is using IIS 5.0 and is not returning a redirect code or
page not found code in any instance. All redirects are done silently with
an OK status returned.
"Glen" <bu****@hotmail.com> wrote in message
news:%2******************@TK2MSFTNGP11.phx.gbl...

Thanks Adam. I'll give that one a look.

- Glen

"Adam Clauss" <ca*****@nospam.tamu.edu> wrote in message
news:OM****************@TK2MSFTNGP10.phx.gbl...

I don't think there is a way using the WebClient class.

If you are using HttpWebRequest, there is the AllowAutoRedirect property
which may allow you to accomplish what you need.

--
Adam Clauss
ca*****@tamu.edu
"Glen" <Bu****@hotmail.com> wrote in message
news:u4****************@TK2MSFTNGP14.phx.gbl...

I'm writing a console utility to download specific files from web sites based on the command line options. In most cases, I can trap the 404
error when the file isn't available because the operator mistyped the URL or it's offline for whatever reason. The problem I'm running into is with certain sites where the admin has set up a redirect to handle the 404
condition and redirects the request to another page.

In this case, the redirected page gets downloaded and saved which is not the desired result. This utility is being used in a scheduling process

to

download and process specific data files from public web sites and the
files must exist, or return an error to halt processing.

Is there anyway to tell the WebClient or WebRequest objects to not

allow redirected content? Or, is there a property in either of the objects

that reflects the actual URL of the source when it's redirected? Or, is there an alternative object that can be used for this purpose to download files from web sites?

Any help would be greatly appreciated.

- Glen

[Adam Clauss]

Hmm, wait, so the website is not telling the client to redirect anywhere?
In this case - there probably is no way to tell that anything was happening.
It sounds like the redirection is actually happening on the server side of
things...

--
Adam Clauss
ca*****@tamu.edu

"Glen" <bu****@hotmail.com> wrote in message
news:O$****************@TK2MSFTNGP10.phx.gbl...

Unfortunately, this parameter doesn't appear to work in some cases. The
site in question is using IIS 5.0 and is not returning a redirect code or
page not found code in any instance. All redirects are done silently with
an OK status returned.
"Glen" <bu****@hotmail.com> wrote in message
news:%2******************@TK2MSFTNGP11.phx.gbl...

Thanks Adam. I'll give that one a look.

- Glen

"Adam Clauss" <ca*****@nospam.tamu.edu> wrote in message
news:OM****************@TK2MSFTNGP10.phx.gbl...

> I don't think there is a way using the WebClient class.
>
> If you are using HttpWebRequest, there is the AllowAutoRedirect
> property
> which may allow you to accomplish what you need.
>
> --
> Adam Clauss
> ca*****@tamu.edu
> "Glen" <Bu****@hotmail.com> wrote in message
> news:u4****************@TK2MSFTNGP14.phx.gbl...
> > I'm writing a console utility to download specific files from web sites > > based on the command line options. In most cases, I can trap the 404
> > error when the file isn't available because the operator mistyped the

URL

> > or it's offline for whatever reason. The problem I'm running into is

with

> > certain sites where the admin has set up a redirect to handle the 404
> > condition and redirects the request to another page.
> >
> > In this case, the redirected page gets downloaded and saved which is not > > the desired result. This utility is being used in a scheduling process

to

> > download and process specific data files from public web sites and
> > the
> > files must exist, or return an error to halt processing.
> >
> > Is there anyway to tell the WebClient or WebRequest objects to not

allow > > redirected content? Or, is there a property in either of the objects

that

> > reflects the actual URL of the source when it's redirected? Or, is

there

> > an alternative object that can be used for this purpose to download

files

> > from web sites?
> >
> > Any help would be greatly appreciated.
> >
> > - Glen
>
>

[Glen]

Hi Adam,

I checked a few things and even in the browser (IE) when you enter and
incorrect path on this particular server it doesn't change the actual
address when it displays the help text. I guess it's more of a content
substitution than a redirect scenario on the server side to handle 404
errors.

I'm going to have to figure out another way to validate the download and
return the appropriate codes to the application.

Thanks for your help.

- Glen

"Adam Clauss" <ca*****@nospam.tamu.edu> wrote in message
news:eh****************@TK2MSFTNGP09.phx.gbl...

Hmm, wait, so the website is not telling the client to redirect anywhere?
In this case - there probably is no way to tell that anything was happening. It sounds like the redirection is actually happening on the server side of
things...

--
Adam Clauss
ca*****@tamu.edu

"Glen" <bu****@hotmail.com> wrote in message
news:O$****************@TK2MSFTNGP10.phx.gbl...

Unfortunately, this parameter doesn't appear to work in some cases. The
site in question is using IIS 5.0 and is not returning a redirect code or page not found code in any instance. All redirects are done silently with an OK status returned.
"Glen" <bu****@hotmail.com> wrote in message
news:%2******************@TK2MSFTNGP11.phx.gbl...

Thanks Adam. I'll give that one a look.

- Glen

"Adam Clauss" <ca*****@nospam.tamu.edu> wrote in message
news:OM****************@TK2MSFTNGP10.phx.gbl...
> I don't think there is a way using the WebClient class.
>
> If you are using HttpWebRequest, there is the AllowAutoRedirect
> property
> which may allow you to accomplish what you need.
>
> --
> Adam Clauss
> ca*****@tamu.edu
> "Glen" <Bu****@hotmail.com> wrote in message
> news:u4****************@TK2MSFTNGP14.phx.gbl...
> > I'm writing a console utility to download specific files from web

sites

> > based on the command line options. In most cases, I can trap the 404 > > error when the file isn't available because the operator mistyped the URL
> > or it's offline for whatever reason. The problem I'm running into is with
> > certain sites where the admin has set up a redirect to handle the 404 > > condition and redirects the request to another page.
> >
> > In this case, the redirected page gets downloaded and saved which is

not

> > the desired result. This utility is being used in a scheduling

process

to
> > download and process specific data files from public web sites and
> > the
> > files must exist, or return an error to halt processing.
> >
> > Is there anyway to tell the WebClient or WebRequest objects to not

allow

> > redirected content? Or, is there a property in either of the

objects that
> > reflects the actual URL of the source when it's redirected? Or, is
there
> > an alternative object that can be used for this purpose to download
files
> > from web sites?
> >
> > Any help would be greatly appreciated.
> >
> > - Glen
>
>

[Adam Clauss]

NP... good luck on that.

Do you know what type of file you are EXPECTING to retrieve? Could you
check the actual content you downloaded to see if it is correct (or some
sort of "error" page?)?

--
Adam Clauss
ca*****@tamu.edu
"Glen" <bu****@hotmail.com> wrote in message
news:%2******************@TK2MSFTNGP12.phx.gbl...

Hi Adam,

I checked a few things and even in the browser (IE) when you enter and
incorrect path on this particular server it doesn't change the actual
address when it displays the help text. I guess it's more of a content
substitution than a redirect scenario on the server side to handle 404
errors.

I'm going to have to figure out another way to validate the download and
return the appropriate codes to the application.

Thanks for your help.

- Glen

"Adam Clauss" <ca*****@nospam.tamu.edu> wrote in message
news:eh****************@TK2MSFTNGP09.phx.gbl...

Hmm, wait, so the website is not telling the client to redirect anywhere?
In this case - there probably is no way to tell that anything was

happening.

It sounds like the redirection is actually happening on the server side
of
things...

--
Adam Clauss
ca*****@tamu.edu

"Glen" <bu****@hotmail.com> wrote in message
news:O$****************@TK2MSFTNGP10.phx.gbl...

> Unfortunately, this parameter doesn't appear to work in some cases.
> The
> site in question is using IIS 5.0 and is not returning a redirect code or > page not found code in any instance. All redirects are done silently with > an OK status returned.
>
>
> "Glen" <bu****@hotmail.com> wrote in message
> news:%2******************@TK2MSFTNGP11.phx.gbl...
>> Thanks Adam. I'll give that one a look.
>>
>> - Glen
>>
>> "Adam Clauss" <ca*****@nospam.tamu.edu> wrote in message
>> news:OM****************@TK2MSFTNGP10.phx.gbl...
>> > I don't think there is a way using the WebClient class.
>> >
>> > If you are using HttpWebRequest, there is the AllowAutoRedirect
>> > property
>> > which may allow you to accomplish what you need.
>> >
>> > --
>> > Adam Clauss
>> > ca*****@tamu.edu
>> > "Glen" <Bu****@hotmail.com> wrote in message
>> > news:u4****************@TK2MSFTNGP14.phx.gbl...
>> > > I'm writing a console utility to download specific files from web
> sites
>> > > based on the command line options. In most cases, I can trap the 404 >> > > error when the file isn't available because the operator mistyped the >> URL
>> > > or it's offline for whatever reason. The problem I'm running into is >> with
>> > > certain sites where the admin has set up a redirect to handle the 404 >> > > condition and redirects the request to another page.
>> > >
>> > > In this case, the redirected page gets downloaded and saved which is > not
>> > > the desired result. This utility is being used in a scheduling
> process
>> to
>> > > download and process specific data files from public web sites and
>> > > the
>> > > files must exist, or return an error to halt processing.
>> > >
>> > > Is there anyway to tell the WebClient or WebRequest objects to not
> allow
>> > > redirected content? Or, is there a property in either of the objects >> that
>> > > reflects the actual URL of the source when it's redirected? Or,
>> > > is
>> there
>> > > an alternative object that can be used for this purpose to
>> > > download
>> files
>> > > from web sites?
>> > >
>> > > Any help would be greatly appreciated.
>> > >
>> > > - Glen
>> >
>> >
>>
>>
>
>

[Glen]

Yes, these particular files are regulated (pipe delimited) with constant
headers. I'll probably just build a simple script to open the file and
verify the header in a seperate step when the transfer is completed. At
least this way, I can keep my C# project generic enough to use with other
processes.

- Glen

"Adam Clauss" <ca*****@nospam.tamu.edu> wrote in message
news:ue****************@TK2MSFTNGP14.phx.gbl...

NP... good luck on that.

Do you know what type of file you are EXPECTING to retrieve? Could you
check the actual content you downloaded to see if it is correct (or some
sort of "error" page?)?

--
Adam Clauss
ca*****@tamu.edu
"Glen" <bu****@hotmail.com> wrote in message
news:%2******************@TK2MSFTNGP12.phx.gbl...

Hi Adam,

I checked a few things and even in the browser (IE) when you enter and
incorrect path on this particular server it doesn't change the actual
address when it displays the help text. I guess it's more of a content
substitution than a redirect scenario on the server side to handle 404
errors.

I'm going to have to figure out another way to validate the download and
return the appropriate codes to the application.

Thanks for your help.

- Glen

"Adam Clauss" <ca*****@nospam.tamu.edu> wrote in message
news:eh****************@TK2MSFTNGP09.phx.gbl...

Hmm, wait, so the website is not telling the client to redirect anywhere? In this case - there probably is no way to tell that anything was

happening.

It sounds like the redirection is actually happening on the server side
of
things...

--
Adam Clauss
ca*****@tamu.edu

"Glen" <bu****@hotmail.com> wrote in message
news:O$****************@TK2MSFTNGP10.phx.gbl...
> Unfortunately, this parameter doesn't appear to work in some cases.
> The
> site in question is using IIS 5.0 and is not returning a redirect code

or

> page not found code in any instance. All redirects are done silently

with

> an OK status returned.
>
>
> "Glen" <bu****@hotmail.com> wrote in message
> news:%2******************@TK2MSFTNGP11.phx.gbl...
>> Thanks Adam. I'll give that one a look.
>>
>> - Glen
>>
>> "Adam Clauss" <ca*****@nospam.tamu.edu> wrote in message
>> news:OM****************@TK2MSFTNGP10.phx.gbl...
>> > I don't think there is a way using the WebClient class.
>> >
>> > If you are using HttpWebRequest, there is the AllowAutoRedirect
>> > property
>> > which may allow you to accomplish what you need.
>> >
>> > --
>> > Adam Clauss
>> > ca*****@tamu.edu
>> > "Glen" <Bu****@hotmail.com> wrote in message
>> > news:u4****************@TK2MSFTNGP14.phx.gbl...
>> > > I'm writing a console utility to download specific files from

web > sites
>> > > based on the command line options. In most cases, I can trap the 404

>> > > error when the file isn't available because the operator

mistyped the

>> URL
>> > > or it's offline for whatever reason. The problem I'm running

into is

>> with
>> > > certain sites where the admin has set up a redirect to handle

the 404

>> > > condition and redirects the request to another page.
>> > >
>> > > In this case, the redirected page gets downloaded and saved

which is

> not
>> > > the desired result. This utility is being used in a scheduling
> process
>> to
>> > > download and process specific data files from public web sites

and >> > > the
>> > > files must exist, or return an error to halt processing.
>> > >
>> > > Is there anyway to tell the WebClient or WebRequest objects to not > allow
>> > > redirected content? Or, is there a property in either of the

objects

>> that
>> > > reflects the actual URL of the source when it's redirected? Or,
>> > > is
>> there
>> > > an alternative object that can be used for this purpose to
>> > > download
>> files
>> > > from web sites?
>> > >
>> > > Any help would be greatly appreciated.
>> > >
>> > > - Glen
>> >
>> >
>>
>>
>
>