Lynx and wget - redirect problems for PHP scripts

[Martin]

I have a PHP script that should be run a few times a day. Currently I run
this script manually as I haven't been able to set up Lynx or wget to run it
as a cronjob. Or to be more specific: the script works, but both wget and
Lynx fail after 10 or 20 redirects, then it warns:

wget: 20 redirections exceeded.
Lynx: Alert!: Redirection limit of 10 URL's reached.

What the scripts does is download various HTML pages (allowed by the
owners) - and some of them have 20-100 pages. The script loops through the
pages and for each new does a redirect. After 10 or 20 it is stopped, due to
Lynx and wget.

Any idea how to fix this ? Can the 10 or 20 limits be modified ?

Martin.

[Jochen Daum]

Hi Martin!

On Wed, 10 Sep 2003 22:31:22 +0200, "Martin" <ma****@home.se> wrote:

I have a PHP script that should be run a few times a day. Currently I run
this script manually as I haven't been able to set up Lynx or wget to run it
as a cronjob. Or to be more specific: the script works, but both wget and
Lynx fail after 10 or 20 redirects, then it warns:

wget: 20 redirections exceeded.
Lynx: Alert!: Redirection limit of 10 URL's reached.

What the scripts does is download various HTML pages (allowed by the
owners) - and some of them have 20-100 pages. The script loops through the
pages and for each new does a redirect. After 10 or 20 it is stopped, due to
Lynx and wget.

Any idea how to fix this ? Can the 10 or 20 limits be modified ?

Maybe ask this question in comp.os.linux.apps, but maybe you can
modify your script, so that it keeps a database table of pages to
download. Then call it every minute and stop it after 55 seconds. This
way, you don't need redirects and its also more reliable towards slow
connection from the sites you download from.

HTH, Jochen
--
PHP DB Edit Toolkit -- PHP scripts for building
database editing interfaces.
http://sourceforge.net/projects/phpdbedittk/

[Nikolai Chuvakhin]

"Martin" <ma****@home.se> wrote in message
news:<yS*****************@news2.bredband.com>...

wget: 20 redirections exceeded.

Any idea how to fix this ? Can the 10 or 20 limits be modified ?

wget seems to have this limit hard-coded in retr.c:

/* Maximum number of allowed redirections. 20 was chosen as a
"reasonable" value, which is low enough to not cause havoc, yet
high enough to guarantee that normal retrievals will not be hurt by
the check. */

#define MAX_REDIRECTIONS 20

Cheers,
NC

[Martin]

Thanks guys - appreciate your help.

Martin.

"Martin" <ma****@home.se> wrote in message
news:yS*****************@news2.bredband.com...

I have a PHP script that should be run a few times a day. Currently I run
this script manually as I haven't been able to set up Lynx or wget to run it as a cronjob. Or to be more specific: the script works, but both wget and
Lynx fail after 10 or 20 redirects, then it warns:

wget: 20 redirections exceeded.
Lynx: Alert!: Redirection limit of 10 URL's reached.

What the scripts does is download various HTML pages (allowed by the
owners) - and some of them have 20-100 pages. The script loops through the
pages and for each new does a redirect. After 10 or 20 it is stopped, due to Lynx and wget.

Any idea how to fix this ? Can the 10 or 20 limits be modified ?

Martin.