custom user agent

[studio]

I have a simple crawler as part of a website. It uses file() to copy
HTML content from a web page on another website and reformat it into a
new HTML page using a template. Is there a simple way of having the
crawler supply a custom HTTP user agent header to identify itself to
the website being crawled? I don't want to alter the php.ini file, I
need to set the user agent value from PHP if possible.

Thanks

[Erwin Moller]

st****@stoxfx.com wrote:

I have a simple crawler as part of a website. It uses file() to copy
HTML content from a web page on another website and reformat it into a
new HTML page using a template. Is there a simple way of having the
crawler supply a custom HTTP user agent header to identify itself to
the website being crawled? I don't want to alter the php.ini file, I
need to set the user agent value from PHP if possible.

Thanks

Hi,

AFAIK: headers are added to the actual HTML page by the software that is
sending the HTML page.
For example: The webserver will add a few headers.

So I do not know how you can 'tag' your pages on the second server, so the
webserver will know it should send some extra headers.

You can however always add META-tags into the document itself, but I don't
know if that is what you need.

just my 2 cents..

Regards,
Erwin Moller

[Steve]

Is there a simple way of having the
crawler supply a custom HTTP user agent header to identify itself

ini_set( "user_agent", "MyCrappyCrawler/1.0\r\n");

---
Steve

[Philip Ronan]

"st****@stoxfx.com" wrote:

I have a simple crawler as part of a website. It uses file() to copy
HTML content from a web page on another website and reformat it into a
new HTML page using a template. Is there a simple way of having the
crawler supply a custom HTTP user agent header to identify itself to
the website being crawled? I don't want to alter the php.ini file, I
need to set the user agent value from PHP if possible.

Thanks

There may be safe-mode restrictions to prevent you doing this, but have a go
anyway:

ini_set("user_agent","Marvin the Paranoid Android");

If you're planning to crawl sites that don't belong to you, be nice and
check the robots.txt file first.

--
phil [dot] ronan @ virgin [dot] net
http://vzone.virgin.net/phil.ronan/

[studio]

We have permission to crawl this page and ini_set() works just fine.

Thanks all!