web searching scripts

[julien.lord]

Does anyone know of a freely available script that can take a given URL
and follow every link within it?

Ideally, I would like to start with this to build a quick application
to grab all the content off a website to publish it to a CD.

Thanks,

jul

[Avell Diroll]

ju*********@gmail.com wrote:

Does anyone know of a freely available script that can take a given URL
and follow every link within it?

Ideally, I would like to start with this to build a quick application
to grab all the content off a website to publish it to a CD.

Thanks,

jul

If you just want to download websites (i.e. not necessarily writing a
program yourself to do that), you may try Httrack, it might suite your
needs.

http://www.httrack.com/

There even seem to be some sort of python bindings ...

http://www.satzbau-gmbh.de/staff/abel/httrack-py/

But there might be some more pythonic solution around ... i would start
looking at twisted or cherrypy, but i never used them myself ...

HIH

regards

Avell

[Jorgen Grahn]

On Fri, 04 Aug 2006 18:11:18 +0200, Avell Diroll <av*********@yahoo.frwrote:

ju*********@gmail.com wrote:

>Does anyone know of a freely available script that can take a given URL
and follow every link within it?

Ideally, I would like to start with this to build a quick application
to grab all the content off a website to publish it to a CD.

....

If you just want to download websites (i.e. not necessarily writing a
program yourself to do that), you may try Httrack, it might suite your
needs.

The well-known Gnu wget is what I always use.

(IMHO, this is a situation where it's a /good/ idea to glue together existing
software, rather than joining many bits of code to a mirror-from-http-to-cdr
application.)

/Jorgen

--
// Jorgen Grahn <grahn@ Ph'nglui mglw'nafh Cthulhu
\X/ snipabacken.dyndns.org R'lyeh wgah'nagl fhtagn!