How to download a web page just like a web browser do ?

[Bo Yang]

Hi ,
It is about one month passed since I post to this list
last time . Yes , I use python , I used it in every day normal
work , whenever I need to do some scripts or other little-scale
works , python is the first one I took consideration in . I must
say it is a powerful tool for me , and what is more important
is there is a friendly and flourish community here .
Oh , I must stop appriciation and go to my question now .

Everyday , I receive a newsletter from NYTimes , but I
didn't want to read the news in evening the time the letter
came in . So , I am about to download the web page
contains the news and read them next morning ! I decide to
use python to write a tool , which should be feeded with a
URL , and then download the page to my disk . This
function just like the Browser's "save as..." function . I
know what shoud I do to accomplish that , I need to parse
the web page , and download all pages in the page , and
modify all the links to conrespond local disk links and ...

So , is there any similar function any one have impelment?
Does anyone can share some code with me ? I really don't
want to some confusing code to process such as text findings
and substitutions .

Thanks in advance !
Bo

[Marc 'BlackJack' Rintsch]

In <11**********************@p79g2000cwp.googlegroups .com>, Bo Yang wrote:

Everyday , I receive a newsletter from NYTimes , but I
didn't want to read the news in evening the time the letter
came in . So , I am about to download the web page
contains the news and read them next morning ! I decide to
use python to write a tool , which should be feeded with a
URL , and then download the page to my disk . This
function just like the Browser's "save as..." function . I
know what shoud I do to accomplish that , I need to parse
the web page , and download all pages in the page , and
modify all the links to conrespond local disk links and ...

Why don't you just use the `wget` program. Not written in Python but much
easier to use instead of writing the functionality yourself.

Ciao,
Marc 'BlackJack' Rintsch

[Gabriel Genellina]

At Wednesday 23/8/2006 11:34, Bo Yang wrote:

Everyday , I receive a newsletter from NYTimes , but I
didn't want to read the news in evening the time the letter
came in . So , I am about to download the web page
contains the news and read them next morning ! I decide to
use python to write a tool , which should be feeded with a
URL , and then download the page to my disk . This
function just like the Browser's "save as..." function . I
know what shoud I do to accomplish that , I need to parse
the web page , and download all pages in the page , and
modify all the links to conrespond local disk links and ...

This tool already exists: wget

Gabriel Genellina
Softlab SRL

__________________________________________________
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas

[Max Penet]

You can also try HarvestMan:

http://harvestman.freezope.org/

Bo Yang wrote:

Hi ,
It is about one month passed since I post to this list
last time . Yes , I use python , I used it in every day normal
work , whenever I need to do some scripts or other little-scale
works , python is the first one I took consideration in . I must
say it is a powerful tool for me , and what is more important
is there is a friendly and flourish community here .
Oh , I must stop appriciation and go to my question now .

Everyday , I receive a newsletter from NYTimes , but I
didn't want to read the news in evening the time the letter
came in . So , I am about to download the web page
contains the news and read them next morning ! I decide to
use python to write a tool , which should be feeded with a
URL , and then download the page to my disk . This
function just like the Browser's "save as..." function . I
know what shoud I do to accomplish that , I need to parse
the web page , and download all pages in the page , and
modify all the links to conrespond local disk links and ...

So , is there any similar function any one have impelment?
Does anyone can share some code with me ? I really don't
want to some confusing code to process such as text findings
and substitutions .

Thanks in advance !
Bo

[Bo Yang]

Thank you , Max !
I think HarvestMan is just what I need !
Thanks again !

[BJörn Lindqvist]

Mechanize (http://wwwsearch.sourceforge.net/mechanize/) is another
option, it can even fill out forms!

--
mvh Björn