How's python's web scraping capabilities (vs LWP) ...

[ArKane]

Hello all,

I've been hacking away at perl for a few months now, mainly using the
LWP module, used for web scraping. Amoung its capabilities include
support for HTTPS and proxies, authentication, cookies (including the
ability to automatically import Internet Explorer cookies), etc.

It seems to me, however, that trying to write more sophisticated apps
using GUIs with perl is somewhat akin to pulling teeth, so I've been
looking at python, but was wondering if there was an LWP equivalent
for python?

[Enigma Curry]

I don't know much about LWP.. but Beautiful Soup is grand!

http://www.crummy.com/software/BeautifulSoup/

Ryan McGuire

[Kent Johnson]

ArKane wrote:

Hello all,

I've been hacking away at perl for a few months now, mainly using the
LWP module, used for web scraping. Amoung its capabilities include
support for HTTPS and proxies, authentication, cookies (including the
ability to automatically import Internet Explorer cookies), etc.

urllib2 (in the standard library) will do most of this and gets you the
HTML from a site. To pull data out of the HTML try BeautifulSoup.

Kent

[Grig Gheorghiu]

Check out twill <http://www.idyll.org/~t/www-tools/twill/>, which is
based on mechanize (<http://wwwsearch.sourceforge.net/>).

Grig