Spidering Hacks for Python, not Perl

[dananrg]

O'Reilly's Spidering Hacks books terrific. One problem. All the code
samples are in Perl. Nothing Pythonic. Is there a book out there for
Python which covers spidering / crawling in depth?

[Enigma Curry]

I've been looking for similar stuff recently. I haven't found much, but
this is the list of links I've come across so far:

Harvest Man - http://harvestman.freezope.org/
Mechanize - http://wwwsearch.sourceforge.net/mechanize/
Beautiful Soup - http://www.crummy.com/software/BeautifulSoup/
(Neither Beautiful Soup, nor Mechanize are complete crawlers but
probably have a lot of the nuts and bolts)

If anyone is aware of a book or other documentation like the OP would
like, I would be pleased to see it as well.

[Duncan Booth]

Enigma Curry wrote:

I've been looking for similar stuff recently. I haven't found much, but
this is the list of links I've come across so far:

Harvest Man - http://harvestman.freezope.org/
Mechanize - http://wwwsearch.sourceforge.net/mechanize/
Beautiful Soup - http://www.crummy.com/software/BeautifulSoup/
(Neither Beautiful Soup, nor Mechanize are complete crawlers but
probably have a lot of the nuts and bolts)

If anyone is aware of a book or other documentation like the OP would
like, I would be pleased to see it as well.

Don't forget webchecker and websucker in the Python distribution Tools
folder.

[gene tani]

Duncan Booth wrote:

Enigma Curry wrote:

I've been looking for similar stuff recently. I haven't found much, but
this is the list of links I've come across so far:

http://awaretek.com/nowak/mygale.html
http://cheeseshop.python.org/pypi/spider.py/0.5

http://sig.levillage.org/?p=599
http://www.robertblum.com/articles/2...n-web-scraping
http://www.rexx.com/~dkuhlman/quixote_htmlscraping.html

http://sig.levillage.org/?p=588

http://sourceforge.net/projects/goog-kongulo/

[gene tani]

gene tani wrote:

Duncan Booth wrote:

Enigma Curry wrote:

a couple more

http://cheeseshop.python.org/pypi/Orchid/1.0
http://cheeseshop.python.org/pypi/webstemmer/0.5.0

[John J. Lee]

da*****@yahoo.com writes:

O'Reilly's Spidering Hacks books terrific. One problem. All the code
samples are in Perl. Nothing Pythonic. Is there a book out there for
Python which covers spidering / crawling in depth?

A fair number of the examples in that book use WWW::Mechanize. I
ported that to Python (it has evolved a bit since then):

http://wwwsearch.sf.net/mechanize
The distribution includes an example from the book ported to Python,
in fact -- porting other examples should be fairly straightforward.

A stable release of mechanize is (finally!) coming soon(-ish).
John