Web crawler on python

[sonich]

I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?

[Mr.SpOOn]

On Sun, Oct 26, 2008 at 9:54 PM, sonich <So************@gmail.comwrote:

I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?

What about BeautifulSoup?

http://www.crummy.com/software/BeautifulSoup/

[James Mills]

On Mon, Oct 27, 2008 at 6:54 AM, sonich <So************@gmail.comwrote:

I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?

Simple, but it works. Extend it all you like.

http://hg.softcircuit.com.au/index.w...ples/spider.py

$ spider.py --help
Usage: spider.py [options] <url>

Options:
--version show program's version number and exit
-h, --help show this help message and exit
-q, --quiet Enable quiet mode
-l, --links Get links for specified url only
-d DEPTH, --depth=DEPTH
Maximum depth to traverse

cheers
James

--
--
-- "Problems are solved by method"

[Support Desk]

-----Original Message-----
From: James Mills [mailto:pr******@shortcircuit.net.au]
Sent: Sunday, October 26, 2008 5:26 PM
To: sonich
Cc: py*********@python.org
Subject: Re: Web crawler on python

On Mon, Oct 27, 2008 at 6:54 AM, sonich <So************@gmail.comwrote:

I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?

Simple, but it works. Extend it all you like.

http://hg.softcircuit.com.au/index.w...330d047ff663/e
xamples/spider.py

$ spider.py --help
Usage: spider.py [options] <url>

Options:
--version show program's version number and exit
-h, --help show this help message and exit
-q, --quiet Enable quiet mode
-l, --links Get links for specified url only
-d DEPTH, --depth=DEPTH
Maximum depth to traverse

cheers
James

--
--
-- "Problems are solved by method"

[Alex]

On Oct 26, 9:54*pm, sonich <[e-mail]:

I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?

You should try Orchid http://pypi.python.org/pypi/Orchid/1.1
or you can have a look at my project on launchpad
https://code.launchpad.net/~esaurito...r/experimental.
It's a single site crawler but you can easily modified it.

Bye.

Alex