By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
432,474 Members | 966 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 432,474 IT Pros & Developers. It's quick & easy.

Web crawler on python

P: n/a
I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?
Oct 26 '08 #1
Share this Question
Share on Google+
4 Replies


P: n/a
On Sun, Oct 26, 2008 at 9:54 PM, sonich <So************@gmail.comwrote:
I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?
What about BeautifulSoup?

http://www.crummy.com/software/BeautifulSoup/
Oct 26 '08 #2

P: n/a
On Mon, Oct 27, 2008 at 6:54 AM, sonich <So************@gmail.comwrote:
I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?
Simple, but it works. Extend it all you like.

http://hg.softcircuit.com.au/index.w...ples/spider.py

$ spider.py --help
Usage: spider.py [options] <url>

Options:
--version show program's version number and exit
-h, --help show this help message and exit
-q, --quiet Enable quiet mode
-l, --links Get links for specified url only
-d DEPTH, --depth=DEPTH
Maximum depth to traverse

cheers
James

--
--
-- "Problems are solved by method"
Oct 26 '08 #3

P: n/a


-----Original Message-----
From: James Mills [mailto:pr******@shortcircuit.net.au]
Sent: Sunday, October 26, 2008 5:26 PM
To: sonich
Cc: py*********@python.org
Subject: Re: Web crawler on python

On Mon, Oct 27, 2008 at 6:54 AM, sonich <So************@gmail.comwrote:
I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?
Simple, but it works. Extend it all you like.

http://hg.softcircuit.com.au/index.w...330d047ff663/e
xamples/spider.py

$ spider.py --help
Usage: spider.py [options] <url>

Options:
--version show program's version number and exit
-h, --help show this help message and exit
-q, --quiet Enable quiet mode
-l, --links Get links for specified url only
-d DEPTH, --depth=DEPTH
Maximum depth to traverse

cheers
James

--
--
-- "Problems are solved by method"
Oct 27 '08 #4

P: n/a
On Oct 26, 9:54*pm, sonich <SoniaTerpug...@gmail.comwrote:
I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?
You should try Orchid http://pypi.python.org/pypi/Orchid/1.1
or you can have a look at my project on launchpad
https://code.launchpad.net/~esaurito...r/experimental.
It's a single site crawler but you can easily modified it.

Bye.

Alex
Oct 28 '08 #5

This discussion thread is closed

Replies have been disabled for this discussion.