web crawler in python or C?

[abhinav]

Hi guys.I have to implement a topical crawler as a part of my
project.What language should i implement
C or Python?Python though has fast development cycle but my concern is
speed also.I want to strke a balance between development speed and
crawler speed.Since Python is an interpreted language it is rather
slow.The crawler which will be working on huge set of pages should be
as fast as possible.One possible implementation would be implementing
partly in C and partly in Python so that i can have best of both
worlds.But i don't know to approach about it.Can anyone guide me on
what part should i implement in C and what should be in Python?

[Paul Rubin]

"abhinav" <ab***********@ gmail.com> writes:

The crawler which will be working on huge set of pages should be
as fast as possible.

What kind of network connection do you have, that's fast enough
that even a fairly cpu-inefficient crawler won't saturate it?

[abhinav]

It is DSL broadband 128kbps.But thats not the point.What i am saying is
that would python be fine for implementing fast crawler algorithms or
should i use C.Handling huge data,multithrea ding,file
handling,heuris tics for ranking,and maintaining huge data
structures.What should be the language so as not to compromise that
much on speed.What is the performance of python based crawlers vs C
based crawlers.Should I use both the languages(partl y C and python).How
should i decide what part to be implemented in C and what should be
done in python?
Please guide me.Thanks.

[Fuzzyman]

abhinav wrote:

It is DSL broadband 128kbps.But thats not the point.What i am saying is
that would python be fine for implementing fast crawler algorithms or
should i use C.
But a web crawler is going to be *mainly* I/O bound - so language
efficiency won't be the main issue. There are several web crawler
implemented in Python.
Handling huge data,multithrea ding,file
handling,heuris tics for ranking,and maintaining huge data
structures.What should be the language so as not to compromise that
much on speed.What is the performance of python based crawlers vs C
based crawlers.Should I use both the languages(partl y C and python).How
If your data processing requirements are fairly heavy you will
*probably* get a speed advantage coding them in C and accessing them
from Python.

The usdual advice (which seems to be applicable to you), is to
prototype in Python (which will be much more fun than in C) then test.

Profile to find your real bottlenecks (if the Python one isn't fast
enough - which it may be), and move your bottlenecks to C.

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml
should i decide what part to be implemented in C and what should be
done in python?
Please guide me.Thanks.

[Paul Rubin]

"abhinav" <ab***********@ gmail.com> writes:

It is DSL broadband 128kbps.But thats not the point.
But it is the point.
What i am saying is that would python be fine for implementing fast
crawler algorithms or should i use C.Handling huge
data,multithrea ding,file handling,heuris tics for ranking,and
maintaining huge data structures.What should be the language so as
not to compromise that much on speed.What is the performance of
python based crawlers vs C based crawlers.Should I use both the
languages(partl y C and python).How should i decide what part to be
implemented in C and what should be done in python? Please guide
me.Thanks.

I think if you don't know how to answer these questions for yourself,
you're not ready to take on projects of that complexity. My advice
is start in Python since development will be much easier. If and when
you start hitting performance problems, you'll have to examine many
combinations of tactics for dealing with them, and switching languages
is just one such tactic.

[gene tani]

Paul Rubin wrote:

"abhinav" <ab***********@ gmail.com> writes:

maintaining huge data structures.What should be the language so as
not to compromise that much on speed.What is the performance of
python based crawlers vs C based crawlers.Should I use both the
languages(partl y C and python).How should i decide what part to be
implemented in C and what should be done in python? Please guide
me.Thanks.

I think if you don't know how to answer these questions for yourself,
you're not ready to take on projects of that complexity. My advice
is start in Python since development will be much easier. If and when
you start hitting performance problems, you'll have to examine many
combinations of tactics for dealing with them, and switching languages
is just one such tactic.

There's another potential bottleneck, parsing HTML and extracting the
text you want, especially when you hit pages that don't meet HTML 4 or
XHTML spec.
http://sig.levillage.org/?p=599

Paul's advice is very sound, given what little info you've provided.

http://trific.ath.cx/resources/python/optimization/
(and look at psyco, pyrex, boost, Swig, Ctypes for bridging C and
python, you have a lot of options. Also look at Harvestman, mechanize,
other existing libs.

[gene tani]

abhinav wrote:

Hi guys.I have to implement a topical crawler as a part of my
project.What language should i implement

Oh, and there's some really good books out there, besides the Orilly
Spidering Hacks. Springer Verlag has a couple books on "Text Mining"
and at least a couple books with "web intelligence" in the title.
Expensive but worth it.

[Andrew Gwozdziewycz]

On 15 Feb 2006 21:56:52 -0800, abhinav <ab***********@ gmail.com> wrote:

Hi guys.I have to implement a topical crawler as a part of my
project.What language should i implement
C or Python?

Why does this keep coming up on here as of late? If you search the
archives, you can find numerous posts about spiders. One interesting
fact is that google itself starting with their spiders in python.
http://www-db.stanford.edu/~backrub/google.html I'm _sure_ it'll work
for you.

--
Andrew Gwozdziewycz <ap****@gmail.c om>
http://ihadagreatview.org
http://plasticandroid.org

[Steven D'Aprano]

On Wed, 15 Feb 2006 21:56:52 -0800, abhinav wrote:

Hi guys.I have to implement a topical crawler as a part of my
project.What language should i implement
C or Python?Python though has fast development cycle but my concern is
speed also.I want to strke a balance between development speed and
crawler speed.Since Python is an interpreted language it is rather
slow.
Python is no more interpreted than Java. Like Java, it is compiled to
byte-code. Unlike Java, it doesn't take three weeks to start the runtime
environment. (Okay, maybe it just *seems* like three weeks.)

The nice clean distinctions between "compiled" and "interprete d" languages
haven't existed in most serious programming languages for a decade or
more. In these days of tokenizers and byte-code compilers and processors
emulating other processors, the difference is more of degree than kind.

It is true that standard Python doesn't compile to platform dependent
machine code, but that is rarely an issue since the bottleneck for most
applications is I/O or human interaction, not language speed. And for
those cases where it is a problem, there are solutions, like Psycho.

After all, it is almost never true that your code must run as fast as
physically possible. That's called "over-engineering". It just needs to
run as fast as needed, that's all. And that's a much simpler problem to
solve cheaply.
The crawler which will be working on huge set of pages should be
as fast as possible.
Web crawler performance is almost certainly going to be I/O bound. Sounds
to me like you are guilty of trying to optimize your code before even
writing a single line of code. What you call "huge" may not be huge to
your computer. Have you tried? The great thing about Python is you can
write a prototype in maybe a tenth the time it would take you to do the
same thing in C. Instead of trying to guess what the performance
bottlenecks will be, you can write your code and profile it and find the
bottlenecks with accuracy.

One possible implementation would be implementing
partly in C and partly in Python so that i can have best of both
worlds.
Sure you can do that, if you need to.
But i don't know to approach about it.Can anyone guide me on
what part should i implement in C and what should be in Python?

Yes. Write it all in Python. Test it, debug it, get it working.

Once it is working, and not before, rigorously profile it. You may find it
is fast enough.

If it is not fast enough, find the bottlenecks. Replace them with better
algorithms. We had an example on comp.lang.pytho n just a day or two ago
where a function which was taking hours to complete was re-written with a
better algorithm which took only seconds. And still in Python.

If it is still too slow after using better algorithms, or if there are no
better algorithms, then and only then re-write those bottlenecks in C for
speed.

--
Steven.

[Steve Holden]

abhinav wrote:

Hi guys.I have to implement a topical crawler as a part of my
project.What language should i implement
C or Python?Python though has fast development cycle but my concern is
speed also.I want to strke a balance between development speed and
crawler speed.Since Python is an interpreted language it is rather
slow.The crawler which will be working on huge set of pages should be
as fast as possible.One possible implementation would be implementing
partly in C and partly in Python so that i can have best of both
worlds.But i don't know to approach about it.Can anyone guide me on
what part should i implement in C and what should be in Python?

Get real. Any web crawler is bound to spend huge amounts of its time
waiting for data to come in over network pipes. Or do you have plans for
massive parallelism previously unheard of in the Python world?

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/