Python Search Engine app

Harlin Seritt

Hi,

Is anyone aware of an available open-source/free search engine app
(something similar to HTDig) written in Python that is out there?
Googling has turned up nothing. Thought maybe I'd mine some of you
guys' minds on this.

thanks,

Harlin Seritt
Internet Villa: www.seritt.org

Sep 14 '05 #1

Alan Meyer

"Harlin Seritt" <ha**********@yahoo.com> wrote in message
news:11**********************@g43g2000cwa.googlegr oups.com...

Hi,

Is anyone aware of an available open-source/free search engine app
(something similar to HTDig) written in Python that is out there?
Googling has turned up nothing. Thought maybe I'd mine some of you
guys' minds on this.

thanks,

Harlin Seritt
Internet Villa: www.seritt.org

I'm not aware of such a thing.

I stand ready to be corrected, but I think Python would not be a
good language for writing search engines. In the ones I've written
for custom projects - in C or PL/1, it has been necessary to
perform very high speed operations on highly compressed binary
structures - which is not Python's forte.

You might be able to put a Python interface over an engine written
in another language.

Alan

Sep 14 '05 #2

gene tani

Yes, there's a bunch. Google for "query parser" + python, "porter
stemming" "stopwords" "text indexer". Maybe lucene has some python
bindings, hmm?

Harlin Seritt wrote:

Hi,

Is anyone aware of an available open-source/free search engine app
(something similar to HTDig) written in Python that is out there?
Googling has turned up nothing. Thought maybe I'd mine some of you
guys' minds on this.

thanks,

Harlin Seritt
Internet Villa: www.seritt.org

Sep 14 '05 #3

Will McGugan

Alan Meyer wrote:

I'm not aware of such a thing.

I stand ready to be corrected, but I think Python would not be a
good language for writing search engines. In the ones I've written
for custom projects - in C or PL/1, it has been necessary to
perform very high speed operations on highly compressed binary
structures - which is not Python's forte.

You might be able to put a Python interface over an engine written
in another language.

Wasn't Google's first search engine actualy written in Python?
Will McGugan
--
http://www.willmcgugan.com
"".join({'*':'@','^':'.'}.get(c,0) or chr(97+(ord(c)-84)%26) for c in
"jvyy*jvyyzpthtna^pbz")

Sep 14 '05 #4

Fredrik Lundh

Harlin Seritt wrote:

Is anyone aware of an available open-source/free search engine app
(something similar to HTDig) written in Python that is out there?
Googling has turned up nothing. Thought maybe I'd mine some of you
guys' minds on this.

http://divmod.org/ has a couple of alternatives:

http://divmod.org/projects/lupy
http://divmod.org/projects/pyndex
http://divmod.org/projects/xapwrap

</F>

Sep 14 '05 #5

Kent Johnson

gene tani wrote:

Yes, there's a bunch. Google for "query parser" + python, "porter
stemming" "stopwords" "text indexer". Maybe lucene has some python
bindings, hmm?
At least two Python versions of Lucene:
http://pylucene.osafoundation.org/
http://divmod.org/projects/lupy

Kent

Harlin Seritt wrote:

Hi,

Is anyone aware of an available open-source/free search engine app
(something similar to HTDig) written in Python that is out there?
Googling has turned up nothing. Thought maybe I'd mine some of you
guys' minds on this.

thanks,

Harlin Seritt
Internet Villa: www.seritt.org

Sep 14 '05 #6

Alan Meyer

"Alan Meyer" <am*****@yahoo.com> wrote in message
news:BJ********************@comcast.com...
....

I stand ready to be corrected, but I think Python would not be a
good language for writing search engines. In the ones I've written
for custom projects - in C or PL/1, it has been necessary to
perform very high speed operations on highly compressed binary
structures - which is not Python's forte.

You might be able to put a Python interface over an engine written
in another language.

It looks like the engines cited by the other responders to the query
are indeed mostly Python wrappings for C++ engines.

One that might not be, "lupy", notes under performance in the FAQ
the simple statement that "Java is faster".

As between Java and C++, I like Java much more as a programmer
friendly programming language, but I think I'd still choose C++ for
a search engine if the number of users or the size of the data was
large.

Alan

Sep 14 '05 #7

Peter Hansen

Will McGugan wrote:

Alan Meyer wrote:

I stand ready to be corrected, but I think Python would not be a
good language for writing search engines.

Wasn't Google's first search engine actualy written in Python?

Not unless it was some kind of quickie prototype that was immediately
discarded in favour of an optimized C version once it was built. The
rumour you're probably referring to is the *web spider* part of the
system, while the indexing/searching stuff is written in C or
something close (memory fails me on that point, but I'm sure Google
itself could easily tell you more).

There's an interesting paper about the early architecture/history of
Google which was a published by the authors (when they were still
in figurative diapers) and which was referenced recently in this
newsgroup. The archives would quickly pull it up for you.

-Peter

Sep 15 '05 #8

Thomas Guettler

Am Wed, 14 Sep 2005 07:39:35 -0700 schrieb Harlin Seritt:

Hi,

Is anyone aware of an available open-source/free search engine app
(something similar to HTDig) written in Python that is out there?
Googling has turned up nothing. Thought maybe I'd mine some of you
guys' minds on this.

Hi,

I use ZODB (not ZOPE) with ZCTextIndex.

HTH,
Thomas

--
Thomas Güttler, http://www.thomas-guettler.de/
E-Mail: guettli (*) thomas-guettler + de
Spam Catcher: ni**************@thomas-guettler.de

Sep 15 '05 #9

Jim

I am finding this discussion illuminating. Can I ask: does it
possibly make sense to use a dB? For an application of mine, I had
been looking at putting the text into PostGres, and using their
OpenFTS, at //http://openfts.sourceforge.net/ (there is a Python module
to talk to OpenFTS, although it is in an early stage of development).

Jim

Sep 15 '05 #10

asmlinkage

Will McGugan wrote:

Wasn't Google's first search engine actualy written in Python?

I would'nt be surprised if they actually used python to do some
prototyping of their algorithms but I cannot imagine the search engine
actually being written in python.

Search engines consist of many sub-systems like spidering, tokenizing,
indexing, compression, ranking etc ... I think the main portion where
google or as a matter of fact many search companies would use python
is probably for spidering and for all the glue that supports the system.

Sep 16 '05 #11