473,323 Members | 1,574 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,323 software developers and data experts.

THE IMPORTANCE OF MAKING THE GOOGLE INDEX DOWNLOADABLE

THE IMPORTANCE OF MAKING THE GOOGLE INDEX DOWNLOADABLE

I write here to make a request on behalf of all the programmers on
earth who have been or are intending to use the Google web search API
for either research purposes or for the development of real world
applications, that Google make their indexes downloadable.

Currently application programmers using the Google web search API are
limited to 1000 queries a day. This on the one hand is a reasonable
decision by Google because; limiting the queries will prevent harm on
the Google system by unnecessary automated queries; but it is also
limiting us programmers severely. The query limit limits the usefulness
of whatever applications we decide to craft out and even limits our
imagination on what is possible with a handful of indexes.

Firstly, I will commend the Google Corporation for opening their
preciously crawled indexes. This is a great service to humanity and
especially to the band of programmers who are interested in
epistemology and are using the Google web search API to enable them
achieve their goals.

Google would be doing another great service for us if they would make
their indexes downloadable to programmers with a good interface for
programmatically accessing the indexes.

The advantages of the above approach would be:

1. Decentralizing the Google system.
2. Reducing the overhead of queries on Google from programmers.
3. Enabling programmers to craft out applications that run on their
local systems (only requiring internet connection when a web page is
needed since the links return on a query are the most important in the
result set) thus enabling them have unlimited number of queries should
these applications go public.
4. Give Google the competing edge in search engine technology and user
satisfaction by gaining programmer loyalty.
5. Encouraging the global adoption and use of the API + INDEXES
provided by Google.
6. Another good thing may be here for Google if they create mechanisms
in the downloaded INDEXES + API that enable programmers update the
indexes from the web. An agreement can now be made that Google will
have unlimited access to the indexes whenever the user's computer is
online and IDLE. So Google update its own indexes from the ones on
various programmers' local machines. Thereby building a truly
distributed global crawler. This can be achieved using grid
technologies thereby possibly cutting down the 300year range for
crawling the world's crawlable information.
Google may still enforce their terms of service by enforcing some kind
of authentication for the use of index already residing on the
programmer's local machine. Though it may not require that the
programmer be on the internet every time he/she wishes to access the
system; since the programmer may wish to tinker with the API and
indexes locally without requiring an internet connection. Online
authentication may be required anytime the user gets online. The
non-commerciability of the indexes must be emphasized through several
schemes.
The Google API can be a tool for epistemological engineers to craft
future Infowares (Information Applications). The most important thing
in the indexes is the links to resources that are returned on queries.

2 versions of the API + INDEXES can be made available.
1. The one without cached pages attached. So that on querying the API
on the local machine with the locally stored indexes, the results are
like those on the regular internet API result set.

2. The one with the cached pages. This one is optional as it will be
large in size.

If you people were good enough to release your API's publicly then
you would also consider this request.

It would be good if the API + INDEX download is accessible by
programmers who program in the following languages:
(a) Python
(b) Java
(c) Perl
(d) Ruby

Or some language independent mechanism can be formulated so programmers
in various languages can access the API + INDEX download.

Page Rank may or may not be included in the package depending on
decisions at Google.

It may also be closed source / open source / or partial source (part
open part closed).

This will be a great service to humanity and to programmers especially.

Thanks,

Ogah Ejini,

Nigeria, West Africa.

Mobile: +234 802 601 5061

Jun 7 '06 #1
2 2004
gen_tricomi wrote:
Currently application programmers using the Google web search API are
limited to 1000 queries a day. This on the one hand is a reasonable
decision by Google because; limiting the queries will prevent harm on
the Google system by unnecessary automated queries; but it is also
limiting us programmers severely. The query limit limits the usefulness
of whatever applications we decide to craft out and even limits our
imagination on what is possible with a handful of indexes.

If you know which sites you are interested in searching then you can
already license hardware from Google which will let you index up to 15
million documents and an api to search the indexed content without
restriction.

If you simply want to compete with Google on searching the entire internet
then they are unlikely to want to help you.

If you fall somewhere in between what can be done with a Google Search
Appliance and competing with Google then you are talking about paying
Google sufficient money that they ought to be interested in sitting round a
table with you. As it says on their website: "For larger deployments,
contact us and we’ll be happy to talk to you about building a custom search
solution for your environment."
Jun 7 '06 #2
gen_tricomi wrote:
THE IMPORTANCE OF MAKING THE GOOGLE INDEX DOWNLOADABLE

I write here to make a request on behalf of all the programmers on
earth who have been or are intending to use the Google web search API
for either research purposes or for the development of real world
applications, that Google make their indexes downloadable.

Frankly I doubt whether the average programmer possesses sufficient
storage or has access to sufficient bandwidth to make downloading the
Google indexes a practical proposition - let alone the cached page
contents too.

There's also the tiny factoid that Google might regard their index
structure, not to mention its contents, as proprietary.

Finally, how frequently would you propose to update your local copy?
Google is adding the results of new spidering to their indices all the time.

A nice idea, perhaps, but surely completely impractical.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Love me, love my blog http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

Jun 7 '06 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: Mike D | last post by:
The code below gives an error. Object doesn't support this property or method: 'myMail.Importance' How do I set importance on an email? Everywhere I have looked it looks like my syntax is right...
19
by: Christian Hvid | last post by:
Hello groups. I have a series of applet computer games on my homepage: http://vredungmand.dk/games/erik-spillet/index.html http://vredungmand.dk/games/nohats/index.html...
3
by: Jason Kistler | last post by:
I am having some serious issues trying to set the "importance" of an ASP email. I am using the CDO.Message object. Here is the code: <% Dim recipients recipients = Request.Form("Jason") &...
9
by: Xah Lee | last post by:
is the Microsoft javascript doc downloadable? http://msdn.microsoft.com/library/default.asp?url=/library/en-us/script56/html/js56jsoriJScript.asp or, is there some other downloadable complete...
19
by: Christian Fowler | last post by:
I have a VERY LARGE pile of geographic data that I am importing into a database (db of choice is postgres, though may hop to oracle if necessary). The data is strictly hierarchical - each node has...
25
by: Amarendra GODBOLE | last post by:
Hi, I am working on a legacy user space app, which has been developed entirely in C, some 15 years ago. Needless to say, it does not even partially conform to any standard. My team is in the...
19
by: elzacho | last post by:
It is my understanding that prototypes in C are purely optional. To my experience, until today I guess, this has been the case (other than eliminating compiler warning messages). However, today...
17
by: =?Utf-8?B?Y2F0aGFyaW51cyB2YW4gZGVyIHdlcmY=?= | last post by:
Hello, I have build a website with approximately 30 html-pages. When I search this website in Google, I see the index.html or home.html on this website, but also other html-pages on this...
9
by: maheswaran | last post by:
Hi all, I developed one application. From that application i created dynamic pages contact us , about us...(like joomla, but application is not in joomla)... These all are comes from database.In...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.