By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
443,760 Members | 1,585 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 443,760 IT Pros & Developers. It's quick & easy.

Web crawlers and dynamic content

P: n/a
Hi,
This must be a very old and well studied question...
I created a site using JSP, that permits the viewing of articles that
are fully stored in a database. The url for the articles is something
like:
.../article.jsp?idi=6&ida=12
My question is, and this is the same for all dynamic content
technologies (server-side scripting technologies): is there a way to
permit for search engines and crawlers (Google, Yahoo...) to find and
categorize these article pages?
Thank you very much,
Araxes Tharsis
Jul 17 '05 #1
Share this Question
Share on Google+
3 Replies


P: n/a
Araxes Tharsis <pn*****@hotmail.com> wrote:
This must be a very old and well studied question...
So an anwser should be easy to find (searchengines and usenet archives).
I created a site using JSP, that permits the viewing of articles that
are fully stored in a database. The url for the articles is something
like:
../article.jsp?idi=6&ida=12
My question is, and this is the same for all dynamic content
technologies (server-side scripting technologies): is there a way to
permit for search engines and crawlers (Google, Yahoo...) to find and
categorize these article pages?


Create a page with the links you want to have indexed, SE will find it
(if this page is linked somewhere offcourse) and index it.

Jul 17 '05 #2

P: n/a
Araxes Tharsis <pn*****@hotmail.com> wrote:

Don't multipost please.

Jul 17 '05 #3

P: n/a

"Araxes Tharsis" <pn*****@hotmail.com> wrote in message
news:fb**************************@posting.google.c om...
Hi,
This must be a very old and well studied question...
I created a site using JSP, that permits the viewing of articles that
are fully stored in a database. The url for the articles is something
like:
../article.jsp?idi=6&ida=12
My question is, and this is the same for all dynamic content
technologies (server-side scripting technologies): is there a way to
permit for search engines and crawlers (Google, Yahoo...) to find and
categorize these article pages?


fwiw, google doesnt seem to have a problem accessing my pages with query
strings (I have scripts set up to email me when it indexes). however, so
far, it has not (if memory serves) got to any page with more than one
variable in the qs.

what you can do is use mod_rewrite to make your urls se friendly. google
it, there's lots of stuff on it. this will help:
http://www.webmaster-toolkit.com/mod...enerator.shtml

here is the script I'm using for the google bot:

$currentPage = $_SERVER['HTTP_HOST'].$_SERVER['REQUEST_URI'];// gives
contents of address bar
$mail = "go************@mydomain.com";
$subject = "GoogleBot - $currentPage ";
$message = "The GoogleBot has indexed this page! $currentPage";
if(eregi("google",$HTTP_USER_AGENT))
{
mail($mail, $subject, $message);
}

here's a list of bots you could use to add to that script, generate a email
for each kind of bot, or write to a database or flatfile:
http://joseluis.pellicer.org/ua/list...ch_Engine.html

--
juglesh

Jul 17 '05 #4

This discussion thread is closed

Replies have been disabled for this discussion.