Can you avoid that googlebot indexes PHPSESSID pages?

[CAH]

Hi

Can you avoid that googlebot indexes PHPSESSID pages? Googlebot is
indexing pages with PHPSESSID, which makes it think my page has a
infinite number of pages. How can one avoid this?

Here is an exsample of url that google register, that might make is
more clear what is happening

http://www.winches.dk/winches.php?ar...6f0d46334659ff...
http://www.winches.dk/winches.php?ar...b6aed41fc142ea...

I do use session registred ID, but if I visit my site I never see those
kind of urls? So how come google gets a hold of them?

Best regards
Mads

[Jerry Stuckle]

CAH wrote:

Hi

Can you avoid that googlebot indexes PHPSESSID pages? Googlebot is
indexing pages with PHPSESSID, which makes it think my page has a
infinite number of pages. How can one avoid this?

Here is an exsample of url that google register, that might make is
more clear what is happening

http://www.winches.dk/winches.php?ar...6f0d46334659ff...
http://www.winches.dk/winches.php?ar...b6aed41fc142ea...

I do use session registred ID, but if I visit my site I never see those
kind of urls? So how come google gets a hold of them?

Best regards
Mads

http://www.php.net/manual/en/ref.session.php

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
js*******@attgl obal.net
=============== ===

[Chung Leong]

CAH wrote:

Hi

Can you avoid that googlebot indexes PHPSESSID pages? Googlebot is
indexing pages with PHPSESSID, which makes it think my page has a
infinite number of pages. How can one avoid this?
Well, one way to handle this is to check the User-Agent header to see
if the client is Googlebot and not enable session. Obviously if a page
is dependent on session then it ceases to be indexible.
Here is an exsample of url that google register, that might make is
more clear what is happening

http://www.winches.dk/winches.php?ar...6f0d46334659ff...
http://www.winches.dk/winches.php?ar...b6aed41fc142ea...

I do use session registred ID, but if I visit my site I never see those
kind of urls? So how come google gets a hold of them?

If session.use_tra ns_sid is enabled, then PHP tries to compensate for
the lack of cookie by inserting the session id into any possible links.

I think you have quite a problem on your hand. Once those links are in
Google's database, the bot will keep returning to them. You'll need to
detect the condition and tell Googlebot to buzz off so it doesn't eat
up your bandwidth quota.

[CAH]

CAH skrev:

Hi

Can you avoid that googlebot indexes PHPSESSID pages? Googlebot is
indexing pages with PHPSESSID, which makes it think my page has a
infinite number of pages. How can one avoid this?

Here is an exsample of url that google register, that might make is
more clear what is happening

http://www.winches.dk/winches.php?ar...6f0d46334659ff...
http://www.winches.dk/winches.php?ar...b6aed41fc142ea...

I do use session registred ID, but if I visit my site I never see those
kind of urls? So how come google gets a hold of them?

Best regards
Mads

I am now testing this as a solution

"Using .htaccess often, you need to put the following two lines in the
..htaccess file, if your host is using PHP as an Apache module:

php_value session.use_onl y_cookies 1
php_value session.use_tra ns_sid 0 "

The downside is my site now only functions when user has cookies
enabled, and I am still not sure whethers this will do the trick.

[noone]

CAH wrote:

CAH skrev:

Hi

Can you avoid that googlebot indexes PHPSESSID pages? Googlebot is
indexing pages with PHPSESSID, which makes it think my page has a
infinite number of pages. How can one avoid this?

Here is an exsample of url that google register, that might make is
more clear what is happening

http://www.winches.dk/winches.php?ar...6f0d46334659ff...
http://www.winches.dk/winches.php?ar...b6aed41fc142ea...

I do use session registred ID, but if I visit my site I never see those
kind of urls? So how come google gets a hold of them?

Best regards
Mads

I am now testing this as a solution "Using .htaccess often, you need to put the following two lines in the
..htaccess file, if your host is using PHP as an Apache module: php_value session.use_onl y_cookies 1
php_value session.use_tra ns_sid 0 " The downside is my site now only functions when user has cookies
enabled, and I am still not sure whethers this will do the trick.

IIRC, google and other sites search for a file called robots.txt that give
directives on what it can and cannot index. Do a google search for
robots.txt to see... (to verify, look in your webserver log files - it
does show up as a request in my apache log files...)

If your robots.txt includes the following directive - it will skip the
entire site.

User-agent: *
Disallow: *

or to limit the scope of it's search:
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: *.php

[CAH]

> IIRC, google and other sites search for a file called robots.txt that give

directives on what it can and cannot index. Do a google search for
robots.txt to see... (to verify, look in your webserver log files - it
does show up as a request in my apache log files...)

If your robots.txt includes the following directive - it will skip the
entire site.

User-agent: *
Disallow: *

or to limit the scope of it's search:
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: *.php

I was testing this robot.txt

User-agent: Googlebot
Disallow: /*PHPSESSID

And that might solve it, I just do not know whether is works or not.

Mads

[Scott]

On Mon, 2006-04-03 at 01:20 -0700, CAH wrote:

Hi

Can you avoid that googlebot indexes PHPSESSID pages? Googlebot is
indexing pages with PHPSESSID, which makes it think my page has a
infinite number of pages. How can one avoid this?

Here is an exsample of url that google register, that might make is
more clear what is happening

http://www.winches.dk/winches.php?ar...6f0d46334659ff...
http://www.winches.dk/winches.php?ar...b6aed41fc142ea...

I do use session registred ID, but if I visit my site I never see those
kind of urls? So how come google gets a hold of them?

Best regards
Mads

There was some discussion of forcing cookies, but the author didn't want
to limit his users, so...

How about doing something like this:

// See if the user agent is Googlebot
$isGoogle = stripos($_SERVE R['HTTP_USER_AGEN T'], 'Googlebot');

// If it is, use ini_set to only allow cookies for the session variable
if ($isGoogle !== false) {
ini_set('sessio n.use_only_cook ies', '1');
}

[CAH]

> There was some discussion of forcing cookies, but the author didn't want

to limit his users, so...

How about doing something like this:

// See if the user agent is Googlebot
$isGoogle = stripos($_SERVE R['HTTP_USER_AGEN T'], 'Googlebot');

// If it is, use ini_set to only allow cookies for the session variable
if ($isGoogle !== false) {
ini_set('sessio n.use_only_cook ies', '1');
}

That is a cool solution, but can one be sure that one can reconize
googlebot? And how about all the other robots? Could one make a "is not
robot test"?

Thanks for the help
Mads

[Scott]

On Mon, 2006-04-03 at 23:57 -0700, CAH wrote:

There was some discussion of forcing cookies, but the author didn't want
to limit his users, so...

How about doing something like this:

// See if the user agent is Googlebot
$isGoogle = stripos($_SERVE R['HTTP_USER_AGEN T'], 'Googlebot');

// If it is, use ini_set to only allow cookies for the session variable
if ($isGoogle !== false) {
ini_set('sessio n.use_only_cook ies', '1');
}

That is a cool solution, but can one be sure that one can reconize
googlebot? And how about all the other robots? Could one make a "is not
robot test"?

Thanks for the help
Mads

I wouldn't expect all (or even most) robots to be easily identified by
the user-agent. Maybe you could make an array of the most common ones
(Googlebot, Inktomi, etc) and loop through it with the logic I
suggested. I also don't think you could check to see if it's a browser,
because firewalls & proxy servers may not send that information through.

Sorry! (It's not my internet. I just work here!)

Scott

[CAH]

> I wouldn't expect all (or even most) robots to be easily identified
by

the user-agent. Maybe you could make an array of the most common ones
(Googlebot, Inktomi, etc) and loop through it with the logic I
suggested. I also don't think you could check to see if it's a browser,
because firewalls & proxy servers may not send that information through.

I see what you mean.

Do you think this solution will work?

"Using .htaccess often, you need to put the following two lines in the
..htaccess file, if your host is using PHP as an Apache module:

php_value session.use_onl y_cookies 1
php_value session.use_tra ns_sid 0 "

I think it does, and even though you then have to rely on cookies, I
think it is the better solution because today this is a small minus,
compared to search engine problems.

If this solutions works

User-agent: Googlebot
Disallow: /*PHPSESSID

it would be by far the simplest, I do however not feel to sure that it
does work, and have no opportunity to check it at this time.

Regards
Mads