How to make dynamic aspx pages indexable by spiders

[moondaddy]

I'm writing an ecommerce app in asp.net/vb.net and need to make the pages
searchable and crawlable by spiders, particularly Google's. As far as I
know, if my pages's contents are mostly populated by user controls on a
single page and I call these different controls by passing one or more
parameters like this: myweb.com/default.aspx?MenuID=44, then the spiders
aren't going to be able do to anything with this. asp.net offers lots of
great technology, but if the success of the site depends on people's ability
to find it, then all of this work is to no avail. I'm surprised to find
very little (none actually) threads or documentation about this. the only
information I can find on optimizing a site for search engine ratings are by
non-ms portals and they all talk about static html pages. its like they
never heard of asp.net.

Can anyone help on this under talked about topic?

historically I've built business applications and never had to worry about
his. I spent tons of time building and optimizing the site for usability
and performance, and now that I'm trying to get it indexed by the search
engine, I'm finding that my high performance database driven site wont be
seen by the spiders, or I'm just not finding the right documentation.

--
mo*******@nospam.com

[Cowboy \(Gregory A. Beamer\)]

Fully dynamic? It is not going to happen as the spider does not know if it
can trust what you are going to deliver.

Predicable, but with a dynamic engine? Consider an HTTP Handler that makes
up a URL like so:

http://mysite.com/august/2004/1/somepage.html

For

http://mysite.com/article.aspx?id=16287674465

the VBPJ site (www.vbpj.com) had an article on hierarchical URLs that was
very nice in showing how to set up the handler for this type of URL. The
spider will see an HTML page, not a dynamic URL, even though you are
creating a page on the fly.

--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA

************************************************** ********************
Think Outside the Box!
************************************************** ********************
"moondaddy" <mo*******@nospam.com> wrote in message
news:ev**************@TK2MSFTNGP09.phx.gbl...

I'm writing an ecommerce app in asp.net/vb.net and need to make the pages
searchable and crawlable by spiders, particularly Google's. As far as I
know, if my pages's contents are mostly populated by user controls on a
single page and I call these different controls by passing one or more
parameters like this: myweb.com/default.aspx?MenuID=44, then the spiders
aren't going to be able do to anything with this. asp.net offers lots of
great technology, but if the success of the site depends on people's ability to find it, then all of this work is to no avail. I'm surprised to find
very little (none actually) threads or documentation about this. the only
information I can find on optimizing a site for search engine ratings are by non-ms portals and they all talk about static html pages. its like they
never heard of asp.net.

Can anyone help on this under talked about topic?

historically I've built business applications and never had to worry about
his. I spent tons of time building and optimizing the site for usability
and performance, and now that I'm trying to get it indexed by the search
engine, I'm finding that my high performance database driven site wont be
seen by the spiders, or I'm just not finding the right documentation.

--
mo*******@nospam.com

[moondaddy]

did you mean this one?
http://www.ftponline.com/vsm/2002_02...r/default.aspx
If so, I didnt see the connect. If not, do you have a url to the article
you were talking about?

Thanks.

--
mo*******@nospam.com
"Cowboy (Gregory A. Beamer)" <No************@comcast.netNoSpamM> wrote in
message news:eq**************@TK2MSFTNGP09.phx.gbl...

Fully dynamic? It is not going to happen as the spider does not know if it
can trust what you are going to deliver.

Predicable, but with a dynamic engine? Consider an HTTP Handler that makes
up a URL like so:

http://mysite.com/august/2004/1/somepage.html

For

http://mysite.com/article.aspx?id=16287674465

the VBPJ site (www.vbpj.com) had an article on hierarchical URLs that was
very nice in showing how to set up the handler for this type of URL. The
spider will see an HTML page, not a dynamic URL, even though you are
creating a page on the fly.

--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA

************************************************** ********************
Think Outside the Box!
************************************************** ********************
"moondaddy" <mo*******@nospam.com> wrote in message
news:ev**************@TK2MSFTNGP09.phx.gbl...

I'm writing an ecommerce app in asp.net/vb.net and need to make the pages searchable and crawlable by spiders, particularly Google's. As far as I
know, if my pages's contents are mostly populated by user controls on a
single page and I call these different controls by passing one or more
parameters like this: myweb.com/default.aspx?MenuID=44, then the spiders aren't going to be able do to anything with this. asp.net offers lots of great technology, but if the success of the site depends on people's ability

to find it, then all of this work is to no avail. I'm surprised to find
very little (none actually) threads or documentation about this. the only information I can find on optimizing a site for search engine ratings are by

non-ms portals and they all talk about static html pages. its like they
never heard of asp.net.

Can anyone help on this under talked about topic?

historically I've built business applications and never had to worry

about his. I spent tons of time building and optimizing the site for usability and performance, and now that I'm trying to get it indexed by the search
engine, I'm finding that my high performance database driven site wont be seen by the spiders, or I'm just not finding the right documentation.

--
mo*******@nospam.com

[moondaddy]

Ok maybe I missed the point if this was in fact the article you were talking
about. The light when on when I read another article about this problem:
http://www.seochat.com/c/a/Search-En...Search-Engine/
where they were talking about php pages, but the same concept (although it
didn't get into the mechanics, just an overview).

so in short is this how it would work: Rather than passing in a parameter,
I'd pass in a url like http://www..mysite/procucts/category/animals and on
the backend I'll use the http handler to intersept this, parse it, and run
select case on the last word for example (in this case animals), then call
the real page using a redirect passing in the real parameter like
products.aspx?89 (where 89 could be the category id animals). Is this the
basic idea?

--
mo*******@nospam.com
"moondaddy" <mo*******@nospam.com> wrote in message
news:uP**************@TK2MSFTNGP11.phx.gbl...

did you mean this one?
http://www.ftponline.com/vsm/2002_02...r/default.aspx If so, I didnt see the connect. If not, do you have a url to the article
you were talking about?

Thanks.

--
mo*******@nospam.com
"Cowboy (Gregory A. Beamer)" <No************@comcast.netNoSpamM> wrote in
message news:eq**************@TK2MSFTNGP09.phx.gbl...

Fully dynamic? It is not going to happen as the spider does not know if it

can trust what you are going to deliver.

Predicable, but with a dynamic engine? Consider an HTTP Handler that makes up a URL like so:

http://mysite.com/august/2004/1/somepage.html

For

http://mysite.com/article.aspx?id=16287674465

the VBPJ site (www.vbpj.com) had an article on hierarchical URLs that was very nice in showing how to set up the handler for this type of URL. The
spider will see an HTML page, not a dynamic URL, even though you are
creating a page on the fly.

--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA

************************************************** ********************
Think Outside the Box!
************************************************** ********************
"moondaddy" <mo*******@nospam.com> wrote in message
news:ev**************@TK2MSFTNGP09.phx.gbl...

I'm writing an ecommerce app in asp.net/vb.net and need to make the

pages searchable and crawlable by spiders, particularly Google's. As far as I know, if my pages's contents are mostly populated by user controls on a single page and I call these different controls by passing one or more
parameters like this: myweb.com/default.aspx?MenuID=44, then the spiders aren't going to be able do to anything with this. asp.net offers lots of great technology, but if the success of the site depends on people's

ability

to find it, then all of this work is to no avail. I'm surprised to find very little (none actually) threads or documentation about this. the only information I can find on optimizing a site for search engine ratings are

by

non-ms portals and they all talk about static html pages. its like they never heard of asp.net.

Can anyone help on this under talked about topic?

historically I've built business applications and never had to worry

about his. I spent tons of time building and optimizing the site for usability and performance, and now that I'm trying to get it indexed by the search engine, I'm finding that my high performance database driven site wont be seen by the spiders, or I'm just not finding the right documentation.

--
mo*******@nospam.com

[clintonG]

"Rewrite URLs on the Fly" by Jeff Prosise (ASP.NET C#)

The article will show up here [1] but I do not know when as it was
just published by asp.netPRO, March 2004 and can only be accessed
via [2] by magazine subscribers.

[1] http://www.wintellect.com/resources/...spx?authorID=1
[2] www.aspnetPRO.com/download

--
<%= Clinton Gallagher
A/E/C Consulting, Web Design, e-Commerce Software Development
Wauwatosa, Milwaukee County, Wisconsin USA
NET cs*********@REMOVETHISTEXTmetromilwaukee.com
URL http://www.metromilwaukee.com/clintongallagher/

"moondaddy" <mo*******@nospam.com> wrote in message
news:#r*************@tk2msftngp13.phx.gbl...

Ok maybe I missed the point if this was in fact the article you were talking about. The light when on when I read another article about this problem:
http://www.seochat.com/c/a/Search-En...Search-Engine/ where they were talking about php pages, but the same concept (although it
didn't get into the mechanics, just an overview).

so in short is this how it would work: Rather than passing in a parameter, I'd pass in a url like http://www..mysite/procucts/category/animals and on
the backend I'll use the http handler to intersept this, parse it, and run
select case on the last word for example (in this case animals), then call
the real page using a redirect passing in the real parameter like
products.aspx?89 (where 89 could be the category id animals). Is this the
basic idea?

--
mo*******@nospam.com
"moondaddy" <mo*******@nospam.com> wrote in message
news:uP**************@TK2MSFTNGP11.phx.gbl...

did you mean this one?

http://www.ftponline.com/vsm/2002_02...r/default.aspx

If so, I didnt see the connect. If not, do you have a url to the article

you were talking about?

Thanks.

--
mo*******@nospam.com
"Cowboy (Gregory A. Beamer)" <No************@comcast.netNoSpamM> wrote in message news:eq**************@TK2MSFTNGP09.phx.gbl...

Fully dynamic? It is not going to happen as the spider does not know if it can trust what you are going to deliver.

Predicable, but with a dynamic engine? Consider an HTTP Handler that makes up a URL like so:

http://mysite.com/august/2004/1/somepage.html

For

http://mysite.com/article.aspx?id=16287674465

the VBPJ site (www.vbpj.com) had an article on hierarchical URLs that was very nice in showing how to set up the handler for this type of URL.

The spider will see an HTML page, not a dynamic URL, even though you are
creating a page on the fly.

--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA

************************************************** ********************
Think Outside the Box!
************************************************** ********************
"moondaddy" <mo*******@nospam.com> wrote in message
news:ev**************@TK2MSFTNGP09.phx.gbl...
> I'm writing an ecommerce app in asp.net/vb.net and need to make the pages

> searchable and crawlable by spiders, particularly Google's. As far as I > know, if my pages's contents are mostly populated by user controls

on

a > single page and I call these different controls by passing one or

more > parameters like this: myweb.com/default.aspx?MenuID=44, then the

spiders

> aren't going to be able do to anything with this. asp.net offers lots

of

> great technology, but if the success of the site depends on people's
ability
> to find it, then all of this work is to no avail. I'm surprised to find > very little (none actually) threads or documentation about this.

the only

> information I can find on optimizing a site for search engine

ratings are

by
> non-ms portals and they all talk about static html pages. its like they > never heard of asp.net.
>
> Can anyone help on this under talked about topic?
>
> historically I've built business applications and never had to worry

about

> his. I spent tons of time building and optimizing the site for

usability

> and performance, and now that I'm trying to get it indexed by the search > engine, I'm finding that my high performance database driven site

wont be

> seen by the spiders, or I'm just not finding the right

documentation. >
> --
> mo*******@nospam.com
>
>

[darrel]

> As far as I

know, if my pages's contents are mostly populated by user controls on a
single page and I call these different controls by passing one or more
parameters like this: myweb.com/default.aspx?MenuID=44, then the spiders
aren't going to be able do to anything with this.

This is incorrect, at least in terms of Google. Google can handle query
strings just fine, with a few caveats:

1) It will index your site slower than if it does not see querystrings (this
is actually because it is polite...if it sees a query string, it assumes a
database, and doesn't want to overload it with queries).

2) It seems to not like more than 3 values in the query string, with two
being the most common limit.

The other issue with query strings, of course, is that they are not very
human readable. If this is a big issue for you, look into a URL rewriting
tool like ISAPI rewrite.

-Darrel

[Steven Cheng[MSFT]]

Thanks for all your informative suggestions.

Hi Moondaddy,

Here are some further resources on ASP.NET Url Rewriting and how to perform
search engine friendly url:

#URL Rewriting in ASP.NET
http://msdn.microsoft.com/library/en...ng.asp?frame=t
rue

#Rewriting with ASP.NET
http://www.codeproject.com/aspnet/URLRewriter.asp

#Search engine friendly URLs using ASP.NET (C#.NET)
http://www.codetoad.com/asp.net_ma_s...nefriendly.asp

Hope also helps. Thanks.
Regards,

Steven Cheng
Microsoft Online Support

Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)

Get Preview at ASP.NET whidbey
http://msdn.microsoft.com/asp.net/whidbey/default.aspx

[moondaddy]

These are great links! Just what the good doctor ordered.

THANKS

--
mo*******@nospam.com
"Steven Cheng[MSFT]" <v-******@online.microsoft.com> wrote in message
news:3S**************@cpmsftngxa10.phx.gbl...

Thanks for all your informative suggestions.

Hi Moondaddy,

Here are some further resources on ASP.NET Url Rewriting and how to perform search engine friendly url:

#URL Rewriting in ASP.NET
http://msdn.microsoft.com/library/en...ng.asp?frame=t rue

#Rewriting with ASP.NET
http://www.codeproject.com/aspnet/URLRewriter.asp

#Search engine friendly URLs using ASP.NET (C#.NET)
http://www.codetoad.com/asp.net_ma_s...nefriendly.asp

Hope also helps. Thanks.
Regards,

Steven Cheng
Microsoft Online Support

Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)

Get Preview at ASP.NET whidbey
http://msdn.microsoft.com/asp.net/whidbey/default.aspx

[George Ter-Saakov]

Not ordered, Prescribed.

Just my 2 cents :)

George.

"moondaddy" <mo*******@nospam.com> wrote in message
news:ex**************@TK2MSFTNGP09.phx.gbl...

These are great links! Just what the good doctor ordered.

THANKS

[Cowboy \(Gregory A. Beamer\)]

I was actually thinking this link:
http://www.ftponline.com/vsm/2002_02...ne/columns/qa/

This one also gives some hints on using it to mask URLs:
http://www.15seconds.com/issue/030522.htm

--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA

************************************************** ********************
Think Outside the Box!
************************************************** ********************
"moondaddy" <mo*******@nospam.com> wrote in message
news:uP**************@TK2MSFTNGP11.phx.gbl...

did you mean this one?
http://www.ftponline.com/vsm/2002_02...r/default.aspx If so, I didnt see the connect. If not, do you have a url to the article
you were talking about?

Thanks.

--
mo*******@nospam.com
"Cowboy (Gregory A. Beamer)" <No************@comcast.netNoSpamM> wrote in
message news:eq**************@TK2MSFTNGP09.phx.gbl...

Fully dynamic? It is not going to happen as the spider does not know if it

can trust what you are going to deliver.

Predicable, but with a dynamic engine? Consider an HTTP Handler that makes up a URL like so:

http://mysite.com/august/2004/1/somepage.html

For

http://mysite.com/article.aspx?id=16287674465

the VBPJ site (www.vbpj.com) had an article on hierarchical URLs that was very nice in showing how to set up the handler for this type of URL. The
spider will see an HTML page, not a dynamic URL, even though you are
creating a page on the fly.

--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA

************************************************** ********************
Think Outside the Box!
************************************************** ********************
"moondaddy" <mo*******@nospam.com> wrote in message
news:ev**************@TK2MSFTNGP09.phx.gbl...

I'm writing an ecommerce app in asp.net/vb.net and need to make the

pages searchable and crawlable by spiders, particularly Google's. As far as I know, if my pages's contents are mostly populated by user controls on a single page and I call these different controls by passing one or more
parameters like this: myweb.com/default.aspx?MenuID=44, then the spiders aren't going to be able do to anything with this. asp.net offers lots of great technology, but if the success of the site depends on people's

ability

to find it, then all of this work is to no avail. I'm surprised to find very little (none actually) threads or documentation about this. the only information I can find on optimizing a site for search engine ratings are

by

non-ms portals and they all talk about static html pages. its like they never heard of asp.net.

Can anyone help on this under talked about topic?

historically I've built business applications and never had to worry

about his. I spent tons of time building and optimizing the site for usability and performance, and now that I'm trying to get it indexed by the search engine, I'm finding that my high performance database driven site wont be seen by the spiders, or I'm just not finding the right documentation.

--
mo*******@nospam.com

[Cowboy \(Gregory A. Beamer\)]

There are some that will completely ignore dynamic strings, however. Google
will do fine, over time. Some will index if you add pages in robots.txt, but
will otherwise ignore any dynamic strings. The ASP.NET redirection will
solve the issue, as the URL appears to be a static URL.

--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA

************************************************** ********************
Think Outside the Box!
************************************************** ********************
"darrel" <no*****@hotmail.com> wrote in message
news:%2****************@tk2msftngp13.phx.gbl...

As far as I
know, if my pages's contents are mostly populated by user controls on a
single page and I call these different controls by passing one or more
parameters like this: myweb.com/default.aspx?MenuID=44, then the spiders aren't going to be able do to anything with this.
This is incorrect, at least in terms of Google. Google can handle query
strings just fine, with a few caveats:

1) It will index your site slower than if it does not see querystrings

(this is actually because it is polite...if it sees a query string, it assumes a
database, and doesn't want to overload it with queries).

2) It seems to not like more than 3 values in the query string, with two
being the most common limit.

The other issue with query strings, of course, is that they are not very
human readable. If this is a big issue for you, look into a URL rewriting
tool like ISAPI rewrite.

-Darrel