Generating "most popular article" lists without crushing the server? ASP.NET caching considerations?

The New York Times and many other online publications automatically generate
"most popular article" lists that cover, say, the last 24 hours.

I am looking for guidance and/or code on the best way to do this in ASP.NET
2 in a way that

a) won't crush the server
and
b) won't be surpressed by any of ASP.NET's caching options: you want the
"hits" to still be registered even if someone is clicking through to a
cached instance of the page.

What are the common solutions for dealing with this? Is it common to use a
stats package and operate on the data store that's generated by the package,
or is this overkill for this intended purpose?

SQL Server is available, and I have full control of the server.

-KF

[John Timney \(MVP\)]

This is the kind of data you want about every hour on a busy site, or every
half day on a less busy site. Ideally, you should be getting the data from
the webserver logs not from the database and analysing it that way. If I
remember correctly, even cached data generates a head request in the IIS log
files so you should be fine querying the that file, which certainly wont
crush your server - but you'll need to check.

Regards

John Timney (MVP)
VISIT MY WEBSITE:
http://www.johntimney.com
http://www.johntimney.com/blog
<ke*****@nospam.nospamwrote in message
news:uS**************@TK2MSFTNGP04.phx.gbl...

The New York Times and many other online publications automatically
generate "most popular article" lists that cover, say, the last 24 hours.

I am looking for guidance and/or code on the best way to do this in
ASP.NET 2 in a way that

a) won't crush the server
and
b) won't be surpressed by any of ASP.NET's caching options: you want the
"hits" to still be registered even if someone is clicking through to a
cached instance of the page.

What are the common solutions for dealing with this? Is it common to use a
stats package and operate on the data store that's generated by the
package, or is this overkill for this intended purpose?

SQL Server is available, and I have full control of the server.

-KF

Thanks for the pointer, John. Interesting, I had never even thought of this
approach, nor have I seen it written about.

So you'd run a simple string analysis and count how many times files have
been hit?

Or maybe has someone written a class for interrogating log files. Does
anyone know of such a beast?

-KF
"John Timney (MVP)" <x_****@timney.eclipse.co.ukwrote in message
news:lp*********************@eclipse.net.uk...

This is the kind of data you want about every hour on a busy site, or
every half day on a less busy site. Ideally, you should be getting the
data from the webserver logs not from the database and analysing it that
way. If I remember correctly, even cached data generates a head request
in the IIS log files so you should be fine querying the that file, which
certainly wont crush your server - but you'll need to check.

Regards

John Timney (MVP)
VISIT MY WEBSITE:
http://www.johntimney.com
http://www.johntimney.com/blog
<ke*****@nospam.nospamwrote in message
news:uS**************@TK2MSFTNGP04.phx.gbl...

>The New York Times and many other online publications automatically
generate "most popular article" lists that cover, say, the last 24 hours.

I am looking for guidance and/or code on the best way to do this in
ASP.NET 2 in a way that

a) won't crush the server
and
b) won't be surpressed by any of ASP.NET's caching options: you want the
"hits" to still be registered even if someone is clicking through to a
cached instance of the page.

What are the common solutions for dealing with this? Is it common to use
a stats package and operate on the data store that's generated by the
package, or is this overkill for this intended purpose?

SQL Server is available, and I have full control of the server.

-KF

[John Timney \(MVP\)]

There is one, cant remember the name though but heres a couple of links to
get you started to prove the idea. Shouldn't take you too long.

http://awstats.sourceforge.net/
http://www.codeproject.com/aspnet/aspnetweblog.asp

Regards

John Timney (MVP)
VISIT MY WEBSITE:
http://www.johntimney.com
http://www.johntimney.com/blog

<ke*****@nospam.nospamwrote in message
news:%2****************@TK2MSFTNGP04.phx.gbl...

Thanks for the pointer, John. Interesting, I had never even thought of
this approach, nor have I seen it written about.

So you'd run a simple string analysis and count how many times files have
been hit?

Or maybe has someone written a class for interrogating log files. Does
anyone know of such a beast?

-KF
"John Timney (MVP)" <x_****@timney.eclipse.co.ukwrote in message
news:lp*********************@eclipse.net.uk...

>This is the kind of data you want about every hour on a busy site, or
every half day on a less busy site. Ideally, you should be getting the
data from the webserver logs not from the database and analysing it that
way. If I remember correctly, even cached data generates a head request
in the IIS log files so you should be fine querying the that file, which
certainly wont crush your server - but you'll need to check.

Regards

John Timney (MVP)
VISIT MY WEBSITE:
http://www.johntimney.com
http://www.johntimney.com/blog
<ke*****@nospam.nospamwrote in message
news:uS**************@TK2MSFTNGP04.phx.gbl...

>>The New York Times and many other online publications automatically
generate "most popular article" lists that cover, say, the last 24
hours.

I am looking for guidance and/or code on the best way to do this in
ASP.NET 2 in a way that

a) won't crush the server
and
b) won't be surpressed by any of ASP.NET's caching options: you want the
"hits" to still be registered even if someone is clicking through to a
cached instance of the page.

What are the common solutions for dealing with this? Is it common to use
a stats package and operate on the data store that's generated by the
package, or is this overkill for this intended purpose?

SQL Server is available, and I have full control of the server.

-KF

[=?Utf-8?B?UGV0ZXIgQnJvbWJlcmcgW0MjIE1WUF0=?=]

As an alternative to John's recommendation, you could try hooking the
Application_PreRequestHandlerExecute event in either global.asax or in an
HttpModule that handles the event. You can thus interrogate the request just
before it gets passed to the page processor, and log all the data you want
into your Sql Server.
Peter

--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net


"ke*****@nospam.nospam" wrote:

The New York Times and many other online publications automatically generate
"most popular article" lists that cover, say, the last 24 hours.

I am looking for guidance and/or code on the best way to do this in ASP.NET
2 in a way that

a) won't crush the server
and
b) won't be surpressed by any of ASP.NET's caching options: you want the
"hits" to still be registered even if someone is clicking through to a
cached instance of the page.

What are the common solutions for dealing with this? Is it common to use a
stats package and operate on the data store that's generated by the package,
or is this overkill for this intended purpose?

SQL Server is available, and I have full control of the server.

-KF

Thank you. Here is an article that discusses the tactic you describe:
http://www.superdotnet.com/Article.aspx?ArticleID=122

They're hooking Application_BeginRequest in global.asax rather than
Application_PreRequestHandlerExecute as you suggested; I wonder if there's a
material difference.

I'm a little surprised nobody has written an all-knowing, all-seeing class
and stored procedure set for addressing site statistic issues. For my needs,
I'm finding after-the-fact log analysis less appealing than the idea of a
SQL database I could interrogate.

Some other options: "Log Parser" allows SQL-type queries against any most
any windows log.
http://www.asp.net/sandbox/app_logparser.aspx
I'm a little disappointed the package doesn't have much in the way of
instructions or example for how to get up and running working with IIS. In
theory, this could be a great solution for my needs.

I'm also surprised that many of the stats packages don't seem to expose
their report data in any sort of structured form, say, in XML. You're stuck
either "rolling your own" solution and coding everything, or working with
packages that seem more directed at Desktop users. Not happy with any of the
prebuilt solutions that I can afford.

-KF
"Peter Bromberg [C# MVP]" <pb*******@yahoo.yabbadabbadoo.comwrote in
message news:1B**********************************@microsof t.com...

As an alternative to John's recommendation, you could try hooking the
Application_PreRequestHandlerExecute event in either global.asax or in an
HttpModule that handles the event. You can thus interrogate the request
just
before it gets passed to the page processor, and log all the data you want
into your Sql Server.
Peter

--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net


"ke*****@nospam.nospam" wrote:

>The New York Times and many other online publications automatically
generate
"most popular article" lists that cover, say, the last 24 hours.

I am looking for guidance and/or code on the best way to do this in
ASP.NET
2 in a way that

a) won't crush the server
and
b) won't be surpressed by any of ASP.NET's caching options: you want the
"hits" to still be registered even if someone is clicking through to a
cached instance of the page.

What are the common solutions for dealing with this? Is it common to use
a
stats package and operate on the data store that's generated by the
package,
or is this overkill for this intended purpose?

SQL Server is available, and I have full control of the server.

-KF

[John Timney \(MVP\)]

>Not happy with any of the prebuilt solutions that I can afford.

....and therein lies the problem. Its a complex subject, and experience
around these areas tends to be expensive as its more niche than mainstream
forms development for example, which is why your struggling to find an all
seeing class. Unfortunately there many ways to crack the same nut and the
pre-built nutcrackers that seem to work are not cheap for all of those
reasons. Whatever solution you turn to - make sure the solution itself
doesn't degrade your web servers performance.

--
--
Regards

John Timney (MVP)
VISIT MY WEBSITE:
http://www.johntimney.com
http://www.johntimney.com/blog
<ke*****@nospam.nospamwrote in message
news:eq*************@TK2MSFTNGP03.phx.gbl...

Thank you. Here is an article that discusses the tactic you describe:
http://www.superdotnet.com/Article.aspx?ArticleID=122

They're hooking Application_BeginRequest in global.asax rather than
Application_PreRequestHandlerExecute as you suggested; I wonder if there's
a material difference.

I'm a little surprised nobody has written an all-knowing, all-seeing class
and stored procedure set for addressing site statistic issues. For my
needs, I'm finding after-the-fact log analysis less appealing than the
idea of a SQL database I could interrogate.

Some other options: "Log Parser" allows SQL-type queries against any most
any windows log.
http://www.asp.net/sandbox/app_logparser.aspx
I'm a little disappointed the package doesn't have much in the way of
instructions or example for how to get up and running working with IIS. In
theory, this could be a great solution for my needs.

I'm also surprised that many of the stats packages don't seem to expose
their report data in any sort of structured form, say, in XML. You're
stuck either "rolling your own" solution and coding everything, or working
with packages that seem more directed at Desktop users. Not happy with any
of the prebuilt solutions that I can afford.

-KF
"Peter Bromberg [C# MVP]" <pb*******@yahoo.yabbadabbadoo.comwrote in
message news:1B**********************************@microsof t.com...

>As an alternative to John's recommendation, you could try hooking the
Application_PreRequestHandlerExecute event in either global.asax or in an
HttpModule that handles the event. You can thus interrogate the request
just
before it gets passed to the page processor, and log all the data you
want
into your Sql Server.
Peter

--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net


"ke*****@nospam.nospam" wrote:

>>The New York Times and many other online publications automatically
generate
"most popular article" lists that cover, say, the last 24 hours.

I am looking for guidance and/or code on the best way to do this in
ASP.NET
2 in a way that

a) won't crush the server
and
b) won't be surpressed by any of ASP.NET's caching options: you want the
"hits" to still be registered even if someone is clicking through to a
cached instance of the page.

What are the common solutions for dealing with this? Is it common to use
a
stats package and operate on the data store that's generated by the
package,
or is this overkill for this intended purpose?

SQL Server is available, and I have full control of the server.

-KF

Thanks, John. I've identified one another option: as of version 8, SurfStats
(a traffic analysis program) will export reports in Excel format, which I
could presumably interrogate with my ASP.NET app. We bought a license for
SurfStats, so this may be a good choice for me.

Microsoft should sell a good analysis program, or bundle it with IIS. It
seems like a basic need for people who may run sites.

-KF
"John Timney (MVP)" <x_****@timney.eclipse.co.ukwrote in message
news:T9*********************@eclipse.net.uk...

>>Not happy with any of the prebuilt solutions that I can afford.

...and therein lies the problem. Its a complex subject, and experience
around these areas tends to be expensive as its more niche than mainstream
forms development for example, which is why your struggling to find an all
seeing class. Unfortunately there many ways to crack the same nut and the
pre-built nutcrackers that seem to work are not cheap for all of those
reasons. Whatever solution you turn to - make sure the solution itself
doesn't degrade your web servers performance.

--
--
Regards

John Timney (MVP)
VISIT MY WEBSITE:
http://www.johntimney.com
http://www.johntimney.com/blog
<ke*****@nospam.nospamwrote in message
news:eq*************@TK2MSFTNGP03.phx.gbl...

>Thank you. Here is an article that discusses the tactic you describe:
http://www.superdotnet.com/Article.aspx?ArticleID=122

They're hooking Application_BeginRequest in global.asax rather than
Application_PreRequestHandlerExecute as you suggested; I wonder if
there's a material difference.

I'm a little surprised nobody has written an all-knowing, all-seeing
class and stored procedure set for addressing site statistic issues. For
my needs, I'm finding after-the-fact log analysis less appealing than the
idea of a SQL database I could interrogate.

Some other options: "Log Parser" allows SQL-type queries against any most
any windows log.
http://www.asp.net/sandbox/app_logparser.aspx
I'm a little disappointed the package doesn't have much in the way of
instructions or example for how to get up and running working with IIS.
In theory, this could be a great solution for my needs.

I'm also surprised that many of the stats packages don't seem to expose
their report data in any sort of structured form, say, in XML. You're
stuck either "rolling your own" solution and coding everything, or
working with packages that seem more directed at Desktop users. Not happy
with any of the prebuilt solutions that I can afford.

-KF
"Peter Bromberg [C# MVP]" <pb*******@yahoo.yabbadabbadoo.comwrote in
message news:1B**********************************@microsof t.com...

>>As an alternative to John's recommendation, you could try hooking the
Application_PreRequestHandlerExecute event in either global.asax or in
an
HttpModule that handles the event. You can thus interrogate the request
just
before it gets passed to the page processor, and log all the data you
want
into your Sql Server.
Peter

--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net


"ke*****@nospam.nospam" wrote:

The New York Times and many other online publications automatically
generate
"most popular article" lists that cover, say, the last 24 hours.

I am looking for guidance and/or code on the best way to do this in
ASP.NET
2 in a way that

a) won't crush the server
and
b) won't be surpressed by any of ASP.NET's caching options: you want
the
"hits" to still be registered even if someone is clicking through to a
cached instance of the page.

What are the common solutions for dealing with this? Is it common to
use a
stats package and operate on the data store that's generated by the
package,
or is this overkill for this intended purpose?

SQL Server is available, and I have full control of the server.

-KF

[Juan T. Llibre]

re:

Microsoft should sell a good analysis program, or bundle it with IIS. It seems like a basic need
for people who may run sites.

Microsoft recently (a few months ago) acquired Livestats from Deepmetrix:
http://www.deepmetrix.com/.
I doubt it will be bundled for free with IIS, but check them out.

I've been using SmarterStats free version:
http://www.smartertools.com/Products...tats/Free.aspx

I understand their paid-for version is quite complete.


Juan T. Llibre, asp.net MVP
asp.net faq : http://asp.net.do/faq/
foros de asp.net, en español : http://asp.net.do/foros/
===================================
<ke*****@nospam.nospamwrote in message news:Oe**************@TK2MSFTNGP04.phx.gbl...

Thanks, John. I've identified one another option: as of version 8, SurfStats (a traffic analysis
program) will export reports in Excel format, which I could presumably interrogate with my ASP.NET
app. We bought a license for SurfStats, so this may be a good choice for me.

Microsoft should sell a good analysis program, or bundle it with IIS. It seems like a basic need
for people who may run sites.
-KF

"John Timney (MVP)" <x_****@timney.eclipse.co.ukwrote in message
news:T9*********************@eclipse.net.uk...

>>>Not happy with any of the prebuilt solutions that I can afford.

...and therein lies the problem. Its a complex subject, and experience around these areas tends
to be expensive as its more niche than mainstream forms development for example, which is why
your struggling to find an all seeing class. Unfortunately there many ways to crack the same nut
and the pre-built nutcrackers that seem to work are not cheap for all of those reasons. Whatever
solution you turn to - make sure the solution itself doesn't degrade your web servers
performance.

--
--
Regards

John Timney (MVP)
VISIT MY WEBSITE:
http://www.johntimney.com
http://www.johntimney.com/blog
<ke*****@nospam.nospamwrote in message news:eq*************@TK2MSFTNGP03.phx.gbl...

>>Thank you. Here is an article that discusses the tactic you describe:
http://www.superdotnet.com/Article.aspx?ArticleID=122

They're hooking Application_BeginRequest in global.asax rather than
Application_PreRequestHandlerExecute as you suggested; I wonder if there's a material
difference.

I'm a little surprised nobody has written an all-knowing, all-seeing class and stored procedure
set for addressing site statistic issues. For my needs, I'm finding after-the-fact log analysis
less appealing than the idea of a SQL database I could interrogate.

Some other options: "Log Parser" allows SQL-type queries against any most any windows log.
http://www.asp.net/sandbox/app_logparser.aspx
I'm a little disappointed the package doesn't have much in the way of instructions or example
for how to get up and running working with IIS. In theory, this could be a great solution for my
needs.

I'm also surprised that many of the stats packages don't seem to expose their report data in any
sort of structured form, say, in XML. You're stuck either "rolling your own" solution and coding
everything, or working with packages that seem more directed at Desktop users. Not happy with
any of the prebuilt solutions that I can afford.

-KF
"Peter Bromberg [C# MVP]" <pb*******@yahoo.yabbadabbadoo.comwrote in message
news:1B**********************************@micros oft.com...
As an alternative to John's recommendation, you could try hooking the
Application_PreRequestHandlerExecute event in either global.asax or in an
HttpModule that handles the event. You can thus interrogate the request just
before it gets passed to the page processor, and log all the data you want
into your Sql Server.
Peter

--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net


"ke*****@nospam.nospam" wrote:

The New York Times and many other online publications automatically generate
"most popular article" lists that cover, say, the last 24 hours.
>
I am looking for guidance and/or code on the best way to do this in ASP.NET
2 in a way that
>
a) won't crush the server
and
b) won't be surpressed by any of ASP.NET's caching options: you want the
"hits" to still be registered even if someone is clicking through to a
cached instance of the page.
>
What are the common solutions for dealing with this? Is it common to use a
stats package and operate on the data store that's generated by the package,
or is this overkill for this intended purpose?
>
SQL Server is available, and I have full control of the server.
>
-KF
>
>
>