OLAP and/or data mining?

Framework fan

Hello,

If I wrote the next ebay (yes I know, yawn-snore) and I had a database
with 5 million auction items in it, what would be a really good
strategy to get a search done very quickly? Would it involve
something called OLAP and/or "data mining"? The only technology I am
familiar with is simply SQL Server databases with stored procedures.
I think I'd be guessing correctly and say that this technology simply
wouldn't be fast enough *on it's own* to do super fast queries against
massive amounts of data.

Any insights would be of great interest. Thanks.

-Frameworker.

Jul 20 '05 #1

Greg D. Moore \(Strider\)

"Framework fan" <te**************@hotmail.com> wrote in message
news:f1**************************@posting.google.c om...

Hello,

If I wrote the next ebay (yes I know, yawn-snore) and I had a database
with 5 million auction items in it, what would be a really good
strategy to get a search done very quickly? Would it involve
something called OLAP and/or "data mining"? The only technology I am
familiar with is simply SQL Server databases with stored procedures.
I think I'd be guessing correctly and say that this technology simply
wouldn't be fast enough *on it's own* to do super fast queries against
massive amounts of data.
OLAP is exactly what you'd want.

Data mining may be useful post-auction to see who bought what, what things
need more promotion etc.

I don't know why you don't think SQL Server wouldn't be fast enough for a
small setup like 5 million auctions.

I have a database with 14 million rows that's probably as complex as an
auction database and most queries are a couple of seconds or less. (some
are longer because of some fairly complex floating point math that has to be
done along the way.)

I have another database that handles at least that many inserts per day w/o
a major problem. And this is all on 3+ y.o. equipment.

Give me a quad Xeon MP box and the right disk subsystem and this thing could
scream.


Any insights would be of great interest. Thanks.

-Frameworker.

Jul 20 '05 #2

mountain man

"Greg D. Moore (Strider)" <mo*****@greenms.com> wrote in message
news:fC******************@twister.nyroc.rr.com...

"Framework fan" <te**************@hotmail.com> wrote in message
news:f1**************************@posting.google.c om...

Hello,

If I wrote the next ebay (yes I know, yawn-snore) and I had a database
with 5 million auction items in it, what would be a really good
strategy to get a search done very quickly? Would it involve
something called OLAP and/or "data mining"? The only technology I am
familiar with is simply SQL Server databases with stored procedures.

Using stored procedures as component building blocks
you can create all forms of data mining and analytical
processing applications.

I think I'd be guessing correctly and say that this technology simply
wouldn't be fast enough *on it's own* to do super fast queries against
massive amounts of data.
OLAP is exactly what you'd want.

Data mining may be useful post-auction to see who bought what, what things
need more promotion etc.

I don't know why you don't think SQL Server wouldn't be fast enough for a
small setup like 5 million auctions.

Seconded.

If you write your procs in the right manner, and maintain
the data integrity in your database then you should have
null performance problems.
I have a database with 14 million rows that's probably as complex as an
auction database and most queries are a couple of seconds or less. (some
are longer because of some fairly complex floating point math that has to be done along the way.)

I have another database that handles at least that many inserts per day w/o a major problem. And this is all on 3+ y.o. equipment.

Give me a quad Xeon MP box and the right disk subsystem and this thing could scream.

Any insights would be of great interest. Thanks.

-Frameworker.

Jul 20 '05 #3

Framework fan

Hi Greg,

I have one follow up question please:

I'd like to incorporate OLAP in to my design. Do I go ahead and
create a sensibly normalised relational database design, with many
dozens of stored procedures to query the tables in "the usual manner",
*then* add on OLAP as a kind of "bolt on", or does OLAP technology
require a totally different database/stored procedures design strategy
right from the word go?

Thanks!

- Frameworker.
"Greg D. Moore \(Strider\)" <mo*****@greenms.com> wrote in message news:<fC******************@twister.nyroc.rr.com>.. .

"Framework fan" <te**************@hotmail.com> wrote in message
news:f1**************************@posting.google.c om...

Hello,

If I wrote the next ebay (yes I know, yawn-snore) and I had a database
with 5 million auction items in it, what would be a really good
strategy to get a search done very quickly? Would it involve
something called OLAP and/or "data mining"? The only technology I am
familiar with is simply SQL Server databases with stored procedures.
I think I'd be guessing correctly and say that this technology simply
wouldn't be fast enough *on it's own* to do super fast queries against
massive amounts of data.

OLAP is exactly what you'd want.

Data mining may be useful post-auction to see who bought what, what things
need more promotion etc.

I don't know why you don't think SQL Server wouldn't be fast enough for a
small setup like 5 million auctions.

I have a database with 14 million rows that's probably as complex as an
auction database and most queries are a couple of seconds or less. (some
are longer because of some fairly complex floating point math that has to be
done along the way.)

I have another database that handles at least that many inserts per day w/o
a major problem. And this is all on 3+ y.o. equipment.

Give me a quad Xeon MP box and the right disk subsystem and this thing could
scream.

Any insights would be of great interest. Thanks.

-Frameworker.

Jul 20 '05 #4

Framework fan

I'm a speed freak! If I can get a query to run .25 seconds faster
using OLAP technology, I'm all for it. :o)
"mountain man" <hobbit@southern_seaweed.com.op> wrote in message news:<f3*****************@news-server.bigpond.net.au>...

"Greg D. Moore (Strider)" <mo*****@greenms.com> wrote in message
news:fC******************@twister.nyroc.rr.com...

"Framework fan" <te**************@hotmail.com> wrote in message
news:f1**************************@posting.google.c om...

Hello,

If I wrote the next ebay (yes I know, yawn-snore) and I had a database
with 5 million auction items in it, what would be a really good
strategy to get a search done very quickly? Would it involve
something called OLAP and/or "data mining"? The only technology I am
familiar with is simply SQL Server databases with stored procedures.

Using stored procedures as component building blocks
you can create all forms of data mining and analytical
processing applications.

I think I'd be guessing correctly and say that this technology simply
wouldn't be fast enough *on it's own* to do super fast queries against
massive amounts of data.

OLAP is exactly what you'd want.

Data mining may be useful post-auction to see who bought what, what things
need more promotion etc.

I don't know why you don't think SQL Server wouldn't be fast enough for a
small setup like 5 million auctions.

Seconded.

If you write your procs in the right manner, and maintain
the data integrity in your database then you should have
null performance problems.

I have a database with 14 million rows that's probably as complex as an
auction database and most queries are a couple of seconds or less. (some
are longer because of some fairly complex floating point math that has to

be

done along the way.)

I have another database that handles at least that many inserts per day

w/o

a major problem. And this is all on 3+ y.o. equipment.

Give me a quad Xeon MP box and the right disk subsystem and this thing

could

scream.

Any insights would be of great interest. Thanks.

-Frameworker.

Jul 20 '05 #5

Greg D. Moore \(Strider\)

"Framework fan" <te**************@hotmail.com> wrote in message
news:f1**************************@posting.google.c om...

I'm a speed freak! If I can get a query to run .25 seconds faster
using OLAP technology, I'm all for it. :o)
I think you completely misunderstand what OLAP is or does. (And I mispoke
below (forgive me, I was tired). You want an OLTP system, not an OLAP.

OLAP and getting better performance are orthogonal to each other. i.e.
using OLAP is not a way to increase performance.


"mountain man" <hobbit@southern_seaweed.com.op> wrote in message

news:<f3*****************@news-server.bigpond.net.au>...

"Greg D. Moore (Strider)" <mo*****@greenms.com> wrote in message
news:fC******************@twister.nyroc.rr.com...

"Framework fan" <te**************@hotmail.com> wrote in message
news:f1**************************@posting.google.c om...
> Hello,
>
> If I wrote the next ebay (yes I know, yawn-snore) and I had a database > with 5 million auction items in it, what would be a really good
> strategy to get a search done very quickly? Would it involve
> something called OLAP and/or "data mining"? The only technology I am > familiar with is simply SQL Server databases with stored procedures.

Using stored procedures as component building blocks
you can create all forms of data mining and analytical
processing applications.

> I think I'd be guessing correctly and say that this technology simply > wouldn't be fast enough *on it's own* to do super fast queries against > massive amounts of data.

OLAP is exactly what you'd want.

Data mining may be useful post-auction to see who bought what, what things need more promotion etc.

I don't know why you don't think SQL Server wouldn't be fast enough for a small setup like 5 million auctions.

Seconded.

If you write your procs in the right manner, and maintain
the data integrity in your database then you should have
null performance problems.

I have a database with 14 million rows that's probably as complex as an auction database and most queries are a couple of seconds or less. (some are longer because of some fairly complex floating point math that has

to be

done along the way.)

I have another database that handles at least that many inserts per

day w/o

a major problem. And this is all on 3+ y.o. equipment.

Give me a quad Xeon MP box and the right disk subsystem and this thing

could

scream.
>
> Any insights would be of great interest. Thanks.
>
> -Frameworker.

Jul 20 '05 #6