Dealing with large amounts of data

[Digety]

We are looking to store a large amount of user data that will be
changed and accessed daily by a large number of people. We expect
around 6-8 million subscribers to our service with each record being
approximately 2000-2500 bytes. The system needs to be running 24/7
and therefore cannot be shut down. What is the best way to implement
this? We were thinking of setting up a cluster of servers to hold the
information and another cluster to backup the information. Is this
practical?
Also, what software is available out there that can distribute query
calls across different servers and to manage large amounts of query
requests?

Thank you in advance.

Ben

[nib]

Digety wrote:

We are looking to store a large amount of user data that will be
changed and accessed daily by a large number of people. We expect
around 6-8 million subscribers to our service with each record being
approximately 2000-2500 bytes. The system needs to be running 24/7
and therefore cannot be shut down. What is the best way to implement
this? We were thinking of setting up a cluster of servers to hold the
information and another cluster to backup the information. Is this
practical?
Also, what software is available out there that can distribute query
calls across different servers and to manage large amounts of query
requests?

Thank you in advance.

Ben

You need to consult with a profession firm that has experience with this
kind of thing. I wouldn't trust us yahoos! :D

Zach

[David Portas]

This question is really too big for a newsgroup. The obvious question is
that if you want to build and run a 24/7 server-farm for 8 million
subscribers then you'll need a team of architecture, development and
operations staff with plenty of years of experience in high-volume,
high-availability environments... so why not ask them this question, since
they'll be in a better position to understand your requirements?

SQL Server's clustering support is for Failover Clustering so that may be
part of your solution for high availability. In SQL Server, distributed
queries can be implemented using partitioned views for load balancing.

You may find some useful information on Microsoft's scalability site:
http://www.microsoft.com/sql/techinf...calability.asp

--
David Portas
SQL Server MVP
--

[Greg D. Moore \(Strider\)]

"Digety" <be******@hotmail.com> wrote in message
news:82**************************@posting.google.c om...

We are looking to store a large amount of user data that will be
changed and accessed daily by a large number of people. We expect
around 6-8 million subscribers to our service with each record being
approximately 2000-2500 bytes. The system needs to be running 24/7
and therefore cannot be shut down. What is the best way to implement
this? We were thinking of setting up a cluster of servers to hold the
information and another cluster to backup the information. Is this
practical?
Practical? What's your budget? What's your response time requirements?

Clustering ain't cheap.

6-8 million rows isn't all that much btw. What's more important is how much
it changes.

Also, what software is available out there that can distribute query
calls across different servers and to manage large amounts of query
requests?
If you're just doing queries, my guess is you won't need this.

To give you an example, I've got a quad CPU Xeon box (700Mhz) that runs at
about 50% CPU these days (some new code just helped). It INSERTS I think 17
million rows a day (which then get moved to another server overnight.)


Thank you in advance.

Ben

[Digety]

Obviously you can tell that I don't know much about this subject, so
I'm sorry for my ignorance. What types of firms out there can handle
something like this? If you could give me some examples, that would
be great. Thank you so much.

[Ben]

[David Portas]

Have you already chosen SQL Server as your database platform? If so then
contact Microsoft Sales, explain your requirements and ask them to suggest a
vendor in your area. I suggest you also hire someone with experience in
databse systems implementation to liaise with the vendor.

If you don't have any given constraints as to what hardware and software
platform to use then you may want to take some advice from an independent IT
consultant to help you decide on the right technology before you talk to
vendors.

--
David Portas
SQL Server MVP
--

[filesiteguy]

David Portas scratched out in the sand

Have you already chosen SQL Server as your database platform? If so then
contact Microsoft Sales, explain your requirements and ask them to suggest
a vendor in your area. I suggest you also hire someone with experience in
databse systems implementation to liaise with the vendor.

If you don't have any given constraints as to what hardware and software
platform to use then you may want to take some advice from an independent
IT consultant to help you decide on the right technology before you talk
to vendors.

As someone else mentioned, your volume really isn't that bad. SQL Server (or
Oracle, MySQL, PostgreSQL) could handle it on one server very easily. I've
done this many times over on MSSQL and MySQL. Both handle the volume on one
server without a hitch.

If, you're worried about 24/7 operations, however, I'd shy away from
MS-based solutions, as they get pricy quickly and are difficult to
maintain. Though Windows is a decent departmental level solution, I've seen
too many cases were $$$ was thrown at it to make it 24/7 and still had it
fail.

I'd lean more towards a Unix-based system, simply because they're better
designed for datacenter operations.
--
kai - kai at 3gproductions dot com
www.gamephreakz.com || www.filesite.org
"friends don't let friends use windows xp"

[Greg D. Moore \(Strider\)]

"filesiteguy" <ab***@127.0.0.1> wrote in message
news:10*************@corp.supernews.com...

As someone else mentioned, your volume really isn't that bad. SQL Server (or Oracle, MySQL, PostgreSQL) could handle it on one server very easily. I've
done this many times over on MSSQL and MySQL. Both handle the volume on one server without a hitch.

I'd agree with this.

If, you're worried about 24/7 operations, however, I'd shy away from
MS-based solutions, as they get pricy quickly and are difficult to
maintain. Though Windows is a decent departmental level solution, I've seen too many cases were $$$ was thrown at it to make it 24/7 and still had it
fail.
But not this. Our main production SQL Server had over the course of the
past few years a 100% uptime (except for a planned move and a few planned
maintenances).

A lot of 24/7 really goes into planning, Unix, Windows or otherwise.

I'd lean more towards a Unix-based system, simply because they're better
designed for datacenter operations.
--
kai - kai at 3gproductions dot com
www.gamephreakz.com || www.filesite.org
"friends don't let friends use windows xp"

[filesiteguy]

Greg D. Moore (Strider) scratched out in the sand

If, you're worried about 24/7 operations, however, I'd shy away from
MS-based solutions, as they get pricy quickly and are difficult to
maintain. Though Windows is a decent departmental level solution, I've

seen

too many cases were $$$ was thrown at it to make it 24/7 and still had it
fail.

But not this. Our main production SQL Server had over the course of the
past few years a 100% uptime (except for a planned move and a few planned
maintenances).

A lot of 24/7 really goes into planning, Unix, Windows or otherwise.

If you were down due to maintenance, then it isn't 100% uptime. Yes,
planning is VERY important. I just have seen that Windows is better being
planned as a small departmental solution and doesn't fit as an enterprise
system.

Eventually you'll learn. You'll also learn not to use Outlook Express,
someday.

--
kai - kai at 3gproductions dot com
www.gamephreakz.com || www.filesite.org
"friends don't let friends use windows xp"

[David Portas]

> I just have seen that Windows is better being

planned as a small departmental solution and doesn't fit as an enterprise
system.

That tells us more about your experience than about Windows. Your
generalization is belied by the reality of thousands of major enterprises
whose experience apparently differs from yours.

--
David Portas
SQL Server MVP
--

[Greg D. Moore \(Strider\)]

"filesiteguy" <ab***@127.0.0.1> wrote in message
news:10*************@corp.supernews.com...

Greg D. Moore (Strider) scratched out in the sand

If, you're worried about 24/7 operations, however, I'd shy away from
MS-based solutions, as they get pricy quickly and are difficult to
maintain. Though Windows is a decent departmental level solution, I've seen

too many cases were $$$ was thrown at it to make it 24/7 and still had it fail.

But not this. Our main production SQL Server had over the course of the
past few years a 100% uptime (except for a planned move and a few planned maintenances).

A lot of 24/7 really goes into planning, Unix, Windows or otherwise.

If you were down due to maintenance, then it isn't 100% uptime.

Depends on your definition. And your budget. (i.e. whether you have the
budget for the hardware and software solutions that allow 0% downtime
maintenance. Same as in a Unix shop.)

And let's put it this way, the Windows solution has had far better uptime
than the Unix/Oracle solution used in a different division.
Yes,
planning is VERY important. I just have seen that Windows is better being
planned as a small departmental solution and doesn't fit as an enterprise
system.

Eventually you'll learn. You'll also learn not to use Outlook Express,
someday.
And perhaps someday you'll learn to not be so condescending.


--
kai - kai at 3gproductions dot com
www.gamephreakz.com || www.filesite.org
"friends don't let friends use windows xp"