473,783 Members | 2,317 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Separate databases for high/low transaction volumes?

I have an existing database with approx 500,000 rows and accessed by a
few hundred users per day creating approx 1,000 new records per day
plus typical reporting - relatively low volume stuff for SQL Server.

I'm about to add a process that will be importing data daily from
legacy databases and summarizing it for reporting purposes, integrating
it with the existing database. This volume of data will be considerably
higher, perhaps 100,000+ rows per day, which will be deleted once it
has been summarized and the results written to some intermediate
tables.

Is there any concern about mixing different levels of volume within one
database? As I'll be creating lots of rows daily and then deleting them
I was wondering about fragmentation, transaction logging etc. and
whether having this processing in a separate database from the main
application would be 'better'.

Jun 22 '06 #1
3 2103

peterc wrote:
I have an existing database with approx 500,000 rows and accessed by a
few hundred users per day creating approx 1,000 new records per day
plus typical reporting - relatively low volume stuff for SQL Server.

I'm about to add a process that will be importing data daily from
legacy databases and summarizing it for reporting purposes, integrating
it with the existing database. This volume of data will be considerably
higher, perhaps 100,000+ rows per day, which will be deleted once it
has been summarized and the results written to some intermediate
tables.
I guess after summarization it will be far less then 100,000 new rows
per day.
Is there any concern about mixing different levels of volume within one
database? As I'll be creating lots of rows daily and then deleting them
I was wondering about fragmentation, transaction logging etc. and
whether having this processing in a separate database from the main
application would be 'better'.


I would import the data into separate stage database with simple model.
I would insert data there in batches and the insert summary data into
reporting tables of the main database.

Jun 23 '06 #2
peterc (su*******@coop erzone.net) writes:
I have an existing database with approx 500,000 rows and accessed by a
few hundred users per day creating approx 1,000 new records per day
plus typical reporting - relatively low volume stuff for SQL Server.

I'm about to add a process that will be importing data daily from
legacy databases and summarizing it for reporting purposes, integrating
it with the existing database. This volume of data will be considerably
higher, perhaps 100,000+ rows per day, which will be deleted once it
has been summarized and the results written to some intermediate
tables.

Is there any concern about mixing different levels of volume within one
database? As I'll be creating lots of rows daily and then deleting them
I was wondering about fragmentation, transaction logging etc. and
whether having this processing in a separate database from the main
application would be 'better'.


It's difficult to say, because the answer depends on many more things than
just the volumes. If you go with two databases, is there lookup data that
needs to be both databases and maintained separately? What about access
to the databases, would you need to maintain users in both databases?

If the other process is completely independent of the stuff which is
in the database now, there is all reason to keep it separate. But if
there is interaction between the two sets of data, I would hesitate to
make the split.

Of course, the database that gets these 100000 rows/day, needs more
attention for maintenance jobs, mainly defragmenting. As for the
transaction log, I get the impression from your description that you
could make it fine with simple recovery. But if you need full recovery,
you will need to back up the transaction log more often.
--
Erland Sommarskog, SQL Server MVP, es****@sommarsk og.se

Books Online for SQL Server 2005 at
http://www.microsoft.com/technet/pro...ads/books.mspx
Books Online for SQL Server 2000 at
http://www.microsoft.com/sql/prodinf...ons/books.mspx
Jun 23 '06 #3
Thanks for the replies - if I can make everything independent (lookup
tables etc.) I think I'll use a separate database.

Peter

Jun 24 '06 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
6139
by: Jeffrey Kilpatrick | last post by:
I have a SQL 6.5 database that we parse some data into everyday using an access program. All this was devises and setup by a programmer that I can't get in contact with anymore and it has actually run for about five years without a hickup! But just a few days ago our parsing program just stops dead before completing and I did get this error message. "exportaLLdataToSQLifnoerror(): number 3146 Description- odbd- call failed. Can't...
3
2092
by: RugbyCoach | last post by:
I'm looking for a tool that can extract only the schema from a database in a form that can be used to generate that schema in another empty database. This is to facilitate our disaster recovery processes where we need the objects only, not the data, and need to replicate this to our disaster recovery site over the WAN. There are plenty of tools that can handle a single database, but does anyone know of any tools that could handle...
18
2091
by: Ron Johnson | last post by:
On Mon, 2003-08-25 at 16:28, Gregory S. Williamson wrote: > One of our sysads sent this link ... wondering if there is any comment on it from the world of actual users of linux and a database. > > <http://story.news.yahoo.com/news?tmpl=story&cid=1738&ncid=738&e=9&u=/zd/20030825/tc_zd/55311> "Weak points include lack of available tools, ease of use and ease of installation" Sounds like he needs point-and-drool tools...
12
2226
by: John Sidney-Woollett | last post by:
I have to convert an java web application currently using an Oracle DB back end to one using a Postgres backend. In Oracle much of the application logic is abstracted away from the java middleware layer using stored procedures within the Oracle database. There are certain features in Oracle that appear to be missing from Postgres which are causing us some concern, and we wondered how much we're going to have to butcher the backend and db...
7
1549
by: elein | last post by:
What is the linux and/or postgres limitation for very large databases, if any? We are looking at 6T-20T. My understanding is that if the hardware supports it, then it can be done in postgres. But can hardware support that? --elein ============================================================ elein@varlena.com Varlena, LLC www.varlena.com
7
24686
by: Altemir | last post by:
I want to create an UPDATE statement that updates a value in a table in database A based on the results of an inner join between tables each residing in deifferent databases. What is the correct syntax for doing this? The following should give you an idea of what I'm trying to do: UPDATE A.dbo.tblCars SET A.dbo.tblCars.Car = 'Ferrari'
2
3792
by: Christian Stooker | last post by:
Part one: ====== Hi ! I want to use SQLite database like the FireBird database: with big isolation level. What's that meaning ? I have an application that periodically check some input directory,
3
4566
by: LurfysMa | last post by:
I would like to hear opinions on the tradeoffs of putting the tables, forms, and queries for several related datasets in separate databases vs one combined database. I am working on an application that will have a number of "subjects". The subjects have a number of commonalities, but are not identical. I am hoping that I will discover that the subjects fall into a few "types" and a common database structure can be used for each type. ...
0
3109
by: jags_32 | last post by:
Hello We use MFG-PRO as our ERP system which in turn uses Progress databases. In the old version of SQL 2000, using DTS packages, we used to set the code page via command prompts and execute DTS packages within that command prompt to fetch data from our double byte ERP databases. In SSIS, we are able to connect and fecth data from NON Double Byte databases, however, when we set the code page first and execute via a command prompt the...
0
9643
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
10313
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10147
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
9946
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
7494
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5378
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5511
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4044
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
3643
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.