473,769 Members | 6,653 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

three table insert

I'm trying to insert data for a real-time application that demands
less than 3 second receipt-to-commit time, has a relatively large
(500,000 records/hr) data rate, and requires access times on the order
of 10-15 seconds. We're storing about 24 hours worth of data on a
rolling basis, so I've partitioned the table by hour. Three of the
columns are (separately) indexed.

I know I can do this by piping data through sqlldr into the live table
as long as partitions are small enough to keep the index modifications
from becoming too taxing. However, I'd like to keep my hardware
requirements to a minimum, since I have about 100 of these streams in
all, and would like to avoid spending $10M on hardware to brute-force
this. So I cooked up a scheme I thought would save on hardware.

I know if I can direct load data into an offline staging table it's
considerably more efficient, but in that case I won't be able to
satisfy my 3 second receipt-to-commit requirement unless my partitions
are impractically small.

I plan to have a partitioned, unindexed table to receive data the
"loading" table). Once I get enough data for a full hour, I'll
exchange the partition with an unpartitioned table's data segment (the
"staging" table). Then I'll build the indexes I need for the
destination ("live") table, and exchange the newly indexed table into
a partition in the "live" table.

The idea here is I can do indexed searches on 23 hours worth of data,
while being able to direct-load my source data without recalculating
indexes. The reason this will work for me is actual _access_ to this
data is relatively uncommon, so I think I would rather deal with a
full table scan on the "loading" table every once in awhile than take
the performance hit from updating indexes every second.

I plan to create a view to access all three tables at once. I have a
couple of questions, though:

1) Can you access a table while you're building indexes? It seems
like I should be able to do non-indexed searches of tables while
they're being indexed. Is that true?

2) If I'm selecting from a table, does the select block the partition
exchange? If it doesn't, is the result set determinate?
3) What I really want to know, with all this table indexing and data
segemnt swapping, is what are the chances some of my data is going to
fall through the cracks?
Jul 19 '05 #1
0 1801

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
72940
by: MEM | last post by:
hello, I'm need some help with some sql to return some data...obviously ;) I have 3 tables. tblMgr, tblSup, and tblRep. I'd like to get all the names of the reps that are assigned to a particular manager. Here is a simplified exmaple of table setup: tblRep
4
6594
by: Gaz | last post by:
Hi, I need to have a table nested within another table. The tables are alongside each other visually speaking, and the nested table (on the right) can vary in size. My problem is that when the nested table has more rows than the first table, the first table (on the left) pads out its rows so that the table matches the height of the nested table. I don't want this it happen.. I just want the nested table to be longer than the first...
14
4300
by: serge | last post by:
I have a scenario where two tables are in a One-to-Many relationship and I need to move the data from the Many table to the One table so that it becomes a One-to-One relationship. I need to salvage the records from the many table and without going into detail, one of the reasons I can't do the opposite as there are records in the ONE table that I need to keep even if they don't have any child records in the MANY table. Below I created...
0
1831
by: WindAndWaves | last post by:
Hi Gurus Here is another question that I have been struggling with for over a year. Basically, I would like to allow users to enter data into three tables in at the same time. The reason it should all be happening on one form is that the data comes in in chuncks and that the users do not like switching between windows (forms) too many times. To enter data into two tables (e.g. organisations and contacts) at the same
6
4193
by: sql_server_user | last post by:
I'm trying to copy all 440 million rows from one table in my SQL Server 2005 db to another table with a different clustering scheme. After a few test inserts that were successful (up to a million rows in 20 seconds with no problem), I crossed my fingers and went whole-hog with the naive "insert into dest_table (column1, column2, ...) select column1, column2, ... from source_table" but of course it ran out of space in *both* the db log file...
1
2978
by: Matik | last post by:
Hey, First, sorry if this post appear twice, because, I can not find my post recently send, trying to post it once again. I'm out of ideas, so, I thought, will search help here again :( I'm trying to prepare a view for ext. app. This is in normal cases very easy, but what if the view structure should be dynamic?! Here is my point (I will siplify the examples).
2
1574
by: robin1983 | last post by:
HI, I have a problem with sql query.. Actually, i have three different table eg. table1, table2, and table3. All the three table have same field name. Actually, my problem is that, whenever I insert any data/value in any table, the data should be entryed to other remaining table too. I don't know whether its possible or not. If its possible plz help me to find out the solution. thanking you...
0
1332
by: ShirishKumar | last post by:
I have three tables 1) Country - columns (PKCountryId(PrimaryKey),CountryName) 2)State - Columns (PKStateId(PrimaryKey),StateName,FKCountryId(ForeignKey)) 3)Address Details -Columns(PKAddressId(PrimaryKey),......,.......,FKCountryId(ForeignKey),FKStateId(ForeignKey),..................) I developed relationship like this. I have three relations
4
2337
by: Yogesh Sharma | last post by:
create table stuinfo(roll int,name varchar(10)) insert into stuinfo values(1,'sonia') insert into stuinfo values(2,'usha') select * from stuinfo create table marks(roll int,grade varchar) insert into marks values(1,'A') insert into marks values(2,'B') select * from marks
0
9589
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
10049
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
8873
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
7413
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
6675
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5309
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5448
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
3565
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2815
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.