473,320 Members | 1,853 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Large Volume Transaction Load

I have developed 10 functions to look up the surrogate key from different dimension tables. For example,
CREATE FUNCTION BIR_GET_DIM_PROD_ID (p_prod_cde VARCHAR(15), p_date DATE, p_ctry_code CHAR(2))
RETURNS BIGINT
LANGUAGE SQL
BEGIN ATOMIC

DECLARE v_id BIGINT;

SET v_id = (SELECT PROD_ID
FROM HEW_PROD_DIM
WHERE REC_RMOVE_DT_TM IS NULL
AND p_date BETWEEN START_DT AND END_DT
AND PROD_CDE = p_prod_cde
AND CTRY_CDE = p_ctry_code);

IF v_id IS NULL
THEN
SET v_id = -1;
END IF;

RETURN v_id;

END @

Provided that I have already built necessary index on CAMP_CDE and PROD_CDE for the working tables. I need to transform 3 million records to get the necessary surrogate keys for each record. Hence, I wonder which of the following approach has better performance:

Approach 1
Insert all records from one working table to another by once together with 10 functions as follows:

INSERT INTO BIR_WK2_TABLE (REC_ID, CAMP_CDE, CAMP_ID, PROD_CDE, PROD_ID, ... REC_UPDT_DT_TM)
SELECT REC_ID,
CAMP_CDE,
BIR_GET_DIM_CAMP_ID(CAMP_CDE),
PROD_CDE,
BIR_GET_DIM_PROD_ID(PROD_CDE),
....
REC_UPDT_DT_TM
FROM BIR_WK1_TABLE;

Approach 2
Insert all records from one working table to another for 10 times. Each time I only join one dimension table to the working table as follows:

BIR_CAMP_DIM table will be joined for CAMP_ID in the 1st round.

INSERT INTO BIR_WK2_TABLE
(REC_ID, CAMP_CDE, CAMP_ID, PROD_CDE, PROD_ID, ... REC_UPDT_DT_TM)
SELECT a.REC_ID,
a.CAMP_CDE,
b.CAMP_ID,
a.PROD_CDE,
a.PROD_ID,
...
REC_UPDT_DT_TM
FROM BIR_WK1_TABLE A, BIR_CAMP_DIM b
WHERE a.CAMP_CDE = b.CAMP_CDE;

BIR_PROD_DIM table will be joined for PROD_ID in the 2nd round.

INSERT INTO BIR_WK3_TABLE
(REC_ID, CAMP_CDE, CAMP_ID, PROD_CDE, PROD_ID, ... REC_UPDT_DT_TM)
SELECT a.REC_ID,
a.CAMP_CDE,
a.CAMP_ID,
a.PROD_CDE,
b.PROD_ID,
...
REC_UPDT_DT_TM
FROM BIR_WK2_TABLE A, BIR_PROD_DIM b
WHERE a.PROD_CDE = b.PROD_CDE;

Repeat the insert action to join other dimension tables until all surrogate keys are retrieved.
Dec 8 '07 #1
0 1165

Sign in to post your reply or Sign up for a free account.

Similar topics

4
by: Jason Murry | last post by:
I have a camera system (Axis) which stores JPG via FTP 1-10fps. There is also a motion jpg live stream. I am trying to store these images either in JPG or in video format so they can be...
1
by: Philipp K. Janert, Ph.D. | last post by:
Dear All! I am trying to load a relatively large table (about 1 Million rows) into an sqlite table, which is kept in memory. The load process is very slow - on the order of 15 minutes or so. ...
1
by: Tim Nelson | last post by:
**** Post for FREE via your newsreader at post.usenet.com **** I am a newbie try to port my applications to Postgres. I have an application that is bulk loading a table with autocommit off (with...
4
by: Leaf | last post by:
Greetings, I've been reading with interest the threads here on deadlocking, as I'm finding my formerly happy app in a production environment suddenly deadlocking left and right. It started...
4
by: oshanahan | last post by:
Does anyone have ideas on the best way to move large amounts of data between tables? I am doing several simple insert/select statements from a staging table to several holding tables, but because...
5
by: Louis LeBlanc | last post by:
Hey folks. I'm new to the list, and not quite what you'd call a DB Guru, so please be patient with me. I'm afraid the lead up here is a bit verbose . . . I am working on an application that...
7
by: Rich Milburn [MVP] | last post by:
Ok I am not a programmer, I copied some code and very painfully got it working in VB6. I can adjust the volume with waveOutSetVolume on winmm.dll. But I could not seem to be able to figure out how...
4
by: =?Utf-8?B?VzFsZDBuZTc0?= | last post by:
When one architects a new project one of the first steps in the decision is to decide on the layers. (In my opinion anyway) One architecture that I have used before is to go solid OO and create...
11
by: Joseph Geretz | last post by:
I've been looking at two approaches for the maintenance of Session state for a Web Service application. One approach uses the old familiar Session object which I've used in the past for Web...
14
by: mfrsousa | last post by:
hi there, i have a huge large text file (350.000 lines) that i want to import to a MS Acccess Database, of course i don't want to use Access, but do it with C#. i already have tried the...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.