hi
For an unavoidable reason, I have to use row-by-row processing
(update) on a temporary table to update a history table every day.
I have around 60,000 records in temporary table and about 2 million in
the history table.
Could any one please suggest different methods to imporve the runtime
of the query?
Would highly appreciate! 9 4564
Is the row-by-row processing done in a cursor? Must you update exactly one
row at a time (if so, why?) or would it be acceptable to update 2,3 or 50
rows at a time?
You can use SET ROWCOUNT and a loop to fine-tune the batch size of rows to
be updated. Bigger batches should improve performance over updating single
rows.
SET ROWCOUNT 50
WHILE 1=1
BEGIN
UPDATE SomeTable
SET ...
WHERE /* row not already updated */
IF @@ROWCOUNT=0
BREAK
END
SET ROWCOUNT 0
--
David Portas
SQL Server MVP
--
Is the row-by-row processing done in a cursor? Must you update exactly one
row at a time (if so, why?) or would it be acceptable to update 2,3 or 50
rows at a time?
You can use SET ROWCOUNT and a loop to fine-tune the batch size of rows to
be updated. Bigger batches should improve performance over updating single
rows.
SET ROWCOUNT 50
WHILE 1=1
BEGIN
UPDATE SomeTable
SET ...
WHERE /* row not already updated */
IF @@ROWCOUNT=0
BREAK
END
SET ROWCOUNT 0
--
David Portas
SQL Server MVP
--
"Muzamil" <mu*****@hotmai l.com> wrote in message
news:5a******** *************** ***@posting.goo gle.com... hi
For an unavoidable reason, I have to use row-by-row processing (update) on a temporary table to update a history table every day. I have around 60,000 records in temporary table and about 2 million in the history table.
Not much you can do if you absolutely HAVE to do row-by-row updating.
You might want to post DDL, etc. so others can take a crack at it. I've
seen many times someone will say, "I have to use a cursor", "I have to
update one row at a time" and then someone posts a much better/faster
solution.
Also, how are you handling transactions? Explicitly or implicitely? If
you're doing them implicitely, are you wrapping each update in its own, or
can up batch say 20 updates?
Finally, where's your log files? Separate physical drives? Could any one please suggest different methods to imporve the runtime of the query?
Would highly appreciate!
Hi
Thanks for your reply.
The row-by-row update is mandatory becuase the leagacy system is
sending us the information such as "Add", "Modify" or "delete" and
this information HAS to be processed in the same order otherwise we'll
get the erroneous data.
I know it's a dumb way of doing things but this is what our and their
IT department has chosen to be correct way of action after several
meetings. Hence the batch idea will not work here.
I am not using Cursors, instead I am using the loop based on the
primary key.
The log files are on different drives.
I've also tried using "WITH (ROWLOCK)" in the update statement but
it's not helping much.
Can you please still throw in some idea? Would be great help!
Thanks
"Greg D. Moore \(Strider\)" <mo************ ****@greenms.co m> wrote in message news:<tO******* ************@tw ister.nyroc.rr. com>... "Muzamil" <mu*****@hotmai l.com> wrote in message news:5a******** *************** ***@posting.goo gle.com... hi
For an unavoidable reason, I have to use row-by-row processing (update) on a temporary table to update a history table every day. I have around 60,000 records in temporary table and about 2 million in the history table.
Not much you can do if you absolutely HAVE to do row-by-row updating.
You might want to post DDL, etc. so others can take a crack at it. I've seen many times someone will say, "I have to use a cursor", "I have to update one row at a time" and then someone posts a much better/faster solution.
Also, how are you handling transactions? Explicitly or implicitely? If you're doing them implicitely, are you wrapping each update in its own, or can up batch say 20 updates?
Finally, where's your log files? Separate physical drives?
Could any one please suggest different methods to imporve the runtime of the query?
Would highly appreciate!
Muzamil (mu*****@hotmai l.com) writes: The row-by-row update is mandatory becuase the leagacy system is sending us the information such as "Add", "Modify" or "delete" and this information HAS to be processed in the same order otherwise we'll get the erroneous data.
Ouch. Life is cruel, sometimes.
I wonder what possibilities there could be to find parallel streams,
that is updates that could be performed independently. Maybe you
can modify 10 rows at a time then. But it does not sound like a very
easy thing to do.
Without knowing the details of the system, it is difficult to give
much advice. But any sort of pre-aggregation you can do, is probably
going to pay back.
--
Erland Sommarskog, SQL Server MVP, so****@algonet. se
Books Online for SQL Server SP3 at http://www.microsoft.com/sql/techinf...2000/books.asp
Details of the system:
The leagcy system sends us records flagged with "Add", "modify" or
"delete".
The purpose of these flags is self-explnatory. But the fun began when
we noticed that within same file , legacy system sends us "Add" and
then "Modify". Thus, we were left with no other option except to do
row-by-row processing.
We came up with the following logic:
a) If record‘s StatusFlag is ‘A' and record‘s key does not exist in
DataWareHouse's Table, then the record is inserted into
DataWareHouse's Table.
b) If record‘s StatusFlag is ‘A', but record‘s key exists in
DataWareHouse's Table, then the record is marked as invalid and will
be inserted into InvalidTable..
c) If record‘s StatusFlag is ‘M' and record‘s key exists in
DataWareHouse's Table and record is active, then the corresponding
record in DataWareHouse's Table will be updated.
d) If record‘s StatusFlag is ‘M' and record‘s key exists in
DataWareHouse's Table but record is inactive, then the record is
marked as invalid and will be inserted into InvalidTable.
e) If record‘s StatusFlag is ‘M' and record‘s key does not exist in
DataWareHouse's Table, then the record is marked as invalid and will
be inserted into InvalidTable.
f) If record‘s StatusFlag is ‘D' and record‘s key exists in
DataWareHouse's Table and record is active, then the corresponding
record in DataWareHouse's Table will be updated as inactive.
g) If record‘s StatusFlag is ‘D' and record‘s key exists in
DataWareHouse's Table but record is inactive, then the record is
marked as invalid and will be inserted into InvalidTable.
h) If record‘s StatusFlag is ‘D' and record‘s key does not exist in
DataWareHouse's Table, then the record is marked as invalid and will
be inserted into InvalidTable.
This logic takes care of ALL the anomalies we were facing before but
at the cost of long processing time.
I await your comments.
Thanks
Erland Sommarskog <so****@algonet .se> wrote in message news:<Xn******* **************@ 127.0.0.1>... Muzamil (mu*****@hotmai l.com) writes: The row-by-row update is mandatory becuase the leagacy system is sending us the information such as "Add", "Modify" or "delete" and this information HAS to be processed in the same order otherwise we'll get the erroneous data.
Ouch. Life is cruel, sometimes.
I wonder what possibilities there could be to find parallel streams, that is updates that could be performed independently. Maybe you can modify 10 rows at a time then. But it does not sound like a very easy thing to do.
Without knowing the details of the system, it is difficult to give much advice. But any sort of pre-aggregation you can do, is probably going to pay back.
Muzamil (mu*****@hotmai l.com) writes: Details of the system: The leagcy system sends us records flagged with "Add", "modify" or "delete". The purpose of these flags is self-explnatory. But the fun began when we noticed that within same file , legacy system sends us "Add" and then "Modify". Thus, we were left with no other option except to do row-by-row processing. We came up with the following logic:
Hm, you might be missing a few cases. What if you get an Add, and record
exists in DW, but is marked inactive? With your current logic, the
input record moved to the Invalid table.
And could that feediug system be as weird as to send Add, Modify, Delete,
and Add again? Well, for a robust solution this is what we should assume.
It's a tricky problem, and I was about to defer the problem, when I
recalled a solution that colleague did for one of our stored procedures.
The secret word for tonight is bucketing! Assuming that there are
only a couple of input records for each key value, this should be
an excellent solution. You create buckets, so that each bucket has
at most one row per key value. Here is an example on how to do it:
UPDATE inputtbl
SET bucket = (SELECT count(*)
FROM inputtbl b
WHERE a.keyval = b.keyval
AND a.rownumber < b.rownumber) + 1
FROM inputtbl a
input.keyval is the keys for the records in the DW table. Rownumber
is a column which as describes the processing order. I assume that
you have such a column.
So now you can iterate over the buckets, and for each bucket, you can do
set- based processing. You still have to iterate, but instead over 60000
rows, only over a couple of buckets.
--
Erland Sommarskog, SQL Server MVP, so****@algonet. se
Books Online for SQL Server SP3 at http://www.microsoft.com/sql/techinf...2000/books.asp
I think I was not articulate enough to convey the logic properly.
Anyways, thanks to everyone for your help.
By using the ROWLOCK and proper indexes, I was ale to reduce the time considerably.
Erland Sommarskog <so****@algonet .se> wrote in message news:<Xn******* *************@1 27.0.0.1>... Muzamil (mu*****@hotmai l.com) writes: Details of the system: The leagcy system sends us records flagged with "Add", "modify" or "delete". The purpose of these flags is self-explnatory. But the fun began when we noticed that within same file , legacy system sends us "Add" and then "Modify". Thus, we were left with no other option except to do row-by-row processing. We came up with the following logic:
Hm, you might be missing a few cases. What if you get an Add, and record exists in DW, but is marked inactive? With your current logic, the input record moved to the Invalid table.
And could that feediug system be as weird as to send Add, Modify, Delete, and Add again? Well, for a robust solution this is what we should assume.
It's a tricky problem, and I was about to defer the problem, when I recalled a solution that colleague did for one of our stored procedures. The secret word for tonight is bucketing! Assuming that there are only a couple of input records for each key value, this should be an excellent solution. You create buckets, so that each bucket has at most one row per key value. Here is an example on how to do it:
UPDATE inputtbl SET bucket = (SELECT count(*) FROM inputtbl b WHERE a.keyval = b.keyval AND a.rownumber < b.rownumber) + 1 FROM inputtbl a
input.keyval is the keys for the records in the DW table. Rownumber is a column which as describes the processing order. I assume that you have such a column.
So now you can iterate over the buckets, and for each bucket, you can do set- based processing. You still have to iterate, but instead over 60000 rows, only over a couple of buckets.
Muzamil (mu*****@hotmai l.com) writes: I think I was not articulate enough to convey the logic properly. Anyways, thanks to everyone for your help. By using the ROWLOCK and proper indexes, I was ale to reduce the time considerably.
Good indexes is always useful, and of course for iterative processing
it is even more imperative, since the cost a less-than-optimal plan
is multiplied.
I'm just curious, would my bucketing idea be applicable to your problem?
It should give you even more speed, but if you have good-ebough now, there
is of course no reason to spend more time on it.
--
Erland Sommarskog, SQL Server MVP, so****@algonet. se
Books Online for SQL Server SP3 at http://www.microsoft.com/sql/techinf...2000/books.asp This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Arni Snorri Eggertsson |
last post by:
Hi
I am trying to design an IO subsystem for my SQL Server and for that I
need to try and predict IO activity on each table in my MSSQL
Database. My idea is to move the hottest tables into special disk
subsystem away from the less hotter tables. So far I have gathered
that we have three tables more hot than the others but I have no
feeling on ratio on how hot each is and how much activity is on the
less hotter tables. I need to...
|
by: NathanG |
last post by:
Hi,
I've run the (SQL2K) Index Tuning Wizard against our 50GB database and
it has recommended changes that will produce a "-3506%" improvement iin
performance.
The recommendations include a lot of indexes to be dropped and a few
new ones to be created, in the Analysis - Query Cost Report screen all
of the indexes shown have a zero or negative percent improvement (e.g.
-96133%).
|
by: Tober |
last post by:
Anyone happen to know of any good books on performance tuning for
MS-SQL 2000? I am currently building a Cognos datawarehouse and would
like to ensure that the db is tuned for peak performance.
Tober
"...when you have eliminated the impossible, whatever remains, however
improbable, must be the truth."
Sherlock Holmes
|
by: pheonix1t |
last post by:
hello,
I've been assigned to do performance tuning on an SQL2000 database
(around 10GB in size, several instances).
So far, I see a single RAID5 array, 4CPU (xeon 700MHZ), 4GB RAM.
I see the raid5 as a bottleneck. I'd setup a raid 10 and seperate the
logs, database and OS(win2k).
The one thing that was a bit odd to me was that I was told this place
doesn't use indexes. The company is a house builder. They are pretty
|
by: George McLean |
last post by:
Hello,
I am a bit confused on tuning parameters for DB2.
I have a VC++ application using ADO for database connectivity. The
application uses server side cursors. For each transaction set the
application opens the connection, issues BeginTrans, performs the updates,
commits or rolls back the updates and closes the connection. I am using MDAC
2.7
| |
by: Fusheng Wang |
last post by:
Hi,
I have an insert intensive database, and it seems that the performance is a
problem.
Any suggestions on performance tuning on insert performance?
Thanks a lot!
Frank
|
by: sacha.prins |
last post by:
Hi,
I read a lot about DB2 INSERT performance here. I have a nice story as
well.
The thing is, I work on 2 installations of DB2 (on completely different
locations) which run on different (but same generation) hardware.
Benchmarking the disk throughput and CPU basically amounts to the same
figures (+/- 10%).
|
by: Jeff S |
last post by:
I'm looking for guidance (tutorials, backgrounders, tips, or otherwise) on
measuring the performance of ASP.NET applications. I'm specifically
interested in acquiring the capability of generating objective data that
quantifies the effects of performance tuning efforts (e.g., performance
measures before and after implementing caching). I want to be able to
demonstrate cases where performance tuning efforts resulted in measurably
faster...
|
by: abcd |
last post by:
I am using one asp web page with code behind and using C#. The page is very
slow when first hit. some time doesnt display but if I wait and request the
page it is displayed. What could be the wrong. Do I missing any caching,
performance tuning. Also I noticed that when I request the page it goes
twice into Page_Load?
thanks in advance
|
by: atlaste |
last post by:
Hi,
I'm currently developing an application that uses a lot of
computational power, disk access and memory caching (to be more exact:
an information retrieval platform). In these kind of applications the
last thing that remains is bare performance tuning.
So for example, you can do an 'if then else' on a bit like a 'case/
switch', an 'if/then/else' and as a multiplication with a static
buffer. Or, you can do sorting with an inline...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| |
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
| |
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| |