To any and all;
I have a very large table (16Mil+ records), for which I want to delete
about 8 Million records. There are a few indexes on this table.
Is there a way that I can run either a query or a series of queries
that will run against each record and delete based on criteria (date)?
If I do a single DELETE query, it will take forever, lock the table,
and my app that runs against it will stop INSERTING, which is bad.
If I do a cursor, I think it locks the table also, so that won't do,
right?
Any help would be appreciated.
Glenn Dekhayser
Contentcatcher.com 6 3962
On 15 Feb 2005 13:19:01 -0800, "gdekhayser" <gd********@voyantinc.com> wrote: To any and all;
I have a very large table (16Mil+ records), for which I want to delete about 8 Million records. There are a few indexes on this table.
Is there a way that I can run either a query or a series of queries that will run against each record and delete based on criteria (date)? If I do a single DELETE query, it will take forever, lock the table, and my app that runs against it will stop INSERTING, which is bad.
If I do a cursor, I think it locks the table also, so that won't do, right?
Any help would be appreciated.
Glenn Dekhayser Contentcatcher.com
With a table that size, have you thought of partitioning it by date? Then,
locks would be held only on the partition(s) that overlap the date range you
are deleting.
Right, but I need to deal with the records in place before I can do
that....don't I? Once I get the table down in size to something
rational, then I can partition it....
gdekhayser (gd********@voyantinc.com) writes: I have a very large table (16Mil+ records), for which I want to delete about 8 Million records. There are a few indexes on this table.
Is there a way that I can run either a query or a series of queries that will run against each record and delete based on criteria (date)? If I do a single DELETE query, it will take forever, lock the table, and my app that runs against it will stop INSERTING, which is bad.
If I do a cursor, I think it locks the table also, so that won't do, right?
First of all, if there is no index on this date column, you can never
avoid the table lock.
If there is a clustered index on the date column, you should be able
to say DELETE tbl "WHERE date < @somedateinthepast" and still have
your insertes coming through. Then again, if you delete day by day,
it may still be leaner.
If there is a non-clustered index on date, it depends on how many
rows there are per date. If there are too many rows per date, a
DELETE per date could table-scan. You could force an index, but
plenty of rows would be locked, for some time.
--
Erland Sommarskog, SQL Server MVP, es****@sommarskog.se
Books Online for SQL Server SP3 at http://www.microsoft.com/sql/techinf...2000/books.asp
For the future, you should consider partitioning , as deleting n
records is o(n) while dropping a table (or truncating it) is O(1).
For now , I would recommend deleting with rowcount< some number ,
probably in the low thousands, which should leave the table in a fairly
operational state while you do it.
Of course after such a massive delete, the table will be in a terribly
fragmented state internally, so the next step is to rebuild the
indexes.
another alternative is to create another table with the same schema but
without the indexes, insert into it the rows that you do want to keep
(the complement of the 'to be deleted') and then build the indexes. you
can then drop the old table and rename the new one.
hope this helps
Tzvika
gdekhayser wrote: To any and all;
I have a very large table (16Mil+ records), for which I want to
delete about 8 Million records. There are a few indexes on this table.
Is there a way that I can run either a query or a series of queries that will run against each record and delete based on criteria
(date)? If I do a single DELETE query, it will take forever, lock the table, and my app that runs against it will stop INSERTING, which is bad.
If I do a cursor, I think it locks the table also, so that won't do, right?
Any help would be appreciated.
Glenn Dekhayser Contentcatcher.com
Erland- I wish I would have read your post earlier. I would have
thought that removing the indexes would make the delete faster. I
goofed and left the 4 indexes in place, the date one wasn't clustered.
I really wouldn't have cared about the row locks if I forced an index,
as they were going to be deleted anyway and not accessed.
I ended up just performing the delete- and watching my logs grow REAL
large.
I now have 3 log files where I really only want one- is there a way to
condense back into one log file when you're split into 3?
gdekhayser (gd********@voyantinc.com) writes: Erland- I wish I would have read your post earlier. I would have thought that removing the indexes would make the delete faster.
Yeah, removing the indexes not useful for the DELETE would have speedied
things up a little. It may not be a good idea to drop the clustered index
though.
I ended up just performing the delete- and watching my logs grow REAL large.
I now have 3 log files where I really only want one- is there a way to condense back into one log file when you're split into 3?
Well, you can say ALTER DATABASE REMOVE FILE, but you would have to
truncate the log first. Maybe the best is to use WITH TRUNCATE_ONLY
and the take a full backup of the database. I guess that then you
should be able to remove some of the files.
But I will have to admit that I have never had reason to drop a log
file, so I am just speculating here.
--
Erland Sommarskog, SQL Server MVP, es****@sommarskog.se
Books Online for SQL Server SP3 at http://www.microsoft.com/sql/techinf...2000/books.asp This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Richard Holliingsworth |
last post by:
Hello:
I have an Access 2K form I built from a SQL Server 7.0 view. I want to
lock certain fields in the database from users so they can see them on
the views and forms, but NOT be able to edit...
|
by: LizP |
last post by:
Hi, guys - looking for a bit of help/advice on moving a database from
SQL 7 to SQL 2000.
We've a third party application which currently uses a SQL7 database.
This database has a number of views...
|
by: M Wells |
last post by:
Hi All,
I have a table that holds pregenerated member IDs.
This table is used to assign an available member id to web site
visitors who choose to register with the site
So, conceptually the...
|
by: extmb |
last post by:
Hi,
I am quite puzzled how SQLServer manages transactions.
Whatever the isolation level I set when performing an insertion, other
connections do not have access to the table in select mode.
...
|
by: Dr NoName |
last post by:
Help!
I have a table that multiple processes must be able to
write to concurrently. However, it for some reason
gets locked in exclusive mode. I narrowed it down to
one SQL statement + some...
|
by: RayPower |
last post by:
I'm having problem with using DAO recordset to append record into a
table and subsequent code to update other tables in a transaction. The
MDB is Access 2000 with the latest service pack of JET 4....
|
by: kavallin |
last post by:
I receives the following in the db2diag.log file many times / day :
2007-03-05-14.55.24.836553+060 E12415C457 LEVEL: Warning
PID : 2785 TID : 1 PROC :...
|
by: wugon.net |
last post by:
question: db2 LUW V8 UNION ALL with table function month() have bad
query performance
Env:
db2 LUW V8 + FP14
Problem :
We have history data from 2005/01/01 ~ 2007/05/xx in single big...
|
by: halex |
last post by:
Hello,
I am having deadlock problem when I have a lot of visitors on my website at the same time. I am using NetTiers templates to generate C# classes for accessing DB layer and problem is in my...
|
by: Charles Arthur |
last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
|
by: ryjfgjl |
last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
|
by: BarryA |
last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers,...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...
| |