To any and all;
I have a very large table (16Mil+ records), for which I want to delete
about 8 Million records. There are a few indexes on this table.
Is there a way that I can run either a query or a series of queries
that will run against each record and delete based on criteria (date)?
If I do a single DELETE query, it will take forever, lock the table,
and my app that runs against it will stop INSERTING, which is bad.
If I do a cursor, I think it locks the table also, so that won't do,
right?
Any help would be appreciated.
Glenn Dekhayser
Contentcatcher. com 6 3976
On 15 Feb 2005 13:19:01 -0800, "gdekhayser " <gd********@voy antinc.com> wrote: To any and all;
I have a very large table (16Mil+ records), for which I want to delete about 8 Million records. There are a few indexes on this table.
Is there a way that I can run either a query or a series of queries that will run against each record and delete based on criteria (date)? If I do a single DELETE query, it will take forever, lock the table, and my app that runs against it will stop INSERTING, which is bad.
If I do a cursor, I think it locks the table also, so that won't do, right?
Any help would be appreciated.
Glenn Dekhayser Contentcatcher .com
With a table that size, have you thought of partitioning it by date? Then,
locks would be held only on the partition(s) that overlap the date range you
are deleting.
Right, but I need to deal with the records in place before I can do
that....don't I? Once I get the table down in size to something
rational, then I can partition it....
gdekhayser (gd********@voy antinc.com) writes: I have a very large table (16Mil+ records), for which I want to delete about 8 Million records. There are a few indexes on this table.
Is there a way that I can run either a query or a series of queries that will run against each record and delete based on criteria (date)? If I do a single DELETE query, it will take forever, lock the table, and my app that runs against it will stop INSERTING, which is bad.
If I do a cursor, I think it locks the table also, so that won't do, right?
First of all, if there is no index on this date column, you can never
avoid the table lock.
If there is a clustered index on the date column, you should be able
to say DELETE tbl "WHERE date < @somedateinthep ast" and still have
your insertes coming through. Then again, if you delete day by day,
it may still be leaner.
If there is a non-clustered index on date, it depends on how many
rows there are per date. If there are too many rows per date, a
DELETE per date could table-scan. You could force an index, but
plenty of rows would be locked, for some time.
--
Erland Sommarskog, SQL Server MVP, es****@sommarsk og.se
Books Online for SQL Server SP3 at http://www.microsoft.com/sql/techinf...2000/books.asp
For the future, you should consider partitioning , as deleting n
records is o(n) while dropping a table (or truncating it) is O(1).
For now , I would recommend deleting with rowcount< some number ,
probably in the low thousands, which should leave the table in a fairly
operational state while you do it.
Of course after such a massive delete, the table will be in a terribly
fragmented state internally, so the next step is to rebuild the
indexes.
another alternative is to create another table with the same schema but
without the indexes, insert into it the rows that you do want to keep
(the complement of the 'to be deleted') and then build the indexes. you
can then drop the old table and rename the new one.
hope this helps
Tzvika
gdekhayser wrote: To any and all;
I have a very large table (16Mil+ records), for which I want to
delete about 8 Million records. There are a few indexes on this table.
Is there a way that I can run either a query or a series of queries that will run against each record and delete based on criteria
(date)? If I do a single DELETE query, it will take forever, lock the table, and my app that runs against it will stop INSERTING, which is bad.
If I do a cursor, I think it locks the table also, so that won't do, right?
Any help would be appreciated.
Glenn Dekhayser Contentcatcher. com
Erland- I wish I would have read your post earlier. I would have
thought that removing the indexes would make the delete faster. I
goofed and left the 4 indexes in place, the date one wasn't clustered.
I really wouldn't have cared about the row locks if I forced an index,
as they were going to be deleted anyway and not accessed.
I ended up just performing the delete- and watching my logs grow REAL
large.
I now have 3 log files where I really only want one- is there a way to
condense back into one log file when you're split into 3?
gdekhayser (gd********@voy antinc.com) writes: Erland- I wish I would have read your post earlier. I would have thought that removing the indexes would make the delete faster.
Yeah, removing the indexes not useful for the DELETE would have speedied
things up a little. It may not be a good idea to drop the clustered index
though.
I ended up just performing the delete- and watching my logs grow REAL large.
I now have 3 log files where I really only want one- is there a way to condense back into one log file when you're split into 3?
Well, you can say ALTER DATABASE REMOVE FILE, but you would have to
truncate the log first. Maybe the best is to use WITH TRUNCATE_ONLY
and the take a full backup of the database. I guess that then you
should be able to remove some of the files.
But I will have to admit that I have never had reason to drop a log
file, so I am just speculating here.
--
Erland Sommarskog, SQL Server MVP, es****@sommarsk og.se
Books Online for SQL Server SP3 at http://www.microsoft.com/sql/techinf...2000/books.asp This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Richard Holliingsworth |
last post by:
Hello:
I have an Access 2K form I built from a SQL Server 7.0 view. I want to
lock certain fields in the database from users so they can see them on
the views and forms, but NOT be able to edit them.
I've looked in BOL, MS SQL Server web page and SQL Server 7.0 books and
could not find how to do this.
Any advise will be greatly appreciated.
|
by: LizP |
last post by:
Hi, guys - looking for a bit of help/advice on moving a database from
SQL 7 to SQL 2000.
We've a third party application which currently uses a SQL7 database.
This database has a number of views which are owned by user
INFORMATION_SCHEMA and are set as system tables. This user doesn't
actually have a login on the SQL server.
We've tried the following to recreate the database on SQL2000, but
whatever we try, the views owned by...
|
by: M Wells |
last post by:
Hi All,
I have a table that holds pregenerated member IDs.
This table is used to assign an available member id to web site
visitors who choose to register with the site
So, conceptually the process has been, from the site (in ASP), to:
- select the top record from the members table where the assigned flag
|
by: extmb |
last post by:
Hi,
I am quite puzzled how SQLServer manages transactions.
Whatever the isolation level I set when performing an insertion, other
connections do not have access to the table in select mode.
Example in SQL Analyzer:
create table foo (
id numeric(10),
data varchar(100)
)
|
by: Dr NoName |
last post by:
Help!
I have a table that multiple processes must be able to
write to concurrently. However, it for some reason
gets locked in exclusive mode. I narrowed it down to
one SQL statement + some weirdness with foreign keys.
To debug this, I opened two psql sessions and typed in
the sql statements manually. Here is the situation:
CREATE TABLE take2
| |
by: RayPower |
last post by:
I'm having problem with using DAO recordset to append record into a
table and subsequent code to update other tables in a transaction. The
MDB is Access 2000 with the latest service pack of JET 4. The system is
client/server, multiusers based. The MDBs are using record locking.
Here is part of the code:
Dim wkSpace As Workspace, db As Database
Dim rstTrans As DAO.Recordset
Set wkSpace = DBEngine.Workspaces(0)
|
by: kavallin |
last post by:
I receives the following in the db2diag.log file many times / day :
2007-03-05-14.55.24.836553+060 E12415C457 LEVEL: Warning
PID : 2785 TID : 1 PROC : db2agent
(dbname)
INSTANCE: db2inst1 NODE : 000 DB : dbname
APPHDL : 0-946 APPID: *LOCAL.db2inst1.070305135434
FUNCTION: DB2 UDB, data management, sqldEscalateLocks, probe:3
MESSAGE : ADM5502W The escalation of...
|
by: wugon.net |
last post by:
question: db2 LUW V8 UNION ALL with table function month() have bad
query performance
Env:
db2 LUW V8 + FP14
Problem :
We have history data from 2005/01/01 ~ 2007/05/xx in single big
table,
we try separate this big table into twelve tables and create a view
|
by: halex |
last post by:
Hello,
I am having deadlock problem when I have a lot of visitors on my website at the same time. I am using NetTiers templates to generate C# classes for accessing DB layer and problem is in my custom Store Procedure.
I have Article table and ArticleLanguage table.
One record from Article (Id, Position, StatusId) table is the same for all languages and in ArticleLanguage (Id, LanguageId, ArticleId, Name) table I have only article names...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
| |
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| |
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |