473,890 Members | 1,353 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Delete duplicate records daily based on Date

Brilstern
208 New Member
I upload names into a database everyday and sometimes I am given duplicate names or names that have previously been submitted. Basically I want to delete all but the latest record based off of the date. I have seen many post about this but I haven't really been able to get anything to work. On my table [Inbound] I have fields [NAME], [SSN], [RANK], [ULN], and [DATE]. I want to delete all duplicate records that have the same SSN excluding Null values. The catch is I want to delete records with the older date. I understand the Min/Max setting on a delete query but I am having trouble getting it to work properly. I figure I am way off so I am starting over. Below is a page that I read with some details regarding the issue but I don't understand access enough to get what I need out of it.

http://bytes.com/topic/access/answer...licate-records

NeoPa,

I think your post might have the closest answer but I am not sure where to go with it.

Sgt B
Dec 28 '11
32 6877
NeoPa
32,584 Recognized Expert Moderator MVP
Lines #4 & #5 of the first batch of SQL from post #17 should have been omitted. Including those lines was an error.

I'm a bit flummoxed on the third one (from post #18) as the table to be deleted from is specified explicitly on line #1. The Help section for the DELETE clause in Access is not very helpful and quite ambiguous, nevertheless I believe I've struggled with this before and found this format both to be consistent with what is there, as well as to work. That said, I knocked up an [Inbound] table with some indicative test data and was unable to get the query to execute :-( It basically wouldn't work, even though it conformed to the syntax indicated in the Help section.

This leaves my best suggestion to date as :
Expand|Select|Wrap|Line Numbers
  1. DELETE *
  2. FROM   [Inbound]
  3. WHERE  ([ID] IN(SELECT DISTINCT tI1.ID
  4.                 FROM   [Inbound] AS [tI1]
  5.                        INNER JOIN
  6.                        [Inbound] AS [tI2]
  7.                   ON   (tI1.SSN = tI2.SSN)
  8.                 WHERE  (tI1.SSN IS NOT NULL)
  9.                   AND  (tI1.Date < DateAdd('d', -10, tI2.Date))))
A couple of small changes beyond simply removing lines #4 & #5, but basically almost identical to the first suggestion of post #17. It worked for me logically (It executed correctly and removed the correct records), but I have no information on its relative performance.

PS. If you find the performance unacceptable then try removing the predicate DISTINCT from line #3 and see how that goes. In my tests both versions worked, but my data wasn't verbose enough to indicate which would handle better under load.
Jan 3 '12 #21
Brilstern
208 New Member
NeoPa,

I appreciate all the help.

Running the latest query at the moment. I will give more feed back after it runs.

!!THANKS!!

Sgt B
Jan 3 '12 #22
Brilstern
208 New Member
Ok so weird outcome. After running it (only took about 30 min on 52000 records) it deleted all records that did not have a SSN. Reason being, it is not reading all the records that don't contain a SSN as NULL, it reads them as BLANK!?! I believe this is because all these records were imported instead of being created in access. So from there I made a logical change (hopefully correct, I was just guessing) and I am running the below query now. Ideas?

Expand|Select|Wrap|Line Numbers
  1.     DELETE *
  2.     FROM   [Inbound]
  3.     WHERE  ([ID] IN(SELECT DISTINCT tI1.ID
  4.                     FROM   [Inbound] AS [tI1]
  5.                            INNER JOIN
  6.                            [Inbound] AS [tI2]
  7.                       ON   (tI1.SSN = tI2.SSN)
  8.                     WHERE  (tI1.SSN = NOT BLANK)
  9.                       AND  (tI1.Date < DateAdd('d', -10, tI2.Date))))
Sgt B
Jan 3 '12 #23
Brilstern
208 New Member
After I ran my query above and it found 0 records to delete...

Sgt B
Jan 3 '12 #24
NeoPa
32,584 Recognized Expert Moderator MVP
Stevan Bias:
Ideas?
Yes. First determine exactly what is in the data that appears to you as blank. The three most common values are :
  1. Null
  2. Empty string ('').
  3. Space(s) (' ').
It appears that these records don't respond to being filtered by Null, so check out what is actually there first and report back. #2 is certainly more likely statistically, so check for that first. Open the table and add a filter and see if the number of records returned matches the number that don't have visible values. To filter by '' simply enter '' into the [SSN] field in the filter.

When we know what we're working with we can move forward, but in case it's the likely option the code for line #8 would be :
Expand|Select|Wrap|Line Numbers
  1.                     WHERE  (tI1.SSN > '')
Jan 3 '12 #25
Brilstern
208 New Member
Ok, so the above is good, running it now...
Jan 3 '12 #26
NeoPa
32,584 Recognized Expert Moderator MVP
Stevan Bias:
Ok, so the above is good
I take that to mean that the 'empty' values are actually recognised as '' (empty strings)?
Jan 3 '12 #27
Brilstern
208 New Member
Yes sir. And it eventually stopped running and froze... This is such bad luck!

I am going to retry in to morning, so I will try to get on with any feedback here in about 12 hours. Please excuse my slow working speed.
Jan 3 '12 #28
NeoPa
32,584 Recognized Expert Moderator MVP
Stevan Bias:
Please excuse my slow working speed.
Go at your own speed. I have plenty to keep me busy until you post another reply ;-)

PS. Don't assume that just because Access freezes that it's stopped running. All long-running queries cause Access to freeze. It doesn't indicate a problem at all. Just that Access is still busy and the interface is a little useless.
Jan 3 '12 #29
Brilstern
208 New Member
I know that but unfortunately I have other databases to work on so I had to stop it to get something accomplished. I just posted a question about that one to...
Jan 3 '12 #30

Sign in to post your reply or Sign up for a free account.

Similar topics

0
4056
by: Joe | last post by:
I used LOAD DATA FILE to load data into a mysql database. I discovered that there are lots of duplicate records. Can anyone give me a simple list of mysql commands to search and destroy duplicate records? Thanks
1
10810
by: Patrizio | last post by:
Hi All, I've the following table with a PK defined on an IDENTITY column (INSERT_SEQ): CREATE TABLE MYDATA ( MID NUMERIC(19,0) NOT NULL, MYVALUE FLOAT NOT NULL, TIMEKEY INTEGER NOT NULL, TIMEKEY_DTTM DATETIME NULL,
2
7159
by: Barbara | last post by:
Hi, I have an sql database that has the primary key set to three fields, but has not been set as unique(I didn't create the table). I have 1 record that has 2 duplicates and I am unable to delete the duplicate entries. If I try to delete any of the three records(they are identical) I get the message 'key column is insufficient or incorrect. Too many rows were affected by update'. I am trying to do this within Enterprise Mgr. Any...
2
5004
by: ms | last post by:
Access 2000: I am trying to delete duplicate records imported to a staging table leaving one of the duplicates to be imported into the live table. A unique record is based on a composite key of 3 fields (vehicleID, BattID, and ChgHrs). VehicleID and BattID are a TEXT datatype and ChrHrs are a number(long int.) datatype. Since records to be imported can have duplicate records of the composite key I need to clean all but one of the...
4
6169
by: KT | last post by:
Is there any one click solution that would do the trick? I would like to create a button, so the person who maintains the database can perform clean up work to delete duplicate records which contain same information in the ID field and the account number field once a week. Thanks in advance! KT
16
16198
by: Theodore70 | last post by:
I am trying to delete duplicate records in an access table using VB. Not a programmer here, but this was one of the codes that I found on the web. I made some modifications to it, but it still did not work for me. Public Function DeleteDuplicate() Dim db As DAO.Database Dim qdf As QueryDef Dim strSQL As String Dim strEmail As String dteEmail = DMax("", "tbl_email") strSQL = "Delete * FROM tbl_email" & _
4
3608
by: smack | last post by:
I was able to get a query out showing all the duplicate records but know I need the duplicates to be sent to another table any ideas?
6
3593
by: hisham123 | last post by:
hi, i had more duplicate records in my table. i want to keep one rest i want to delete. can you give me sql server 2000 query
7
9886
by: AccessHunter | last post by:
I am using the following code to find and delete records in a table. The logic will go through each record and if a duplicate row is found will delete it. I ran this code and it worked the first time. Its not deleting the rows when I tried the second time. I debugged the code and its actually going through the delete step but the row is not getting deleted as it did the first time. Please help. Thanks in advance. Function...
6
5947
by: Dilip1983 | last post by:
Hi All, I want to delete duplicate records from a large table. There is one index(INDEX_U1) on 4 columns(col1,col2,col3,col4) which is in unusable state. First of all when i tried to rebuild index it showed error as unique key violation. So i want to delete duplicate records for col1,col2,col3,col4 combination. How can i delete the duplicate records from this large table?
0
9979
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9825
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
11234
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
10925
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10467
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
9638
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
8018
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
6052
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
3
3282
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.