473,722 Members | 2,161 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

REQ HELP: Problem eliminating duplicates

Hi,

I have a table with duplicate records.
Some of the duplicates need to be eliminated from the table and some
need not.
A duplicate record does not need to be eliminated if the one record
appears multiple times on one employee's report.
However, if the same record(s) appear on any other employee's report,
it means they collaborated on the item, and the item from the second
employee's report should be eliminated from the table.

Each record includes the following fields:

ID
Period
Report
AgentID
Date
Invoice
JobID
Description
Country
Gross
Comm

A duplicate record is identified if the following fields are NOT the
same:

ID
AgentID

the following fields are identical:

Date
Invoice
JobID
Description
Country
Gross

I have created an append query to update a copy of the original table
(structure only) with the fields that must be identical designated as
primary key fields.

Unfortunately, when this table is updated with the append query, some
duplicates are not eliminated (seems this happens when more than two
employees are involved so there are three or more duplicates of a
record), and (I think) some records that should not be eliminated
(suplicates from same employee) are eliminated.

Will someone please help me figure this out?

Thanks,
MHenry

Some sample data (Description column omitted)

ID Period Report AgentID Date Invoice JobID
Country Gross Comm
1913 403 S DA 3/30/04 I0019013 LEP HFL DA 21
… England $2,775.00 $555.00
1914 403 S HFL 3/30/04 I0019013 LEP HFL DA 21
… England $2,775.00 $555.00
1915 403 S LEP 3/30/04 I0019013 LEP HFL DA 21
.... England $2,775.00 $555.00
1916 403 S DA 3/30/04 I0019014 LEP HFL DA 21
… USA $3,000.00 $600.00
1917 403 S HFL 3/30/04 I0019014 LEP HFL DA 21
… USA $3,000.00 $600.00
1918 403 S LEP 3/30/04 I0019014 LEP HFL DA 21
.... USA $3,000.00 $600.00
1919 403 S HFL 3/30/04 I0019016 LEP HFL 200304
… USA $1,000.00 $300.00
1920 403 S LEP 3/30/04 I0019016 LEP HFL 200304
… USA $1,000.00 $300.00
1922 403 S LEP 3/30/04 I0019034 LEP HFL 200304
… USA $500.00 $150.00
1921 403 S HFL 3/30/04 I0019034 LEP HFL 200304
… USA $500.00 $150.00
1923 403 S LEP 3/31/04 I0019050 LEP 090104 F
England $370.00 $111.00
1924 403 S JFO 3/31/04 I0019050 LEP JFO 090104
England $370.00 $111.00
Nov 13 '05 #1
1 1669
Hi

I have copied and converted your data, and am re-posting it for you as CSV
text in case anyone else wants to attempt to help you, too:

"ID","Period"," Report","AgentI D","Date","Invo ice","JobID","C ountry","Gross" ,
"Comm"
1913.00,403.00, "S","DA",3/30/04 0:00:00,"I00190 13","LEP HFL DA
21","England",$ 2775.00,$555.00
1914.00,403.00, "S","HFL",3/30/04 0:00:00,"I00190 13","LEP HFL DA
21","England",$ 2775.00,$555.00
1915.00,403.00, "S","LEP",3/30/04 0:00:00,"I00190 13","LEP HFL DA
21","England",$ 2775.00,$555.00
1916.00,403.00, "S","DA",3/30/04 0:00:00,"I00190 14","LEP HFL DA
21","USA",$3000 .00,$600.00
1917.00,403.00, "S","HFL",3/30/04 0:00:00,"I00190 14","LEP HFL DA
21","USA",$3000 .00,$600.00
1918.00,403.00, "S","LEP",3/30/04 0:00:00,"I00190 14","LEP HFL DA
21","USA",$3000 .00,$600.00
1919.00,403.00, "S","HFL",3/30/04 0:00:00,"I00190 16","LEP HFL
200304","USA",$ 1000.00,$300.00
1920.00,403.00, "S","LEP",3/30/04 0:00:00,"I00190 16","LEP HFL
200304","USA",$ 1000.00,$300.00
1922.00,403.00, "S","LEP",3/30/04 0:00:00,"I00190 34","LEP HFL
200304","USA",$ 500.00,$150.00
1921.00,403.00, "S","HFL",3/30/04 0:00:00,"I00190 34","LEP HFL
200304","USA",$ 500.00,$150.00
1923.00,403.00, "S","LEP",3/31/04 0:00:00,"I00190 50","LEP 090104
F","England",$3 70.00,$111.00
1924.00,403.00, "S","JFO",3/31/04 0:00:00,"I00190 50","LEP JFO
090104","Englan d",$370.00,$111 .00

Observations:
--------------

1.) All of the data appears to be a classic case of "flat-file" database
thinking, where you seem to think that all data needs to be in a single
table. Access is a relational database, where different elements of data are
stored in different, but related tables. This methodology allows you to
enter items ONCE, and then link them to related items in another table in
order to re-combine the elements and *display* them as though they are all
in one record.

I'd "study up" a bit on relational table design, then attempt to seperate
this data into the following tables if I were you:

tblJobs
tblInvoices
tblCountries
tblAgents
tblCommissions (if each Agent gets paid a seperate commission)

2.) A lot of the data appears to be just cocatenated or formatted data from
other fields.
i.e (from the first record) JobID = LEP HFL DA 21
I'm assuming that to mean that 3 "AgentID's (LEP, HFL, and DA) all
contributed to the completion of some task or sale... What does "21"
represent?
Period = 403... does that represent the 3rd month of 2004?
Gross = The total profit on the job?
Comm = Commision paid. Is that amount paid to each Agent, or is split evenly
amongst them? Divided up in diferently-sized portions?

3.) These appear to be the relationships, although I am not sure where some
fields (i.e. Period, Report) "fit" because the data is identical in all
records:
1 Job ==> Many Invoices
1 Job ==> Many Countries
1 Invoice ==> Many Agents
1 Invoice ==> 1 Gross, 1 Comm? (This implies that all agents share
equally?)

Oh waitaminute ...
1 Invoice ==> Many Jobs also? (Inv# I0019050 -- JobID's LEP 090104 F,
and LEP JFO 090104)
If so, then we're talking about a "Many-to-Many" relationship which
requires a join table.
1 Job ==> Many - [tblJoinInvoiceJ obs] - Many <== 1 Invoice

So, there... that should get you started :-)

P.S.
I'd also like to advise you to not use *Reserved Words* like "Date" for your
field names... it *will* come back to haunt you eventually.
"MHenry" <MH****@NoSpam. net> wrote in message
news:22******** *************** *********@4ax.c om...
Hi,

I have a table with duplicate records.
Some of the duplicates need to be eliminated from the table and some
need not.
A duplicate record does not need to be eliminated if the one record
appears multiple times on one employee's report.
However, if the same record(s) appear on any other employee's report,
it means they collaborated on the item, and the item from the second
employee's report should be eliminated from the table.

Each record includes the following fields:

ID
Period
Report
AgentID
Date
Invoice
JobID
Description
Country
Gross
Comm

A duplicate record is identified if the following fields are NOT the
same:

ID
AgentID

the following fields are identical:

Date
Invoice
JobID
Description
Country
Gross

I have created an append query to update a copy of the original table
(structure only) with the fields that must be identical designated as
primary key fields.

Unfortunately, when this table is updated with the append query, some
duplicates are not eliminated (seems this happens when more than two
employees are involved so there are three or more duplicates of a
record), and (I think) some records that should not be eliminated
(suplicates from same employee) are eliminated.

Will someone please help me figure this out?

Thanks,
MHenry

Some sample data (Description column omitted)

ID Period Report AgentID Date Invoice JobID
Country Gross Comm
1913 403 S DA 3/30/04 I0019013 LEP HFL DA 21
. England $2,775.00 $555.00
1914 403 S HFL 3/30/04 I0019013 LEP HFL DA 21
. England $2,775.00 $555.00
1915 403 S LEP 3/30/04 I0019013 LEP HFL DA 21
... England $2,775.00 $555.00
1916 403 S DA 3/30/04 I0019014 LEP HFL DA 21
. USA $3,000.00 $600.00
1917 403 S HFL 3/30/04 I0019014 LEP HFL DA 21
. USA $3,000.00 $600.00
1918 403 S LEP 3/30/04 I0019014 LEP HFL DA 21
... USA $3,000.00 $600.00
1919 403 S HFL 3/30/04 I0019016 LEP HFL 200304
. USA $1,000.00 $300.00
1920 403 S LEP 3/30/04 I0019016 LEP HFL 200304
. USA $1,000.00 $300.00
1922 403 S LEP 3/30/04 I0019034 LEP HFL 200304
. USA $500.00 $150.00
1921 403 S HFL 3/30/04 I0019034 LEP HFL 200304
. USA $500.00 $150.00
1923 403 S LEP 3/31/04 I0019050 LEP 090104 F
England $370.00 $111.00
1924 403 S JFO 3/31/04 I0019050 LEP JFO 090104
England $370.00 $111.00

Nov 13 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

9
2307
by: Paul | last post by:
Can anyone suggest an efficient way to eliminate duplicate entries in a list? The naive approach below works fine, but is very slow with lists containing tens of thousands of entries: def uniq(list): u = for x in list: if x not in u: u.append(x) return u
2
1723
by: Sebastian | last post by:
The following query needs about 2 minutes to complete (finding dupes) on a table of about 10000 addresses. Does anyone have an idea on how to speed this up ? Thanks in advance !!! Sebastian
4
16773
by: Phillo | last post by:
Hello, I'm new at Javascript, and have written a script for a series of random roll-over button images, but one thing I would like to add is a function that checks to make sure that there are no duplicates in the randomly generated variables that choose the pictures. Can anyone give me a hand with this? One other thing I can't seem to figure out is how to manage the "onLoad" aspect of caching my roll-over images (DW has locked the...
2
1346
by: nikkii | last post by:
I currently have a spreadsheet with more than 1000 records. Within this spreadsheet are many duplicates. In the past I've been using he filter feature to find the duplictes. However, as the spreadsheet getts longer that is becoming more difficult. Is there any other way to delete the duplicates and keep the unique the values? Please respond through the site as I will not be responding to any presonal emails. Thanks in advance!!
6
2402
by: Marlene | last post by:
Hi All I have the following scenario, where I have found all the duplicates in a table, based on an order number and a part number (item).I might have something like this: Order PODate Rec Qty Invoice# Item Supplier Status POReceivedDate 570133 03/09/2004 50 0 DMEDIUM L0010 PENDING 03/09/2004 570133 03/09/2004 50 0 DMEDIUM L0010 PENDING 03/09/2004 570133 03/09/2004 50 0 DMEDIUM L0010 PENDING 03/09/2004
3
2171
by: AK | last post by:
Hi Our product uses MS-SQL Server 2000. One of our customer has 10 installations with each installation stroring data in its own database. Now the customer wants to consolidate these databases into one and we already have plan for that by consolidating one DB at a time. But first they want to find how many unique or duplicate entries they have across all the 10 databases Assumptions:
5
2653
by: EP | last post by:
This inquiry may either turn out to be about the suitability of the SHA-1 (160 bit digest) for file identification, the sha function in Python ... or about some error in my script. Any insight appreciated in advance. I am trying to reduce duplicate files in storage at home - I have a large number files (e.g. MP3s) which have been stored on disk multiple times under different names or on different paths. The using applications will...
1
1213
by: capdownlondon | last post by:
Im using the following code to duplicate a record varCnt(retrieved from a combo box on the form) many times, and it only duplicates the record with the fields present on the form for that record. The duplicate fields are and (upto target 5) and then it duplicates the but adds 1 to each duplicate for this field. But if the targets are updated again it keeps the old duplicates for future weeks, causing there to be 2 sets of records for...
118
4665
by: Chuck Cheeze | last post by:
This might be in the wrong group, but... Here is an example of my data: entry_id cat_id 1 20 2 25 3 30 4 25 5 35
0
8863
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
8739
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9238
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
9157
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
9088
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
6681
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
4502
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
2
2602
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2147
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.