468,248 Members | 1,533 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 468,248 developers. It's quick & easy.

How to check Internal and External duplicates?

Recently, I have a requirement to develop a vb.net application wherein the input excel sheet data which has an average of 5000 records should be checked for Internal duplicates (duplicates within the same sheet) and external duplicates (duplicates which exist outside this sheet). I have gone through lot of logics..some of which are ...

- Common and currently testing out..

- First insert the excel sheet data into DB..then query by following

Expand|Select|Wrap|Line Numbers
  1.    select distinct id,mobilenumber from scrubmaster where calltablename=<calltablename given by the uploader> 
Then loop for duplicates...

Expand|Select|Wrap|Line Numbers
  1.   For jj As Integer = 0 To ds3.Tables("dup").Rows.Count - 1
  2.  
  3.             stor1 = "SELECT count(mobilenumber), calltablename from scrubmaster WHERE mobilenumber =<mobilenumber> and flag is NULL GROUP BY calltablename"
  4.             da3 = New OleDbDataAdapter(stor1, con3)
  5.             da3.Fill(dse3, "dupp")
********* internal duplicates logic **************

Expand|Select|Wrap|Line Numbers
  1.             If dse3.Tables("dupp").Rows(0).Item(0) > 1 Then
  2.                 dupcountid += 1
  3.                 stor2 = "update scrubmaster set flag='ID' where ID=" & ds3.Tables("dup").Rows(jj).Item(0) & " and flag IS NULL"
  4.                 comm.Connection = con3
  5.                 comm.CommandText = stor2
  6.                 comm.ExecuteNonQuery()
  7.                 stor2 = ""
  8.             End If
********* External duplicates logic **************

Expand|Select|Wrap|Line Numbers
  1.             If dse3.Tables("dupp").Rows.Count > 1 Then
  2.                 dupcounted += 1
  3.                 stor2 = "update scrubmaster set flag='ED' where ID=" & ds3.Tables("dup").Rows(jj).Item(0) & " and flag IS NULL"
  4.                 comm.Connection = con3
  5.                 comm.CommandText = stor2
  6.                 comm.ExecuteNonQuery()
  7.                 stor2 = ""
  8.             End If
  9.             stor1 = ""
  10.                     Next
The above logic works but it takes very long time..I have tried other logics from web but they "Query timeout"...

Any good logic pls or correct my logic?
Aug 24 '09 #1
1 1913
tlhintoq
3,525 Expert 2GB
TIP: When you are writing your question, there is a button on the tool bar that wraps the [code] tags around your copy/pasted code. It helps a bunch. Its the button with a '#' on it. More on tags. They're cool. Check'em out.
Aug 24 '09 #2

Post your reply

Sign in to post your reply or Sign up for a free account.

Similar topics

9 posts views Thread by qazmlp | last post: by
1 post views Thread by Razvan | last post: by
6 posts views Thread by Christopher Benson-Manica | last post: by
5 posts views Thread by Mr. B | last post: by
8 posts views Thread by Carlos J. Quintero | last post: by
3 posts views Thread by al.cpwn | last post: by
1 post views Thread by =?Utf-8?B?VCBSYXkgSHVtcGhyZXk=?= | last post: by
reply views Thread by NPC403 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.