472,353 Members | 1,000 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,353 software developers and data experts.

How to check Internal and External duplicates?

Recently, I have a requirement to develop a vb.net application wherein the input excel sheet data which has an average of 5000 records should be checked for Internal duplicates (duplicates within the same sheet) and external duplicates (duplicates which exist outside this sheet). I have gone through lot of logics..some of which are ...

- Common and currently testing out..

- First insert the excel sheet data into DB..then query by following

Expand|Select|Wrap|Line Numbers
  1.    select distinct id,mobilenumber from scrubmaster where calltablename=<calltablename given by the uploader> 
Then loop for duplicates...

Expand|Select|Wrap|Line Numbers
  1.   For jj As Integer = 0 To ds3.Tables("dup").Rows.Count - 1
  3.             stor1 = "SELECT count(mobilenumber), calltablename from scrubmaster WHERE mobilenumber =<mobilenumber> and flag is NULL GROUP BY calltablename"
  4.             da3 = New OleDbDataAdapter(stor1, con3)
  5.             da3.Fill(dse3, "dupp")
********* internal duplicates logic **************

Expand|Select|Wrap|Line Numbers
  1.             If dse3.Tables("dupp").Rows(0).Item(0) > 1 Then
  2.                 dupcountid += 1
  3.                 stor2 = "update scrubmaster set flag='ID' where ID=" & ds3.Tables("dup").Rows(jj).Item(0) & " and flag IS NULL"
  4.                 comm.Connection = con3
  5.                 comm.CommandText = stor2
  6.                 comm.ExecuteNonQuery()
  7.                 stor2 = ""
  8.             End If
********* External duplicates logic **************

Expand|Select|Wrap|Line Numbers
  1.             If dse3.Tables("dupp").Rows.Count > 1 Then
  2.                 dupcounted += 1
  3.                 stor2 = "update scrubmaster set flag='ED' where ID=" & ds3.Tables("dup").Rows(jj).Item(0) & " and flag IS NULL"
  4.                 comm.Connection = con3
  5.                 comm.CommandText = stor2
  6.                 comm.ExecuteNonQuery()
  7.                 stor2 = ""
  8.             End If
  9.             stor1 = ""
  10.                     Next
The above logic works but it takes very long time..I have tried other logics from web but they "Query timeout"...

Any good logic pls or correct my logic?
Aug 24 '09 #1
1 2060
3,525 Expert 2GB
TIP: When you are writing your question, there is a button on the tool bar that wraps the [code] tags around your copy/pasted code. It helps a bunch. Its the button with a '#' on it. More on tags. They're cool. Check'em out.
Aug 24 '09 #2

Sign in to post your reply or Sign up for a free account.

Similar topics

by: qazmlp | last post by:
const has internal linkage in C++, but external linkage in C. Am I right ? But, linker reports multiply-defined error if the following header is...
by: Razvan | last post by:
Hi What is the difference between an internal and an external entity ? The first one is defined in the internal subset (not in a separate...
by: Christopher Benson-Manica | last post by:
Just FMI, if an external stylesheet is linked to a page (with <link>), will subsequent <style> tags override the values in the external stylesheet?...
by: Mr. B | last post by:
The following code is how I check for duplicates in a List box. This is simple enough as there is only one column of stuff to check. ' Check for...
by: Carlos J. Quintero | last post by:
Hi, As you know the current keywords "protected internal" (C#) or "Protected Friend" (VB.Net) means "Protected Or internal" (C#) or "Protected Or...
by: al.cpwn | last post by:
do static and inline functions or members have internal linkage? I have been reading this newsgroup on google and found conflicting ideas. Can...
by: Daniel Di Vita | last post by:
I have an application that needs to pull files from an internal server. This is the setup. The web server is external facing, meaning exposed to the...
by: Gnus | last post by:
Hi all! I have a little(?) problem with external schema location... I'm using xerces/xalan (C++) for doing some actions with some set of...
by: =?Utf-8?B?VCBSYXkgSHVtcGhyZXk=?= | last post by:
I have an ASP.NET 2.0 web app using forms authentication and an ASP.NET Membership database. Internal users access the app from the intranet, but...
by: Naresh1 | last post by:
What is WebLogic Admin Training? WebLogic Admin Training is a specialized program designed to equip individuals with the skills and knowledge...
by: jalbright99669 | last post by:
Am having a bit of a time with URL Rewrite. I need to incorporate http to https redirect with a reverse proxy. I have the URL Rewrite rules made...
by: Matthew3360 | last post by:
Hi there. I have been struggling to find out how to use a variable as my location in my header redirect function. Here is my code. ...
by: AndyPSV | last post by:
HOW CAN I CREATE AN AI with an .executable file that would suck all files in the folder and on my computerHOW CAN I CREATE AN AI with an .executable...
by: Arjunsri | last post by:
I have a Redshift database that I need to use as an import data source. I have configured the DSN connection using the server, port, database, and...
by: WisdomUfot | last post by:
It's an interesting question you've got about how Gmail hides the HTTP referrer when a link in an email is clicked. While I don't have the specific...
by: Matthew3360 | last post by:
Hi, I have been trying to connect to a local host using php curl. But I am finding it hard to do this. I am doing the curl get request from my web...
by: Carina712 | last post by:
Setting background colors for Excel documents can help to improve the visual appeal of the document and make it easier to read and understand....
by: Rahul1995seven | last post by:
Introduction: In the realm of programming languages, Python has emerged as a powerhouse. With its simplicity, versatility, and robustness, Python...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.