what is best algorithm to check duplicated rows

[Ryan Liu]

Hi,

If I have tens of thousands DataRow in a DataTable and allow the end user to
pick any DataColumn(s) to check for duplicated lines, the data is so large,
is there a better API, algorithm can be used for this purpose?

Thanks a lot!
Ryan

[JR]

SELECT?

JR

"Ryan Liu" <Ry*******@gmail.comëúá
áäåãòä:OJ**************@TK2MSFTNGP06.phx.gbl...

Hi,

If I have tens of thousands DataRow in a DataTable and allow the end user
to pick any DataColumn(s) to check for duplicated lines, the data is so
large, is there a better API, algorithm can be used for this purpose?

Thanks a lot!
Ryan

[Michael Nemtsev]

Hello Ryan,

Hmm,
I see two ways - using the hashtable or sorting + binary search

---
WBR, Michael Nemtsev [.NET/C# MVP].
My blog: http://spaces.live.com/laflour
Team blog: http://devkids.blogspot.com/

"The greatest danger for most of us is not that our aim is too high and we
miss it, but that it is too low and we reach it" (c) Michelangelo

RLHi,
RL>
RLIf I have tens of thousands DataRow in a DataTable and allow the end
RLuser to pick any DataColumn(s) to check for duplicated lines, the
RLdata is so large, is there a better API, algorithm can be used for
RLthis purpose?
RL>
RLThanks a lot!
RLRyan

[=?Utf-8?B?UGV0ZXIgQnJvbWJlcmcgW0MjIE1WUF0=?=]

Ryan,
The first question I would ask is "how did you get tens of thousands of
rows" into this Datatable? If they came out of a database, shouldn't that be
where you are enforcing your referential and unique column integrity?
Peter

--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net


"Ryan Liu" wrote:

Hi,

If I have tens of thousands DataRow in a DataTable and allow the end user to
pick any DataColumn(s) to check for duplicated lines, the data is so large,
is there a better API, algorithm can be used for this purpose?

Thanks a lot!
Ryan

[Ryan Liu]

Hi Peter,

The data is imported by end user from external file, most time it is a csv
text file.

After import to datatable, then the end user specify the criteria which is
used to pick rows from datatable.

Then for selected rows, the end user want to check duplicated lines before
insert them to database. The criteria for checking duplicated lines is
also specified by the end user.

And I am also required check duplicated entries against data already in
database.

Thank you and everyone replyed to this message!

Ryan

"Peter Bromberg [C# MVP]" <pb*******@yahoo.yabbadabbadoo.comwrote in
message news:55**********************************@microsof t.com...

Ryan,
The first question I would ask is "how did you get tens of thousands of
rows" into this Datatable? If they came out of a database, shouldn't that
be
where you are enforcing your referential and unique column integrity?
Peter

--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net


"Ryan Liu" wrote:

>Hi,

If I have tens of thousands DataRow in a DataTable and allow the end user
to
pick any DataColumn(s) to check for duplicated lines, the data is so
large,
is there a better API, algorithm can be used for this purpose?

Thanks a lot!
Ryan

[Ryan Liu]

Thanks!

Just hope hash algorithm for string is as efficient as int.

And the criteria for checking duplicated datarows could be based on multiple
dataColumns (AND logic), this make it difficult for me to come out a hash
algorithm.

Ryan

"Michael Nemtsev" <ne*****@msn.comwrote in message
news:a2***************************@msnews.microsof t.com...

Hello Ryan,

Hmm,
I see two ways - using the hashtable or sorting + binary search

---
WBR, Michael Nemtsev [.NET/C# MVP]. My blog:
http://spaces.live.com/laflour
Team blog: http://devkids.blogspot.com/

"The greatest danger for most of us is not that our aim is too high and we
miss it, but that it is too low and we reach it" (c) Michelangelo

RLHi,
RLRLIf I have tens of thousands DataRow in a DataTable and allow the
end
RLuser to pick any DataColumn(s) to check for duplicated lines, the
RLdata is so large, is there a better API, algorithm can be used for
RLthis purpose?
RLRLThanks a lot!
RLRyan

[=?Utf-8?B?UGV0ZXIgQnJvbWJlcmcgW0MjIE1WUF0=?=]

You can try adding a unique constraint to one or more columns in the DataTable.
I'm not sure exactly how to treat exceptions or error messages during an
import, but I'm sure if you look it up in the MSDN documentation you can find
some examples.
Peter

--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net


"Ryan Liu" wrote:

Hi Peter,

The data is imported by end user from external file, most time it is a csv
text file.

After import to datatable, then the end user specify the criteria which is
used to pick rows from datatable.

Then for selected rows, the end user want to check duplicated lines before
insert them to database. The criteria for checking duplicated lines is
also specified by the end user.

And I am also required check duplicated entries against data already in
database.

Thank you and everyone replyed to this message!

Ryan

"Peter Bromberg [C# MVP]" <pb*******@yahoo.yabbadabbadoo.comwrote in
message news:55**********************************@microsof t.com...

Ryan,
The first question I would ask is "how did you get tens of thousands of
rows" into this Datatable? If they came out of a database, shouldn't that
be
where you are enforcing your referential and unique column integrity?
Peter

--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net


"Ryan Liu" wrote:

Hi,

If I have tens of thousands DataRow in a DataTable and allow the end user
to
pick any DataColumn(s) to check for duplicated lines, the data is so
large,
is there a better API, algorithm can be used for this purpose?

Thanks a lot!
Ryan