Avoiding dupes when merging files

[google_groups3]

Hi all.

I currently have 2 text files which contain lists of file names. These
text files are updated by my code. What I want to do is be able to
merge these text files discarding the duplicates.

And to make it harder (or not???!!) my criteria for defining the
duplicate is the left 15 (or so) characters of the file path.
Help, as always, is greatly appreciated!

Thanks

[Lucas Tam]

go************@ hotmail.com wrote in news:1101328833 .131813.52400
@c13g2000cwb.go oglegroups.com:

Hi all.

I currently have 2 text files which contain lists of file names. These
text files are updated by my code. What I want to do is be able to
merge these text files discarding the duplicates.

And to make it harder (or not???!!) my criteria for defining the
duplicate is the left 15 (or so) characters of the file path.
Help, as always, is greatly appreciated!

Take a look at the Microsoft Text Driver - you can run SQL queries on the
text file. Perhaps you can just query each file checking for dupes?

Or you could load the data into a datatable (or hash table type object?),
with the PK set as the filename... if a duplicate shows up, the datatable
should throw a duplicate PK exception which you would catch and ignore.

Or lastly... perhaps you should think of a different method of storing the
data? Maybe a database is a better idea than text files?

--
Lucas Tam (RE********@rog ers.com)
Please delete "REMOVE" from the e-mail address when replying.
http://members.ebay.com/aboutme/coolspot18/

[Bob Hollness]

> Take a look at the Microsoft Text Driver - you can run SQL queries on the

text file. Perhaps you can just query each file checking for dupes?

Or you could load the data into a datatable (or hash table type object?),
with the PK set as the filename... if a duplicate shows up, the datatable
should throw a duplicate PK exception which you would catch and ignore.

Or lastly... perhaps you should think of a different method of storing the
data? Maybe a database is a better idea than text files?

I like the idea of the PK exception as it will give an error that i can
trap. I am being forced to use text files though for simplicity. Do you
have any sample code for implementing a datatable/PK exception as this is
new to me!

Bob

[Lucas Tam]

"Bob Hollness" <bo*@blockbuste r.com> wrote in
news:uH******** ******@TK2MSFTN GP11.phx.gbl:

I like the idea of the PK exception as it will give an error that i
can trap. I am being forced to use text files though for simplicity.
Do you have any sample code for implementing a datatable/PK exception
as this is new to me!

Here's the example from MSDN:

http://msdn.microsoft.com/library/de...l=/library/en-
us/cpref/html/frlrfsystemdata datatableclassp rimarykeytopic. asp

I've used it a couple of times and it works fine.

Here is what you do in short:

1. Add your columns to a datatable.
2. Add the same column from step 2 into a primary key array.
3. Add the primary key array to the DataTable.Prima ryKey property.

--
Lucas Tam (RE********@rog ers.com)
Please delete "REMOVE" from the e-mail address when replying.
http://members.ebay.com/aboutme/coolspot18/

[Bob Hollness]

>

Take a look at the Microsoft Text Driver - you can run SQL queries on the
text file. Perhaps you can just query each file checking for dupes?

Or you could load the data into a datatable (or hash table type object?),
with the PK set as the filename... if a duplicate shows up, the datatable
should throw a duplicate PK exception which you would catch and ignore.

Or lastly... perhaps you should think of a different method of storing the
data? Maybe a database is a better idea than text files?

--
Lucas Tam (RE********@rog ers.com)
Please delete "REMOVE" from the e-mail address when replying.
http://members.ebay.com/aboutme/coolspot18/

Thanks for the fast reply. I have to use text files so that really is not
an option. Any pointers or some sample code on how to use the datatable? I
like the idea of being able to trap a dupicate OK error.

Bob

[Lucas Tam]

"Bob Hollness" <bo*@blockbuste r.com> wrote in
news:#U******** ******@TK2MSFTN GP11.phx.gbl:

Take a look at the Microsoft Text Driver - you can run SQL queries on
the text file. Perhaps you can just query each file checking for
dupes?

Or you could load the data into a datatable (or hash table type
object?), with the PK set as the filename... if a duplicate shows up,
the datatable should throw a duplicate PK exception which you would
catch and ignore.

Or lastly... perhaps you should think of a different method of
storing the data? Maybe a database is a better idea than text files?

--
Lucas Tam (RE********@rog ers.com)
Please delete "REMOVE" from the e-mail address when replying.
http://members.ebay.com/aboutme/coolspot18/

Thanks for the fast reply. I have to use text files so that really is
not an option. Any pointers or some sample code on how to use the
datatable? I like the idea of being able to trap a dupicate OK error.

I replied to your message a bit early in the day, but I'm not sure if
you received it:

Here's the example from MSDN (particularly the SetPrimaryKeys Sub):

http://msdn.microsoft.com/library/de...l=/library/en-
us/cpref/html/frlrfsystemdata datatableclassp rimarykeytopic. asp

I've used it a couple of times and it works fine.

Here is what you do in short:

1. Add your columns to a datatable.
2. Add the same column from step 2 into a primary key array.
3. Add the primary key array to the DataTable.Prima ryKey property.
--
Lucas Tam (RE********@rog ers.com)
Please delete "REMOVE" from the e-mail address when replying.
http://members.ebay.com/aboutme/coolspot18/

[Bob Hollness]

Thanks for this. But I guess i need something a little more basic. Also to
do it in memory or straight to disk. I guess i'll keep playing with the
loops

--
Bob Hollness

-------------------------------------
I'll have a B please Bob
"Lucas Tam" <RE********@rog ers.com> wrote in message
news:Xn******** *************** ****@140.99.99. 130...

"Bob Hollness" <bo*@blockbuste r.com> wrote in
news:uH******** ******@TK2MSFTN GP11.phx.gbl:

I like the idea of the PK exception as it will give an error that i
can trap. I am being forced to use text files though for simplicity.
Do you have any sample code for implementing a datatable/PK exception
as this is new to me!

Here's the example from MSDN:

http://msdn.microsoft.com/library/de...l=/library/en-
us/cpref/html/frlrfsystemdata datatableclassp rimarykeytopic. asp

I've used it a couple of times and it works fine.

Here is what you do in short:

1. Add your columns to a datatable.
2. Add the same column from step 2 into a primary key array.
3. Add the primary key array to the DataTable.Prima ryKey property.

--
Lucas Tam (RE********@rog ers.com)
Please delete "REMOVE" from the e-mail address when replying.
http://members.ebay.com/aboutme/coolspot18/

[Bob Hollness]

> Hi all.

I currently have 2 text files which contain lists of file names. These
text files are updated by my code. What I want to do is be able to
merge these text files discarding the duplicates.

And to make it harder (or not???!!) my criteria for defining the
duplicate is the left 15 (or so) characters of the file path.
Help, as always, is greatly appreciated!

Thanks

OK. This is the solution I came up with. Not as elegant as one would have
hoped. but then again, only I get to see how it functions under the bonnet
(hood for the Americans) !!! And of course, this is still to be tidied up
and made pretty. Feel free to pull it apart and embarrass me.......
Sub FindDupes(ByVal File2Compare As String, ByVal OriginalFile As
String, ByVal OutputFile As String)

Dim File1Reader As New StreamReader(Fi le2Compare)
Dim File2Reader 'As New StreamReader(Or iginalFile)
Dim File3Writer As New StreamWriter(Ou tputFile)
Dim Line1 As String = ""
Dim Line2 As String = ""
Dim Found As Boolean

Do
Line1 = File1Reader.Rea dLine
Found = False

If Not Line1 Is Nothing Then

File2Reader = New StreamReader(Or iginalFile)

Do
Line2 = File2Reader.Rea dLine()
If Line1 = Line2 Then
Found = True
Exit Do
End If
Loop Until Line2 Is Nothing

If Found = False Then
File3Writer.Wri teLine(Line1)
End If

Found = False

File2Reader.Clo se()

End If
Loop Until Line1 Is Nothing

File1Reader.Clo se()
File2Reader.Clo se()
File3Writer.Clo se()

--
Bob Hollness

-------------------------------------
I'll have a B please Bob

[Anon-E-Moose]

"Bob Hollness" <bo*@blockbuste r.com> wrote in news:uUD3YV00EH A.1392
@TK2MSFTNGP14.p hx.gbl:

Feel free to pull it apart and embarrass me.......

Very inefficent when compared to Cor's elegant example of a hash table!

[Larry Serflaten]

"Bob Hollness" <bo*@blockbuste r.com> wrote

I currently have 2 text files which contain lists of file names. These
text files are updated by my code. What I want to do is be able to
merge these text files discarding the duplicates.
And to make it harder (or not???!!) my criteria for defining the
duplicate is the left 15 (or so) characters of the file path.
Help, as always, is greatly appreciated!

OK. This is the solution I came up with. Not as elegant as one would have
hoped. but then again, only I get to see how it functions under the bonnet
(hood for the Americans) !!! And of course, this is still to be tidied up
and made pretty. Feel free to pull it apart and embarrass me.......

As Cor suggested use a Hashtable, (or you might call it a Dictionary) it will
be much more efficient, and easier to code....

Paste the following in to a routine to see it in action:

HTH
LFS
Dim item As String
Dim hash As New System.Collecti ons.Hashtable
Dim file1 As String() = New String() { _
"Pretend this is text from a file.", _
"It is contained in an array only for", _
"demo purposes."}
Dim file2 As String() = New String() { _
"This is the text from a second file.", _
"The next line is a duplicate line and", _
"will overwrite the original entry:", _
"It is contained (DUPLICATE)", _
"Only the first 10 characters", _
"were used toward duplicate testing."}

For Each item In file1
hash.Item(item. Substring(0, 10)) = item
Next

For Each item In file2
hash.Item(item. Substring(0, 10)) = item
Next

Dim entry As System.Collecti ons.DictionaryE ntry
For Each entry In hash
Debug.WriteLine (entry.Value)
Next

Debug.WriteLine ("")
Debug.WriteLine ("Note that the order is not maintained, and")
Debug.WriteLine ("the duplicate line's original value was")
Debug.WriteLine ("overwritte n by the later (duplicate) entry.")