Hi,
I have a job where it will involve checking large volumes of addresses
against a master address table.
The master address table contains approx 1.7 million addresses.
The files (normally CSV) I must check against will contain anything
between 100,000 records and 40 million records.
Due to spelling mistakes, different address formats and other errors I
can't just load up the data into SQL and do simple queries to get
matches. So I am writting some vb.net code to pre-process the data in
order to match to the SQL master address table.
At the moment I've played around with an open SQL connection and
SQLdatareader that I'm feeding new SQLcommands to. It works but as I'm a
relative beginner to vb.net it's probably not the most efficient method.
Would loading the master address table into a dataset or datatable be
effective. Bear in mind that the master address table will be approx
600mb in size.
Any help/advice/suggestions greatly appreciated.
thanks
Dave