Quickly Inserting many 'unique' rows when there's duplicates in source data.

Ok so this has got me stumped. I need to insert 400,000 words into a (unique) column and i have two problems:

1. My source data is not clean, there are around 1% repeated words.. i'm pretty sure that if i try an INSERT INTO a 'unique' column with duplicate entries it will stop the insert. What is the best way to INSERT this much data without having to clean it first? or perhaps somebody can recommend a quick way to remove duplicates before INSERT?

2. I assume the best way to make a bulk INSERT like this is to upload it as a file.. does the file have to be formatted as an INSERT query? ...my host is godaddy (don't hold it against me:) and i'm using phpMyAdmin.. if it makes a difference.

Thanks in advance!

BugBoy

re: Quickly Inserting many 'unique' rows when there's duplicates in source data.

Use INSERT IGNORE INTO instead of INSERT INTO. But if the text file then you have to remove UNIQUE index and then import the data. After importing data, remove the duplicate rows.