Accented characters getting corrupted when parsing CSV file

[Jake]

I am writing a script that allows people to upload a CSV file for
processing and I am having trouble with accented characters. If a
UTF-8 encoded file is uploaded, everything works fine. The problem
happens when someone uploads a non UTF8 file that happens to have an
accented character inside, like é. In this case, the character is
converted to a funky symbol, like �.

I noticed this explanation in the PHP documentation for the "fgetcsv"
that I am using to read the file: "Note: Locale setting is taken into
account by this function. If LANG is e.g. en_US.UTF-8, files in one-
byte encoding are read wrong by this function."

So my question is, how do I get around this? My script needs to be
able to handle a UTF8 file and a non-UTF8 file with accents. It seems
like I can only pick one. Anyone know of a way to convert the funky
symbol back to the accented character?

Thanks

[Jake]

Here is a simple way to reproduce the problem.

1) Create an empty file with this string inside "a é É ñ". Save this
with ISO-8859-1 encoding.
2) Create a PHP page that has this inside: echo
file_get_conten ts("myfile.txt" );

The output from that will be gobbledygook. If you resave the file
with UTF-8 encoding, it will work fine. I can't control how people
are encoding their files so I need to be able to handle both cases.

I've tried using mb_convert_enco ding and utf8_decode without luck.

[=?iso-8859-1?Q?=C1lvaro?= G. Vicario]

*** Jake escribió/wrote (Tue, 21 Oct 2008 12:05:03 -0700 (PDT)):

I am writing a script that allows people to upload a CSV file for
processing and I am having trouble with accented characters. If a
UTF-8 encoded file is uploaded, everything works fine. The problem
happens when someone uploads a non UTF8 file that happens to have an
accented character inside, like é. In this case, the character is
converted to a funky symbol, like �.

You can use iconv() to convert between charsets but you must know what
charset the CSV is using.

--
-- http://alvaro.es - �lvaro G. Vicario - Burgos, Spain
-- Mi sitio sobre programación web: http://bits.demogracia.com
-- Mi web de humor en cubitos: http://www.demogracia.com
--

[Curtis]

�lvaro G. Vicario wrote:

*** Jake escribió/wrote (Tue, 21 Oct 2008 12:05:03 -0700 (PDT)):

>I am writing a script that allows people to upload a CSV file for
processing and I am having trouble with accented characters. If a
UTF-8 encoded file is uploaded, everything works fine. The problem
happens when someone uploads a non UTF8 file that happens to have an
accented character inside, like é. In this case, the character is
converted to a funky symbol, like �.

You can use iconv() to convert between charsets but you must know what
charset the CSV is using.

Also, a couple functions to look up for detecting and converting encoding:

<URL:http://php.net/manual/en/function.mb-detect-encoding.php>
<URL:http://php.net/manual/en/function.mb-convert-encoding.php>

--
Curtis