How to determine if a file is UTF8 encoded?

[Thomas Podlesak]

I need a check, if a file is utf8 encoded. I only found the php-functions
'iconv' and 'recode'. But it seems it´s not possible to determine the
encoding with them. Isn´t there any similar function to the 'file'-command
on linux for php?

[daemon]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thomas Podlesak wrote:

I need a check, if a file is utf8 encoded. I only found the php-functions
'iconv' and 'recode'. But it seems it´s not possible to determine the
encoding with them. Isn´t there any similar function to the 'file'-command
on linux for php?

Ok, I'm barly understanding your request. UTF/UTF8/UTF16 affects the
character set of your file. Now all HTML files can configure the browser
to determin the character set just be defining:

<meta http-equiv="Content-Type" content="application/xml+xhtml;
charset=UTF-8" />

As well, if your sending PHP encoded file, you can pre-determin the
filetype just by defining the character encoding threw the headers:

header("Content-Type: application/xml+xhtml; charset=utf-8");

Other then that, the rest is done threw your web server, picking the
file... that if you wanted to encode it further you would have to
convert it using the functions you said above.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (MingW32)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFDfmPQ/WE0aXnOUiYRArvoAJ4tE6s7WPSRtBZg1tLdFgJmFL5MNQCbBKU e
GR3X5SE21hzNzW4k2UOh7f8=
=xJ0s
-----END PGP SIGNATURE-----

[Thomas Podlesak]

That's not my problem, daemon.

The Problem is: The client uploads a csv-file. The php-script has to ensure
that the uploaded file is utf-8 encoded.

daemon wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thomas Podlesak wrote:

I need a check, if a file is utf8 encoded. I only found the php-functions
'iconv' and 'recode'. But it seems it´s not possible to determine the
encoding with them. Isn´t there any similar function to the
'file'-command on linux for php?

Ok, I'm barly understanding your request. UTF/UTF8/UTF16 affects the
character set of your file. Now all HTML files can configure the browser
to determin the character set just be defining:

<meta http-equiv="Content-Type" content="application/xml+xhtml;
charset=UTF-8" />

As well, if your sending PHP encoded file, you can pre-determin the
filetype just by defining the character encoding threw the headers:

header("Content-Type: application/xml+xhtml; charset=utf-8");

Other then that, the rest is done threw your web server, picking the
file... that if you wanted to encode it further you would have to
convert it using the functions you said above.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (MingW32)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFDfmPQ/WE0aXnOUiYRArvoAJ4tE6s7WPSRtBZg1tLdFgJmFL5MNQCbBKU e
GR3X5SE21hzNzW4k2UOh7f8=
=xJ0s
-----END PGP SIGNATURE-----

[Philip Ronan]

"Thomas Podlesak" wrote:

I need a check, if a file is utf8 encoded. I only found the php-functions
'iconv' and 'recode'. But it seems it´s not possible to determine the
encoding with them. Isn´t there any similar function to the 'file'-command
on linux for php?

Try this: <http://php.net/mb-detect-encoding> (but make sure you read the
notes at the bottom, especially <http://php.net/mb-detect-encoding#50087>).

--
phil [dot] ronan @ virgin [dot] net
http://vzone.virgin.net/phil.ronan/

[Chung Leong]

Try this:

$text = file_get_contents("test.txt");
echo preg_match('/./u', $text);

The u modifier tell PCRE the input is UTF-8. If it's not properly
encoded, then it'll return false.

[Ewoud Dronkert]

Chung Leong wrote:

echo preg_match('/./u', $text);

That will match on any single utf8 character, which could potentially be
followed by non-utf8 data... Also, I'm not sure about its behaviour when
encountering such data.

--
E. Dronkert

[Chung Leong]

Ewoud Dronkert wrote:

Chung Leong wrote:

echo preg_match('/./u', $text);

That will match on any single utf8 character, which could potentially be
followed by non-utf8 data... Also, I'm not sure about its behaviour when
encountering such data.

--
E. Dronkert

PCRE validates the string before it runs the expression.

pcre.c:8037
if (valid_utf8((uschar *)subject, length) >= 0)
return PCRE_ERROR_BADUTF8;

If it isn't valid all the way through, then there's no match.

[Ewoud Dronkert]

Chung Leong wrote:

PCRE validates the string before it runs the expression.
If it isn't valid all the way through, then there's no match.

OK, but aren't charsets like latin1 (8859-1) subsets of utf8, and us-ascii
of them? So those would also be considered utf8.

--
E. Dronkert

[Chung Leong]

Ewoud Dronkert wrote:

Chung Leong wrote:

PCRE validates the string before it runs the expression.
If it isn't valid all the way through, then there's no match.

OK, but aren't charsets like latin1 (8859-1) subsets of utf8, and us-ascii
of them? So those would also be considered utf8.

The Latin 1 is a subset of Unicode, true enough, with matching
codepoints. But when encoded as UTF-8, characters in the U+00F0 -
U+00FF will become 2 byte sequences. So text in 8859-1 with curly
quotes and such won't be identified as UTF-8. Text with characters only
in the basic Latin range (i.e. ASCII) would be identical to UTF-8.

It's of course possible to construct a text encoded in 8859-1, KOI8-R,
or whatever, that would appear as valid UTF-8. It'd be total gibberish
though. In a UTF-8 byte sequence, a byte with bit-6 on has to be
followed by a byte with bit-6 off. In a 8-bit charset, that means a
separation of at least 32 code points--too far apart to stay within the
alphabet.

[Thomas Podlesak]

In a other group somebody recommended the PECL Fileinfo extension.
http://pecl.php.net/package/Fileinfo

Seems ok for me.

Thanks for your help!
Thomas