473,732 Members | 2,201 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

How to determine if a file is UTF8 encoded?

I need a check, if a file is utf8 encoded. I only found the php-functions
'iconv' and 'recode'. But it seems it´s not possible to determine the
encoding with them. Isn´t there any similar function to the 'file'-command
on linux for php?
Nov 22 '05 #1
9 24205
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thomas Podlesak wrote:
I need a check, if a file is utf8 encoded. I only found the php-functions
'iconv' and 'recode'. But it seems it´s not possible to determine the
encoding with them. Isn´t there any similar function to the 'file'-command
on linux for php?


Ok, I'm barly understanding your request. UTF/UTF8/UTF16 affects the
character set of your file. Now all HTML files can configure the browser
to determin the character set just be defining:

<meta http-equiv="Content-Type" content="applic ation/xml+xhtml;
charset=UTF-8" />

As well, if your sending PHP encoded file, you can pre-determin the
filetype just by defining the character encoding threw the headers:

header("Content-Type: application/xml+xhtml; charset=utf-8");

Other then that, the rest is done threw your web server, picking the
file... that if you wanted to encode it further you would have to
convert it using the functions you said above.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (MingW32)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFDfmPQ/WE0aXnOUiYRArvo AJ4tE6s7WPSRtBZ g1tLdFgJmFL5MNQ CbBKUe
GR3X5SE21hzNzW4 k2UOh7f8=
=xJ0s
-----END PGP SIGNATURE-----
Nov 22 '05 #2
That's not my problem, daemon.

The Problem is: The client uploads a csv-file. The php-script has to ensure
that the uploaded file is utf-8 encoded.

daemon wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thomas Podlesak wrote:
I need a check, if a file is utf8 encoded. I only found the php-functions
'iconv' and 'recode'. But it seems it´s not possible to determine the
encoding with them. Isn´t there any similar function to the
'file'-command on linux for php?


Ok, I'm barly understanding your request. UTF/UTF8/UTF16 affects the
character set of your file. Now all HTML files can configure the browser
to determin the character set just be defining:

<meta http-equiv="Content-Type" content="applic ation/xml+xhtml;
charset=UTF-8" />

As well, if your sending PHP encoded file, you can pre-determin the
filetype just by defining the character encoding threw the headers:

header("Content-Type: application/xml+xhtml; charset=utf-8");

Other then that, the rest is done threw your web server, picking the
file... that if you wanted to encode it further you would have to
convert it using the functions you said above.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (MingW32)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFDfmPQ/WE0aXnOUiYRArvo AJ4tE6s7WPSRtBZ g1tLdFgJmFL5MNQ CbBKUe
GR3X5SE21hzNzW4 k2UOh7f8=
=xJ0s
-----END PGP SIGNATURE-----


Nov 22 '05 #3
"Thomas Podlesak" wrote:
I need a check, if a file is utf8 encoded. I only found the php-functions
'iconv' and 'recode'. But it seems its not possible to determine the
encoding with them. Isnt there any similar function to the 'file'-command
on linux for php?


Try this: <http://php.net/mb-detect-encoding> (but make sure you read the
notes at the bottom, especially <http://php.net/mb-detect-encoding#50087> ).

--
phil [dot] ronan @ virgin [dot] net
http://vzone.virgin.net/phil.ronan/

Nov 22 '05 #4
Try this:

$text = file_get_conten ts("test.txt") ;
echo preg_match('/./u', $text);

The u modifier tell PCRE the input is UTF-8. If it's not properly
encoded, then it'll return false.

Nov 22 '05 #5
Chung Leong wrote:
echo preg_match('/./u', $text);


That will match on any single utf8 character, which could potentially be
followed by non-utf8 data... Also, I'm not sure about its behaviour when
encountering such data.

--
E. Dronkert
Nov 22 '05 #6

Ewoud Dronkert wrote:
Chung Leong wrote:
echo preg_match('/./u', $text);


That will match on any single utf8 character, which could potentially be
followed by non-utf8 data... Also, I'm not sure about its behaviour when
encountering such data.

--
E. Dronkert


PCRE validates the string before it runs the expression.

pcre.c:8037
if (valid_utf8((us char *)subject, length) >= 0)
return PCRE_ERROR_BADU TF8;

If it isn't valid all the way through, then there's no match.

Nov 22 '05 #7
Chung Leong wrote:
PCRE validates the string before it runs the expression.
If it isn't valid all the way through, then there's no match.


OK, but aren't charsets like latin1 (8859-1) subsets of utf8, and us-ascii
of them? So those would also be considered utf8.

--
E. Dronkert
Nov 22 '05 #8
Ewoud Dronkert wrote:
Chung Leong wrote:
PCRE validates the string before it runs the expression.
If it isn't valid all the way through, then there's no match.


OK, but aren't charsets like latin1 (8859-1) subsets of utf8, and us-ascii
of them? So those would also be considered utf8.


The Latin 1 is a subset of Unicode, true enough, with matching
codepoints. But when encoded as UTF-8, characters in the U+00F0 -
U+00FF will become 2 byte sequences. So text in 8859-1 with curly
quotes and such won't be identified as UTF-8. Text with characters only
in the basic Latin range (i.e. ASCII) would be identical to UTF-8.

It's of course possible to construct a text encoded in 8859-1, KOI8-R,
or whatever, that would appear as valid UTF-8. It'd be total gibberish
though. In a UTF-8 byte sequence, a byte with bit-6 on has to be
followed by a byte with bit-6 off. In a 8-bit charset, that means a
separation of at least 32 code points--too far apart to stay within the
alphabet.

Nov 22 '05 #9
In a other group somebody recommended the PECL Fileinfo extension.
http://pecl.php.net/package/Fileinfo

Seems ok for me.

Thanks for your help!
Thomas
Nov 22 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

21
39463
by: Sami Viitanen | last post by:
Hello, How can I check if a file is binary or text? There was some easy way but I forgot it.. Thanks in adv.
1
1284
by: Asra | last post by:
Hello, I have a frame's onload event calling a function which needs to know which file was loaded. Is there any java object/method to determine the file name or any type of file id? Thanks, Asra
2
12110
by: Mark Anderson | last post by:
Hi, I've a problem with code that should produce a Windows(ANSI) encoded text file but doesn't. Server is IIS 5 on Win 2k, with ASP ver? My ASP uses data from an upstream HTML form on a UTF-8 encoded page. The latter is output by a server-side system and I can't alter that source format. Here's some of the ASP code (may wrap): ' Following vars are longer strings but of this type - just more Request
10
10098
by: Marc Jennings | last post by:
Hi there, Can anyone point out any really obvious flaws in the methodology below to determine the likely encoding of a file, please? I know the number of types of encoding is small, but that is only because the possibilities I need to work with is a small list. > private string determineFileEncoding(FileStream strm) > { > long originalSize = strm.Length;
8
9807
by: Dinesh Jain | last post by:
Hi all, I encountered a serious problem while working with clipboard class in VB.NET. I want to simulate cut-copy-paste operation with my application and Windows Explorer. User can copy files from Explorer and paste it into my application & vice-a-versa. My question is- How can I determine if user has copied or cut the files from Windows Explorer? I want to differentiate cut & copy.
3
2432
by: Mark Gibson | last post by:
Is there an equivalent to the unix 'file' command? $ file min.txt min.txt: ASCII text $ file trunk trunk: directory $ file compliance.tgz compliance.tgz: gzip compressed data, from Unix What I really want to do is determine if a file is 1) a directory, 2) a
1
10048
by: nalinibala | last post by:
I'm having issues using XmlTextWriter, saving it out to a file with UTF8 encoding, and seeing "human unreadable" characters show up *right before* the XML declaration. I need to have the XML declaration state "encoding = utf-8", but also get rid of the dirty characters. I was hoping that using System.Text.Encoding.UTF8 in the constructor of the XmlTextWriter class would prove nice and simple, but I'm finding out I'm having some issues with...
0
1010
by: Tim Golden | last post by:
Lawrence, Anna K (US SSA) wrote: From where I'm sitting, I can't see enough to help. The crucial thing seems to be in this phrase: "When I try to take a string that is coming out of the database and export it to a Word document, no amount of encoding/decoding will make the Word doc display properly". It presumably doesn't matter what web framework etc. you're using: you either get a unicode object or an encoded string from the database...
4
2518
by: Samuel | last post by:
Hi I am trying to write to a string text encoded to utf8 as oppose to utf16 Since the data comes from an XML object (and I serialize it) I need to pass either StreamWriter or a StringWriter object, I don't want to create a file so I want to use a StringWriter (passing to it's constructor a StringBuilder) The problem is that the StringWriter encodes utf16 (I don't know how to change it)
0
8774
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9447
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9307
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
8186
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development projectplanning, coding, testing, and deploymentwithout human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6735
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupr who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
6031
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4550
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4809
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
3
2180
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.