473,322 Members | 1,610 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,322 software developers and data experts.

How to detect the character encoding of a file ?

JB
Hi All,

I have an application that reads text files and does various things
with their content.
I'd like to know how to detect the character encoding of each text
file so my app can handle them correctly (e.g. ANSI, UTF8, etc).

Thanks for any tips
JB
Aug 14 '08 #1
4 4796
>I have an application that reads text files and does various things
>with their content.
I'd like to know how to detect the character encoding of each text
file so my app can handle them correctly (e.g. ANSI, UTF8, etc).
The StreamReader class already has some support for this, check out
the contructors that take a boolean detectEncodingFromByteOrderMarks
parameter.
Mattias

--
Mattias Sjögren [C# MVP] mattias @ mvps.org
http://www.msjogren.net/dotnet/ | http://www.dotnetinterop.com
Please reply only to the newsgroup.
Aug 15 '08 #2
Also note that a lot of older systems will refuse files because they can`t
handle the BOM at the begining of the file
because of this reasson there are a lot of files still out there without
BOM .

michel
"Mattias Sjögren" <ma********************@mvps.orgschreef in bericht
news:ez****************@TK2MSFTNGP04.phx.gbl...
I have an application that reads text files and does various things
with their content.
I'd like to know how to detect the character encoding of each text
file so my app can handle them correctly (e.g. ANSI, UTF8, etc).

The StreamReader class already has some support for this, check out
the contructors that take a boolean detectEncodingFromByteOrderMarks
parameter.
Mattias

--
Mattias Sjögren [C# MVP] mattias @ mvps.org
http://www.msjogren.net/dotnet/ | http://www.dotnetinterop.com
Please reply only to the newsgroup.

Aug 16 '08 #3
Michel Posseth [MCP] wrote:
because of this reasson there are a lot of files still out there
without BOM.

Thumbs up for UTF8 with BOM ;-) Most people think a character can only
consist of 2 bytes with UTF16

Regards,

Peter
Aug 19 '08 #4
JB
On 19 Aug, 22:39, Peter Gibbons <Pe...@initech.localwrote:
Michel Posseth [MCP] wrote:
because of this reasson there are a lot of files still out there
without BOM.

Thumbs up for UTF8 with BOM ;-) Most people think a character can only
consist of 2 bytes with UTF16

Regards,

Peter
Thank you all for your suggestions.
Regards
JB
Aug 20 '08 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
by: R. Rajesh Jeba Anbiah | last post by:
Here is a nice code to detect utf-8 <http://in2.php.net/utf8_encode#39986> But, I couldn't find out the logic behind the script. If anyone knows that please share. Particularly I would like to...
8
by: Agnes | last post by:
In my .net ,i need to generate an xml file , however, user may input a chinese character, Then , the xml will got something unknow characters. the following is my code, Does anyone know how to...
18
by: james | last post by:
Hi, I am loading a CSV file ( Comma Seperated Value) into a Richtext box. I have a routine that splits the data up when it hits the "," and then copies the results into a listbox. The data also...
13
by: Michal | last post by:
Hello, is there any way how to detect string encoding in Python? I need to proccess several files. Each of them could be encoded in different charset (iso-8859-2, cp1250, etc). I want to detect...
1
by: Kenneth McDonald | last post by:
I am going to demonstrate my complete lack of understanding as to going back and forth between character encodings, so I hope someone out there can shed some light on this. I have always...
4
by: Marc Scheuner | last post by:
Folks, I have a text file which contains some XML. In its XML header, it claims to be of UTF-8 encoding - however, it's really not, it's a ANSI / Windows-1252 / ISO-8859-1 encoding. Trouble...
20
by: FAQ server | last post by:
----------------------------------------------------------------------- FAQ Topic - How do I detect Opera/Netscape/IE? ----------------------------------------------------------------------- The...
13
by: =?Utf-8?B?YXVsZGg=?= | last post by:
i have come across a situation in my project where i read a text file with some characters greater than hex 0x7f. i need to write character (0xE0) to a new file as an exception. however when i...
10
by: Paul W | last post by:
Hi all, I have an application that reads data in from a text file and stores it in a database. My problem is that there are some characters in the file that aren't being handled properly. For...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.