How to read html files AS IS. Encoding seems to change the characters.

[Zoro]

My task is to read html files from disk and save them onto SQL Server
database field. I have created an nvarchar(max) field to hold them.
The problem is that some characters, particularly html entities, and
French/German special characters are lost and/or replaced by a
question mark.
This is really frustrating. I have tried using StreamReader with ALL
the encodings available and none work correctly. Each encoding handles
some characters and but loses others. I also tried reading into byte
array, but as soon as I converted the array to string the encoding
ruined the text.
Maybe the solution is not to convert to string? but then how will I
save it to the database?

Is there a way to get this html text AS IS - with no encoding and no
changes into the database?
I could do it in Delphi and many other pre .NET, there must be a way
in C# too - surely?!

Thanks a lot for your help,
zoro.

[=?ISO-8859-1?Q?G=F6ran_Andersson?=]

Zoro wrote:

Thanks again Goran for your help.

>You are writing it back as UTF-8, as you are not specifying any encoding
in the WriteAllText method call.

It looks like I may be able to do it with string after all.
It looks like the problem with my test was - like you suggested - that
i didn't specify the write encoding. When I do, as long as I use the
same encoding when reading and writing, it worked with all 3 codes you
have suggested (but not with any of the built in codes - e.g. UTF-n!).

Then you have successfully decoded the file into text, as you are not
losing any characters.

If you save the file using utf-8, all the characters will still be
there, as strings are unicode and utf-8 can store any unicode
characters. The reason that your test did not succeed with the unicode
encodings is because the utility that you are using doesn't support
unicode. You would need the "Pro" version for that.

I am still not clear on how it's going to work away from the test -
using the database situation, but I am HOPING it would work as
follows:
1. I will use 1 of these codes to read the file
2. then store the string into nvarchar field and add a note informing
users of the encoding I used
3. specify the same encoding when creating the file, after reading the
string from the db.

Do you think this would work?
Thanks again,
zoro.

As you successfully decoded the file to a string, you can store that in
a nvarchar/ntext field and you are done. You can also store the encoding
used if you like to recreate the file exactly, but you can create a file
using any encoding that supports the characters in the text.

One advantage with using utf-8 encoding is that it places a BOM (byte
order mark) at the beginning of the file, that can be used to identify
the encoding used. If you use the File.ReadAllTex t to read a file that
contains a BOM, it will read the file correctly, even if you specify a
completely different encoding.

--
Göran Andersson
_____
http://www.guffa.com

[Zoro]

As you successfully decoded the file to a string, you can store that in

a nvarchar/ntext field and you are done. You can also store the encoding
used if you like to recreate the file exactly, but you can create a file
using any encoding that supports the characters in the text.

This doesn't seem to work. When I tried to read the files using
28591encoding and writing using 65001 (UTF-8), I seem to have lost the
dashes (-) in the text. I am not sure why this happens, but it looks
like I must save the text with the same encoding I read from the file.
btw, this is not only observed in the utility, I also opened the file
in a browser and the browser interpreted it differently too.

Thank again,
zoro.

[=?ISO-8859-1?Q?G=F6ran_Andersson?=]

Zoro wrote:

>As you successfully decoded the file to a string, you can store that in
a nvarchar/ntext field and you are done. You can also store the encoding
used if you like to recreate the file exactly, but you can create a file
using any encoding that supports the characters in the text.

This doesn't seem to work. When I tried to read the files using
28591encoding and writing using 65001 (UTF-8), I seem to have lost the
dashes (-) in the text. I am not sure why this happens, but it looks
like I must save the text with the same encoding I read from the file.
btw, this is not only observed in the utility, I also opened the file
in a browser and the browser interpreted it differently too.

Thank again,
zoro.

That is strange. A dash is a regular ASCII character, which should work
in any encoding.

However, the browser might misinterpret the encoding, have you tried to
open the file in Notepad?

--
Göran Andersson
_____
http://www.guffa.com

[Zoro]

That is strange. A dash is a regular ASCII character, which should work

in any encoding.

However, the browser might misinterpret the encoding, have you tried to
open the file in Notepad?

I did open them in Notepad. In the original file it's a dash and in
the new file it's a question mark (?) or missing char - depending on
the encoding.
However, I suspect this is because they are Em-dash or En-dash - ascii
150, 151.
Thanks,
zoro.

[Mihai N.]

As you successfully decoded the file to a string, you can store that in

a nvarchar/ntext field and you are done. You can also store the encoding
used if you like to recreate the file exactly, but you can create a file
using any encoding that supports the characters in the text.

On problem with loading with an encoding and saving with another is that
if the original encoding was not correct, you get junk.
Imagine loading a Greek file using the Russian codepage (cp1250), then
saving it as utf-8.

The second problem with this is if some files have in fact a encoding
meta-data in the head (or will start having, at some point).
This means that you end up writing a file as UTF-8 while the meta says
something else (the original encoding).

Other than this is should be ok.
--
Mihai Nita [Microsoft MVP, Windows - SDK]
http://www.mihai-nita.net
------------------------------------------
Replace _year_ with _ to get the real email