UTF-8 Character Decoding Problem

[seedstorm]

I am using a HtmlInputFile control in ASP.NET 2.0 to upload a file in a UserControl. After upload, I am examining the HttpPostedFile property of this object to read the bytes of the uploaded file's input stream, convert them to characters, and store them in a database.

For the most part, the process I had in place was working, but I noticed certain characters in my test file were not decoded correctly. I investigated the values of the corresponding byte values I received from the input stream, and came to the conclusion that the stream was encoded as UTF-8 ( which I have also come to understand is the default in many places in .NET when no other encoding scheme is specified ). I think the problem is that all the characters in the stream were encoded using a single byte, so in the case of the few characters in the file whose UTF-8 encodings required more than one byte, they are improperly decoded.

An example:

One of the characters I am having an issue with is the dash ( hyphen, minus-sign operator ) character (-). It is encoded from the input stream as a single byte with the value 0x96, but I found a resource that lists this character as requiring two bytes ( 0xc2, 0x96 ) in the UTF-8 encoding. The result is that when I convert the byte that is supposed to represent this character to a char, it ends up with the value ( bit-wise ) 0xfffd.

Here is the code I am trying to use to accomplish this:

private string serializeFile( )
    {
    StringBuilder fileContents = new StringBuilder( );
    if( this._file != null )
        {
        HttpPostedFile file = this._file;
 
        byte[ ] fileBytes = new byte[file.ContentLength];
        file.InputStream.Read( fileBytes, 0, file.ContentLength );
        foreach( byte fileByte in fileBytes )
            {
            Char character = Convert.ToChar( fileByte );
            fileContents.Append( character );
            }
        }
...
 

My question is this: can anyone see what I am doing wrong? Or, if I'm not doing anything obviously wrong (which would surprise me), how can I properly decode these characters without resorting to statically testing for the characters I know to be a problem?

Thank you very much for taking the time to read this, and if you choose to help. Let me know if you need any more information.

[Plater]

Take a look at the
System.Text.Encoding section.

Probably in particular:
System.Text.Encoding.UTF8

and say: System.Text.Encoding.UTF8.GetString(byte[])

[seedstorm]

I'm sorry. I forgot to mention that I've tried several different solutions involving the System.Text.Encoding class. I have tried using the UTF8 property to decode my byte array. The result is the same as using the Convert.ToChar method. I have also tried creating a UTF8 Decoder, and using its GetChars method; that led me to trying to detect when the byte value was over 127 ( outside of ASCII range ) and create a UTF32 character, using three other bytes, each with value 0x0, for padding. I then tried to use the UTF32 property to decode that byte array, with the same output all around.

I think I am well and truly not getting something here.

[Plater]

Hmm, well what you described seemed more like utf-16
(System.Text.Encoding.BigEndianUnicode)
utf-8 means 8bits per character, seems like there shouldn't BE any utf-8 encoded character that takes 2 8bit values to create.
That would make it utf-16?

[seedstorm]

I looked into the UTF-8 characters in question, and am now very suspicious of the characters themselves, as they are only supported by a handful of fonts.

Furthermore, after editing my test file in Notepad, to replace the offending characters in an environment in which I was sure the resultant character would be encoded as ASCII, voila!, no more problems when uploading the file.

I think the problem is therefore with the encoding of the files themselves; it is still puzzling, but I guess no longer within the scope of this forum. I'll shut up about it now =).

Thanks for your suggestions, Plater.