By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
459,509 Members | 1,227 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 459,509 IT Pros & Developers. It's quick & easy.

UTF8/UTF7/ASCII problem while reading from text file

P: n/a
hi!

I have the following problem. I need to read data from a TXT file our
company receives.
I would use StreamReader, and process it line by line using ReadLine,
however, the following problem occurs.

The file contains characters with ASCII codes above 128. But the file is
still text (nothing like UTF7/8 or the like). It also might contain + signs.
As a result:

UTF8 encoding doesn't read characters above 128
UTF7 encoding reads everything ok, except eats the + signs, and some
characters after them
ASCII encoding reads the + sign ok, however, characters above 128 are
disappear.

Because the file arrives in this form, I do not have any control on how it
looks like. The best idea so far was to create an own ReadLine method, that
reads the file byte after byte, and converts using UTF7, while taking
special care to feed the + character (ASCII code 46) to an ASCII encoder.
This way I could build a string from a line, that contains exactly what's in
the file.

But would there be a nicer way, or just this do-it-yourself-manually?

thanx

-Lenard
Nov 16 '05 #1
Share this Question
Share on Google+
5 Replies


P: n/a
Lenard Gunda <fr****@fbi.hu> wrote:
I have the following problem. I need to read data from a TXT file our
company receives.
I would use StreamReader, and process it line by line using ReadLine,
however, the following problem occurs.

The file contains characters with ASCII codes above 128.
No it doesn't, because there are no such things. ASCII is a 7-bit
encoding.
But the file is still text (nothing like UTF7/8 or the like).
UTF-7 and UTF-8 are text encodings - a file containing text in UTF-8
encoding is still a text file.
It also might contain + signs. As a result:

UTF8 encoding doesn't read characters above 128
UTF7 encoding reads everything ok, except eats the + signs, and some
characters after them
ASCII encoding reads the + sign ok, however, characters above 128 are
disappear.

Because the file arrives in this form, I do not have any control on how it
looks like. The best idea so far was to create an own ReadLine method, that
reads the file byte after byte, and converts using UTF7, while taking
special care to feed the + character (ASCII code 46) to an ASCII encoder.
This way I could build a string from a line, that contains exactly what's in
the file.

But would there be a nicer way, or just this do-it-yourself-manually?


It sounds like you really need to know what encoding your file is
*really* in. Have you tried Encoding.Default?

See http://www.pobox.com/~skeet/csharp/unicode.html for more
information.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
Nov 16 '05 #2

P: n/a
> UTF-7 and UTF-8 are text encodings - a file containing text in UTF-8
encoding is still a text file.
Yup I know that. I wanted to mean plain text file, one that can be read
without conversion.

It sounds like you really need to know what encoding your file is
*really* in. Have you tried Encoding.Default?


Well, it contains ASCII characters, extended-ascii characters that are (in
this case) Finnish language characters. There's probably a code-page number
that would describe it. But ... Encoding.Default solved my problem, so it
would seem. Thanks very much!

-Lenard
Nov 16 '05 #3

P: n/a
Lenard Gunda <fr****@fbi.hu> wrote:
UTF-7 and UTF-8 are text encodings - a file containing text in UTF-8
encoding is still a text file.
Yup I know that. I wanted to mean plain text file, one that can be read
without conversion.


There's *always* conversion involved. The file is binary data, and you
want text data. There's a conversion involved, even if it's ASCII.
It sounds like you really need to know what encoding your file is
*really* in. Have you tried Encoding.Default?


Well, it contains ASCII characters, extended-ascii characters that are (in
this case) Finnish language characters.


"Extended-ascii" isn't a well-defined character set (there are many
character sets which are extensions of ASCII) and anything above 127 is
*not* ASCII.
There's probably a code-page number
that would describe it. But ... Encoding.Default solved my problem, so it
would seem. Thanks very much!


To find out the code page of Encoding.Default, just look at
Encoding.Default.CodePage.

I'm glad it's working for you though.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
Nov 16 '05 #4

P: n/a

"Jon Skeet [C# MVP]" <sk***@pobox.com> wrote in message
news:MP************************@msnews.microsoft.c om...
Lenard Gunda <fr****@fbi.hu> wrote:
UTF-7 and UTF-8 are text encodings - a file containing text in UTF-8
encoding is still a text file.


Yup I know that. I wanted to mean plain text file, one that can be read
without conversion.


There's *always* conversion involved. The file is binary data, and you
want text data. There's a conversion involved, even if it's ASCII.
It sounds like you really need to know what encoding your file is
*really* in. Have you tried Encoding.Default?


Well, it contains ASCII characters, extended-ascii characters that are (in this case) Finnish language characters.


"Extended-ascii" isn't a well-defined character set (there are many
character sets which are extensions of ASCII) and anything above 127 is
*not* ASCII.
There's probably a code-page number
that would describe it. But ... Encoding.Default solved my problem, so it would seem. Thanks very much!


To find out the code page of Encoding.Default, just look at
Encoding.Default.CodePage.


And note that it's working because your default encoding is the one with the
Finnish characters. If you need it to work on machines where this is not
the case, take Jon's advice: look at Encoding.Default.CodePage, and add
codes that explicitly uses that encoding to read the file.
Nov 16 '05 #5

P: n/a
Hi,
And note that it's working because your default encoding is the one with the Finnish characters. If you need it to work on machines where this is not
the case, take Jon's advice: look at Encoding.Default.CodePage, and add
codes that explicitly uses that encoding to read the file.


Yup, I finally managed to understand how these Encoders work, and found it
how to create one for a particular code page. Could be useful in the future,
but because this product is supposed to run on our server, it will have the
correct settings. But good advice, anyway.

Thanks for the help.

-Lenard
Nov 16 '05 #6

This discussion thread is closed

Replies have been disabled for this discussion.