unicode text file

I have some unicode (utf8) text file. I _tried_ to write a simple
program that read one of them and write it to the standard output but...
of course it doesn't work. There is an easy way to do it? Thanks, K.

This is my program.

#include <fstream>
#include <iostream>
#include <string>

using namespace std;

int main(){
ifstream infile ("in.txt");
string s;
while (infile >> s) {
cout << s;
}
}

re: unicode text file

"Koulbak" <turbolence@gmail.com> wrote in message
news:428de2b4$1_1@news.bluewin.ch...[color=blue]
>I have some unicode (utf8) text file. I _tried_ to write a simple program
>that read one of them and write it to the standard output but... of course
>it doesn't work. There is an easy way to do it? Thanks, K.
>
> This is my program.
>
> #include <fstream>
> #include <iostream>
> #include <string>
>
> using namespace std;
>
> int main(){
> ifstream infile ("in.txt");[/color]

You should here check that file was opened successfully
before attempting to read from it.
[color=blue]
> string s;
> while (infile >> s) {
> cout << s;
> }
> }[/color]

Try using 'wifstream' and 'wcout'.

-Mike

re: unicode text file

Mike Wahler wrote:
[read unicode text file][color=blue][color=green]
>>int main(){
>> ifstream infile ("in.txt");[/color]
>
> You should here check that file was opened successfully
> before attempting to read from it.[/color]

In the real program of course I do it, but in my post I put only the
essential part of the question.
[color=blue][color=green]
>> string s;
>> while (infile >> s) {
>> cout << s;
>> }
>>}[/color]
>
>
> Try using 'wifstream' and 'wcout'.[/color]

1 Tried, it doesn't compile.

error C2679: binary '>>' : no operator found which takes a right-hand
operand of type 'std::string' (or there is no acceptable conversion)

I added also wstring and it compile but it doens't work correctly: it
prints a lot of garbage.

2 I thought that with C++ there was the possibility to use exactly the
standard way (avoid special construct as wcout) maybe setting some
library option. Is it not at all true?

Thanks a lot, K.

re: unicode text file

Koulbak wrote:
[color=blue]
> 1 Tried, it doesn't compile.
>
> error C2679: binary '>>' : no operator found which takes a right-hand
> operand of type 'std::string' (or there is no acceptable conversion)[/color]


You should use wstring. A wchar_t string literal is prefixed with L. For example:


wstring s= L"Some string";


[color=blue]
> I added also wstring and it compile but it doens't work correctly: it
> prints a lot of garbage.
>
> 2 I thought that with C++ there was the possibility to use exactly the
> standard way (avoid special construct as wcout) maybe setting some
> library option. Is it not at all true?[/color]

These *are* standard facilities. All string facilities come with their wchar_t equivalents
(including the facilities of the C-subset).



--
Ioannis Vranos

http://www23.brinkster.com/noicys

re: unicode text file

Ioannis Vranos wrote:[color=blue]
> You should use wstring. [...][/color]

I add wstring, it doesn't works.
[color=blue][color=green]
>> 2 I thought that with C++ there was the possibility to use exactly the
>> standard way (avoid special construct as wcout) maybe setting some
>> library option. Is it not at all true?[/color]
>
>
> These *are* standard facilities. All string facilities come with their
> wchar_t equivalents (including the facilities of the C-subset).[/color]

Sorry I was not clear at all. I would like to avoid as mush as possible
the implementation details. I don't want to use explicitely unicode
function but simply say to the compiler or to the library that my
character code is unicode and then read a file exactly in the usual way.

I would like to avoid to learn a new set of function to read and
manipulate unicode character, unicode string and so on. Of course if it
is possible.

Thanks, K.

re: unicode text file

Koulbak wrote:[color=blue][color=green][color=darkred]
> >> string s;
> >> while (infile >> s) {
> >> cout << s;
> >> }
> >>}[/color][/color]
>
> 1 Tried, it doesn't compile.
>
> error C2679: binary '>>' : no operator found which takes a right-hand[/color]
[color=blue]
> operand of type 'std::string' (or there is no acceptable conversion)[/color]

You have not included all necessary or the wrong header files (or have
the wrong files in your include path).
[color=blue]
> I added also wstring and it compile but it doens't work correctly: it[/color]
[color=blue]
> prints a lot of garbage.[/color]

wstring is not appropriate for UTF-8.

R.C.

re: unicode text file

[....][color=blue][color=green]
>>I added also wstring and it compile but it doens't work correctly: it[/color]
>
>[color=green]
>>prints a lot of garbage.[/color]
>
>
> wstring is not appropriate for UTF-8.[/color]

Ok, that' s the problem. My encoding is UTF-8.

Any solution?
Thanks, K.

re: unicode text file

Koulbak wrote:[color=blue][color=green]
> >
> > wstring is not appropriate for UTF-8.[/color]
>
> Ok, that' s the problem. My encoding is UTF-8.
>
> Any solution?[/color]

Maybe I've been wrong. See e.g.
http://www.cl.cam.ac.uk/~mgk25/unicode.html
http://www-106.ibm.com/developerwork.../l-linuni.html
or search for other 'UTF-8' resources.

re: unicode text file

Koulbak wrote:[color=blue]
> I have some unicode (utf8) text file. I _tried_ to write a
> simple program that read one of them and write it to the
> standard output but... of course it doesn't work. There
> is an easy way to do it? Thanks, K.
>
> This is my program.
>
> #include <fstream>
> #include <iostream>
> #include <string>
>
> using namespace std;
>
> int main(){
> ifstream infile ("in.txt");
> string s;
> while (infile >> s) {
> cout << s;
> }
> }[/color]

ostream >> string reads a word (up to whitespace), and then
ignores any adjacent whitespace and newlines.
To do line-by-line reading, you would go:

while (getline(infile, s))
cout << s;

But this is not good for UTF-8 files because newline characters
might be part of a UTF-8 code.

To output the whole file at once:

cout << infile.rdbuf();

I'm assuming you want to output UTF-8 on stdout (Standard
C++ offers no facilities for converting UTF-8 to a stream
of wide characters). Can you clarify your intention?

The best thing to do (IMHO) would be to open the file in
binary mode, and also force std::cout into binary mode. (This
would require a system-specific code). Then, no translation
will occur and it will work correctly.

If you can't force cout to binary, then it *might* work to
open the input in text mode too, and hope that the input
conversions match the output conversions!

re: unicode text file

Koulbak wrote:
[color=blue]
> Sorry I was not clear at all. I would like to avoid as mush as possible
> the implementation details. I don't want to use explicitely unicode
> function but simply say to the compiler or to the library that my
> character code is unicode and then read a file exactly in the usual way.
>
> I would like to avoid to learn a new set of function to read and
> manipulate unicode character, unicode string and so on. Of course if it
> is possible.[/color]

wchar_t represents the largest character set of a system, char mainly represents a byte
and 1 byte character sets. If you have to deal with various character sets, then better
stick to wchar_t and the corresponding facilities for it (which are the same with plain
char facilities, with an additional w in their name) .



--
Ioannis Vranos

http://www23.brinkster.com/noicys

re: unicode text file

Rapscallion wrote:
[color=blue]
> wstring is not appropriate for UTF-8.[/color]

Why?


--
Ioannis Vranos

http://www23.brinkster.com/noicys

re: unicode text file

Old Wolf wrote:
[color=blue]
> The best thing to do (IMHO) would be to open the file in
> binary mode, and also force std::cout into binary mode. (This
> would require a system-specific code). Then, no translation
> will occur and it will work correctly.
>
> If you can't force cout to binary, then it *might* work to
> open the input in text mode too, and hope that the input
> conversions match the output conversions![/color]


What is wrong with the use of wcout?



--
Ioannis Vranos

http://www23.brinkster.com/noicys

re: unicode text file

>> The best thing to do (IMHO) would be to open the file in[color=blue][color=green]
>> binary mode, and also force std::cout into binary mode.
>> (This would require a system-specific code). Then, no
>> translation will occur and it will work correctly.
>>
>> If you can't force cout to binary, then it *might* work to
>> open the input in text mode too, and hope that the input
>> conversions match the output conversions![/color][/color]
[color=blue]
> What is wrong with the use of wcout?[/color]

UTF-8 is a stream of 1-byte chars with characters beyond ASCII
coded as multi byte sequences. I guess that you need to read such
a stream as a char or binary stream and then decode each line
with appropriate routine to UTF-16 Unicode. Say
MultiByteToWideChar and WideCharToMultiByte strings on Win32
platform. Other API exists on *nix platform in iconv ets.

--
Serge

re: unicode text file

> I have some unicode (utf8) text file. I _tried_ to write a simple[color=blue]
> program that read one of them and write it to the standard output[/color]
but...[color=blue]
> of course it doesn't work[/color]

What character set do you want to use when writing to standard output?

If you want it to write using a character set other than the UTF-8 that
it read in, you need to do some conversion. You have to do this
explicitly. It will not happen automatically.

Assuming that your program is going to actually do something with the
text, rather than just reading it in and then writing it out again, you
need to decide what character set you want to use internally. I mostly
use UTF-8 internally and for input/output, so there is rarely any
conversion. I store this in chars. This is on Unix, and I'm in the
"western hemisphere". I understand that Windows programmers tend to
use UTF-16 quite often and that would also be sensible for non-European
languages. For that you should use wchars. You should not be using
ASCII for any new applications.

To actually perform the conversion you need something like the iconv
library. This is supported just about everywhere, but you'll want a
C++ wrapper for it to make it more palateable.

Regards, Phil.

re: unicode text file

phil_gg04@treefic.com wrote:[color=blue][color=green]
>>I have some unicode (utf8) text file. I _tried_ to write a simple
>>program that read one of them and write it to the standard output
>> but... of course it doesn't work[/color]
>
> What character set do you want to use when writing to standard output? [..]
> If you want it to write using a character set other than the UTF-8 that
> it read in, you need to do some conversion. You have to do this
> explicitly. It will not happen automatically.[/color]


Thanks! I think I perfectly understood the problem.

My program was only an exercise, but the goal was learn how to "set" the
library (?) to read unicode (or eventually another encoding), manipulate
it using the string functionality of the standard library and then
write it back in a particular encoding on a file or to the standard output.
[color=blue]
> Assuming that your program is going to actually do
> something with the text, rather than just reading
> it in and then writing it out again, you need to
> decide what character set you want to use internally.[/color]

It's really necessary that I specify the internal encoding? At my level
(scholastic level) I have no performance problem so if does exists a
default encoding this is ok for me.

So I would like specify the input file encoding and the ouput file
encoding, than use my program, for example:

string s;
while (infile >> s) {
if (s=="hello")
{;} //delete "hello" from input
else
{cout << s;}
}

I don't want, if it's possible, to specify wstring, wcut and so on because
1 I don't want to change the program the day I will need a diffent encoding
2 The program written without wstring, wcut etc. is more natural and
general and don't touch the implementation level

[Old Wolf write][color=blue]
> I'm assuming you want to output UTF-8 on stdout (Standard
> C++ offers no facilities for converting UTF-8 to a stream
> of wide characters). Can you clarify your intention?[/color]

I hope now it's more clear.

Thanks to all for the help. K.