unicode text file

[Koulbak]

I have some unicode (utf8) text file. I _tried_ to write a simple
program that read one of them and write it to the standard output but...
of course it doesn't work. There is an easy way to do it? Thanks, K.

This is my program.

#include <fstream>
#include <iostream>
#include <string>

using namespace std;

int main(){
ifstream infile ("in.txt");
string s;
while (infile >> s) {
cout << s;
}
}

[Mike Wahler]

"Koulbak" <tu********@gma il.com> wrote in message
news:42******** **@news.bluewin .ch...

I have some unicode (utf8) text file. I _tried_ to write a simple program
that read one of them and write it to the standard output but... of course
it doesn't work. There is an easy way to do it? Thanks, K.

This is my program.

#include <fstream>
#include <iostream>
#include <string>

using namespace std;

int main(){
ifstream infile ("in.txt");
You should here check that file was opened successfully
before attempting to read from it.
string s;
while (infile >> s) {
cout << s;
}
}

Try using 'wifstream' and 'wcout'.

-Mike

[Koulbak]

Mike Wahler wrote:
[read unicode text file]

int main(){
ifstream infile ("in.txt");

You should here check that file was opened successfully
before attempting to read from it.

In the real program of course I do it, but in my post I put only the
essential part of the question.

string s;
while (infile >> s) {
cout << s;
}
}

Try using 'wifstream' and 'wcout'.

1 Tried, it doesn't compile.

error C2679: binary '>>' : no operator found which takes a right-hand
operand of type 'std::string' (or there is no acceptable conversion)

I added also wstring and it compile but it doens't work correctly: it
prints a lot of garbage.

2 I thought that with C++ there was the possibility to use exactly the
standard way (avoid special construct as wcout) maybe setting some
library option. Is it not at all true?

Thanks a lot, K.

[Ioannis Vranos]

Koulbak wrote:

1 Tried, it doesn't compile.

error C2679: binary '>>' : no operator found which takes a right-hand
operand of type 'std::string' (or there is no acceptable conversion)

You should use wstring. A wchar_t string literal is prefixed with L. For example:
wstring s= L"Some string";
I added also wstring and it compile but it doens't work correctly: it
prints a lot of garbage.

2 I thought that with C++ there was the possibility to use exactly the
standard way (avoid special construct as wcout) maybe setting some
library option. Is it not at all true?

These *are* standard facilities. All string facilities come with their wchar_t equivalents
(including the facilities of the C-subset).

--
Ioannis Vranos

http://www23.brinkster.com/noicys

[Koulbak]

Ioannis Vranos wrote:

You should use wstring. [...]

I add wstring, it doesn't works.

2 I thought that with C++ there was the possibility to use exactly the
standard way (avoid special construct as wcout) maybe setting some
library option. Is it not at all true?

These *are* standard facilities. All string facilities come with their
wchar_t equivalents (including the facilities of the C-subset).

Sorry I was not clear at all. I would like to avoid as mush as possible
the implementation details. I don't want to use explicitely unicode
function but simply say to the compiler or to the library that my
character code is unicode and then read a file exactly in the usual way.

I would like to avoid to learn a new set of function to read and
manipulate unicode character, unicode string and so on. Of course if it
is possible.

Thanks, K.

[Rapscallion]

Koulbak wrote:

string s;
while (infile >> s) {
cout << s;
}
}

1 Tried, it doesn't compile.

error C2679: binary '>>' : no operator found which takes a right-hand

operand of type 'std::string' (or there is no acceptable conversion)
You have not included all necessary or the wrong header files (or have
the wrong files in your include path).
I added also wstring and it compile but it doens't work correctly: it prints a lot of garbage.

wstring is not appropriate for UTF-8.

R.C.

[Koulbak]

[....]

I added also wstring and it compile but it doens't work correctly: it

prints a lot of garbage.

wstring is not appropriate for UTF-8.

Ok, that' s the problem. My encoding is UTF-8.

Any solution?
Thanks, K.

[Rapscallion]

Koulbak wrote:

wstring is not appropriate for UTF-8.

Ok, that' s the problem. My encoding is UTF-8.

Any solution?

Maybe I've been wrong. See e.g.
http://www.cl.cam.ac.uk/~mgk25/unicode.html
http://www-106.ibm.com/developerwork.../l-linuni.html
or search for other 'UTF-8' resources.

[Old Wolf]

Koulbak wrote:

I have some unicode (utf8) text file. I _tried_ to write a
simple program that read one of them and write it to the
standard output but... of course it doesn't work. There
is an easy way to do it? Thanks, K.

This is my program.

#include <fstream>
#include <iostream>
#include <string>

using namespace std;

int main(){
ifstream infile ("in.txt");
string s;
while (infile >> s) {
cout << s;
}
}

ostream >> string reads a word (up to whitespace), and then
ignores any adjacent whitespace and newlines.
To do line-by-line reading, you would go:

while (getline(infile , s))
cout << s;

But this is not good for UTF-8 files because newline characters
might be part of a UTF-8 code.

To output the whole file at once:

cout << infile.rdbuf();

I'm assuming you want to output UTF-8 on stdout (Standard
C++ offers no facilities for converting UTF-8 to a stream
of wide characters). Can you clarify your intention?

The best thing to do (IMHO) would be to open the file in
binary mode, and also force std::cout into binary mode. (This
would require a system-specific code). Then, no translation
will occur and it will work correctly.

If you can't force cout to binary, then it *might* work to
open the input in text mode too, and hope that the input
conversions match the output conversions!

[Ioannis Vranos]

Koulbak wrote:

Sorry I was not clear at all. I would like to avoid as mush as possible
the implementation details. I don't want to use explicitely unicode
function but simply say to the compiler or to the library that my
character code is unicode and then read a file exactly in the usual way.

I would like to avoid to learn a new set of function to read and
manipulate unicode character, unicode string and so on. Of course if it
is possible.

wchar_t represents the largest character set of a system, char mainly represents a byte
and 1 byte character sets. If you have to deal with various character sets, then better
stick to wchar_t and the corresponding facilities for it (which are the same with plain
char facilities, with an additional w in their name) .

--
Ioannis Vranos

http://www23.brinkster.com/noicys