Chinese/Japanese Characters from ofstream

[MrPickle]

I am tokenizing a string and sending it to a ofstream but I am getting strange results.

The \n sequence isn't working; it doesn't go to a new line.
I'm getting Chinese/Japanese characters in the file.

I think this may be due to compiling the project with the unicode character set rather than multibyte character set, but that's a guess.

Here's my code:

void Tokenize(std::string str, std::vector<std::string>& vs)
{
    std::string s;
    std::stringstream ss(str);
    vs.clear();
    while(ss >> s) vs.push_back(s);
}
 
 
//...
 
std::ofstream osp("prt.txt");
std::string line = "begin particle \"name\"";
std::vector<std::string> tokens;
Tokenize(line, tokens);
if(!tokens.empty())
{
    if(tokens[0] == "begin")
    {
        if(tokens[1] == "particle")
        {
            char open = tokens[2][0];
            char close = tokens[tokens.size()-1][tokens[tokens.size()-1].length()];
            osp << open << " " << tokens[2] << " " << close << tokens[tokens.size()-1] << "\n";
        }
    }
}

Output:
斖斖斖쀀

[weaknessforcats]

Firstly, you cannot use a Unicode string with a C++ string object. You have to use a wstring object.

Second, the basic_string<>::operator>> skips whitespace. Since the \n is whitesace, it gets skipped.

What is it you are trying to do?