What are the Minimum requirements for all 17 planes of Unicode in C++?

[SwissProgrammer]

What are the Minimum requirements for all 17 planes of Unicode in C++?


In YOUR EXPERIENCE !

(Sometimes official descriptions have not been accurate. I want experienced answers.)

C++0 to C++20; Which version is the minimum that can work with ALL of the 17 planes without requiring a work-around or a third party dll?

I am not interested in Visual Studio or .net . I used these in the past and I am aware that they are powerful but I specifically do not want them now. Just C++.

In case you might not know what Unicode "planes" are, see https://en.wikipedia.org/wiki/Plane_%28Unicode%29


I am currently only using plane 0 and C++11. I want to be able to use all of the planes 0-16. I want to know the minimum requirements in C++.


Thank you.

[Banfa]

I am partly replying because I'd like to know the answer if someone else replies and partly because I'm not sure you are asking the right question.

Support for Unicode is not a programming language or version matter but rather it is related to the execution environment supported character encoding.

Execution environment and source code (or build environment) character sets can be different, although they often aren't in the case of building and executing on the same platform.

So take this example program

int main()
{
    wchar_t c = '\u0444';
 
    cout << "cout: ф" << endl;
    cout << "cout: " << u8"\u0444" << endl;
    cout << "cout: " << c << endl;
 
    wcout << "wcout: ф" << endl << flush;
    wcout << "wcout: " << u8"\u0444" << endl << flush;
    wcout << "wcout: " << c << endl << flush;
 
    return 0;
}

Compiled as C++14 and run in Power Shell I get the following output

cout: Ðä
cout: Ðä
cout: d184
wcout: Ð

Because Power Shell does not understand Unicode characters; run this command in the Power Shell

$OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = New-Object System.Text.UTF8Encoding

To tell Power Shell to use UTF8; re-run the same program without recompilation and you get this output

cout: ф
cout: ф
cout: d184
wcout:

Recompile the program using C++98 and you get this output

cout: ф
cout: ф
cout: 53636
wcout:

The only thing that has changed is the wchar_t variable is being displayed in decimal instead of hexidecimal.

Support for Unicode is depressingly non-standard across platforms so it is hard to write portable code using Unicode.

P.S. I have no idea why only 1 of the 3 wcout lines is producing output in all cases.

[Banfa]

Sorry that post didn't even try to answer the question, the point I was trying to make was, ignoring your problem of how to put plane 1+ Unicode characters into standard C++ code, outputting them requires a system that understands them.

Actually using them in code is complicated by standard C++ only really having support for UTF8 (and in theory that only came in with C++11) so outputting a character from plane 1+ then becomes rather painful because, for example character U+1FA0F (Black King rotate from plane 1), even if the system understands this plane and can display it, which isn't a given, in standard C++ your only option would be to use UTF8 encoding which looks something like u8"\xF0\x9f\xA8\x8F" which has to be looked up by hand and is a pain to type and because I haven't got an environment that knows how to interpret it I don't even know if it is correct.

I realise this post also doesn't answer the question, (still hoping someone else can) but at least it doesn't answer the question as opposed to not answering a different question.

[SwissProgrammer]

Banfa,

I did not want to lead someone with an answer, but your answer says close to what I have found.

UTF-8, in my opinion, is the most universal of the UTF options. I have not found any limit to the expandability of the UTF-8 encoding.

UTF-8, if my memory is correct, was what I was using back in Windows 2000 and (I think) in Windows NT. But, I was not programming in C++ at that time. Therefore, I thought to ask the local C++ experts here.

Before the Unicode consortium expanded their published scope past plane 0, I used Unicode a lot. Currently, as I transition into C++11, my coding time is greatly limited by my struggling through the learning curve. I was wanting someone with experience in the planes above 0 to speak to the issues encountered.

Your response, though you might not think it was so appropriate, I enjoyed.

Thank you.



One, but not the only, goal that I have with C++11 and Unicode is to be able to have a text box in which someone pastes a Unicode character or sentence and my program automatically shows in another text box the Unicode representation of that input:

Example input 办法 .

My program would show the UTF-8 encoding for that and maybe even split it apart into the two words that it contains 办 (ban) , and 法 (fa), each with their own UTF8 encoding.

But, first I wanted to know more about what I am dealing with in C++. Am I using at least the C++ version that can give me that response? Am I using the version that can give me the correct response for every plane?

So, I started with the simplest of the questions: Am I using the minimum version of C++ that can do all of this.

Later, I might have struggled to get the example to work, being confident that the C++ version was capable of doing the job.

Again, Thank you Banfa.

.

[dev7060]

P.S. I have no idea why only 1 of the 3 wcout lines is producing output in all cases.

int main() {
  wchar_t c = '\u0444';
  wcout << "wcout: ф" << endl << flush;
  if (wcout.fail()) {
    cout << "\nwide to narrow conversion didn't succeed; Unicode is not representable in the codepage";
    cout << endl;
    wcout << "\nThis won't get printed. Other wcouts don't have any effect at this point";
    wcout.clear();
  }
  wcout << "wcout: " << u8"\u0444" << endl << flush;
  if (wcout.fail()) {
    cout << "\nattempt #2 didn't succeed";
    cout << endl;
    wcout << "not shown on the console";
    wcout.clear();
    wcout << "hello user\n";
  }
  wcout << "wcout: " << c << endl << flush;
  wcout << "not available on the console as well ";
  return 0;
}
 

also,

A program should not mix output operations on wcout with output operations on cout
(or with other narrow-oriented output operations on stdout): Once an output operation has
been performed on either, the standard output stream acquires an
orientation (either narrow or wide) that can only be safely changed by calling freopen on stdout.

https://www.cplusplus.com/reference/iostream/wcout/

In YOUR EXPERIENCE !
(Sometimes official descriptions have not been accurate. I want experienced answers.)

Disclaimer: As you specifically asked for an experienced answer, I'm a student and not experienced at all when it comes to professional development. The below is just how I view it with my understanding.

My understanding of Unicode is that it isn't concerned with a language, as pointed out by Banfa. Every system or environment has kind of its way of dealing with it, has its own character set, and uses workarounds to set up compatibilities with others for exchanging the data. It depends on how encoding is done; what code points are being used, how many bytes for a character, which two code points are combined to represent a new character, what byte order, endian system, etc. Mapping is implementation-dependent.

Here's a char : 🮕 (it is not showing up on my screen, just copied a random off of Wikipedia)
In JS console,

console.log("🮕".length) 

shows the output 2.
In PHP,

echo strlen("🮕")

shows the output 4.

One solution is to use a fixed-length encoding across everything like UTF-32 that uses 4 bytes per code point. the con is that it's space inefficient. Imagine a 5 bytes character array in UTF-8 taking 20 bytes in the UTF-32 representation. literally a mess on a larger scale. ASCII's representation will have many leading 0s consuming memory for no reason. Variable-length encoding like in UTF-8 or UTF-16 allocates memory bytes according to the needs and situation.

Let's say you write a program in an ide. You converted the encodings between char* and wchar_t* back and forth in between function calls. The libraries would process em (or not?) all using the implemented mappings, but the output produced on the terminal may show undefined behavior because it may be supporting the encoding and mappings of the OS. Whatever representation code is sent by our binary to show may not be available in the character map of the OS to produce a relevant output. Windows have UTF-16 implementation hence apps use the same. If you run the same code in Unix or Linux, the output may be different (in UTF-8).

For having uniformity; I guess the engine, language, os, environment, third party compilers, linkers, ide, libraries, dependencies, databases, binaries, web connections, etc. (whatever is interacting with your encoded data in between) all have to agree on a common set of rules to represent the chars; which would be a hypothetical concept (maybe). I mean, for example, to communicate over the networks, you'd need maximum compression for the fast travel of the packets, hence would choose a variable size encoding. And if you see Java, it stores data as UTF-16 internally and on the other hand, UTF-16 is not used in internet websites because it's incompatible with ASCII. Workarounds seem to be the only solution to build the bridge and for the devs; trial and error if you don't know how the encodings are being done in a system. For example, Java docs states clearly:

The Java programming language is based on the Unicode character set, and several libraries implement the Unicode standard. Unicode is an international character set standard which supports all of the major scripts of the world, as well as common technical symbols. The original Unicode specification defined characters as fixed-width 16-bit entities, but the Unicode standard has since been changed to allow for characters whose representation requires more than 16 bits. The range of legal code points is now U+0000 to U+10FFFF. An encoding defined by the standard, UTF-16, allows to represent all Unicode code points using one or two 16-bit units.
The primitive data type char in the Java programming language is an unsigned 16-bit integer that can represent a Unicode code point in the range U+0000 to U+FFFF, or the code units of UTF-16. The various types and classes in the Java platform that represent character sequences - char[], implementations of java.lang.CharSequence (such as the String class), and implementations of java.text.CharacterIterator - are UTF-16 sequences. Most Java source code is written in ASCII, a 7-bit character encoding, or ISO-8859-1, an 8-bit character encoding, but is translated into UTF-16 before processing.

Ref: https://www.oracle.com/technical-res...lementary.html
https://docs.oracle.com/javase/8/doc.../overview.html
I've used java references just for the demonstration of a system.

One, but not the only, goal that I have with C++11 and Unicode is to be able to have a text box in which someone pastes a Unicode character or sentence and my program automatically shows in another text box the Unicode representation of that input:

Here's my guess: Whatever gui library you're using is probably making calls to winapi behind the scenes and using the OS's layout and character set to display everything. When you paste something in the text box if the OS's encoding character set couldn't map it, you probably won't see it properly in the text box field in the first place. Even though you may be able to pass the character to the called event handlers' functions of the library and the background processing may interpret everything correctly (or not), but you have to depend on an external OS to see through the output i.e. you need an external environment to interact with your app anyway same as you need a third party compiler (mingw, gcc, etc.) to process the text and produce the binaries. That's where workarounds come into play. You need a way to make the text recognizable to be displayed properly using third part libs or your logic if you can figure out how stuff is happening behind the scenes.

[SwissProgrammer]

dev7060,

You pointed directly to the issues. I agree.

Maybe if I get help one tiny step at a time.

I do not yet know how to make a Unicode capable, drag-n-drop text box in C++11.

I would like to be able to input to, and to read from, that text box with both wchar_t* and TCHAR*.

Help.

4 hours later (I told you I am new at this) I have the following.

 
#define _UNICODE
#define UNICODE
 
#include <windows.h>
 
#include<iostream>
#include <string>
using namespace std;
 
#define MAX_LOADSTRING 100
 
HWND Handle_Main_Window = NULL;
 
#include <windows.h>
 
void create_controls( const HWND hwnd );
 
 
LRESULT CALLBACK WndProc(HWND hWnd, UINT msg, WPARAM wParam, LPARAM lParam);
 
wchar_t g_szClassName[] = L"myWindowClass";
 
int APIENTRY WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance,
    LPSTR lpCmdLine, int nCmdShow)
{
    WNDCLASSEX wc;
    MSG Msg;
 
    wc.cbSize        = sizeof(WNDCLASSEX);
    wc.style         = 0;
    wc.lpfnWndProc   = WndProc;
    wc.cbClsExtra    = 0;
    wc.cbWndExtra    = 0;
    wc.hInstance     = hInstance;
    wc.hIcon         = LoadIcon(nullptr, IDI_APPLICATION);
    wc.hCursor       = LoadCursor(nullptr, IDC_ARROW);
    wc.hbrBackground = (HBRUSH)(COLOR_WINDOW+1);
    wc.lpszMenuName  = nullptr;
    wc.lpszClassName = g_szClassName;
    wc.hIconSm       = LoadIcon(nullptr, IDI_APPLICATION);
 
    if(!RegisterClassEx(&wc))
    {
        MessageBox(nullptr, L"Window Registration Failed!", L"Error!",
            MB_ICONEXCLAMATION | MB_OK);
        return 0;
    }
 
    Handle_Main_Window = CreateWindowEx(
        WS_EX_CLIENTEDGE,
        g_szClassName,
        L"Title",
        WS_OVERLAPPEDWINDOW,
        CW_USEDEFAULT,
        CW_USEDEFAULT,
        500,
        500,
        nullptr,
        nullptr,
        hInstance,
        nullptr);
 
    if(Handle_Main_Window == NULL)
    {
        MessageBox(nullptr, L"Window Creation Failed!", L"Error!",
            MB_ICONEXCLAMATION | MB_OK);
        return 0;
    }
 
    ShowWindow(Handle_Main_Window, nCmdShow);
    UpdateWindow(Handle_Main_Window);
 
    while(GetMessage(&Msg, nullptr, 0, 0) > 0)
    {
        TranslateMessage(&Msg);
        DispatchMessage(&Msg);
    }
    return Msg.wParam;
}
 
 
LRESULT CALLBACK WndProc(HWND hWnd, UINT msg, WPARAM wParam, LPARAM lParam)
    {
        switch (msg)
        {
 
            case WM_CREATE:
                create_controls( hWnd );
                break;
 
            case WM_COMMAND:
                switch(LOWORD(wParam)) {
                case 1:{
                        ::MessageBox( hWnd, L"PUSH BUTTON 1 was clicked", L"message from PUSH BUTTON 1", MB_SETFOREGROUND );
                        break;
                    }
 
                case 2:{
                        const HWND text_box = GetDlgItem( hWnd, 3 );
                        const int n = GetWindowTextLength( text_box );
                        wstring text( n + 1, L'#' );
                        if( n > 0 )
                            {
                                GetWindowText( text_box, &text[0], text.length() );
                            }
                        text.resize( n );
                        ::MessageBox(hWnd, text.c_str(), L"The INPUT TEXT WINDOW", MB_SETFOREGROUND );
                        break;
                    }
 
                case 3:{
                        break;
                    }
 
                case 4:{
                        break;
                    }
 
                case 5:{
                        const HWND text_box = GetDlgItem( hWnd, 5 );
                        const int n = GetWindowTextLength( text_box );
                        wstring text( n + 1, L'#' );
                        if( n > 0 )
                            {
                                GetWindowText( text_box, &text[0], text.length() );
                            }
                        text.resize( n );
                        ::MessageBox(hWnd, L"SAVE BUTTON was clicked", L"message from SAVE BUTTON", MB_SETFOREGROUND );
                        break;
                    }
 
                default:{
                    }
            }
            break;
 
            case WM_CLOSE:{
                    DestroyWindow(hWnd);
                    break;
                }
 
            case WM_DESTROY:{
                    PostQuitMessage(0);
                }
 
            default:{
                    return DefWindowProc(hWnd, msg, wParam, lParam);
                }
        }
        return FALSE;
    }
 
void create_controls( const HWND hwnd )
    {
 
        CreateWindow( L"BUTTON",
            L"PUSH BUTTON 1",
            WS_VISIBLE | WS_CHILD | WS_BORDER,
            10,10,
            130,20,
            hwnd, (HMENU) 1, GetModuleHandle( nullptr ), nullptr
            )  ;
 
        CreateWindow( L"EDIT",
            L"INPUT TEXT WINDOW",
            WS_VISIBLE | WS_CHILD | WS_BORDER,
            10,50,
            200,25,
            hwnd, (HMENU) 3, GetModuleHandle( nullptr ), nullptr
            );
 
        CreateWindow( L"BUTTON",
            L"SAVE BUTTON",
            WS_VISIBLE | WS_CHILD | WS_BORDER,
            10,80,
            110,20,
            hwnd, (HMENU) 5, GetModuleHandle( nullptr ), nullptr
            );
 
        CreateWindow( L"EDIT",
            L"OUTPUT TEXT WINDOW",
            WS_VISIBLE | WS_CHILD | WS_BORDER,
            10,130,
            300,300,
            hwnd, (HMENU) 4, GetModuleHandle( nullptr ), nullptr
            );
    }
 
 
 

I can paste "Example input 办法 ." into the INPUT BOX, but what do I do with it next? I want to be able to click the button below that and see the UTF8 representation in the bottom box.




Help please.

Banfa said: "Support for Unicode is not a programming language or version matter but rather it is related to the execution environment supported character encoding."

I agree.
This is a start to having my program to be able to adapt to that.

I am trying to get to the final answer of my original question in this post. One step at a time.

Thank you.

[SioSio]

I did some research.
UTF-8: In order to be compatible with ASCII characters, the same part as ASCII is encoded with 1 byte, and the other parts are encoded with 2-6 bytes. In a 4-byte sequence, up to 21 bits (0x1FFFFF) can be expressed, but those representing 17 or more planes outside the Unicode range (larger than U + 10FFFF) are not accepted.
UTF-16, UTF-32: Unlike UTF-8, it is not ASCII compatible.
Therefore, the condition that meets the requirement of # 1 is to look for a version of C++ that supports UTF-8.

Support status of UTF-8 depending on the version of C++

C++17 can process UTF-8 data as "char" data. This allows you to use std::regex, std::fstream, std::cout, etc. without loss.
In C++20, we added char8_t and std::u8string for UTF-8. However, UTF is not supported at all due to the lack of std::u8fstream. Therefore, we need a way to convert between UTF-8 and the execution character set.

[SioSio]

I forgot to write.
In the C++11 Standard Library, UTF-8 is not supported for string and integer conversion functions, and I/O functions. Therefore, it needs to be converted to the system multibyte character code.

[Banfa]

Looks like you are using WIN32 API. Windows GUI natively uses UTF16, I believe and the WIN32 API has wide char and multibyte versions of many characters, signified by a post fix W or A.

It also has a set of helper functions Unicode and Character Set Functions and I think the one you are interested in is WideCharToMultiByte.

I know there are Googleable examples out there.

[dev7060]

I can paste "Example input 办法 ." into the INPUT BOX, but what do I do with it next? I want to be able to click the button below that and see the UTF8 representation in the bottom box.

Like this?

case 5: {
  HWND InputTextBox = GetDlgItem(hWnd, 3);
  const int n = GetWindowTextLength(InputTextBox);
  wstring text(n + 1, L '#');
  if (n > 0) {
    GetWindowText(InputTextBox, & text[0], text.length());
  }
  const wchar_t * wcs = text.c_str();
  SetDlgItemText(hWnd, 4, wcs);
}
 

If you're using UTF-8 chars inside the Code::Blocks,
Settings -> Editor -> Encoding -> Change it from 'default' to UTF-8

Code::Blocks is smart enough to change the encoding automatically to prevent losing data. But it would do that temporarily for every time you click on build.

Cygwin environment can be used for CLI testing. It supports UTF-8. https://www.cygwin.com/

[Banfa]

How's it going?

You probably need a couple of helper functions

Convert wide character to multibyte character aka UTF16 to UTF8

// Convert a wide Unicode string to an UTF8 string
std::string utf8_encode(const std::wstring &wstr)
{
    if (wstr.empty())
    {
        return std::string();
    }
 
    int size_needed = WideCharToMultiByte(CP_UTF8, 0, wstr.c_str(), (int)wstr.size(), NULL, 0, NULL, NULL);
 
    char buffer[size_needed+1];
    WideCharToMultiByte(CP_UTF8, 0, wstr.c_str(), (int)wstr.size(), buffer, size_needed+1, NULL, NULL);
 
    std::string strTo( buffer );
    return strTo;
}

Get the character values of the multibyte character string

std::wstring utf8_byte_values(const std::string &str)
{
    if (str.empty())
    {
        return std::wstring();
    }
 
    bool first = true;
    std::wstringstream out;
 
    for(auto iter = str.begin(); iter != str.end(); ++iter)
    {
        if (first)
        {
            first = false;
        }
        else
        {
            out << L" ";
        }
 
        unsigned int value = ((unsigned)*iter) & 0xFF;
        out << L"0x" << std::hex << std::setw(2) << std::setfill(L'0') << value;
    }
 
    return out.str();
}

Then you Save Button code could look something like

       case BTN_SAVE:
        {
            const HWND in_box = GetDlgItem( hWnd, EDT_INPUT_TEXT );
            const int n = GetWindowTextLength( in_box  );
            if( n > 0 )
            {
                wchar_t text[n+1]; // +1 for terminator
                GetWindowText( in_box, text, n+1 );
                string utf8 = utf8_encode(wstring(text));
                // Force calling of ASCII/UTF8 version
                SetDlgItemText( hWnd, EDT_OUTPUT_TEXT, utf8_byte_values(utf8).c_str());
            }
//            text.resize( n );
            break;
        }

Note I defined symbols for all your dialog item ids to aid readability.

More importantly note that WIN32 API is a C API and expects C style strings, that is '\0' terminated. This does not play nicely with C++ particularly C++ strings because they are not '\0' terminated which makes passing &text[0] to a WIN32 API where text is a std::string or std::wstring a very risky business. Instead, if the WIN32 API accepts a constant pointer prefer text.c_str() or if the WIN32 API function expects a non-constant pointer use a standard C array and convert to a std::(w)string later.

Of course I have been slightly naughty in my code and used variable length arrays which a C rather than a C++ feature but my GNU compiler lets me get away with that with a warning :D

[SwissProgrammer]

SioSio, Thank you.

I feel like I should parse the input into ASCII and non-ASCII first.

Then, I should parse the non-ASCII incoming text and characters and test each as to how well they work in UTF-8 first, then in UTF-16 (to see if it is larger than U + 10FFFF). Compare the results. Thus at least finding out if I am receiving input that is in plane 0 or plane 1+.

Then respond into the second text box with the resultant U.



Separately:
You said, "In the C++11 Standard Library, UTF-8 is not supported for string and integer conversion functions, and I/O functions. Therefore, it needs to be converted to the system multibyte character code."

In my CODE::BLOCKS 17.12 Settings/Editor/General settings/Encoding settings I have been using UTF-8 with the following choices chosen:
/ "As default encoding (bypassing C::B's auto-detection)"
/ "If conversion fails using the settings above, try system local settings".

But, I am concerned about system local settings on a user's computer that is different from my tested systems. Maybe then I should just catch any errors of such and deal with that separately.

I think that this is correct. What do you think? How would you handle this?

Thank you.

[SwissProgrammer]

Banfa, Thank you.

When I started learning C++11 I used WideCharToMultiByte and MultiByteToWideChar.

They seemed to work. But I read, maybe 2 or 3 places, that these should be avoided. I should have asked here at that time, but I did not. Since I see you using them, I shall use them with more confidence.


You used:

        std::wstringstream out;
 

For that I got
error: aggregate 'std::wstringstream out' has incomplete type and cannot be defined

For future readers:
I added

#include <sstream>

which fixed that.

You used:

             out << L"0x" << std::hex << std::setw(2) << std::setfill(L'0') << value;

For that I got
error: 'setw' is not a member of 'std'

For future readers:
I added

#include <iomanip>

which fixes that.

You used:

                const HWND in_box = GetDlgItem( hWnd, EDT_INPUT_TEXT );

which I changed to:

const HWND in_box = GetDlgItem(hWnd, 3);

I like the EDT_INPUT_TEXT but I am not certain how to get my CreateWindow to use that. So, I used 3 instead.

You used:

                    SetDlgItemText( hWnd, EDT_OUTPUT_TEXT, utf8_byte_values(utf8).c_str());

which I changed to

                    SetDlgItemText(hWnd, 4, utf8_byte_values(utf8).c_str());

Again, I like the way that you did it, but I am having difficulty getting your line to work.

I am not certain what the

//            text.resize( n );

is. But thank you.


It works. Thank you. I am getting closer to being able to test on different platforms in different versions of C++.

For 办
I get 0xe5 0x8a 0x9e

Getting closer.

Lots of times I have wanted to see an update of the progress of code changes that other people were working on.

For future readers here is what currently works for me:

#define _UNICODE
#define UNICODE
 
#include <windows.h>
 
#include <iostream>
#include <sstream>      // for std::wstringstream
#include <iomanip>      // for std::setw
#include <string>
using namespace std;
 
#define MAX_LOADSTRING 100
 
HWND Handle_Main_Window = NULL;
 
#include <windows.h>
 
void create_controls( const HWND hwnd );
 
 
LRESULT CALLBACK WndProc(HWND hWnd, UINT msg, WPARAM wParam, LPARAM lParam);
 
wchar_t g_szClassName[] = L"myWindowClass";
 
// Previous declarations
    std::string utf8_encode(const std::wstring &wstr);
    std::wstring utf8_byte_values(const std::string &str);
 
 
    // Convert a wide Unicode string to an UTF8 string
    std::string utf8_encode(const std::wstring &wstr)
    {
        if (wstr.empty())
        {
            return std::string();
        }
 
        int size_needed = WideCharToMultiByte(CP_UTF8, 0, wstr.c_str(), (int)wstr.size(), nullptr, 0, nullptr, nullptr);
 
        char buffer[size_needed+1];
        WideCharToMultiByte(CP_UTF8, 0, wstr.c_str(), (int)wstr.size(), buffer, size_needed+1, nullptr, nullptr);
 
        std::string strTo( buffer );
        return strTo;
    }
 
 
 
    std::wstring utf8_byte_values(const std::string &str)
    {
        if (str.empty())
        {
            return std::wstring();
        }
 
        bool first = true;
        std::wstringstream out;
        // error: aggregate 'std::wstringstream out' has incomplete type and cannot be defined
 
        // I found this in <iosfwd>
        // Class for @c wchar_t mixed input and output memory streams.
        //   typedef basic_stringstream<wchar_t>     wstringstream;
        // Is that something from Visual Studio or maybe a later version of Code:Blocks?
 
        for(auto iter = str.begin(); iter != str.end(); ++iter)
        {
            if (first)
            {
                first = false;
            }
            else
            {
                out << L" ";
            }
 
            unsigned int value = ((unsigned)*iter) & 0xFF;
            out << L"0x" << std::hex << std::setw(2) << std::setfill(L'0') << value;
        }
 
        return out.str();
    }
 
 
 
int APIENTRY WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance,
    LPSTR lpCmdLine, int nCmdShow)
{
    WNDCLASSEX wc;
    MSG Msg;
 
    wc.cbSize        = sizeof(WNDCLASSEX);
    wc.style         = 0;
    wc.lpfnWndProc   = WndProc;
    wc.cbClsExtra    = 0;
    wc.cbWndExtra    = 0;
    wc.hInstance     = hInstance;
    wc.hIcon         = LoadIcon(nullptr, IDI_APPLICATION);
    wc.hCursor       = LoadCursor(nullptr, IDC_ARROW);
    wc.hbrBackground = (HBRUSH)(COLOR_WINDOW+1);
    wc.lpszMenuName  = nullptr;
    wc.lpszClassName = g_szClassName;
    wc.hIconSm       = LoadIcon(nullptr, IDI_APPLICATION);
 
    if(!RegisterClassEx(&wc))
    {
        MessageBox(nullptr, L"Window Registration Failed!", L"Error!",
            MB_ICONEXCLAMATION | MB_OK);
        return 0;
    }
 
    Handle_Main_Window = CreateWindowEx(
        WS_EX_CLIENTEDGE,
        g_szClassName,
        L"Title",
        WS_OVERLAPPEDWINDOW,
        CW_USEDEFAULT,
        CW_USEDEFAULT,
        500,
        500,
        nullptr,
        nullptr,
        hInstance,
        nullptr);
 
    if(Handle_Main_Window == NULL)
    {
        MessageBox(nullptr, L"Window Creation Failed!", L"Error!",
            MB_ICONEXCLAMATION | MB_OK);
        return 0;
    }
 
    ShowWindow(Handle_Main_Window, nCmdShow);
    UpdateWindow(Handle_Main_Window);
 
    while(GetMessage(&Msg, nullptr, 0, 0) > 0)
    {
        TranslateMessage(&Msg);
        DispatchMessage(&Msg);
    }
    return Msg.wParam;
}
 
 
LRESULT CALLBACK WndProc(HWND hWnd, UINT msg, WPARAM wParam, LPARAM lParam)
    {
        switch (msg)
        {
 
            case WM_CREATE:
                create_controls( hWnd );
                break;
 
            case WM_COMMAND:
                switch(LOWORD(wParam)) {
                case 1:{
                        ::MessageBox( hWnd, L"PUSH BUTTON 1 was clicked", L"message from PUSH BUTTON 1", MB_SETFOREGROUND );
                        break;
                    }
 
                case 2:{
                        const HWND text_box = GetDlgItem( hWnd, 3 );
                        const int n = GetWindowTextLength( text_box );
                        wstring text( n + 1, L'#' );
                        if( n > 0 )
                            {
                                GetWindowText( text_box, &text[0], text.length() );
                            }
                        text.resize( n );
                        ::MessageBox(hWnd, text.c_str(), L"The INPUT TEXT WINDOW", MB_SETFOREGROUND );
                        break;
                    }
 
                case 3:{
                        break;
                    }
 
                case 4:{
                        break;
                    }
 
               case 5:  //BTN_SAVE:
                {
//                    const HWND in_box = GetDlgItem( hWnd, EDT_INPUT_TEXT );
                    const HWND in_box = GetDlgItem(hWnd, 3);
                    const int n = GetWindowTextLength( in_box  );
                    if( n > 0 )
                    {
                        wchar_t text[n+1]; // +1 for terminator
                        GetWindowText( in_box, text, n+1 );
                        string utf8 = utf8_encode(wstring(text));
                        // Force calling of ASCII/UTF8 version
//                        SetDlgItemText( hWnd, EDT_OUTPUT_TEXT, utf8_byte_values(utf8).c_str());
                        SetDlgItemText(hWnd, 4, utf8_byte_values(utf8).c_str());
 
                    }
        //            text.resize( n );
                    break;
                }
 
                default:{
                    }
            }
            break;
 
            case WM_CLOSE:{
                    DestroyWindow(hWnd);
                    break;
                }
 
            case WM_DESTROY:{
                    PostQuitMessage(0);
                }
 
            default:{
                    return DefWindowProc(hWnd, msg, wParam, lParam);
                }
        }
        return FALSE;
    }
 
void create_controls( const HWND hwnd )
    {
 
        CreateWindow( L"BUTTON",
            L"PUSH BUTTON 1",
            WS_VISIBLE | WS_CHILD | WS_BORDER,
            10,10,
            130,20,
            hwnd, (HMENU) 1, GetModuleHandle( nullptr ), nullptr
            )  ;
 
        CreateWindow( L"EDIT",
            L"办",
            WS_VISIBLE | WS_CHILD | WS_BORDER,
            10,50,
            200,25,
            hwnd, (HMENU) 3, GetModuleHandle( nullptr ), nullptr
            );
 
        CreateWindow( L"BUTTON",
            L"SAVE BUTTON",
            WS_VISIBLE | WS_CHILD | WS_BORDER,
            10,80,
            110,20,
            hwnd, (HMENU) 5, GetModuleHandle( nullptr ), nullptr
            );
 
        CreateWindow( L"EDIT",
            L"OUTPUT TEXT WINDOW",
            WS_VISIBLE | WS_CHILD | WS_BORDER,
            10,130,
            300,300,
            hwnd, (HMENU) 4, GetModuleHandle( nullptr ), nullptr
            );
    }
 
 

Thank you.

[Banfa]

You used:

**SetDlgItemText(*hWnd,*EDT_OUTPUT_TEXT,*utf8_byte_values(utf8).c_str());

which I changed to

**SetDlgItemText(hWnd,*4,*utf8_byte_values(utf8).c_str());

Again, I like the way that you did it, but I am having difficulty getting your line to work.

#define EDT_OUTPUT_TEXT 4

At the top of the file.

If you have to use a number more than once it is a magic number. Magic numbers are very poor practice and you remove them by assigning them to a symbol, actually in C++ a const variable should be preferred to create this type of constant.

const int EDT_OUTPUT_TEXT = 4;

But this is WIN32 which I used with C so a reverted to #define.

[SioSio]

Tip 1.
An example of determining whether a character string contains non-alphanumeric symbols.

#include <iostream>
#include <regex>
 
/**
 * @brief Determine if it is an alphanumeric symbol.
 *
 * @return true:only alphanumeric / false:Contains non-alphanumeric symbols
 */
bool IsAlphabetNumericSymbol(std::string src)
{
    std::regex pattern("^[a-zA-Z0-9!-/:-@\[-`{-~]+$");
    std::smatch sm;
    if (std::regex_match(src, sm, pattern))
    {
        return true;
    }
    else
    {
        return false;
    }
}
 
int main()
{
    // Only alphanumeric case
    std::cout << IsAlphabetNumericSymbol("abc012@") << std::endl;
 
    // Contains non-alphanumeric symbols case
    std::cout << IsAlphabetNumericSymbol("1漢字A") << std::endl;
    return 0;
}

Tip 2.
"123漢字ABC" shown in UTF-16 is 16 bytes.

Tip 3.
Mutual conversion UTF-8 <=> UTF-16

inline std::wstring convertUtf8ToUtf16(char const* iString)
{
    std::wstring_convert<std::codecvt_utf8<wchar_t>, wchar_t> converter;
    return converter.from_bytes(iString);
}
 
inline std::string convertUtf16ToUtf8(wchar_t const* iString)
{
    std::wstring_convert<std::codecvt_utf8<wchar_t>, wchar_t> converter;
    return converter.to_bytes(iString);
}

Referenced URL.
https://docs.microsoft.com/en-us/arc...and-win32-apis

I hope you find this information helpful.

[SwissProgrammer]

I used

                         ::MessageBox(hWnd, L"SAVE BUTTON was clicked", L"message from SAVE BUTTON", MB_SETFOREGROUND );
 

I did not need the scope resolution operator :: before the Messagebox. I think it is sometimes used in Visual Studio. I have cleaned that out.

                         MessageBox(hWnd, L"SAVE BUTTON was clicked", L"message from SAVE BUTTON", MB_SETFOREGROUND );
 

I have been reading that

using namespace std;

adds a huge amount of code to a program. I am trying to avoid that by using

    using std::string;
    using std::wstring;
    using std::wstringstream;
    using std::hex;
    using std::setw;
    using std::setfill;

I am open to comments on that.



I worked on those "magic numbers" and I think that I fixed them.

const int PUSH_BUTTON_1     = 1;
const int IN_put_text_box   = 2;
const int SAVE_button       = 3;
const int OUT_put_text_box  = 4;

That took a lot longer that I had expected.



For

办 办

I get

0xe5 0x8a 0x9e 0x20 0x20 0x20 0xe5 0x8a 0x9e

I know that

办

is

0xe5 0x8a 0x9e

and the blanks are

0x20

but how do I separate it out automatically?
A simple response like

0xe5 0x8a 0x9e, 0x20, 0x20, 0x20, 0xe5 0x8a 0x9e

would tell me at least where the separation logic is. Then I could go forward and work with each. Someone please?

Here what I have so far

#define _UNICODE
#define UNICODE
 
#include <windows.h>
 
#include <iostream>
#include <sstream>      // for std::wstringstream
#include <iomanip>      // for std::setw
#include <string>
 
//using namespace std;  // do not need all of this namespace for this small program.
 
// Shortened version of namespace std;
    using std::string;
    using std::wstring;
    using std::wstringstream;
    using std::hex;
    using std::setw;
    using std::setfill;
 
#define MAX_LOADSTRING 100
 
HWND Handle_Main_Window = NULL;
 
#include <windows.h>
 
wchar_t g_szClassName[] = L"myWindowClass";
 
const int PUSH_BUTTON_1     = 1;
const int IN_put_text_box   = 2;
const int SAVE_button       = 3;
const int OUT_put_text_box  = 4;
 
 
// Previous declarations
    void create_controls( const HWND hwnd );
    LRESULT CALLBACK WndProc(HWND hWnd, UINT msg, WPARAM wParam, LPARAM lParam);
    string utf8_encode(const wstring &wstr);
    wstring utf8_byte_values(const string &str);
 
 
// Convert a wide Unicode string to an UTF8 string
string utf8_encode(const wstring &wstr)
    {
        if (wstr.empty())
            {
                return string();
            }
 
        int size_needed = WideCharToMultiByte(CP_UTF8, 0, wstr.c_str(), (int)wstr.size(), nullptr, 0, nullptr, nullptr);
 
        char buffer[size_needed+1];
        WideCharToMultiByte(CP_UTF8, 0, wstr.c_str(), (int)wstr.size(), buffer, size_needed+1, nullptr, nullptr);
 
        string strTo( buffer );
        return strTo;
    }
 
 
 
wstring utf8_byte_values(const string &str)
    {
        if (str.empty())
            {
                return wstring();
            }
 
        bool first = true;
        wstringstream out;
 
        for(auto iter = str.begin(); iter != str.end(); ++iter)
            {
                if (first)
                    {
                        first = false;
                    }
                else
                    {
                        out << L" ";
                    }
 
                unsigned int value = ((unsigned)*iter) & 0xFF;
                out << L"0x" << hex << setw(2) << setfill(L'0') << value;
            }
 
        return out.str();
    }
 
 
 
int APIENTRY WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance,
    LPSTR lpCmdLine, int nCmdShow)
    {
        WNDCLASSEX wc;
        MSG Msg;
 
        wc.cbSize        = sizeof(WNDCLASSEX);
        wc.style         = 0;
        wc.lpfnWndProc   = WndProc;
        wc.cbClsExtra    = 0;
        wc.cbWndExtra    = 0;
        wc.hInstance     = hInstance;
        wc.hIcon         = LoadIcon(nullptr, IDI_APPLICATION);
        wc.hCursor       = LoadCursor(nullptr, IDC_ARROW);
        wc.hbrBackground = (HBRUSH)(COLOR_WINDOW+1);
        wc.lpszMenuName  = nullptr;
        wc.lpszClassName = g_szClassName;
        wc.hIconSm       = LoadIcon(nullptr, IDI_APPLICATION);
 
        if(!RegisterClassEx(&wc))
            {
                MessageBox(nullptr, L"Window Registration Failed!", L"Error!", MB_ICONEXCLAMATION | MB_OK);
                return 0;
            }
 
        Handle_Main_Window = CreateWindowEx(
            WS_EX_CLIENTEDGE,
            g_szClassName,
            L"Title",
            WS_OVERLAPPEDWINDOW,
            CW_USEDEFAULT,
            CW_USEDEFAULT,
            630,
            470,
            nullptr,
            nullptr,
            hInstance,
            nullptr);
 
        if(Handle_Main_Window == NULL)
            {
                MessageBox(nullptr, L"Window Creation Failed!", L"Error!", MB_ICONEXCLAMATION | MB_OK);
                return 0;
            }
 
        ShowWindow(Handle_Main_Window, nCmdShow);
        UpdateWindow(Handle_Main_Window);
 
        while(GetMessage(&Msg, nullptr, 0, 0) > 0)
            {
                TranslateMessage(&Msg);
                DispatchMessage(&Msg);
            }
        return Msg.wParam;
    }
 
 
LRESULT CALLBACK WndProc(HWND hWnd, UINT msg, WPARAM wParam, LPARAM lParam)
    {
        switch (msg)
            {
 
                case WM_CREATE:
                    {
                        create_controls( hWnd );
                        break;
                    }
 
                case WM_COMMAND:
                    {
                        switch(LOWORD(wParam))
                            {
 
                                case PUSH_BUTTON_1:
                                    {
                                        MessageBox( hWnd, L"PUSH BUTTON 1 was clicked", L"message from PUSH BUTTON 1", MB_SETFOREGROUND );
                                        break;
                                    }
 
                                case IN_put_text_box:
                                    {
                                        const HWND in_box = GetDlgItem(hWnd, IN_put_text_box);
                                        const int n = GetWindowTextLength( in_box  );
                                        if( n > 0 )
                                            {
                                                wchar_t text[n+1]; // +1 for terminator
                                                GetWindowText( in_box, text, n+1 );
                                                string utf8 = utf8_encode(wstring(text));
                                                // Force calling of ASCII/UTF8 version
                                                SetDlgItemText(hWnd, OUT_put_text_box, utf8_byte_values(utf8).c_str());
 
                                            }
                                        else    //if( n = 0 )
                                            {
                                                wchar_t text[n+1]; // +1 for terminator
                                                GetWindowText( in_box, text, n+1 );
                                                string utf8 = utf8_encode(wstring(text));
                                                // Force calling of ASCII/UTF8 version
                                                SetDlgItemText(hWnd, OUT_put_text_box, utf8_byte_values(utf8).c_str());
                                            }
                                        break;
                                    }
 
                                case SAVE_button:  //BTN_SAVE:
                                    {
                                        const HWND in_box = GetDlgItem(hWnd, IN_put_text_box);
                                        const int n = GetWindowTextLength( in_box  );
                                        if( n > 0 )
                                            {
                                                wchar_t text[n+1]; // +1 for terminator
                                                GetWindowText( in_box, text, n+1 );
                                                string utf8 = utf8_encode(wstring(text));
                                                // Force calling of ASCII/UTF8 version
                                                SetDlgItemText(hWnd, OUT_put_text_box, utf8_byte_values(utf8).c_str());
                                            }
                                        else    //if( n = 0 )
                                            {
                                                wchar_t text[n+1]; // +1 for terminator
                                                GetWindowText( in_box, text, n+1 );
                                                string utf8 = utf8_encode(wstring(text));
                                                // Force calling of ASCII/UTF8 version
                                                SetDlgItemText(hWnd, OUT_put_text_box, utf8_byte_values(utf8).c_str());
                                            }
                                        break;
                                    }
 
                                default:
                                    {
                                    }
                        }
 
                        break;
                    }
 
                case WM_CLOSE:
                    {
                        DestroyWindow(hWnd);
                        break;
                    }
 
                case WM_DESTROY:
                    {
                        PostQuitMessage(0);
                    }
 
                default:
                    {
                        return DefWindowProc(hWnd, msg, wParam, lParam);
                    }
            }
 
        return FALSE;
    }
 
void create_controls( const HWND hwnd )
    {
 
        CreateWindowW
            (
                L"BUTTON",
                L"PUSH BUTTON 1",
                WS_VISIBLE | WS_CHILD | WS_BORDER,
                10,10,
                130,20,
                hwnd,
                (HMENU) PUSH_BUTTON_1,
                GetModuleHandle( nullptr ),
                nullptr
            );
 
        CreateWindowW
            (
                L"EDIT",
                L"办   办",
                WS_VISIBLE | WS_CHILD | WS_BORDER,
                10,50,
                200,25,
                hwnd,
                (HMENU) IN_put_text_box,
                GetModuleHandle( nullptr ),
                nullptr
            );
 
        CreateWindowW
            (
                L"BUTTON",
                L"SAVE BUTTON",
                WS_VISIBLE | WS_CHILD | WS_BORDER,
                10,80,
                110,20,
                hwnd,
                (HMENU) SAVE_button,
                GetModuleHandle( nullptr ),
                nullptr
            );
 
        CreateWindowW
            (
                L"EDIT",
                L"OUTPUT TEXT WINDOW",
                WS_VISIBLE | WS_CHILD | WS_BORDER,
                10,130,
                600,300,
                hwnd,
                (HMENU) OUT_put_text_box,
                GetModuleHandle( nullptr ),
                nullptr
            );
    }
 
 

Thank you.

[Banfa]

Mostly the scope resolution operator :: is unnecessary but it is there for the odd occasion where there is a clash between a class member name and a top level symbol name, i.e. suppose you are trying to call ::MessageBox from within a class with a member called MessageBox.

Using using <symbol> rather than using namespace <NamespaceName> is exactly what we do in production code in work for the very same reason, to reduce the number of symbols being imported into the global namespace. I'd definately recommend it although you can end up with a large section of using directives at the top of your source files.

Once you get used to using constants for magic numbers from project start it becomes easier. Make use you decide on a naming convention for these constants an stick to it, all caps with underscores is a common standard.

Maybe try treating the input string 1 character at a time discarding the space characters or try using std::string::find_first_of to locate spaces in either the input or output.

[SwissProgrammer]

Banfa,

Update: I have been looking at std::string::find_first_of, but It does not seem to work with Unicode. And, I get messages of Visual Studio problems with similar attempts. I am trying to not use VS in any way, so I am still working on this. Thank you. I might eventually get the ability to parse or split Unicode strings into each single or combined character. Thanks for now.

[SwissProgrammer]

SioSio,

You said, "In the C++11 Standard Library, UTF-8 is not supported for string and integer conversion functions, and I/O functions. Therefore, it needs to be converted to the system multibyte character code."

I have read that std::wstring is better for parsing than multibyte. I have also read that multibyte is more useful. Which direction should I study to be able to parse the following into each individual characters?

For now I am just trying to do the test with Unicode plane 0.


If someone pastes or places into my text box the following:

123漢字ABC

I want the second text box to show the following:

Full Sentence in UTF-8
\x31\x32\x33\xe6\xbc\xa2\xe5\xad\x97\x41\x42\x43


Individual Single Characters in UTF-8

1 = \x31
2 = \x32
3 = \x33
漢 = \xe6\xbc\xa2
字 = \xe5\xad\x97
A = \x41
B = \x42
C = \x43

All that in a single text box showing all those lines.

Should I work at doing this with multibyte or with std::wstring?

I almost got it to work a few times, but I am not certain what I did.

I thought to split the input sentence into individual characters in the following area, but I lost it. Maybe later I can show you what I did if I get it close again.

                                case SAVE_button:  //BTN_SAVE:
                                    {
                                        const HWND in_box = GetDlgItem(hWnd, IN_put_text_box);
                                        const int n = GetWindowTextLength( in_box  );
                                        if( n > 0 )
                                            {
                                                wchar_t text[n+1]; // +1 for terminator
                                                GetWindowText( in_box, text, n+1 );
                                                string utf8 = utf8_encode(wstring(text));
                                                // Force calling of ASCII/UTF8 version
                                                SetDlgItemText(hWnd, OUT_put_text_box, utf8_byte_values(utf8).c_str());
                                            }
                                        else    //if( n = 0 )
                                            {
                                                wchar_t text[n+1]; // +1 for terminator
                                                GetWindowText( in_box, text, n+1 );
                                                string utf8 = utf8_encode(wstring(text));
                                                // Force calling of ASCII/UTF8 version
                                                SetDlgItemText(hWnd, OUT_put_text_box, utf8_byte_values(utf8).c_str());
                                            }
                                        break;
                                    }

Or should I use UTF-16 or UTF-32?

I also found, "The better way is to use std::u16string (std::basic_string<char16_t>) and std::u32string (std::basic_string<char32_t>). They'll work regardless of system and encoding of the source file" [here].

I have been struggling with this for a while, and there is lots of advice for and against lots of stuff, most of which I have not gotten to work except temporarily and now I know that something did work, but I do not remember what did.

Thank you.

[SioSio]

Correspondence between UTF-8 bytes and code point range.
The number of bytes when a character is encoded in UTF-8 can be derived as follows.

For 1 byte, the bit pattern is 0xxx xxxx.
The number of valid bits is 7, the maximum value is 0111 1111 = 0x7F.
The range to be represented is 0x0000 to 0x007F (within ASCII range)
For 2 bytes, the bit pattern is 110x xxxx 10xx xxxx.
The effective number of bits is 5 + 6 = 11, the maximum value is 0111 1111 1111 = 0x7FF.
The range to represent is 0x0080 ~ 0x07FF.
For 3 bytes, the bit pattern is 1110 xxxx 10xx xxxx 10xx xxxx.
The number of valid bits is 4 + 6 + 6 = 16 (exactly 2 bytes), and the maximum value is 0xFFFF.
The range to be represented is all the remaining characters in the UCS-2 range.
4 bytes are only the surrogate pair part (surface 00 to surface 10H).

In other words, since the number of continuous bytes can be known from the bit pattern of the first byte, it can be divided character by character by using this.

[SwissProgrammer]

I am a beginner at C++ and this is where I am in this current process. Maybe other beginners can learn from this.


HELPFUL COMMENTS on the following ARE APPRECIATED !



I will now try to find the following

0xxx
110x
1110

In analyzing equivalents of the string "123漢字ABC"

Using online converters [with reference links];

Unicode text = UTF-8[1] = Decimal[1] = Binary[2]

I added the Decimal equivalents so that I could have that option.



1 = \x31 = 00049 = 00110001

2 = \x32 = 00050 = 00110010

3 = \x33 = 00051 = 00110011

漢 = \xe6\xbc\xa2 = 28450 = 11100110 10111100 10100010

字 = \xe5\xad\x97 = 23383 = 11100101 10101101 10010111

A = \x41 = 00065 = 01000001

B = \x42 = 00066 = 01000010

C = \x43 = 00067 = 01000011


If I start with the Unicode string "123漢字ABC" as it appears in binary form on [2] using full length bytes and placing a space after every byte.

I get the following:

With spacing

00110001 00110010 00110011 11100110 10111100 10100010
11100101 10101101 10010111 01000001 01000010 01000011

or

Without spacing

00110001001100100011001111100110101111001010001011 1001011010110110010111010000010100001001000011

If I split that as SioSio says:

I first look for 0xxx or 110x or 1110 and set those as separate. I am not certain how that works with Unicode planes above 0 (1 to 16), but this is where I am now.

00110001 starts with 0xxx,

00110010 starts with 0xxx,

00110010 starts with 0xxx,

11100110 starts with 1110,

10111100 does not start with any of those so it is a continuation of the previous,

10100010 does not start with any of those so it is a continuation of the previous,

Thus: 11100110 10111100 10100010 are together.

11100101 starts with 1110,

10101101 does not start with any of those so it is a continuation of the previous,

10010111 does not start with any of those so it is a continuation of the previous,

Thus: 11100101 10101101 10010111 are together.

01000001 starts with 0xxx,

01000010 starts with 0xxx,

01000011 starts with 0xxx,


Result (without spaces):

00110001
00110010
00110010
111001101011110010100010
111001011010110110010111
01000001
01000010
01000011

Should I convert this to UTF-8 or just use it in binary form at this time before I convert it to readable characters?

On the link [3] they go from binary to text characters. Maybe I could use C++ to go directly from binary to text characters. (?)



If the previous is correct, I now have a process to put into C++11 code.

1. Convert text (in the textbox) to binary.
2. Split the binary into (parts or sections or some other phrase) with spaces.
3. Search for 0xxx, and 110x, and 1110 in that binary to get the starting of each (what that thing is called).
4. Save those [things] to an array of them for later use.
5. Convert each of those [things] to readable Unicode characters.
6. Save those Unicode characters to an array of them for later use.
7. Parse the text, change the text, edit the text, by using the arrays.

Comments PLEASE !

Am I doing this right?


Thank you.

[SioSio]

Hi.
It doesn't have to be that complicated.

The code to split the UTF-8 string character by character is as follows.

#include <iostream>
#include <string>
 
void print_char(std::string);
 
int main(void)
{
    using namespace std;
    string str = u8"123漢字ABC";
    print_char(str);
    return 0;
}
 
void print_char(std::string str)
{
    using namespace std;
    int pos;
    unsigned char ch;
    int char_size;
    for (pos = 0; pos < str.size(); pos += char_size) {
        ch = str[pos];
        if (ch < 0x80) {
            char_size = 1;
        }
        else if (ch < 0xE0) {
            char_size = 2;
        }
        else if (ch < 0xF0) {
            char_size = 3;
        }
        else {
            char_size = 4;
        }
        cout << str.substr(pos, char_size) << '\n';
    }
}

[SwissProgrammer]

SioSio,

        for (pos = 0; pos < str.size(); pos += char_size) {

I get
warning: comparison between signed and unsigned integer expressions [-Wsign-compare]|

and

I tried

void print_char(std::string str)
    {
        using namespace std;
        unsigned int pos;
        unsigned char ch;
        int char_size;
        for (pos = 0; pos < str.size(); pos += char_size) {

I get
Process terminated with status 0 (0 minute(s), 0 second(s))

and

I tried

{
        using namespace std;
        int pos;
        unsigned char ch;
        unsigned int char_size;
        for (pos = 0; pos < str.size(); pos += char_size) {

I get
Process terminated with status 0 (0 minute(s), 0 second(s))

and

I tried

{
        using namespace std;
        int pos;
        char ch;
        int char_size;
        for (pos = 0; pos < str.size(); pos += char_size) {

I get
warning: comparison between signed and unsigned integer expressions [-Wsign-compare]|
With more warnings.

It looks impressive, but why will it not let me make these adjustments to work?

I am working on this.

Thank you.

[SioSio]

I think that warningdisappears with the following modifications.

    unsigned int pos;
    unsigned char ch;
    unsigned int char_size;

std::wstring_convert is available in C++11.
How to convert std::wstring to a byte string with std::wstring_convert and display it with std::cout.

#include <iostream>
#include <string>
#include <codecvt>
 
int main(void)
{
    using namespace std;
    wstring_convert<codecvt_utf8<wchar_t>, wchar_t> cv;
    wstring wstr = L"123漢字ABC";
 
    for (auto v : wstr) {
        cout << cv.to_bytes(v) << '\n';
    }
    return 0;
}

[SwissProgrammer]

For

        wstring_convert<codecvt_utf8<wchar_t>, wchar_t> cv;

I get: wstring_convert was not declared in this scope

I found class wstring_convert in locale_conv.h .
When I added the String conversions from that header file, it gave more errors.

I added #include <locale>
and that fixed that part.

[SwissProgrammer]

OK I got it to work as a console application.

The 漢字 do not show up right, but they are separated out like the rest as they should be. I think that I am beginning to understand what you are doing.

#include <iostream>
#include <string>
#include <codecvt>
#include <locale>
 
#include <conio.h>
#include <stdio.h>
 
// Previous declarations
    void PressAKeyToContinue();
 
 
int main(void)
    {
        using namespace std;
        wstring_convert<codecvt_utf8<wchar_t>, wchar_t> cv;
        wstring wstr = L"123漢字ABC";
 
        for (auto v : wstr) 
            {
                cout << cv.to_bytes(v) << '\n';
                // .to_bytes  Reference: https://en.cppreference.com/w/cpp/locale/wstring_convert/to_bytes
            }
 
        PressAKeyToContinue();
        return 0;
    }
 
 
void PressAKeyToContinue()
    {
        int c;
        printf( "\nPress a key to continue..." );
        c = getch();
        if (c == 0 || c == 224) getch();
    }
 

Thank you.

[SwissProgrammer]

I think that the code is probably working correctly, but my CLI might be the problem and not showing the correct response.


I am working on putting this into the previous code with the textboxes, etc. I will try to get back to you on that if I get it to work.


Thank you.

[SwissProgrammer]

Notes for future readers relating to reports on codecvt_utf8 and surrogate pairs.

I have been studying codecvt_utf8 from

wstring_convert<codecvt_utf8<wchar_t>, wchar_t> cv;

I have read that codecvt_utf8 is reported to be buggy.

"Microsoft's implementation of std::codecvt_utf8 appears to successfully convert any UTF-16 code unit into UTF-8—including surrogate pairs. This is a bug, as surrogates are not encodable." There seem to be others with this opinion.

I tried

wstring_convert<std::codecvt_utf8_utf16<wchar_t>, wchar_t> cv;

Which worked fine as a replacement. And, if I understand it, it assumes that the wchar_t is utf-16 [X} for the conversion, which I have been reading that Microsoft Windows seems to use internally for all Unicode. Later I read suggestions that this is also buggy.

Why the reports of buggyness? I found the answer:

After further research I found that this buggyness is reported to only apply to narrow character surrogates. Narrow. Not for wide. Not for wide characters which I am using, example wchar or for wchar_t.

I am considering both codecvt_utf8 and codecvt_utf8_utf16 safe to use.

The use of either of these did not cause my CLI to show the correct Unicode characters. So, I researched and found, "Windows 95/98/Me: ExtTextOutW is supported by the Microsoft Layer for Unicode." That meant, to me, that something should work. For CLI, I do not know yet. I found on an Oracle page [X] "James Kass's Code2001 font" which I downloaded from a link on here to Code2000 font by James Kass (CODE2001.TTF). I unzipped it and placed it into the Windows/fonts directory, where it was automatically installed by XP. Now, a test page shows up correctly in my Firefox 43.0b9 browser.

[SwissProgrammer]

How do I completely delete this one message?

[SioSio]

https://bytes.com/forums/feedback/87...ow-delete-post

[SwissProgrammer]

"Use the report button (the triangle icon at the top of each post) and detail the reason why you think the post should be deleted and a moderator will look into the matter for you."

I do not see any triangle icon at the top or bottom of my post.

If that should mean the "report abuse" button or icon below the message, then would that go as a record of the poster being reported as abusive?

I would like to be able to make a reply, then after posting the reply, if I find that it was wrong, to be able to (as the poster) delete or remove the reply. I tried to edit it down to zero text, but I get an error message.

OK, I will try that.

It says, "Note: This is ONLY to be used to report spam, advertising messages, and problematic (harassment, fighting, or rude) posts." I can not un-post?

[SioSio]

"the triangle icon at the top of each post"
Certainly, I can't find it.
There are cases where I want to delete post, such as when I have posted wrong information. I also want to know how to delete post.

[SwissProgrammer]

I have fought with this .to_bytes for a week to convert it from working with the CLI to working with my GUI.


I think that I should replace in

                                    case IN_put_text_box:
                                        {
                                            const HWND in_box = GetDlgItem(hWnd, IN_put_text_box);
                                            const int n = GetWindowTextLength( in_box  );
                                            if( n > 0 )
                                                {
                                                    wchar_t text[n+1]; // +1 for terminator
                                                    GetWindowText( in_box, text, n+1 );
                                                    string utf8 = utf8_encode(wstring(text));
                                                    // Force calling of ASCII/UTF8 version
                                                    SetDlgItemText(hWnd, OUT_put_text_box, utf8_byte_values(utf8).c_str());
 
                                                }
                                            else    //if( n = 0 )
                                                {
                                                    wchar_t text[n+1]; // +1 for terminator
                                                    GetWindowText( in_box, text, n+1 );
                                                    string utf8 = utf8_encode(wstring(text));
                                                    // Force calling of ASCII/UTF8 version
                                                    SetDlgItemText(hWnd, OUT_put_text_box, utf8_byte_values(utf8).c_str());
                                                }
                                            break;
 

of where I was so far:

 
    #define _UNICODE
    #define UNICODE
 
    #include <windows.h>
 
    #include <iostream>
    #include <sstream>      // for std::wstringstream
    #include <iomanip>      // for std::setw
    #include <string>
 
    //using namespace std;  // do not need all of this namespace for this small program.
 
    // Shortened version of namespace std;
        using std::string;
        using std::wstring;
        using std::wstringstream;
        using std::hex;
        using std::setw;
        using std::setfill;
 
    #define MAX_LOADSTRING 100
 
    HWND Handle_Main_Window = NULL;
 
    #include <windows.h>
 
    wchar_t g_szClassName[] = L"myWindowClass";
 
    const int PUSH_BUTTON_1     = 1;
    const int IN_put_text_box   = 2;
    const int SAVE_button       = 3;
    const int OUT_put_text_box  = 4;
 
 
    // Previous declarations
        void create_controls( const HWND hwnd );
        LRESULT CALLBACK WndProc(HWND hWnd, UINT msg, WPARAM wParam, LPARAM lParam);
        string utf8_encode(const wstring &wstr);
        wstring utf8_byte_values(const string &str);
 
 
    // Convert a wide Unicode string to an UTF8 string
    string utf8_encode(const wstring &wstr)
        {
            if (wstr.empty())
                {
                    return string();
                }
 
            int size_needed = WideCharToMultiByte(CP_UTF8, 0, wstr.c_str(), (int)wstr.size(), nullptr, 0, nullptr, nullptr);
 
            char buffer[size_needed+1];
            WideCharToMultiByte(CP_UTF8, 0, wstr.c_str(), (int)wstr.size(), buffer, size_needed+1, nullptr, nullptr);
 
            string strTo( buffer );
            return strTo;
        }
 
 
 
    wstring utf8_byte_values(const string &str)
        {
            if (str.empty())
                {
                    return wstring();
                }
 
            bool first = true;
            wstringstream out;
 
            for(auto iter = str.begin(); iter != str.end(); ++iter)
                {
                    if (first)
                        {
                            first = false;
                        }
                    else
                        {
                            out << L" ";
                        }
 
                    unsigned int value = ((unsigned)*iter) & 0xFF;
                    out << L"0x" << hex << setw(2) << setfill(L'0') << value;
                }
 
            return out.str();
        }
 
 
 
    int APIENTRY WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance,
        LPSTR lpCmdLine, int nCmdShow)
        {
            WNDCLASSEX wc;
            MSG Msg;
 
            wc.cbSize        = sizeof(WNDCLASSEX);
            wc.style         = 0;
            wc.lpfnWndProc   = WndProc;
            wc.cbClsExtra    = 0;
            wc.cbWndExtra    = 0;
            wc.hInstance     = hInstance;
            wc.hIcon         = LoadIcon(nullptr, IDI_APPLICATION);
            wc.hCursor       = LoadCursor(nullptr, IDC_ARROW);
            wc.hbrBackground = (HBRUSH)(COLOR_WINDOW+1);
            wc.lpszMenuName  = nullptr;
            wc.lpszClassName = g_szClassName;
            wc.hIconSm       = LoadIcon(nullptr, IDI_APPLICATION);
 
            if(!RegisterClassEx(&wc))
                {
                    MessageBox(nullptr, L"Window Registration Failed!", L"Error!", MB_ICONEXCLAMATION | MB_OK);
                    return 0;
                }
 
            Handle_Main_Window = CreateWindowEx(
                WS_EX_CLIENTEDGE,
                g_szClassName,
                L"Title",
                WS_OVERLAPPEDWINDOW,
                CW_USEDEFAULT,
                CW_USEDEFAULT,
                630,
                470,
                nullptr,
                nullptr,
                hInstance,
                nullptr);
 
            if(Handle_Main_Window == NULL)
                {
                    MessageBox(nullptr, L"Window Creation Failed!", L"Error!", MB_ICONEXCLAMATION | MB_OK);
                    return 0;
                }
 
            ShowWindow(Handle_Main_Window, nCmdShow);
            UpdateWindow(Handle_Main_Window);
 
            while(GetMessage(&Msg, nullptr, 0, 0) > 0)
                {
                    TranslateMessage(&Msg);
                    DispatchMessage(&Msg);
                }
            return Msg.wParam;
        }
 
 
    LRESULT CALLBACK WndProc(HWND hWnd, UINT msg, WPARAM wParam, LPARAM lParam)
        {
            switch (msg)
                {
 
                    case WM_CREATE:
                        {
                            create_controls( hWnd );
                            break;
                        }
 
                    case WM_COMMAND:
                        {
                            switch(LOWORD(wParam))
                                {
 
                                    case PUSH_BUTTON_1:
                                        {
                                            MessageBox( hWnd, L"PUSH BUTTON 1 was clicked", L"message from PUSH BUTTON 1", MB_SETFOREGROUND );
                                            break;
                                        }
 
                                    case IN_put_text_box:
                                        {
                                            const HWND in_box = GetDlgItem(hWnd, IN_put_text_box);
                                            const int n = GetWindowTextLength( in_box  );
                                            if( n > 0 )
                                                {
                                                    wchar_t text[n+1]; // +1 for terminator
                                                    GetWindowText( in_box, text, n+1 );
                                                    string utf8 = utf8_encode(wstring(text));
                                                    // Force calling of ASCII/UTF8 version
                                                    SetDlgItemText(hWnd, OUT_put_text_box, utf8_byte_values(utf8).c_str());
 
                                                }
                                            else    //if( n = 0 )
                                                {
                                                    wchar_t text[n+1]; // +1 for terminator
                                                    GetWindowText( in_box, text, n+1 );
                                                    string utf8 = utf8_encode(wstring(text));
                                                    // Force calling of ASCII/UTF8 version
                                                    SetDlgItemText(hWnd, OUT_put_text_box, utf8_byte_values(utf8).c_str());
                                                }
                                            break;
                                        }
 
                                    case SAVE_button:  //BTN_SAVE:
                                        {
                                            const HWND in_box = GetDlgItem(hWnd, IN_put_text_box);
                                            const int n = GetWindowTextLength( in_box  );
                                            if( n > 0 )
                                                {
                                                    wchar_t text[n+1]; // +1 for terminator
                                                    GetWindowText( in_box, text, n+1 );
                                                    string utf8 = utf8_encode(wstring(text));
                                                    // Force calling of ASCII/UTF8 version
                                                    SetDlgItemText(hWnd, OUT_put_text_box, utf8_byte_values(utf8).c_str());
                                                }
                                            else    //if( n = 0 )
                                                {
                                                    wchar_t text[n+1]; // +1 for terminator
                                                    GetWindowText( in_box, text, n+1 );
                                                    string utf8 = utf8_encode(wstring(text));
                                                    // Force calling of ASCII/UTF8 version
                                                    SetDlgItemText(hWnd, OUT_put_text_box, utf8_byte_values(utf8).c_str());
                                                }
                                            break;
                                        }
 
                                    default:
                                        {
                                        }
                            }
 
                            break;
                        }
 
                    case WM_CLOSE:
                        {
                            DestroyWindow(hWnd);
                            break;
                        }
 
                    case WM_DESTROY:
                        {
                            PostQuitMessage(0);
                        }
 
                    default:
                        {
                            return DefWindowProc(hWnd, msg, wParam, lParam);
                        }
                }
 
            return FALSE;
        }
 
    void create_controls( const HWND hwnd )
        {
 
            CreateWindowW
                (
                    L"BUTTON",
                    L"PUSH BUTTON 1",
                    WS_VISIBLE | WS_CHILD | WS_BORDER,
                    10,10,
                    130,20,
                    hwnd,
                    (HMENU) PUSH_BUTTON_1,
                    GetModuleHandle( nullptr ),
                    nullptr
                );
 
            CreateWindowW
                (
                    L"EDIT",
                    L"办   办",
                    WS_VISIBLE | WS_CHILD | WS_BORDER,
                    10,50,
                    200,25,
                    hwnd,
                    (HMENU) IN_put_text_box,
                    GetModuleHandle( nullptr ),
                    nullptr
                );
 
            CreateWindowW
                (
                    L"BUTTON",
                    L"SAVE BUTTON",
                    WS_VISIBLE | WS_CHILD | WS_BORDER,
                    10,80,
                    110,20,
                    hwnd,
                    (HMENU) SAVE_button,
                    GetModuleHandle( nullptr ),
                    nullptr
                );
 
            CreateWindowW
                (
                    L"EDIT",
                    L"OUTPUT TEXT WINDOW",
                    WS_VISIBLE | WS_CHILD | WS_BORDER,
                    10,130,
                    600,300,
                    hwnd,
                    (HMENU) OUT_put_text_box,
                    GetModuleHandle( nullptr ),
                    nullptr
                );
        }
 
 
 
 

with this

                                    case IN_put_text_box:
                                        {
                                            const HWND in_box = GetDlgItem(hWnd, IN_put_text_box);
                                            const int n = GetWindowTextLength( in_box  );
                                            if( n > 0 )
                                                {
                                                    wchar_t text[n+1]; // +1 for terminator
                                                    GetWindowText( in_box, text, n+1 );
                                                    string utf8 = utf8_encode(wstring(text.to_bytes));   // CHANGED HERE
                                                    // Force calling of ASCII/UTF8 version
                                                    SetDlgItemText(hWnd, OUT_put_text_box, (utf8).c_str());   // CHANGED HERE
 
                                                }
                                            else    //if( n = 0 )
                                                {
                                                    wchar_t text[n+1]; // +1 for terminator
                                                    GetWindowText( in_box, text, n+1 );
                                                    string utf8 = utf8_encode(wstring(text));
                                                    // Force calling of ASCII/UTF8 version
                                                    SetDlgItemText(hWnd, OUT_put_text_box, utf8_byte_values(utf8).c_str());
                                                }
                                            break;
                                        }
 

I have tried to vary it. It is not working.

What is the logic of how to make this work?

I have been studying .to_bytes and most of what I get is for CLI (which I worked through). I am having difficulty with understanding how to make it work in GUI.

I get that .to_bytes gets each entire character, one at a time. Then what? Then why for that what.

Thank you if you show me how to make it work with GUI, but please tell me why it works that way. Other online explanations are not helping me enough.

Thank you.

[SwissProgrammer]

SioSio,

I tried using .to-bytes, but I am not certain that it is cross-platform (Windows, Mac, Unix) usable. Is it?

Thank you.

You said,
"the number of continuous bytes can be known from the bit pattern of the first byte"

I might want to reference how I found more of the following and what I found later, and other people might want to read this years later to understand, so I will explain:

I tried the following with a Command Line Interface via Code::Blocks 17.12 on Windows XP Professional (service pack 2) 32-bit:

0xxx
110x
1110
1111

I tested 0xxx and 110x and 1110. I found them to work for me in Plane 0 Unicode.

In testing, by myself of more than 10 Unicode characters from Plane 1, I found that 1111 works.

I found this on a Microsoft page [X]:

All the following bytes start with the mark "10" and the xxx's denote the binary representation of the encoding within the given range.

Unicode Range

UTF-8 Encoded Bytes

0x0000-0x007F

0 xxxxxxx

0x0080-0x07FF

110 xxxxx 10xxxxxx

0x0800-0xFFFF

1110 xxxx 10xxxxxx 10xxxxxx

0x10000-0x1FFFFF

11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

Similar here: [X], [X], [X], [X]

Since Plane 16 Unicode (currently the highest) goes up to 10FFFF, and the last code point of 11110xxx is the same 10FFFF therefore I think that you have helped me greatly with this.

Thank you SioSio.

Thank you everyone else that helped me to understand what Unicode is doing and what C and C++ are doing as it relates to Unicode.

Please remember that I am still a beginner at C++ and I am still struggling with understanding a lot, but I am not allowing that to stop me.

Here is where I am with this:

 
#define _UNICODE
#define UNICODE
 
#include <windows.h>
#include <commctrl.h>
 
#include <iostream>
#include <bitset>
using namespace std;
 
string TextToBinaryString(string words) {
    string tempString = "";
    string splitIntoBytes = "Spliting [" + words + "] into Bytes\n";
    string binaryString = "";
    string stringByte;
 
    string combined_1;
    string combined_3;
    string combined_4;
 
    std::string Number1;
    std::string Number2;
    std::string Number3;
    std::string Number4;
 
    int i=1;
    string numToString = "numToString is Not Working";
 
    for (char& _char : words) {
 
        // From Plane 0 Unicode Check for 0xxx or 110x or 1110
        // From plane 1 Unicode Check for 1111                  Reference: http://www.i18nguy.com/unicode-plane1-utf8.html
 
        numToString = to_string (i);;
        stringByte = bitset<8>(_char).to_string();
 
        i = i +1;
 
        binaryString = binaryString + "\n\n" + numToString + " Testing       " +  stringByte + "\n";
 
        binaryString =binaryString + "     _char====" + _char + "\n";
 
        Number1 = stringByte[0];
        Number2 = stringByte[1];
        Number3 = stringByte[2];
        Number4 = stringByte[3];
 
        // 0
        combined_1 = Number1;
 
        // 110 , 111
        combined_3 = Number1 + Number2 + Number3;
 
        // 1111 with verification of not being 1110
        combined_4 = Number1 + Number2 + Number3 + Number4;
 
 
        binaryString = binaryString + "     Number1=[" + Number1 + "]\n";
 
        binaryString = binaryString + "     Number2=[" + Number2 + "]\n";
 
        binaryString = binaryString + "     Number3=[" + Number3 + "]\n";
 
        binaryString = binaryString + "     Number4=[" + Number4 + "]\n\n";
 
 
        binaryString = binaryString + "     combined_1=[" + combined_1 + "]\n";
 
        binaryString = binaryString + "     combined_3=[" + combined_3 + "]\n";
 
        binaryString = binaryString + "     combined_4=[" + combined_4 + "]\n\n";
 
 
        if (combined_1 == "0")  // 0
            {
                tempString = "\n      [" + _char ;
                tempString = tempString  + "]=<Combined_1 PLANE 0>" + stringByte;
                // I get the following error if I do not split up the previous two lines
                // error: invalid operands of types 'const char*' and 'const char [23]' to binary 'operator+'|
                // Please answer in separate question here https://bytes.com/topic/c/answers/974655-invalid-operands-const-char-binary-operator Thank you.
 
 
                binaryString = binaryString += tempString;
                splitIntoBytes = splitIntoBytes + "\n Unicode Plane 0 [" + _char + "]=" + stringByte;
            }
        else
            {
 
                if (combined_3 == "110")
                    {                                tempString = "\n      [" + _char ;
                        tempString = tempString  + "]=<Combined_3 PLANE 0>" + stringByte;
 
                        binaryString = binaryString += tempString;
                        splitIntoBytes = splitIntoBytes + "\n Unicode Plane 0 [" + _char + "]=" + stringByte;
                    }
                else
                    if (combined_4 == "1110")   // Verifying that there is a zero "0" past the 111.
                                                // Therefore it is still in Plane 0.
                        {
                            tempString = "\n      [" + _char ;
                            tempString = tempString  + "]=<Combined_4 PLANE 0>" + stringByte;
 
                            binaryString = binaryString += tempString;
                            splitIntoBytes = splitIntoBytes + "\n Unicode Plane 0 [" + _char + "]=" + stringByte;
                        }
                    else if (combined_4 == "1111")  // Verifying that there is a one "1" past the 111.
                                                    // Therefore it is now in Plane 1.
                        {
                            tempString = "\n      [" + _char ;
                            tempString = tempString  + "]=<Combined_4 PLANE 0>" + stringByte;
 
                            binaryString = binaryString += tempString;
                            splitIntoBytes = splitIntoBytes + "\n Unicode Plane 1 [" + _char + "]=" + stringByte;
                        }
                    else
                        {
                            tempString = "\n      [" + _char ;
                            tempString = tempString  + "]=<EXTENDED>" + stringByte;
 
                            binaryString = binaryString += tempString;
                            splitIntoBytes = splitIntoBytes + " " + _char + "=" + stringByte;
                        }
 
            }
        binaryString = binaryString + "\n";
 
    }
 
    // For Line by Line testing:
//    binaryString = splitIntoBytes + "\n\n-Line by line testing is next:\n\n" + binaryString + "\n\n";
 
    // For quick results:
    binaryString = splitIntoBytes + "\n\n";
 
    return binaryString;
}
 
int main()
{
//    string testText = "123办456   办"; // From Plane 0 Unicode
    string testText = "a𐑒𐑦𐑙𐑛𐑳𐑥b" ; // From Plane 1 Unicode
 
    // I went to http://www.i18nguy.com/unicode-example-plane1.html and copied some of their characters
    // which are Plane 1 Unicode and then pasted them into the code here
    // and got a bunch of boxes
    // string testText = "a[][][][][][]b";
    //
    // And, when I compiled it sometimes I got warnings that the line did not do anything,
    // but I ran it and IT WORKED !
    //
    // Maybe, even though my laptop might not have the correct fonts to show the higher plane in Unicode characters,
    // maybe C++11 via CODE::BLOCKS 17.12 accepted it and compiled it and ran it correctly.
    //
    // Wow. Thank you bytes.com
 
 
    cout << "Convert [" << testText << "] to Binary\n\n";
    cout << TextToBinaryString(testText) << "\n";
 
    return 0;
}