Problem with curses and UTF-8

[Ian Ward]

When I run the following code in a terminal with the encoding set to
UTF-8 I get garbage on the first line, but the correct output on the second.
import curses
s = curses.initscr()
s.addstr('\xc3\x85 U+00C5 LATIN CAPITAL LETTER A WITH RING ABOVE\n')
s.addstr('\xc3\xa5 U+00F5 LATIN SMALL LETTER O WITH TILDE')
s.refresh()
s.getstr()
curses.endwin()
I tested with gnome-terminal, Python 2.4 and Ubuntu breezy. The output
is correct when I run the following code:
print '\xc3\x85 U+00C5 LATIN CAPITAL LETTER A WITH RING ABOVE'
print '\xc3\xa5 U+00F5 LATIN SMALL LETTER O WITH TILDE'
Any Ideas?

Ian Ward

[Martin v. Löwis]

Ian Ward wrote:

Any Ideas?

I think there is one or more ncurses bugs somewhere.

The ncurses documentation suggests that you should link with
ncurses_w instead of linking with ncurses - you might try
that as well. If it helps, please do report back.

Ultimately, somebody will need to debug ncurses to find out
what precisely happens, and why.

Regards,
Martin

[Ian Ward]

Martin v. Löwis wrote:

Ian Ward wrote:

Any Ideas?

I think there is one or more ncurses bugs somewhere.

The ncurses documentation suggests that you should link with
ncurses_w instead of linking with ncurses - you might try
that as well. If it helps, please do report back.

Ultimately, somebody will need to debug ncurses to find out
what precisely happens, and why.

Thank you for your response. I see there are other people that have run
into the same problem.

I've had to work around many curses issues while developing Urwid (a
console UI library). Even if the bugs are fixed I'm going to have to
bypass the curses module to support UTF-8 in a reliable way for all users.

I think there are enough escape sequences common to all modern terminals
so that I can build a generic curses-replacement for my library.
However, if someone is already working on something similar I don't want
to reinvent the wheel.

Ian Ward

[Thomas Dickey]

"Martin v. Löwis" <ma****@v.loewis.de> wrote:

Ian Ward wrote:

Any Ideas?

I think there is one or more ncurses bugs somewhere.
indeed. It might be nice to report them rather than jawing about it.
The ncurses documentation suggests that you should link with
ncurses_w instead of linking with ncurses - you might try
that as well. If it helps, please do report back.
ncursesw
Ultimately, somebody will need to debug ncurses to find out
what precisely happens, and why.

no need for debugging - it's a well-known problem. UTF-8 uses more than
one byte per cell, normal curses uses one byte per cell. To handle UTF-8,
you need ncursesw.

--
Thomas E. Dickey
http://invisible-island.net
ftp://invisible-island.net

[Thomas Dickey]

Ian Ward <ia*@excess.org> wrote:

I've had to work around many curses issues while developing Urwid (a

hmm - I've read Urwid, and most of the comments I've read in that regard
reflect problems in Urwid. Perhaps it's time for you to do a little analysis.

(looking forward to bug reports, rather than line noise)

--
Thomas E. Dickey
http://invisible-island.net
ftp://invisible-island.net

[Grant Edwards]

On 2006-02-08, Ian Ward <ia*@excess.org> wrote:

I think there are enough escape sequences common to all modern terminals
so that I can build a generic curses-replacement for my library.

Why not use termcap/terminfo?

--
Grant Edwards grante Yow! Where does it go when
at you flush?
visi.com

[Ian Ward]

Thomas Dickey wrote:

hmm - I've read Urwid, and most of the comments I've read in that regard
reflect problems in Urwid. Perhaps it's time for you to do a little analysis.

(looking forward to bug reports, rather than line noise)

A fair request. My appologies for the inflammatory subject :-)

When trying to check for user input without waiting I use code like:
window_object.nodelay(1)
curses.cbreak()
input = window_object.getch()

Occasionally (hard to reproduce reliably) the cbreak() call will raise
an exception, but if I call it a second time before calling getch the
code will work properly. This problem might be related to a signal
interrupting the function call, I'm not sure.

Also, screen resizing only seems to be reported once by getch() even if
the user continues to resize the window. I have worked around this by
calling curses.doupdate() between calls to getch(). Maybe this is by design?

Finally, the curses escape sequence detection could be broadened. The
top part of the curses_display module in Urwid defines many escape
sequences I've run into that curses doesn't detect.

Ian Ward

[Ian Ward]

Grant Edwards wrote:

Why not use termcap/terminfo?

That's a good idea, but I'd have to wrap the c library myself, wouldn't
I? Also, what happens when a user has an incorrect TERM setting (I've
run into this before)

I don't want to reimpliment all the nice speed optimizations that the
curses library has, I just want something simple that should work for as
many people as possible.

Ian Ward

[Grant Edwards]

On 2006-02-08, Ian Ward <ia*@excess.org> wrote:

Grant Edwards wrote:

Why not use termcap/terminfo?
That's a good idea, but I'd have to wrap the c library myself,
wouldn't I?

Probably. I don't remember seeing a python module for them.
Also, what happens when a user has an incorrect TERM setting
(I've run into this before)
Then things (besides your program) won't work.
I don't want to reimpliment all the nice speed optimizations
that the curses library has, I just want something simple that
should work for as many people as possible.

Depending on what you're tring to do, slang might be an option,
but I don't think there's a Python binding. There is a
(largely unsupported) Python binding for the "newt" widget set
that runs on top of slang. The old text-mode "red dialog
windows on a blue background" RedHat installer and admin apps
were written in Python using the newt widget library. The
"newt" Python module is called "snack".

--
Grant Edwards grante Yow! DIDI... is that a
at MARTIAN name, or, are we
visi.com in ISRAEL?

[Ian Ward]

Thomas Dickey wrote:

"Martin v. Löwis" <ma****@v.loewis.de> wrote:

The ncurses documentation suggests that you should link with
ncurses_w instead of linking with ncurses - you might try
that as well. If it helps, please do report back.

ncursesw

I'll test it if someone would dumb down "link with ncursesw instead of
ncurses" a little for me.

I tried:
../configure --with-libs="ncursesw5"

and it failed saying:
checking size of wchar_t... configure: error: cannot compute sizeof
(wchar_t), 77

Ian Ward

[Martin v. Löwis]

Thomas Dickey wrote:

no need for debugging - it's a well-known problem. UTF-8 uses more than
one byte per cell, normal curses uses one byte per cell. To handle UTF-8,
you need ncursesw.

I tried that, but it didn't improve anything.

Regards,
Martin

[Ian Ward]

Grant Edwards wrote:

Depending on what you're tring to do, slang might be an option,

I've looked at newt and snack, but all I really need is:
- a way to position the cursor at (0,0)
- a way to hide and show the cursor
- a way to detect when the terminal is resized
- a way to query the terminal size

Ian Ward

[Martin v. Löwis]

> I'll test it if someone would dumb down "link with ncursesw instead of

ncurses" a little for me.

I tried:
./configure --with-libs="ncursesw5"

and it failed saying:
checking size of wchar_t... configure: error: cannot compute sizeof
(wchar_t), 77

If that was Python's configure: don't do that. Instead, hack setup.py
to make it change the compiler/linker settings, or even edit the
compiler/linker line manually at first.

Regards.
Martin

[Martin v. Löwis]

> I'll test it if someone would dumb down "link with ncursesw instead of

ncurses" a little for me.

I tried:
./configure --with-libs="ncursesw5"

and it failed saying:
checking size of wchar_t... configure: error: cannot compute sizeof
(wchar_t), 77

If that was Python's configure: don't do that. Instead, hack setup.py
to make it change the compiler/linker settings, or even edit the
compiler/linker line manually at first.

Regards.
Martin

[Ian Ward]

Martin v. Löwis wrote:

If that was Python's configure: don't do that. Instead, hack setup.py
to make it change the compiler/linker settings, or even edit the
compiler/linker line manually at first.

Ok, that compiled.

Now when I run the same test:

import curses
s = curses.initscr()
s.addstr('\xc3\x85 U+00C5 LATIN CAPITAL LETTER A WITH RING ABOVE\n')
s.addstr('\xc3\xa5 U+00F5 LATIN SMALL LETTER O WITH TILDE')
s.refresh()
s.getstr()
curses.endwin()
This is what I see:

+00C5 LATIN CAPITAL LETTER A WITH RING ABOVE
+00F5 LATIN SMALL LETTER O WITH TILDE
so, the UTF-8 characters didn't appear and the " U" at the beginning
became just " ".

Ian Ward

[Thomas Dickey]

"Martin v. Löwis" <ma****@v.loewis.de> wrote:

I'll test it if someone would dumb down "link with ncursesw instead of
ncurses" a little for me.

I tried:
./configure --with-libs="ncursesw5"

and it failed saying:
checking size of wchar_t... configure: error: cannot compute sizeof
(wchar_t), 77

If that was Python's configure: don't do that. Instead, hack setup.py
yes - python's configure script needs a lot of work
(alternatively, it is not the sort of script I would write).
to make it change the compiler/linker settings, or even edit the
compiler/linker line manually at first.

that works

--
Thomas E. Dickey
http://invisible-island.net
ftp://invisible-island.net

[Thomas Dickey]

Ian Ward <ia*@excess.org> wrote:

Martin v. Löwis wrote:

If that was Python's configure: don't do that. Instead, hack setup.py
to make it change the compiler/linker settings, or even edit the
compiler/linker line manually at first.

Ok, that compiled.
same here - though it was not immediately not clear which copy of ncurses it's
using (not the shared libraries since I installed those with tracing - a
little odd for it to use the static library, but that's what the access time
tells me).

To check on that (since I wanted to read the ncurses trace),
I ran strace and ltrace to look for clues.
Now when I run the same test: import curses
s = curses.initscr()
s.addstr('\xc3\x85 U+00C5 LATIN CAPITAL LETTER A WITH RING ABOVE\n')
s.addstr('\xc3\xa5 U+00F5 LATIN SMALL LETTER O WITH TILDE')
s.refresh()
s.getstr()
curses.endwin()
Testing this, and looking to see what's going on, I notice that python
is doing a

setlocale(LC_ALL, "C");

before the addstr is actually called. (ncurses never sets the locale;
it calls setlocale in one place to ask what it is).

That makes ncurses think it's not really doing UTF-8, of course. What I
see on the screen is the U+00C5 comes out with a box and a "~E" (the
latter being ncurses' representation in POSIX for \0x85).
This is what I see: +00C5 LATIN CAPITAL LETTER A WITH RING ABOVE
+00F5 LATIN SMALL LETTER O WITH TILDE
so, the UTF-8 characters didn't appear and the " U" at the beginning
became just " ".

well - running in uxterm I see the second line properly. But some more
tinkering is needed to make python work properly.

--
Thomas E. Dickey
http://invisible-island.net
ftp://invisible-island.net

[Thomas Dickey]

Grant Edwards <gr****@visi.com> wrote:

Depending on what you're tring to do, slang might be an option,

perhaps not - he's trying to use UTF-8. I haven't seen any plausible
comment that indicates John Davis is interested in updating newt to
work with slang2 (though of course he's welcome to show the code ;-)

--
Thomas E. Dickey
http://invisible-island.net
ftp://invisible-island.net

[Thomas Dickey]

Ian Ward <ia*@excess.org> wrote:

Grant Edwards wrote:

Depending on what you're tring to do, slang might be an option,

I've looked at newt and snack, but all I really need is:
- a way to position the cursor at (0,0)
- a way to hide and show the cursor
- a way to detect when the terminal is resized
- a way to query the terminal size

....and send UTF-8 text, keeping track of where you really are on the screen.

--
Thomas E. Dickey
http://invisible-island.net
ftp://invisible-island.net

[Thomas Dickey]

Ian Ward <ia*@excess.org> wrote:

Thomas Dickey wrote:

hmm - I've read Urwid, and most of the comments I've read in that regard
reflect problems in Urwid. Perhaps it's time for you to do a little analysis.

(looking forward to bug reports, rather than line noise)

A fair request. My appologies for the inflammatory subject :-) When trying to check for user input without waiting I use code like:
window_object.nodelay(1)
curses.cbreak()
input = window_object.getch() Occasionally (hard to reproduce reliably) the cbreak() call will raise
an exception, but if I call it a second time before calling getch the
code will work properly. This problem might be related to a signal
interrupting the function call, I'm not sure.
perhaps a more complete test-case would let me test it and see.
Also, screen resizing only seems to be reported once by getch() even if
the user continues to resize the window. I have worked around this by
calling curses.doupdate() between calls to getch(). Maybe this is by design?
Or perhaps it's some interaction with python - I don't know.
The applications that I use with resizing (and ncurses' test
programs) work smoothly enough.
Finally, the curses escape sequence detection could be broadened. The
top part of the curses_display module in Urwid defines many escape
sequences I've run into that curses doesn't detect.

That's data (terminfo). ncurses is data-driven, doesn't "detect"
features of the terminal (though it does of course use environment
variables for locale, etc.).

xterm's terminfo lists a lot of function keys, for instance.

The limit for predefined function-key names for terminfo is 60,
but ncurses can accept extended terminfo descriptions (but I like to
limit the length and style of names so it's possible to access them
from termcap). One could define names like shift_f1, but then termcap
applications couldn't see them. (The last I knew, slang doesn't either,
but that's a different thread).

That's been true for about 6 years.

Current xterm's terminfo includes these names which apply to your
comment: The ones on the end are extended names that ncurses' tic
deduces from the terminfo file when it compiles it:

comparing xterm-new to xterm-xf86-v44.
comparing booleans.
comparing numbers.
comparing strings.
kf49: '\EO3P', NULL.
kf50: '\EO3Q', NULL.
kf51: '\EO3R', NULL.
kf52: '\EO3S', NULL.
kf53: '\E[15;3~', NULL.
kf54: '\E[17;3~', NULL.
kf55: '\E[18;3~', NULL.
kf56: '\E[19;3~', NULL.
kf57: '\E[20;3~', NULL.
kf58: '\E[21;3~', NULL.
kf59: '\E[23;3~', NULL.
kf60: '\E[24;3~', NULL.
kf61: '\EO4P', NULL.
kf62: '\EO4Q', NULL.
kf63: '\EO4R', NULL.
kind: '\E[1;2B', NULL.
kri: '\E[1;2A', NULL.
kDN: '\E[1;2B', NULL.
kDN5: '\E[1;5B', NULL.
kDN6: '\E[1;6B', NULL.
kLFT5: '\E[1;5D', NULL.
kLFT6: '\E[1;6D', NULL.
kRIT5: '\E[1;5C', NULL.
kRIT6: '\E[1;6C', NULL.
kUP: '\E[1;2A', NULL.
kUP5: '\E[1;5A', NULL.
kUP6: '\E[1;6A', NULL.

--
Thomas E. Dickey
http://invisible-island.net
ftp://invisible-island.net

[Donn Cave]

In article <11*************@corp.supernews.com>,
Thomas Dickey <di****@saltmine.radix.net> wrote:

Ian Ward <ia*@excess.org> wrote:

....

Also, screen resizing only seems to be reported once by getch() even if
the user continues to resize the window. I have worked around this by
calling curses.doupdate() between calls to getch(). Maybe this is by
design?

Or perhaps it's some interaction with python - I don't know.
The applications that I use with resizing (and ncurses' test
programs) work smoothly enough.

I have no idea about the present application, but just as a
general observation, when Python traps a signal, it saves
the signal number, and makes a note to check for trapped signals
as the next Python operation. That check iterates through the
list of possible signals to see if any have been caught, and
execute their respective handlers if any.

Since an external function call is an operation, no signal
handler will execute until it returns. At that time, the
signal handler will execute once, at most.

Finally, the curses escape sequence detection could be broadened. The
top part of the curses_display module in Urwid defines many escape
sequences I've run into that curses doesn't detect.

That's data (terminfo). ncurses is data-driven, doesn't "detect"
features of the terminal (though it does of course use environment
variables for locale, etc.).

xterm's terminfo lists a lot of function keys, for instance.

This is just my opinion, but any application that depends
on function keys in terminfo is broken, automatically.
Optional support for function keys is a nice touch, but the
data isn't good enough out there to depend on it.

Donn Cave, do**@u.washington.edu

[Damjan]

I just recompiled my python to link to ncursesw, and tried your example
with a little modification:

import curses, locale
locale.setlocale(locale.LC_ALL, '')
s = curses.initscr()
s.addstr(u'\u00c5 U+00C5 LATIN CAPITAL LETTER A WITH RING
ABOVE\n'.encode('utf-8') )
s.addstr(u'\u00f5 U+00F5 LATIN SMALL LETTER O WITH
TILDE\n'.encode('utf-8'))
s.refresh()
s.getstr()
curses.endwin()

And it works ok for me, Slackware-10.2, python-2.4.2, ncurses-5.4 all
in KDE's konsole.
My locale is mk_MK.UTF-8.

Now it would be great if python's curses module worked with unicode
strings directly.

[Ross Ridge]

Thomas Dickey wrote:

...and send UTF-8 text, keeping track of where you really are on the screen.

You make that sound so easy.

Ross Ridge

[Ian Ward]

Ross Ridge wrote:

Thomas Dickey wrote:

...and send UTF-8 text, keeping track of where you really are on the screen.

You make that sound so easy.

I'll have to deal with that anyway, since I'm doing all my own wrapping,
justification and clipping of text. (don't talk to me about RtoL text,
I'm getting to it)

I'm going to look at the Mined text editor for some terminal behavior
detection code. Mined is able to produce good UTF-8 output on a variety
of terminals, and it links agains ncurses, not ncursesw... Interesting.

Ian Ward

[Thomas Dickey]

Ian Ward <ia*@excess.org> wrote:

I'm going to look at the Mined text editor for some terminal behavior
mined_2000 (there's more than one program named mined, and the other
doesn't do UTF-8).
detection code. Mined is able to produce good UTF-8 output on a variety
of terminals, and it links agains ncurses, not ncursesw... Interesting.

It's probably using termcap (and the wide-character functions declared
in wchar.h).

--
Thomas E. Dickey
http://invisible-island.net
ftp://invisible-island.net

[Ian Ward]

Damjan wrote:

import curses, locale
locale.setlocale(locale.LC_ALL, '')
s = curses.initscr()

Hey, that works for me. Combined characters and wide characters are
working too.

Now the real problem.. how do I convince the python higher-ups to link
against cursesw by default?

At the very least all distros that use UTF-8 as their default encoding
should switch to cursesw.

Ian Ward

[Ross Ridge]

Ian Ward wrote:

I'll have to deal with that anyway, since I'm doing all my own wrapping,
justification and clipping of text.

In general it's impossible to know how many display positions some
random Unicode character might use. For example, Chinese characters
normally take two display positions, but the terminal your using might
not support them and display a single width replacement character.
Hopefully, you're limitted in the character set you actually need to
support and the terminals that your applicaiton will be using.

[Martin v. Löwis]

Ian Ward wrote:

Hey, that works for me. Combined characters and wide characters are
working too.

Now the real problem.. how do I convince the python higher-ups to link
against cursesw by default?

That's very easy. Contribute a working patch. That patch should support
all possible situations (e.g. curses is ncurses, and ncursesw is
available, curses is ncurses, and ncursesw is not available, curses
is not ncurses), and submit that patch to sf.net/projects/python.

Regards,
Martin

[Martin v. Löwis]

Ian Ward wrote:

Hey, that works for me. Combined characters and wide characters are
working too.

Now the real problem.. how do I convince the python higher-ups to link
against cursesw by default?

That's very easy. Contribute a working patch. That patch should support
all possible situations (e.g. curses is ncurses, and ncursesw is
available, curses is ncurses, and ncursesw is not available, curses
is not ncurses), and submit that patch to sf.net/projects/python.

Regards,
Martin

[Ian Ward]

Martin v. Löwis wrote:

That's very easy. Contribute a working patch. That patch should support
all possible situations (e.g. curses is ncurses, and ncursesw is
available, curses is ncurses, and ncursesw is not available, curses
is not ncurses), and submit that patch to sf.net/projects/python.

Done.

http://sourceforge.net/tracker/index...70&atid=305470

Ian Ward

[Ian Ward]

Ross Ridge wrote:

In general it's impossible to know how many display positions some
random Unicode character might use. For example, Chinese characters
normally take two display positions, but the terminal your using might
not support them and display a single width replacement character.
Hopefully, you're limitted in the character set you actually need to
support and the terminals that your applicaiton will be using.

I'm not so lucky -- I'm writing a console UI library (Urwid) that anyone
could use, and I'm trying to support as many encodings and terminals as
possible.

I hope that the different terminal behaviors can be enumerated so that
the console interface will degrade gracefully with less capable
terminals. The mined_2000 unicode text editor is a program that does
this by detecting the terminal's behavior on startup. I'll probably take
a similar approach.

Ian Ward