Python & Unicode decimal interpretation

[Scott David Daniels]

In reading over the source for CPython's PyUnicode_EncodeDecimal,
I see a dance to handle characters which are neither dec-equiv nor
in Latin-1. Does anyone know about the intent of such a conversion?

As far as I can tell, error handling is one of:
strict, replace, ignore, xmlcharrefreplace, or something_else
What I don't understand is whether, in the ignore or something_else
cases, there is any chance that digits will show up anywhere that
they would not if these characters were treated as a character like '?'?

Can someone either give me definitive "why not" or (preferably) give
me a test case that shows where that interpretation does not hold.

--Scott David Daniels
sc***********@acm.org

[Martin v. Löwis]

Scott David Daniels wrote:

In reading over the source for CPython's PyUnicode_EncodeDecimal,
I see a dance to handle characters which are neither dec-equiv nor
in Latin-1. Does anyone know about the intent of such a conversion?
To support this:

int(u"\N{DEVANAGARI DIGIT SEVEN}")

7

As far as I can tell, error handling is one of:
strict, replace, ignore, xmlcharrefreplace, or something_else
What I don't understand is whether, in the ignore or something_else
cases, there is any chance that digits will show up anywhere that
they would not if these characters were treated as a character like '?'?

Can someone either give me definitive "why not" or (preferably) give
me a test case that shows where that interpretation does not hold.

In the "ignore" case, no output is produced at all, for the unencodable
character; this is the same way that '?' would be treated (it is
also unencodable).

In the something_else case, a user-defined exception handler could
treat the error in any way it liked, e.g. encoding all letters
u'A' to digit '0'. This might be different from the way this error
handler would treat '?'.

Regards,
Martin

[Scott David Daniels]

Martin v. Löwis wrote:

Scott David Daniels wrote:

In reading over the source for CPython's PyUnicode_EncodeDecimal,
I see a dance to handle characters which are neither dec-equiv nor
in Latin-1. Does anyone know about the intent of such a conversion?
To support this:

>>> int(u"\N{DEVANAGARI DIGIT SEVEN}") 7

OK, That much I have handled. I am fiddling with direct-to-number
conversions and wondering about cases like

int(u"\N{DEVANAGARI DIGIT SEVEN}" + XXX

+ u"\N{DEVANAGARI DIGIT SEVEN}")

Where XXX does not pass the digit test, but must either:
(A) be dropped, giving a result of 77
or (B) get translated (e.g. to u'234') giving 72347
or (C) get translated (to u'2' + YYY + u'4') where YYY will
require further handling ...

I don't really understand how the "ignore" or "something_else"
cases get caused by python source [where they come from]. Are they
only there for C-program access?
In the "ignore" case, no output is produced at all, for the unencodable
character; this is the same way that '?' would be treated (it is
also unencodable). If I understand you correctly -- I can consider the digit stream to stop
as soon as I hit a non-digit (except for handling bases 11-36).
In the something_else case, a user-defined exception handler could
treat the error in any way it liked, e.g. encoding all letters
u'A' to digit '0'. This might be different from the way this error
handler would treat '?'.

--Scott David Daniels
sc***********@acm.org

[Martin v. Löwis]

Scott David Daniels wrote:

>>> int(u"\N{DEVANAGARI DIGIT SEVEN}") 7

OK, That much I have handled. I am fiddling with direct-to-number
conversions and wondering about cases like

>>> int(u"\N{DEVANAGARI DIGIT SEVEN}" + XXX + u"\N{DEVANAGARI DIGIT SEVEN}")

int() passes NULL as error mode, equalling strict. So if you get an
unencodable character, you get the UnicodeError.
I don't really understand how the "ignore" or "something_else"
cases get caused by python source [where they come from]. Are they
only there for C-program access?

Neither, nor. This code is dead.
In the "ignore" case, no output is produced at all, for the unencodable
character; this is the same way that '?' would be treated (it is
also unencodable).

If I understand you correctly -- I can consider the digit stream to stop
as soon as I hit a non-digit (except for handling bases 11-36).

No. In "ignore" mode, a codec doesn't stop at the unencodable character.
Instead, it skips it, continuing with the next character.

I mistakenly said that this would happen to '?' (question mark) also;
this is incorrect: PyUnicode_EncodeDecimal copies all Latin-1 characters
to the output, latin-1-encoded. So '?' would appear in the output,
even in "ignore" mode.

Handling of bases is not done in the function at all. Instead, the
callers of PyUnicode_EncodeDecimal will deal with number formats
(base, prefix, exponent syntax, etc.) They will assume ASCII
bytes.

Regards,
Martin