Where is the ucs-32 codec?

[beni.cherniavsky]

Python seems to be missing a UCS-32 codec, even in wide builds (not
that it the build should matter).
Is there some deep reason or should I just contribute a patch?

If it's just a bug, should I call the codec 'ucs-32' or 'utf-32'? Or
both (aliased)?
There should be '-le' and '-be' variats, I suppose. Should there be a
variant without explicit endianity, using a BOM to decide (like
'utf-16')?
And it should combine surrogates into valid characters (on all builds),
like the 'utf-8' codec does, right?

--
Beni Cherniavsky <cb**@users.sf.net>, who can only read email on
weekends.

[Erik Max Francis]

be**************@gmail.com wrote:

Python seems to be missing a UCS-32 codec, even in wide builds (not
that it the build should matter).
Is there some deep reason or should I just contribute a patch?

If it's just a bug, should I call the codec 'ucs-32' or 'utf-32'? Or
both (aliased)?
There should be '-le' and '-be' variats, I suppose. Should there be a
variant without explicit endianity, using a BOM to decide (like
'utf-16')?
And it should combine surrogates into valid characters (on all builds),
like the 'utf-8' codec does, right?

Note that UTF-32 is UCS-4. UCS-32 ("Universial Character Set in 32
octets") wouldn't make much sense.

Not that Python has a UCS-4 encoding available either. I'm really not
sure why.

--
Erik Max Francis && ma*@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
Could it be / That we need loving to survive
-- Neneh Cherry

[Méta-MCI]

Hi!

Look at: http://cjkpython.berlios.de (iconvcodec)

(Serge Orlov has built a version for Python 2.4 "special for me"; thanks to
him).
@-salutations
--
Michel Claveau

[Martin v. Löwis]

be**************@gmail.com wrote:

Python seems to be missing a UCS-32 codec, even in wide builds (not
that it the build should matter).
Is there some deep reason or should I just contribute a patch?
The only reason is that nobody has needed one so far, and because
it is quite some work to do if done correctly. Why do you need it?
There should be '-le' and '-be' variats, I suppose. Should there be a
variant without explicit endianity, using a BOM to decide (like
'utf-16')?
Right.
And it should combine surrogates into valid characters (on all builds),
like the 'utf-8' codec does, right?

Right.

Also, it should support the incremental interface (as any multi-byte
codec should).

If you want it complete, it should also support line-oriented input.
Notice that .readline/.readlines is particularly difficult to implement,
as you can't rely on the underlying stream's .readline implementation
to provide meaningful results.

While we are discussing problems: there also is the issue whether
..readline/.readlines should take the additional Unicode linebreak
characters into account (e.g. U+2028, U+2029), and if so, whether
that should be restricted to "universal newlines" mode.

Regards,
Martin

[Erik Max Francis]

Martin v. Löwis wrote:

The only reason is that nobody has needed one so far, and because
it is quite some work to do if done correctly. Why do you need it?

Why would it be "quite some work"? Converting from UTF-16 to UTF-32 is
pretty straightforward, and UTF-16 is already supported.

--
Erik Max Francis && ma*@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
Democritus may have come from Abdera, but he was no dummy.
-- Carl Sagan

[Martin v. Löwis]

Erik Max Francis wrote:

The only reason is that nobody has needed one so far, and because
it is quite some work to do if done correctly. Why do you need it?

Why would it be "quite some work"? Converting from UTF-16 to UTF-32 is
pretty straightforward, and UTF-16 is already supported.

I would like to see it correct, unlike the current UTF-16 codec. Perhaps
whoever contributes an UTF-32 codec could also deal with the defects of
the UTF-16 codec.

Regards,
Martin

[cben]

Méta-MCI wrote:

Hi!

Look at: http://cjkpython.berlios.de (iconvcodec)

(Serge Orlov has built a version for Python 2.4 "special for me"; thanks to
him).

Thanks for the pointer.
iconvcodec should do the job, but I still want a native implementation
to be included with any python.

[cben]

Martin v. Löwis wrote:

Erik Max Francis wrote:

The only reason is that nobody has needed one so far, and because
it is quite some work to do if done correctly. Why do you need it?

Somebody asked me about generating UTF-32 (he didn't have choice of the
output format).
I was about to propose the obvious ``u.encode('utf-32')`` but
discovered it's missing.
Someone proposed 'unicode-internal' but it depends on the build and is
an ugly answer.
Next time, I want Guido's Time Machine to just work, so I have to fix
this ;-).
Why would it be "quite some work"? Converting from UTF-16 to UTF-32 is
pretty straightforward, and UTF-16 is already supported.

I would like to see it correct, unlike the current UTF-16 codec. Perhaps
whoever contributes an UTF-32 codec could also deal with the defects of
the UTF-16 codec.

Now this is interesting, as I hoped to base my code on UTF-16 (and
perhaps UTF-8 for combining surrogates)... Can you elaborate?

I could attempt to fix UTF-16 as well but I don't have the expertise to
choose the right behaviour,
so you'll have to specify precisely what it should do (that it doesn't
do now).

[Fredrik Lundh]

cb**@users.sf.net wrote:

Somebody asked me about generating UTF-32 (he didn't have choice of the
output format). I was about to propose the obvious ``u.encode('utf-32')``
but discovered it's missing.

hint 1:

u = u"Hello"
a = array.array("I", map(ord, u))
a.tostring() 'H\x00\x00\x00e\x00\x00\x00l\x00\x00\x00l\x00\x00\ x00o\x00\x00\x00' a.byteswap()
a.tostring() '\x00\x00\x00H\x00\x00\x00e\x00\x00\x00l\x00\x00\x 00l\x00\x00\x00o'

hint 2:
import sys
sys.byteorder

'little'

</F>

[Martin v. Löwis]

cb**@users.sf.net wrote:

I would like to see it correct, unlike the current UTF-16 codec. Perhaps
whoever contributes an UTF-32 codec could also deal with the defects of
the UTF-16 codec.

Now this is interesting, as I hoped to base my code on UTF-16 (and
perhaps UTF-8 for combining surrogates)... Can you elaborate?

The codec doesn't do line-oriented input correctly (i.e. readline);
it raises NotImplementedError.

Regards,
Martin