473,394 Members | 1,149 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,394 software developers and data experts.

Revised PEP 349: Allow str() to return unicode strings

[Please mail followups to py********@python.org.]

The PEP has been rewritten based on a suggestion by Guido to change
str() rather than adding a new built-in function. Based on my
testing, I believe the idea is feasible. It would be helpful if
people could test the patched Python with their own applications and
report any incompatibilities.
PEP: 349
Title: Allow str() to return unicode strings
Version: $Revision: 1.3 $
Last-Modified: $Date: 2005/08/22 21:12:08 $
Author: Neil Schemenauer <na*@arctrix.com>
Status: Draft
Type: Standards Track
Content-Type: text/plain
Created: 02-Aug-2005
Post-History: 06-Aug-2005
Python-Version: 2.5
Abstract

This PEP proposes to change the str() built-in function so that it
can return unicode strings. This change would make it easier to
write code that works with either string type and would also make
some existing code handle unicode strings. The C function
PyObject_Str() would remain unchanged and the function
PyString_New() would be added instead.
Rationale

Python has had a Unicode string type for some time now but use of
it is not yet widespread. There is a large amount of Python code
that assumes that string data is represented as str instances.
The long term plan for Python is to phase out the str type and use
unicode for all string data. Clearly, a smooth migration path
must be provided.

We need to upgrade existing libraries, written for str instances,
to be made capable of operating in an all-unicode string world.
We can't change to an all-unicode world until all essential
libraries are made capable for it. Upgrading the libraries in one
shot does not seem feasible. A more realistic strategy is to
individually make the libraries capable of operating on unicode
strings while preserving their current all-str environment
behaviour.

First, we need to be able to write code that can accept unicode
instances without attempting to coerce them to str instances. Let
us label such code as Unicode-safe. Unicode-safe libraries can be
used in an all-unicode world.

Second, we need to be able to write code that, when provided only
str instances, will not create unicode results. Let us label such
code as str-stable. Libraries that are str-stable can be used by
libraries and applications that are not yet Unicode-safe.

Sometimes it is simple to write code that is both str-stable and
Unicode-safe. For example, the following function just works:

def appendx(s):
return s + 'x'

That's not too surprising since the unicode type is designed to
make the task easier. The principle is that when str and unicode
instances meet, the result is a unicode instance. One notable
difficulty arises when code requires a string representation of an
object; an operation traditionally accomplished by using the str()
built-in function.

Using the current str() function makes the code not Unicode-safe.
Replacing a str() call with a unicode() call makes the code not
str-stable. Changing str() so that it could return unicode
instances would solve this problem. As a further benefit, some code
that is currently not Unicode-safe because it uses str() would
become Unicode-safe.
Specification

A Python implementation of the str() built-in follows:

def str(s):
"""Return a nice string representation of the object. The
return value is a str or unicode instance.
"""
if type(s) is str or type(s) is unicode:
return s
r = s.__str__()
if not isinstance(r, (str, unicode)):
raise TypeError('__str__ returned non-string')
return r

The following function would be added to the C API and would be the
equivalent to the str() built-in (ideally it be called PyObject_Str,
but changing that function could cause a massive number of
compatibility problems):

PyObject *PyString_New(PyObject *);

A reference implementation is available on Sourceforge [1] as a
patch.
Backwards Compatibility

Some code may require that str() returns a str instance. In the
standard library, only one such case has been found so far. The
function email.header_decode() requires a str instance and the
email.Header.decode_header() function tries to ensure this by
calling str() on its argument. The code was fixed by changing
the line "header = str(header)" to:

if isinstance(header, unicode):
header = header.encode('ascii')

Whether this is truly a bug is questionable since decode_header()
really operates on byte strings, not character strings. Code that
passes it a unicode instance could itself be considered buggy.
Alternative Solutions

A new built-in function could be added instead of changing str().
Doing so would introduce virtually no backwards compatibility
problems. However, since the compatibility problems are expected to
rare, changing str() seems preferable to adding a new built-in.

The basestring type could be changed to have the proposed behaviour,
rather than changing str(). However, that would be confusing
behaviour for an abstract base type.
References

[1] http://www.python.org/sf/1266570
Copyright

This document has been placed in the public domain.

Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
End:

Aug 22 '05 #1
2 2603
neil,

i just intended to worry that returning a unicode object from ``str()``
would break assumptions about the way that 'type definers' like
``str()``, ``int()``, ``float()`` and so on work, but i quickly
realized that e.g. ``int()`` does return a long where appropriate!
since the principle works there one may surmise it will also work for
``str()`` in the long run.

one point i don't seem to understand right now is why it says in the
function definition::

if type(s) is str or type(s) is unicode:
...

instead of using ``isinstance()``.

Testing for ``type()`` means that instances of derived classes (that
may or may not change nothing or almost nothing to the underlying
class) when passed to a function that uses ``str()`` will behave in a
different way!

isn't it more realistic and commonplace to assume that derivatives of a
class do fulfill the requirements of the underlying class? -- which may
turn out to be wrong! but still...

the code as it stands means i have to remember that *in this special
case only* (when deriving from ``unicode``), i have to add a
``__str__()`` method myself that simply returns ``self``.

then of course, one could change ``unicode.__str__()`` to return
``self``, itself, which should work. but then, why so complicated?

i suggest to change said line to::

if isinstance( s, ( str, unicode ) ):
...

any objections?

_wolf

Aug 23 '05 #2
Neil Schemenauer <na*@arctrix.com> writes on Mon, 22 Aug 2005 15:31:42 -0600:
...
Some code may require that str() returns a str instance. In the
standard library, only one such case has been found so far. The
function email.header_decode() requires a str instance and the
email.Header.decode_header() function tries to ensure this by
calling str() on its argument. The code was fixed by changing
the line "header = str(header)" to:

if isinstance(header, unicode):
header = header.encode('ascii')
Note, that this is not equivalent to the old "str(header)":

"str(header)" used Python's "default encoding" while the
new code uses 'ascii'.

The new code might be more correct than the old one has been.

...
Alternative Solutions

A new built-in function could be added instead of changing str().
Doing so would introduce virtually no backwards compatibility
problems. However, since the compatibility problems are expected to
rare, changing str() seems preferable to adding a new built-in.


Can we get a new builtin with the exact same behaviour as
the current "str" which can be used when we do require an "str"
(and cannot use a "unicode").

Dieter
Aug 24 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

31
by: Raymond Hettinger | last post by:
Based on your extensive feedback, PEP 322 has been completely revised. The response was strongly positive, but almost everyone preferred having a function instead of multiple object methods. The...
30
by: Hallvard B Furuseth | last post by:
Now that the '-*- coding: <charset> -*-' feature has arrived, I'd like to see an addition: # -*- str7bit:True -*- After the source file has been converted to Unicode, cause a parse error if a...
27
by: John Roth | last post by:
PEP 263 is marked finished in the PEP index, however I haven't seen the specified Phase 2 in the list of changes for 2.4 which is when I expected it. Did phase 2 get cancelled, or is it just not...
20
by: Mike Meyer | last post by:
This version includes the input from various and sundry people. Thanks to everyone who contributed. <mike PEP: XXX Title: A rational number module for Python Version: $Revision: 1.4 $...
46
by: Leo Breebaart | last post by:
I've tried Googling for this, but practically all discussions on str.join() focus on the yuck-ugly-shouldn't-it-be-a-list-method? issue, which is not my problem/question at all. What I can't...
11
by: cjl | last post by:
Hey all: I want to convert strings (ex. '3', '32') to strings with left padded zeroes (ex. '003', '032'), so I tried this: string1 = '32' string2 = "%03s" % (string1) print string2 >32
14
by: Russell E. Owen | last post by:
I have code like this: except Exception, e: self.setState(self.Failed, str(e)) which fails if the exception contains a unicode argument. I did, of course, try unicode(e) but that fails. The...
3
by: Andrii V. Mishkovskyi | last post by:
2008/5/7 Alexandr N Zamaraev <tonal@promsoft.ru>: Unicode and str objects are not the same. Why do you think that this is a bug? Anyway, you can always use 'encode' method of unicode objects: ...
19
by: est | last post by:
From python manual str( ) Return a string containing a nicely printable representation of an object. For strings, this returns the string itself. The difference with repr(object) is that...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.