possible unicode bug in implicit string concatenation?

[Fahd Khan]

Hi team! While troubleshooting a crash I had while using BitTorrent
where the torrent's target file names didn't fall into the ascii range
I was playing around in the interpreter and noticed this behaviour:

u'\u12345' + 'foo' u'\u12345foo' u'\u12345' u'foo' u'\u12345foo' u'\u12345' + u'foo'.encode(' ascii') u'\u12345foo' u'\u12345' u'foo'.encode(' ascii') Traceback (most recent call last):
File "<interacti ve input>", line 1, in ?
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\u1234' in
position 0: ordinal not in range(128)

Is this a bug, or is my understanding of how Python works flawed? I
tried tracing it within the interpreter itself bug got lost after a
little while... I'm familiar with the interpreter loop, but not the
parser, and I suspect this is something to do with implicit string
concatenation being parsed differently from the explicit version, i.e.
the explicit version uses the + operator slot, while the implicit
version does something else. Any ideas?

Fahd Khan
ICON | Clinical Research
W: 281-295-4834

[Martin v. Löwis]

Fahd Khan wrote:

u'\u12345 ' u'foo'.encode(' ascii')

Traceback (most recent call last):
File "<interacti ve input>", line 1, in ?
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\u1234' in
position 0: ordinal not in range(128)
Is this a bug, or is my understanding of how Python works flawed?

Yes :-) Your understanding is flawed.
I
tried tracing it within the interpreter itself bug got lost after a
little while... I'm familiar with the interpreter loop, but not the
parser, and I suspect this is something to do with implicit string
concatenation being parsed differently from the explicit version, i.e.
the explicit version uses the + operator slot, while the implicit
version does something else. Any ideas?

During parsing, strings are concatenated. And concatenation is
the same as +. So the expression at the top of this message is
the same as u'\u12345foo'.e ncode('ascii'). That fails because
\u1234 is not supported in ASCII. Now,

u'\u12345'+u'fo o'.encode('asci i')

is something completely different: concatenation does not happen
during parsing, but only at execution. The computation of this
expression is as follows

u'foo'.encode(' ascii') gives 'foo'
u'\u12345'+'foo ' finds that Unicode and byte strings are to be
added. This causes the byte string to be coerced to Unicode,
computing
'foo'.decode(sy s.getdefaultenc oding())
sys.getdefaulte ncoding() gives 'ascii'
'foo'.decode('a scii') gives u'foo'
u'\u12345'+u'fo o' gives u'\u12345foo'

Regards,
Martin