473,324 Members | 2,179 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,324 software developers and data experts.

possible unicode bug in implicit string concatenation?

Hi team! While troubleshooting a crash I had while using BitTorrent
where the torrent's target file names didn't fall into the ascii range
I was playing around in the interpreter and noticed this behaviour:
u'\u12345' + 'foo' u'\u12345foo' u'\u12345' u'foo' u'\u12345foo' u'\u12345' + u'foo'.encode('ascii') u'\u12345foo' u'\u12345' u'foo'.encode('ascii') Traceback (most recent call last):
File "<interactive input>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\u1234' in
position 0: ordinal not in range(128)


Is this a bug, or is my understanding of how Python works flawed? I
tried tracing it within the interpreter itself bug got lost after a
little while... I'm familiar with the interpreter loop, but not the
parser, and I suspect this is something to do with implicit string
concatenation being parsed differently from the explicit version, i.e.
the explicit version uses the + operator slot, while the implicit
version does something else. Any ideas?

Fahd Khan
ICON | Clinical Research
W: 281-295-4834
Jul 18 '05 #1
1 2329
Fahd Khan wrote:
u'\u12345' u'foo'.encode('ascii')

Traceback (most recent call last):
File "<interactive input>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\u1234' in
position 0: ordinal not in range(128)
Is this a bug, or is my understanding of how Python works flawed?


Yes :-) Your understanding is flawed.
I
tried tracing it within the interpreter itself bug got lost after a
little while... I'm familiar with the interpreter loop, but not the
parser, and I suspect this is something to do with implicit string
concatenation being parsed differently from the explicit version, i.e.
the explicit version uses the + operator slot, while the implicit
version does something else. Any ideas?


During parsing, strings are concatenated. And concatenation is
the same as +. So the expression at the top of this message is
the same as u'\u12345foo'.encode('ascii'). That fails because
\u1234 is not supported in ASCII. Now,

u'\u12345'+u'foo'.encode('ascii')

is something completely different: concatenation does not happen
during parsing, but only at execution. The computation of this
expression is as follows

u'foo'.encode('ascii') gives 'foo'
u'\u12345'+'foo' finds that Unicode and byte strings are to be
added. This causes the byte string to be coerced to Unicode,
computing
'foo'.decode(sys.getdefaultencoding())
sys.getdefaultencoding() gives 'ascii'
'foo'.decode('ascii') gives u'foo'
u'\u12345'+u'foo' gives u'\u12345foo'

Regards,
Martin
Jul 18 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
by: Ivan Voras | last post by:
When concatenating strings (actually, a constant and a string...) i get the following error: UnicodeDecodeError: 'ascii' codec can't decode byte 0xfc in position 1: ordinal not in range(128) ...
4
by: fowlertrainer | last post by:
Hi ! I want to get the WMI infos from Windows machines. I use Py from HU (iso-8859-2) charset. Then I wrote some utility for it, because I want to write it to an XML file. def...
1
by: jrs_14618 | last post by:
Hello All, This post is essentially a reply a previous post/thread here on this mailing.database.myodbc group titled: MySQL 4.0, FULL-TEXT Indexing and Search Arabic Data, Unicode I was...
8
by: sonald | last post by:
Hi, I am using python2.4.1 I need to pass russian text into python and validate the same. Can u plz guide me on how to make my existing code support the russian text. Is there any module...
17
by: Adam Olsen | last post by:
As was seen in another thread, there's a great deal of confusion with regard to surrogates. Most programmers assume Python's unicode type exposes only complete characters. Even CPython's own...
7
by: 7stud | last post by:
Based on this example and the error: ----- u_str = u"abc\u9999" print u_str UnicodeEncodeError: 'ascii' codec can't encode character u'\u9999' in position 3: ordinal not in range(128) ------
7
by: Robert Latest | last post by:
Here's a test snippet... import sys for k in sys.stdin: print '%s -%s' % (k, k.decode('iso-8859-1')) ....but it barfs when actually fed with iso8859-1 characters. How is this done right? ...
3
by: RN1 | last post by:
Consider the following code snippet: <% Dim intA, strA, strB intA=5 strA="Hello World" strB=strA+intA Response.Write(strB) %>
5
by: Thierry | last post by:
Hello fellow pythonists, I'm a relatively new python developer, and I try to adjust my understanding about "how things works" to python, but I have hit a block, that I cannot understand. I...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.