urllib.urlencod e wrongly encoding ± character

[sleytr]

Hi, I'm trying to make a gui for a web service. Site using ±
character in value of some fields. But I can't encode this character
properly.

data = {'key':'±'}
urllib.urlencod e(data)

'key=%C2%B1'

but it should be only %B1 not %C2%B1. where is this %C2 coming from?

[Serge Orlov]

sl****@gmail.co m wrote:

Hi, I'm trying to make a gui for a web service. Site using ±
character in value of some fields. But I can't encode this character
properly.

data = {'key':'±'}
urllib.urlencod e(data) 'key=%C2%B1'

but it should be only %B1 not %C2%B1.
It should be %C2%B1, because de-facto urls are encoded as utf-8. I've
just tried entering ± into four input field: firefox 1.5 search
toolbar, www.google.com search in firefox 1.5, google toolbar in IE 6,
www.google.com search in IE 6. Everywhere ± is encoded as %C2%B1. In
older browsers YMMV.
where is this %C2 coming from?

Your console must be utf-8.

u'±'.encode('ut f-8')

'\xc2\xb1'

[sleytr]

you are right. but when I capture traffic in firefox via
livehttpheaders extension, it shows me that ± is encoded to %B1.
Addition to that, I found lots of page about urlencoding they have a
conversation tables or scripts. All of them defines ± as %B1 .

realy confused? I can copy and use urlencoded values from firefox, but
I'm realy want to do things with right way.

[Serge Orlov]

sl****@gmail.co m wrote:

you are right. but when I capture traffic in firefox via
livehttpheaders extension, it shows me that ± is encoded to %B1.
It depends on whether user entered url into address bar or clicked on
submit button on a page. In the first case there were no standard how
to deal with non-ascii characters for a long time. Only rfc 3986 in
2005 said: use utf-8. In the second case browsers submit forms in the
encoding of the page where the form is defined. Most likely that is
what you see when you capture traffic.

Addition to that, I found lots of page about urlencoding they have a
conversation tables or scripts. All of them defines ± as %B1 .
I guess it is because web pages usually serve pretty closed language
communities. Some people just encode urls as latin-1, and it works for
99.9999% of their users. They just don't care that they don't handle
chinese characters since they have no chinese users.

realy confused? I can copy and use urlencoded values from firefox, but
I'm realy want to do things with right way.

It is not clear what you do. Are you interacting with independant 3rd
party web service or you control both server and client?

[sleytr]

I have no control over server side.

I'm using Ubuntu Breezy at home and Ubuntu Dapper at work. Now I'm at
work and same code working properly here! (returning %B1) I'm not sure
and not checked yet but locale settings and/or installed Python version
may be different between two computers.

I think there should be way to encode ± to %B1 on any platform/locale
combination. While searching for a real solution, I'm going to add a
search&destroy filter for %C2 on urlencoded dictionary as a workaround.
Because my queries are constant and %C2 is the only problem for now.

[Fredrik Lundh]

sl****@gmail.co m wrote:

I think there should be way to encode ± to %B1 on any platform/locale
combination. While searching for a real solution, I'm going to add a
search&destroy filter for %C2 on urlencoded dictionary as a workaround.
Because my queries are constant and %C2 is the only problem for now.

I'm obviously missing some context here, but "encoding ± to %B1 on any
platform" is exactly what urlencode does:

import urllib
urllib.urlencod e([("key", chr(0xb1))])

'key=%B1'

(however, if you pass in unicode values with non-ascii characters, url-
encode will give you an error).

are you sure the conversion to UTF-8 isn't happening *before* you pass
your data to urlencode ? what does

print "1", repr(data)
print "2", repr(urllib.url encode(data))

print for the kind of data you're encoding ?

</F>

[Richard Brodie]

"Fredrik Lundh" <fr*****@python ware.com> wrote in message
news:ma******** *************** *************** *@python.org...

I'm obviously missing some context here, but "encoding ± to %B1 on any
platform" is exactly what urlencode does:

>>> import urllib
>>> urllib.urlencod e([("key", chr(0xb1))])

'key=%B1'

Yeah but you're cheating by using the platform independent chr(0xb1)
instead of a literal '±' in an unspecified encoding.

[Evren Esat Ozkan]

when I remove "# -*- coding: utf-8 -*-" line from start of the script
it worked properly. So I moved variable decleration to another file and
imported than it worked too.

Now it's working but I dont understand what I'm doing wrong? I'm new to
Python and unicode encoding. I'm tried
encode/decode(ascii,ut f-8,latin-1,iso-8859-9) on this string. None of
them worked and gave fallowing error:

UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0xc2 in position 5.
I think I must read more docs about Python and Unicode strings :)

[Evren Esat Ozkan]

I'm just discovered that I don't have to remove that line, just change
utf-8 to iso-8859-9 and it worked again. But I want to use utf-8.
Please advise...

[Fredrik Lundh]

Evren Esat Ozkan wrote:

when I remove "# -*- coding: utf-8 -*-" line from start of the script
it worked properly. So I moved variable decleration to another file and
imported than it worked too.

the coding directive controls how *unicode* literals in the *source code*
are parsed into unicode string objects. it has absolutely nothing to do with
how urlencode works.

if would help if you posted a short self-contained code snippet, so we
don't have to keep guessing.

</F>