473,594 Members | 2,839 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

urllib.urlencod e wrongly encoding character

Hi, I'm trying to make a gui for a web service. Site using
character in value of some fields. But I can't encode this character
properly.

data = {'key':''}
urllib.urlencod e(data)

'key=%C2%B1'

but it should be only %B1 not %C2%B1. where is this %C2 coming from?

Apr 6 '06 #1
12 4912
sl****@gmail.co m wrote:
Hi, I'm trying to make a gui for a web service. Site using
character in value of some fields. But I can't encode this character
properly.

data = {'key':''}
urllib.urlencod e(data) 'key=%C2%B1'

but it should be only %B1 not %C2%B1.
It should be %C2%B1, because de-facto urls are encoded as utf-8. I've
just tried entering into four input field: firefox 1.5 search
toolbar, www.google.com search in firefox 1.5, google toolbar in IE 6,
www.google.com search in IE 6. Everywhere is encoded as %C2%B1. In
older browsers YMMV.
where is this %C2 coming from?


Your console must be utf-8.
u''.encode('ut f-8')

'\xc2\xb1'

Apr 6 '06 #2
you are right. but when I capture traffic in firefox via
livehttpheaders extension, it shows me that is encoded to %B1.
Addition to that, I found lots of page about urlencoding they have a
conversation tables or scripts. All of them defines as %B1 .

realy confused? I can copy and use urlencoded values from firefox, but
I'm realy want to do things with right way.

Apr 6 '06 #3
sl****@gmail.co m wrote:
you are right. but when I capture traffic in firefox via
livehttpheaders extension, it shows me that is encoded to %B1.
It depends on whether user entered url into address bar or clicked on
submit button on a page. In the first case there were no standard how
to deal with non-ascii characters for a long time. Only rfc 3986 in
2005 said: use utf-8. In the second case browsers submit forms in the
encoding of the page where the form is defined. Most likely that is
what you see when you capture traffic.

Addition to that, I found lots of page about urlencoding they have a
conversation tables or scripts. All of them defines as %B1 .
I guess it is because web pages usually serve pretty closed language
communities. Some people just encode urls as latin-1, and it works for
99.9999% of their users. They just don't care that they don't handle
chinese characters since they have no chinese users.

realy confused? I can copy and use urlencoded values from firefox, but
I'm realy want to do things with right way.


It is not clear what you do. Are you interacting with independant 3rd
party web service or you control both server and client?

Apr 6 '06 #4
I have no control over server side.

I'm using Ubuntu Breezy at home and Ubuntu Dapper at work. Now I'm at
work and same code working properly here! (returning %B1) I'm not sure
and not checked yet but locale settings and/or installed Python version
may be different between two computers.

I think there should be way to encode to %B1 on any platform/locale
combination. While searching for a real solution, I'm going to add a
search&destroy filter for %C2 on urlencoded dictionary as a workaround.
Because my queries are constant and %C2 is the only problem for now.

Apr 6 '06 #5
sl****@gmail.co m wrote:
I think there should be way to encode to %B1 on any platform/locale
combination. While searching for a real solution, I'm going to add a
search&destroy filter for %C2 on urlencoded dictionary as a workaround.
Because my queries are constant and %C2 is the only problem for now.


I'm obviously missing some context here, but "encoding to %B1 on any
platform" is exactly what urlencode does:
import urllib
urllib.urlencod e([("key", chr(0xb1))])

'key=%B1'

(however, if you pass in unicode values with non-ascii characters, url-
encode will give you an error).

are you sure the conversion to UTF-8 isn't happening *before* you pass
your data to urlencode ? what does

print "1", repr(data)
print "2", repr(urllib.url encode(data))

print for the kind of data you're encoding ?

</F>

Apr 6 '06 #6

"Fredrik Lundh" <fr*****@python ware.com> wrote in message
news:ma******** *************** *************** *@python.org...
I'm obviously missing some context here, but "encoding to %B1 on any
platform" is exactly what urlencode does:
>>> import urllib
>>> urllib.urlencod e([("key", chr(0xb1))])

'key=%B1'


Yeah but you're cheating by using the platform independent chr(0xb1)
instead of a literal '' in an unspecified encoding.
Apr 6 '06 #7
when I remove "# -*- coding: utf-8 -*-" line from start of the script
it worked properly. So I moved variable decleration to another file and
imported than it worked too.

Now it's working but I dont understand what I'm doing wrong? I'm new to
Python and unicode encoding. I'm tried
encode/decode(ascii,ut f-8,latin-1,iso-8859-9) on this string. None of
them worked and gave fallowing error:

UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0xc2 in position 5.
I think I must read more docs about Python and Unicode strings :)

Apr 6 '06 #8
I'm just discovered that I don't have to remove that line, just change
utf-8 to iso-8859-9 and it worked again. But I want to use utf-8.
Please advise...

Apr 6 '06 #9
Evren Esat Ozkan wrote:
when I remove "# -*- coding: utf-8 -*-" line from start of the script
it worked properly. So I moved variable decleration to another file and
imported than it worked too.


the coding directive controls how *unicode* literals in the *source code*
are parsed into unicode string objects. it has absolutely nothing to do with
how urlencode works.

if would help if you posted a short self-contained code snippet, so we
don't have to keep guessing.

</F>

Apr 6 '06 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
28397
by: Joshua Beall | last post by:
Hi All, I can see from the manual that the difference between urlencode and rawurlencode is that urlencode translates spaces to '+' characters, whereas rawurlencode translates it into it's hex code. My question is, is there any real world difference between these two functions? Or perhaps another way of asking the question: *why* are there two different functions? In what situation would you need one, and not be able to use the...
7
2324
by: Stuart McGraw | last post by:
I just spent a $*#@!*&^&% hour registering at ^$#@#%^ Sourceforce and trying to submit a Python bug report but it still won't let me. I give up. Maybe someone who cares will see this post, or maybe it will save time for someone else who runs into this problem... ================================================ Environment: - Microsoft Windows 2000 Pro
0
3577
by: Pieter Edelman | last post by:
Hi all, I'm trying to submit some data using a POST request to a HTTP server with BASIC authentication with python, but I can't get it to work. Since it's driving me completely nuts, so here's my cry for help. The server is an elog logbook server (http://midas.psi.ch/elog/). It is protected with a password and an empty username. I can login both using urllib and urllib2 (suppose the password is "foobar", the logbook is running on port...
3
5148
by: George Hester | last post by:
http://support.microsoft.com/default.aspx?scid=kb;en-us;301464 Look down at the MyPage.asp example. You will see that Microsoft does this: 'Costruct the URL for the current page s = "http://" s = s & Request.ServerVariables("HTTP_HOST") s = s & Request.ServerVariables("URL") If Request.QueryString.Count > 0 Then s = s & "?" & Request.QueryString
5
2409
by: vtreddy | last post by:
Hi All: In my application I am sending the input values through query string from a datagrid template column,I used URLEncode to encode the URL String, please find the input below, I am facing a problem when the input string contains character , Here from ASP.NET I am redirecting to the ASP based system,when I give same input from ASP based system, it is showing Microsoft%AE%20Access%20 ,with this I am getting the correct results, where as...
9
4494
by: Mark_Rarefy | last post by:
Trying to urlencode this string: »ÁÏŒŠ˜ªŒ›h^aYh in vb.net (using either HttpUtility.UrlEncode(strEncrypted, encoding.UTF8) orServer.UrlEncode) I get: %c2%90%c2%bb%c3%81%c3%8f%7f%c5%92%c5%a0%cb%9c%c2%aa%c5%92%c2%9d%e2%80%bah%5eaYh in ASP script (using Server.URLEncode) I get: %90%BB%C1%CF%7F%8C%8A%98%AA%8C%9D%9Bh%5EaYh
1
496
by: evanpmeth | last post by:
I have tried multiple ways of posting information to a website and have failed. I have seen this problem on other forums can someone explain or point me to information on how POST works through urllib an different broweser (what is the difference). my first attempt was out of the docs: Code: import httplib, urllib params = urllib.urlencode({'email' : 'a@a.com',
11
9496
by: George Sakkis | last post by:
The following snippet results in different outcome for (at least) the last three major releases: # Python 2.3.4 u'%94' # Python 2.4.2 UnicodeDecodeError: 'ascii' codec can't decode byte 0x94 in position 0: ordinal not in range(128)
0
1338
by: gmguyx | last post by:
I tried using urllib.urlopen to open a personalized webpage (my.yahoo.com) but it doesn't work: print urllib.urlopen(http://my.yahoo.com).read() Instead of returning my.yahoo.com, it returns a page asking me to log in. So then I tried urllib.urlencode:
0
7877
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
8253
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
8374
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
8009
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
1
5739
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupr who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
3867
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
2389
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
1482
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
1216
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.