473,581 Members | 2,915 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Unicode problem

Hi to all, I have a little problem with unicode handling under Python.

I have this code

s = u'A unicode string with this damn apostrophe \x2019'

outf = codecs.open('fi lename.txt', 'w', 'iso-8859-15')
outf.write(s)

what I obtain is a UnicodeEncodeEr ror that says me that character \x2019
maps to undefined.

But the character \x2019 is the apostrophe and in the unicode table it has
\x0027 as an equivalent, so the codecs should convert \x2019 to \x27 ( as
defined in iso-8859-15 )....

The problem is that my software deals with italian strings that has a lot
of apostrophe and other similar simbols mapped between 2000 and 206F

Have can I resolve this issue? Should I prepocess the unicode strings or
is there a way to instruct Python to do the conversion?
Jul 7 '07 #1
6 6773
On Sat, 07 Jul 2007 16:06:03 +0000, pa******@giochi nternet.com wrote:
Hi to all, I have a little problem with unicode handling under Python.

I have this code

s = u'A unicode string with this damn apostrophe \x2019'

outf = codecs.open('fi lename.txt', 'w', 'iso-8859-15')
outf.write(s)

what I obtain is a UnicodeEncodeEr ror that says me that character \x2019
maps to undefined.

But the character \x2019 is the apostrophe and in the unicode table it has
\x0027 as an equivalent, so the codecs should convert \x2019 to \x27 ( as
defined in iso-8859-15 )....
No it shouldn't because \x2019 is a "right single quotation mark" and not
an apostrophe.

Ciao,
Marc 'BlackJack' Rintsch
Jul 7 '07 #2
No it shouldn't because \x2019 is a "right single quotation mark" and not
an apostrophe.

Ciao,
Marc 'BlackJack' Rintsch

I agree, but the problem is much subtle. I have coverted a text from
iso-8859-1 to utf-8 and the codecs have translated \x27 ( the iso
apostrophe ) to \xe28099 in utf-8 ( or u'2019' in unicode code point
notation )

So if convert an apostrophe to a "right single quotation mark" why not
translate the "right single quotation mark" to "apostrophe "

As I can see it works in one direction but not in the other
Jul 7 '07 #3
I agree, but the problem is much subtle. I have coverted a text from
iso-8859-1 to utf-8 and the codecs have translated \x27 ( the iso
apostrophe ) to \xe28099 in utf-8 ( or u'2019' in unicode code point
notation )
What software did you use to make that so? The Python codec certainly
never would do such a thing.

Are you sure it was latin-1 and \x27, and not windows-1252 and \x92?

Regards,
Martin
Jul 7 '07 #4
pa******@giochi nternet.com <pa******@gioch internet.comwro te:
...
Ah, I answered you on the Italian NG before seeing you had also posted
the same request here. What I proposed there was (untested):

import codecs

_rimedi = { u'\x2019': "'" }

def rimedia(exc):
if isinstance(exc, (UnicodeEncodeE rror, UnicodeTranslat eError)):
erore = exc.object[exc.start:exc.e nd]
if len(erore)==1 and erore in _rimedi: return _rimedi[erore]
raise exc
codecs.register _error('rimedia ', rimedia)

outf = codecs.open('fi lename.txt', 'w', 'iso-8859-15', errors='rimedia ')
Alex
Jul 7 '07 #5
pa******@giochi nternet.com wrote:
Hi to all, I have a little problem with unicode handling under Python.

I have this code

s = u'A unicode string with this damn apostrophe \x2019'

outf = codecs.open('fi lename.txt', 'w', 'iso-8859-15')
outf.write(s)

what I obtain is a UnicodeEncodeEr ror that says me that character \x2019
maps to undefined.

But the character \x2019 is the apostrophe and in the unicode table it has
\x0027 as an equivalent, so the codecs should convert \x2019 to \x27 ( as
defined in iso-8859-15 )....
U+2019 is RIGHT SINGLE QUOTATION MARK. The APOSTROPHE (U+0027) is a
cross-reference as a similar code point, but they're not the same thing.

Your problem is that ISO-8859-15 doesn't have the RIGHT SINGLE QUOTATION
MARK, so you'll have to do the translation yourself if you want to turn
it into a true APOSTROPHE.

--
Erik Max Francis && ma*@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM, Y!M erikmaxfrancis
She glanced at her watch ... It was 9:23.
-- James Clavell
Jul 7 '07 #6
>
What software did you use to make that so? The Python codec certainly
never would do such a thing.

Are you sure it was latin-1 and \x27, and not windows-1252 and \x92?

Regards,
Martin
you're right...the source of text are html pages and obviously webmasters
have poor knowledge of encodings, so the meta declared the encoding as
ISO-8859-1 but the real encoding is Windows-1252 and yes it uses \x92 as
apostrophe, so the problem isn't Python
Jul 8 '07 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
7078
by: sebastien.hugues | last post by:
Hi I would like to retrieve the application data directory path of the logged user on windows XP. To achieve this goal i use the environment variable APPDATA. The logged user has this name: sťbastien. The second character is not an ascii one and when i try to encode the path that contains this name in utf-8,
8
5259
by: Bill Eldridge | last post by:
I'm trying to grab a document off the Web and toss it into a MySQL database, but I keep running into the various encoding problems with Unicode (that aren't a problem for me with GB2312, BIG 5, etc.) What I'd like is something as simple as: CREATE TABLE junk (junklet VARCHAR(2500) CHARACTER SET UTF8)); import MySQLdb, re,urllib
14
2834
by: wolfgang haefelinger | last post by:
Hi, I wonder whether someone could explain me a bit what's going on here: import sys # I'm running Mandrake 1o and Windows XP. print sys.version ## 2.3.3 (#2, Feb 17 2004, 11:45:40)
19
5648
by: Svennglenn | last post by:
I'm working on a program that is supposed to save different information to text files. Because the program is in swedish i have to use unicode text for Ňń÷ letters. When I run the following testscript I get an error message. # -*- coding: cp1252 -*-
6
5600
by: peter pilsl | last post by:
postgres 7.3.2 I store unicode-data in postgresql. The data is retrieved via webinterfaces, processed with perl and then stored in postgresql (and viceversa). All is going nice with one problem. If performing a "select * order by field"-query the result is not what I expected. German umlauts (ie: ÷) are stored as doublechars ("÷" is...
1
4438
by: Paul | last post by:
Hi, I am extending an existing MFC app to use Unicode (for a Japanese version of the interface elements). The app's purpose is to control a peripheral device through the serial port, and the device only understands ansi strings (in fact, I synthesize a character stream and send it as though it was typed into a terminal). The problem seems...
3
5461
by: Kidus Yared | last post by:
I am having a problem displaying Unicode characters on my Forms labels and buttons. After coding Button1.Text = unicode; where the unicode is a Unicode character or string (‚Äė\u1234‚Äô or ‚Äú\u1234‚ÄĚ) It seems to work the first time I set the button to the Unicode character. After a while, when saving my code, I get a pop-up window stating...
1
4838
by: jrs_14618 | last post by:
Hello All, This post is essentially a reply a previous post/thread here on this mailing.database.myodbc group titled: MySQL 4.0, FULL-TEXT Indexing and Search Arabic Data, Unicode I was wondering if anybody has experienced the same issues
5
3026
by: Holger Joukl | last post by:
Hi there, I consider the behaviour of unicode() inconvenient wrt to conversion of non-string arguments. While you can do: u'17.3' you cannot do:
5
2585
by: Thierry | last post by:
Hello fellow pythonists, I'm a relatively new python developer, and I try to adjust my understanding about "how things works" to python, but I have hit a block, that I cannot understand. I needed to output unicode datas back from a web service, and could not get back unicode/multibyte text before applying an hack that I don't understand...
0
7883
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, weíll explore What is ONU, What Is Router, ONU & Routerís main...
0
8182
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
0
6567
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development projectóplanning, coding, testing, and deploymentówithout human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
1
5685
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Duprť who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
5366
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3809
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
1
2311
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
1413
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
1147
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.