Is there a way to get utf-8 out of a Unicode string?

[thebjorn]

I've got a database (ms sqlserver) that's (way) out of my control,
where someone has stored utf-8 encoded Unicode data in regular varchar
fields, so that e.g. the string 'Blåbærsyltetøy' is in the database
as 'Bl\xc3\xa5b\xc3\xa6rsyltet\xc3\xb8y' :-/

Then I read the data out using adodbapi (which returns all strings as
Unicode) and I get u'Bl\xc3\xa5b\xc3\xa6rsyltet\xc3\xb8y'. I couldn't
find any way to get back to the original short of:

def unfk(s):
return eval(repr(s)[1:]).decode('utf-8')

i.e. chopping off the u in the repr of a unicode string, and relying on
eval to interpret the \xHH sequences.

Is there a less hack'ish way to do this?

-- bjorn

[Fredrik Lundh]

thebjorn wrote:

I've got a database (ms sqlserver) that's (way) out of my control,
where someone has stored utf-8 encoded Unicode data in regular varchar
fields, so that e.g. the string 'Blåbærsyltetøy' is in the database
as 'Bl\xc3\xa5b\xc3\xa6rsyltet\xc3\xb8y' :-/

Then I read the data out using adodbapi (which returns all strings as
Unicode) and I get u'Bl\xc3\xa5b\xc3\xa6rsyltet\xc3\xb8y'. I couldn't
find any way to get back to the original short of:

def unfk(s):
return eval(repr(s)[1:]).decode('utf-8')

i.e. chopping off the u in the repr of a unicode string, and relying on
eval to interpret the \xHH sequences.

Is there a less hack'ish way to do this?

first, check if you can get your database adapter to understand that the
database contains UTF-8 and not ISO-8859-1. if that's not possible, you
can roundtrip via ISO-8859-1 yourself:

>>u = u'Bl\xc3\xa5b\xc3\xa6rsyltet\xc3\xb8y'
u

u'Bl\xc3\xa5b\xc3\xa6rsyltet\xc3\xb8y'

>>u.encode("iso-8859-1")

'Bl\xc3\xa5b\xc3\xa6rsyltet\xc3\xb8y'

>>u.encode("iso-8859-1").decode("utf-8")

u'Bl\xe5b\xe6rsyltet\xf8y'

>>print u.encode("iso-8859-1").decode("utf-8")

Blåbærsyltetøy

</F>

[Gerrit Holl]

Hei,

On 2006-10-30 08:25:41 +0100, thebjorn wrote:

def unfk(s):
return eval(repr(s)[1:]).decode('utf-8')

i.e. chopping off the u in the repr of a unicode string, and relying on
eval to interpret the \xHH sequences.

Is there a less hack'ish way to do this?

Slightly lack hackish:

return ''.join(chr(ord(c)) for c in s)

Gerrit.

[thebjorn]

Fredrik Lundh wrote:

thebjorn wrote:

I've got a database (ms sqlserver) that's (way) out of my control,
where someone has stored utf-8 encoded Unicode data in regular varchar
fields, so that e.g. the string 'Blåbærsyltetøy' is in the database
as 'Bl\xc3\xa5b\xc3\xa6rsyltet\xc3\xb8y' :-/

...

first, check if you can get your database adapter to understand that the
database contains UTF-8 and not ISO-8859-1.

It would be the way to go, however it looks like they've managed to get
Latin-1 data in exactly two columns in the entire database (this is a
commercial product of course, so there's no way for us to fix things).
And just to make things more interesting, I think I'm running into an
ADO bug where capital letters (outside the U+0000 to U+007F range) are
returning strange values:

>>c.execute('create table utf8 (f1 varchar(15))')
u'ÆØÅÉ'.encode('utf-8')

'\xc3\x86\xc3\x98\xc3\x85\xc3\x89'

>>x = _
c.execute('insert into utf8 (f1) values (?)', (x,))
c.execute('select * from utf8')
c.fetchall()

((u'\xc3\u2020\xc3\u02dc\xc3\u2026\xc3\u2030',),)

>>>

I haven't tested this through C[#/++] to verify that it's an ADO issue,
but it seems unlikely that MS would view this as anything but incorrect
usage no matter where the issue is...

Anyway, sorry for venting :-)

if that's not possible, you can roundtrip via ISO-8859-1 yourself:

>>u = u'Bl\xc3\xa5b\xc3\xa6rsyltet\xc3\xb8y'

....

>>print u.encode("iso-8859-1").decode("utf-8")

Blåbærsyltetøy

That's very nice!

-- bjorn

[thebjorn]

Gerrit Holl wrote:

Hei,

On 2006-10-30 08:25:41 +0100, thebjorn wrote:

def unfk(s):
return eval(repr(s)[1:]).decode('utf-8')

....

Is there a less hack'ish way to do this?

Slightly lack hackish:

return ''.join(chr(ord(c)) for c in s)

Much less hackish :-)

-- bjorn