473,699 Members | 2,383 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

converting to and from octal escaped UTF--8

Hi,

I am writing unicode stings into a special text file that requires to
have non-ascii characters as as octal-escaped UTF-8 codes.

For example, the letter "Í" (latin capital I with acute, code point 205)
would come out as "\303\215".

I will also have to read back from the file later on and convert the
escaped characters back into a unicode string.

Does anyone have any suggestions on how to go from "Í" to "\303\215" and
vice versa?

I know I can get the code point by doing
>>"Í".decode(' utf-8').encode('uni code_escape')
but there doesn't seem to be any similar method for getting the octal
escaped version.

Thanks,
Michael
Dec 3 '07 #1
9 11557
Michael Goerz wrote:
Hi,

I am writing unicode stings into a special text file that requires to
have non-ascii characters as as octal-escaped UTF-8 codes.

For example, the letter "Í" (latin capital I with acute, code point 205)
would come out as "\303\215".

I will also have to read back from the file later on and convert the
escaped characters back into a unicode string.

Does anyone have any suggestions on how to go from "Í" to "\303\215" and
vice versa?

I know I can get the code point by doing
>>>"Í".decode( 'utf-8').encode('uni code_escape')
but there doesn't seem to be any similar method for getting the octal
escaped version.

Thanks,
Michael
I've come up with the following solution. It's not very pretty, but it
works (no bugs, I hope). Can anyone think of a better way to do it?

Michael
_________

import binascii

def escape(s):
hexstring = binascii.b2a_he x(s)
result = ""
while len(hexstring) 0:
(hexbyte, hexstring) = (hexstring[:2], hexstring[2:])
octbyte = oct(int(hexbyte , 16)).zfill(3)
result += "\\" + octbyte[-3:]
return result

def unescape(s):
result = ""
while len(s) 0:
if s[0] == "\\":
(octbyte, s) = (s[1:4], s[4:])
try:
result += chr(int(octbyte , 8))
except ValueError:
result += "\\"
s = octbyte + s
else:
result += s[0]
s = s[1:]
return result

print escape("\303\21 5")
print unescape('adf\\ 303\\215adf')
Dec 3 '07 #2
On Dec 2, 8:38 pm, Michael Goerz <answer...@8439 .e4ward.comwrot e:
Michael Goerz wrote:
Hi,
I am writing unicode stings into a special text file that requires to
have non-ascii characters as as octal-escaped UTF-8 codes.
For example, the letter "" (latin capital I with acute, code point 205)
would come out as "\303\215".
I will also have to read back from the file later on and convert the
escaped characters back into a unicode string.
Does anyone have any suggestions on how to go from "" to "\303\215" and
vice versa?
I know I can get the code point by doing
>>"".decode('u tf-8').encode('uni code_escape')
but there doesn't seem to be any similar method for getting the octal
escaped version.
Thanks,
Michael

I've come up with the following solution. It's not very pretty, but it
works (no bugs, I hope). Can anyone think of a better way to do it?

Michael
_________

import binascii

def escape(s):
hexstring = binascii.b2a_he x(s)
result = ""
while len(hexstring) 0:
(hexbyte, hexstring) = (hexstring[:2], hexstring[2:])
octbyte = oct(int(hexbyte , 16)).zfill(3)
result += "\\" + octbyte[-3:]
return result

def unescape(s):
result = ""
while len(s) 0:
if s[0] == "\\":
(octbyte, s) = (s[1:4], s[4:])
try:
result += chr(int(octbyte , 8))
except ValueError:
result += "\\"
s = octbyte + s
else:
result += s[0]
s = s[1:]
return result

print escape("\303\21 5")
print unescape('adf\\ 303\\215adf')
Looks like escape() can be a bit simpler...

def escape(s):
result = []
for char in s:
result.append(" \%o" % ord(char))
return ''.join(result)

Regards,
Jordan
Dec 3 '07 #3
MonkeeSage wrote:
Looks like escape() can be a bit simpler...

def escape(s):
result = []
for char in s:
result.append(" \%o" % ord(char))
return ''.join(result)

Regards,
Jordan
Very neat! Thanks a lot...
Michael
Dec 3 '07 #4
Michael Goerz wrote:
Hi,

I am writing unicode stings into a special text file that requires to
have non-ascii characters as as octal-escaped UTF-8 codes.

For example, the letter "Í" (latin capital I with acute, code point 205)
would come out as "\303\215".

I will also have to read back from the file later on and convert the
escaped characters back into a unicode string.

Does anyone have any suggestions on how to go from "Í" to "\303\215" and
vice versa?
Perhaps something along the lines of:
>>def encode(source):
... return "".join("\% o" % ord(c) for c in source.encode(' utf8'))
...
>>def decode(encoded) :
... bytes = "".join(chr(int (c, 8)) for c in encoded.split(' \\')[1:])
... return bytes.decode('u tf8')
...
>>encode(u"Í ")
'\\303\\215'
>>print decode(_)
Í
>>>
HTH
Michael

Dec 3 '07 #5
On Dec 2, 11:46 pm, Michael Spencer <m...@telcopart ners.comwrote:
Michael Goerz wrote:
Hi,
I am writing unicode stings into a special text file that requires to
have non-ascii characters as as octal-escaped UTF-8 codes.
For example, the letter "" (latin capital I with acute, code point 205)
would come out as "\303\215".
I will also have to read back from the file later on and convert the
escaped characters back into a unicode string.
Does anyone have any suggestions on how to go from "" to "\303\215" and
vice versa?

Perhaps something along the lines of:
>>def encode(source):
... return "".join("\% o" % ord(c) for c in source.encode(' utf8'))
...
>>def decode(encoded) :
... bytes = "".join(chr(int (c, 8)) for c in encoded.split(' \\')[1:])
... return bytes.decode('u tf8')
...
>>encode(u"" )
'\\303\\215'
>>print decode(_)
>>>

HTH
Michael
Nice one. :) If I might suggest a slight variation to handle cases
where the "encoded" string contains plain text as well as octal
escapes...

def decode(encoded) :
for octc in (c for c in re.findall(r'\\ (\d{3})', encoded)):
encoded = encoded.replace (r'\%s' % octc, chr(int(octc, 8)))
return encoded.decode( 'utf8')

This way it can handle both "\\141\\144\\14 6\\303\\215\\14 1\\144\\146"
as well as "adf\\303\\215a df".

Regards,
Jordan
Dec 3 '07 #6
On Dec 3, 1:31 am, MonkeeSage <MonkeeS...@gma il.comwrote:
On Dec 2, 11:46 pm, Michael Spencer <m...@telcopart ners.comwrote:
Michael Goerz wrote:
Hi,
I am writing unicode stings into a special text file that requires to
have non-ascii characters as as octal-escaped UTF-8 codes.
For example, the letter "" (latin capital I with acute, code point 205)
would come out as "\303\215".
I will also have to read back from the file later on and convert the
escaped characters back into a unicode string.
Does anyone have any suggestions on how to go from "" to "\303\215"a nd
vice versa?
Perhaps something along the lines of:
>>def encode(source):
... return "".join("\% o" % ord(c) for c in source.encode(' utf8'))
...
>>def decode(encoded) :
... bytes = "".join(chr(int (c, 8)) for c in encoded.split(' \\')[1:])
... return bytes.decode('u tf8')
...
>>encode(u"" )
'\\303\\215'
>>print decode(_)
HTH
Michael

Nice one. :) If I might suggest a slight variation to handle cases
where the "encoded" string contains plain text as well as octal
escapes...

def decode(encoded) :
for octc in (c for c in re.findall(r'\\ (\d{3})', encoded)):
encoded = encoded.replace (r'\%s' % octc, chr(int(octc, 8)))
return encoded.decode( 'utf8')

This way it can handle both "\\141\\144\\14 6\\303\\215\\14 1\\144\\146"
as well as "adf\\303\\215a df".

Regards,
Jordan
err...

def decode(encoded) :
for octc in re.findall(r'\\ (\d{3})', encoded):
encoded = encoded.replace (r'\%s' % octc, chr(int(octc, 8)))
return encoded.decode( 'utf8')
Dec 3 '07 #7
MonkeeSage wrote:
On Dec 3, 1:31 am, MonkeeSage <MonkeeS...@gma il.comwrote:
>On Dec 2, 11:46 pm, Michael Spencer <m...@telcopart ners.comwrote:
>>Michael Goerz wrote:
Hi,
I am writing unicode stings into a special text file that requires to
have non-ascii characters as as octal-escaped UTF-8 codes.
For example, the letter "" (latin capital I with acute, code point 205)
would come out as "\303\215".
I will also have to read back from the file later on and convert the
escaped characters back into a unicode string.
Does anyone have any suggestions on how to go from "" to "\303\215" and
vice versa?
Perhaps something along the lines of:
>>def encode(source):
... return "".join("\% o" % ord(c) for c in source.encode(' utf8'))
...
>>def decode(encoded) :
... bytes = "".join(chr(int (c, 8)) for c in encoded.split(' \\')[1:])
... return bytes.decode('u tf8')
...
>>encode(u"" )
'\\303\\215'
>>print decode(_)

HTH
Michael
Nice one. :) If I might suggest a slight variation to handle cases
where the "encoded" string contains plain text as well as octal
escapes...

def decode(encoded) :
for octc in (c for c in re.findall(r'\\ (\d{3})', encoded)):
encoded = encoded.replace (r'\%s' % octc, chr(int(octc, 8)))
return encoded.decode( 'utf8')

This way it can handle both "\\141\\144\\14 6\\303\\215\\14 1\\144\\146"
as well as "adf\\303\\215a df".

Regards,
Jordan

err...

def decode(encoded) :
for octc in re.findall(r'\\ (\d{3})', encoded):
encoded = encoded.replace (r'\%s' % octc, chr(int(octc, 8)))
return encoded.decode( 'utf8')
Great suggestions from both of you! I came up with my "final" solution
based on them. It encodes only non-ascii and non-printables, and stays
in unicode strings for both input and output. Also, low ascii values now
encode into a 3-digit octal sequence also, so that decode can catch them
properly.

Thanks a lot,
Michael

____________

import re

def encode(source):
encoded = ""
for character in source:
if (ord(character) < 32) or (ord(character) 128):
for byte in character.encod e('utf8'):
encoded += ("\%03o" % ord(byte))
else:
encoded += character
return encoded.decode( 'utf-8')

def decode(encoded) :
decoded = encoded.encode( 'utf-8')
for octc in re.findall(r'\\ (\d{3})', decoded):
decoded = decoded.replace (r'\%s' % octc, chr(int(octc, 8)))
return decoded.decode( 'utf8')
orig = u"blablub" + chr(10)
enc = encode(orig)
dec = decode(enc)
print orig
print enc
print dec

Dec 3 '07 #8
>>>>Michael Goerz <an*******@8439 .e4ward.com(MG) wrote:
>MG if (ord(character) < 32) or (ord(character) 128):
If you encode chars < 32 it seems more appropriate to also encode 127.

Moreover your code is quadratic in the size of the string so if you use
long strings it would be better to use join.
--
Piet van Oostrum <pi**@cs.uu.n l>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C 4]
Private email: pi**@vanoostrum .org
Dec 4 '07 #9
On Dec 3, 8:10 am, Michael Goerz <answer...@8439 .e4ward.comwrot e:
MonkeeSage wrote:
On Dec 3, 1:31 am, MonkeeSage <MonkeeS...@gma il.comwrote:
On Dec 2, 11:46 pm, Michael Spencer <m...@telcopart ners.comwrote:
>Michael Goerz wrote:
Hi,
I am writing unicode stings into a special text file that requires to
have non-ascii characters as as octal-escaped UTF-8 codes.
For example, the letter "" (latin capital I with acute, code point205)
would come out as "\303\215".
I will also have to read back from the file later on and convert the
escaped characters back into a unicode string.
Does anyone have any suggestions on how to go from "" to "\303\215" and
vice versa?
Perhaps something along the lines of:
>>def encode(source):
... return "".join("\% o" % ord(c) for c in source.encode(' utf8'))
...
>>def decode(encoded) :
... bytes = "".join(chr(int (c, 8)) for c in encoded.split(' \\')[1:])
... return bytes.decode('u tf8')
...
>>encode(u"" )
'\\303\\215'
>>print decode(_)

HTH
Michael
Nice one. :) If I might suggest a slight variation to handle cases
where the "encoded" string contains plain text as well as octal
escapes...
def decode(encoded) :
for octc in (c for c in re.findall(r'\\ (\d{3})', encoded)):
encoded = encoded.replace (r'\%s' % octc, chr(int(octc, 8)))
return encoded.decode( 'utf8')
This way it can handle both "\\141\\144\\14 6\\303\\215\\14 1\\144\\146"
as well as "adf\\303\\215a df".
Regards,
Jordan
err...
def decode(encoded) :
for octc in re.findall(r'\\ (\d{3})', encoded):
encoded = encoded.replace (r'\%s' % octc, chr(int(octc, 8)))
return encoded.decode( 'utf8')

Great suggestions from both of you! I came up with my "final" solution
based on them. It encodes only non-ascii and non-printables, and stays
in unicode strings for both input and output. Also, low ascii values now
encode into a 3-digit octal sequence also, so that decode can catch them
properly.

Thanks a lot,
Michael

____________

import re

def encode(source):
encoded = ""
for character in source:
if (ord(character) < 32) or (ord(character) 128):
for byte in character.encod e('utf8'):
encoded += ("\%03o" % ord(byte))
else:
encoded += character
return encoded.decode( 'utf-8')

def decode(encoded) :
decoded = encoded.encode( 'utf-8')
for octc in re.findall(r'\\ (\d{3})', decoded):
decoded = decoded.replace (r'\%s' % octc, chr(int(octc, 8)))
return decoded.decode( 'utf8')

orig = u"blablub" + chr(10)
enc = encode(orig)
dec = decode(enc)
print orig
print enc
print dec
An optimization... in decode() store matches as keys in a dict, so you
only do the string replacement once for each unique character...

def decode(encoded) :
decoded = encoded.encode( 'utf-8')
matches = {}
for octc in re.findall(r'\\ (\d{3})', decoded):
matches[octc] = None
for octc in matches:
decoded = decoded.replace (r'\%s' % octc, chr(int(octc, 8)))
return decoded.decode( 'utf8')

Untested...

Regards,
Jordan
Dec 4 '07 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
3216
by: Sverre Bakke | last post by:
Hi I am using the base_convert() function to convert numbers between binary, hexadecimal, decimal, etc... It works great, but I have problems converting numbers with .'s Like this number: 15.20 decimal will be 1111 as binary...
2
4277
by: flamingivanova | last post by:
I have several ascii files that contain '\ooo' strings which represent the octal value for a character. I want to convert these files to unicode, and I came up with the following script. But it seems to me that there must be a much simpler way to do it. Could someone more experienced suggest some improvements? I want to convert a file eg. containing: hello \326du
3
7770
by: hunterb | last post by:
I have a file which has no BOM and contains mostly single byte chars. There are numerous double byte chars (Japanese) which appear throughout. I need to take the resulting Unicode and store it in a DB and display it onscreen. No matter which way I open the file, convert it to Unicode/leave it as is or what ever, I see all single bytes ok, but double bytes become 2 seperate single bytes. Surely there is an easy way to convert these mixed...
1
3650
by: James | last post by:
this is a console program to convert ANSI to UTF8 format. Although in notepad i open the source file (which is ansi), and after running the program below, and re-open in notepad (it shows utf8 encoding), does it mean that it has been correctly converted ? Pls let me know what i have done wrong in conversion ... Module Module1
2
3128
by: ranjithkumar | last post by:
I am using mysql and have some data in my application in the latin1 charset. I have a necessity to support the utf 8 charset. Now I want to migrate the data between these two charset. The normal way I do migration is as follows: Taking a dump of the data with the currently running mysql converting the necessary parameters in the mysql settings and starting the mysql with utf8 support droping the database.
15
35496
by: jaks.maths | last post by:
How to convert negative integer to hexadecimal or octal number? Ex: -568 What is the equivalent hexadecimal and octal number??
0
5444
by: Alci | last post by:
I am getting some Korean characters data from MS SQL server. These data were submitted as UTF-8 into the database, but stored as normal varchars. So, when I getting them out of database by using Gridview +SqlDataSource, they are actually ASCII format, but I couldn't just convert the encoding of the page to get the proper UTF-8 format Korean texts. So, i modified a function from MS site, try to convert ASCII to UTF-8, as below protected...
1
4295
by: HaifaCarina | last post by:
Here's the code is used but...but still something is wrong... i need help... /*CONVERTING DECIMAL TO OCTAL*/ String inputDeci,octal=""; int deci,count=0, i, h; inputDeci = JOptionPane.showInputDialog("Enter a decimal number "); deci = Integer.parseInt(inputDeci);
0
3343
by: Terry Reedy | last post by:
A. Joseph wrote: These are number representation systems that can be applied to or used with integral, rational (numberator,denominator), and 'point' numbers. Try Wikipedia or any search engine. There are standard algorithms for converting between representations. See above. Good programmer calculators have these built in.
0
8615
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9174
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
8883
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
6534
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupr who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5874
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4376
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4629
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
2347
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2009
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.