not quite 1252

[Anton Vredegoor]

I'm trying to import text from an open office document (save as .sxw and
read the data from content.xml inside the sxw-archive using
elementtree and such tools).

The encoding that gives me the least problems seems to be cp1252,
however it's not completely perfect because there are still characters
in it like \93 or \94. Has anyone handled this before? I'd rather not
reinvent the wheel and start translating strings 'by hand'.

Anton

[Serge Orlov]

Anton Vredegoor wrote:

Serge Orlov wrote:

Anton Vredegoor wrote:

In fact there are a lot of printable things that haven't got a text
attribute, for example some items with tag (xxxx)s.

In my sample file I see <text:s text:c="2"/>, is that you're talking
about? Since my file is small I can say for sure this tag represents
two space characters.

Or for example in firefox:

<text:s/>
in Amsterdam
<text:s/>

So, probably yes. If it doesn't have a text attribrute if you iterate
over it using OOopy for example:

o = OOoPy (infile = fname)
c = o.read ('content.xml')
for x in c.getiterator() :
if x.text:

Then we know for sure you have recreated my other problem.

I'm tweaking a small test file and see that
<text:s/> is one space character
<text:s text:c="2"/> is two space characters
<text:s text:c="3"/> is three space characters

[Anton Vredegoor]

Martin v. Löwis wrote:

So if that is the case: What is the problem then? If you interpret
the document as cp1252, and it contains \x93 and \x94, what is
it that you don't like about that? In yet other words: what actions
are you performing, what are the results you expect to get, and
what are the results that you actually get?

Well, where do these cp1252 codes come from? The xml-file claims it's
utf-8.

I just tried out some random decodings and cp1252 seemed to work. I
don't like to have to guess this way. I think John wouldn't even allow
it :-)

Anton

[Martin v. Löwis]

Anton Vredegoor wrote:

So if that is the case: What is the problem then? If you interpret
the document as cp1252, and it contains \x93 and \x94, what is
it that you don't like about that? In yet other words: what actions
are you performing, what are the results you expect to get, and
what are the results that you actually get?
Well, where do these cp1252 codes come from? The xml-file claims it's
utf-8.

Ah. Then the document is most likely right: \x94 can very well occur
in an UTF-8 file.
I just tried out some random decodings and cp1252 seemed to work. I
don't like to have to guess this way. I think John wouldn't even allow
it :-)

Well, if the document is UTF-8, you should decode it as UTF-8, of
course.

Regards,
Martin

[Anton Vredegoor]

Martin v. Löwis wrote:

Well, if the document is UTF-8, you should decode it as UTF-8, of
course.

Thanks. This and:

http://en.wikipedia.org/wiki/UTF-8

solved my problem with understanding the encoding.

Anton

proof that I understand it now (please anyone, prove me wrong if you can):

from zipfile import ZipFile, ZIP_DEFLATED

def by80(seq):
it = iter(seq)
while it:
yield ''.join(it.next () for i in range(80))

def utfCheck(infn):
zin = ZipFile(infn, 'r', ZIP_DEFLATED)
data = zin.read('conte nt.xml').decode ('utf-8')
for line in by80(data):
print line.encode('12 52')

def test():
infn = "xxx.sxw"
utfCheck(infn)

if __name__=='__ma in__':
test()