473,324 Members | 2,002 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,324 software developers and data experts.

Unicode and MoinMoin

Dear all,

My lab has been using a Movable Type blog for internal communication
and announcement for a couple of years, but we've now seen the light
and I've set up a MoinMoin wiki. Everything's installed beautifully, so
I'm writing scripts to export all our Movable Type blog posts to wiki
pages. So far so good.

The only issue I'm having relates to Unicode. MoinMoin and python are
pretty unforgiving about files that contain Unicode characters that
aren't included in the coding properly. I've spent hours reading about
Unicode, and playing with different encoding/decoding commands, but at
this point, I just want a hacky solution that will ignore the
improperly coded characters or replace them with placeholders.

Can anyone recommend a simple surefire Unix/Python/Perl command that
will help me avoid errors like the one below? Any suggestions would be
hugely appreciated.

Thank you very much for your time,

Yours,
Greg

----

'utf8' codec can't decode byte 0x96 in position 4910: unexpected code
byte

* args = ('utf8', 'AUTHOR: blahblah\n\nTITLE: Reading Course
Readings... G. A. \x96 For references see blahblah.\n\n\n-----\n\n',
4910, 4911, 'unexpected code byte')
* encoding = 'utf8'
* end = 4911
* object = 'AUTHOR: blahblah\n\nTITLE: Reading Course Readings...
G. A. \x96 For references see blahblah.\n\n\n-----\n\n'
* reason = 'unexpected code byte'
* start = 4910

Feb 27 '06 #1
2 2515
Greg:
The only issue I'm having relates to Unicode. MoinMoin and python are
pretty unforgiving about files that contain Unicode characters that
aren't included in the coding properly. I've spent hours reading about
Unicode, and playing with different encoding/decoding commands, but at
this point, I just want a hacky solution that will ignore the
improperly coded characters or replace them with placeholders.


Call the codec with the errors argument set to "ignore" or "replace".
unicode('AUTHOR: blahblah\n\nTITLE: Reading Course Readings... G. A. \x96 For references see blahblah.\n\n\n-----\n\n', 'utf8')
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
File "c:\python24\lib\encodings\utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x96 in position 58:
unexpected code byte unicode('AUTHOR: blahblah\n\nTITLE: Reading Course Readings... G.

A. \x96 For references see blahblah.\n\n\n-----\n\n', 'utf8', 'replace')
u'AUTHOR: blahblah\n\nTITLE: Reading Course Readings... G. A. \ufffd For
references see blahblah.\n\n\n-----\n\n'

BTW, its probably in Windows-1252 where it would be a dash.
Depending on your context it may pay to handle the exception instead of
using "replace" and attempt interpreting as Windows-1252.

Neil
Feb 27 '06 #2
Neil Hodgson wrote:
The only issue I'm having relates to Unicode. MoinMoin and python are
pretty unforgiving about files that contain Unicode characters that
aren't included in the coding properly. I've spent hours reading about
Unicode, and playing with different encoding/decoding commands, but at
this point, I just want a hacky solution that will ignore the
improperly coded characters or replace them with placeholders.


Call the codec with the errors argument set to "ignore" or "replace".
>>> unicode('AUTHOR: blahblah\n\nTITLE: Reading Course Readings... G. A. \x96 For references see blahblah.\n\n\n-----\n\n', 'utf8')
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
File "c:\python24\lib\encodings\utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x96 in position 58:
unexpected code byte >>> unicode('AUTHOR: blahblah\n\nTITLE: Reading Course Readings... G.

A. \x96 For references see blahblah.\n\n\n-----\n\n', 'utf8', 'replace')
u'AUTHOR: blahblah\n\nTITLE: Reading Course Readings... G. A. \ufffd For
references see blahblah.\n\n\n-----\n\n'

BTW, its probably in Windows-1252 where it would be a dash.
Depending on your context it may pay to handle the exception instead of
using "replace" and attempt interpreting as Windows-1252.


here's one way to explicitly deal with 1252 gremlins:

http://effbot.org/zone/unicode-gremlins.htm

</F>

Feb 27 '06 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: asdf sdf | last post by:
ActiveState python 2.3.2 Win2k Pro SP4 Apache 2.0 browsing all of http://moinmaster.wikiwikiweb.de/HelpForBeginners in vain. Running http://localhost/mywiki?test yields: MoinMoin CGI...
0
by: asdf sdf | last post by:
i have an install issue with moinmoin on win2k. i've tried the mailing list, but it appears to be a very low traffic, low subscriber list. can i get a reality check? is anyone using moinmoin in...
6
by: Ara.T.Howard | last post by:
hi- i know nada about python so please forgive me if this is way off base. i'm trying to fix a bug in MoinMoin whereby WordsWithTwoCapsInARowLike ^^ ^^ ^^
7
by: Mark | last post by:
Hi, I have Moinmoin 1.3.4 installed and working on Linux RHEL3.0. However, all screen elements are lined up on the left hand side. How can I get it displayed like the wiki at: ...
1
by: funny_leech | last post by:
Please tell me if there's a better group to post this in-- I've been trying to get a MoinMoin installation up-and-running for the last two days. I've installed the latest MoinMoin version,...
0
by: Marcus | last post by:
Apologies if this seems like it's off-topic, but since moinmoin is written in Python, I am hoping to solicit some good thoughts in this group. The problem: I'm setting up a wiki farm (small...
1
by: Daniel Klein | last post by:
I have a pressing need to get a wiki up and running in a fairly short timeframe. I did some investigations and the Python MoinMoin wiki seemed to be the best choice for me based on simplicity, the...
2
by: kyosohma | last post by:
Hi, I am working on a new site about Python to help document some of the modules that don't have much for documentation. Anyway, I chose to use MoinMoin because it has very easy-to-use code...
1
by: John [H2O] | last post by:
I've installed MoinMoin easily enough, but i have two problems now. 1) Some pages, namely the initial FrontPage, fail to load completely.... 2) It seems after logging in, if I navigate away...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.