473,322 Members | 1,510 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,322 software developers and data experts.

Unicode is driving me nuts!

I am trying to parse a huge Chinese corpus uing
python, and I am having a hard time handling the
Chinese characters.

I need to get some particular Chinese characters that
meet a certain standard one by one from the corpus.

Before I parse a sentence and try to locate the
character, I unicode the whole string I read in like
so:

str = unicode(raw_str, myencoding)

I used 'gbk' and 'cp936' encoding for example.

This works just fine with a small sample Chinese
document.

But when I attempted to run the script on the entire
corpus, I get the typical "incomplete multibyte
sequence error" or "UnicodeEncodeError: 'ascii' codec
can't encode characters in position 0-23: ordinal not
in range(128)"

I am at my wit's end, so frustrated at handling
non-ascii texts.

Any hint would be highly appreciated.

__________________________________________________ _______
Do You Yahoo!?
ÍêÈ«Ãâ·ÑµÄÑÅ»¢µçÓÊ£¬ÂíÉÏ×¢²á»ñÔù¶îÍâ60Õ×ÍøÂç´æ´¢¿Õ ¼ä
http://cn.rd.yahoo.com/mail_cn/tag/?...mail.yahoo.com

Jul 18 '05 #1
0 915

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
by: Keiron Waites | last post by:
Please see the problem in action here: http://www.leadbullet.biz/contact.php If you mouse over the fields, you will see that text is shown on the right. The text makes the other fields move when...
1
by: Paul | last post by:
Hi, I am extending an existing MFC app to use Unicode (for a Japanese version of the interface elements). The app's purpose is to control a peripheral device through the serial port, and the...
12
by: Marty | last post by:
It seems all of the sudden that user controls that contain images are referencing image sources relative to the document that I drop the control on. This obviously does not work beacuase the...
0
by: Simon Harris | last post by:
Ok, this really is driving me nuts!!! :( 'All' I am trying to do is get the value of a named element. My XML doc is: <?xml version="1.0" encoding="utf-16" standalone="yes" ?> - <Page>...
4
by: trond | last post by:
Hello all, Before I start I'd like to point out that I am a complete novice when it comes to asp.net - My background is in network and operating systems, and although I have been doing a bit of...
2
by: mitsura | last post by:
Hi, I need to read a simle XML file. For this I use the SAX parser. So far so good. The XML file consist out of number of "Service" object with each object a set of attributes. I read...
5
by: fidtz | last post by:
The code: import codecs udlASCII = file("c:\\temp\\CSVDB.udl",'r') udlUNI = codecs.open("c:\\temp\\CSVDB2.udl",'w',"utf_16") udlUNI.write(udlASCII.read()) udlUNI.close()
4
by: mattlightbourn | last post by:
Hi all, I have a problem which has been driving me nuts. Crosstab queries! I have a database witch a few different tables to do with garment manufacturing. I have a table for a client...
3
by: DuncanIdaho | last post by:
Hello experts IE 7.0.5730.11 Opera 9.27 Firefox 2.0.0.14 This problem only occurs in Opera and Firefox (amazing, IE does something right, or maybe not) Anyway, the problem is that when I...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.