472,331 Members | 2,001 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,331 software developers and data experts.

newbie: how do I test a byte string?

How do I test a byte string in Python? I want to manually convert (no libraries or functions) a UTF-8 string into UTF-16.

My basic solution is to read from the stream some number of UTF-8 bytes, convert them into codepoints, then convert those codepoints into UTF-16 bytes. I want to code this myself, but I don't understand how to test the actual byte sequence.

Let's say I use the following code to ensure I have a UTF-8 encoding (from Evan Jones' Scratch Pad: http://evanjones.ca/python-utf8.html)

Expand|Select|Wrap|Line Numbers
  1. s = "hello normal string"
  2. u = unicode( s, "utf-8" )
  3. backToBytes = u.encode( "utf-8" )
  4.  
Now, I need to test the lead byte of the sequence for each character in "backToBytes", right? Is there a function that does this? Any help would be appreciated.
Jul 30 '08 #1
1 1916
I guess I get to solve my own thread (thanks again to the Natural Language Toolkit's online tutorial). The function repr() appears to give me what I need:

Expand|Select|Wrap|Line Numbers
  1. line = u'\u0144'
  2. line_utf = line.encode('utf8')
  3.  
  4. print 'line = ', line_utf
  5. print 'line repr = ', repr(line_utf)
  6.  
Output:
line = ń
line repr = '\xc5\x84'

It's the '\xc5\x84' part that I needed.
Jul 31 '08 #2

Sign in to post your reply or Sign up for a free account.

Similar topics

9
by: lawrence | last post by:
Someone on www.php.net suggested using a seems_utf8() method to test text for UTF-8 character encoding but didn't specify how to write such a...
12
by: Nollie | last post by:
I need to write a couple of my own string manipulation routines (e.g. a strcpy() alternative that returns the number of chars copied). I've started...
14
by: ThazKool | last post by:
I want to see if this code works the way it should on a Big-Endian system. Also if anyone has any ideas on how determine this at compile-time so...
6
by: tchaiket | last post by:
Hey all, I'm using NUNIT to test our classes. However, I don't want to hard code test data into the NUNIT classes. For example, to test adding...
4
by: Spam Catcher | last post by:
Anyone know of any form upload utilities which will allow me to upload files to an ASP.NET page without the need of a web browser? I like to test...
11
by: itgetsharder | last post by:
can anyone help me? code errors! -------------------------------------------------------------------------------- im creating a code for a...
11
by: new4cprog | last post by:
I need your guidance as to how to approach this question. I don't even know where to begin. It seems a bit general for me. Thanks alot. Devise a...
0
by: Madmartigan | last post by:
Hi I'm a newbie to C# and have been instructed to create a Hangman game in SharpDevelop. I don't want the answer to the full code, just some help...
2
by: Damfino | last post by:
Hi all, Newbie question here wrt defining a class that will work on bits read from a binary file. How would you go about doing it? As an example...
0
by: tammygombez | last post by:
Hey fellow JavaFX developers, I'm currently working on a project that involves using a ComboBox in JavaFX, and I've run into a bit of an issue....
0
by: tammygombez | last post by:
Hey everyone! I've been researching gaming laptops lately, and I must say, they can get pretty expensive. However, I've come across some great...
0
by: concettolabs | last post by:
In today's business world, businesses are increasingly turning to PowerApps to develop custom business applications. PowerApps is a powerful tool...
0
better678
by: better678 | last post by:
Question: Discuss your understanding of the Java platform. Is the statement "Java is interpreted" correct? Answer: Java is an object-oriented...
0
by: Kemmylinns12 | last post by:
Blockchain technology has emerged as a transformative force in the business world, offering unprecedented opportunities for innovation and...
0
by: CD Tom | last post by:
This happens in runtime 2013 and 2016. When a report is run and then closed a toolbar shows up and the only way to get it to go away is to right...
0
by: CD Tom | last post by:
This only shows up in access runtime. When a user select a report from my report menu when they close the report they get a menu I've called Add-ins...
0
by: antdb | last post by:
Ⅰ. Advantage of AntDB: hyper-convergence + streaming processing engine In the overall architecture, a new "hyper-convergence" concept was...
0
by: Matthew3360 | last post by:
Hi there. I have been struggling to find out how to use a variable as my location in my header redirect function. Here is my code. ...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.