473,545 Members | 2,006 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

UTF32 CodePoints, UTF8 Combining Chars / Surrogate Pairs, and .NET

I've spent a bit of time over the last year trying to implement RFC 3454
(Preparation of Internationaliz ed Strings, aka 'StringPrep').

This RFC is also a dependency for RFC 3491 (Internationali zed Domain Names /
IDNA) which is something that I also need to support.

The problem that I've been struggling with in .NET is that of Unicode Code
Points > 0xFFFF. These points are encoded into UTF8 using the Surrogate Pair
encoding scheme that the Unicode Spec defined in section 3.7 of the Unicode
Spec (http://www.unicode.org/book/ch03.pdf).

Related to Surrogate Pairs, are the whole set of Unicode Combining
characters.

The problem, then, is this:

When I iterate over a string using the .NET StringInfo class I get a set of
graphemes. These graphemes correctly handle the combining characters and
surrogate pairs, and end up giving me a single UTF-32 Code Point for each
grapheme.

BUT, let's say the original string had U:0x10FF1 encoded as a UTF8 surrogate
pair. This character is illegal in a particular stringprep profile.

The original string also had a combining character sequence U:301 + U:302
(for example) and the grapheme that the StringInfo class reports for this is
also U:0x10FF1.

The problem is that each of the combining characters IS legal in the
stringprep profile, but I have no way of telling if the original data was
the (illegal) UTF-32 code point, or the (legal) combining characters.

Has anyone implemented any of this stuff in .NET ?

--

Chris Mullins
Jul 21 '05 #1
2 3245
Hello Chris,

I noticed that you posted this question in
microsoft.publi c.dotnet.framew ork also. I have replied you and will follow
up on it there. If you have free time, please check that thread for this
question.

Thanks very much.

Best regards,
Yanhong Huang
Microsoft Community Support

Get Secure! ¨C www.microsoft.com/security
This posting is provided "AS IS" with no warranties, and confers no rights.

Jul 21 '05 #2
Hello Chris,

I noticed that you posted this question in
microsoft.publi c.dotnet.framew ork also. I have replied you and will follow
up on it there. If you have free time, please check that thread for this
question.

Thanks very much.

Best regards,
Yanhong Huang
Microsoft Community Support

Get Secure! ¨C www.microsoft.com/security
This posting is provided "AS IS" with no warranties, and confers no rights.

Jul 21 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
1740
by: Sagi Bashari | last post by:
Hello, I would like to know the status of the UTF8 support in MySQL 4.1. I tried to create a table using utf8 charset, and inserting hebrew text into it. it seems like it still treats this text as binary - for example the length() function returns 8 on 4 chars string, or when cretting a column using varchar(4) i can only insert 2 letters...
12
4055
by: Chris Mullins | last post by:
I'm implementing RFC 3491 in .NET, and running into a strange issue. Step 1 of RFC 3491 is performing a set of mappings dicated by tables B.1 and B.2. I'm having trouble with the following mappings though, and it seems like a shortcoming of the .NET framework: When I see Unicode value 0x10400, I'm supposed to map it to value 0x10428....
3
5260
by: Chris Mullins | last post by:
I've got a big unicode character, and i'm trying to build it into a string. The unicode character is in the range "0x10400", so it's going to require a surrogate pair. I've been through all the logic to iterate over strings that already have these pairs in them, but how do I encode this Unicode Character INTO the string? The string is...
3
7738
by: hunterb | last post by:
I have a file which has no BOM and contains mostly single byte chars. There are numerous double byte chars (Japanese) which appear throughout. I need to take the resulting Unicode and store it in a DB and display it onscreen. No matter which way I open the file, convert it to Unicode/leave it as is or what ever, I see all single bytes ok, but...
4
11493
by: H Lee | last post by:
Hi, I'm an XML newbie, and not sure if this is the appropriate newsgroup to post my question, so feel free to suggest other newgroups where I should post this message if this is the case. I'm having issues using XmlTextWriter, saving it out to a file with UTF8 encoding, and seeing "dirty", or "human unreadable" characters show up *right...
2
372
by: Chris Mullins | last post by:
I've spent a bit of time over the last year trying to implement RFC 3454 (Preparation of Internationalized Strings, aka 'StringPrep'). This RFC is also a dependency for RFC 3491 (Internationalized Domain Names / IDNA) which is something that I also need to support. The problem that I've been struggling with in .NET is that of Unicode Code...
6
2611
by: guy | last post by:
if a string contains surrogate chars (i.e. Unicode characters that consiste of more than 1 char) do functions that use an indexer or a string length into the string e.g. Mid, Len work correctly? guy
0
7664
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
7921
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7437
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
5982
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
1
5343
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
4958
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3446
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
1023
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
720
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.