Hi,
we have to convert a quite large ISO-application (Mysql4, PHP5) to
UTF8. We found out so far that there is no collation for MySQL which
is able to sort all character sets correctly. So this has to be done.
And there might be some problems with string functions as not all
string functions have multibyte aquivalents.
What are your experiences with this topic? Any hints?
In case you are an expert for unicode in combination with MYsql and
PHP or for sorting algorithmn of asian characters please drop me a
line at
luebbert-AT-globalpark-Dot-de. We are willing to pay for consultants.
Thanx
Dorthe Luebbert 8 2791
Dorthe Luebbert wrote: Hi,
we have to convert a quite large ISO-application (Mysql4, PHP5) to UTF8. We found out so far that there is no collation for MySQL which is able to sort all character sets correctly. So this has to be done. And there might be some problems with string functions as not all string functions have multibyte aquivalents.
What are your experiences with this topic? Any hints?
In case you are an expert for unicode in combination with MYsql and PHP or for sorting algorithmn of asian characters please drop me a line at luebbert-AT-globalpark-Dot-de. We are willing to pay for consultants.
Thanx
Dorthe Luebbert
yup, somewhat of a problem, essentially, it can't be done. Its
inpossible to map multibyte character sets to 8 byte character sets.
NSpam wrote: Dorthe Luebbert wrote:
Hi,
we have to convert a quite large ISO-application (Mysql4, PHP5) to UTF8. We found out so far that there is no collation for MySQL which is able to sort all character sets correctly. So this has to be done. And there might be some problems with string functions as not all string functions have multibyte aquivalents.
What are your experiences with this topic? Any hints? In case you are an expert for unicode in combination with MYsql and PHP or for sorting algorithmn of asian characters please drop me a line at luebbert-AT-globalpark-Dot-de. We are willing to pay for consultants.
Thanx
Dorthe Luebbert
yup, somewhat of a problem, essentially, it can't be done. Its inpossible to map multibyte character sets to 8 byte character sets.
Whoops, should that read "8 bit character sets"
NSpam (ch*********@gmail.com) wrote:
: Dorthe Luebbert wrote:
: > Hi,
: >
: > we have to convert a quite large ISO-application (Mysql4, PHP5) to
: > UTF8. We found out so far that there is no collation for MySQL which
: > is able to sort all character sets correctly. So this has to be done.
: > And there might be some problems with string functions as not all
: > string functions have multibyte aquivalents.
: >
: > What are your experiences with this topic? Any hints?
: >
: > In case you are an expert for unicode in combination with MYsql and
: > PHP or for sorting algorithmn of asian characters please drop me a
: > line at
: > luebbert-AT-globalpark-Dot-de. We are willing to pay for consultants.
: >
: > Thanx
: >
: > Dorthe Luebbert
: yup, somewhat of a problem, essentially, it can't be done. Its
: inpossible to map multibyte character sets to 8 [bit]byte character
sets.
Yes, but many parts can be done, just not quite in the straight forward
way.
If you view utf-8 data as 8 bit characters, then each utf-8 "character"
becomes a _unique_ string of one to three (four?) characters. Which means
that simply handling the data as 8 bit character strings can often work
correctly, as long as you don't chop the string in the middle of a
utf-character.
So lets say you want to search for the unicode character with codepoint
123,456 (what character that might be I have no idea). That value is
represented as something like a three byte string. Now use your 8 bit
string routines to search through the utf-string, but instead of using
utf-8 aware routines and looking for a character, insted you use 8 bit
routines and look for that three byte string. Your program will find it
correctly exactly the same as if you were using utf-8 aware string
routines. And so for example, if you do a search and replace then
replacing that three byte string with a new string will do exactly the
same thing as a character replace, as long as you use the correct little
substrings.
If you are manipulating data from outside the program, and the data comes
in as utf-8, like in a web form, then for many tasks you simply ignore
that fact it is utf-8 as treat it as 8 bit data and everything will work.
E.g. If the user fills in a search dialog and the data from the dialog is
utf-8, then your program simply searches for the string exactly as
received and the fact that it's utf-8 whereas you are using 8 bit string
routines will make no difference to whether you find that string in some
other utf-8 data. And when you send the data back to the client, then as
long as their browser displays it as utf-8, then they will see all the
correct data.
Sorting may not appear to work because the resulting character order
doesn't make sense, but I think you'll find that at a low level even
routines like strcmp "work" in the sense that utf-8 characters with higher
code points create 8 bits strings that sort in the same order as if you
were sorting them using the numerical code points.
--
This space not for rent.
"Malcolm Dew-Jones" <yf***@vtn1.victoria.tc.ca> wrote in message
news:42******@news.victoria.tc.ca... So lets say you want to search for the unicode character with codepoint 123,456 (what character that might be I have no idea). That value is represented as something like a three byte string. Now use your 8 bit string routines to search through the utf-string, but instead of using utf-8 aware routines and looking for a character, insted you use 8 bit routines and look for that three byte string. Your program will find it correctly exactly the same as if you were using utf-8 aware string routines.
Not so. You will match the codepoint, as you say, but you will also match
codepoints that bridge UTF characters.
Let's say that the string "AAA" is a unicode sequence (in other words, it's
some codepoint) and "AAB" is also a codepoint.
Now if you have the two-character unicode string AAA followed by AAB, it
looks like AAAAAB when viewed as 8-bit bytes.
If you naively search for the codepoint AAA in this string you'll get 3
places it matches. But only the first of these is valid. The second two
matches are bogus because they bridge codepoints.
"Dorthe Luebbert" <do*************@gmx.de> wrote in message
news:27**************************@posting.google.c om... Hi,
we have to convert a quite large ISO-application (Mysql4, PHP5) to UTF8. We found out so far that there is no collation for MySQL which is able to sort all character sets correctly. So this has to be done. And there might be some problems with string functions as not all string functions have multibyte aquivalents.
What are your experiences with this topic? Any hints?
In case you are an expert for unicode in combination with MYsql and PHP or for sorting algorithmn of asian characters please drop me a line at luebbert-AT-globalpark-Dot-de. We are willing to pay for consultants.
Thanx
Dorthe Luebbert
Some guy was working on creating a PHP extension for the ICU library. I
don't know what the status is at this point. Worth googling.
If MySQL doesn't support Unicode collation, I guess there's no much you can
do. Perhaps time to consider a commercial database? I know MSSQL can handle
sorting in a number of languages. Oracle can too probably.
Dana Cartwright wrote: "Malcolm Dew-Jones" <yf***@vtn1.victoria.tc.ca> wrote in message news:42******@news.victoria.tc.ca...So lets say you want to search for the unicode character with codepoint 123,456 (what character that might be I have no idea). That value is represented as something like a three byte string. Now use your 8 bit string routines to search through the utf-string, but instead of using utf-8 aware routines and looking for a character, insted you use 8 bit routines and look for that three byte string. Your program will find it correctly exactly the same as if you were using utf-8 aware string routines.
Not so. You will match the codepoint, as you say, but you will also match codepoints that bridge UTF characters.
Since no UTF-8 character's byte sequence is a subsequence of any other
UTF-8 character's byte sequence, that's just not true. A bytewise search
will indeed find only correct matches; this is one of the things that
sets UTF-8 apart from pre-Unicode multibyte encodings.
-- brion vibber (brion @ pobox.com)
Chung Leong wrote: Some guy was working on creating a PHP extension for the ICU library. I don't know what the status is at this point. Worth googling.
Can't help much on collation currently, but I've written a partial PHP
extension wrapper for ICU's normalizer and a pure-PHP equivalent for
validation and normalization of UTF-8 input. It's under GPL license and
is bundled with MediaWiki 1.4 (www.mediawiki.org)
I ended up writing that rather than trying to wade through the full ICU
extension; I had a hard enough time trying to track it down, and didn't
want to rely on ICU and a custom PHP extension as they aren't always
available.
If MySQL doesn't support Unicode collation, I guess there's no much you can do. Perhaps time to consider a commercial database? I know MSSQL can handle sorting in a number of languages. Oracle can too probably.
MySQL 4.1 and higher do have UTF-8 support. I don't know how well it
works at this stage or whether the collation support is suitable for the
original poster's needs.
-- brion vibber (brion @ pobox.com)
Brion Vibber wrote: Dana Cartwright wrote: Not so. You will match the codepoint, as you say, but you will also match codepoints that bridge UTF characters.
Since no UTF-8 character's byte sequence is a subsequence of any other UTF-8 character's byte sequence, that's just not true. A bytewise search will indeed find only correct matches; this is one of the things that sets UTF-8 apart from pre-Unicode multibyte encodings.
Some details; if character encodings bore you, stop reading now. ;)
Having a byte substring that bridges characters is impossible because of
the way a UTF-8 byte sequence is structured. Any UTF-8 character will be
laid out into bytes like one of these sequences:
[ASCII]
[head1][tail]
[head2][tail][tail]
[head3][tail][tail][tail]
The number of tail bytes is determined by the high-order bits of the
head byte; any given head byte will always be followed by the same
number of tail bytes. Note that each category of byte is in a distinct,
non-overlapping range:
ASCII: 0x00-0x7f
tail: 0x80-0xbf
head1: 0xc0-0xdf
head2: 0xe0-0xef
head3: 0xf0-0xf7
No ASCII byte ever appears in a non-ASCII sequence, and no sequence head
byte ever appears in the tail of any other sequence. The only way for a
match to bridge characters is if you're working with corrupt data that's
not actually valid UTF-8. Let's say that the string "AAA" is a unicode sequence (in other words, it's some codepoint) and "AAB" is also a codepoint.
Now if you have the two-character unicode string AAA followed by AAB, it looks like AAAAAB when viewed as 8-bit bytes.
If you naively search for the codepoint AAA in this string you'll get 3 places it matches. But only the first of these is valid. The second two matches are bogus because they bridge codepoints.
This scenario is clearly impossible due to the distinction between head
and tail bytes; sequences cannot run together or overlap in this way.
-- brion vibber (brion @ pobox.com) This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: neur0maniak |
last post by:
Hi,
I've been eager to try out PHP5, so I've dumped it on my little dev
machine. It's running WinXP with IIS5. I've put the php-cgi.exe in the
"mappings" page as I'm used to doing with PHP4....
|
by: Suzanne Vogel |
last post by:
I'd like to convert a double to a binary representation. I can use the
"&" bit operation with a bit mask to convert *non* float types to binary
representations, but I can't use "&" on doubles.
...
|
by: DAVID SCHULMAN |
last post by:
I've been trying to perform a calculation that has been running into
an underflow (insufficient precision) problem in Microsoft Excel, which
calculates using at most 15 significant digits. For this...
|
by: bq |
last post by:
Hello,
Two questions related to floating point support:
What C compilers for the wintel (MS Windows + x86) platform are C99
compliant as far as <math.h> and <tgmath.h> are concerned?
What...
|
by: RoSsIaCrIiLoIA |
last post by:
I have rewrote the malloc() function of K&R2 chapter 8.7
typedef long Align;
^^^^
Here, should I write 'long', 'double' or 'long double'?
I know that in my pc+compiler sizeof(long)=4,...
|
by: Steve |
last post by:
Hi,
I've developed a testing application that stores formatted results in a
database. Recently it was requested that I add the ability to generate
graphs from the raw, un formatted test results...
|
by: Terry |
last post by:
I have some old VB5 functions that specify types of Long and Double that
make calls to an unmanaged 3rd party DLL probably just as old. The source
for the DLL is not available.
I'm getting some...
|
by: xhe |
last post by:
I have just upgraded my php version form php4 to php5. and I met this
problem, and don't know if you know the solution.
My site was written in PHP4, and most parts can be running smoothly in
PHP5,...
|
by: Virtual_X |
last post by:
As in IEEE754
double consist of
sign bit
11 bits for exponent
52 bits for fraction
i write this code to print double parts as it explained in ieee754
i want to know if the code contain any...
|
by: ryjfgjl |
last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: nemocccc |
last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
|
by: Sonnysonu |
last post by:
This is the data of csv file
1 2 3
1 2 3
1 2 3
1 2 3
2 3
2 3
3
the lengths should be different i have to store the data by column-wise with in the specific length.
suppose the i have to...
|
by: Hystou |
last post by:
There are some requirements for setting up RAID:
1. The motherboard and BIOS support RAID configuration.
2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
| |