473,233 Members | 1,543 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,233 software developers and data experts.

PHP5 and Double Byte (experts wanted)

Hi,

we have to convert a quite large ISO-application (Mysql4, PHP5) to
UTF8. We found out so far that there is no collation for MySQL which
is able to sort all character sets correctly. So this has to be done.
And there might be some problems with string functions as not all
string functions have multibyte aquivalents.

What are your experiences with this topic? Any hints?

In case you are an expert for unicode in combination with MYsql and
PHP or for sorting algorithmn of asian characters please drop me a
line at
luebbert-AT-globalpark-Dot-de. We are willing to pay for consultants.

Thanx

Dorthe Luebbert
Jul 17 '05 #1
8 2778
Dorthe Luebbert wrote:
Hi,

we have to convert a quite large ISO-application (Mysql4, PHP5) to
UTF8. We found out so far that there is no collation for MySQL which
is able to sort all character sets correctly. So this has to be done.
And there might be some problems with string functions as not all
string functions have multibyte aquivalents.

What are your experiences with this topic? Any hints?

In case you are an expert for unicode in combination with MYsql and
PHP or for sorting algorithmn of asian characters please drop me a
line at
luebbert-AT-globalpark-Dot-de. We are willing to pay for consultants.

Thanx

Dorthe Luebbert

yup, somewhat of a problem, essentially, it can't be done. Its
inpossible to map multibyte character sets to 8 byte character sets.
Jul 17 '05 #2
NSpam wrote:
Dorthe Luebbert wrote:
Hi,

we have to convert a quite large ISO-application (Mysql4, PHP5) to
UTF8. We found out so far that there is no collation for MySQL which
is able to sort all character sets correctly. So this has to be done.
And there might be some problems with string functions as not all
string functions have multibyte aquivalents.

What are your experiences with this topic? Any hints?
In case you are an expert for unicode in combination with MYsql and
PHP or for sorting algorithmn of asian characters please drop me a
line at
luebbert-AT-globalpark-Dot-de. We are willing to pay for consultants.

Thanx

Dorthe Luebbert


yup, somewhat of a problem, essentially, it can't be done. Its
inpossible to map multibyte character sets to 8 byte character sets.

Whoops, should that read "8 bit character sets"
Jul 17 '05 #3
NSpam (ch*********@gmail.com) wrote:
: Dorthe Luebbert wrote:
: > Hi,
: >
: > we have to convert a quite large ISO-application (Mysql4, PHP5) to
: > UTF8. We found out so far that there is no collation for MySQL which
: > is able to sort all character sets correctly. So this has to be done.
: > And there might be some problems with string functions as not all
: > string functions have multibyte aquivalents.
: >
: > What are your experiences with this topic? Any hints?
: >
: > In case you are an expert for unicode in combination with MYsql and
: > PHP or for sorting algorithmn of asian characters please drop me a
: > line at
: > luebbert-AT-globalpark-Dot-de. We are willing to pay for consultants.
: >
: > Thanx
: >
: > Dorthe Luebbert
: yup, somewhat of a problem, essentially, it can't be done. Its
: inpossible to map multibyte character sets to 8 [bit]byte character
sets.

Yes, but many parts can be done, just not quite in the straight forward
way.

If you view utf-8 data as 8 bit characters, then each utf-8 "character"
becomes a _unique_ string of one to three (four?) characters. Which means
that simply handling the data as 8 bit character strings can often work
correctly, as long as you don't chop the string in the middle of a
utf-character.

So lets say you want to search for the unicode character with codepoint
123,456 (what character that might be I have no idea). That value is
represented as something like a three byte string. Now use your 8 bit
string routines to search through the utf-string, but instead of using
utf-8 aware routines and looking for a character, insted you use 8 bit
routines and look for that three byte string. Your program will find it
correctly exactly the same as if you were using utf-8 aware string
routines. And so for example, if you do a search and replace then
replacing that three byte string with a new string will do exactly the
same thing as a character replace, as long as you use the correct little
substrings.

If you are manipulating data from outside the program, and the data comes
in as utf-8, like in a web form, then for many tasks you simply ignore
that fact it is utf-8 as treat it as 8 bit data and everything will work.
E.g. If the user fills in a search dialog and the data from the dialog is
utf-8, then your program simply searches for the string exactly as
received and the fact that it's utf-8 whereas you are using 8 bit string
routines will make no difference to whether you find that string in some
other utf-8 data. And when you send the data back to the client, then as
long as their browser displays it as utf-8, then they will see all the
correct data.

Sorting may not appear to work because the resulting character order
doesn't make sense, but I think you'll find that at a low level even
routines like strcmp "work" in the sense that utf-8 characters with higher
code points create 8 bits strings that sort in the same order as if you
were sorting them using the numerical code points.
--

This space not for rent.
Jul 17 '05 #4
"Malcolm Dew-Jones" <yf***@vtn1.victoria.tc.ca> wrote in message
news:42******@news.victoria.tc.ca...
So lets say you want to search for the unicode character with codepoint
123,456 (what character that might be I have no idea). That value is
represented as something like a three byte string. Now use your 8 bit
string routines to search through the utf-string, but instead of using
utf-8 aware routines and looking for a character, insted you use 8 bit
routines and look for that three byte string. Your program will find it
correctly exactly the same as if you were using utf-8 aware string
routines.


Not so. You will match the codepoint, as you say, but you will also match
codepoints that bridge UTF characters.

Let's say that the string "AAA" is a unicode sequence (in other words, it's
some codepoint) and "AAB" is also a codepoint.

Now if you have the two-character unicode string AAA followed by AAB, it
looks like AAAAAB when viewed as 8-bit bytes.

If you naively search for the codepoint AAA in this string you'll get 3
places it matches. But only the first of these is valid. The second two
matches are bogus because they bridge codepoints.
Jul 17 '05 #5
"Dorthe Luebbert" <do*************@gmx.de> wrote in message
news:27**************************@posting.google.c om...
Hi,

we have to convert a quite large ISO-application (Mysql4, PHP5) to
UTF8. We found out so far that there is no collation for MySQL which
is able to sort all character sets correctly. So this has to be done.
And there might be some problems with string functions as not all
string functions have multibyte aquivalents.

What are your experiences with this topic? Any hints?

In case you are an expert for unicode in combination with MYsql and
PHP or for sorting algorithmn of asian characters please drop me a
line at
luebbert-AT-globalpark-Dot-de. We are willing to pay for consultants.

Thanx

Dorthe Luebbert


Some guy was working on creating a PHP extension for the ICU library. I
don't know what the status is at this point. Worth googling.

If MySQL doesn't support Unicode collation, I guess there's no much you can
do. Perhaps time to consider a commercial database? I know MSSQL can handle
sorting in a number of languages. Oracle can too probably.
Jul 17 '05 #6
Dana Cartwright wrote:
"Malcolm Dew-Jones" <yf***@vtn1.victoria.tc.ca> wrote in message
news:42******@news.victoria.tc.ca...
So lets say you want to search for the unicode character with codepoint
123,456 (what character that might be I have no idea). That value is
represented as something like a three byte string. Now use your 8 bit
string routines to search through the utf-string, but instead of using
utf-8 aware routines and looking for a character, insted you use 8 bit
routines and look for that three byte string. Your program will find it
correctly exactly the same as if you were using utf-8 aware string
routines.


Not so. You will match the codepoint, as you say, but you will also match
codepoints that bridge UTF characters.


Since no UTF-8 character's byte sequence is a subsequence of any other
UTF-8 character's byte sequence, that's just not true. A bytewise search
will indeed find only correct matches; this is one of the things that
sets UTF-8 apart from pre-Unicode multibyte encodings.

-- brion vibber (brion @ pobox.com)
Jul 17 '05 #7
Chung Leong wrote:
Some guy was working on creating a PHP extension for the ICU library. I
don't know what the status is at this point. Worth googling.
Can't help much on collation currently, but I've written a partial PHP
extension wrapper for ICU's normalizer and a pure-PHP equivalent for
validation and normalization of UTF-8 input. It's under GPL license and
is bundled with MediaWiki 1.4 (www.mediawiki.org)

I ended up writing that rather than trying to wade through the full ICU
extension; I had a hard enough time trying to track it down, and didn't
want to rely on ICU and a custom PHP extension as they aren't always
available.
If MySQL doesn't support Unicode collation, I guess there's no much you can
do. Perhaps time to consider a commercial database? I know MSSQL can handle
sorting in a number of languages. Oracle can too probably.


MySQL 4.1 and higher do have UTF-8 support. I don't know how well it
works at this stage or whether the collation support is suitable for the
original poster's needs.

-- brion vibber (brion @ pobox.com)
Jul 17 '05 #8
Brion Vibber wrote:
Dana Cartwright wrote:
Not so. You will match the codepoint, as you say, but you will also
match codepoints that bridge UTF characters.


Since no UTF-8 character's byte sequence is a subsequence of any other
UTF-8 character's byte sequence, that's just not true. A bytewise search
will indeed find only correct matches; this is one of the things that
sets UTF-8 apart from pre-Unicode multibyte encodings.


Some details; if character encodings bore you, stop reading now. ;)
Having a byte substring that bridges characters is impossible because of
the way a UTF-8 byte sequence is structured. Any UTF-8 character will be
laid out into bytes like one of these sequences:

[ASCII]
[head1][tail]
[head2][tail][tail]
[head3][tail][tail][tail]

The number of tail bytes is determined by the high-order bits of the
head byte; any given head byte will always be followed by the same
number of tail bytes. Note that each category of byte is in a distinct,
non-overlapping range:

ASCII: 0x00-0x7f
tail: 0x80-0xbf
head1: 0xc0-0xdf
head2: 0xe0-0xef
head3: 0xf0-0xf7

No ASCII byte ever appears in a non-ASCII sequence, and no sequence head
byte ever appears in the tail of any other sequence. The only way for a
match to bridge characters is if you're working with corrupt data that's
not actually valid UTF-8.
Let's say that the string "AAA" is a unicode sequence (in other words, it's
some codepoint) and "AAB" is also a codepoint.

Now if you have the two-character unicode string AAA followed by AAB, it
looks like AAAAAB when viewed as 8-bit bytes.

If you naively search for the codepoint AAA in this string you'll get 3
places it matches. But only the first of these is valid. The second two
matches are bogus because they bridge codepoints.


This scenario is clearly impossible due to the distinction between head
and tail bytes; sequences cannot run together or overlap in this way.

-- brion vibber (brion @ pobox.com)
Jul 17 '05 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

11
by: neur0maniak | last post by:
Hi, I've been eager to try out PHP5, so I've dumped it on my little dev machine. It's running WinXP with IIS5. I've put the php-cgi.exe in the "mappings" page as I'm used to doing with PHP4....
17
by: Suzanne Vogel | last post by:
I'd like to convert a double to a binary representation. I can use the "&" bit operation with a bit mask to convert *non* float types to binary representations, but I can't use "&" on doubles. ...
5
by: DAVID SCHULMAN | last post by:
I've been trying to perform a calculation that has been running into an underflow (insufficient precision) problem in Microsoft Excel, which calculates using at most 15 significant digits. For this...
22
by: bq | last post by:
Hello, Two questions related to floating point support: What C compilers for the wintel (MS Windows + x86) platform are C99 compliant as far as <math.h> and <tgmath.h> are concerned? What...
3
by: RoSsIaCrIiLoIA | last post by:
I have rewrote the malloc() function of K&R2 chapter 8.7 typedef long Align; ^^^^ Here, should I write 'long', 'double' or 'long double'? I know that in my pc+compiler sizeof(long)=4,...
6
by: Steve | last post by:
Hi, I've developed a testing application that stores formatted results in a database. Recently it was requested that I add the ability to generate graphs from the raw, un formatted test results...
17
by: Terry | last post by:
I have some old VB5 functions that specify types of Long and Double that make calls to an unmanaged 3rd party DLL probably just as old. The source for the DLL is not available. I'm getting some...
3
by: xhe | last post by:
I have just upgraded my php version form php4 to php5. and I met this problem, and don't know if you know the solution. My site was written in PHP4, and most parts can be running smoothly in PHP5,...
29
by: Virtual_X | last post by:
As in IEEE754 double consist of sign bit 11 bits for exponent 52 bits for fraction i write this code to print double parts as it explained in ieee754 i want to know if the code contain any...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.