470,874 Members | 1,400 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 470,874 developers. It's quick & easy.

UTF-8 Headache --

I have a function that (by fluke or whatever) used to work perfectly
and seems to have changed behaviour on me. The function was meant to
take a string and convert it from have characters with diacritics to
there non-diacritic equivalent. For example Drer would become Durer
-- except all of a sudden its becoming DA?rer. This is a problem :)
The function and some sample HTML are below -- any clues or hints would
be appreciated. I do see my extended character represented by the two
-- I understand what has kinda happened I just dont know how to deal
with it ...

<?php

function kill_diacritic ($word_string) {

global $dbtype;

if (empty($word_string)) {
return $word_string;
}
else {
$string_length = strlen($word_string);
for ($x=0;$x<$string_length;$x++) {
$ascii = ord(substr($word_string,$x,1));

switch($ascii){

case 224: //
case 225: //
case 226: //
case 227: //
case 228: //
case 229: //
$tmp = "a";
break;

case 231: //
$tmp = "c";
break;

case 232: //
case 233: //
case 234: //
case 235: //
$tmp = "e";
break;

case 236: //
case 237: //
case 238: //
case 239: //
$tmp = "i";
break;

case 241: //
$tmp = "n";
break;

case 240: //
case 242: //
case 243: //
case 244: //
case 245: //
case 246: //
case 248: //
$tmp = "o";
break;

case 154: //
$tmp = "s";
break;

case 249: //
case 251: //
case 252: //
$tmp = "u";
break;

case 253: //
$tmp = "y";
break;

case 158: //
$tmp = "z";
break;

case 192: //
case 193: //
case 194: //
case 195: //
case 196: //
case 197: //
$tmp = "A";
break;

/*
// Oracle represents as a ?. Not sure what MySQL will
// Do with this character. Pretty sure nobody will ever
// search using it but its there regardless.

case 198: //
$tmp = "?";
break;
*/

case 200: //
case 201: //
case 202: //
case 203: //
$tmp = "E";
break;

case 208: //
$tmp = "D";
break;

case 204: //
case 205: //
case 206: //
case 207: //
$tmp = "I";
break;

case 209: //
$tmp = "N";
break;

case 210: //
case 211: //
case 212: //
case 213: //
case 214: //
case 216: //
$tmp = "O";
break;

case 138: //
$tmp = "S";
break;

case 217: //
case 218: //
case 219: //
case 220: //
$tmp = "U";
break;

case 159: //
case 221: //
$tmp = "Y";
break;
} // switch

if (!empty($tmp) or $tmp=="_") {
$word_string = str_replace(chr($ascii),$tmp,$word_string);
$tmp="";
}
} // for
}

return $word_string;
}
?>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>

<head>
<title>UTF8 Testing</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<link rel='stylesheet' href='styles/default/stylesheet.css'
type='text/css'>
<script type="text/JavaScript" src="javascript.js"></script>

</head>

<body>

<form method="GET" action="index.php">
<input type="text" name="s" size="10" value="<?php echo
$_GET['s']; ?>">
<input type="submit" value="Search">
</form>

<p>
<?php echo $_GET['s']; ?>
</p>

<p>
<?php echo kill_diacritic($_GET['s']); ?>
</p>

</body>

</html>

Jan 10 '06 #1
4 1662
James wrote:
I have a function that (by fluke or whatever) used to work perfectly
and seems to have changed behaviour on me. The function was meant to
take a string and convert it from have characters with diacritics to
there non-diacritic equivalent. For example D?rer would become Durer
-- except all of a sudden its becoming DA?rer. This is a problem :)
The function and some sample HTML are below -- any clues or hints would
be appreciated. I do see my extended character represented by the two
-- I understand what has kinda happened I just dont know how to deal
with it ...


UTF-8 is a variable-length encoding. Accented characters are stored
with two bytes so your code is not going to work. Instead of looping
through the string manually, use the strtr() with an array as the
translation table. Figuring out what the letters are in UTF-8 encoding
won't be fun though.

In general it's best to avoid using Unicode unless you're actually
doing multi-lingual stuff.

Jan 10 '06 #2
James wrote:
I have a function that (by fluke or whatever) used to work perfectly
and seems to have changed behaviour on me. The function was meant to
take a string and convert it from have characters with diacritics to
there non-diacritic equivalent. For example Drer would become Durer
-- except all of a sudden its becoming DA?rer. This is a problem :)
The function and some sample HTML are below -- any clues or hints would
be appreciated. I do see my extended character represented by the two
-- I understand what has kinda happened I just dont know how to deal
with it ...

<?php

function kill_diacritic ($word_string) {

<snip>
Imagine this:

function translate($from_portuguese) {
if ($from_portuguese == 'bom dia') return 'good morning';
}

and now do

echo translate('bonjour');
You won't expect that to work, will you? :)

For the same reason that you can't accept French when you expect
Portuguese, accepting UTF-8 when you're expecting iso-8859-1 will not
work:
All single-byte utf-8 characters are < 128;
when a utf-8 character starts with 128 or greater you need two or more
bytes to identify the character specified.

http://www.w3.org/TR/html4/interact/...accept-charset
The default value for this attribute is the reserved string
"UNKNOWN". User agents may interpret this value as the character
encoding that was used to transmit the document containing this
FORM element.

So, your web page was sent with utf-8 encoding (did you also configure
your server for utf-8, or did you simply add the meta tag?), but
there's no indication for what character sets you accept in forms.
Maybe not all browsers interpret "UNKNOWN" as what is specified in the
META tag, or even in a Content-Type HTTP header.

Further reading:
http://www.w3.org/International/tuto...rial-char-enc/

--
Mail to my "From:" address is readable by all at http://www.dodgeit.com/
== ** ## !! ------------------------------------------------ !! ## ** ==
TEXT-ONLY mail to the whole "Reply-To:" address ("My Name" <my@address>)
may bypass my spam filter. If it does, I may reply from another address!
Jan 10 '06 #3
Pedro Graca wrote:
<snip>
For the same reason that you can't accept French when you expect
Portuguese, accepting UTF-8 when you're expecting iso-8859-1 will not
work:

<snip>

Good explanation:-) Glad that you're back here.

--
<?php echo 'Just another PHP saint'; ?>
Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/

Jan 11 '06 #4
R. Rajesh Jeba Anbiah wrote:
Pedro Graca wrote:
For the same reason that you can't accept French when you expect
Portuguese, accepting UTF-8 when you're expecting iso-8859-1 will not
work:
Good explanation:-)

Thank you. The rest of the post was a mess though :)
Glad that you're back here.

Thank you very much. It's good to be back.

--
Mail to my "From:" address is readable by all at http://www.dodgeit.com/
== ** ## !! ------------------------------------------------ !! ## ** ==
TEXT-ONLY mail to the whole "Reply-To:" address ("My Name" <my@address>)
may bypass my spam filter. If it does, I may reply from another address!
Jan 11 '06 #5

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

9 posts views Thread by lawrence | last post: by
4 posts views Thread by Alban Hertroys | last post: by
38 posts views Thread by Haines Brown | last post: by
6 posts views Thread by jmgonet | last post: by
6 posts views Thread by archana | last post: by
7 posts views Thread by Jimmy Shaw | last post: by
1 post views Thread by sheldon.regular | last post: by
23 posts views Thread by Allan Ebdrup | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.