473,396 Members | 1,852 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,396 software developers and data experts.

UTF-8 Headache --

I have a function that (by fluke or whatever) used to work perfectly
and seems to have changed behaviour on me. The function was meant to
take a string and convert it from have characters with diacritics to
there non-diacritic equivalent. For example Dürer would become Durer
-- except all of a sudden its becoming DA?rer. This is a problem :)
The function and some sample HTML are below -- any clues or hints would
be appreciated. I do see my extended character represented by the two
-- I understand what has kinda happened I just dont know how to deal
with it ...

<?php

function kill_diacritic ($word_string) {

global $dbtype;

if (empty($word_string)) {
return $word_string;
}
else {
$string_length = strlen($word_string);
for ($x=0;$x<$string_length;$x++) {
$ascii = ord(substr($word_string,$x,1));

switch($ascii){

case 224: // à
case 225: // á
case 226: // â
case 227: // ã
case 228: // ä
case 229: // å
$tmp = "a";
break;

case 231: // ç
$tmp = "c";
break;

case 232: // è
case 233: // é
case 234: // ê
case 235: // ë
$tmp = "e";
break;

case 236: // ì
case 237: // í
case 238: // î
case 239: // ï
$tmp = "i";
break;

case 241: // ñ
$tmp = "n";
break;

case 240: // ð
case 242: // ò
case 243: // ó
case 244: // ô
case 245: // õ
case 246: // ö
case 248: // ø
$tmp = "o";
break;

case 154: // š
$tmp = "s";
break;

case 249: // ù
case 251: // û
case 252: // ü
$tmp = "u";
break;

case 253: // ý
$tmp = "y";
break;

case 158: // ž
$tmp = "z";
break;

case 192: // À
case 193: // Á
case 194: // Â
case 195: // Ã
case 196: // Ä
case 197: // Å
$tmp = "A";
break;

/*
// Oracle represents Æ as a ?. Not sure what MySQL will
// Do with this character. Pretty sure nobody will ever
// search using it but its there regardless.

case 198: // Æ
$tmp = "?";
break;
*/

case 200: // È
case 201: // É
case 202: // Ê
case 203: // Ë
$tmp = "E";
break;

case 208: // Ð
$tmp = "D";
break;

case 204: // Ì
case 205: // Í
case 206: // Î
case 207: // Ï
$tmp = "I";
break;

case 209: // Ñ
$tmp = "N";
break;

case 210: // Ò
case 211: // Ó
case 212: // Ô
case 213: // Õ
case 214: // Ö
case 216: // Ø
$tmp = "O";
break;

case 138: // Š
$tmp = "S";
break;

case 217: // Ù
case 218: // Ú
case 219: // Û
case 220: // Ü
$tmp = "U";
break;

case 159: // Ÿ
case 221: // Ý
$tmp = "Y";
break;
} // switch

if (!empty($tmp) or $tmp=="_") {
$word_string = str_replace(chr($ascii),$tmp,$word_string);
$tmp="";
}
} // for
}

return $word_string;
}
?>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>

<head>
<title>UTF8 Testing</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<link rel='stylesheet' href='styles/default/stylesheet.css'
type='text/css'>
<script type="text/JavaScript" src="javascript.js"></script>

</head>

<body>

<form method="GET" action="index.php">
<input type="text" name="s" size="10" value="<?php echo
$_GET['s']; ?>">
<input type="submit" value="Search">
</form>

<p>
<?php echo $_GET['s']; ?>
</p>

<p>
<?php echo kill_diacritic($_GET['s']); ?>
</p>

</body>

</html>

Jan 10 '06 #1
4 1759
James wrote:
I have a function that (by fluke or whatever) used to work perfectly
and seems to have changed behaviour on me. The function was meant to
take a string and convert it from have characters with diacritics to
there non-diacritic equivalent. For example D?rer would become Durer
-- except all of a sudden its becoming DA?rer. This is a problem :)
The function and some sample HTML are below -- any clues or hints would
be appreciated. I do see my extended character represented by the two
-- I understand what has kinda happened I just dont know how to deal
with it ...


UTF-8 is a variable-length encoding. Accented characters are stored
with two bytes so your code is not going to work. Instead of looping
through the string manually, use the strtr() with an array as the
translation table. Figuring out what the letters are in UTF-8 encoding
won't be fun though.

In general it's best to avoid using Unicode unless you're actually
doing multi-lingual stuff.

Jan 10 '06 #2
James wrote:
I have a function that (by fluke or whatever) used to work perfectly
and seems to have changed behaviour on me. The function was meant to
take a string and convert it from have characters with diacritics to
there non-diacritic equivalent. For example Dürer would become Durer
-- except all of a sudden its becoming DA?rer. This is a problem :)
The function and some sample HTML are below -- any clues or hints would
be appreciated. I do see my extended character represented by the two
-- I understand what has kinda happened I just dont know how to deal
with it ...

<?php

function kill_diacritic ($word_string) {

<snip>
Imagine this:

function translate($from_portuguese) {
if ($from_portuguese == 'bom dia') return 'good morning';
}

and now do

echo translate('bonjour');
You won't expect that to work, will you? :)

For the same reason that you can't accept French when you expect
Portuguese, accepting UTF-8 when you're expecting iso-8859-1 will not
work:
All single-byte utf-8 characters are < 128;
when a utf-8 character starts with 128 or greater you need two or more
bytes to identify the character specified.

http://www.w3.org/TR/html4/interact/...accept-charset
The default value for this attribute is the reserved string
"UNKNOWN". User agents may interpret this value as the character
encoding that was used to transmit the document containing this
FORM element.

So, your web page was sent with utf-8 encoding (did you also configure
your server for utf-8, or did you simply add the meta tag?), but
there's no indication for what character sets you accept in forms.
Maybe not all browsers interpret "UNKNOWN" as what is specified in the
META tag, or even in a Content-Type HTTP header.

Further reading:
http://www.w3.org/International/tuto...rial-char-enc/

--
Mail to my "From:" address is readable by all at http://www.dodgeit.com/
== ** ## !! ------------------------------------------------ !! ## ** ==
TEXT-ONLY mail to the whole "Reply-To:" address ("My Name" <my@address>)
may bypass my spam filter. If it does, I may reply from another address!
Jan 10 '06 #3
Pedro Graca wrote:
<snip>
For the same reason that you can't accept French when you expect
Portuguese, accepting UTF-8 when you're expecting iso-8859-1 will not
work:

<snip>

Good explanation:-) Glad that you're back here.

--
<?php echo 'Just another PHP saint'; ?>
Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/

Jan 11 '06 #4
R. Rajesh Jeba Anbiah wrote:
Pedro Graca wrote:
For the same reason that you can't accept French when you expect
Portuguese, accepting UTF-8 when you're expecting iso-8859-1 will not
work:
Good explanation:-)

Thank you. The rest of the post was a mess though :)
Glad that you're back here.

Thank you very much. It's good to be back.

--
Mail to my "From:" address is readable by all at http://www.dodgeit.com/
== ** ## !! ------------------------------------------------ !! ## ** ==
TEXT-ONLY mail to the whole "Reply-To:" address ("My Name" <my@address>)
may bypass my spam filter. If it does, I may reply from another address!
Jan 11 '06 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

9
by: lawrence | last post by:
Someone on www.php.net suggested using a seems_utf8() method to test text for UTF-8 character encoding but didn't specify how to write such a method. Can anyone suggest a test that might work?...
4
by: Alban Hertroys | last post by:
Another python/psycopg question, for which the solution is probably quite simple; I just don't know where to look. I have a query that inserts data originating from an utf-8 encoded XML file....
12
by: Mike Dee | last post by:
A very very basic UTF-8 question that's driving me nuts: If I have this in the beginning of my Python script in Linux: #!/usr/bin/env python # -*- coding: UTF-8 -*- should I - or should I...
38
by: Haines Brown | last post by:
I'm having trouble finding the character entity for the French abbreviation for "number" (capital N followed by a small supercript o, period). My references are not listing it. Where would I...
6
by: jmgonet | last post by:
Hello everybody, I'm having troubles loading a Xml string encoded in UTF-8. If I try this code: ------------------------------ XmlDocument doc=new XmlDocument(); String s="<?xml...
6
by: archana | last post by:
Hi all, can someone tell me difference between unicode and utf 8 or utf 18 and which one is supporting more character set. whic i should use to support character ucs-2. I want to use ucs-2...
7
by: Jimmy Shaw | last post by:
Hi everybody, Is there any SIMPLE way to convert from UTF-16 to UTF-32? I may be mixed up, but is it possible that all UTF-16 "code points" that are 16 bits long appear just the same in UTF-32,...
1
by: sheldon.regular | last post by:
I am new to unicode so please bear with my stupidity. I am doing the following in a Python IDE called Wing with Python 23. äöü äöü '\xc3\xa4\xc3\xb6\xc3\xbc' u'\xe4\xf6\xfc'...
10
by: Jed | last post by:
I have a form that needs to handle international characters withing the UTF-8 character set. I have tried all the recommended strategies for getting utf-8 characters from form input to email...
23
by: Allan Ebdrup | last post by:
I hava an ajax web application where i hvae problems with UTF-8 encoding oc chineese chars. My Ajax webapplication runs in a HTML page that is UTF-8 Encoded. I copy and paste some chineese chars...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.