question about php script handling/displaying UTF8 email

[chris_fieldhouse]

Hi,

I'm almost done with a php driven email filter and automated forwarder,
I've tested it out with various emails and ironed out plain text and
html.
But this final item has me stumped.

When processing an email which contains UTF8 encoded characters, I
can't work out how to detect the presence of the UTF8 characters, so I
get =E2=80=99 displayed instead of a '.
And when forwarded, the =E2=80=99 is sent as plain text losing the
information that it is a utf8 encoded character as opposed to plain
text.

This is the last item I need to resolve.

I tried looking up/searching on:
utf8_decode
utf8_encode
mb_convert_encoding (actually I get function not found when I try it in
the script)
imap_utf8

but can't work it out.
I want to do either of the following
1) convert the utf8 encoded characters to closest local characters or
html codes
or
2) inject headers when the email is forwarded so that the recipents
email client will recognise correctly the presence of the utf8
characters.

Also, what is the easiest way to detect the presence of the utf8
characters, is there a header record I can search for?

Any help would be appreciated as this is furstrating me.

Chris.

[chris_fieldhouse]

followup,

I enabled the mbstring library in my php.ini, so I could test with the
mb_convert_encoding function, but that still leaves utf8 encoded
characters in the text body.

I tried
$body = mb_convert_encoding($body, 'HTML-ENTITIES', "UTF-8");
based on an example at www.php.net, but to no avail.

Any ideas?

Chris

[Andrew]

ch**************@hotmail.com wrote:

Hi,

I'm almost done with a php driven email filter and automated forwarder,
I've tested it out with various emails and ironed out plain text and
html.
But this final item has me stumped.

When processing an email which contains UTF8 encoded characters, I
can't work out how to detect the presence of the UTF8 characters, so I
get =E2=80=99 displayed instead of a '.
And when forwarded, the =E2=80=99 is sent as plain text losing the
information that it is a utf8 encoded character as opposed to plain
text.

Here's a function I use for utf8 detection (I think I grabbed it from
the manual somewhere):

/************************************************** ***********/
/* Returns TRUE if a string is UTF-8. */
/* Returns FALSE if a string is not UTF-8. */
/* Compatible with 31-bit encoding scheme of Unicode 3.x */
/************************************************** ***********/

function seems_utf8 ($Str) {
for ($i=0; $i<strlen($Str); $i++) {
if (ord($Str[$i]) < 0x80) continue; # 0bbbbbbb
elseif ((ord($Str[$i]) & 0xE0) == 0xC0) $n=1; # 110bbbbb
elseif ((ord($Str[$i]) & 0xF0) == 0xE0) $n=2; # 1110bbbb
elseif ((ord($Str[$i]) & 0xF8) == 0xF0) $n=3; # 11110bbb
elseif ((ord($Str[$i]) & 0xFC) == 0xF8) $n=4; # 111110bb
elseif ((ord($Str[$i]) & 0xFE) == 0xFC) $n=5; # 1111110b
else return false; # Does not match any model
for ($j=0; $j<$n; $j++) { # n bytes matching 10bbbbbb follow ?
if ((++$i == strlen($Str)) || ((ord($Str[$i]) & 0xC0) != 0x80))
return false;
}
}
return true;
}

Then to decode:

/************************************************** ***********/
/* utf8_encode will encode a string that is already encoded! */
/* This means that everytime you utf8_encode a string it */
/* will grow and grow exponentially! */
/* Use this function instead of utf8_encode to check if a */
/* string is already encoded before encoding. */
/* Needs the seems_utf8 function. */
/************************************************** ***********/

function utf8_ensure ($str) {
return seems_utf8($str)? $str: utf8_encode($str);
}

--
Andrew @ Rockface
np: (Winamp is not active ;-)
www.rockface-records.co.uk

[finalwebsites.com]

I have a similar problem:

I have a form where I didn't know what kind of languages is used by
visitor. I think the best way is to convert everything to utf-8 (if you
can). In my case this is the best solution, also if you sore the data
inside mysql.

Just use overall utf-8 encoding (forms, pages, email scripts)

--

gr. Olaf
PHP classes and scripts: http://www.finalwebsites.com

[chris_fieldhouse]

Yes I am storing it in a MySQL database,
I wonder if it would be safer just to store the body text in the DB as
a 'blob', cos I wonder just how messed up the text gets if trying to
store utf-8 as a text field.

I'll try Andrew's detection/encode functions as well as using a blob
field and report back.

Thanks,
Chris