html_entity_decode and unicode cyrillic characters

[Markus Ernst]

Hi

I have a string such as

Добро"

that shows the cyrillic word "?????" in the browser. Now I played around
with lots of examples and contributed functions in the manual
(html_entity_decode, strtr, convert_cyr_string ...) but was not able to
decode these entities. As browsers displays them correctly on my PC and Mac
I think I am missing something...

Somebody can point me to the right direction?

(The problem is: A user posts russian text in an application. The text
arrives at the server encoded like that. Now for security reasons the output
is converted with htmlentities(). So I have to decode the russian entities
before re-encoding them.)

Thanks
Markus

[Daniel Tryba]

Markus Ernst <derernst@no#sp#amgmx.ch> wrote:

I have a string such as

Добро"

that shows the cyrillic word "?????" in the browser. Now I played around
with lots of examples and contributed functions in the manual
(html_entity_decode, strtr, convert_cyr_string ...) but was not able to
decode these entities. As browsers displays them correctly on my PC and Mac
I think I am missing something...
Decode them into what?
(The problem is: A user posts russian text in an application. The text
arrives at the server encoded like that. Now for security reasons the output
is converted with htmlentities(). So I have to decode the russian entities
before re-encoding them.)

So don't use htmlentities (I can't imagine any use of it anyway), use
htmlspecialchars instead.

[Markus Ernst]

Daniel Tryba wrote:

Markus Ernst <derernst@no#sp#amgmx.ch> wrote:

I have a string such as

Добро"

that shows the cyrillic word "?????" in the browser. Now I played
around with lots of examples and contributed functions in the manual
(html_entity_decode, strtr, convert_cyr_string ...) but was not able
to decode these entities. As browsers displays them correctly on my
PC and Mac I think I am missing something...

Decode them into what?

Into anything that does not contain an ampersand or other special character,
and can be properly displayed or decoded for display.

(The problem is: A user posts russian text in an application. The
text arrives at the server encoded like that. Now for security
reasons the output is converted with htmlentities(). So I have to
decode the russian entities before re-encoding them.)

So don't use htmlentities (I can't imagine any use of it anyway), use
htmlspecialchars instead.

That does not make any difference, as both convert Д into
&amp;#1044; - which is exactly the problem.

--
Markus

[R. Rajesh Jeba Anbiah]

Markus Ernst wrote:

I have a string such as

Добро"

that shows the cyrillic word "?????" in the browser. Now I played around with lots of examples and contributed functions in the manual
(html_entity_decode, strtr, convert_cyr_string ...) but was not able to decode these entities. As browsers displays them correctly on my PC and Mac I think I am missing something...

Somebody can point me to the right direction?

http://in2.php.net/utf8_encode#49336

--
<?php echo 'Just another PHP saint'; ?>
Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/

[Daniel Tryba]

Markus Ernst <derernst@no#sp#amgmx.ch> wrote:

I have a string such as

Добро"

Decode them into what?

Into anything that does not contain an ampersand or other special character,
and can be properly displayed or decoded for display.

Ahhh, so why have them encoded in the first place, you know that you can
specify in what character encoding browsers are supposed do send the
characters (see w3)!

So attack your problem at the source:
use a decend encoding (either the proper iso8859-x for
cyrillic, or better yet UTF-8 so you can "mix chactersets").

So don't use htmlentities (I can't imagine any use of it anyway), use
htmlspecialchars instead.

That does not make any difference, as both convert Д into
&amp;#1044; - which is exactly the problem.

Doh! :)

[Markus Ernst]

Daniel Tryba wrote:

Ahhh, so why have them encoded in the first place, you know that you
can specify in what character encoding browsers are supposed do send
the characters (see w3)!

So attack your problem at the source:
use a decend encoding (either the proper iso8859-x for
cyrillic, or better yet UTF-8 so you can "mix chactersets").

Thank you, this is definitely the way to go. Anyway, <meta
http-equiv="content-type" content="text/html;charset=utf-8"> does not make
IE use UTF-8. But when manually settin UTF-8, the characters are actually
transferred unencoded.

For a quick and dirty fix, this did the trick for proper display:
$text = preg_replace("/&amp;#([0-9a-fx]+);/mi", "&#\$1;", $text);

But I will definitely try to go the Unicode way.

--
Markus

[Daniel Tryba]

Markus Ernst <derernst@no#sp#amgmx.ch> wrote:

So attack your problem at the source:
use a decend encoding (either the proper iso8859-x for
cyrillic, or better yet UTF-8 so you can "mix chactersets").
Thank you, this is definitely the way to go. Anyway, <meta
http-equiv="content-type" content="text/html;charset=utf-8"> does not make
IE use UTF-8. But when manually settin UTF-8, the characters are actually
transferred unencoded.

When using PHP please don't use those silly http-equiv thingies, use the
real headers instead: header('Content-Type: text/html; charset=UTF-8');

You also might want to add the correct attribute to the form element
(IIRC accept-encoding, but see w3c to be sure).
For a quick and dirty fix, this did the trick for proper display:
$text = preg_replace("/&amp;#([0-9a-fx]+);/mi", "&#\$1;", $text);
Nice one, I was wondering if using chr() worked, but this is much nicer
(although still a hack :)
But I will definitely try to go the Unicode way.

_The_ way to go. Take a look at the MultiByte String section of the
manual, setting output and internal encoding to UTF-8 works for me (I
never heard any complaints so far).

[Chung Leong]

"Markus Ernst" <derernst@NO#SP#AMgmx.ch> wrote in message
news:42**********@news.cybercity.ch...

For a quick and dirty fix, this did the trick for proper display:
$text = preg_replace("/&amp;#([0-9a-fx]+);/mi", "&#\$1;", $text);
The &$xxxx; format accepts decimal, so the a-f part is unnecessary :-)
But I will definitely try to go the Unicode way.

Switching from a single byte charset to a multibyte encoding is a fairly big
change. At the least you're looking at transcoding all your existing data.

[Markus Ernst]

Chung Leong wrote:

"Markus Ernst" <derernst@NO#SP#AMgmx.ch> wrote in message
news:42**********@news.cybercity.ch...

For a quick and dirty fix, this did the trick for proper display:
$text = preg_replace("/&amp;#([0-9a-fx]+);/mi", "&#\$1;", $text);

The &$xxxx; format accepts decimal, so the a-f part is unnecessary :-)

But I will definitely try to go the Unicode way.

Switching from a single byte charset to a multibyte encoding is a
fairly big change. At the least you're looking at transcoding all
your existing data.

Thank you for this input - I will work on fixing that whole thing before the
application goes productive and the users begin to submit real text.

--
Markus

[Markus Ernst]

Daniel Tryba wrote:

When using PHP please don't use those silly http-equiv thingies, use
the real headers instead: header('Content-Type: text/html;
charset=UTF-8');
Oh - I was proud of having created an application where the proper encoding
for the selected language was automatically used in the http-equiv thingy...
But I admit that from the beginning I was afraid of the moment somebody
decided to make an arab or hebrew or thai web site with it :-) I am working
on a CMS system that can handle an unlimited number of language versions for
every page of the site, but did not think of the possibility of mixing up
languages inside of a page. I am actually lucky that a user of a previous
version of the system tried to do exactly this :-)
You also might want to add the correct attribute to the form element
(IIRC accept-encoding, but see w3c to be sure).

I thought I knew really a lot about HTML, but there are still lots of things
to learn (it is accept-charset).

For a quick and dirty fix, this did the trick for proper display:
$text = preg_replace("/&amp;#([0-9a-fx]+);/mi", "&#\$1;", $text);

Nice one, I was wondering if using chr() worked, but this is much
nicer (although still a hack :)

I tried all examples with chr() I found in the manual, and it was
interesting how different the results can be with different operations
performed before or after. There must be conflicts between different
encodings of cyrillic and I doubt that a solution found for those characters
I get would reliably work for cyrillic characters sent from any other
system.

But I will definitely try to go the Unicode way.

_The_ way to go. Take a look at the MultiByte String section of the
manual, setting output and internal encoding to UTF-8 works for me (I
never heard any complaints so far).

I will spend more time on learning about that. Thanks a lot for your inputs.

--
Markus

[Markus Ernst]

R. Rajesh Jeba Anbiah wrote:

Somebody can point me to the right direction?

http://in2.php.net/utf8_encode#49336

Thank you, this works well! I even had come across it before, but as I was
not aware of the UTF-8 header stuff I (and thus got bad results) had not
given it much attention.

--
Markus