character sets

[WindAndWaves]

Hi Folk
Here I am writing my first php / mysql site, almost ready, and now this... charactersets....

The encoding that I use on my webpage is:

<META HTTP-EQUIV="content-type" CONTENT="text/html; charset=UTF-8">

When people enter new data I use

$newvalue = htmlentities($_POST["newvalue"], ENT_QUOTES)

I then SQL this into my table and next I display the value

e.g. <DIV CLASS="content">'.$newvalue.'</DIV>

All of this works fine, BUT, funny characters that may have been entered through the form (e.g. Word-Style quotation marks,
e-accent-grave, etc..) are taking on a whole new life. I put in an e with an accent and it changed into a chinese character.

I tried to run

$link = mysql_connect($host, $username, $password);
$charset = mysql_character_set_name($link);
printf ("character set is %s\n", $charset);

but that only gave me an error.

I searched on google, but many of the notes are in other languages.... ;-)

Does anyone have any hints in English?

TIA

- Nicolaas

[NC]

WindAndWaves wrote:

The encoding that I use on my webpage is:
<META HTTP-EQUIV="content-type" CONTENT="text/html; charset=UTF-8">

When people enter new data I use
$newvalue = htmlentities($_POST["newvalue"], ENT_QUOTES)

I then SQL this into my table and next I display the value
e.g. <DIV CLASS="content">'.$newvalue.'</DIV>

All of this works fine, BUT, funny characters that may have been
entered through the form (e.g. Word-Style quotation marks,
e-accent-grave, etc..) are taking on a whole new life.
I put in an e with an accent and it changed into a chinese
character.

You have two options to fix this:

1. Convert your strings from UTF-8 into, say, ISO-8859-1,
before storing them in the database:

$string = iconv('UTF-8', 'ISO-8859-1', $string);

You will need your PHP installation to be compiled
with iconv extension to do that.

2. Set your MySQL server's character set to UTF-8.

First, check if you currently have UTF-8 support.
Run this query:

SHOW VARIABLES;

find the `character_sets` variable in the output and
verify that `utf8` is listed among the character sets
currently supported. If there's no support for UTF-8,
install or configure it (see MySQL documentation for
details).

If and when you have UTF-8 support, you can set
UTF-8 as the default character set for your database:

ALTER DATABASE db_name
DEFAULT CHARACTER SET utf8;

Alternatively, you can change character set setting
on a per-connection basis by sending this query:

SET NAMES 'utf8';

first thing after establishing a connection to the
database.

Cheers,
NC

[Andy Hassall]

On 1 Feb 2005 13:40:47 -0800, "NC" <[e-mail]> wrote:

You have two options to fix this:

1. Convert your strings from UTF-8 into, say, ISO-8859-1,
before storing them in the database:

$string = iconv('UTF-8', 'ISO-8859-1', $string);
That's a lossy conversion though, so be careful.
You will need your PHP installation to be compiled
with iconv extension to do that.

2. Set your MySQL server's character set to UTF-8.

First, check if you currently have UTF-8 support.
Run this query:

SHOW VARIABLES;

find the `character_sets` variable in the output and
verify that `utf8` is listed among the character sets
currently supported. If there's no support for UTF-8,
install or configure it (see MySQL documentation for
details).

If and when you have UTF-8 support, you can set
UTF-8 as the default character set for your database:

ALTER DATABASE db_name
DEFAULT CHARACTER SET utf8;

Alternatively, you can change character set setting
on a per-connection basis by sending this query:

SET NAMES 'utf8';

first thing after establishing a connection to the
database.

If you don't mind any length functions returning the wrong values (i.e.
returning byte length not character length), you could probably even get away
with storing UTF-8 in MySQL without setting anything - provided it doesn't
attempt to do any character set conversions, just stores strings as-is.

But basically you have to be very careful when working with character set
encodings, since you've got to know what you're dealing with at each step, and
whether any function's going to try and interpret the encoded bytes into a
character, or just pass it on.

--
Andy Hassall / <an**@andyh.co.uk> / <http://www.andyh.co.uk>
<http://www.andyhsoftware.co.uk/space> Space: disk usage analysis tool

[Andy Hassall]

On Tue, 1 Feb 2005 21:02:15 +1300, "WindAndWaves" <ac****@ngaru.com> wrote:

Here I am writing my first php / mysql site, almost ready, and now this... charactersets....

The encoding that I use on my webpage is:

<META HTTP-EQUIV="content-type" CONTENT="text/html; charset=UTF-8">
Send a proper character set header; using <meta> for content type and
encodings is generally for situations where HTTP headers don't exist, e.g.
reading off a filesystem.

header("Content-type: text/html; charset=utf-8");

http://uk2.php.net/header
When people enter new data I use

$newvalue = htmlentities($_POST["newvalue"], ENT_QUOTES)
That defaults to ISO-8859-1; if you pass it UTF-8 without setting the third
parameter, you'll corrupt your data.

http://uk2.php.net/htmlentities
I then SQL this into my table and next I display the value

e.g. <DIV CLASS="content">'.$newvalue.'</DIV>

All of this works fine, BUT, funny characters that may have been entered through the form (e.g. Word-Style quotation marks,
e-accent-grave, etc..)
These characters all exist in UTF-8 - as does almost every character.
are taking on a whole new life. I put in an e with an accent and it changed into a chinese character.
Can you give a short self-contained example demonstrating it?
I tried to run

$link = mysql_connect($host, $username, $password);
$charset = mysql_character_set_name($link);
printf ("character set is %s\n", $charset);

but that only gave me an error.

According to the manul there's no such function. There's
mysqli_character_set_name, from the new PHP5 mysqli extension - but not in the
old mysql extension.

--
Andy Hassall / <an**@andyh.co.uk> / <http://www.andyh.co.uk>
<http://www.andyhsoftware.co.uk/space> Space: disk usage analysis tool

[WindAndWaves]

"Andy Hassall" <an**@andyh.co.uk> wrote in message news:3m********************************@4ax.com...

On Tue, 1 Feb 2005 21:02:15 +1300, "WindAndWaves" <ac****@ngaru.com> wrote:

Here I am writing my first php / mysql site, almost ready, and now this... charactersets....

The encoding that I use on my webpage is:

<META HTTP-EQUIV="content-type" CONTENT="text/html; charset=UTF-8">

Send a proper character set header; using <meta> for content type and
encodings is generally for situations where HTTP headers don't exist, e.g.
reading off a filesystem.

header("Content-type: text/html; charset=utf-8");

http://uk2.php.net/header

When people enter new data I use

$newvalue = htmlentities($_POST["newvalue"], ENT_QUOTES)

That defaults to ISO-8859-1; if you pass it UTF-8 without setting the third
parameter, you'll corrupt your data.

http://uk2.php.net/htmlentities

I then SQL this into my table and next I display the value

e.g. <DIV CLASS="content">'.$newvalue.'</DIV>

All of this works fine, BUT, funny characters that may have been entered through the form (e.g. Word-Style quotation marks,
e-accent-grave, etc..)

These characters all exist in UTF-8 - as does almost every character.

are taking on a whole new life. I put in an e with an accent and it changed into a chinese character.

Can you give a short self-contained example demonstrating it?

I tried to run

$link = mysql_connect($host, $username, $password);
$charset = mysql_character_set_name($link);
printf ("character set is %s\n", $charset);

but that only gave me an error.

According to the manul there's no such function. There's
mysqli_character_set_name, from the new PHP5 mysqli extension - but not in the
old mysql extension.

--
Andy Hassall / <an**@andyh.co.uk> / <http://www.andyh.co.uk>

Hi Andy and NC

I since discovered that a lot of functions arenot supported by my provider. Namely, UTF-8 is not supported in MySQL and PHP does
not support, for example, the conversion functions that you mention above.

I think I will have to stick with a pretty plane type of characterset and make the Japanese pages by hand.

Thank you for your helpful answers.

- Nicolaas

[Tony Marston]

"WindAndWaves" <ac****@ngaru.com> wrote in message
news:OC*********************@news.xtra.co.nz...

"Andy Hassall" <an**@andyh.co.uk> wrote in message
news:3m********************************@4ax.com...

On Tue, 1 Feb 2005 21:02:15 +1300, "WindAndWaves" <ac****@ngaru.com>
wrote:

>Here I am writing my first php / mysql site, almost ready, and now
>this... charactersets....
>
>The encoding that I use on my webpage is:
>
><META HTTP-EQUIV="content-type" CONTENT="text/html; charset=UTF-8">
Send a proper character set header; using <meta> for content type and
encodings is generally for situations where HTTP headers don't exist,
e.g.
reading off a filesystem.

header("Content-type: text/html; charset=utf-8");

http://uk2.php.net/header

>
>When people enter new data I use
>
>$newvalue = htmlentities($_POST["newvalue"], ENT_QUOTES)

That defaults to ISO-8859-1; if you pass it UTF-8 without setting the
third
parameter, you'll corrupt your data.

http://uk2.php.net/htmlentities

>I then SQL this into my table and next I display the value
>
>e.g. <DIV CLASS="content">'.$newvalue.'</DIV>
>
>All of this works fine, BUT, funny characters that may have been entered
>through the form (e.g. Word-Style quotation marks,
>e-accent-grave, etc..)

These characters all exist in UTF-8 - as does almost every character.

>are taking on a whole new life. I put in an e with an accent and it
>changed into a chinese character.

Can you give a short self-contained example demonstrating it?

>I tried to run
>
>$link = mysql_connect($host, $username, $password);
>$charset = mysql_character_set_name($link);
>printf ("character set is %s\n", $charset);
>
>but that only gave me an error.

According to the manul there's no such function. There's
mysqli_character_set_name, from the new PHP5 mysqli extension - but not
in the
old mysql extension.

--
Andy Hassall / <an**@andyh.co.uk> / <http://www.andyh.co.uk>

Hi Andy and NC

I since discovered that a lot of functions arenot supported by my
provider. Namely, UTF-8 is not supported in MySQL

Wrong. MySQL 4.1 supports various character sets. Take a look at
http://dev.mysql.com/doc/mysql/en/charset.html
and PHP does
not support, for example, the conversion functions that you mention above.
Wrong again. Take a look at the multi-byte string conversion functions at
http://www.php.net/manual/en/ref.mbstring.php
--
Tony Marston

http://www.tonymarston.net
I think I will have to stick with a pretty plane type of characterset and
make the Japanese pages by hand.

Thank you for your helpful answers.

- Nicolaas

[WindAndWaves]

"Tony Marston" <to**@NOSPAM.demon.co.uk> wrote in message
[...........]

Wrong. MySQL 4.1 supports various character sets. Take a look at
http://dev.mysql.com/doc/mysql/en/charset.html

This is what my ISP dude said:
My understanding is unicode support is only in version 4.1 and above and we have no servers running

4.1 yet

this may change in the future

all our newver servers are running 4.0x

and PHP does
not support, for example, the conversion functions that you mention above.

Wrong again. Take a look at the multi-byte string conversion functions at
http://www.php.net/manual/en/ref.mbstring.php

and the same sort of thing seems to apply to PHP, although I am running
PHP Version 4.3.4. PHP simply does not recognise the functions.

It seems like I need to have my own dedicated server and this will costs...

Thanks for your answer.

- Nicolaas

[Tony Marston]

"WindAndWaves" <ac****@ngaru.com> wrote in message
news:RI*********************@news.xtra.co.nz...

> and PHP does
> not support, for example, the conversion functions that you mention
> above.
Wrong again. Take a look at the multi-byte string conversion functions at
http://www.php.net/manual/en/ref.mbstring.php

and the same sort of thing seems to apply to PHP, although I am running
PHP Version 4.3.4. PHP simply does not recognise the functions.

This is an optional extension, so it must be explictly enabled when your
version of PHP is built. A proper ISP would be prepared to configure in this
option for you.
Also, a proper ISP would be running PHP 4.3.10, not 4.3.4

--
Tony Marston

http://www.tonymarston.net
It seems like I need to have my own dedicated server and this will
costs...

Thanks for your answer.

- Nicolaas