473,324 Members | 2,400 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,324 software developers and data experts.

In the opinion of PHP, what is a character?


I'm worried about idiot users that write long essays in Microsoft Word,
then log into their accounts and bring up an HTML form and copy and
paste the essay and hit submit. Or perhaps they do this using
WordPerfect. Or perhaps they use MacWrite.

I output everything from my site as UTF-8. I'd like to check the input
for characters that are not UTF-8 and then turn the bad ones to an
ASCII question mark. I could loop through a string as if it was an
array and test each character, but what does PHP think a character is?
Does PHP understand what a multi-byte character is? Would this work?
// the $string is a form input, possibly containing characters
// written in any of the world's word processors
$finalString = "";
for ($i=0; $i < strlen($string); $i++) {
$char = $string[$i];
$encoding = mb_detect_encoding($char);
if ($encoding != "UTF-8") {
$char = "?";
}
$finalString .= $char;
}

They offer this on www.php.net, in the comments, but, again, I'm not
sure it would work on individual characters, and I'm about reading
Regx.

===========================

Much simpler UTF-8-ness checker using a regular expression created by
the W3C:

<?php

// Returns true if $string is valid UTF-8 and false otherwise.
function is_utf8($string) {

// From http://w3.org/International/question...rms-utf-8.html
return preg_match('%^(?:
[\x09\x0A\x0D\x20-\x7E] # ASCII
| [\xC2-\xDF][\x80-\xBF] # non-overlong 2-byte
| \xE0[\xA0-\xBF][\x80-\xBF] # excluding overlongs
| [\xE1-\xEC\xEE\xEF][\x80-\xBF]{2} # straight 3-byte
| \xED[\x80-\x9F][\x80-\xBF] # excluding surrogates
| \xF0[\x90-\xBF][\x80-\xBF]{2} # planes 1-3
| [\xF1-\xF3][\x80-\xBF]{3} # planes 4-15
| \xF4[\x80-\x8F][\x80-\xBF]{2} # plane 16
)*$%xs', $string);

} // function is_utf8

?>

Jul 21 '05 #1
5 1597
<lk******@geocities.com> wrote in message
news:11**********************@g49g2000cwa.googlegr oups.com...
what does PHP think a character is?
Does PHP understand what a multi-byte character is?


The user's manual is your friend here...

"In PHP, a character is the same as a byte, that is, there are exactly 256
different characters possible. This also implies that PHP has no native
support of Unicode. See utf8_encode() and utf8_decode() for some Unicode
support."

quoted from

http://us2.php.net/manual/en/language.types.string.php
Jul 21 '05 #2
On 21 Jul 2005 00:29:54 -0700, lk******@geocities.com wrote:
I output everything from my site as UTF-8. I'd like to check the input
for characters that are not UTF-8 and then turn the bad ones to an
ASCII question mark.
Déja vu all over again. ;-p
I could loop through a string as if it was an
array and test each character, but what does PHP think a character is?
PHP's string data type has no knowledge of character encodings. It treats
strings as a meaning-free series of bytes. Not characters.
Does PHP understand what a multi-byte character is?
No. The mbstring extension does, though.
Would this work?

// the $string is a form input, possibly containing characters
// written in any of the world's word processors
$finalString = "";
for ($i=0; $i < strlen($string); $i++) {
$char = $string[$i];
$encoding = mb_detect_encoding($char);
if ($encoding != "UTF-8") {
$char = "?";
}
$finalString .= $char;
}


No. This goes byte-by-byte. There's no reason why mb_detect_encoding should
return UTF-8, since for anything <127 then it could equally be ASCII, or for
other values some other encoding such as ISO-8859-15.

To find invalid UTF-8 encoded byte sequences you have to consider more than
one byte at a time.

As I believe was covered the previous times you've asked this:

You can tell whether a series of bytes is not a series of UTF-8 encoded
characters, by looking for byte sequences that are not valid UTF-8 - look for
lead bytes and the corresponding numbers of trail bytes.

Therefore, your current request (replace byte sequences that cannot be UTF-8
encodings with a "?" character) is quite possible, but you need to consider
more than one byte at a time and will probably have to backtrack a bit if you
get an invalid sequence.

In one previous incarnation of this thread I posted a script to detect invalid
UTF-8 byte sequences; looks like this could be quite easily adapted to your
current request:

http://groups.google.co.uk/group/com...0075dcf?hl=en&

Just remove the various returns that exit when it finds bad characters, and
instead, whenever $charSize drops to zero, append it to a string for output, or
if it finds a bad encoing, append a "?".

However, you cannot tell whether a byte sequence is actually a series of UTF-8
characters, because it could be encoded in something else that happens to share
the same byte representation.

--
Andy Hassall / <an**@andyh.co.uk> / <http://www.andyh.co.uk>
<http://www.andyhsoftware.co.uk/space> Space: disk usage analysis tool
Jul 21 '05 #3
>>I output everything from my site as UTF-8. I'd like to check the input
for characters that are not UTF-8 and then turn the bad ones to an
ASCII question mark.


Déja vu all over again. ;-p


I know. Every 3 or 4 months I come back to the issue and try to fix it,
but I never get it fixed. This has been going on for 2 years now. Maybe
I'll get it this time.

Jul 21 '05 #4
This should work as well:

preg_match('/./u', $text);

Badly encoded UTF-8 text would cause PCRE to go poopoo and return false
even where the string isn't empty.

Jul 21 '05 #5
lk******@geocities.com wrote:
I output everything from my site as UTF-8. I'd like to check the input
for characters that are not UTF-8 and then turn the bad ones to an
ASCII question mark. [...]


This function simply drop all the illegal sequences and return a legal
UTF-8 string:

function Force_UTF_8($s)
{
return mb_convert_encoding($s, 'UTF-8', 'UTF-8');
}

For example:

Force_UTF_8("A\x00B\xc0\x80C") ==> "A\000BC"

Note that the control characters (here \000) aren't removed.

Should be enought for many cases. You may send a warning to the user if
the resulting string differ from the original one.

Regards,
___
/_|_\ Umberto Salsi
\/_\/ www.icosaedro.it

Jul 22 '05 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
by: Ali Eghtebas | last post by:
Hi, I've made this regex to catch the start of a valid multiline comment such as "/*" in e.g. T-SQL code. "(?<=^(?:*'*')*?*)(?<!^(?:*'*')*?--.*)/\*.*?$" With Multiline option on. As we know...
35
by: Sandeep Sharma | last post by:
Right from the time the first edition of K&R was released, the advantages of using symbolic constants, as opposed to "magic numbers", has been emphasized ---- and for good reason. I don't dispute...
250
by: Sugapablo | last post by:
Just out of curiosity, while checking on a site I was working on, I decided to throw a couple of the web's most popular URLs into the W3C Markup Validator. Out of microsoft.com, google.com,...
69
by: Ravi | last post by:
#define A B #define B A main() { int A=5; float B=6.0; printf("\n %d %f",A,B); printf(" %d %f",B,A); }
28
by: SStory | last post by:
Hello Group. I am a VB/VB.NET programmer. I have done some C/C++ years ago. I am considering a job opportunity which is unfortunately in C# and not VB.NET. What is the group's opinion on...
49
by: Allen Browne | last post by:
If you are looking for opinon on what's useful in Access 2007, there's a new article at: http://allenbrowne.com/Access2007.html Covers what's good (useful features), what's mixed (good and bad),...
16
by: Mark Chambers | last post by:
Hi there, I'm seeking opinions on the use of regular expression searching. Is there general consensus on whether it's now a best practice to rely on this rather than rolling your own (string)...
15
by: Devon Null | last post by:
I was simply wondering if this was a mal-formed class header: #ifndef _items_h_ #define _items_h_ #include <iostream> #include <string> #include <vector> using namespace std;
3
KevinADC
by: KevinADC | last post by:
Purpose The purpose of this article is to discuss the difference between characters inside a character class and outside a character class and some special characters inside a character class....
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.