What chars are considered safe?

[Derek Fountain]

I was just writing a sanitisation route for a bit of user input. The data is
an English text description of a product, and will go into a DB, then back
out to other user's browsers.

As per normal practise, I was working on the basis of leaving in all
characters that I considered safe and stripping out everything else. This
led me to think of what characters are actually safe, given that the user
will want to be able to use at least basic punctuation, currency symbols
and so on. Avoiding < and > seemed obvious, but most other things have a
use I think.

My current line looks like this:

$data = preg_replace( '/[^\s\w\d@"\'()[]{}:#~!$%&*_-+.,]/', "", $data );

(Note that's a list of chars that are *not* to be replaced.) Are any of
these dangerous? Or have I left out some that are harmless and should be in
there?

--
The email address used to post is a spam pit. Contact me at
http://www.derekfountain.org : <a
href="http://www.derekfountain.org/">Derek Fountain</a>

[John Dunlop]

Derek Fountain wrote:

I was just writing a sanitisation route for a bit of user input. The data is
an English text description of a product, and will go into a DB, then back
out to other user's browsers.

As per normal practise, I was working on the basis of leaving in all
characters that I considered safe and stripping out everything else. This
led me to think of what characters are actually safe, given that the user
will want to be able to use at least basic punctuation, currency symbols
and so on. Avoiding < and > seemed obvious, but most other things have a
use I think.

My current line looks like this:

$data = preg_replace( '/[^\s\w\d@"\'()[]{}:#~!$%&*_-+.,]/', "", $data );
Two remarks on your pattern syntax: First, the second of
the two square brackets that appear to be in the character
class is in fact not. It is the closing square bracket, and
every non-metacharacter afterwards must be in the subject
string for the replacement to occur. Second, the hyphen, if
it were inside a character class, would cause a warning.
Either escape it or put it where it is not interpreted as a
metacharacter; that is, at the beginning or at the end.
(Note that's a list of chars that are *not* to be replaced.) Are any of
these dangerous? Or have I left out some that are harmless and should be in
there?

How about semicolons and pound signs ('£')?

--
Jock

[Michael Fesser]

.oO(Derek Fountain)

I was just writing a sanitisation route for a bit of user input. The data is
an English text description of a product, and will go into a DB, then back
out to other user's browsers.

As per normal practise, I was working on the basis of leaving in all
characters that I considered safe and stripping out everything else. This
led me to think of what characters are actually safe
Safe for what? In normal text every character can be safe if handled
properly.
given that the user
will want to be able to use at least basic punctuation, currency symbols
and so on. Avoiding < and > seemed obvious, but most other things have a
use I think.

What's wrong with < and >?

Micha

[Chung Leong]

"Derek Fountain" <no****@example.com> wrote in message
news:42**********************@per-qv1-newsreader-01.iinet.net.au...

I was just writing a sanitisation route for a bit of user input. The data is an English text description of a product, and will go into a DB, then back
out to other user's browsers.

As per normal practise, I was working on the basis of leaving in all
characters that I considered safe and stripping out everything else. This
led me to think of what characters are actually safe, given that the user
will want to be able to use at least basic punctuation, currency symbols
and so on. Avoiding < and > seemed obvious, but most other things have a
use I think.

My current line looks like this:

$data = preg_replace( '/[^\s\w\d@"\'()[]{}:#~!$%&*_-+.,]/', "", $data );

(Note that's a list of chars that are *not* to be replaced.) Are any of
these dangerous? Or have I left out some that are harmless and should be in there?

What encoding are you using? None of the characters above (maybe except 255)
is special, so I think can be safely included. People like to have their
curly quotes and m-dashes.

[NC]

Derek Fountain wrote:

I was just writing a sanitisation route for a bit of user input.
The data is an English text description of a product, and will
go into a DB, then back out to other user's browsers.

I think you should clarify your definition of "safe". Safe against
what? There are at least three issues that need to be worked on
here: use of potentially improper HTML formatting by users,
malicious Javascript, and SQL injection...

Also, do you plan to store user inputs as HTML or plain text?

Cheers,
NC

[Derek Fountain]

>> $data = preg_replace( '/[^\s\w\d@"\'()[]{}:#~!$%&*_-+.,]/', "", $data );

Two remarks on your pattern syntax: First, the second of
the two square brackets that appear to be in the character
class is in fact not. It is the closing square bracket, and
every non-metacharacter afterwards must be in the subject
string for the replacement to occur. Second, the hyphen, if
it were inside a character class, would cause a warning.

Ahem, yeah, I spotted those not long after posting... :o} Thanks for the
pointer though!

--
The email address used to post is a spam pit. Contact me at
http://www.derekfountain.org : <a
href="http://www.derekfountain.org/">Derek Fountain</a>

[Derek Fountain]

Michael Fesser wrote:

will want to be able to use at least basic punctuation, currency symbols
and so on. Avoiding < and > seemed obvious, but most other things have a
use I think.

What's wrong with < and >?

When returned to an innocent user's browser they allow cross site scripting.

--
The email address used to post is a spam pit. Contact me at
http://www.derekfountain.org : <a
href="http://www.derekfountain.org/">Derek Fountain</a>

[Michael Fesser]

.oO(Derek Fountain)

Michael Fesser wrote:

will want to be able to use at least basic punctuation, currency symbols
and so on. Avoiding < and > seemed obvious, but most other things have a
use I think.

What's wrong with < and >?

When returned to an innocent user's browser they allow cross site scripting.

Yep, I know. But that's not a problem with the chars itself, but rather
because of improper I/O handling. Use htmlspecialchars() whenever you
print text out to an HTML page and there's no reason anymore to forbid
such "special" chars.

Micha

[Derek Fountain]

NC wrote:

Derek Fountain wrote:

I was just writing a sanitisation route for a bit of user input.
The data is an English text description of a product, and will
go into a DB, then back out to other user's browsers.
I think you should clarify your definition of "safe". Safe against
what?

Safe for sending through a database API into the DB, then sending back out
to another user's browser...
There are at least three issues that need to be worked on
here: use of potentially improper HTML formatting by users,
malicious Javascript, and SQL injection...
....and not facilitating exploits based on those or any other attack vectors.

The fact several people asked me to define "safe" puzzled me somewhat. How
many interpretations are there of the word in the context of sending data
strings into a DB and back out to a user's browser? I've a feeling I'm
missing something! :o)
Also, do you plan to store user inputs as HTML or plain text?

As typed into a browser textarea input field. I'm expecting plain text, but
want the user to be able to provide as much punctuation, slang, smilies,
etc. as they like, as long as nothing goes in that might compromise the
system or another user's browser/session.

I know to leave < and > out to prevent XSS. I'm not sure about the entity
constructors & and ;. I'm tempted to leave the single quote out because it
is too useful in SQL injection. I'm suspicious of backticks because of
command injection - that's from my Perl background, although I'm not sure
if that's justified in PHP. I have a natural urge to surpress anything that
might be used in scripting, like #, !, and slashes, but again, not for a
reason I actually claim to understand.

Perhaps I'm going about this the wrong way? What do other PHP coders do with
a text value before considering it "safe" to store and relay back to other
user's browsers?
--
The email address used to post is a spam pit. Contact me at
http://www.derekfountain.org : <a
href="http://www.derekfountain.org/">Derek Fountain</a>

[Michael Fesser]

.oO(Derek Fountain)

Perhaps I'm going about this the wrong way? What do other PHP coders do with
a text value before considering it "safe" to store and relay back to other
user's browsers?

Every input/output operation of (string) data may need some special
escaping/encoding applied before to be safe, dependent on the target
media. In your case I would simply do it this way:

1) Every user-submitted string data is run through stripslashes() if
magic quotes are enabled (check with get_magic_quotes_gpc()). This way
you get the raw data, don't have to rely on some obscure configuration
setting and can apply a proper escaping yourself whenever necessary.

2) Before storing the data into the DB all the strings are run through
mysql_real_escape_string() to escape certain chars (like single quotes)
which might cause trouble for the DB.

3) When printing the data out again to an HTML page htmlspecialchars()
is called on all strings, which converts some chars that have a special
meaning in HTML (<, >, &, ") to named entity references.

That's all. There's no need to take special care of some particular
chars, the mentioned functions above take care of problems like SQL
injection and XSS.

Micha

[Matt Mitchell]

"Derek Fountain" <no****@example.com> wrote in message
news:42***********************@per-qv1-newsreader-01.iinet.net.au...

: I know to leave < and > out to prevent XSS. I'm not sure about the entity
: constructors & and ;. I'm tempted to leave the single quote out because it
: is too useful in SQL injection. I'm suspicious of backticks because of
: command injection - that's from my Perl background, although I'm not sure
: if that's justified in PHP. I have a natural urge to surpress anything
that
: might be used in scripting, like #, !, and slashes, but again, not for a
: reason I actually claim to understand.
:
: Perhaps I'm going about this the wrong way? What do other PHP coders do
with
: a text value before considering it "safe" to store and relay back to other
: user's browsers?

Well, for starters *anything* that I insert into a web page using php ALWAYS
goes through htmlentities() first, which entitity escapes &quot; &apos; &gt;
&lt; etc - otherwise you'll be making invalid html.

You might want to clean up stuff that people input too - there's a good
chance that someone putting
<script language='javascript'...
or
<?php
isn't necessarily entering something you'd like to display...

Matt

[Derek Fountain]

Michael Fesser wrote:

Every input/output operation of (string) data may need some special
escaping/encoding applied before to be safe, dependent on the target
media. In your case I would simply do it this way:

1) Every user-submitted string data is run through stripslashes() if
magic quotes are enabled (check with get_magic_quotes_gpc()). This way
you get the raw data, don't have to rely on some obscure configuration
setting and can apply a proper escaping yourself whenever necessary.

2) Before storing the data into the DB all the strings are run through
mysql_real_escape_string() to escape certain chars (like single quotes)
which might cause trouble for the DB.

3) When printing the data out again to an HTML page htmlspecialchars()
is called on all strings, which converts some chars that have a special
meaning in HTML (<, >, &, ") to named entity references.

That's all. There's no need to take special care of some particular
chars, the mentioned functions above take care of problems like SQL
injection and XSS.

This makes a lot of sense, thanks. One more question though. Assume I take
what the user has given me and do the DB escaping before I put it in the
database. I then ensure that whenever this string is returned to a browser
I pass it through htmlentities() first. But... what about when I want to
return the string to the user so they can edit it? The user will want to
see the quotes and angle brackets, etc., not the escaped versions of those
characters. Is it safe to hand the string back in a <textarea> or similar
without escaping it?

I experimented with Mozilla, and it seems to do the right thing. If I put
<script>alert('hello');<script> into my DB then send it out into a textarea
the browser doesn't run the script, but just offers it for editing. Is that
dependable? It would appear to be a browser feature, and not something PHP
has any control over.

--
The email address used to post is a spam pit. Contact me at
http://www.derekfountain.org : <a
href="http://www.derekfountain.org/">Derek Fountain</a>