RFC3986, backslash in URI/URLs

[Alan J. Flavell]

[Sorry, there isn't a newsgroup for discussing URLs as such - this
seemed a reasonably on-topic place to discuss it...?]

The story so far: on somewhat unrelated newsgroup, my attention
fell upon the URL:
http://www.speedtouchdsl.com/prod706.htm
which contains a link to the purported URL:
http://www.speedtouchdsl.com/pdf\dat...06WL-780WL.pdf

Comparing the latter with other URLs in that area, it appeared that
the "\" was a probable blunder for "/". However, since their web
server is IIS, it appears that their server silently fixes-up this
blunder[1], and delivers the intended content. My recollection of
RFC1738 was that an unencoded "\" ought not to appear in a URL, so I
was initially inclined to rate this URL as broken...

However, this then led me down the trail of RFC2396, which 'updates
and merges "Uniform Resource Locators" [RFC1738] and "Relative Uniform
Resource Locators" [RFC1808]', and RFC3986, which "obsoletes rfc 1808
and updates rfc 1738".

In RFC2396 2.4.3, the backslash is listed under "Excluded US-ASCII
characters", under the subcategory of "unwise", with the "must"
requirement:

|Data corresponding to excluded characters must be escaped in order to
|be properly represented within a URI.

So far, so good.

But in RFC3986, this character "\" seems to have been stealthily
dropped from the list of characters needint to be escaped. I find no
mention of this change in Appendix D, "Changes from RFC2396".

The only substantive mention of "\" which I can find is in section 7.3
under the main heading of "7. Security Considerations":

|Special care should be taken when the URI path interpretation process
| involves the use of a back-end file system or related system
| functions. File systems typically assign an operational meaning to
| special characters, such as the "/", "\", ":", "[", and "]"

Aside from this potential security exposure, it appears to me that the
cited URL, which I would like to have categorised as defective, would
be rated as OK by this latest RFC. And since the server returns the
desired resource when this misbegotten URL is presented, I can't even
rate it as a blunder - can I?

Any suggestions why this apparently risky, and IMHO undesirable,
change was smuggled into the RFC without mentioning it in the changes?

In http://lists.w3.org/Archives/Public/...5May/0004.html ,
I found a top-posted "answer" which is resolutely ignoring the
bottom-quoted question -

| > Shouldn't backslash itself be included in the must-be-escaped
| > list?

Shouldn't it?

regards

[1] Of course, this isn't a situation that I meet in my own
serveradmin-ing using Apache. If the author codes "\" instead of "/"
in a URL, and attempts to follow the link with a www-conforming
browser, the link does not work. If they use IE instead, however, it
appears that it silently fixes-up the error on the *client* side. It
seems from my tests that IE6 makes no attempt to access the cited URL
directly - it replaces the "\" by "/" before even trying (whereas
Mozilla replaces the "\" by "%5C", after which, Apache, he say "no").

So it looks as if MS give themselves two bites at this fuxup: once in
their browser-like object, and once in their web server.

(Another reason why authors are misguided if they use MS software as
their only test of their web pages. But I digress.)

--

[Jack]

Alan J. Flavell inquired:

In http://lists.w3.org/Archives/Public/...5May/0004.html ,
I found a top-posted "answer" which is resolutely ignoring the
bottom-quoted question -

| > Shouldn't backslash itself be included in the must-be-escaped | >
list?

Shouldn't it?
I think it should. I think it's rather obvious[1]; regardless of
language, an escape character should always itself be escaped if it is
to take its "normal" value in some expression.
It seems from my tests that IE6 makes no attempt to access the cited
URL directly - it replaces the "\" by "/" before even trying

Yes, IE mangles URLs from the address-bar in several ways before sending
them off over the interweb.

--
Jack.

[1] The expressions "obviously..." and "it's obvious that..." are
frequently encountered when the author is about to perpetrate some
inadvertent fallacy.

[Harlan Messinger]

Jack wrote:

Alan J. Flavell inquired:

In http://lists.w3.org/Archives/Public/...5May/0004.html , I
found a top-posted "answer" which is resolutely ignoring the
bottom-quoted question -

| > Shouldn't backslash itself be included in the must-be-escaped | >
list?

Shouldn't it?

I think it should. I think it's rather obvious[1]; regardless of
language, an escape character should always itself be escaped if it is
to take its "normal" value in some expression.

How can you mean "regardless of language", given that which characters
are escape characters depends entirely on what language is in use?

[Harlan Messinger]

Alan J. Flavell wrote:

The only substantive mention of "\" which I can find is in section 7.3
under the main heading of "7. Security Considerations":

|Special care should be taken when the URI path interpretation process
| involves the use of a back-end file system or related system
| functions. File systems typically assign an operational meaning to
| special characters, such as the "/", "\", ":", "[", and "]"

Aside from this potential security exposure, it appears to me that the
cited URL, which I would like to have categorised as defective, would
be rated as OK by this latest RFC. And since the server returns the
desired resource when this misbegotten URL is presented, I can't even
rate it as a blunder - can I?

Any suggestions why this apparently risky, and IMHO undesirable,
change was smuggled into the RFC without mentioning it in the changes?

May I ask what the source of risk is? You mention that backslash being
the path delimiter on a back-end file system, but that can't be the
problem, since the forward slash is the path delimiter on other file
systems, and the interpretation of the forward slash in URIs as a path
delimiter doesn't create risk on that account.

[Jukka K. Korpela]

Harlan Messinger <hm*******************@comcast.net> scripsit:

Jack wrote:

- -

I think it should. I think it's rather obvious[1]; regardless of
language, an escape character should always itself be escaped if it
is to take its "normal" value in some expression.

How can you mean "regardless of language", given that which characters
are escape characters depends entirely on what language is in use?

Well, I'd say that the _principle_ of escaping an escape character,
regardless of language (notation), is adequate within broad limits. Jack's
error is that he assumes that the backslash is an escape character in the
"language" of URLs, i.e. URL syntax.

--
Jukka K. Korpela ("Yucca")
http://www.cs.tut.fi/~jkorpela/

[Alan J. Flavell]

On Sat, 10 Jun 2006, Harlan Messinger wrote:

Alan J. Flavell wrote:

The only substantive mention of "\" which I can find is in section
7.3 under the main heading of "7. Security Considerations":

|Special care should be taken when the URI path interpretation process
| involves the use of a back-end file system or related system
| functions. File systems typically assign an operational meaning to
| special characters, such as the "/", "\", ":", "[", and "]" [...]
Any suggestions why this apparently risky, and IMHO undesirable,
change was smuggled into the RFC without mentioning it in the
changes?
May I ask what the source of risk is? You mention that backslash
being the path delimiter on a back-end file system,

Well, I might have done so, if I had thought about it; but in fairness
it wasn't *my* mention, it was a quote from the RFC. :-}
but that can't be the problem, since
the forward slash is the path delimiter on other file systems,
I don't agree. In principle, the "/" has a defined meaning in a URL
(it's a hierarchy separator, if I can put it loosely), and anyone
interpreting a URL is required to attribute that meaning to it - no
matter what their local file system separator might be.

Whereas "\" has no defined meaning in the structure of a URL, and
could (given an insufficiently paranoid parser) possibly find its way
into a filesystem reference. Which could have significant
consequences on, say, Windows.
and the interpretation of the forward slash in URIs as a path
delimiter doesn't create risk on that account.

Because the URL "/" (the one that functions as a URL hierarchy
separator) never gets that far. By then it would have been turned
into the filesystem hiararchy separator, whatever that might be.
Yes, it might sometimes be "/", but don't let that fool you. It might
just as well been turned into ":" for a different filesystem, or into
a hierarchical database query or whatever, in the general case.

I think that's the sort of thing that the RFC authors have in mind,
anyway.

[Jack]

Harlan Messinger wrote:

Jack wrote:

Alan J. Flavell inquired:

In
http://lists.w3.org/Archives/Public/...5May/0004.html ,
I found a top-posted "answer" which is resolutely ignoring the
bottom-quoted question -

| > Shouldn't backslash itself be included in the must-be-escaped
| > list?

Shouldn't it?

I think it should. I think it's rather obvious[1]; regardless of
language, an escape character should always itself be escaped if it
is to take its "normal" value in some expression.

How can you mean "regardless of language", given that which
characters are escape characters depends entirely on what language is
in use?

Try this:

"For any language x, that set of characters which are escape characters
in x should themselvesd be escaped if they are to take their normal
values in some expression."

I thought that was obviously my meaning, and it seems to require some
perverse gymnastics to get my original utterance to mean something
different.

--
Jack.

[John Dunlop]

Alan J. Flavell

In http://lists.w3.org/Archives/Public/...5May/0004.html ,
I found a top-posted "answer" which is resolutely ignoring the
bottom-quoted question -

| > Shouldn't backslash itself be included in the must-be-escaped
| > list?

Shouldn't it?

I suppose you could say backslashes *are* included in the
must-be-escaped list, if you recognise the list as implied. The
explicit list seems to have been silently dropped: the word 'excluded'
appears in RFC3986 only in unrelated contexts, and I can't find mention
of this removal anywhere in the changeover notes:

http://www.gbiv.com/protocols/uri/rev-2002/issues.html

Anyway, backslashes still can't occur in URLs, since no production
allows them.

--
Jock

[David E. Ross]

John Dunlop wrote:

Alan J. Flavell

In http://lists.w3.org/Archives/Public/...5May/0004.html ,
I found a top-posted "answer" which is resolutely ignoring the
bottom-quoted question -

| > Shouldn't backslash itself be included in the must-be-escaped
| > list?

Shouldn't it?

I suppose you could say backslashes *are* included in the
must-be-escaped list, if you recognise the list as implied. The
explicit list seems to have been silently dropped: the word 'excluded'
appears in RFC3986 only in unrelated contexts, and I can't find mention
of this removal anywhere in the changeover notes:

http://www.gbiv.com/protocols/uri/rev-2002/issues.html

Anyway, backslashes still can't occur in URLs, since no production
allows them.

The real questions are: What is the meaning of a back-slash in a URL?
Does it have a special meaning, the way reserved characters (/, $, &, ?,
etc) have? If it has a special meaning, where is it documented so that
browser developers and Web page authors will know about it?

Note that it cannot just another character in the name of a path or
file. RFC 3986, Appendix A, indicates a path can have a name consisting
only of alphabetic characters, numerals, -, +, ., _, ~, and
percent-encoded characters. A path may also have @, :, and certain
reserved characters; but all these have special meainings within a path
(taking us back to the third question in my first paragraph).

In a very loose sense, percent-encoding is a form of escaping a
character. However, a percent-encoded character might have a different
meaing in a URL than the related literal character. For example, "%25"
represents the character "%". Obviously, the former (just a character
in a string of characters) is not treated the same as the latter (the
signal for percent-encoding).

I have yet to see a use of back-slash in a URL that was not an error,
generally a typo by the Web page author.

--

David E. Ross
<http://www.rossde.com/>

Concerned about someone (e.g., Pres. Bush) snooping
into your E-mail? Use PGP.
See my <http://www.rossde.com/PGP/>

[John Dunlop]

David E. Ross:

The real questions are:
I think these are different points of discussion, no more real or
imaginary than the original, but more off-topic. The original was
about the status of backslashes wrt URLs, which has a direct bearing on
whether or not a doc violates or conforms to the spec.
What is the meaning of a back-slash in a URL?
Since a sequence of characters containing a backslash can't be a URL,
I'll take that as a backslash percent-encoded, %5C. My answer then
would be it means pretty much whatever you want it to mean.
Does it have a special meaning, the way reserved characters (/, $, &, ?,
etc) have?
No, else it would be included in the Reserved set and its use would
be documented in RFC3986.
A path may also have @, :, and certain reserved characters; but all these have
special meainings within a path

Only scheme-specifically. 'Aside from dot-segments in hierarchical
paths, a path segment is considered opaque by the generic syntax.'
(sec. 3.3)

--
Jock

[Harlan Messinger]

Jack wrote:

Harlan Messinger wrote:

Jack wrote:

Alan J. Flavell inquired:
In
http://lists.w3.org/Archives/Public/...5May/0004.html ,
I found a top-posted "answer" which is resolutely ignoring the
bottom-quoted question -

| > Shouldn't backslash itself be included in the must-be-escaped
| > list?

Shouldn't it?

I think it should. I think it's rather obvious[1]; regardless of
language, an escape character should always itself be escaped if it
is to take its "normal" value in some expression.

How can you mean "regardless of language", given that which
characters are escape characters depends entirely on what language is
in use?

Try this:

"For any language x, that set of characters which are escape characters
in x should themselvesd be escaped if they are to take their normal
values in some expression."

I thought that was obviously my meaning, and it seems to require some
perverse gymnastics to get my original utterance to mean something
different.

It was obvious that that's what you should have meant, but it wasn't
obvious, if you did mean that, why you would have thought a remark about
the escaping of escape characters to be relevant, given that backslashes
*aren't* escape characters in the context being discussed (URL syntax).

[Harlan Messinger]

Alan J. Flavell wrote:

On Sat, 10 Jun 2006, Harlan Messinger wrote:

Alan J. Flavell wrote:

The only substantive mention of "\" which I can find is in section
7.3 under the main heading of "7. Security Considerations":

|Special care should be taken when the URI path interpretation process
| involves the use of a back-end file system or related system
| functions. File systems typically assign an operational meaning to
| special characters, such as the "/", "\", ":", "[", and "]" [...]
Any suggestions why this apparently risky, and IMHO undesirable,
change was smuggled into the RFC without mentioning it in the
changes?

May I ask what the source of risk is? You mention that backslash
being the path delimiter on a back-end file system,

Well, I might have done so, if I had thought about it; but in fairness
it wasn't *my* mention, it was a quote from the RFC. :-}

but that can't be the problem, since
the forward slash is the path delimiter on other file systems,

I don't agree. In principle, the "/" has a defined meaning in a URL
(it's a hierarchy separator, if I can put it loosely), and anyone
interpreting a URL is required to attribute that meaning to it - no
matter what their local file system separator might be.

Whereas "\" has no defined meaning in the structure of a URL, and
could (given an insufficiently paranoid parser) possibly find its way
into a filesystem reference. Which could have significant
consequences on, say, Windows.

and the interpretation of the forward slash in URIs as a path
delimiter doesn't create risk on that account.

Because the URL "/" (the one that functions as a URL hierarchy
separator) never gets that far. By then it would have been turned
into the filesystem hiararchy separator, whatever that might be.
Yes, it might sometimes be "/", but don't let that fool you. It might
just as well been turned into ":" for a different filesystem, or into
a hierarchical database query or whatever, in the general case.

OK, I see. Thanks for the explanation.

[Jack]

Harlan Messinger wrote:

"For any language x, that set of characters which are escape
characters in x should themselvesd be escaped if they are to take
their normal values in some expression."

I thought that was obviously my meaning, and it seems to require
some perverse gymnastics to get my original utterance to mean
something different.

It was obvious that that's what you should have meant, but it wasn't
obvious, if you did mean that, why you would have thought a remark
about the escaping of escape characters to be relevant, given that
backslashes *aren't* escape characters in the context being discussed
(URL syntax).

Indeed. Without giving it much thought, I had temporarily formed the
mistaken impression that backslashes *were* escape characters in URLs,
and that was the context of my post. Sorry for confusion.

--
Jack.

[Alan J. Flavell]

On Sun, 10 Jun 2006, John Dunlop wrote:

Alan J. Flavell

In http://lists.w3.org/Archives/Public/...5May/0004.html ,
I found a top-posted "answer" which is resolutely ignoring the
bottom-quoted question -

| > Shouldn't backslash itself be included in the must-be-escaped
| > list?

Shouldn't it?
I suppose you could say backslashes *are* included in the
must-be-escaped list, if you recognise the list as implied. The
explicit list seems to have been silently dropped:

ah, thanks for explaining that...
the word 'excluded' appears in RFC3986 only in unrelated contexts,
and I can't find mention of this removal anywhere in the changeover
notes:

http://www.gbiv.com/protocols/uri/rev-2002/issues.html
Well, appendix D does say:

Section 2, on characters, has been rewritten to explain what
characters are reserved, when they are reserved, and why they are
reserved, even when they are not used as delimiters by the generic
syntax.

So that, at least, does still define "reserved" characters.

But it also says

... URI normalizers
are now given license to decode any percent-encoded octets
corresponding to unreserved characters.

Oh, hang on! "Reserved Characters" are listed in 2.2, while
"Unreserved characters" are listed in 2.3. And the backslash is
neither "reserved" nor "unreserved".

So, "unreserved" does not mean "any characters which are not
reserved". There are characters which are neither. Is that
confusing?

Indeed, if I take an ASCII table, and mark off the (non-control)
characters which they designate as reserved, and as unreserved, I'm
left with quite a few which are neither, as follows.

"%" is obviously special ...

Leaving (by my reckoning) space, ", <, >, \, ^, `, {, |, and }

Is there a name for this category of characters - that are neither
reserved nor unreserved?
Anyway, backslashes still can't occur in URLs, since no production
allows them.

I must admit I hadn't tried approaching the problem from that
direction, but, now that you put it that way, it does make sense.

thanks

[David E. Ross]

Alan J. Flavell wrote:

On Sun, 10 Jun 2006, John Dunlop wrote:

Alan J. Flavell

In http://lists.w3.org/Archives/Public/...5May/0004.html ,
I found a top-posted "answer" which is resolutely ignoring the
bottom-quoted question -

| > Shouldn't backslash itself be included in the must-be-escaped
| > list?

Shouldn't it?

I suppose you could say backslashes *are* included in the
must-be-escaped list, if you recognise the list as implied. The
explicit list seems to have been silently dropped:

ah, thanks for explaining that...

the word 'excluded' appears in RFC3986 only in unrelated contexts,
and I can't find mention of this removal anywhere in the changeover
notes:

http://www.gbiv.com/protocols/uri/rev-2002/issues.html

Well, appendix D does say:

Section 2, on characters, has been rewritten to explain what
characters are reserved, when they are reserved, and why they are
reserved, even when they are not used as delimiters by the generic
syntax.

So that, at least, does still define "reserved" characters.

But it also says

... URI normalizers
are now given license to decode any percent-encoded octets
corresponding to unreserved characters.

Oh, hang on! "Reserved Characters" are listed in 2.2, while
"Unreserved characters" are listed in 2.3. And the backslash is
neither "reserved" nor "unreserved".

So, "unreserved" does not mean "any characters which are not
reserved". There are characters which are neither. Is that
confusing?

Indeed, if I take an ASCII table, and mark off the (non-control)
characters which they designate as reserved, and as unreserved, I'm
left with quite a few which are neither, as follows.

"%" is obviously special ...

Leaving (by my reckoning) space, ", <, >, \, ^, `, {, |, and }

Is there a name for this category of characters - that are neither
reserved nor unreserved?

Anyway, backslashes still can't occur in URLs, since no production
allows them.

I must admit I hadn't tried approaching the problem from that
direction, but, now that you put it that way, it does make sense.

thanks

My inference is that, if a character is neither reserved nor unreserved,
it must be prohibited.

--

David E. Ross
<http://www.rossde.com/>

Concerned about someone (e.g., Pres. Bush) snooping
into your E-mail? Use PGP.
See my <http://www.rossde.com/PGP/>

[David E. Ross]

John Dunlop wrote:

David E. Ross:

The real questions are:

I think these are different points of discussion, no more real or
imaginary than the original, but more off-topic. The original was
about the status of backslashes wrt URLs, which has a direct bearing on
whether or not a doc violates or conforms to the spec.

I used the phrase "The real questions are" somewhat facitiously to point
out that the original poster should tell us how he thinks back-slash
should be interpreted. I don't think he can, thus concluding the
discussion.

--

David E. Ross
<http://www.rossde.com/>

Concerned about someone (e.g., Pres. Bush) snooping
into your E-mail? Use PGP.
See my <http://www.rossde.com/PGP/>

[Michael Winter]

On 11/06/2006 21:15, Alan J. Flavell wrote:

[snip]

Oh, hang on! "Reserved Characters" are listed in 2.2, while
"Unreserved characters" are listed in 2.3. And the backslash is
neither "reserved" nor "unreserved".

So, "unreserved" does not mean "any characters which are not
reserved". There are characters which are neither. Is that confusing?
I wouldn't say so. An unreserved character is one that will never have a
special meaning, whereas reserved characters do either in generic- or
scheme-specific URI syntax.

The definition of reserved and unreserved characters has, as far as I
can see, the most significance within the percent-encoding mechanism
and, by extension, syntax-based normalisation where URI normalisers may
decode a subset of percent-encoded sequences. The remaining set of
characters have no real significance; a scheme should not use them as
delimiters. As such, special treatment from URI normalisers is not
necessary, so they do not feature in either category.

<rationale>

2.2 Reserved Characters says:

URI producing applications should percent-encode data octets
that correspond to characters in the reserved set unless
these characters are specifically allowed by the URI scheme
to represent data in that component.

So, unless present to delimit subcomponents within a URI, all
reserved characters should be percent-encoded.

Furthermore, "characters in the reserved set are protected from
normalization". That is to say that premature decoding may
render delimiters ambiguous so reserved characters should not
be decoded until the URI has been completely parsed, as noted
in 2.4 When to Encode or Decode:

When a URI is dereferenced, the components and subcomponents
significant to the scheme-specific dereferencing process (if
any) must be parsed and separated before the percent-encoded
octets within those components can be safely decoded, as
otherwise the data may be mistaken for component delimiters.

Finally, 2.3 Unreserved Characters states:

For consistency, percent-encoded octets in the ranges of
ALPHA (%41-%5A and %61-%7A), DIGIT (%30-%39), hyphen (%2D),
period (%2E), underscore (%5F), or tilde (%7E) [ed.: the
unreserved set] should not be created by URI producers and,
when found in a URI, should be decoded to their corresponding
unreserved characters by URI normalizers.

</rationale>

[snip]
Is there a name for this category of characters - that are neither
reserved nor unreserved?

Not to my knowledge, and I doubt it warrants a name, either. :-)

[snip]

Mike
Apologies for the quantity of quotes, though at least it saves searching
the RFC. For the record, the text was taken exclusively from #3986.

--
Michael Winter
Prefix subject with [News] before replying by e-mail.

[Andy Mabbett]

In message <Pi*******************************@ppepc20.ph.gla. ac.uk>,
Alan J. Flavell <fl*****@physics.gla.ac.uk> writes

Sorry, there isn't a newsgroup for discussing URLs as such

alt.www.webmaster ?

uk.net.web.authoring ?

ISTR that the use of "authoring" rather than "html" in the latter
newsgroup's name was deliberate; to allow for such discussions.

--
Andy Mabbett
Say "NO!" to compulsory ID Cards: <http://www.no2id.net/>

Free Our Data: <http://www.freeourdata.org.uk>

[Pierre Goiffon]

Alan J. Flavell wrote:

The story so far: on somewhat unrelated newsgroup, my attention
fell upon the URL:
http://www.speedtouchdsl.com/prod706.htm
which contains a link to the purported URL:
http://www.speedtouchdsl.com/pdf\dat...06WL-780WL.pdf

Comparing the latter with other URLs in that area, it appeared that
the "\" was a probable blunder for "/". However, since their web
server is IIS, it appears that their server silently fixes-up this
blunder[1], and delivers the intended content.

A set of tools for IIS, IIS Lockdown (it should certainly be installed
on all IIS servers), provides URL Scan. This latter disable by default
URL containing "\" (see http://www.securityfocus.com/infocus/1755 and
the default values for the DenyUrlSequences parameter)

[John Dunlop]

Alan J. Flavell:

So, "unreserved" does not mean "any characters which are not
reserved". There are characters which are neither. Is that
confusing?
I can see how it could be.
Indeed, if I take an ASCII table, and mark off the (non-control)
characters which they designate as reserved, and as unreserved, I'm
left with quite a few which are neither, as follows.

"%" is obviously special ...

Leaving (by my reckoning) space, ", <, >, \, ^, `, {, |, and }

Is there a name for this category of characters - that are neither
reserved nor unreserved?

I've called them 'disallowed' informally, following in 2396's
footsteps. 3986, leaving the figuring out of what's not allowed to
your process of deduction, doesn't assign a label to that set of
characters.

--
Jock