URLs and ampersands

[Steven D'Aprano]

I'm using urllib.urlretrieve() to download HTML pages, and I've hit a
snag with URLs containing ampersands:

http://www.example.com/parrot.php?x=1&y=2

Somewhere in the process, urls like the above are escaped to:

http://www.example.com/parrot.php?x=1&y=2

which naturally fails to exist.

I could just do a string replace, but is there a "right" way to escape
and unescape URLs? I've looked through the standard lib, but I can't find
anything helpful.
--
Steven

[Gabriel Genellina]

En Mon, 04 Aug 2008 20:43:45 -0300, Steven D'Aprano
<st***@REMOVE-THIS-cybersource.com.auescribi�:

I'm using urllib.urlretrieve() to download HTML pages, and I've hit a
snag with URLs containing ampersands:

http://www.example.com/parrot.php?x=1&y=2

Somewhere in the process, urls like the above are escaped to:

http://www.example.com/parrot.php?x=1&amp;y=2

which naturally fails to exist.

I could just do a string replace, but is there a "right" way to escape
and unescape URLs? I've looked through the standard lib, but I can't find
anything helpful.

This works fine for me:

pyimport urllib
pyfn =
urllib.urlretrieve("http://c7.amazingcounters.com/counter.php?i=1516903
&c=4551022")[0]
pyopen(fn,"rb").read()
'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00...

So it's not urlretrieve escaping the url, but something else in your
code...

--
Gabriel Genellina

[Steven D'Aprano]

On Mon, 04 Aug 2008 23:16:46 -0300, Gabriel Genellina wrote:

En Mon, 04 Aug 2008 20:43:45 -0300, Steven D'Aprano
<st***@REMOVE-THIS-cybersource.com.auescribi�:

>I'm using urllib.urlretrieve() to download HTML pages, and I've hit a
snag with URLs containing ampersands:

http://www.example.com/parrot.php?x=1&y=2

Somewhere in the process, urls like the above are escaped to:

http://www.example.com/parrot.php?x=1&amp;y=2

which naturally fails to exist.

I could just do a string replace, but is there a "right" way to escape
and unescape URLs? I've looked through the standard lib, but I can't
find anything helpful.

This works fine for me:

pyimport urllib
pyfn =
urllib.urlretrieve("http://c7.amazingcounters.com/counter.php?i=1516903
&c=4551022")[0]
pyopen(fn,"rb").read()
'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00...

So it's not urlretrieve escaping the url, but something else in your
code...

I didn't say it urlretrieve was escaping the URL. I actually think the
URLs are pre-escaped when I scrape them from a HTML file. I have searched
for, but been unable to find, standard library functions that escapes or
unescapes URLs. Are there any such functions?

--
Steven

[Wojtek Walczak]

Dnia 05 Aug 2008 09:59:20 GMT, Steven D'Aprano napisa³(a):

I didn't say it urlretrieve was escaping the URL. I actually think the
URLs are pre-escaped when I scrape them from a HTML file. I have searched
for, but been unable to find, standard library functions that escapes or
unescapes URLs. Are there any such functions?

$ cd /usr/lib/python2.5/
$ grep "\&amp\;" *.py
BaseHTTPServer.py: return html.replace("&", "&amp;").replace("<",
"&lt;").replace(">", "&gt;")
cgi.py: s = s.replace("&", "&amp;") # Must be done first!
cgitb.py: doc = doc.replace('&', '&amp;').replace('<',
'&lt;')
difflib.py:
text=text.replace("&","&amp;").replace(">","&gt;") .replace("<","&lt;")
HTMLParser.py: s = s.replace("&amp;", "&") # Must be last
pydoc.py: return replace(text, '&', '&amp;', '<', '&lt;', '>',
'&gt;')
xmlrpclib.py: s = replace(s, "&", "&amp;")

So it could be BaseHTTPServer, cgi, cgitb, difflib, HTMLParser,
pydoc or xmlrpclib. Do you use any of these? Or maybe some other
external module?

--
Regards,
Wojtek Walczak,
http://www.stud.umk.pl/~wojtekwa/

[Richard Brodie]

"Steven D'Aprano" <st***@REMOVE-THIS-cybersource.com.auwrote in message
news:00**********************@news.astraweb.com...

I could just do a string replace, but is there a "right" way to escape
and unescape URLs?

The right way is to parse your HTML with an HTML parser. URLs are not
exempt from the normal HTML escaping rules, although there are an awful lot
of pages that get this wrong.

You didn't post any code, so it's hard to tell but maybe something like
ElementTree or lxml would be a better tool than the ones you are currently using.

[Duncan Booth]

Steven D'Aprano <st****@REMOVE.THIS.cybersource.com.auwrote:

I didn't say it urlretrieve was escaping the URL. I actually think the
URLs are pre-escaped when I scrape them from a HTML file. I have
searched for, but been unable to find, standard library functions that
escapes or unescapes URLs. Are there any such functions?

Whenever you put a URL into an HTML file you need to escape it, so
naturally you will also need to unescape it when it is retrieved from the
file. However, whatever you use to parse the HMTL ought to be unescaping
text and attributes as part of the parsing process, so you shouldn't need a
separate function for this.

e.g.

>>from BeautifulSoup import BeautifulSoup
soup = BeautifulSoup('''<a href="http://www.example.com/parrot.php?x=1

&amp;y=2">link</a>''')

>>soup.contents[0]['href']

u'http://www.example.com/parrot.php?x=1&y=2'

>>>

Even Python's builtin HTMLParser class will do this for you. What parser
are you using?

--
Duncan Booth http://kupuguy.blogspot.com

[Gabriel Genellina]

En Tue, 05 Aug 2008 06:59:20 -0300, Steven D'Aprano <st****@remove.this.cybersource.com.auescribió:

On Mon, 04 Aug 2008 23:16:46 -0300, Gabriel Genellina wrote:

>En Mon, 04 Aug 2008 20:43:45 -0300, Steven D'Aprano
<st***@REMOVE-THIS-cybersource.com.auescribi�:

>>I'm using urllib.urlretrieve() to download HTML pages, and I've hit a
snag with URLs containing ampersands:

http://www.example.com/parrot.php?x=1&y=2

Somewhere in the process, urls like the above are escaped to:

http://www.example.com/parrot.php?x=1&amp;y=2

which naturally fails to exist.

I could just do a string replace, but is there a "right" way to escape
and unescape URLs? I've looked through the standard lib, but I can't
find anything helpful.

This works fine for me:

pyimport urllib
pyfn =
urllib.urlretrieve("http://c7.amazingcounters.com/counter.php?i=1516903
&c=4551022")[0]
pyopen(fn,"rb").read()
'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00...

So it's not urlretrieve escaping the url, but something else in your
code...

I didn't say it urlretrieve was escaping the URL. I actually think the
URLs are pre-escaped when I scrape them from a HTML file.

(Ok, you didn't even menction you were scraping HTML pages...)

I have searched
for, but been unable to find, standard library functions that escapes or
unescapes URLs. Are there any such functions?

Yes: cgi.escape/unescape, and xml.sax.saxutils.escape/unescape.
How are you scraping the HTML source? Both BeautifulSoup and ElementTree.HTMLTreeBuilder already do that work for you.

--
Gabriel Genellina

[Matthew Woodcraft]

Steven D'Aprano wrote:

I'm using urllib.urlretrieve() to download HTML pages, and I've hit a
snag with URLs containing ampersands:

http://www.example.com/parrot.php?x=1&y=2

Somewhere in the process, urls like the above are escaped to:

http://www.example.com/parrot.php?x=1&amp;y=2

which naturally fails to exist.

I could just do a string replace, but is there a "right" way to escape
and unescape URLs? I've looked through the standard lib, but I can't find
anything helpful.

I don't believe there is a concept of 'escaping a URL' as such. How you
escape or unescape a URL depends on what context you're embedding it in
or extracting it from.

In this case, it looks like you have URLs which have been escaped to go
into an html CDATA attribute value (such as <a href="...">).

I believe there is no documented function in the Python standard library
which reverses this escaping (short of putting your string into a
larger document and parsing that with a full html or xml parser).

-M-

[Matthew Woodcraft]

Gabriel Genellina wrote:

Steven D'Aprano wrote:

>I have searched for, but been unable to find, standard library
functions that escapes or unescapes URLs. Are there any such
functions?

Yes: cgi.escape/unescape, and xml.sax.saxutils.escape/unescape.

I don't see a cgi.unescape in the standard library.

I don't think xml.sax.saxutils.unescape will be suitable for Steven's
purpose, because it doesn't process numeric character references (which
are both legal and seen in the wild in /href/ attributes).

-M-

[Steven D'Aprano]

On Tue, 05 Aug 2008 12:07:39 +0000, Duncan Booth wrote:

Whenever you put a URL into an HTML file you need to escape it, so
naturally you will also need to unescape it when it is retrieved from
the file. However, whatever you use to parse the HMTL ought to be
unescaping text and attributes as part of the parsing process, so you
shouldn't need a separate function for this.

....

Even Python's builtin HTMLParser class will do this for you. What parser
are you using?

A regex.

I know, I know, now I have two problems :-)

It's a quick and dirty hack, not a production piece of code, and I have a
quick and dirty fix by just using url.replace('&amp;', '&').

Thanks to everybody who replied. I guess I really have to bite the bullet
and learn how to use a proper HTML parser.

--
Steven

[Paul Rubin]

Steven D'Aprano <st***@REMOVE-THIS-cybersource.com.auwrites:

I could just do a string replace, but is there a "right" way to escape
and unescape URLs? I've looked through the standard lib, but I can't find
anything helpful.

xml.sax.utils.unescape()

[Duncan Booth]

Matthew Woodcraft <ma******@chiark.greenend.org.ukwrote:

Gabriel Genellina wrote:

>Steven D'Aprano wrote:

>>I have searched for, but been unable to find, standard library
functions that escapes or unescapes URLs. Are there any such
functions?

>Yes: cgi.escape/unescape, and xml.sax.saxutils.escape/unescape.

I don't see a cgi.unescape in the standard library.

I don't think xml.sax.saxutils.unescape will be suitable for Steven's
purpose, because it doesn't process numeric character references (which
are both legal and seen in the wild in /href/ attributes).

Here's the code I use. It handles decimal and hex entity references as well
as all html named entities.

import re
from htmlentitydefs import name2codepoint
name2codepoint = name2codepoint.copy()
name2codepoint['apos']=ord("'")

EntityPattern = re.compile('&(?:#(\d+)|(?:#x([\da-fA-F]+))|([a-zA-Z]+));')
def decodeEntities(s, encoding='utf-8'):
def unescape(match):
code = match.group(1)
if code:
return unichr(int(code, 10))
else:
code = match.group(2)
if code:
return unichr(int(code, 16))
else:
code = match.group(3)
if code in name2codepoint:
return unichr(name2codepoint[code])
return match.group(0)

if isinstance(s, str):
s = s.decode(encoding)
return EntityPattern.sub(unescape, s)

--
Duncan Booth http://kupuguy.blogspot.com