473,221 Members | 1,931 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,221 software developers and data experts.

Unescaping URLs in Python

Here's a URL from a link on the home page of a major company.

<a href="/adsk/servlet/index?siteID=123112&amp;id=1860142">About Us</a>

Yes, that "&amp;" is in the source text of the page.

This is, in fact, correct HTML. See

http://www.htmlhelp.com/tools/valida...blems.html#amp

What's the appropriate Python function to call to unescape a URL which might
contain things like that? Will this interfere with the usual "%" type escapes
in URLs?

What's actually needed to get this right is something that goes from
HTML escaped form to URL escaped form, because, in general, there is no
unescaped form that will work for all URLs.

There's "htmldecode" at "http://zesty.ca/python/scrape.py", which works,
but this should be a standard library function.

John Nagle
Dec 25 '06 #1
3 2848
In message <hW******************@newssvr21.news.prodigy.net >, John Nagle
wrote:
Here's a URL from a link on the home page of a major company.

<a href="/adsk/servlet/index?siteID=123112&amp;id=1860142">About Us</a>

What's the appropriate Python function to call to unescape a URL
which might contain things like that?
Just use any HTML-parsing library. I think the standard Python HTMLParser
will do the trick, provided there aren't any errors in the HTML.
Will this interfere with the usual "%" type escapes in URLs?
No. Just think of it as an HTML attribute value; the fact that it's a URL is
a question of later interpretation, nothing to do with the fact that it
comes from an HTML attribute.

Dec 25 '06 #2
Lawrence D'Oliveiro wrote:
In message <hW******************@newssvr21.news.prodigy.net >, John Nagle
wrote:

>>Here's a URL from a link on the home page of a major company.

<a href="/adsk/servlet/index?siteID=123112&amp;id=1860142">About Us</a>

What's the appropriate Python function to call to unescape a URL
which might contain things like that?


Just use any HTML-parsing library. I think the standard Python HTMLParser
will do the trick, provided there aren't any errors in the HTML.
I'm using BeautifulSoup, because I need to process real world
HTML. At least by default, it doesn't unescape URLs like that.

Nor, on the output side, does it escape standalone "&" characters,
as in text like "Sales & Advertising Department".
But there are various BeautifulSoup options; more on this later.

John Nagle
Dec 25 '06 #3
John Nagle wrote:
What's the appropriate Python function to call to unescape a URL which
might contain things like that?
xml.sax.saxutils.unescape()

Will this interfere with the usual "%"
type escapes in URLs?
Nope, and urllib.unquote() can be used to translate URL escapes manually.

Jeffrey
Dec 25 '06 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Daniel | last post by:
I'm working with strings that contain xml escape codes, such as '0' and need a way in python to unescape these back to their ascii representation, such as '&' but can't seem to find a python method...
3
by: pinkfloydhomer | last post by:
I am writing a program that has to do some lightweight HTTP communication with a webserver on the internet. I haven't checked, but I'm sure I could do something lowlevel like opening a socket...
2
by: Vance Kessler | last post by:
We are trying write a new ASP.NET page to work with an existing stateless ASP application. The ASP application creates a cookie and of course stores the cookie values as escaped strings (using the...
0
by: Kurt B. Kaiser | last post by:
Patch / Bug Summary ___________________ Patches : 391 open ( +7) / 3028 closed (+12) / 3419 total (+19) Bugs : 906 open ( -3) / 5519 closed (+19) / 6425 total (+16) RFE : 207 open...
7
by: Ben Finney | last post by:
Howdy all, I'm working on a web application that is starting to gain a lot of back-end code written in Python. However, all the current interface code is written in legacy PHP. I'd like to...
9
by: Frank Potter | last post by:
I want to find a multithreaded downloading lib in python, can someone recommend one for me, please? Thanks~
11
by: Steven D'Aprano | last post by:
I'm using urllib.urlretrieve() to download HTML pages, and I've hit a snag with URLs containing ampersands: http://www.example.com/parrot.php?x=1&y=2 Somewhere in the process, urls like the...
0
by: kshw | last post by:
Hi, Iím new to programming. Iím currently learning python to write a web crawler to extract all text from a web page, in addition to, crawling to further URLs and collecting the text there. The...
0
by: veera ravala | last post by:
ServiceNow is a powerful cloud-based platform that offers a wide range of services to help organizations manage their workflows, operations, and IT services more efficiently. At its core, ServiceNow...
0
by: VivesProcSPL | last post by:
Obviously, one of the original purposes of SQL is to make data query processing easy. The language uses many English-like terms and syntax in an effort to make it easy to learn, particularly for...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
2
by: jimatqsi | last post by:
The boss wants the word "CONFIDENTIAL" overlaying certain reports. He wants it large, slanted across the page, on every page, very light gray, outlined letters, not block letters. I thought Word Art...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.