473,695 Members | 2,788 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Unescaping URLs in Python

Here's a URL from a link on the home page of a major company.

<a href="/adsk/servlet/index?siteID=12 3112&amp;id=186 0142">About Us</a>

Yes, that "&amp;" is in the source text of the page.

This is, in fact, correct HTML. See

http://www.htmlhelp.com/tools/valida...blems.html#amp

What's the appropriate Python function to call to unescape a URL which might
contain things like that? Will this interfere with the usual "%" type escapes
in URLs?

What's actually needed to get this right is something that goes from
HTML escaped form to URL escaped form, because, in general, there is no
unescaped form that will work for all URLs.

There's "htmldecode " at "http://zesty.ca/python/scrape.py", which works,
but this should be a standard library function.

John Nagle
Dec 25 '06 #1
3 2878
In message <hW************ ******@newssvr2 1.news.prodigy. net>, John Nagle
wrote:
Here's a URL from a link on the home page of a major company.

<a href="/adsk/servlet/index?siteID=12 3112&amp;id=186 0142">About Us</a>

What's the appropriate Python function to call to unescape a URL
which might contain things like that?
Just use any HTML-parsing library. I think the standard Python HTMLParser
will do the trick, provided there aren't any errors in the HTML.
Will this interfere with the usual "%" type escapes in URLs?
No. Just think of it as an HTML attribute value; the fact that it's a URL is
a question of later interpretation, nothing to do with the fact that it
comes from an HTML attribute.

Dec 25 '06 #2
Lawrence D'Oliveiro wrote:
In message <hW************ ******@newssvr2 1.news.prodigy. net>, John Nagle
wrote:

>>Here's a URL from a link on the home page of a major company.

<a href="/adsk/servlet/index?siteID=12 3112&amp;id=186 0142">About Us</a>

What's the appropriate Python function to call to unescape a URL
which might contain things like that?


Just use any HTML-parsing library. I think the standard Python HTMLParser
will do the trick, provided there aren't any errors in the HTML.
I'm using BeautifulSoup, because I need to process real world
HTML. At least by default, it doesn't unescape URLs like that.

Nor, on the output side, does it escape standalone "&" characters,
as in text like "Sales & Advertising Department".
But there are various BeautifulSoup options; more on this later.

John Nagle
Dec 25 '06 #3
John Nagle wrote:
What's the appropriate Python function to call to unescape a URL which
might contain things like that?
xml.sax.saxutil s.unescape()

Will this interfere with the usual "%"
type escapes in URLs?
Nope, and urllib.unquote( ) can be used to translate URL escapes manually.

Jeffrey
Dec 25 '06 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
7726
by: Daniel | last post by:
I'm working with strings that contain xml escape codes, such as '0' and need a way in python to unescape these back to their ascii representation, such as '&' but can't seem to find a python method for this. I tried xml.sax.saxutils.unescape(s), but while it works with '&amp;', it doesn't work with '0' and other numeric codes. Any suggestions on how to decode the numeric xml escape codes such as this? Thanks. -- To reply to me directly,...
3
2119
by: pinkfloydhomer | last post by:
I am writing a program that has to do some lightweight HTTP communication with a webserver on the internet. I haven't checked, but I'm sure I could do something lowlevel like opening a socket myself and then send/receive everything myself on this (how do I do that?), but I'd bet that Python have some module which is more high level. Something that would just let me connect using an URL, send a few GETs, and receive the answer as a...
2
8421
by: Vance Kessler | last post by:
We are trying write a new ASP.NET page to work with an existing stateless ASP application. The ASP application creates a cookie and of course stores the cookie values as escaped strings (using the vbscript escape function). I am have a terrible time 'unescaping' that string with C# under ASP.NET. I have tried the following: HttpUtility.UrlDecode HttpUtility.HtmlDecode (this should not work but I tried it)...
0
1576
by: Kurt B. Kaiser | last post by:
Patch / Bug Summary ___________________ Patches : 391 open ( +7) / 3028 closed (+12) / 3419 total (+19) Bugs : 906 open ( -3) / 5519 closed (+19) / 6425 total (+16) RFE : 207 open ( -1) / 197 closed ( +1) / 404 total ( +0) New / Reopened Patches ______________________
7
1819
by: Ben Finney | last post by:
Howdy all, I'm working on a web application that is starting to gain a lot of back-end code written in Python. However, all the current interface code is written in legacy PHP. I'd like to slowly introduce new features as Python WSGI programs. Is it possible to write a Python WSGI program that talks to a PHP program as its "back end"? Where can I find out how to do this, preferably with examples?
9
2464
by: Frank Potter | last post by:
I want to find a multithreaded downloading lib in python, can someone recommend one for me, please? Thanks~
11
8954
by: Steven D'Aprano | last post by:
I'm using urllib.urlretrieve() to download HTML pages, and I've hit a snag with URLs containing ampersands: http://www.example.com/parrot.php?x=1&y=2 Somewhere in the process, urls like the above are escaped to: http://www.example.com/parrot.php?x=1&amp;y=2 which naturally fails to exist.
0
1655
by: kshw | last post by:
Hi, Iím new to programming. Iím currently learning python to write a web crawler to extract all text from a web page, in addition to, crawling to further URLs and collecting the text there. The idea is to place all the extracted text in a .txt file with each word in a single line. So the text has to be tokenized. All punctuation marks, duplicate words and non-stop words have to be removed. The program should crawl the web to a certain...
0
8649
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, weíll explore What is ONU, What Is Router, ONU & Routerís main usage, and What is the difference between ONU and Router. Letís take a closer look ! Part I. Meaning of...
0
8586
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9137
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
5842
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4351
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4592
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3025
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2289
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
1986
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.