473,847 Members | 1,439 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

BeautifulSoup vs. loose & chars

I've been parsing existing HTML with BeautifulSoup, and occasionally
hit content which has something like "Design & Advertising", that is,
an "&" instead of an "&". Is there some way I can get BeautifulSoup
to clean those up? There are various parsing options related to "&"
handling, but none of them seem to do quite the right thing.

If I write the BeautifulSoup parse tree back out with "prettify",
the loose "&" is still in there. So the output is
rejected by XML parsers. Which is why this is a problem.
I need valid XML out, even if what went in wasn't quite valid.

John Nagle
Dec 26 '06 #1
7 4637

John Nagle wrote:
I've been parsing existing HTML with BeautifulSoup, and occasionally
hit content which has something like "Design & Advertising", that is,
an "&" instead of an "&". Is there some way I can get BeautifulSoup
to clean those up? There are various parsing options related to "&"
handling, but none of them seem to do quite the right thing.

If I write the BeautifulSoup parse tree back out with "prettify",
the loose "&" is still in there. So the output is
rejected by XML parsers. Which is why this is a problem.
I need valid XML out, even if what went in wasn't quite valid.

John Nagle

So do you want to remove "&" or replace them with "&" ? If you want
to replace it try the following;

import urllib, sys

try:
location = urllib.urlopen( url)
except IOError, (errno, strerror):
sys.exit("I/O error(%s): %s" % (errno, strerror))

content = location.read()
content = content.replace ("&", "&")
To do this with BeautifulSoup, i think you need to go through every
Tag, get its content, see if it contains an "&" and then replace the
Tag with the same Tag but the content contains "&"

Hope this helps.
Cheers

Dec 26 '06 #2
On 26 Dec 2006 04:22:38 -0800, placid <Bu****@gmail.c omwrote:
So do you want to remove "&" or replace them with "&amp;" ? If you want
to replace it try the following;
I think he wants to replace them, but just the invalid ones. I.e.,

This & this &amp; that

would become

This &amp; this &amp; that
No, i don't know how to do this efficiently. =/...
I think some kind of regex could do it.

--
Felipe.
Dec 26 '06 #3
"Felipe Almeida Lessa" <fe**********@g mail.comwrote:
On 26 Dec 2006 04:22:38 -0800, placid <Bu****@gmail.c omwrote:
>So do you want to remove "&" or replace them with "&amp;" ? If you
want to replace it try the following;

I think he wants to replace them, but just the invalid ones. I.e.,

This & this &amp; that

would become

This &amp; this &amp; that
No, i don't know how to do this efficiently. =/...
I think some kind of regex could do it.
Since he's asking for valid xml as output, it isn't sufficient just to
ignore entity definitions: HTML has a lot of named entities such as
&nbsp; but xml only has a very limited set of predefined named entities.
The safest technique is to convert them all to numeric escapes except
for the very limited set also guaranteed to be available in xml.

Try this:

from cgi import escape
import re
from htmlentitydefs import name2codepoint
name2codepoint = name2codepoint. copy()
name2codepoint['apos']=ord("'")

EntityPattern =
re.compile('&(? :#(\d+)|(?:#x([\da-fA-F]+))|([a-zA-Z]+));')

def decodeEntities( s, encoding='utf-8'):
def unescape(match) :
code = match.group(1)
if code:
return unichr(int(code , 10))
else:
code = match.group(2)
if code:
return unichr(int(code , 16))
else:
return unichr(name2cod epoint[match.group(3)])
return EntityPattern.s ub(unescape, s)
>>escape(
decodeEntities( "This & this &amp; that&nbsp;&eacu te;")).encode(
'ascii', 'xmlcharrefrepl ace')
'This &amp; this &amp; that é'
P.S. apos is handled specially as it isn't technically a
valid html entity (and Python doesn't include it in its entity
list), but it is an xml entity and recognised by many browsers so some
people might use it in html.

Dec 26 '06 #4
Felipe Almeida Lessa wrote:
On 26 Dec 2006 04:22:38 -0800, placid <Bu****@gmail.c omwrote:
>So do you want to remove "&" or replace them with "&amp;" ? If you want
to replace it try the following;


I think he wants to replace them, but just the invalid ones. I.e.,

This & this &amp; that

would become

This &amp; this &amp; that
No, i don't know how to do this efficiently. =/...
I think some kind of regex could do it.
Yes, and the appropriate one is:

krefindamp = re.compile(r'&( ?!(\w|#)+;)')
...
xmlsection = re.sub(krefinda mp,'&amp;',xmls ection)

This will replace an '&' with '&amp' if the '&' isn't
immediately followed by some combination of letters, numbers,
and '#' ending with a ';' Admittedly this would let something
like '&xx#2;', which isn't a legal entity, through unmodified.

There's still a potential problem with unknown entities in the output XML, but
at least they're recognized as entities.

John Nagle
Dec 26 '06 #5


Duncan Booth skrev:
"Felipe Almeida Lessa" <fe**********@g mail.comwrote:

>On 26 Dec 2006 04:22:38 -0800, placid <Bu****@gmail.c omwrote:
>>So do you want to remove "&" or replace them with "&amp;" ? If you
want to replace it try the following;
I think he wants to replace them, but just the invalid ones. I.e.,

This & this &amp; that

would become

This &amp; this &amp; that
No, i don't know how to do this efficiently. =/...
I think some kind of regex could do it.


Since he's asking for valid xml as output, it isn't sufficient just to
ignore entity definitions: HTML has a lot of named entities such as
&nbsp; but xml only has a very limited set of predefined named entities.
The safest technique is to convert them all to numeric escapes except
for the very limited set also guaranteed to be available in xml.

Try this:

from cgi import escape
import re
from htmlentitydefs import name2codepoint
name2codepoint = name2codepoint. copy()
name2codepoint['apos']=ord("'")

EntityPattern =
re.compile('&(? :#(\d+)|(?:#x([\da-fA-F]+))|([a-zA-Z]+));')

def decodeEntities( s, encoding='utf-8'):
def unescape(match) :
code = match.group(1)
if code:
return unichr(int(code , 10))
else:
code = match.group(2)
if code:
return unichr(int(code , 16))
else:
return unichr(name2cod epoint[match.group(3)])
return EntityPattern.s ub(unescape, s)

>>>escape(
decodeEntities( "This & this &amp; that&nbsp;&eacu te;")).encode(
'ascii', 'xmlcharrefrepl ace')
'This &amp; this &amp; that é'
P.S. apos is handled specially as it isn't technically a
valid html entity (and Python doesn't include it in its entity
list), but it is an xml entity and recognised by many browsers so some
people might use it in html.
Hey i fund this site:
http://www.htmlhelp.com/reference/ht...s/symbols.html

I hope that its what you mean.

/Scripter47

Dec 26 '06 #6
Andreas Lysdal <an*****@ridder garn.dkwrote:
>P.S. apos is handled specially as it isn't technically a
valid html entity (and Python doesn't include it in its entity
list), but it is an xml entity and recognised by many browsers so some
people might use it in html.
Hey i fund this site:
http://www.htmlhelp.com/reference/ht...s/symbols.html

I hope that its what you mean.
Try
http://www.w3.org/TR/html4/sgml/entities.html#entities
for a more complete list.
Dec 26 '06 #7
John Nagle wrote:
Felipe Almeida Lessa wrote:
>On 26 Dec 2006 04:22:38 -0800, placid <Bu****@gmail.c omwrote:

>>So do you want to remove "&" or replace them with "&amp;" ? If you want
to replace it try the following;
I think he wants to replace them, but just the invalid ones. I.e.,

This & this &amp; that

would become

This &amp; this &amp; that
No, i don't know how to do this efficiently. =/...
I think some kind of regex could do it.

Yes, and the appropriate one is:

krefindamp = re.compile(r'&( ?!(\w|#)+;)')
...
xmlsection = re.sub(krefinda mp,'&amp;',xmls ection)

This will replace an '&' with '&amp' if the '&' isn't
immediately followed by some combination of letters, numbers,
and '#' ending with a ';' Admittedly this would let something
like '&xx#2;', which isn't a legal entity, through unmodified.

There's still a potential problem with unknown entities in the output XML, but
at least they're recognized as entities.

John Nagle
Here's another idea:
>>s = '''<htmlhtm tag should not translate
& should be &amp;
&xx#2; isn't a legal entity and should translate
{ is a legal entity and should not translate
</html>
>>import SE # http://cheeseshop.python.org/pypi/SE/2.3
HTM_Escapes = SE.SE (definitions) # See definitions below the
dotted line
>>print HTM_Escapes (s)
<htmlhtm tag should not translate
&gt; &amp; should be &amp;
&gt; &amp;xx#2; isn&quot;t a legal entity and should translate
&gt; { is a legal entity and should not translate
</html>

Regards

Frederic
------------------------------------------------------------------------------
definitions = '''

# Do # Don't do
# " =&nbsp;" &nbsp;== # 32 20
(34)=&dquot; &dquot;== # 34 22
&=&amp; &amp;== # 38 26
'=&quot; &quot;== # 39 27
<=&lt; &lt;== # 60 3c
>=&gt; &gt;== # 62 3e
=&copy; &copy;== # 169 a9
=&middot; &middot;== # 183 b7
=&raquo; &raquo;== # 187 bb
=&Agrave; &Agrave;== # 192 c0
=&Aacute; &Aacute;== # 193 c1
=&Acirc; &Acirc;== # 194 c2
=&Atilde; &Atilde;== # 195 c3
=&Auml; &Auml;== # 196 c4
=&Aring; &Aring;== # 197 c5
=&AElig; &AElig;== # 198 c6
=&Ccedil; &Ccedil;== # 199 c7
=&Egrave; &Egrave;== # 200 c8
=&Eacute; &Eacute;== # 201 c9
=&Ecirc; &Ecirc;== # 202 ca
=&Euml; &Euml;== # 203 cb
=&Igrave; &Igrave;== # 204 cc
=&Iacute; &Iacute;== # 205 cd
=&Icirc; &Icirc;== # 206 ce
=&Iuml; &Iuml;== # 207 cf
=&Eth; &Eth;== # 208 d0
=&Ntilde; &Ntilde;== # 209 d1
=&Ograve; &Ograve;== # 210 d2
=&Oacute; &Oacute;== # 211 d3
=&Ocirc; &Ocirc;== # 212 d4
=&Otilde; &Otilde;== # 213 d5
=&Ouml; &Ouml;== # 214 d6
=&Oslash; &Oslash;== # 216 d8
=&Ugrve; &Ugrve;== # 217 d9
=&Uacute; &Uacute;== # 218 da
=&Ucirc; &Ucirc;== # 219 db
=&Uuml; &Uuml;== # 220 dc
=&Yacute; &Yacute;== # 221 dd
=&Thorn; &Thorn;== # 222 de
=&szlig; &szlig;== # 223 df
=&agrave; &agrave;== # 224 e0
=&aacute; &aacute;== # 225 e1
=&acirc; &acirc;== # 226 e2
=&atilde; &atilde;== # 227 e3
=&auml; &auml;== # 228 e4
=&aring; &aring;== # 229 e5
=&aelig; &aelig;== # 230 e6
=&ccedil; &ccedil;== # 231 e7
=&egrave; &egrave;== # 232 e8
=&eacute; &eacute;== # 233 e9
=&ecirc; &ecirc;== # 234 ea
=&euml; &euml;== # 235 eb
=&igrave; &igrave;== # 236 ec
=&iacute; &iacute;== # 237 ed
=&icirc; &icirc;== # 238 ee
=&iuml; &iuml;== # 239 ef
=&eth; &eth;== # 240 f0
=&ntilde; &ntilde;== # 241 f1
=&ograve; &ograve;== # 242 f2
=&oacute; &oacute;== # 243 f3
=&ocric; &ocric;== # 244 f4
=&otilde; &otilde;== # 245 f5
=&ouml; &ouml;== # 246 f6
=&oslash; &oslash;== # 248 f8
=&ugrave; &ugrave;== # 249 f9
=&uacute; &uacute;== # 250 fa
=&ucirc; &ucirc;== # 251 fb
=&uuml; &uuml;== # 252 fc
=&yacute; &yacute;== # 253 fd
=&thorn; &thorn;== # 254 fe
(xff)=ÿ # 255 ff
&#== # All numeric codes
"~<(.|\n)*?>~== " # All HTM tags '''

If the ampersand is all you need to handle you can erase the others
in the first column. You need to keep the second column though, except
the last entry, because the tags don't need protection if '<' and
'>' in the first column are gone.
Definitions are easily edited and can be kept in text files.
The SE constructor accepts a file name instead of a definitions string:
>>HTM_Escapes = SE.SE ('definition_fi le_name')

-------------------------------------------------------------------

Dec 26 '06 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
3130
by: Steve Young | last post by:
I tried using BeautifulSoup to make changes to the url links on html pages, but when the page was displayed, it was garbled up and didn't look right (even when I didn't actually change anything on the page yet). I ran these steps in python to see what was up: >>from BeautifulSoup import BeautifulSoup >>from urllib2 import build_opener, Request >> >>req = Request('http://www.python.org/')
7
8612
by: Gonzillaaa | last post by:
I'm trying to get the data on the "Central London Property Price Guide" box at the left hand side of this page http://www.findaproperty.com/regi0018.html I have managed to get the data :) but when I start looking for tables I only get tables of depth 1 how do I go about accessing inner tables? same happens for links... this is what I've go so far
4
2738
by: William Xu | last post by:
Hi, all, This piece of code used to work well. i guess the error occurs after some upgrade. >>> import urllib >>> from BeautifulSoup import BeautifulSoup >>> url = 'http://www.google.com' >>> port = urllib.urlopen(url).read() >>> soup = BeautifulSoup()
5
2666
by: John Nagle | last post by:
This, which is from a real web site, went into BeautifulSoup: <param name="movie" value="/images/offersBanners/sw04.swf?binfot=We offer fantastic rates for selected weeks or days!!&blinkt=Click here And this came out, via prettify: <addresssnippet siteurl="http%3A//apartmentsapart.com" url="http%3A//www.apartmentsapart.com/Europe/Spain/Madrid/FAQ"> <param name="movie" value="/images/offersBanners/sw04.swf?binfot=We offer
9
1774
by: Mizipzor | last post by:
Is there a way to "subscribe" to individual topics? im currently getting bombarded with daily digests and i wish to only receive a mail when there is activity in a topic that interests me. Can this be done? Thanks in advance.
6
1898
by: John Nagle | last post by:
Here's a construct with which BeautifulSoup has problems. It's from "http://support.microsoft.com/contactussupport/?ws=support". This is the original: <a href="http://www.microsoft.com/usability/enroll.mspx" id="L_75998" title="<!--http://www.microsoft.com/usability/information.mspx->" onclick="return MS_HandleClick(this,'C_32179', true);">
5
3782
by: Larry Bates | last post by:
Info: Python version: ActivePython 2.5.1.1 Platform: Windows I wanted to install BeautifulSoup today for a small project and decided to use easy_install. I can install other packages just fine. Unfortunately I get the following error from BeautifulSoup installation attempt: C:\Python25\Lib\SITE-P~1>easy_install BeautifulSoup
3
9820
by: Magnus.Moraberg | last post by:
Hi, I wish to extract all the words on a set of webpages and store them in a large dictionary. I then wish to procuce a list with the most common words for the language under consideration. So, my code below reads the page - http://news.bbc.co.uk/welsh/hi/newsid_7420000/newsid_7420900/7420967.stm a welsh language page. I hope to then establish the 1000 most commonly
3
2177
by: bsagert | last post by:
I downloaded BeautifulSoup.py from http://www.crummy.com/software/BeautifulSoup/ and being a n00bie, I just placed it in my Windows c:\python25\lib\ file. When I type "import beautifulsoup" from the interactive prompt it works like a charm. This seemed too easy in retrospect. Then I downloaded the PIL (Python Imaging Library) module from http://www.pythonware.com/products/pil/. Instead of a simple file that BeautifulSoup sent me, PIL is...
0
9892
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, well explore What is ONU, What Is Router, ONU & Routers main usage, and What is the difference between ONU and Router. Lets take a closer look ! Part I. Meaning of...
0
9734
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10991
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10653
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10718
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10347
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7061
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5725
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
2
4129
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.