(Was prevously posted as a followup to something else by accident.)
I'm running a website page through BeautifulSoup. It parses OK
with Python 2.4, but Python 2.5 fails with an exception:
Traceback (most recent call last):
File "./sitetruth/InfoSitePage.py ", line 268, in httpfetch
self.pagetree = BeautifulSoup.B eautifulSoup(si tetext) # parse into tree form
File "./sitetruth/BeautifulSoup.p y", line 1326, in __init__
BeautifulStoneS oup.__init__(se lf, *args, **kwargs)
File "./sitetruth/BeautifulSoup.p y", line 973, in __init__
self._feed()
File "./sitetruth/BeautifulSoup.p y", line 998, in _feed
SGMLParser.feed (self, markup or "")
File "/usr/lib/python2.5/sgmllib.py", line 99, in feed
self.goahead(0)
File "/usr/lib/python2.5/sgmllib.py", line 133, in goahead
k = self.parse_star ttag(i)
File "/usr/lib/python2.5/sgmllib.py", line 291, in parse_starttag
self.finish_sta rttag(tag, attrs)
File "/usr/lib/python2.5/sgmllib.py", line 340, in finish_starttag
self.handle_sta rttag(tag, method, attrs)
File "/usr/lib/python2.5/sgmllib.py", line 376, in handle_starttag
method(attrs)
File "./sitetruth/BeautifulSoup.p y", line 1416, in start_meta
self._feed(self .declaredHTMLEn coding)
File "./sitetruth/BeautifulSoup.p y", line 998, in _feed
SGMLParser.feed (self, markup or "")
File "/usr/lib/python2.5/sgmllib.py", line 99, in feed
self.goahead(0)
File "/usr/lib/python2.5/sgmllib.py", line 133, in goahead
k = self.parse_star ttag(i)
File "/usr/lib/python2.5/sgmllib.py", line 285, in parse_starttag
self._convert_r ef, attrvalue)
UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0xa7 in position 0: ordinal
not in range(128)
The code that's failing is in "_convert_r ef", which is new in Python 2.5.
That function wasn't present in 2.4. I think the code is trying to
handle single quotes inside of double quotes, or something like that.
To replicate, run http://www.bankofamerica.com
or http://www.gm.com
through BeautifulSoup.
Something about this code doesn't like big companies. Web sites of smaller
companies are going through OK.
Also reported as a bug:
[ 1651995 ] sgmllib _convert_ref UnicodeDecodeEr ror exception, new in 2.5
John Nagle 2  1582 
on 05.02.2007 03:49 John Nagle said the following:
(Was prevously posted as a followup to something else by accident.)
I'm running a website page through BeautifulSoup. It parses OK
with Python 2.4, but Python 2.5 fails with an exception:
Traceback (most recent call last):
File "./sitetruth/InfoSitePage.py ", line 268, in httpfetch
self.pagetree = BeautifulSoup.B eautifulSoup(si tetext) # parse into tree form
File "./sitetruth/BeautifulSoup.p y", line 1326, in __init__
BeautifulStoneS oup.__init__(se lf, *args, **kwargs)
File "./sitetruth/BeautifulSoup.p y", line 973, in __init__
self._feed()
File "./sitetruth/BeautifulSoup.p y", line 998, in _feed
SGMLParser.feed (self, markup or "")
File "/usr/lib/python2.5/sgmllib.py", line 99, in feed
self.goahead(0)
File "/usr/lib/python2.5/sgmllib.py", line 133, in goahead
k = self.parse_star ttag(i)
File "/usr/lib/python2.5/sgmllib.py", line 291, in parse_starttag
self.finish_sta rttag(tag, attrs)
File "/usr/lib/python2.5/sgmllib.py", line 340, in finish_starttag
self.handle_sta rttag(tag, method, attrs)
File "/usr/lib/python2.5/sgmllib.py", line 376, in handle_starttag
method(attrs)
File "./sitetruth/BeautifulSoup.p y", line 1416, in start_meta
self._feed(self .declaredHTMLEn coding)
File "./sitetruth/BeautifulSoup.p y", line 998, in _feed
SGMLParser.feed (self, markup or "")
File "/usr/lib/python2.5/sgmllib.py", line 99, in feed
self.goahead(0)
File "/usr/lib/python2.5/sgmllib.py", line 133, in goahead
k = self.parse_star ttag(i)
File "/usr/lib/python2.5/sgmllib.py", line 285, in parse_starttag
self._convert_r ef, attrvalue)
UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0xa7 in position 0: ordinal
not in range(128)
The code that's failing is in "_convert_r ef", which is new in Python 2.5.
That function wasn't present in 2.4. I think the code is trying to
handle single quotes inside of double quotes, or something like that.
To replicate, run
http://www.bankofamerica.com
or
http://www.gm.com
through BeautifulSoup.
Something about this code doesn't like big companies. Web sites of smaller
companies are going through OK.
Also reported as a bug:
[ 1651995 ] sgmllib _convert_ref UnicodeDecodeEr ror exception, new in 2.5
John Nagle
Hi,
I had a similar problem recently and did not have time to file a
bug-report. Thanks for doing that.
The problem is the code that handles entity and character references in
SGMLParser.pars e_starttag. Seems that it is not careful about
unicode/str issues.
My quick'n'dirty workaround was to remove the offending char-entity from
the website before feeding it to Beautifulsoup::
text = text.replace('® ', '') # remove rights reserved sign entity
cheers,
stefan
John Nagle wrote:
(Was prevously posted as a followup to something else by accident.)
I'm running a website page through BeautifulSoup. It parses OK
with Python 2.4, but Python 2.5 fails with an exception:
Traceback (most recent call last):
File "./sitetruth/InfoSitePage.py ", line 268, in httpfetch
self.pagetree = BeautifulSoup.B eautifulSoup(si tetext) # parse into
tree form
File "./sitetruth/BeautifulSoup.p y", line 1326, in __init__
BeautifulStoneS oup.__init__(se lf, *args, **kwargs)
File "./sitetruth/BeautifulSoup.p y", line 973, in __init__
self._feed()
File "./sitetruth/BeautifulSoup.p y", line 998, in _feed
SGMLParser.feed (self, markup or "")
File "/usr/lib/python2.5/sgmllib.py", line 99, in feed
self.goahead(0)
File "/usr/lib/python2.5/sgmllib.py", line 133, in goahead
k = self.parse_star ttag(i)
File "/usr/lib/python2.5/sgmllib.py", line 291, in parse_starttag
self.finish_sta rttag(tag, attrs)
File "/usr/lib/python2.5/sgmllib.py", line 340, in finish_starttag
self.handle_sta rttag(tag, method, attrs)
File "/usr/lib/python2.5/sgmllib.py", line 376, in handle_starttag
method(attrs)
File "./sitetruth/BeautifulSoup.p y", line 1416, in start_meta
self._feed(self .declaredHTMLEn coding)
File "./sitetruth/BeautifulSoup.p y", line 998, in _feed
SGMLParser.feed (self, markup or "")
File "/usr/lib/python2.5/sgmllib.py", line 99, in feed
self.goahead(0)
File "/usr/lib/python2.5/sgmllib.py", line 133, in goahead
k = self.parse_star ttag(i)
File "/usr/lib/python2.5/sgmllib.py", line 285, in parse_starttag
self._convert_r ef, attrvalue)
UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0xa7 in position 0:
ordinal
not in range(128)
The code that's failing is in "_convert_r ef", which is new in
Python 2.5.
That function wasn't present in 2.4. I think the code is trying to
handle single quotes inside of double quotes, or something like that.
To replicate, run
http://www.bankofamerica.com
or
http://www.gm.com
through BeautifulSoup.
Something about this code doesn't like big companies. Web sites of smaller
companies are going through OK.
Also reported as a bug:
[ 1651995 ] sgmllib _convert_ref UnicodeDecodeEr ror exception, new in 2.5
Found the problem and updated the bug report with a fix. But someone
else will have to check it in.
There's a place in SGMLParser where someone assumed that values 0..255
were valid ASCII characters. But in fact the allowed range is 0..127.
The effect is that Unicode strings containing values between 128 and 255
will blow up SGMLParser.
In fact, you can even make this happen with an ASCII
source file by using an HTML entity which has a Unicode representation
between 128 and 255, (such as "§"), then using something
Unicode-oriented like BeautifulSoup on it.
John Nagle This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics | |
by: C. Titus Brown |
last post by:
Hi all,
while playing with PBP/mechanize/ClientForm, I ran into a problem with
the way htmllib.HTMLParser was handling encoded tag attributes.
Specifically, the following HTML was not being handled correctly:
<option value="Small (6")">Small (6)</option>
The 'value' attr was being given the escaped value, not the
|
by: Harlin Seritt |
last post by:
I am trying to use SGMLlib module to extract all links from some data I
pulled from the web (via urllib). I have looked at the documentation
online and can not make sense of it. As a quick example, how would I
get the hyperlinks for an html file?
thanks,
Harlin
|
by: Sakcee |
last post by:
I want to build a simple validator for rss2 feeds, that checks basic
structure and reports channels , items , and their attributes etc.
I have been reading Mark Pilgrims articles on xml.com, diveintopython
and someother stuff on sgmllib, sax.handlers and content handlers,
xml.dom.minidom
why is all of this necessary, what is the difference between all these
libraries, it seems to me that I can parse the rss2 feed with any of
these...
|
by: Richard Hsu |
last post by:
code:-
# Internal -- finish processing of end tag
def finish_endtag(self, tag):
if not tag: # <---- i am confused about this
found = len(self.stack) - 1
if found < 0:
self.unknown_endtag(tag) # <---- and this
return
|
by: Martitza |
last post by:
Hi. I work for a small company (actually in process of forming)
interested in embedding or extending python as part of our commercial
non-open-source product. We have legal counsel, but are interested in
the spirit as well as the letter of the law. Not much seems to have
been written about the python license since version 2, so pointers to
more recent discussions or contacts are appreciated. If this is not
the right place to ask these...
| | |
by: Michael Butscher |
last post by:
Hi,
if I execute the following two lines in Python 2.5 (to feed in a
*unicode* string):
import sgmllib
sgmllib.SGMLParser().feed(u'<a title="teßt"></a>')
|
by: siggi |
last post by:
@Ben Sizer
Hi Ben,
in January I received your message re Pygame and Python 2.5:
As a Python (and programming ) newbie allow me a - certainly naive -
question:
|
by: Eric Anderson |
last post by:
I mainly work in other languages (mostly Ruby lately) but my text
editor (Scribes) is python. With python being everywhere for dynamic
scripting I thought I would read the source to learn the language
better (I've gone through some basic tutorials but I always prefer to
learn from real source).
So right from the start I see the following:
from operator import truth
if truth(argv):
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
| | |
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
|
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| | |
