471,873 Members | 1,767 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,873 software developers and data experts.

Python and decimal character entities over 128.

Some web feeds use decimal character entities that seem to confuse
Python (or me). For example, the string "doesn't" may be coded as
"doesn’t" which should produce a right leaning apostrophe.
Python hates decimal entities beyond 128 so it chokes unless you do
something like string.encode('utf-8'). Even then, what should have
been a right-leaning apostrophe ends up as "’". The following script
does just that. Look for the string "The Canuck iPhone: Apple doesn
t care" after running it.

# coding: UTF-8
import feedparser

s = ''
d = feedparser.parse('http://feeds.feedburner.com/Mathewingramcom/
work')
title = d.feed.title
link = d.feed.link
for i in range(0,4):
title = d.entries[i].title
link = d.entries[i].link
s += title +'\n' + link + '\n'

f = open('c:/x/test.txt', 'w')
f.write(s.encode('utf-8'))
f.close()

This useless script is adapted from a "useful" script. Its only
purpose is to ask the Python community how I can deal with decimal
entities 128. Thanks in advance, Bill
Jul 9 '08 #1
3 1933
On Wed, 09 Jul 2008 16:39:24 -0700, bsagert wrote:
Some web feeds use decimal character entities that seem to confuse
Python (or me).
I guess they confuse you. Python is fine.
For example, the string "doesn't" may be coded as "doesn’t" which
should produce a right leaning apostrophe. Python hates decimal entities
beyond 128 so it chokes unless you do something like
string.encode('utf-8').
Python doesn't hate nor chokes on these entities. It just refuses to
guess which encoding you want, if you try to write *unicode* objects into
a file. Files contain byte values not characters.
Even then, what should have been a right-leaning apostrophe ends up as
"’". The following script does just that. Look for the string "The
Canuck iPhone: Apple doesnâ €™t care" after running it.
Then you didn't tell the application you used to look at the result, that
the text is UTF-8 encoded. I guess you are using Windows and
the application expects cp1252 encoded text because an UTF-8 encoded
apostrophe looks like '’' in cp1252.

Choose the encoding you want the result to have and anything is fine.
Unless you stumble over a feed using characters which can't be encoded
in the encoding of your choice. That's why UTF-8 might have been a good
idea.

Ciao,
Marc 'BlackJack' Rintsch
Jul 10 '08 #2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

bs*****@gmail.com wrote:
Some web feeds use decimal character entities that seem to confuse
Python (or me). For example, the string "doesn't" may be coded as
"doesn’t" which should produce a right leaning apostrophe.
Python hates decimal entities beyond 128 so it chokes unless you do
something like string.encode('utf-8'). Even then, what should have
been a right-leaning apostrophe ends up as "’". The following script
does just that. Look for the string "The Canuck iPhone: Apple doesnâ
€™t care" after running it.

# coding: UTF-8
import feedparser

s = ''
d = feedparser.parse('http://feeds.feedburner.com/Mathewingramcom/
work')
title = d.feed.title
link = d.feed.link
for i in range(0,4):
title = d.entries[i].title
link = d.entries[i].link
s += title +'\n' + link + '\n'

f = open('c:/x/test.txt', 'w')
f.write(s.encode('utf-8'))
f.close()

This useless script is adapted from a "useful" script. Its only
purpose is to ask the Python community how I can deal with decimal
entities 128. Thanks in advance, Bill
--
http://mail.python.org/mailman/listinfo/python-list
This is a two-fold issue: encodings/charsets and entities. Encodings are
a way to _encode_ charsets to a sequence of octets. Entities are a way
to avoid a (harder) encoding/decoding process at the expense of
readability: when you type #8217; no one actually see the intended
character, but those are easily encoded in ascii.

When dealing with multiples sources of information, like your script may
be, I always include a middleware of normalization to Python's Unicode
Type. Web sites may use whatever encoding they please.

The whole process is like this:
1. Fetch the content
2. Use whatever clue in the contents to guess the encoding used by the
document, e.g Content-type HTTP header; <meta http-equiv="content-type"
....>; <?xml version="1.0" encoding="utf-8"?>, and so on.
3. If none are present, then use chardet to guess for an acceptable decoder.
4. Decode ignoring those character that cannot be decoded.
5. The result is further processed to find entities and "decode" them to
actual Unicode characters. (See below)

You may find these helpful:
http://effbot.org/zone/unicode-objects.htm
http://www.mozilla.org/projects/intl...Detection.html
http://www.amk.ca/python/howto/unicode

This is function I have used to process entities:
Expand|Select|Wrap|Line Numbers
  1. from htmlentitydefs import name2codepoint
  2. def __processhtmlentities__(text):
  3. assert type(text) is unicode, "Non-normalized text"
  4. html = []
  5. (buffer, amp, text) = text.partition('&')
  6. while amp:
  7. html.append(buffer)
  8. (entity, semicolon, text) = text.partition(';')
  9. if entity[0] != '#':
  10. if entity in name2codepoint:
  11. html.append(unichr(name2codepoint[entity]))
  12. else:
  13. html.append(int(entity[1:])))
  14. (buffer, amp, text) = text.partition('&')
  15. html.append(buffer)
  16. return u''.join(html)
  17.  

Best regards,
Manuel.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkh2S+sACgkQI2zpkmcEAhil6gCgkAnRE4s5b8 oQHamk6utkbAl7
m+YAoIZH2/u73hDcs0G/u294use27v17
=mXuK
-----END PGP SIGNATURE-----
Jul 10 '08 #3
I don't have an answer for why Python might be mis-handling the data,
but wanted to make a factual correction:

bs*****@gmail.com writes:
Some web feeds use decimal character entities that seem to confuse
Python (or me). For example, the string "doesn't" may be coded as
"doesn’t" which should produce a right leaning apostrophe.
That character isn't a "right leaning apostrophe"; it has nothing to
do with apostrophes. It is the character called "right single
quotation mark" in <URL:http://www.w3.org/TR/html4/sgml/entities.html>
and in Unicode (code point U+2019).

It's a typographical error to use a quotation mark as an apostrophe.
Use the apostrophe character (U+0027) where an apostrophe is intended,
and quotation mark characters where those are intended.

This is directed, of course, at the person generating that output.

--
\ “If you go to a costume party at your boss's house, wouldn't |
`\ you think a good costume would be to dress up like the boss's |
_o__) wife? Trust me, it's not.” —Jack Handey |
Ben Finney
Jul 10 '08 #4

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

19 posts views Thread by Ian | last post: by
2 posts views Thread by kamp | last post: by
1 post views Thread by suresh | last post: by
4 posts views Thread by lianciana | last post: by
reply views Thread by Mike McGranahan | last post: by
1 post views Thread by Tony | last post: by
4 posts views Thread by Paul Rubin | last post: by
reply views Thread by YellowAndGreen | last post: by
reply views Thread by zermasroor | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.