By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
457,979 Members | 960 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 457,979 IT Pros & Developers. It's quick & easy.

Using Beautiful Soup to entangle bookmarks.html

P: n/a
Hi,

I'm trying to use the Beautiful Soup package to parse through the
"bookmarks.html" file which Firefox exports all your bookmarks into.
I've been struggling with the documentation trying to figure out how to
extract all the urls. Has anybody got a couple of longer examples using
Beautiful Soup I could play around with?

Thanks,
Martin.

Sep 7 '06 #1
Share this Question
Share on Google+
15 Replies


P: n/a
Francach schrieb:
Hi,

I'm trying to use the Beautiful Soup package to parse through the
"bookmarks.html" file which Firefox exports all your bookmarks into.
I've been struggling with the documentation trying to figure out how to
extract all the urls. Has anybody got a couple of longer examples using
Beautiful Soup I could play around with?
Why do you use BeautifulSoup on that? It's generated content, and I
suppose it is well-formed, most probably even xml. So use a standard
parser here, better yet somthing like lxml/elementtree

Diez
Sep 7 '06 #2

P: n/a

Diez B. Roggisch wrote:
suppose it is well-formed, most probably even xml.
Maybe not. Otherwise, why would there be a script like this one[1]?
Anyway, I found that and other scripts that work with firefox
bookmarks.html files with a quick search [2]. Perhaps you will find
something there that is helpful.

[1]:
http://www.physic.ut.ee/~kkannike/en...e/bookmarks.py
[2]: http://www.google.com/search?q=firef...ks.html+python

Waylan

Sep 7 '06 #3

P: n/a
Diez B. Roggisch wrote:
Francach schrieb:
>Hi,

I'm trying to use the Beautiful Soup package to parse through the
"bookmarks.html" file which Firefox exports all your bookmarks into.
I've been struggling with the documentation trying to figure out how to
extract all the urls. Has anybody got a couple of longer examples using
Beautiful Soup I could play around with?


Why do you use BeautifulSoup on that? It's generated content, and I
suppose it is well-formed, most probably even xml. So use a standard
parser here, better yet somthing like lxml/elementtree

Diez
Once upon a time I have written for my own purposes some code on this
subject, so maybe it can be used as a starter (tested a bit, but
consider its status as a kind of alpha release):

<code>
from urllib import urlopen
from sgmllib import SGMLParser

class mySGMLParserClassProvidingListOf_HREFs(SGMLParser) :
# provides only HREFs <a href="someURL"for links to another pages skipping
# references to:
# - internal links on same page : "#..."
# - email adresses : "mailto:..."
# and skipping part with appended internal link info, so that e.g.:
# - "LinkSpec#internalLinkID" will be listed as "LinkSpec" only
# ---
# reset() overwrites an empty function available in SGMLParser class
def reset(self):
SGMLParser.reset(self)
self.A_HREFs = []
#: def reset(self)

# start_a() overwrites an empty function available in SGMLParser class
# from which this class is derived. start_a() will be called each
time the
# SGMLParser detects an <a ...tag within the feed(ed) HTML document:
def start_a(self, tagAttributes_asListOfNameValuePairs):
for attrName, attrValue in tagAttributes_asListOfNameValuePairs:
if attrName=='href':
if attrValue[0] != '#' and attrValue[:7] !='mailto:':
if attrValue.find('#') >= 0:
attrValue = attrValue[:attrValue.find('#')]
#: if
self.A_HREFs.append(attrValue)
#: if
#: if
#: for
#: def start_a(self, attributes_NamesAndValues_AsListOfTuples)
#: class mySGMLParserClassProvidingListOf_HREFs(SGMLParser)
#
------------------------------------------------------------------------------
# ---
# Execution block:
fileLikeObjFrom_urlopen = urlopen('www.google.com') # set URL
mySGMLParserClassObj_withListOfHREFs =
mySGMLParserClassProvidingListOf_HREFs()
mySGMLParserClassObj_withListOfHREFs.feed(fileLike ObjFrom_urlopen.read())
mySGMLParserClassObj_withListOfHREFs.close()
fileLikeObjFrom_urlopen.close()

for href in mySGMLParserClassObj_withListOfHREFs.A_HREFs:
print href
#: for
</code>

Claudio Grondi
Sep 7 '06 #4

P: n/a
waylan schrieb:
Diez B. Roggisch wrote:
>suppose it is well-formed, most probably even xml.

Maybe not. Otherwise, why would there be a script like this one[1]?
Anyway, I found that and other scripts that work with firefox
bookmarks.html files with a quick search [2]. Perhaps you will find
something there that is helpful.
I have to admit: I didn't check on that file, and simply couldn't
believe it was so badly written as it apparently is.

But I was at least capable of shoving it through HTMLParser. But I'm not
sure if that is of any use.

Excuse me causing confusion.

Diez
Sep 7 '06 #5

P: n/a

Francach wrote:
Hi,

I'm trying to use the Beautiful Soup package to parse through the
"bookmarks.html" file which Firefox exports all your bookmarks into.
I've been struggling with the documentation trying to figure out how to
extract all the urls. Has anybody got a couple of longer examples using
Beautiful Soup I could play around with?

Thanks,
Martin.
If the only thing you want out of the document is the URL's why not
search for: href="..." ? You could get a regular expression that
matches that pretty easily. I think this should just about get you
there, but my regular expressions have gotten very rusty.

/href=\".+\"/

Sep 7 '06 #6

P: n/a
On 7 Sep 2006 14:30:25 -0700, Adam Jones <aj*****@gmail.comwrote:
>
Francach wrote:
Hi,

I'm trying to use the Beautiful Soup package to parse through the
"bookmarks.html" file which Firefox exports all your bookmarks into.
I've been struggling with the documentation trying to figure out how to
extract all the urls. Has anybody got a couple of longer examples using
Beautiful Soup I could play around with?

Thanks,
Martin.

If the only thing you want out of the document is the URL's why not
search for: href="..." ? You could get a regular expression that
matches that pretty easily. I think this should just about get you
there, but my regular expressions have gotten very rusty.

/href=\".+\"/
I doubt the bookmarks file is huge so something simple like

f = open('bookmarks.html').readlines()
data = [x for x in f if x.strip().startswith('<DT><A ')]

would get you started.

On my exported firefox bookmarks, this gives me all the urls, they
just need to be parsed a bit more accurately, I might be tempted to
just use a couple of splits() to keep it real simple.

HTH
--

Tim Williams
Sep 7 '06 #7

P: n/a
Francach wrote:
Hi,

I'm trying to use the Beautiful Soup package to parse through the
"bookmarks.html" file which Firefox exports all your bookmarks into.
I've been struggling with the documentation trying to figure out how to
extract all the urls. Has anybody got a couple of longer examples using
Beautiful Soup I could play around with?

Thanks,
Martin.
from BeautifulSoup import BeautifulSoup
urls = [tag['href'] for tag in
BeautifulSoup(open('bookmarks.html')).findAll('a')]

Regards,
George

Sep 8 '06 #8

P: n/a
Hi,

thanks for the helpful reply.
I wanted to do two things - learn to use Beautiful Soup and bring out
all the information
in the bookmarks file to import into another application. So I need to
be able to travel down the tree in the bookmarks file. bookmarks seems
to use header tags which can then contain a tags where the href
attributes are. What I don't understand is how to create objects which
can then be used to return the information in the next level of the
tree.

Thanks again,
Martin.

George Sakkis wrote:
Francach wrote:
Hi,

I'm trying to use the Beautiful Soup package to parse through the
"bookmarks.html" file which Firefox exports all your bookmarks into.
I've been struggling with the documentation trying to figure out how to
extract all the urls. Has anybody got a couple of longer examples using
Beautiful Soup I could play around with?

Thanks,
Martin.

from BeautifulSoup import BeautifulSoup
urls = [tag['href'] for tag in
BeautifulSoup(open('bookmarks.html')).findAll('a')]

Regards,
George
Sep 8 '06 #9

P: n/a
Francach wrote:
George Sakkis wrote:
Francach wrote:
Hi,
>
I'm trying to use the Beautiful Soup package to parse through the
"bookmarks.html" file which Firefox exports all your bookmarks into.
I've been struggling with the documentation trying to figure out how to
extract all the urls. Has anybody got a couple of longer examples using
Beautiful Soup I could play around with?
>
Thanks,
Martin.
from BeautifulSoup import BeautifulSoup
urls = [tag['href'] for tag in
BeautifulSoup(open('bookmarks.html')).findAll('a')]
Hi,

thanks for the helpful reply.
I wanted to do two things - learn to use Beautiful Soup and bring out
all the information
in the bookmarks file to import into another application. So I need to
be able to travel down the tree in the bookmarks file. bookmarks seems
to use header tags which can then contain a tags where the href
attributes are. What I don't understand is how to create objects which
can then be used to return the information in the next level of the
tree.

Thanks again,
Martin.
I'm not sure I understand what you want to do. Originally you asked to
extract all urls and BeautifulSoup can do this for you in one line. Why
do you care about intermediate objects or if the anchor tags are nested
under header tags or not ? Read and embrace BeautifulSoup's philosophy:
"You didn't write that awful page. You're just trying to get some data
out of it. Right now, you don't really care what HTML is supposed to
look like."

George

Sep 8 '06 #10

P: n/a
Hi George,

Firefox lets you group the bookmarks along with other information into
directories and sub-directories. Firefox uses header tags for this
purpose. I'd like to get this grouping information out aswell.

Regards,
Martin.
the idea is to extract.
George Sakkis wrote:
Francach wrote:
George Sakkis wrote:
Francach wrote:
Hi,

I'm trying to use the Beautiful Soup package to parse through the
"bookmarks.html" file which Firefox exports all your bookmarks into.
I've been struggling with the documentation trying to figure out how to
extract all the urls. Has anybody got a couple of longer examples using
Beautiful Soup I could play around with?

Thanks,
Martin.
>
from BeautifulSoup import BeautifulSoup
urls = [tag['href'] for tag in
BeautifulSoup(open('bookmarks.html')).findAll('a')]
Hi,

thanks for the helpful reply.
I wanted to do two things - learn to use Beautiful Soup and bring out
all the information
in the bookmarks file to import into another application. So I need to
be able to travel down the tree in the bookmarks file. bookmarks seems
to use header tags which can then contain a tags where the href
attributes are. What I don't understand is how to create objects which
can then be used to return the information in the next level of the
tree.

Thanks again,
Martin.

I'm not sure I understand what you want to do. Originally you asked to
extract all urls and BeautifulSoup can do this for you in one line. Why
do you care about intermediate objects or if the anchor tags are nested
under header tags or not ? Read and embrace BeautifulSoup's philosophy:
"You didn't write that awful page. You're just trying to get some data
out of it. Right now, you don't really care what HTML is supposed to
look like."

George
Sep 8 '06 #11

P: n/a
Francach wrote:
>
Firefox lets you group the bookmarks along with other information into
directories and sub-directories. Firefox uses header tags for this
purpose. I'd like to get this grouping information out aswell.
import libxml2dom # http://www.python.org/pypi/libxml2dom
d = libxml2dom.parse("bookmarks.html", html=1)
for node in d.xpath("html/body//dt/*[1]"):
if node.localName == "h3":
print "Section:", node.nodeValue
elif node.localName == "a":
print "Link:", node.getAttribute("href")

One exercise, using the above code as a starting point, would be to
reproduce the hierarchy exactly, rather than just showing the section
names and the links which follow them. Ultimately, you may be looking
for a way to just convert the HTML into a simple XML document or into
another hierarchical representation which excludes the HTML baggage and
details irrelevant to your problem.

Paul

Sep 8 '06 #12

P: n/a
Francach wrote:
Hi George,

Firefox lets you group the bookmarks along with other information into
directories and sub-directories. Firefox uses header tags for this
purpose. I'd like to get this grouping information out aswell.

Regards,
Martin.
Here's what I came up with:
http://rafb.net/paste/results/G91EAo70.html. Tested only on my
bookmarks; see if it works for you.

For each subfolder there is a recursive call that walks the respective
subtree, so it's probably not the most efficient solution, but I
couldn't think of any one-pass way to do it using BeautifulSoup.

George

Sep 8 '06 #13

P: n/a
Hallo George,

thanks a lot! This is exactly the direction I had in mind.
Your script demonstrates nicely how Beautiful Soup works.

Regards,
Martin.

George Sakkis wrote:
Francach wrote:
Hi George,

Firefox lets you group the bookmarks along with other information into
directories and sub-directories. Firefox uses header tags for this
purpose. I'd like to get this grouping information out aswell.

Regards,
Martin.

Here's what I came up with:
http://rafb.net/paste/results/G91EAo70.html. Tested only on my
bookmarks; see if it works for you.

For each subfolder there is a recursive call that walks the respective
subtree, so it's probably not the most efficient solution, but I
couldn't think of any one-pass way to do it using BeautifulSoup.

George
Sep 9 '06 #14

P: n/a
"George Sakkis" <ge***********@gmail.comwrote:
>Here's what I came up with:
http://rafb.net/paste/results/G91EAo70.html. Tested only on my
bookmarks; see if it works for you.
That URL is dead. Got another?

-----
robin
noisetheatre.blogspot.com
Sep 21 '06 #15

P: n/a
robin wrote:
"George Sakkis" <ge***********@gmail.comwrote:
Here's what I came up with:
http://rafb.net/paste/results/G91EAo70.html. Tested only on my
bookmarks; see if it works for you.

That URL is dead. Got another?
Yeap, try this one:
http://gsakkis-utils.googlecode.com/...s/bookmarks.py

George

Sep 21 '06 #16

This discussion thread is closed

Replies have been disabled for this discussion.