473,573 Members | 2,773 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Using Beautiful Soup to entangle bookmarks.html

Hi,

I'm trying to use the Beautiful Soup package to parse through the
"bookmarks.html " file which Firefox exports all your bookmarks into.
I've been struggling with the documentation trying to figure out how to
extract all the urls. Has anybody got a couple of longer examples using
Beautiful Soup I could play around with?

Thanks,
Martin.

Sep 7 '06 #1
15 5968
Francach schrieb:
Hi,

I'm trying to use the Beautiful Soup package to parse through the
"bookmarks.html " file which Firefox exports all your bookmarks into.
I've been struggling with the documentation trying to figure out how to
extract all the urls. Has anybody got a couple of longer examples using
Beautiful Soup I could play around with?
Why do you use BeautifulSoup on that? It's generated content, and I
suppose it is well-formed, most probably even xml. So use a standard
parser here, better yet somthing like lxml/elementtree

Diez
Sep 7 '06 #2

Diez B. Roggisch wrote:
suppose it is well-formed, most probably even xml.
Maybe not. Otherwise, why would there be a script like this one[1]?
Anyway, I found that and other scripts that work with firefox
bookmarks.html files with a quick search [2]. Perhaps you will find
something there that is helpful.

[1]:
http://www.physic.ut.ee/~kkannike/en...e/bookmarks.py
[2]: http://www.google.com/search?q=firef...ks.html+python

Waylan

Sep 7 '06 #3
Diez B. Roggisch wrote:
Francach schrieb:
>Hi,

I'm trying to use the Beautiful Soup package to parse through the
"bookmarks.htm l" file which Firefox exports all your bookmarks into.
I've been struggling with the documentation trying to figure out how to
extract all the urls. Has anybody got a couple of longer examples using
Beautiful Soup I could play around with?


Why do you use BeautifulSoup on that? It's generated content, and I
suppose it is well-formed, most probably even xml. So use a standard
parser here, better yet somthing like lxml/elementtree

Diez
Once upon a time I have written for my own purposes some code on this
subject, so maybe it can be used as a starter (tested a bit, but
consider its status as a kind of alpha release):

<code>
from urllib import urlopen
from sgmllib import SGMLParser

class mySGMLParserCla ssProvidingList Of_HREFs(SGMLPa rser):
# provides only HREFs <a href="someURL"f or links to another pages skipping
# references to:
# - internal links on same page : "#..."
# - email adresses : "mailto:... "
# and skipping part with appended internal link info, so that e.g.:
# - "LinkSpec#inter nalLinkID" will be listed as "LinkSpec" only
# ---
# reset() overwrites an empty function available in SGMLParser class
def reset(self):
SGMLParser.rese t(self)
self.A_HREFs = []
#: def reset(self)

# start_a() overwrites an empty function available in SGMLParser class
# from which this class is derived. start_a() will be called each
time the
# SGMLParser detects an <a ...tag within the feed(ed) HTML document:
def start_a(self, tagAttributes_a sListOfNameValu ePairs):
for attrName, attrValue in tagAttributes_a sListOfNameValu ePairs:
if attrName=='href ':
if attrValue[0] != '#' and attrValue[:7] !='mailto:':
if attrValue.find( '#') >= 0:
attrValue = attrValue[:attrValue.find ('#')]
#: if
self.A_HREFs.ap pend(attrValue)
#: if
#: if
#: for
#: def start_a(self, attributes_Name sAndValues_AsLi stOfTuples)
#: class mySGMLParserCla ssProvidingList Of_HREFs(SGMLPa rser)
#
------------------------------------------------------------------------------
# ---
# Execution block:
fileLikeObjFrom _urlopen = urlopen('www.go ogle.com') # set URL
mySGMLParserCla ssObj_withListO fHREFs =
mySGMLParserCla ssProvidingList Of_HREFs()
mySGMLParserCla ssObj_withListO fHREFs.feed(fil eLikeObjFrom_ur lopen.read())
mySGMLParserCla ssObj_withListO fHREFs.close()
fileLikeObjFrom _urlopen.close( )

for href in mySGMLParserCla ssObj_withListO fHREFs.A_HREFs:
print href
#: for
</code>

Claudio Grondi
Sep 7 '06 #4
waylan schrieb:
Diez B. Roggisch wrote:
>suppose it is well-formed, most probably even xml.

Maybe not. Otherwise, why would there be a script like this one[1]?
Anyway, I found that and other scripts that work with firefox
bookmarks.html files with a quick search [2]. Perhaps you will find
something there that is helpful.
I have to admit: I didn't check on that file, and simply couldn't
believe it was so badly written as it apparently is.

But I was at least capable of shoving it through HTMLParser. But I'm not
sure if that is of any use.

Excuse me causing confusion.

Diez
Sep 7 '06 #5

Francach wrote:
Hi,

I'm trying to use the Beautiful Soup package to parse through the
"bookmarks.html " file which Firefox exports all your bookmarks into.
I've been struggling with the documentation trying to figure out how to
extract all the urls. Has anybody got a couple of longer examples using
Beautiful Soup I could play around with?

Thanks,
Martin.
If the only thing you want out of the document is the URL's why not
search for: href="..." ? You could get a regular expression that
matches that pretty easily. I think this should just about get you
there, but my regular expressions have gotten very rusty.

/href=\".+\"/

Sep 7 '06 #6
On 7 Sep 2006 14:30:25 -0700, Adam Jones <aj*****@gmail. comwrote:
>
Francach wrote:
Hi,

I'm trying to use the Beautiful Soup package to parse through the
"bookmarks.html " file which Firefox exports all your bookmarks into.
I've been struggling with the documentation trying to figure out how to
extract all the urls. Has anybody got a couple of longer examples using
Beautiful Soup I could play around with?

Thanks,
Martin.

If the only thing you want out of the document is the URL's why not
search for: href="..." ? You could get a regular expression that
matches that pretty easily. I think this should just about get you
there, but my regular expressions have gotten very rusty.

/href=\".+\"/
I doubt the bookmarks file is huge so something simple like

f = open('bookmarks .html').readlin es()
data = [x for x in f if x.strip().start swith('<DT><A ')]

would get you started.

On my exported firefox bookmarks, this gives me all the urls, they
just need to be parsed a bit more accurately, I might be tempted to
just use a couple of splits() to keep it real simple.

HTH
--

Tim Williams
Sep 7 '06 #7
Francach wrote:
Hi,

I'm trying to use the Beautiful Soup package to parse through the
"bookmarks.html " file which Firefox exports all your bookmarks into.
I've been struggling with the documentation trying to figure out how to
extract all the urls. Has anybody got a couple of longer examples using
Beautiful Soup I could play around with?

Thanks,
Martin.
from BeautifulSoup import BeautifulSoup
urls = [tag['href'] for tag in
BeautifulSoup(o pen('bookmarks. html')).findAll ('a')]

Regards,
George

Sep 8 '06 #8
Hi,

thanks for the helpful reply.
I wanted to do two things - learn to use Beautiful Soup and bring out
all the information
in the bookmarks file to import into another application. So I need to
be able to travel down the tree in the bookmarks file. bookmarks seems
to use header tags which can then contain a tags where the href
attributes are. What I don't understand is how to create objects which
can then be used to return the information in the next level of the
tree.

Thanks again,
Martin.

George Sakkis wrote:
Francach wrote:
Hi,

I'm trying to use the Beautiful Soup package to parse through the
"bookmarks.html " file which Firefox exports all your bookmarks into.
I've been struggling with the documentation trying to figure out how to
extract all the urls. Has anybody got a couple of longer examples using
Beautiful Soup I could play around with?

Thanks,
Martin.

from BeautifulSoup import BeautifulSoup
urls = [tag['href'] for tag in
BeautifulSoup(o pen('bookmarks. html')).findAll ('a')]

Regards,
George
Sep 8 '06 #9
Francach wrote:
George Sakkis wrote:
Francach wrote:
Hi,
>
I'm trying to use the Beautiful Soup package to parse through the
"bookmarks.html " file which Firefox exports all your bookmarks into.
I've been struggling with the documentation trying to figure out how to
extract all the urls. Has anybody got a couple of longer examples using
Beautiful Soup I could play around with?
>
Thanks,
Martin.
from BeautifulSoup import BeautifulSoup
urls = [tag['href'] for tag in
BeautifulSoup(o pen('bookmarks. html')).findAll ('a')]
Hi,

thanks for the helpful reply.
I wanted to do two things - learn to use Beautiful Soup and bring out
all the information
in the bookmarks file to import into another application. So I need to
be able to travel down the tree in the bookmarks file. bookmarks seems
to use header tags which can then contain a tags where the href
attributes are. What I don't understand is how to create objects which
can then be used to return the information in the next level of the
tree.

Thanks again,
Martin.
I'm not sure I understand what you want to do. Originally you asked to
extract all urls and BeautifulSoup can do this for you in one line. Why
do you care about intermediate objects or if the anchor tags are nested
under header tags or not ? Read and embrace BeautifulSoup's philosophy:
"You didn't write that awful page. You're just trying to get some data
out of it. Right now, you don't really care what HTML is supposed to
look like."

George

Sep 8 '06 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

13
2887
by: NoSpamThankYouMam | last post by:
I am looking for a product that I am not sure exists. I have bookmarks to webpages in Internet Explorer, Mozilla Firefox, Opera, Netscape Navigator, and on a "Favorite Links" page on my website. Every time I add a bookmarked page, I have to do it five times. Is there a product or combination of products that will synchronize my bookmarks...
15
17862
by: could ildg | last post by:
In re, the punctuation "^" can exclude a single character, but I want to exclude a whole word now. for example I have a string "hi, how are you. hello", I want to extract all the part before the world "hello", I can't use ".*" because "^" only exclude single char "h" or "e" or "l" or "o". Will somebody tell me how to do it? Thanks.
2
2161
by: meyerkp | last post by:
Hi all, I'm trying to extract some information from an html file using beautiful soup. The strings I want get are after br tags, eg: <font size='6'> <br>this info <br>more info <br>and more info </font>
3
1407
by: rh0dium | last post by:
Hi all, I am trying to parse into a dictionary a table and I am having all kinds of fun. Can someone please help me out. What I want is this: dic={'Division Code':'SALS','Employee':'LOO ABLE'} Here is what I have..
1
2715
by: Tempo | last post by:
Heya. I have never used a module/script before, and the first problem I have run into is that I do not know how to install a module/script. I have downloaded Beautiful Soup, but how do I use it in one of my own programs? I know that I use an "include" statement, but do I first need to make a copy of BeautifulSoup.pyc or BeautifulSoup.py into...
0
679
by: Anthra Norell | last post by:
Hi, Martin, SE is a stream editor that does not introduce the overhead and complications of overkill parsing. See if it suits your needs: http://cheeseshop.python.org/pypi/SE/2.2%20beta <EAT # delete all unmatched input "~(?i)<a.*?href.*?>~==\n" # keep hrefs and add a new line
3
3115
by: PicURLPy | last post by:
Hello, I want to extract some image links from different html pages, in particular i want extract those image tags which height values are greater than 200. Is there an elegant way in BeautifulSoup to do this?
3
10693
by: cjl | last post by:
I am learning python and beautiful soup, and I'm stuck. A web page has a table that contains data I would like to scrape. The table has a unique class, so I can use: soup.find("table", {"class": "class_name"}) This isolates the table. So far, so good. Next, this table has a certain number of rows (I won't know ahead of time how many),...
2
5588
by: Alexnb | last post by:
Okay, I have used BeautifulSoup a lot lately, but I am wondering, how do you open a local html file? Usually I do something like this for a url soup = BeautifulSoup(urllib.urlopen('http://www.website.com') but the file extension doesn't work. So how do I open one? -- View this message in context:...
0
7784
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7705
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
8033
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
1
7796
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
8077
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
0
6426
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
0
3734
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
0
3739
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2224
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.