HTML page into a string

Tempo

In my last post I received some advice to use urllib.read() to get a
whole html page as a string, which will then allow me to use
BeautifulSoup to do what I want with the string. But when I was
researching the 'urllib' module I couldn't find anything about its
sub-section '.read()' ? Is that the right module to get a html page
into a string? Or am I completely missing something here? I'll take
this as the more likely of the two cases. Thanks for any and all help.

Feb 8 '06 #1

Subscribe Post Reply

1566

Steve Holden

Tempo wrote:

In my last post I received some advice to use urllib.read() to get a
whole html page as a string, which will then allow me to use
BeautifulSoup to do what I want with the string. But when I was
researching the 'urllib' module I couldn't find anything about its
sub-section '.read()' ? Is that the right module to get a html page
into a string? Or am I completely missing something here? I'll take
this as the more likely of the two cases. Thanks for any and all help.

I think you've misunderstood. You call urllib.urlopen() with a URL as an
argument. The object that this call returns is file-like (in so far as
you can read it to get the content of the web page):

import urllib
page = urllib.urlopen("http://www.holdenweb.com/")
data = page.read()
print data <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="content-type" content="text/html;charset=ISO-8859-1">
<meta name="generator" content="Adobe GoLive 6">
<meta http-equiv="DESCRIPTION" content="Holden Web provides
architectural design of databases and information systems, with
full-service implementation and support">
...
</tr>
</tbody>
</table>
</div>
</body>
</html>

You will find there are lots of other things you can do with that
file-like object too, but reading it is the important one as far as
using BeautifulSoup goes.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

Feb 8 '06 #2

Jason Earl

"Tempo" <br*******@gmail.com> writes:

In my last post I received some advice to use urllib.read() to get a
whole html page as a string, which will then allow me to use
BeautifulSoup to do what I want with the string. But when I was
researching the 'urllib' module I couldn't find anything about its
sub-section '.read()' ? Is that the right module to get a html page
into a string? Or am I completely missing something here? I'll take
this as the more likely of the two cases. Thanks for any and all help.

Here's a short example of how this all works:

#!/usr/bin/env python

import urllib2
from BeautifulSoup import BeautifulSoup

response = urllib2.urlopen('http://www.cnn.com')
soup = BeautifulSoup(response)
print soup.prettify()

It's not a particularly useful example, unless, of course, you wish to
prettify cnn's html, but it should get you to the point where
BeautifulSoup's documentation starts to make sense.

Jason

Feb 8 '06 #3

Tempo

Perfect. Thanks a bunch for clearing that all up for me. You have
delayed some long lost hours for me.

Feb 8 '06 #4

by: lkrubner | last post by:

I'd like to write a PHP script to be used from the command line on a Unix machine. I'd like for the script to put together a string, turn it into a web page, print it, then return control the...

PHP

How to bring aspx code (in HTML view) to the aspx.vb code-behind?

by: Paolo Pignatelli | last post by:

I have an aspx code behind page that goes something like this in the HTML view: <asp:HyperLink id=HyperLink1 runat="server" NavigateUrl='<%#"mailto:" &...

ASP.NET

Insert Server Control Into HTML String (via Literal) at runtime

by: Guadala Harry | last post by:

In an ASCX, I have a Literal control into which I inject a at runtime. litInjectedContent.Text = dataClass.GetHTMLSnippetFromDB(someID); This works great as long as the contains just...

ASP.NET

Capturing the HTML from an ASP.NET 2.0 page

by: Ken Cox - Microsoft MVP | last post by:

I'm trying to find a way to program in ASP.NET 2.0 but capture the HTML output. I found the following routine in ASP.NET 2.0 Cookbook from O'Reilly. It doesn't work if I include a server-side...

ASP.NET

Best way to dump HTML strings to a page in ASP.Net?

by: mathewda | last post by:

I'm currently work'n on a project where I'm dynamically generating some HTML as text into a string builder. My ASP page has a <span> tag on it and after I gernerate HTML to my string builder I...

ASP.NET

Show HTML String

by: Just D. | last post by:

All, What's the simplest way to show my own HTML string on the ASPX page assuming that this page is just created using the wizard and it has nothing on it? We're free to use any control adding...

ASP.NET

Resizable, sortable, findable and scrollable HTML table

by: since | last post by:

I figured I would post my solution to the following. Resizable column tables. Search and replace values in a table. (IE only) Scrollable tables. Sortable tables. It is based on a lot...

Javascript

Send email with html body using System.net.mail

by: anu b | last post by:

Now i am sending email to my friend using session variable... but my code is as below private bool SendEmail(string email) { try

C# / C Sharp

Send Email with html body

by: anu b | last post by:

Hii I am using System.net.mail for sending email... i need to send a webpage with its html content as my email body .....for that i used mail.Isbodyhtml as true...but it is not working for me...

Visual Basic .NET

Easy Steps to Fix "Canon Printer Won't Connect to WiFi Network"

by: taylorcarr | last post by:

A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...

General

How to turn on java script in a villaon keypad mobile phone

by: Charles Arthur | last post by:

How do i turn on java script on a villaon, callus and itel keypad mobile phone

Java

Batch import of multiple excel files into the database

by: ryjfgjl | last post by:

If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...

Data Management

Merging data from multiple Excel files

by: ryjfgjl | last post by:

In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...

Data Management

Migrating Website to Cloud - Emmanuel Katto

by: emmanuelkatto | last post by:

Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel

General

Navigating the Data Structures and Algorithms (DSA)

by: BarryA | last post by:

What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...

Algorithms / Advanced Math

Looking to do Android software development, any suggestions? Is flutter better?

by: nemocccc | last post by:

hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?

General

How to build RAID in BIOS?

by: Hystou | last post by:

There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...

Computer Hardware

Problem With Comparison Operator <=> in G++

by: Oralloy | last post by:

Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...

C / C++

HTML page into a string

Similar topics