473,883 Members | 2,177 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

HTML page into a string

In my last post I received some advice to use urllib.read() to get a
whole html page as a string, which will then allow me to use
BeautifulSoup to do what I want with the string. But when I was
researching the 'urllib' module I couldn't find anything about its
sub-section '.read()' ? Is that the right module to get a html page
into a string? Or am I completely missing something here? I'll take
this as the more likely of the two cases. Thanks for any and all help.

Feb 8 '06 #1
3 1585
Tempo wrote:
In my last post I received some advice to use urllib.read() to get a
whole html page as a string, which will then allow me to use
BeautifulSoup to do what I want with the string. But when I was
researching the 'urllib' module I couldn't find anything about its
sub-section '.read()' ? Is that the right module to get a html page
into a string? Or am I completely missing something here? I'll take
this as the more likely of the two cases. Thanks for any and all help.

I think you've misunderstood. You call urllib.urlopen( ) with a URL as an
argument. The object that this call returns is file-like (in so far as
you can read it to get the content of the web page):
import urllib
page = urllib.urlopen( "http://www.holdenweb.c om/")
data = page.read()
print data <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="content-type" content="text/html;charset=IS O-8859-1">
<meta name="generator " content="Adobe GoLive 6">
<meta http-equiv="DESCRIPT ION" content="Holden Web provides
architectural design of databases and information systems, with
full-service implementation and support">
...
</tr>
</tbody>
</table>
</div>
</body>
</html>


You will find there are lots of other things you can do with that
file-like object too, but reading it is the important one as far as
using BeautifulSoup goes.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

Feb 8 '06 #2
"Tempo" <br*******@gmai l.com> writes:
In my last post I received some advice to use urllib.read() to get a
whole html page as a string, which will then allow me to use
BeautifulSoup to do what I want with the string. But when I was
researching the 'urllib' module I couldn't find anything about its
sub-section '.read()' ? Is that the right module to get a html page
into a string? Or am I completely missing something here? I'll take
this as the more likely of the two cases. Thanks for any and all help.

Here's a short example of how this all works:

#!/usr/bin/env python

import urllib2
from BeautifulSoup import BeautifulSoup

response = urllib2.urlopen ('http://www.cnn.com')
soup = BeautifulSoup(r esponse)
print soup.prettify()

It's not a particularly useful example, unless, of course, you wish to
prettify cnn's html, but it should get you to the point where
BeautifulSoup's documentation starts to make sense.

Jason
Feb 8 '06 #3
Perfect. Thanks a bunch for clearing that all up for me. You have
delayed some long lost hours for me.

Feb 8 '06 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
3186
by: lkrubner | last post by:
I'd like to write a PHP script to be used from the command line on a Unix machine. I'd like for the script to put together a string, turn it into a web page, print it, then return control the command line. Obviously it is quite easy to make a string that is valid HTML and send it to the printer. The tough part is making look like a web page. In particular, image tags should print out as images, not image tags. Does anyone have an idea...
6
5528
by: Paolo Pignatelli | last post by:
I have an aspx code behind page that goes something like this in the HTML view: <asp:HyperLink id=HyperLink1 runat="server" NavigateUrl='<%#"mailto:" &amp; DataBinder.Eval(Container.DataItem,"StoreEmail") &amp; "&amp;Subject=" &amp; DataBinder.Eval(Container.DataItem,"ProductName") ....
20
5677
by: Guadala Harry | last post by:
In an ASCX, I have a Literal control into which I inject a at runtime. litInjectedContent.Text = dataClass.GetHTMLSnippetFromDB(someID); This works great as long as the contains just client-side HTML, CSS, etc. What I want to do is somehow insert a *server control* into the , then set the server control's properties at runtime.
2
1733
by: Ken Cox - Microsoft MVP | last post by:
I'm trying to find a way to program in ASP.NET 2.0 but capture the HTML output. I found the following routine in ASP.NET 2.0 Cookbook from O'Reilly. It doesn't work if I include a server-side dropdownlist control on the page. The error is RegisterForEventValidation can only be called during Render(); Any ideas? Ken Microsoft MVP
3
1984
by: mathewda | last post by:
I'm currently work'n on a project where I'm dynamically generating some HTML as text into a string builder. My ASP page has a <span> tag on it and after I gernerate HTML to my string builder I concat all the strings and set the span's innerHTML property to this output. What I'm wondering is if this is the best way to do this or is there a better method to take a bunch of strings and dump them out onto an ASP page so it shows up as...
3
2893
by: Just D. | last post by:
All, What's the simplest way to show my own HTML string on the ASPX page assuming that this page is just created using the wizard and it has nothing on it? We're free to use any control adding it to this page just to show this HTML string. I know that in some cases we can use .InnerHtml property of some controls to inject the HTML string created in C# code, but it didn't work in my case. I tried to use TEXTAREA1.InnerHtml and it shows me...
1
16989
by: since | last post by:
I figured I would post my solution to the following. Resizable column tables. Search and replace values in a table. (IE only) Scrollable tables. Sortable tables. It is based on a lot examples I found on the web. Works in IE and mozilla. http://www.imaputz.com/cssStuff/bigFourVersion.html
2
18184
by: anu b | last post by:
Now i am sending email to my friend using session variable... but my code is as below private bool SendEmail(string email) { try
3
7248
by: anu b | last post by:
Hii I am using System.net.mail for sending email... i need to send a webpage with its html content as my email body .....for that i used mail.Isbodyhtml as true...but it is not working for me so pls help me....
0
9945
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9798
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
11160
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10766
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
9588
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
5807
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
6007
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4622
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
3
3241
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.