470,591 Members | 2,282 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 470,591 developers. It's quick & easy.

Obtaining Webpage Source with Python

How can I obtain the source of a remote webpage (e.g.
http://www.python.org/index.html) using Python?

Something like:

pyPage = open('http://www.python.org/index.html',r).read()

Obviously that won't work but how can I do something to that effect?
Thanks,
Ryan Kaskel

--I posed this before but it seems it is not showing up...
Jul 18 '05 #1
4 2968
ry**@ryankaskel.com (Ryan Kaskel) writes:
Something like:

pyPage = open('http://www.python.org/index.html',r).read()

Obviously that won't work but how can I do something to that effect?


import urllib
pyPage = urllib.urlopen('http://www.python.org/index.html',r).read()
Jul 18 '05 #2
Paul Rubin <http://ph****@NOSPAM.invalid> writes:
import urllib
pyPage = urllib.urlopen('http://www.python.org/index.html',r).read()


oops:

import urllib
pyPage = urllib.urlopen('http://www.python.org/index.html').read()

i.e. omit the 'r' argument to urlib.urlopen.
Jul 18 '05 #3
pyPage = open('http://www.python.org/index.html',r).read()


using open() for local files and ORLs is called url-fopen and works in
PHP, which is a major security hole, because it even allows one to
include() code files from the web without knowing it, that kind of thing...

python has two functions so you know what you're doing.

If your webpage needs cookies or something, you'll need urllib2

If you wanna parse it afterwards use Htmllib or BeautifulSoup
Jul 18 '05 #4
Take a look at the urllib module:

http://python.org/doc/2.3.3/lib/module-urllib.html

On Wed, Jun 23, 2004 at 10:03:04PM -0700, Ryan Kaskel wrote:
How can I obtain the source of a remote webpage (e.g.
http://www.python.org/index.html) using Python?

Something like:

pyPage = open('http://www.python.org/index.html',r).read()

Obviously that won't work but how can I do something to that effect?
Thanks,
Ryan Kaskel

--I posed this before but it seems it is not showing up...


Jul 18 '05 #5

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

3 posts views Thread by hendedav | last post: by
15 posts views Thread by qwweeeit | last post: by
2 posts views Thread by brianlum | last post: by
6 posts views Thread by Anton Vredegoor | last post: by
6 posts views Thread by Jetus | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.