469,276 Members | 1,488 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,276 developers. It's quick & easy.

urllib.request

103 100+
I do not understand what additional parameters i have to pas to urllib.request to download a full webpage.

1)

For instance, if you open
http://www.vocabulary.com/dictionary/reiterate
on the right bottom you will find "usage examples".

If you try to save wabpage and than reopen it:
file:///C:/Users/Gintare/Downloads/reiterate%20-%20Dictionary%20Definition%20%20%20Vocabulary.com. htm
the "usage examples" will be displayed.

But if you use:
Expand|Select|Wrap|Line Numbers
  1. res="http://www.vocabulary.com/dictionary/reiterate"
  2. resp = urllib.request.urlopen(res)
  3. html = resp.read()             
  4. soup=BeautifulSoup(html)                          shtml=BeautifulSoup.prettify(soup)                           fFI=open('requested_file.html','w',encoding='utf-8')
  5. fFI.writelines(shtml)
  6. fFI.close()
The 'requested_file.html' will not have the section "usage examples".

How to download full webpage with the section "usage examples" using urllib.request?

2) How to retrieve other examples, which normally are accessible by pressing hyperlink "next". There are more than 40 examples for this word, but hey are displayed only in portions by 4.
Mar 28 '15 #1
2 1693
numberwhun
3,503 Expert Mod 2GB
Have you tried simply using:

Expand|Select|Wrap|Line Numbers
  1. urllib.urlopen
Take out the 'request' portion and try that instead.
Apr 7 '15 #2
gintare
103 100+
I am getting error:
AttributeError at /search1

'module' object has no attribute 'urlopen'

Request Method: GET
Request URL: http://127.0.0.1:8000/search1
Django Version: 1.6.2
Exception Type: AttributeError
Exception Value:


on line 1000: resp = urllib.urlopen(res)
Apr 7 '15 #3

Post your reply

Sign in to post your reply or Sign up for a free account.

Similar topics

1 post views Thread by Chris Lyon | last post: by
11 posts views Thread by Pater Maximus | last post: by
reply views Thread by Pieter Edelman | last post: by
3 posts views Thread by Chris Tavares | last post: by
8 posts views Thread by Gabriel Zachmann | last post: by
1 post views Thread by oyster | last post: by
reply views Thread by zhoujie | last post: by
reply views Thread by suresh191 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.