By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
424,963 Members | 1,261 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 424,963 IT Pros & Developers. It's quick & easy.

urllib.request

100+
P: 103
I do not understand what additional parameters i have to pas to urllib.request to download a full webpage.

1)

For instance, if you open
http://www.vocabulary.com/dictionary/reiterate
on the right bottom you will find "usage examples".

If you try to save wabpage and than reopen it:
file:///C:/Users/Gintare/Downloads/reiterate%20-%20Dictionary%20Definition%20%20%20Vocabulary.com. htm
the "usage examples" will be displayed.

But if you use:
Expand|Select|Wrap|Line Numbers
  1. res="http://www.vocabulary.com/dictionary/reiterate"
  2. resp = urllib.request.urlopen(res)
  3. html = resp.read()             
  4. soup=BeautifulSoup(html)                          shtml=BeautifulSoup.prettify(soup)                           fFI=open('requested_file.html','w',encoding='utf-8')
  5. fFI.writelines(shtml)
  6. fFI.close()
The 'requested_file.html' will not have the section "usage examples".

How to download full webpage with the section "usage examples" using urllib.request?

2) How to retrieve other examples, which normally are accessible by pressing hyperlink "next". There are more than 40 examples for this word, but hey are displayed only in portions by 4.
Mar 28 '15 #1
Share this Question
Share on Google+
2 Replies


numberwhun
Expert Mod 2.5K+
P: 3,503
Have you tried simply using:

Expand|Select|Wrap|Line Numbers
  1. urllib.urlopen
Take out the 'request' portion and try that instead.
Apr 7 '15 #2

100+
P: 103
I am getting error:
AttributeError at /search1

'module' object has no attribute 'urlopen'

Request Method: GET
Request URL: http://127.0.0.1:8000/search1
Django Version: 1.6.2
Exception Type: AttributeError
Exception Value:


on line 1000: resp = urllib.urlopen(res)
Apr 7 '15 #3

Post your reply

Sign in to post your reply or Sign up for a free account.