473,708 Members | 2,458 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Can not get urllib.urlopen to work

I am trying to implement the recipe listed at
http://aspn.activestate.com/ASPN/Coo.../Recipe/211886

However, I can not get to first base. When I try to run

import urllib
fo=urllib.urlop en("http://www.dictionary. com/")
page = fo.read()

I get:

Traceback (most recent call last):
File "C:/Program Files/Python/Lib/idlelib/testurl", line 2, in -toplevel-
fo=urllib.urlop en("http://www.dictionary. com/")
File "C:\PROGRA~1\PY THON\lib\urllib .py", line 76, in urlopen
return opener.open(url )
File "C:\PROGRA~1\PY THON\lib\urllib .py", line 181, in open
return getattr(self, name)(url)
File "C:\PROGRA~1\PY THON\lib\urllib .py", line 297, in open_http
h.endheaders()
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 712, in endheaders
self._send_outp ut()
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 597, in _send_output
self.send(msg)
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 564, in send
self.connect()
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 548, in connect
raise socket.error, msg
IOError: [Errno socket error] (10061, 'Connection refused')
Jul 18 '05 #1
11 5050
More information:

I am running Python 2.3.4, Windows 2000 Pro, along with Norton Internet
Security Professional. I do not get any messages from Norton.
Jul 18 '05 #2
Pater Maximus wrote:
I am trying to implement the recipe listed at
http://aspn.activestate.com/ASPN/Coo.../Recipe/211886

However, I can not get to first base. When I try to run

import urllib
fo=urllib.urlop en("http://www.dictionary. com/")
page = fo.read()

I get:
Traceback (most recent call last):
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 548, in connect
raise socket.error, msg
IOError: [Errno socket error] (10061, 'Connection refused')


Do you have any reason to think that this site is currently
hosting a web server?

Attempting to connect with IE or Telnet on port 80 results in
a similar response.

(Telnet is always a good thing to try, before posting about
server problems...)

-Peter
Jul 18 '05 #3

"Pater Maximus" <em***@home.com > wrote in message
news:3J******** ************@co mcast.com...
IOError: [Errno socket error] (10061, 'Connection refused')


My browser says the same thing. ;-)
Alan Isaac
Jul 18 '05 #4
Pater Maximus wrote:
I am trying to implement the recipe listed at
http://aspn.activestate.com/ASPN/Coo.../Recipe/211886

However, I can not get to first base. When I try to run

import urllib
fo=urllib.urlop en("http://www.dictionary. com/")
page = fo.read()


I can't even connect to it with my web browser. Can you?
If you can, they are probably checking the user-agent sent
by urllib, to make it harder to do this sort of automated
screen scraping.

See the docs at
http://www.python.org/doc/current/li...le-urllib.html

for an example of how to change the default user-agent.

Here's one for MSIE under Win2K

Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)

Andrew
da***@dalkescie ntific.com
Jul 18 '05 #5
>I am trying to implement the recipe listed at
http://aspn.activestate.com/ASPN/Coo.../Recipe/211886

However, I can not get to first base. When I try to run

import urllib
fo=urllib.urlop en("http://www.dictionary. com/")
page = fo.read()

I get:

Traceback (most recent call last):
File "C:/Program Files/Python/Lib/idlelib/testurl", line 2, in -toplevel-
fo=urllib.urlop en("http://www.dictionary. com/")
File "C:\PROGRA~1\PY THON\lib\urllib .py", line 76, in urlopen
return opener.open(url )
File "C:\PROGRA~1\PY THON\lib\urllib .py", line 181, in open
return getattr(self, name)(url)
File "C:\PROGRA~1\PY THON\lib\urllib .py", line 297, in open_http
h.endheaders()
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 712, in endheaders
self._send_outp ut()
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 597, in _send_output
self.send(msg)
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 564, in send
self.connect()
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 548, in connect
raise socket.error, msg
IOError: [Errno socket error] (10061, 'Connection refused')


Connection refused is the key. Apparently dictionary.com figured out that
people were trying to use their resourses without giving them credit. I
have done some research on how to accomplish this so that people can not use
the cgi-bin programs (and others) that I write.

A few years back I needed to get information about the weather based on zip
code. I used to use weather.com but they fixed the hole. They made any
port 80 request forward to another address, which in turn forwarded you to
the original request. So if you west to www.weather.com, it would forward
you to www2.weather.co m, then back to www.weather.com. The index page at
www.weather.com would then check the referrer to see if it came from
www2.weather.co m. If the referrer was correct, then you got the page
content. If it was wrong, there was no page.

Similarly, www2.weather.co m would check to see that the referrer was
www.weather.com... so there was no way around it.

I use another method to protect my programs... but it still does the same
thing ultimatly... stops people from using my programs.

Like it has been mentioned, a good starting place is Telnet. I tried
telnetting when I first read this post and got a connection refused, but now
I can get through - weird. I also tried a source dump from lynx... which 10
minutes ago did not work, but now it does.

# telnet www.dictionary.com 80

# lynx -source -preparse -dump http://www.dictionary.com

These both work to show the source of the index page. And the python recipe
even works now.... go figure.
Jul 18 '05 #6
By the way, this does not work as well as it could.

It only lists the definitions if they are presented in listed <LI> tags.

So for the word greedy... it works. But, for the word greed, it does not.

I am sure some better parsing of the pages could be done for those
definitions lot listed in li tags.

Maybe I will play around with it and see what I can come up with.
Jul 18 '05 #7

Sean Berry wrote:
A few years back I needed to get information about the weather based on zip code. I used to use weather.com but they fixed the hole. They made any
port 80 request forward to another address, which in turn forwarded you to
the original request. So if you west to www.weather.com, it would forward
you to www2.weather.co m, then back to www.weather.com. The index page at
www.weather.com would then check the referrer to see if it came from
www2.weather.co m. If the referrer was correct, then you got the page
content. If it was wrong, there was no page.

Similarly, www2.weather.co m would check to see that the referrer was
www.weather.com... so there was no way around it.
Au contraire, monsieur... there is no reason I can't set the referrer
header (using httplib).
I use another method to protect my programs... but it still does the same
thing ultimatly... stops people from using my programs.


It is a level of discouragement, and may be sufficient to stop the
simplest of abuses, but the reality is that, given enough work, one can have
their Python program recreate any HTTP transaction or series of transactions
needed. If you want to redirect me to a dynamically generated URL, fine...
I'll read the Location header and go there, if you want to set a cookie
there, fine, I'll read the Set-Cookie header, set it, then return it to you,
just as a browser does. If you want to check the User-Agent, I can send
headers that look exactly like what MSIE or any other browser would send.
You can make it difficult, no doubt, and doing so may be a wise thing to do,
but it is wrong to say that just becasue A sends you to B, and B refers you
back to A, there is no way around it.

I am currently doing exactly this sort of thing, but not to abuse
others' work. We pay for a service where data is published on a web site. We
need to get at this data several times an hour, 24X7. There is a login page,
and one or more HTML forms that must be filled out and submitted, and
cookies set and checked to get to the pages that contain the data we need.
It was not designed to be machine read, but that doesn't mean reading it in
an automated manner is "wrong" or abusive - we're paying for that data, they
just happen to present it in a format that's not all that conducive to
automated machine parsing. Fine, I'll do the extra work to get at it in that
manner.

I suppose you could meter the number of requests that a particular IP is
making or something on the server side, and there isn't much that can be
done on the requester's end. With access to several machines, though, even
this hurdle could be cleared.

So, anyway, if the OP wants to get more sophisticated about automated
surfing, check out the httplib module:
http://docs.python.org/lib/module-httplib.html

FYI... www.dictionary.com loads (for me, now) through a simple telnet
request:

ej@sand:~/src/python> telnet www.dictionary.com 80
Trying 66.161.12.81...
Connected to www.dictionary.com.
Escape character is '^]'.
GET / HTTP/1.0

HTTP/1.1 200 OK
Date: Wed, 27 Oct 2004 22:57:49 GMT
Server: Apache
Cache-Control: must-revalidate
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Last-Modified: Wed, 27 Oct 2004 22:58:07 GMT
Pragma: no-cache
Connection: close
Content-Type: text/html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/1999/REC-html401-19991224/loose.dtd">
<html>

<snip!>

HTH,
-ej
Jul 18 '05 #8
> Au contraire, monsieur... there is no reason I can't set the referrer
header (using httplib).


I stand corrected.

Jul 18 '05 #9
Pater Maximus wrote:
I am trying to implement the recipe listed at
http://aspn.activestate.com/ASPN/Coo.../Recipe/211886

However, I can not get to first base. When I try to run

import urllib
fo=urllib.urlop en("http://www.dictionary. com/")
page = fo.read()

I get:

Traceback (most recent call last):
File "C:/Program Files/Python/Lib/idlelib/testurl", line 2, in -toplevel-
fo=urllib.urlop en("http://www.dictionary. com/")
File "C:\PROGRA~1\PY THON\lib\urllib .py", line 76, in urlopen
return opener.open(url )
File "C:\PROGRA~1\PY THON\lib\urllib .py", line 181, in open
return getattr(self, name)(url)
File "C:\PROGRA~1\PY THON\lib\urllib .py", line 297, in open_http
h.endheaders()
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 712, in endheaders
self._send_outp ut()
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 597, in _send_output
self.send(msg)
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 564, in send
self.connect()
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 548, in connect
raise socket.error, msg
IOError: [Errno socket error] (10061, 'Connection refused')

Suspect the action of a firewall or just assume you were unlucky and the
server was down when you hit it. I had no problem just now:

$ python
Python 2.3.4 (#1, Jun 13 2004, 11:21:03)
[GCC 3.3.1 (cygming special)] on cygwin
Type "help", "copyright" , "credits" or "license" for more information.
import urllib
fo=urllib.urlop en("http://www.dictionary. com/")
page = fo.read()


regards
Steve
--
http://www.holdenweb.com
http://pydish.holdenweb.com
Holden Web LLC +1 800 494 3119
Jul 18 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
5191
by: Gary Feldman | last post by:
I think I've found a deficiency in the design of urllib related to https. In order to complete an https connection, it appears that URLOpener and hence FancyURLOpener require the key and cert files. Or at least, it's not clear from the description of socket.ssl what it does if they're omitted. However, urlopen has no way to specify such things. Nor should it - for typical uses, a person simply trying to retrieve data from an ssl site...
0
3595
by: Pieter Edelman | last post by:
Hi all, I'm trying to submit some data using a POST request to a HTTP server with BASIC authentication with python, but I can't get it to work. Since it's driving me completely nuts, so here's my cry for help. The server is an elog logbook server (http://midas.psi.ch/elog/). It is protected with a password and an empty username. I can login both using urllib and urllib2 (suppose the password is "foobar", the logbook is running on port...
1
2072
by: Timothy Wu | last post by:
Hi, I'm trying to fill the form on page http://www.cbs.dtu.dk/services/TMHMM/ using urllib. There are two peculiarities. First of all, I am filling in incorrect key/value pairs in the parameters on purpose because that's the only way I can get it to work.. For "version" I am suppose to leave it unchecked, having value of empty string. And for name "outform" I am suppose to assign it a value of "-short". Instead, I left out
0
1669
by: Chris | last post by:
hello, I have an odd behaviour. I try to download some files connected to a specific webpage (e.g. all stylesheets) with urllib2.urlopen(x) This seems to hang on the 2nd file or so. Doing the exact same thing via urllib.urlopen(x) does work without a hitch...
4
3590
by: william | last post by:
I've got a strange problem on windows (not very familiar with that OS). I can ping a host, but cannot get it via urllib (see here under). I can even telnet the host on port 80. Thus network seems good, but not for python ;-(. Does any windows specialist can guide me (a poor linux user) to get Network functionalitiies with python on windows ?
6
14531
by: JabaPyth | last post by:
Hello, I'm trying to use the urllib module, but when i try urllib.urlopen, it gives me a socket error: >>import urllib >>print urllib.urlopen('http://www.google.com/').read() Traceback (most recent call last): File "<input>", line 1, in ? File "C:\Python24\lib\urllib.py", line 77, in urlopen return opener.open(url)
0
2447
by: Ali.Sabil | last post by:
hello all, I just maybe hit a bug in both urllib and urllib2, actually urllib doesn't support proxy authentication, and if you setup the http_proxy env var to http_proxy=http://user:passwd@host:port/ and https_proxy=$http_proxy i get a traceback : Traceback (most recent call last): File "test_urllib.py", line 2, in ? urllib.urlopen("https://sf.net/")
4
9343
by: kgrafals | last post by:
Hi, I'm just trying to read from a webpage with urllib but I'm getting IOErrors. This is my code: import urllib sock = urllib.urlopen("http://www.google.com/") and this is the error:
0
1341
by: gmguyx | last post by:
I tried using urllib.urlopen to open a personalized webpage (my.yahoo.com) but it doesn't work: print urllib.urlopen(http://my.yahoo.com).read() Instead of returning my.yahoo.com, it returns a page asking me to log in. So then I tried urllib.urlencode:
0
8788
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9290
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9159
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
9061
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
7925
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
5939
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4713
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
2508
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2097
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.