473,575 Members | 3,313 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Can not get urllib.urlopen to work

I am trying to implement the recipe listed at
http://aspn.activestate.com/ASPN/Coo.../Recipe/211886

However, I can not get to first base. When I try to run

import urllib
fo=urllib.urlop en("http://www.dictionary. com/")
page = fo.read()

I get:

Traceback (most recent call last):
File "C:/Program Files/Python/Lib/idlelib/testurl", line 2, in -toplevel-
fo=urllib.urlop en("http://www.dictionary. com/")
File "C:\PROGRA~1\PY THON\lib\urllib .py", line 76, in urlopen
return opener.open(url )
File "C:\PROGRA~1\PY THON\lib\urllib .py", line 181, in open
return getattr(self, name)(url)
File "C:\PROGRA~1\PY THON\lib\urllib .py", line 297, in open_http
h.endheaders()
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 712, in endheaders
self._send_outp ut()
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 597, in _send_output
self.send(msg)
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 564, in send
self.connect()
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 548, in connect
raise socket.error, msg
IOError: [Errno socket error] (10061, 'Connection refused')
Jul 18 '05 #1
11 5035
More information:

I am running Python 2.3.4, Windows 2000 Pro, along with Norton Internet
Security Professional. I do not get any messages from Norton.
Jul 18 '05 #2
Pater Maximus wrote:
I am trying to implement the recipe listed at
http://aspn.activestate.com/ASPN/Coo.../Recipe/211886

However, I can not get to first base. When I try to run

import urllib
fo=urllib.urlop en("http://www.dictionary. com/")
page = fo.read()

I get:
Traceback (most recent call last):
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 548, in connect
raise socket.error, msg
IOError: [Errno socket error] (10061, 'Connection refused')


Do you have any reason to think that this site is currently
hosting a web server?

Attempting to connect with IE or Telnet on port 80 results in
a similar response.

(Telnet is always a good thing to try, before posting about
server problems...)

-Peter
Jul 18 '05 #3

"Pater Maximus" <em***@home.com > wrote in message
news:3J******** ************@co mcast.com...
IOError: [Errno socket error] (10061, 'Connection refused')


My browser says the same thing. ;-)
Alan Isaac
Jul 18 '05 #4
Pater Maximus wrote:
I am trying to implement the recipe listed at
http://aspn.activestate.com/ASPN/Coo.../Recipe/211886

However, I can not get to first base. When I try to run

import urllib
fo=urllib.urlop en("http://www.dictionary. com/")
page = fo.read()


I can't even connect to it with my web browser. Can you?
If you can, they are probably checking the user-agent sent
by urllib, to make it harder to do this sort of automated
screen scraping.

See the docs at
http://www.python.org/doc/current/li...le-urllib.html

for an example of how to change the default user-agent.

Here's one for MSIE under Win2K

Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)

Andrew
da***@dalkescie ntific.com
Jul 18 '05 #5
>I am trying to implement the recipe listed at
http://aspn.activestate.com/ASPN/Coo.../Recipe/211886

However, I can not get to first base. When I try to run

import urllib
fo=urllib.urlop en("http://www.dictionary. com/")
page = fo.read()

I get:

Traceback (most recent call last):
File "C:/Program Files/Python/Lib/idlelib/testurl", line 2, in -toplevel-
fo=urllib.urlop en("http://www.dictionary. com/")
File "C:\PROGRA~1\PY THON\lib\urllib .py", line 76, in urlopen
return opener.open(url )
File "C:\PROGRA~1\PY THON\lib\urllib .py", line 181, in open
return getattr(self, name)(url)
File "C:\PROGRA~1\PY THON\lib\urllib .py", line 297, in open_http
h.endheaders()
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 712, in endheaders
self._send_outp ut()
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 597, in _send_output
self.send(msg)
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 564, in send
self.connect()
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 548, in connect
raise socket.error, msg
IOError: [Errno socket error] (10061, 'Connection refused')


Connection refused is the key. Apparently dictionary.com figured out that
people were trying to use their resourses without giving them credit. I
have done some research on how to accomplish this so that people can not use
the cgi-bin programs (and others) that I write.

A few years back I needed to get information about the weather based on zip
code. I used to use weather.com but they fixed the hole. They made any
port 80 request forward to another address, which in turn forwarded you to
the original request. So if you west to www.weather.com, it would forward
you to www2.weather.co m, then back to www.weather.com. The index page at
www.weather.com would then check the referrer to see if it came from
www2.weather.co m. If the referrer was correct, then you got the page
content. If it was wrong, there was no page.

Similarly, www2.weather.co m would check to see that the referrer was
www.weather.com... so there was no way around it.

I use another method to protect my programs... but it still does the same
thing ultimatly... stops people from using my programs.

Like it has been mentioned, a good starting place is Telnet. I tried
telnetting when I first read this post and got a connection refused, but now
I can get through - weird. I also tried a source dump from lynx... which 10
minutes ago did not work, but now it does.

# telnet www.dictionary.com 80

# lynx -source -preparse -dump http://www.dictionary.com

These both work to show the source of the index page. And the python recipe
even works now.... go figure.
Jul 18 '05 #6
By the way, this does not work as well as it could.

It only lists the definitions if they are presented in listed <LI> tags.

So for the word greedy... it works. But, for the word greed, it does not.

I am sure some better parsing of the pages could be done for those
definitions lot listed in li tags.

Maybe I will play around with it and see what I can come up with.
Jul 18 '05 #7

Sean Berry wrote:
A few years back I needed to get information about the weather based on zip code. I used to use weather.com but they fixed the hole. They made any
port 80 request forward to another address, which in turn forwarded you to
the original request. So if you west to www.weather.com, it would forward
you to www2.weather.co m, then back to www.weather.com. The index page at
www.weather.com would then check the referrer to see if it came from
www2.weather.co m. If the referrer was correct, then you got the page
content. If it was wrong, there was no page.

Similarly, www2.weather.co m would check to see that the referrer was
www.weather.com... so there was no way around it.
Au contraire, monsieur... there is no reason I can't set the referrer
header (using httplib).
I use another method to protect my programs... but it still does the same
thing ultimatly... stops people from using my programs.


It is a level of discouragement, and may be sufficient to stop the
simplest of abuses, but the reality is that, given enough work, one can have
their Python program recreate any HTTP transaction or series of transactions
needed. If you want to redirect me to a dynamically generated URL, fine...
I'll read the Location header and go there, if you want to set a cookie
there, fine, I'll read the Set-Cookie header, set it, then return it to you,
just as a browser does. If you want to check the User-Agent, I can send
headers that look exactly like what MSIE or any other browser would send.
You can make it difficult, no doubt, and doing so may be a wise thing to do,
but it is wrong to say that just becasue A sends you to B, and B refers you
back to A, there is no way around it.

I am currently doing exactly this sort of thing, but not to abuse
others' work. We pay for a service where data is published on a web site. We
need to get at this data several times an hour, 24X7. There is a login page,
and one or more HTML forms that must be filled out and submitted, and
cookies set and checked to get to the pages that contain the data we need.
It was not designed to be machine read, but that doesn't mean reading it in
an automated manner is "wrong" or abusive - we're paying for that data, they
just happen to present it in a format that's not all that conducive to
automated machine parsing. Fine, I'll do the extra work to get at it in that
manner.

I suppose you could meter the number of requests that a particular IP is
making or something on the server side, and there isn't much that can be
done on the requester's end. With access to several machines, though, even
this hurdle could be cleared.

So, anyway, if the OP wants to get more sophisticated about automated
surfing, check out the httplib module:
http://docs.python.org/lib/module-httplib.html

FYI... www.dictionary.com loads (for me, now) through a simple telnet
request:

ej@sand:~/src/python> telnet www.dictionary.com 80
Trying 66.161.12.81...
Connected to www.dictionary.com.
Escape character is '^]'.
GET / HTTP/1.0

HTTP/1.1 200 OK
Date: Wed, 27 Oct 2004 22:57:49 GMT
Server: Apache
Cache-Control: must-revalidate
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Last-Modified: Wed, 27 Oct 2004 22:58:07 GMT
Pragma: no-cache
Connection: close
Content-Type: text/html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/1999/REC-html401-19991224/loose.dtd">
<html>

<snip!>

HTH,
-ej
Jul 18 '05 #8
> Au contraire, monsieur... there is no reason I can't set the referrer
header (using httplib).


I stand corrected.

Jul 18 '05 #9
Pater Maximus wrote:
I am trying to implement the recipe listed at
http://aspn.activestate.com/ASPN/Coo.../Recipe/211886

However, I can not get to first base. When I try to run

import urllib
fo=urllib.urlop en("http://www.dictionary. com/")
page = fo.read()

I get:

Traceback (most recent call last):
File "C:/Program Files/Python/Lib/idlelib/testurl", line 2, in -toplevel-
fo=urllib.urlop en("http://www.dictionary. com/")
File "C:\PROGRA~1\PY THON\lib\urllib .py", line 76, in urlopen
return opener.open(url )
File "C:\PROGRA~1\PY THON\lib\urllib .py", line 181, in open
return getattr(self, name)(url)
File "C:\PROGRA~1\PY THON\lib\urllib .py", line 297, in open_http
h.endheaders()
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 712, in endheaders
self._send_outp ut()
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 597, in _send_output
self.send(msg)
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 564, in send
self.connect()
File "C:\PROGRA~1\PY THON\lib\httpli b.py", line 548, in connect
raise socket.error, msg
IOError: [Errno socket error] (10061, 'Connection refused')

Suspect the action of a firewall or just assume you were unlucky and the
server was down when you hit it. I had no problem just now:

$ python
Python 2.3.4 (#1, Jun 13 2004, 11:21:03)
[GCC 3.3.1 (cygming special)] on cygwin
Type "help", "copyright" , "credits" or "license" for more information.
import urllib
fo=urllib.urlop en("http://www.dictionary. com/")
page = fo.read()


regards
Steve
--
http://www.holdenweb.com
http://pydish.holdenweb.com
Holden Web LLC +1 800 494 3119
Jul 18 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
5176
by: Gary Feldman | last post by:
I think I've found a deficiency in the design of urllib related to https. In order to complete an https connection, it appears that URLOpener and hence FancyURLOpener require the key and cert files. Or at least, it's not clear from the description of socket.ssl what it does if they're omitted. However, urlopen has no way to specify such...
0
3576
by: Pieter Edelman | last post by:
Hi all, I'm trying to submit some data using a POST request to a HTTP server with BASIC authentication with python, but I can't get it to work. Since it's driving me completely nuts, so here's my cry for help. The server is an elog logbook server (http://midas.psi.ch/elog/). It is protected with a password and an empty username. I can...
1
2063
by: Timothy Wu | last post by:
Hi, I'm trying to fill the form on page http://www.cbs.dtu.dk/services/TMHMM/ using urllib. There are two peculiarities. First of all, I am filling in incorrect key/value pairs in the parameters on purpose because that's the only way I can get it to work.. For "version" I am suppose to leave it unchecked, having value of empty string....
0
1652
by: Chris | last post by:
hello, I have an odd behaviour. I try to download some files connected to a specific webpage (e.g. all stylesheets) with urllib2.urlopen(x) This seems to hang on the 2nd file or so. Doing the exact same thing via urllib.urlopen(x) does work without a hitch...
4
3583
by: william | last post by:
I've got a strange problem on windows (not very familiar with that OS). I can ping a host, but cannot get it via urllib (see here under). I can even telnet the host on port 80. Thus network seems good, but not for python ;-(. Does any windows specialist can guide me (a poor linux user) to get Network functionalitiies with python on...
6
14515
by: JabaPyth | last post by:
Hello, I'm trying to use the urllib module, but when i try urllib.urlopen, it gives me a socket error: >>import urllib >>print urllib.urlopen('http://www.google.com/').read() Traceback (most recent call last): File "<input>", line 1, in ? File "C:\Python24\lib\urllib.py", line 77, in urlopen return opener.open(url)
0
2438
by: Ali.Sabil | last post by:
hello all, I just maybe hit a bug in both urllib and urllib2, actually urllib doesn't support proxy authentication, and if you setup the http_proxy env var to http_proxy=http://user:passwd@host:port/ and https_proxy=$http_proxy i get a traceback : Traceback (most recent call last): File "test_urllib.py", line 2, in ?...
4
9336
by: kgrafals | last post by:
Hi, I'm just trying to read from a webpage with urllib but I'm getting IOErrors. This is my code: import urllib sock = urllib.urlopen("http://www.google.com/") and this is the error:
0
1338
by: gmguyx | last post by:
I tried using urllib.urlopen to open a personalized webpage (my.yahoo.com) but it doesn't work: print urllib.urlopen(http://my.yahoo.com).read() Instead of returning my.yahoo.com, it returns a page asking me to log in. So then I tried urllib.urlencode:
0
7843
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
8115
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
8278
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
0
8141
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
0
5336
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3776
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
0
3795
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2283
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
1378
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.