472,950 Members | 2,056 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,950 software developers and data experts.

urllib leaves sockets open?

Hi all. I'm currently tracking down a problem in a little script[1] I have,
and I was hoping that those more experienced than myself could weigh in.

The script's job is to grab the status page off a DLink home router. This is
a really simple job: I just use urllib.urlopen() to grab the status page.
The router uses HTTP Basic authentication, so I've subclassed FancyURLOpener
to supply the credentials.

This all worked fine with an older router, but with the newer model there's
a long delay between sending the authentication information and actually
getting the response back. When just going in via a brower, there is no such
delay.

I did a little work with a tracing proxy, and I noticed something
interesting. urllib first makes an HTTP request without authentication
information. This gets back an HTTP 401 error code, as expected. urllib then
opens a second socket, and sends the Authentication header, again just as
expected.

Here's what I noticed: The socket for the first request that failed is still
connected. It looks like what's happening is that the router's only allowing
a single HTTP connection at a time. As a result, the second, authenticated
request, doesn't get it's response until there's some kind of timeout and
the first socket disconnects.

Is this normal behavior for urllib? Is there a way to force that initial
socket closed earlier? Is there something else I need to do?

Thanks for any insight,

-Chris

[1] The script in question is:

router_address = "xxx"
router_port = 80
router_user = "user"
router_password = "password"

class DI604Opener( urllib.FancyURLopener ):
def prompt_user_passwd( self, host, realm ):
return ( router_user, router_password )

urllib._urlopener = DI604Opener()

#
# Kick off the process when run from the command line
#
if __name__ == "__main__":
status_page = urllib.urlopen( "http://%s:%s/status.htm" % ( router_address,
router_port ) )
print status_page.read()

Aug 21 '05 #1
3 2176
"Chris Tavares" <cc*@tavaresstudios.com> writes:
Is this normal behavior for urllib? Is there a way to force that initial
socket closed earlier? Is there something else I need to do?


I'd say open a sourceforge bug. There may be a way around it with the
fancy opener methods of urllib2, but it's a bug if regular urllib
opens a second socket without closing the first one. For http 1.1
it should be able to use just one socket anyway.
Aug 21 '05 #2
"Paul Rubin" <http://ph****@NOSPAM.invalid> wrote in message
news:7x************@ruckus.brouhaha.com...
"Chris Tavares" <cc*@tavaresstudios.com> writes:
Is this normal behavior for urllib? Is there a way to force that initial
socket closed earlier? Is there something else I need to do?


I'd say open a sourceforge bug. There may be a way around it with the
fancy opener methods of urllib2, but it's a bug if regular urllib
opens a second socket without closing the first one. For http 1.1
it should be able to use just one socket anyway.


Thanks, I'll do some poking around in urllib first and see if I can narrow
it down.

Is there a way to do HTTP 1.1 with urllib? The docs say 0.9 and 1.0 only.

Thanks,

-Chris
Aug 21 '05 #3
"Chris Tavares" <cc*@tavaresstudios.com> writes:
Is there a way to do HTTP 1.1 with urllib? The docs say 0.9 and 1.0 only.


I'm not sure. Try urllib2, but I'm still not sure.
Aug 21 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Andreas Dahl | last post by:
Hi, I use urllib to retrieve data via HTTP. Unfortunately my program crashes after a while (after some loops) because the connection timed out. raise socket.error, msg IOError: (60,...
3
by: Haim Ashkenazi | last post by:
Hi I'm writing a script that uses urllib on win98. until now I used python 2.3.x (x < 4) and it worked ok. I re-installed windows and installed python 2.3.4 and now I get an error when trying to...
1
by: John Hunter | last post by:
I have a test script below which I use to fetch urls into strings, either over https or http. When over https, I use m2crypto.urllib and when over http I use the standard urllib. Whenever, I...
0
by: Pieter Edelman | last post by:
Hi all, I'm trying to submit some data using a POST request to a HTTP server with BASIC authentication with python, but I can't get it to work. Since it's driving me completely nuts, so here's...
11
by: George Sakkis | last post by:
The following snippet results in different outcome for (at least) the last three major releases: # Python 2.3.4 u'%94' # Python 2.4.2 UnicodeDecodeError: 'ascii' codec can't decode byte...
11
by: Stefan Palme | last post by:
Hi all, is there a way to modify the time a call of urllib.open(...) waits for an answer from the other side? Have a tool which automatically checks a list of websites for certain content....
5
by: supercooper | last post by:
I am downloading images using the script below. Sometimes it will go for 10 mins, sometimes 2 hours before timing out with the following error: Traceback (most recent call last): File...
5
by: John Nagle | last post by:
I thought I had all the timeout problems with urllib worked around, but no. socket.setdefaulttimeout is useful, but not always effective. I'm setting that to 15 seconds. If the host end won't...
2
by: ShashiGowda | last post by:
Hey there i made a script to download all images from a web site but it runs damn slow though I have a lot of bandwidth waiting to be used please tell me a way to use urllib to open many...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Sept 2023 starting at 18:00 UK time (6PM UTC+1) and finishing at about 19:15 (7.15PM) The start time is equivalent to 19:00 (7PM) in Central...
0
by: Mushico | last post by:
How to calculate date of retirement from date of birth
2
by: DJRhino | last post by:
Was curious if anyone else was having this same issue or not.... I was just Up/Down graded to windows 11 and now my access combo boxes are not acting right. With win 10 I could start typing...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 4 Oct 2023 starting at 18:00 UK time (6PM UTC+1) and finishing at about 19:15 (7.15PM) The start time is equivalent to 19:00 (7PM) in Central...
0
tracyyun
by: tracyyun | last post by:
Hello everyone, I have a question and would like some advice on network connectivity. I have one computer connected to my router via WiFi, but I have two other computers that I want to be able to...
4
NeoPa
by: NeoPa | last post by:
Hello everyone. I find myself stuck trying to find the VBA way to get Access to create a PDF of the currently-selected (and open) object (Form or Report). I know it can be done by selecting :...
3
NeoPa
by: NeoPa | last post by:
Introduction For this article I'll be using a very simple database which has Form (clsForm) & Report (clsReport) classes that simply handle making the calling Form invisible until the Form, or all...
1
by: Teri B | last post by:
Hi, I have created a sub-form Roles. In my course form the user selects the roles assigned to the course. 0ne-to-many. One course many roles. Then I created a report based on the Course form and...
3
by: nia12 | last post by:
Hi there, I am very new to Access so apologies if any of this is obvious/not clear. I am creating a data collection tool for health care employees to complete. It consists of a number of...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.