Hi all. I'm currently tracking down a problem in a little script[1] I have,
and I was hoping that those more experienced than myself could weigh in.
The script's job is to grab the status page off a DLink home router. This is
a really simple job: I just use urllib.urlopen( ) to grab the status page.
The router uses HTTP Basic authentication, so I've subclassed FancyURLOpener
to supply the credentials.
This all worked fine with an older router, but with the newer model there's
a long delay between sending the authentication information and actually
getting the response back. When just going in via a brower, there is no such
delay.
I did a little work with a tracing proxy, and I noticed something
interesting. urllib first makes an HTTP request without authentication
information. This gets back an HTTP 401 error code, as expected. urllib then
opens a second socket, and sends the Authentication header, again just as
expected.
Here's what I noticed: The socket for the first request that failed is still
connected. It looks like what's happening is that the router's only allowing
a single HTTP connection at a time. As a result, the second, authenticated
request, doesn't get it's response until there's some kind of timeout and
the first socket disconnects.
Is this normal behavior for urllib? Is there a way to force that initial
socket closed earlier? Is there something else I need to do?
Thanks for any insight,
-Chris
[1] The script in question is:
router_address = "xxx"
router_port = 80
router_user = "user"
router_password = "password"
class DI604Opener( urllib.FancyURL opener ):
def prompt_user_pas swd( self, host, realm ):
return ( router_user, router_password )
urllib._urlopen er = DI604Opener()
#
# Kick off the process when run from the command line
#
if __name__ == "__main__":
status_page = urllib.urlopen( "http://%s:%s/status.htm" % ( router_address,
router_port ) )
print status_page.rea d() 3 2245
"Chris Tavares" <cc*@tavaresstu dios.com> writes: Is this normal behavior for urllib? Is there a way to force that initial socket closed earlier? Is there something else I need to do?
I'd say open a sourceforge bug. There may be a way around it with the
fancy opener methods of urllib2, but it's a bug if regular urllib
opens a second socket without closing the first one. For http 1.1
it should be able to use just one socket anyway.
"Paul Rubin" <http://ph****@NOSPAM.i nvalid> wrote in message
news:7x******** ****@ruckus.bro uhaha.com... "Chris Tavares" <cc*@tavaresstu dios.com> writes: Is this normal behavior for urllib? Is there a way to force that initial socket closed earlier? Is there something else I need to do?
I'd say open a sourceforge bug. There may be a way around it with the fancy opener methods of urllib2, but it's a bug if regular urllib opens a second socket without closing the first one. For http 1.1 it should be able to use just one socket anyway.
Thanks, I'll do some poking around in urllib first and see if I can narrow
it down.
Is there a way to do HTTP 1.1 with urllib? The docs say 0.9 and 1.0 only.
Thanks,
-Chris
"Chris Tavares" <cc*@tavaresstu dios.com> writes: Is there a way to do HTTP 1.1 with urllib? The docs say 0.9 and 1.0 only.
I'm not sure. Try urllib2, but I'm still not sure. This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Andreas Dahl |
last post by:
Hi,
I use urllib to retrieve data via HTTP. Unfortunately my program crashes
after a while (after some loops) because the connection timed out.
raise socket.error, msg
IOError: (60, 'Connection timed out')
I am not so familiar with python, but is there a possibility to
configure the 'waiting time'? Or how can I handle such an event? To skip
|
by: Haim Ashkenazi |
last post by:
Hi
I'm writing a script that uses urllib on win98. until now I used python
2.3.x (x < 4) and it worked ok. I re-installed windows and installed
python 2.3.4 and now I get an error when trying to open a url "no host
given":
Python 2.3.4 (#53, May 25 2004, 21:17:02) on win32
Type "copyright", "credits" or "license()" for more information.
|
by: John Hunter |
last post by:
I have a test script below which I use to fetch urls into strings,
either over https or http. When over https, I use m2crypto.urllib and
when over http I use the standard urllib. Whenever, I import sockets
and setdefaulttimeout, however, using m2crypto.urllib tends to cause a
http.BadStatusLine to be raised, even if the timeout is set to be very
large. All of the documents in the test script can be accessed
publicly.
Any ideas? Is...
|
by: Pieter Edelman |
last post by:
Hi all,
I'm trying to submit some data using a POST request to a HTTP server with
BASIC authentication with python, but I can't get it to work. Since it's
driving me completely nuts, so here's my cry for help.
The server is an elog logbook server (http://midas.psi.ch/elog/). It is
protected with a password and an empty username. I can login both using
urllib and urllib2 (suppose the password is "foobar", the logbook is
running on port...
|
by: George Sakkis |
last post by:
The following snippet results in different outcome for (at least) the
last three major releases:
# Python 2.3.4
u'%94'
# Python 2.4.2
UnicodeDecodeError: 'ascii' codec can't decode byte 0x94 in position 0:
ordinal not in range(128)
| |
by: Stefan Palme |
last post by:
Hi all,
is there a way to modify the time a call of
urllib.open(...)
waits for an answer from the other side? Have a tool
which automatically checks a list of websites for
certain content. The tool "hangs" when one of the
contacted websites behaves badly and "never" answers...
|
by: supercooper |
last post by:
I am downloading images using the script below. Sometimes it will go
for 10 mins, sometimes 2 hours before timing out with the following
error:
Traceback (most recent call last):
File "ftp_20070326_Downloads_cooperc_FetchLibreMapProjectDRGs.py",
line 108, i
n ?
urllib.urlretrieve(fullurl, localfile)
File "C:\Python24\lib\urllib.py", line 89, in urlretrieve
|
by: John Nagle |
last post by:
I thought I had all the timeout problems with urllib worked around,
but no.
socket.setdefaulttimeout is useful, but not always effective.
I'm setting that to 15 seconds.
If the host end won't open the connection within 15 seconds,
urllib times out. But if the host end opens the connection,
then never sends anything, urllib waits for many minutes before
timing out. Any idea how to deal with this? And don't just
say "use urllib2"...
|
by: ShashiGowda |
last post by:
Hey there i made a script to download all images from a web site but
it runs damn slow though I have a lot of bandwidth waiting to be used
please tell me a way to use urllib to open many connections to the
server to download many pics simultaneously.... Any off question
suggestions are also ok...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
| |
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |