By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
445,645 Members | 1,048 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 445,645 IT Pros & Developers. It's quick & easy.

urlretrieve hangs on large files (sometimes)

P: n/a
Hi. I am writing a script that downloads lots of zips from a usgs site.
All is going well except that occasionally, in the midst of
downloading a file, the script just hangs and i must either ctrl-break
it (running from command line) or, if I wait about 20 minutes after it
hangs, even that won't allow me to close it. Either way, I must then
restart the program and have it pick up where it left off.

I don't know much about python or http, but I believe that the problem
is website sometimes does not respond to the 'get' (or whatever
urlretrieve ) request is in a timely manner: certainly the site
sometimes takes a bit of a break even when downloading with IE, for
example. However IE or other download managers continue -- my python
script just hangs.

Here is the relevant code, I think:

out = file(outFile, "wb")
(filename, headers,) = urllib.urlretrieve(dlinks[count], outFile,
reporthook=report)
out.close()

I am making a haphazard guess that somewhere in the bowels of
urlretrieve there is a timeout setting for the http get command, and if
the site does not respond in time it gets all upset and confused.
Whether I am wrong or right -- any suggestions?

Thanks a lot.

-Josh
Jul 18 '05 #1
Share this Question
Share on Google+
1 Reply


P: n/a
Josh wrote:
Hi. I am writing a script that downloads lots of zips from a usgs site.
All is going well except that occasionally, in the midst of downloading
a file, the script just hangs and i must either ctrl-break it (running
from command line) or, if I wait about 20 minutes after it hangs, even
that won't allow me to close it. Either way, I must then restart the
program and have it pick up where it left off.


Use socket.setdefaulttimeout().
Marcin

Jul 18 '05 #2

This discussion thread is closed

Replies have been disabled for this discussion.