urlretrieve hangs on large files (sometimes)

[Josh]

Hi. I am writing a script that downloads lots of zips from a usgs site.
All is going well except that occasionally, in the midst of
downloading a file, the script just hangs and i must either ctrl-break
it (running from command line) or, if I wait about 20 minutes after it
hangs, even that won't allow me to close it. Either way, I must then
restart the program and have it pick up where it left off.

I don't know much about python or http, but I believe that the problem
is website sometimes does not respond to the 'get' (or whatever
urlretrieve ) request is in a timely manner: certainly the site
sometimes takes a bit of a break even when downloading with IE, for
example. However IE or other download managers continue -- my python
script just hangs.

Here is the relevant code, I think:

out = file(outFile, "wb")
(filename, headers,) = urllib.urlretrieve(dlinks[count], outFile,
reporthook=report)
out.close()

I am making a haphazard guess that somewhere in the bowels of
urlretrieve there is a timeout setting for the http get command, and if
the site does not respond in time it gets all upset and confused.
Whether I am wrong or right -- any suggestions?

Thanks a lot.

-Josh

[Marcin Ciura]

Josh wrote:

Hi. I am writing a script that downloads lots of zips from a usgs site.
All is going well except that occasionally, in the midst of downloading
a file, the script just hangs and i must either ctrl-break it (running
from command line) or, if I wait about 20 minutes after it hangs, even
that won't allow me to close it. Either way, I must then restart the
program and have it pick up where it left off.

Use socket.setdefaulttimeout().
Marcin