By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
455,539 Members | 1,289 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 455,539 IT Pros & Developers. It's quick & easy.

urllib (in thread) never returns

P: n/a
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

I've been having an intermittent problem with urllib.
With an interval of around 15 minutes (i.e.: this is run every 15m)
this code runs fine for about 1-2 weeks, but then gets it's
knickers in a twist, and never seems to return, nor except.

try:
log("ABOUT TO USE urllib TO FETCH ["+self.url+"]")
f = urllib.urlopen(self.url)
temp_xml = f.read()
[1] log("DONE! GOT %d bytes for [%s]" % (len(str(temp_xml)),self.url))
f.close()
refreshed = True
except:
[2] log("Error: fetching ["+self.url+"], retrying ... ")
...

In the 'broken' state, I never see the log message [1] or [2], it
just sits in either the urlopen() or read() forever.

The code is running in a separate thread.
TCP timeout is set to 45 seconds.
Running on Linux (gentoo)

<complete-guess>
What I think is wrong, is that the server is sitting
behind a somewhat-dodgey ADSL connection. The server it's
contacting is also on a dodgey ADSL connection.

I'm guessing that the socket is opened with
the ADSL modem is NATing the connection, but when
the backend internet connection bounces, the NAT'ed
connection somehow stays up, leaving the connection
somehow dangling; alive, but dead too...
</complete-guess>

I would have thought that some urllib-internal timeout would
fix this?!

I could watch the thread from another thread, implementing
my own timeout... But then (AFAIK) there's no way to terminate
a mis-behaving thread anyways.

Any suggestions?

thanks,
- -Kingsley
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEut+ISo2jKyi4JlgRApIdAJ4iIWDLXR9am659XAS1aj Lv1ry12wCfeOiI
FUNbrisdyo4j3Yle4zN2ESk=
=EPU2
-----END PGP SIGNATURE-----
Jul 17 '06 #1
Share this Question
Share on Google+
1 Reply


P: n/a
Kingsley:
>it just sits in either the urlopen() or read() forever.
[...]
>I would have thought that some urllib-internal timeout
would fix this?!
Yes, but you'll need to enable it. See socket.setdefaulttimeout() :
http://docs.python.org/lib/module-socket.html

--
René Pijlman
Jul 17 '06 #2

This discussion thread is closed

Replies have been disabled for this discussion.