socket.connect() hangs in SYN_SENT state.

[bukzor]

I'm having an issue where my program hangs while doing
socket.connect() for a couple minutes, then times out the connection
and crashes. I'm connecting to an apache2 process on the same machine,
for testing. When looking at netstat, the socket is in the SYN_SENT
state, like this:

$netstat -a -tcp
tcp 0 0 *:www *:*
LISTEN 7635/apache2
tcp 0 1 bukzor:38234 adsl-75-61-84-249.d:www
SYN_SENT 9139/python

Anyone know a general reason this might happen? Even better, a way to
fix it?
Doing a minimal amount of research, I found this in the netstat
manual:
The state SYN_SENT means that an application has made arequest for a
TCP session, but has not yet received the return SYN+ACK packet.

This would indicate it's a server issue, but it seems very stable when
I look at it via a browser.
Here's the server. If you browse to it, it documents the exported
functions:
http://bukzor.hopto.org/modpython/xmlrpc.py

Here's my test client that's hanging. Turn 'verbose' to True to get
more debugging info.

#!/usr/bin/env python
from xmlrpclib import ServerProxy
 
s = ServerProxy("http://bukzor.hopto.org/modpython/xmlrpc.py",
verbose=False)
 
print s.helloworld()
print s.add(1,2)
print s.subtract(1,2)
 

Thanks,
--Buck

[Miles]

On Sat, Jul 12, 2008 at 11:23 PM, bukzor <wo**********@gmail.comwrote:

I'm connecting to an apache2 process on the same machine,
for testing. When looking at netstat, the socket is in the SYN_SENT
state, like this:

$netstat -a -tcp
tcp 0 0 *:www *:* LISTEN 7635/apache2
tcp 0 1 bukzor:38234 adsl-75-61-84-249.d:www SYN_SENT 9139/python

Anyone know a general reason this might happen? Even better, a way to
fix it?

That socket connection is to a remote machine, not the same one. Your
test code works fine for me. The "hang then crash" (and I'm assuming
"crash" here means an uncaught exception) just means that your packets
are being silently ignored by whatever machine you're actually
attempting to connect to. It's possible that your machine has odd DNS
settings causing buzkor.hopto.org to resolve to the wrong address.

-Miles

[bukzor]

On Jul 13, 1:14*am, Miles <[e-mail]:

On Sat, Jul 12, 2008 at 11:23 PM, bukzor <[e-mail]:

I'm connecting to an apache2 process on the same machine,
for testing. When looking at netstat, the socket is in the SYN_SENT
state, like this:

$netstat -a -tcp
tcp * * * *0 * * *0 *:www * * * * * * * ** *:* LISTEN * * *7635/apache2
tcp * * * *0 * * *1 bukzor:38234 * * * * * *adsl-75-61-84-249.d:www SYN_SENT * *9139/python

Anyone know a general reason this might happen? Even better, a way to
fix it?

That socket connection is to a remote machine, not the same one. *Your
test code works fine for me. *The "hang then crash" (and I'm assuming
"crash" here means an uncaught exception) just means that your packets
are being silently ignored by whatever machine you're actually
attempting to connect to. It's possible that your machine has odd DNS
settings causing buzkor.hopto.org to resolve to the wrong address.

-Miles

I'm connecting to my machine through the internet, and the resolved
URL of my router is what you're seeing above. If you run the code
above you'll see what I mean.

Thanks tho,
--Buck

[Miles]

On Sun, Jul 13, 2008 at 2:32 PM, bukzor <wo**********@gmail.comwrote:

On Jul 13, 1:14 am, Miles <[e-mail]:

>On Sat, Jul 12, 2008 at 11:23 PM, bukzor <[e-mail]:

I'm connecting to an apache2 process on the same machine,
for testing. When looking at netstat, the socket is in the SYN_SENT
state, like this:

$netstat -a -tcp
tcp 0 0 *:www *:* LISTEN 7635/apache2
tcp 0 1 bukzor:38234 adsl-75-61-84-249.d:www SYN_SENT 9139/python

Anyone know a general reason this might happen? Even better, a way to
fix it?

That socket connection is to a remote machine, not the same one. Your
test code works fine for me. The "hang then crash" (and I'm assuming
"crash" here means an uncaught exception) just means that your packets
are being silently ignored by whatever machine you're actually
attempting to connect to. It's possible that your machine has odd DNS
settings causing buzkor.hopto.org to resolve to the wrong address.

I'm connecting to my machine through the internet, and the resolved
URL of my router is what you're seeing above. If you run the code
above you'll see what I mean.

I did run the code, and as I said, it works fine. Your description of
the setup is not consistent. The netstat output unambiguously states
that a Python script on "buzkor" is attempting to open a connection to
the HTTP port on the "adsl" machine (and failing because "adsl" is not
responding). The problem here is not Python; you seem to be confused
about which machine is connecting to which.

-Miles

[bukzor]

On Jul 13, 1:08 pm, Miles <[e-mail]:

On Sun, Jul 13, 2008 at 2:32 PM, bukzor <[e-mail]:

On Jul 13, 1:14 am, Miles <[e-mail]:

On Sat, Jul 12, 2008 at 11:23 PM, bukzor <[e-mail]:
I'm connecting to an apache2 process on the same machine,
for testing. When looking at netstat, the socket is in the SYN_SENT
state, like this:

$netstat -a -tcp
tcp 0 0 *:www *:* LISTEN 7635/apache2
tcp 0 1 bukzor:38234 adsl-75-61-84-249.d:www SYN_SENT 9139/python

Anyone know a general reason this might happen? Even better, a way to
fix it?

That socket connection is to a remote machine, not the same one. Your
test code works fine for me. The "hang then crash" (and I'm assuming
"crash" here means an uncaught exception) just means that your packets
are being silently ignored by whatever machine you're actually
attempting to connect to. It's possible that your machine has odd DNS
settings causing buzkor.hopto.org to resolve to the wrong address.

I'm connecting to my machine through the internet, and the resolved
URL of my router is what you're seeing above. If you run the code
above you'll see what I mean.

I did run the code, and as I said, it works fine. Your description of
the setup is not consistent. The netstat output unambiguously states
that a Python script on "buzkor" is attempting to open a connection to
the HTTP port on the "adsl" machine (and failing because "adsl" is not
responding). The problem here is not Python; you seem to be confused
about which machine is connecting to which.

-Miles

The problem only manifests about 1 in 20 runs. Below there's code for
a client that shows the problem 100% of the time.

The two URL's that I seem to be "confused" about point to the same IP.
Maybe this will make it clear:

PING bukzor.hopto.org (75.61.84.249) 56(84) bytes of data.
64 bytes from adsl-75-61-84-249.dsl.pltn13.sbcglobal.net
(75.61.84.249): icmp_seq=1 ttl=255 time=1.68 ms
64 bytes from adsl-75-61-84-249.dsl.pltn13.sbcglobal.net
(75.61.84.249): icmp_seq=2 ttl=255 time=0.493 ms
64 bytes from adsl-75-61-84-249.dsl.pltn13.sbcglobal.net
(75.61.84.249): icmp_seq=3 ttl=255 time=0.602 ms
Apparently netstat truncated the URL before. Here's the code I
mentioned:

#!/usr/bin/env python
from xmlrpclib import ServerProxy
from time import time
 
s = ServerProxy("http://bukzor.hopto.org/modpython/xmlrpc.py",
verbose=True)
 
i = 0
start = time()
while True:
print s.helloworld()
print s.add(1,2)
print s.subtract(1,2)
i += 3
print "AVG: %.2fs" % ((time() - start) / i)
 

Thanks,
--Buck

[Miles]

On Sun, Jul 13, 2008 at 8:35 PM, bukzor <wo**********@gmail.comwrote:

The problem only manifests about 1 in 20 runs. Below there's code for
a client that shows the problem 100% of the time.

The two URL's that I seem to be "confused" about point to the same IP.
Maybe this will make it clear:

PING bukzor.hopto.org (75.61.84.249) 56(84) bytes of data.
64 bytes from adsl-75-61-84-249.dsl.pltn13.sbcglobal.net
(75.61.84.249): icmp_seq=1 ttl=255 time=1.68 ms

For me, buzkor.hopto.org resolves to 69.65.19.125, which I hope
explains why I thought you were confused, and increases my own
suspicion that DNS settings are to blame. I let the script run for
about five minutes without it failing.

Does your luck change if you use "localhost" or a numeric IP address
in the ServerProxy URL?

-Miles

[Miles]

On Sun, Jul 13, 2008 at 9:31 PM, Miles <se*********@gmail.comwrote:

On Sun, Jul 13, 2008 at 8:35 PM, bukzor <wo**********@gmail.comwrote:

>The problem only manifests about 1 in 20 runs. Below there's code for
a client that shows the problem 100% of the time.

The two URL's that I seem to be "confused" about point to the same IP.
Maybe this will make it clear:

PING bukzor.hopto.org (75.61.84.249) 56(84) bytes of data.
64 bytes from adsl-75-61-84-249.dsl.pltn13.sbcglobal.net
(75.61.84.249): icmp_seq=1 ttl=255 time=1.68 ms

For me, buzkor.hopto.org resolves to 69.65.19.125

Ahhh... "bukzor". Well, that makes sense. Pardon my temporary dyslexia.

-Miles

[Miles]

On Sat, Jul 12, 2008 at 11:23 PM, bukzor <wo**********@gmail.comwrote:

Anyone know a general reason this might happen? Even better, a way to
fix it?

Another reason that a socket can hang in the SYN_SENT state (besides
trying to connect to an unreachable host without getting an ICMP
destination-unreachable message in response): if the server's listen
queue is full, it will silently ignore SYN packets until there is room
in the queue.

Sorry again about the "bukzor" vs. "buzkor" thing. I don't know
what's causing your problem (and it's probably not a DNS issue after
all) but it's more likely to be a server issue than a client one.
Maybe your client has an unusually low socket timeout for some reason,
though; does increasing it (with socket.setdefaulttimeout) help? Mine
seems to default to about 75 seconds.

If you can't work out the root cause, but it only happens every once
in a while, you could try changing your client code to catch the
socket exception and retry a limited number of times.

-Miles

[Miles]

On Sun, Jul 13, 2008 at 10:29 PM, Miles <se*********@gmail.comwrote:

On Sat, Jul 12, 2008 at 11:23 PM, bukzor <wo**********@gmail.comwrote:

>Anyone know a general reason this might happen? Even better, a way to
fix it?

Maybe your client has an unusually low socket timeout for some reason,
though; does increasing it (with socket.setdefaulttimeout) help?

Never mind on that, as you already said that it hangs for about two
minutes. Clearly my reading comprehension and retention rate are at
an all-time low today.

low-signal-to-noise-ratio-ly yours,
Miles

[bukzor]

On Jul 13, 6:31 pm, Miles <[e-mail]:

On Sun, Jul 13, 2008 at 8:35 PM, bukzor <[e-mail]:

The problem only manifests about 1 in 20 runs. Below there's code for
a client that shows the problem 100% of the time.

The two URL's that I seem to be "confused" about point to the same IP.
Maybe this will make it clear:

PING bukzor.hopto.org (75.61.84.249) 56(84) bytes of data.
64 bytes from adsl-75-61-84-249.dsl.pltn13.sbcglobal.net
(75.61.84.249): icmp_seq=1 ttl=255 time=1.68 ms

For me, buzkor.hopto.org resolves to 69.65.19.125, which I hope
explains why I thought you were confused, and increases my own
suspicion that DNS settings are to blame. I let the script run for
about five minutes without it failing.

Does your luck change if you use "localhost" or a numeric IP address
in the ServerProxy URL?

-Miles

It seems to work fairly perfectly If i use localhost or even my LAN IP
address, but starts to fail if I go beyond that.

You said you ran it for five minuts without error. Did it error out
after that? If you can't reproduce it, that would indicate something
else.