sockets: How to know when your data is sent

[Marc Ederis]

Hello,

I'm having a problem with sending data with sockets over a dial-up
connection. When I use the send function, it will happily send a
buffer of a megabyte and more in one shot. But of course, the data is
still in the network buffer... Meaning you can't disconnect for awhile
(but for how long...). The problem is, how can I know when it's done?
Is there a way to be notified when the data has truly been sent?

I tried using setsockopt and SO_SNDBUF to reduce the send buffer size,
thinking that the data would actually send in smaller increments,
instead of putting the whole thing in a buffer and taking it out of my
hands... No luck. It all works, mind you, but I'm left clueless as to
when the data is really off my computer!

I'm just following the example in the Python sockets How-To. Here's
the send function:

def mysend(msg):
totalsent = 0
while totalsent < MSGLEN:
sent = self.sock.send( msg[totalsent:])
if sent == 0:
raise RuntimeError, \\
"socket connection broken"
totalsent = totalsent + sent

Is there something I'm missing, or is there a better way to do this?

Thanks,
-Marc

[Grant Edwards]

On 2004-11-09, Marc Ederis <me*****@hotmai l.com> wrote:

I'm having a problem with sending data with sockets over a
dial-up connection. When I use the send function, it will
happily send a buffer of a megabyte and more in one shot. But
of course, the data is still in the network buffer... Meaning
you can't disconnect for awhile (but for how long...). The
problem is, how can I know when it's done? Is there a way to
be notified when the data has truly been sent?
What OS are you using?

Under Linux, there is no way to tell. IIRC, there's no way to
tell under BSD either. I actually wrote some kernel mode code
once upon a time to provide an ioctl() that I could call from
user space to find out when the data I had written to a socket
was actually sent.
Is there something I'm missing, or is there a better way to do
this?

The way to tell that data has been sent is to use an
application-level protocol that acknowleges the data transferr.

--
Grant Edwards grante Yow! Remember, in 2039,
at MOUSSE & PASTA will
visi.com be available ONLY by
prescription!!

[Sion Arrowsmith]

Grant Edwards <gr****@visi.co m> wrote:

On 2004-11-09, Marc Ederis <me*****@hotmai l.com> wrote:

When I use the send function, it will
happily send a buffer of a megabyte and more in one shot. But
of course, the data is still in the network buffer...
[ ... ] The
problem is, how can I know when it's done?Under Linux, there is no way to tell. IIRC, there's no way to
tell under BSD either. I actually wrote some kernel mode code
once upon a time to provide an ioctl() that I could call from
user space to find out when the data I had written to a socket
was actually sent.

I may well be missing the point of the problem here, but wouldn't
just turning TCP_NODELAY on guarantee that send() won't return
until the last ACK has returned? And if you're not using TCP:
The way to tell that data has been sent is to use an
application-level protocol that acknowleges the data transferr.

applies in spades if you care about reliability.

--
\S -- si***@chiark.gr eenend.org.uk -- http://www.chaos.org.uk/~sion/
___ | "Frankly I have no feelings towards penguins one way or the other"
\X/ | -- Arthur C. Clarke
her nu becomeþ se bera eadward ofdun hlæddre heafdes bæce bump bump bump

[Grant Edwards]

On 2004-11-09, Sion Arrowsmith <si***@chiark.g reenend.org.uk> wrote:

When I use the send function, it will happily send a buffer of
a megabyte and more in one shot. But of course, the data is
still in the network buffer... [ ... ] The problem is, how
can I know when it's done?
Under Linux, there is no way to tell. IIRC, there's no way to
tell under BSD either. I actually wrote some kernel mode code
once upon a time to provide an ioctl() that I could call from
user space to find out when the data I had written to a socket
was actually sent.

I may well be missing the point of the problem here, but wouldn't
just turning TCP_NODELAY on guarantee that send() won't return
until the last ACK has returned?

Nope. Send returns as soon as the the data is copied to a a
kernel-space buffer. TCP_NODELAY has no effect on that. It
only changes the stack's behavior as it's taking data from that
buffer and putting it on the wire. Eventually, the TCP window
will fill up, and after than the tx buffer will fill up, and
after that, send will block. There's no practical way to
predict the size of either the TCP window or the tx buffer.
There's also no practical way to determine how much data is
still waiting to be acked (unless you want to do kernel
hacking).
And if you're not using TCP:

The way to tell that data has been sent is to use an
application-level protocol that acknowleges the data transferr.

applies in spades if you care about reliability.

Unfortunately, many people who write network applications (and
protocols) make all sorts of bizarre assumptions, and somebody
ends up jumping through hoops later to try to fix things when
those assumptions turn out to be wrong. :/

--
Grant Edwards grante Yow! I wonder if I should
at put myself in ESCROW!!
visi.com

[Marc Ederis]

Grant Edwards <gr****@visi.co m> wrote in message news:<41******* *************** *@visi.com>...

On 2004-11-09, Sion Arrowsmith <si***@chiark.g reenend.org.uk> wrote:

When I use the send function, it will happily send a buffer of
a megabyte and more in one shot. But of course, the data is
still in the network buffer... [ ... ] The problem is, how
can I know when it's done?

Under Linux, there is no way to tell. IIRC, there's no way to
tell under BSD either. I actually wrote some kernel mode code
once upon a time to provide an ioctl() that I could call from
user space to find out when the data I had written to a socket
was actually sent.

I may well be missing the point of the problem here, but wouldn't
just turning TCP_NODELAY on guarantee that send() won't return
until the last ACK has returned?

Nope. Send returns as soon as the the data is copied to a a
kernel-space buffer. TCP_NODELAY has no effect on that. It
only changes the stack's behavior as it's taking data from that
buffer and putting it on the wire. Eventually, the TCP window
will fill up, and after than the tx buffer will fill up, and
after that, send will block. There's no practical way to
predict the size of either the TCP window or the tx buffer.
There's also no practical way to determine how much data is
still waiting to be acked (unless you want to do kernel
hacking).

And if you're not using TCP:

The way to tell that data has been sent is to use an
application-level protocol that acknowleges the data transferr.

applies in spades if you care about reliability.

Unfortunately, many people who write network applications (and
protocols) make all sorts of bizarre assumptions, and somebody
ends up jumping through hoops later to try to fix things when
those assumptions turn out to be wrong. :/

I see, so the right way to do it is to have the server respond on a
successful receipt of the data. Thanks Grant. And thanks for
mentioning TCP_NODELAY, Sion, because I was about to try that too, but
this saves me the trouble.

-Marc

[Jaime Wyant]

On 09 Nov 2004 17:26:03 GMT, Grant Edwards <gr****@visi.co m> wrote:
[ snip! ]

Unfortunately, many people who write network applications (and
protocols) make all sorts of bizarre assumptions, and somebody
ends up jumping through hoops later to try to fix things when
those assumptions turn out to be wrong. :/

Woah! This is all enlightening news to me. Of course, what little
bit of socket level stuff I've done, I've always had a
"send/acknowledge" type protocol wrapped around. I guess I've just
been lucky so far :-).

jw

[Grant Edwards]

On 2004-11-10, Jaime Wyant <pr***********@ gmail.com> wrote:

On 09 Nov 2004 17:26:03 GMT, Grant Edwards <gr****@visi.co m> wrote:
[ snip! ]

Unfortunately, many people who write network applications (and
protocols) make all sorts of bizarre assumptions, and somebody
ends up jumping through hoops later to try to fix things when
those assumptions turn out to be wrong. :/

Woah! This is all enlightening news to me. Of course, what little
bit of socket level stuff I've done, I've always had a
"send/acknowledge" type protocol wrapped around. I guess I've just
been lucky so far :-).

While we're on the topic, another common problem occurs when
people assume that there is a 1:1 correspondance between
write() and read() operations when using TCP. IOW, they assume
that if they write() (or send()) a block of data of size N,
that it will arrive at the other end as a block and be read as
a block of size N. Under some conditions, this will often be
true. I ran into one app that worked under this assumption for
years, but broken when a satellite link was placed in the path.

--
Grant Edwards grante Yow! With YOU, I can be
at MYSELF... We don't NEED
visi.com Dan Rather...

[Richie Hindle]

[Marc]

When I use the send function, it will
happily send a buffer of a megabyte and more in one shot. But
of course, the data is still in the network buffer... Meaning
you can't disconnect for awhile (but for how long...). The
problem is, how can I know when it's done? Is there a way to
be notified when the data has truly been sent?
[Grant] Under Linux, there is no way to tell. IIRC, there's no way to
tell under BSD either. [...]
The way to tell that data has been sent is to use an
application-level protocol that acknowleges the data transferr.

If you want to ensure that all the data has been sent before *closing* the
socket, which sounds like Marc's requirement, you can use
setsockopt(SO_L INGER):

SO_LINGER
Sets or gets the SO_LINGER option. The argument is
a linger structure.

struct linger {
int l_onoff; /* linger active */
int l_linger; /* how many seconds to linger for */
};

When enabled, a close(2) or shutdown(2) will not
return until all queued messages for the socket
have been successfully sent or the linger timeout
has been reached. Otherwise, the call returns imme*
diately and the closing is done in the background.
When the socket is closed as part of exit(2), it
always lingers in the background.

(from the GNU manpages). I don't believe that's true for shutdown() on
all platforms, but I do believe it's true for close() / closesocket().
Calling it from Python is a bit of a chore involving the struct module,
and is left as an exercise for the reader. 8-)

--
Richie Hindle
ri****@entrian. com

[Gandalf]

Woah! This is all enlightening news to me. Of course, what little
bit of socket level stuff I've done, I've always had a
"send/acknowledge" type protocol wrapped around. I guess I've just
been lucky so far :-).

Oh yeah. I did the same and I realized that the response time can be
very long.
I had to reconsider my protocol, and use as few send/ack pairs as possible.
No matter if you have a broadband connection or not - it can be VERY slow
if your protocol uses many send/ack pairs.

[Grant Edwards]

On 2004-11-10, Gandalf <ga*****@geoche msource.com> wrote:

Woah! This is all enlightening news to me. Of course, what
little bit of socket level stuff I've done, I've always had a
"send/acknowledge" type protocol wrapped around. I guess I've
just been lucky so far :-).
Oh yeah. I did the same and I realized that the response time
can be very long.

For a badly designed protocol, it can.
I had to reconsider my protocol, and use as few send/ack pairs
as possible. No matter if you have a broadband connection or
not - it can be VERY slow if your protocol uses many send/ack
pairs.

The answer is to use a sliding window protocol.

--
Grant Edwards grante Yow! I'm having a
at RELIGIOUS EXPERIENCE... and
visi.com I don't take any DRUGS