473,322 Members | 1,314 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,322 software developers and data experts.

sockets: How to know when your data is sent

Hello,

I'm having a problem with sending data with sockets over a dial-up
connection. When I use the send function, it will happily send a
buffer of a megabyte and more in one shot. But of course, the data is
still in the network buffer... Meaning you can't disconnect for awhile
(but for how long...). The problem is, how can I know when it's done?
Is there a way to be notified when the data has truly been sent?

I tried using setsockopt and SO_SNDBUF to reduce the send buffer size,
thinking that the data would actually send in smaller increments,
instead of putting the whole thing in a buffer and taking it out of my
hands... No luck. It all works, mind you, but I'm left clueless as to
when the data is really off my computer!

I'm just following the example in the Python sockets How-To. Here's
the send function:

def mysend(msg):
totalsent = 0
while totalsent < MSGLEN:
sent = self.sock.send(msg[totalsent:])
if sent == 0:
raise RuntimeError, \\
"socket connection broken"
totalsent = totalsent + sent

Is there something I'm missing, or is there a better way to do this?

Thanks,
-Marc
Jul 18 '05 #1
12 2246
On 2004-11-09, Marc Ederis <me*****@hotmail.com> wrote:
I'm having a problem with sending data with sockets over a
dial-up connection. When I use the send function, it will
happily send a buffer of a megabyte and more in one shot. But
of course, the data is still in the network buffer... Meaning
you can't disconnect for awhile (but for how long...). The
problem is, how can I know when it's done? Is there a way to
be notified when the data has truly been sent?
What OS are you using?

Under Linux, there is no way to tell. IIRC, there's no way to
tell under BSD either. I actually wrote some kernel mode code
once upon a time to provide an ioctl() that I could call from
user space to find out when the data I had written to a socket
was actually sent.
Is there something I'm missing, or is there a better way to do
this?


The way to tell that data has been sent is to use an
application-level protocol that acknowleges the data transferr.

--
Grant Edwards grante Yow! Remember, in 2039,
at MOUSSE & PASTA will
visi.com be available ONLY by
prescription!!
Jul 18 '05 #2
Grant Edwards <gr****@visi.com> wrote:
On 2004-11-09, Marc Ederis <me*****@hotmail.com> wrote:
When I use the send function, it will
happily send a buffer of a megabyte and more in one shot. But
of course, the data is still in the network buffer...
[ ... ] The
problem is, how can I know when it's done?Under Linux, there is no way to tell. IIRC, there's no way to
tell under BSD either. I actually wrote some kernel mode code
once upon a time to provide an ioctl() that I could call from
user space to find out when the data I had written to a socket
was actually sent.


I may well be missing the point of the problem here, but wouldn't
just turning TCP_NODELAY on guarantee that send() won't return
until the last ACK has returned? And if you're not using TCP:
The way to tell that data has been sent is to use an
application-level protocol that acknowleges the data transferr.


applies in spades if you care about reliability.

--
\S -- si***@chiark.greenend.org.uk -- http://www.chaos.org.uk/~sion/
___ | "Frankly I have no feelings towards penguins one way or the other"
\X/ | -- Arthur C. Clarke
her nu becomeþ se bera eadward ofdun hlæddre heafdes bæce bump bump bump
Jul 18 '05 #3
On 2004-11-09, Sion Arrowsmith <si***@chiark.greenend.org.uk> wrote:
When I use the send function, it will happily send a buffer of
a megabyte and more in one shot. But of course, the data is
still in the network buffer... [ ... ] The problem is, how
can I know when it's done?
Under Linux, there is no way to tell. IIRC, there's no way to
tell under BSD either. I actually wrote some kernel mode code
once upon a time to provide an ioctl() that I could call from
user space to find out when the data I had written to a socket
was actually sent.


I may well be missing the point of the problem here, but wouldn't
just turning TCP_NODELAY on guarantee that send() won't return
until the last ACK has returned?


Nope. Send returns as soon as the the data is copied to a a
kernel-space buffer. TCP_NODELAY has no effect on that. It
only changes the stack's behavior as it's taking data from that
buffer and putting it on the wire. Eventually, the TCP window
will fill up, and after than the tx buffer will fill up, and
after that, send will block. There's no practical way to
predict the size of either the TCP window or the tx buffer.
There's also no practical way to determine how much data is
still waiting to be acked (unless you want to do kernel
hacking).
And if you're not using TCP:
The way to tell that data has been sent is to use an
application-level protocol that acknowleges the data transferr.


applies in spades if you care about reliability.


Unfortunately, many people who write network applications (and
protocols) make all sorts of bizarre assumptions, and somebody
ends up jumping through hoops later to try to fix things when
those assumptions turn out to be wrong. :/

--
Grant Edwards grante Yow! I wonder if I should
at put myself in ESCROW!!
visi.com
Jul 18 '05 #4
Grant Edwards <gr****@visi.com> wrote in message news:<41***********************@visi.com>...
On 2004-11-09, Sion Arrowsmith <si***@chiark.greenend.org.uk> wrote:
When I use the send function, it will happily send a buffer of
a megabyte and more in one shot. But of course, the data is
still in the network buffer... [ ... ] The problem is, how
can I know when it's done?

Under Linux, there is no way to tell. IIRC, there's no way to
tell under BSD either. I actually wrote some kernel mode code
once upon a time to provide an ioctl() that I could call from
user space to find out when the data I had written to a socket
was actually sent.


I may well be missing the point of the problem here, but wouldn't
just turning TCP_NODELAY on guarantee that send() won't return
until the last ACK has returned?


Nope. Send returns as soon as the the data is copied to a a
kernel-space buffer. TCP_NODELAY has no effect on that. It
only changes the stack's behavior as it's taking data from that
buffer and putting it on the wire. Eventually, the TCP window
will fill up, and after than the tx buffer will fill up, and
after that, send will block. There's no practical way to
predict the size of either the TCP window or the tx buffer.
There's also no practical way to determine how much data is
still waiting to be acked (unless you want to do kernel
hacking).
And if you're not using TCP:
The way to tell that data has been sent is to use an
application-level protocol that acknowleges the data transferr.


applies in spades if you care about reliability.


Unfortunately, many people who write network applications (and
protocols) make all sorts of bizarre assumptions, and somebody
ends up jumping through hoops later to try to fix things when
those assumptions turn out to be wrong. :/

I see, so the right way to do it is to have the server respond on a
successful receipt of the data. Thanks Grant. And thanks for
mentioning TCP_NODELAY, Sion, because I was about to try that too, but
this saves me the trouble.

-Marc
Jul 18 '05 #5
On 09 Nov 2004 17:26:03 GMT, Grant Edwards <gr****@visi.com> wrote:
[ snip! ]
Unfortunately, many people who write network applications (and
protocols) make all sorts of bizarre assumptions, and somebody
ends up jumping through hoops later to try to fix things when
those assumptions turn out to be wrong. :/


Woah! This is all enlightening news to me. Of course, what little
bit of socket level stuff I've done, I've always had a
"send/acknowledge" type protocol wrapped around. I guess I've just
been lucky so far :-).

jw
Jul 18 '05 #6
On 2004-11-10, Jaime Wyant <pr***********@gmail.com> wrote:
On 09 Nov 2004 17:26:03 GMT, Grant Edwards <gr****@visi.com> wrote:
[ snip! ]
Unfortunately, many people who write network applications (and
protocols) make all sorts of bizarre assumptions, and somebody
ends up jumping through hoops later to try to fix things when
those assumptions turn out to be wrong. :/


Woah! This is all enlightening news to me. Of course, what little
bit of socket level stuff I've done, I've always had a
"send/acknowledge" type protocol wrapped around. I guess I've just
been lucky so far :-).


While we're on the topic, another common problem occurs when
people assume that there is a 1:1 correspondance between
write() and read() operations when using TCP. IOW, they assume
that if they write() (or send()) a block of data of size N,
that it will arrive at the other end as a block and be read as
a block of size N. Under some conditions, this will often be
true. I ran into one app that worked under this assumption for
years, but broken when a satellite link was placed in the path.

--
Grant Edwards grante Yow! With YOU, I can be
at MYSELF... We don't NEED
visi.com Dan Rather...
Jul 18 '05 #7

[Marc]
When I use the send function, it will
happily send a buffer of a megabyte and more in one shot. But
of course, the data is still in the network buffer... Meaning
you can't disconnect for awhile (but for how long...). The
problem is, how can I know when it's done? Is there a way to
be notified when the data has truly been sent?
[Grant] Under Linux, there is no way to tell. IIRC, there's no way to
tell under BSD either. [...]
The way to tell that data has been sent is to use an
application-level protocol that acknowleges the data transferr.


If you want to ensure that all the data has been sent before *closing* the
socket, which sounds like Marc's requirement, you can use
setsockopt(SO_LINGER):

SO_LINGER
Sets or gets the SO_LINGER option. The argument is
a linger structure.

struct linger {
int l_onoff; /* linger active */
int l_linger; /* how many seconds to linger for */
};

When enabled, a close(2) or shutdown(2) will not
return until all queued messages for the socket
have been successfully sent or the linger timeout
has been reached. Otherwise, the call returns imme*
diately and the closing is done in the background.
When the socket is closed as part of exit(2), it
always lingers in the background.

(from the GNU manpages). I don't believe that's true for shutdown() on
all platforms, but I do believe it's true for close() / closesocket().
Calling it from Python is a bit of a chore involving the struct module,
and is left as an exercise for the reader. 8-)

--
Richie Hindle
ri****@entrian.com

Jul 18 '05 #8
Woah! This is all enlightening news to me. Of course, what little
bit of socket level stuff I've done, I've always had a
"send/acknowledge" type protocol wrapped around. I guess I've just
been lucky so far :-).

Oh yeah. I did the same and I realized that the response time can be
very long.
I had to reconsider my protocol, and use as few send/ack pairs as possible.
No matter if you have a broadband connection or not - it can be VERY slow
if your protocol uses many send/ack pairs.
Jul 18 '05 #9
On 2004-11-10, Gandalf <ga*****@geochemsource.com> wrote:
Woah! This is all enlightening news to me. Of course, what
little bit of socket level stuff I've done, I've always had a
"send/acknowledge" type protocol wrapped around. I guess I've
just been lucky so far :-).
Oh yeah. I did the same and I realized that the response time
can be very long.


For a badly designed protocol, it can.
I had to reconsider my protocol, and use as few send/ack pairs
as possible. No matter if you have a broadband connection or
not - it can be VERY slow if your protocol uses many send/ack
pairs.


The answer is to use a sliding window protocol.

--
Grant Edwards grante Yow! I'm having a
at RELIGIOUS EXPERIENCE... and
visi.com I don't take any DRUGS
Jul 18 '05 #10
me*****@hotmail.com (Marc Ederis) wrote in message news:<f0**************************@posting.google. com>...
I'm having a problem with sending data with sockets over a dial-up
connection. When I use the send function, it will happily send a
buffer of a megabyte and more in one shot. But of course, the data is
still in the network buffer... Meaning you can't disconnect for awhile
(but for how long...). The problem is, how can I know when it's done?
Is there a way to be notified when the data has truly been sent?


You can get pretty close if you are willing to close the socket (but
since you want to disconnect, that should be fine).

What you need to do is turn on linger mode with a very large timeout (large
enough so that your slow connection has enough time to send the data).
Once linger is turned on, close() will block until all the data has
been sent.

The one issue that you can run into is if the linger time runs out before
all the data is sent then the rest of the data is discarded. But as long
as you set linger long enough (how about 10 years), you will run out of
patience long before the data is discarded.

samuel
Jul 18 '05 #11
Richie Hindle <ri****@entrian.com> wrote in message news:<ma**************************************@pyt hon.org>...
If you want to ensure that all the data has been sent before *closing* the
socket, which sounds like Marc's requirement, you can use
setsockopt(SO_LINGER):

SO_LINGER
Sets or gets the SO_LINGER option. The argument is
a linger structure.

struct linger {
int l_onoff; /* linger active */
int l_linger; /* how many seconds to linger for */
};

When enabled, a close(2) or shutdown(2) will not
return until all queued messages for the socket
have been successfully sent or the linger timeout
has been reached. Otherwise, the call returns imme*
diately and the closing is done in the background.
When the socket is closed as part of exit(2), it
always lingers in the background.

(from the GNU manpages). I don't believe that's true for shutdown() on
all platforms, but I do believe it's true for close() / closesocket().
Calling it from Python is a bit of a chore involving the struct module,
and is left as an exercise for the reader. 8-)


SO_LINGER... Interesting, I'll keep that in mind for the future. In
the end, acknowledging receipt of the data, as it was arriving at the
server, turned out to be the best solution for me. I wanted to be able
to show a progress bar (at the client) of the data being sent.

Thanks guys,
-Marc
Jul 18 '05 #12
Sion Arrowsmith <si***@chiark.greenend.org.uk> wrote:
The way to tell that data has been sent is to use an
application-level protocol that acknowleges the data transferr.


applies in spades if you care about reliability.


Oh yep:
http://web.mit.edu/Saltzer/www/publi...d/endtoend.pdf
Alex
Jul 18 '05 #13

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: Robert A. van Ginkel | last post by:
In news:OZ0W9RsdDHA.2432@TK2MSFTNGP10.phx.gbl... I ask the question how I can see if all the data is on the other side of the connection. I got as answer that I should use the blocking property. I...
6
by: Laxmikant Rashinkar | last post by:
Is there any way to use a C# socket in promiscuous mode? Any sample code that shows how this is done? any assistance is much appreciated! thanks LK
0
by: Stuart Norris | last post by:
Dear Group, I am having a problem setting SocketOptionName.SendTimeout on a client TCPIP application using the sockets in .NET. From the on-line help it is possible to set a...
3
by: Michael Maercker | last post by:
hi! i'm really not into networking at all and have now been asigned the task of porting a vb6-code into vb.net (compact framework, in this case) and the code uses the winsock-control. i quickly...
2
by: a_agaga | last post by:
Do you know are there some reasons why many do not make processes to communicate through memory? Why network connections (sockets) are used so commonly in IPC (inter process communication)...
14
by: eliss.carmine | last post by:
I'm using TCP/IP to send a Bitmap object over Sockets. This is my first time using C# at all so I don't know if this is the "right" way to do it. I've already found out several times the way I was...
5
by: Cichy | last post by:
Hello, I'm writing a Client-Server application using sockets (asynchronous). There is a Server (Master) which accepts incoming connections, and Client (Slave). Afetr establishing connections...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.