Hello,
I'm having a problem with sending data with sockets over a dial-up
connection. When I use the send function, it will happily send a
buffer of a megabyte and more in one shot. But of course, the data is
still in the network buffer... Meaning you can't disconnect for awhile
(but for how long...). The problem is, how can I know when it's done?
Is there a way to be notified when the data has truly been sent?
I tried using setsockopt and SO_SNDBUF to reduce the send buffer size,
thinking that the data would actually send in smaller increments,
instead of putting the whole thing in a buffer and taking it out of my
hands... No luck. It all works, mind you, but I'm left clueless as to
when the data is really off my computer!
I'm just following the example in the Python sockets How-To. Here's
the send function:
def mysend(msg):
totalsent = 0
while totalsent < MSGLEN:
sent = self.sock.send(msg[totalsent:])
if sent == 0:
raise RuntimeError, \\
"socket connection broken"
totalsent = totalsent + sent
Is there something I'm missing, or is there a better way to do this?
Thanks,
-Marc 12 2246
On 2004-11-09, Marc Ederis <me*****@hotmail.com> wrote: I'm having a problem with sending data with sockets over a dial-up connection. When I use the send function, it will happily send a buffer of a megabyte and more in one shot. But of course, the data is still in the network buffer... Meaning you can't disconnect for awhile (but for how long...). The problem is, how can I know when it's done? Is there a way to be notified when the data has truly been sent?
What OS are you using?
Under Linux, there is no way to tell. IIRC, there's no way to
tell under BSD either. I actually wrote some kernel mode code
once upon a time to provide an ioctl() that I could call from
user space to find out when the data I had written to a socket
was actually sent.
Is there something I'm missing, or is there a better way to do this?
The way to tell that data has been sent is to use an
application-level protocol that acknowleges the data transferr.
--
Grant Edwards grante Yow! Remember, in 2039,
at MOUSSE & PASTA will
visi.com be available ONLY by
prescription!!
Grant Edwards <gr****@visi.com> wrote: On 2004-11-09, Marc Ederis <me*****@hotmail.com> wrote: When I use the send function, it will happily send a buffer of a megabyte and more in one shot. But of course, the data is still in the network buffer... [ ... ] The problem is, how can I know when it's done?Under Linux, there is no way to tell. IIRC, there's no way to tell under BSD either. I actually wrote some kernel mode code once upon a time to provide an ioctl() that I could call from user space to find out when the data I had written to a socket was actually sent.
I may well be missing the point of the problem here, but wouldn't
just turning TCP_NODELAY on guarantee that send() won't return
until the last ACK has returned? And if you're not using TCP:
The way to tell that data has been sent is to use an application-level protocol that acknowleges the data transferr.
applies in spades if you care about reliability.
--
\S -- si***@chiark.greenend.org.uk -- http://www.chaos.org.uk/~sion/
___ | "Frankly I have no feelings towards penguins one way or the other"
\X/ | -- Arthur C. Clarke
her nu becomeþ se bera eadward ofdun hlæddre heafdes bæce bump bump bump
On 2004-11-09, Sion Arrowsmith <si***@chiark.greenend.org.uk> wrote: When I use the send function, it will happily send a buffer of a megabyte and more in one shot. But of course, the data is still in the network buffer... [ ... ] The problem is, how can I know when it's done? Under Linux, there is no way to tell. IIRC, there's no way to tell under BSD either. I actually wrote some kernel mode code once upon a time to provide an ioctl() that I could call from user space to find out when the data I had written to a socket was actually sent.
I may well be missing the point of the problem here, but wouldn't just turning TCP_NODELAY on guarantee that send() won't return until the last ACK has returned?
Nope. Send returns as soon as the the data is copied to a a
kernel-space buffer. TCP_NODELAY has no effect on that. It
only changes the stack's behavior as it's taking data from that
buffer and putting it on the wire. Eventually, the TCP window
will fill up, and after than the tx buffer will fill up, and
after that, send will block. There's no practical way to
predict the size of either the TCP window or the tx buffer.
There's also no practical way to determine how much data is
still waiting to be acked (unless you want to do kernel
hacking).
And if you're not using TCP:
The way to tell that data has been sent is to use an application-level protocol that acknowleges the data transferr.
applies in spades if you care about reliability.
Unfortunately, many people who write network applications (and
protocols) make all sorts of bizarre assumptions, and somebody
ends up jumping through hoops later to try to fix things when
those assumptions turn out to be wrong. :/
--
Grant Edwards grante Yow! I wonder if I should
at put myself in ESCROW!!
visi.com
Grant Edwards <gr****@visi.com> wrote in message news:<41***********************@visi.com>... On 2004-11-09, Sion Arrowsmith <si***@chiark.greenend.org.uk> wrote:
When I use the send function, it will happily send a buffer of a megabyte and more in one shot. But of course, the data is still in the network buffer... [ ... ] The problem is, how can I know when it's done?
Under Linux, there is no way to tell. IIRC, there's no way to tell under BSD either. I actually wrote some kernel mode code once upon a time to provide an ioctl() that I could call from user space to find out when the data I had written to a socket was actually sent.
I may well be missing the point of the problem here, but wouldn't just turning TCP_NODELAY on guarantee that send() won't return until the last ACK has returned?
Nope. Send returns as soon as the the data is copied to a a kernel-space buffer. TCP_NODELAY has no effect on that. It only changes the stack's behavior as it's taking data from that buffer and putting it on the wire. Eventually, the TCP window will fill up, and after than the tx buffer will fill up, and after that, send will block. There's no practical way to predict the size of either the TCP window or the tx buffer. There's also no practical way to determine how much data is still waiting to be acked (unless you want to do kernel hacking).
And if you're not using TCP:
The way to tell that data has been sent is to use an application-level protocol that acknowleges the data transferr.
applies in spades if you care about reliability.
Unfortunately, many people who write network applications (and protocols) make all sorts of bizarre assumptions, and somebody ends up jumping through hoops later to try to fix things when those assumptions turn out to be wrong. :/
I see, so the right way to do it is to have the server respond on a
successful receipt of the data. Thanks Grant. And thanks for
mentioning TCP_NODELAY, Sion, because I was about to try that too, but
this saves me the trouble.
-Marc
On 09 Nov 2004 17:26:03 GMT, Grant Edwards <gr****@visi.com> wrote:
[ snip! ] Unfortunately, many people who write network applications (and protocols) make all sorts of bizarre assumptions, and somebody ends up jumping through hoops later to try to fix things when those assumptions turn out to be wrong. :/
Woah! This is all enlightening news to me. Of course, what little
bit of socket level stuff I've done, I've always had a
"send/acknowledge" type protocol wrapped around. I guess I've just
been lucky so far :-).
jw
On 2004-11-10, Jaime Wyant <pr***********@gmail.com> wrote: On 09 Nov 2004 17:26:03 GMT, Grant Edwards <gr****@visi.com> wrote: [ snip! ] Unfortunately, many people who write network applications (and protocols) make all sorts of bizarre assumptions, and somebody ends up jumping through hoops later to try to fix things when those assumptions turn out to be wrong. :/
Woah! This is all enlightening news to me. Of course, what little bit of socket level stuff I've done, I've always had a "send/acknowledge" type protocol wrapped around. I guess I've just been lucky so far :-).
While we're on the topic, another common problem occurs when
people assume that there is a 1:1 correspondance between
write() and read() operations when using TCP. IOW, they assume
that if they write() (or send()) a block of data of size N,
that it will arrive at the other end as a block and be read as
a block of size N. Under some conditions, this will often be
true. I ran into one app that worked under this assumption for
years, but broken when a satellite link was placed in the path.
--
Grant Edwards grante Yow! With YOU, I can be
at MYSELF... We don't NEED
visi.com Dan Rather...
[Marc] When I use the send function, it will happily send a buffer of a megabyte and more in one shot. But of course, the data is still in the network buffer... Meaning you can't disconnect for awhile (but for how long...). The problem is, how can I know when it's done? Is there a way to be notified when the data has truly been sent?
[Grant] Under Linux, there is no way to tell. IIRC, there's no way to tell under BSD either. [...] The way to tell that data has been sent is to use an application-level protocol that acknowleges the data transferr.
If you want to ensure that all the data has been sent before *closing* the
socket, which sounds like Marc's requirement, you can use
setsockopt(SO_LINGER):
SO_LINGER
Sets or gets the SO_LINGER option. The argument is
a linger structure.
struct linger {
int l_onoff; /* linger active */
int l_linger; /* how many seconds to linger for */
};
When enabled, a close(2) or shutdown(2) will not
return until all queued messages for the socket
have been successfully sent or the linger timeout
has been reached. Otherwise, the call returns imme*
diately and the closing is done in the background.
When the socket is closed as part of exit(2), it
always lingers in the background.
(from the GNU manpages). I don't believe that's true for shutdown() on
all platforms, but I do believe it's true for close() / closesocket().
Calling it from Python is a bit of a chore involving the struct module,
and is left as an exercise for the reader. 8-)
--
Richie Hindle ri****@entrian.com Woah! This is all enlightening news to me. Of course, what little bit of socket level stuff I've done, I've always had a "send/acknowledge" type protocol wrapped around. I guess I've just been lucky so far :-).
Oh yeah. I did the same and I realized that the response time can be
very long.
I had to reconsider my protocol, and use as few send/ack pairs as possible.
No matter if you have a broadband connection or not - it can be VERY slow
if your protocol uses many send/ack pairs.
On 2004-11-10, Gandalf <ga*****@geochemsource.com> wrote: Woah! This is all enlightening news to me. Of course, what little bit of socket level stuff I've done, I've always had a "send/acknowledge" type protocol wrapped around. I guess I've just been lucky so far :-). Oh yeah. I did the same and I realized that the response time can be very long.
For a badly designed protocol, it can.
I had to reconsider my protocol, and use as few send/ack pairs as possible. No matter if you have a broadband connection or not - it can be VERY slow if your protocol uses many send/ack pairs.
The answer is to use a sliding window protocol.
--
Grant Edwards grante Yow! I'm having a
at RELIGIOUS EXPERIENCE... and
visi.com I don't take any DRUGS me*****@hotmail.com (Marc Ederis) wrote in message news:<f0**************************@posting.google. com>... I'm having a problem with sending data with sockets over a dial-up connection. When I use the send function, it will happily send a buffer of a megabyte and more in one shot. But of course, the data is still in the network buffer... Meaning you can't disconnect for awhile (but for how long...). The problem is, how can I know when it's done? Is there a way to be notified when the data has truly been sent?
You can get pretty close if you are willing to close the socket (but
since you want to disconnect, that should be fine).
What you need to do is turn on linger mode with a very large timeout (large
enough so that your slow connection has enough time to send the data).
Once linger is turned on, close() will block until all the data has
been sent.
The one issue that you can run into is if the linger time runs out before
all the data is sent then the rest of the data is discarded. But as long
as you set linger long enough (how about 10 years), you will run out of
patience long before the data is discarded.
samuel
Richie Hindle <ri****@entrian.com> wrote in message news:<ma**************************************@pyt hon.org>... If you want to ensure that all the data has been sent before *closing* the socket, which sounds like Marc's requirement, you can use setsockopt(SO_LINGER):
SO_LINGER Sets or gets the SO_LINGER option. The argument is a linger structure.
struct linger { int l_onoff; /* linger active */ int l_linger; /* how many seconds to linger for */ };
When enabled, a close(2) or shutdown(2) will not return until all queued messages for the socket have been successfully sent or the linger timeout has been reached. Otherwise, the call returns imme* diately and the closing is done in the background. When the socket is closed as part of exit(2), it always lingers in the background.
(from the GNU manpages). I don't believe that's true for shutdown() on all platforms, but I do believe it's true for close() / closesocket(). Calling it from Python is a bit of a chore involving the struct module, and is left as an exercise for the reader. 8-)
SO_LINGER... Interesting, I'll keep that in mind for the future. In
the end, acknowledging receipt of the data, as it was arriving at the
server, turned out to be the best solution for me. I wanted to be able
to show a progress bar (at the client) of the data being sent.
Thanks guys,
-Marc
Sion Arrowsmith <si***@chiark.greenend.org.uk> wrote: The way to tell that data has been sent is to use an application-level protocol that acknowleges the data transferr.
applies in spades if you care about reliability.
Oh yep: http://web.mit.edu/Saltzer/www/publi...d/endtoend.pdf
Alex This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Robert A. van Ginkel |
last post by:
In news:OZ0W9RsdDHA.2432@TK2MSFTNGP10.phx.gbl... I ask the question how I
can see if all the data is on the other side of the connection.
I got as answer that I should use the blocking property.
I...
|
by: Laxmikant Rashinkar |
last post by:
Is there any way to use a C# socket in promiscuous mode?
Any sample code that shows how this is done?
any assistance is much appreciated!
thanks
LK
|
by: Stuart Norris |
last post by:
Dear Group,
I am having a problem setting SocketOptionName.SendTimeout on a client
TCPIP application using the sockets in .NET. From the on-line help it
is possible to set a...
|
by: Michael Maercker |
last post by:
hi!
i'm really not into networking at all and have now been asigned the task of
porting a vb6-code into vb.net (compact framework, in this case) and the
code uses the winsock-control. i quickly...
|
by: a_agaga |
last post by:
Do you know are there some reasons why many do not make processes to
communicate through memory?
Why network connections (sockets) are used so commonly in IPC (inter
process communication)...
|
by: eliss.carmine |
last post by:
I'm using TCP/IP to send a Bitmap object over Sockets. This is my first
time using C# at all so I don't know if this is the "right" way to do
it. I've already found out several times the way I was...
|
by: Cichy |
last post by:
Hello,
I'm writing a Client-Server application using sockets (asynchronous).
There is a Server (Master) which accepts incoming connections, and
Client (Slave).
Afetr establishing connections...
|
by: DolphinDB |
last post by:
Tired of spending countless mintues downsampling your data? Look no further!
In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
|
by: ryjfgjl |
last post by:
ExcelToDatabase: batch import excel into database automatically...
|
by: isladogs |
last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM).
In this month's session, we are pleased to welcome back...
|
by: Vimpel783 |
last post by:
Hello!
Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
|
by: PapaRatzi |
last post by:
Hello,
I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
|
by: CloudSolutions |
last post by:
Introduction:
For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
|
by: Shællîpôpï 09 |
last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
|
by: af34tf |
last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome former...
| |