Hi,
I'd like to know if there's a way to determine which is the best
buffer size to use when you have to send() and recv() some data over
the network.
I have an FTP server application which, on data channel, uses 8192
bytes as buffer for both incoming and outgoing data.
Some time ago I received a report from a guy [1] who stated that
changing the buffers from 8192 to 4096 results in a drastical speed
improvement.
I tried to make some tests by using different buffer sizes, from 4 Kb
to 256 Kb, but I'm not sure which one to use as default in my
application since I noticed they can vary from different OSes.
Is there a recommended way to determine the best buffer size to use?
Thanks in advance
[1] http://groups.google.com/group/pyftp...thread/f13a82b...
--- Giampaolo http://code.google.com/p/pyftpdlib/ 3  13232 
On Nov 14, 9:56*am, "Giampaolo Rodola'" <gne...@gmail.c omwrote:
Hi,
I'd like to know if there's a way to determine which is the best
buffer size to use when you have to send() and recv() some data over
the network.
I have an FTP server application which, on data channel, uses 8192
bytes as buffer for both incoming and outgoing data.
Some time ago I received a report from a guy [1] who stated that
changing the buffers from 8192 to 4096 results in a drastical speed
improvement.
I tried to make some tests by using different buffer sizes, from 4 Kb
to 256 Kb, but I'm not sure which one to use as default in my
application since I noticed they can vary from different OSes.
Is there a recommended way to determine the best buffer size to use?
Thanks in advance
[1]http://groups.google.c om/group/pyftpdlib/browse_thread/thread/f13a82b....
--- Giampaolohttp://code.google.com/p/pyftpdlib/
As you stated, the answer is obviously OS/stack dependant. Regardless,
I believe you'll likely find the best answer is between 16K-64K. Once
you consider the various TCP stack improvements which are now
available and the rapid increase of available bandwidth, you'll likely
want to use the largest buffers which do not impose scalability issues
for your system/application. Unless you have reason to use a smaller
buffer, use 64K buffers and be done with it. This helps minimize the
number of context switches and helps ensure the stack always has data
to keep pumping.
To look at it another way, using 64k buffers requires 1/8 the number
of system calls and less time actually spent in python code.
If as you say someone actually observed a performance improvement when
changing from 8k buffers to 4k buffers, it likely has something to do
with python's buffer allocation overhead but even that seems contrary
to my expectation. The referenced article was not available to me so I
was not able to follow and read.
Another possibility is 4k buffers require less fragmentation and is
likely to perform better on lossy connections. Is it possible he/she
was testing on a high lossy connection? In short, performance wise,
TCP stinks on lossy connections.
On Nov 14, 5:27*pm, Greg Copeland <gtcopel...@gma il.comwrote:
On Nov 14, 9:56*am, "Giampaolo Rodola'" <gne...@gmail.c omwrote:
Hi,
I'd like to know if there's a way to determine which is the best
buffer size to use when you have to send() and recv() some data over
the network.
I have an FTP server application which, on data channel, uses 8192
bytes as buffer for both incoming and outgoing data.
Some time ago I received a report from a guy [1] who stated that
changing the buffers from 8192 to 4096 results in a drastical speed
improvement.
I tried to make some tests by using different buffer sizes, from 4 Kb
to 256 Kb, but I'm not sure which one to use as default in my
application since I noticed they can vary from different OSes.
Is there a recommended way to determine the best buffer size to use?
Thanks in advance
[1]http://groups.google.c om/group/pyftpdlib/browse_thread/thread/f13a82b...
--- Giampaolohttp://code.google.com/p/pyftpdlib/
As you stated, the answer is obviously OS/stack dependant. Regardless,
I believe you'll likely find the best answer is between 16K-64K. Once
you consider the various TCP stack improvements which are now
available and the rapid increase of available bandwidth, you'll likely
want to use the largest buffers which do not impose scalability issues
for your system/application. Unless you have reason to use a smaller
buffer, use 64K buffers and be done with it. This helps minimize the
number of context switches and helps ensure the stack always has data
to keep pumping.
To look at it another way, using 64k buffers requires 1/8 the number
of system calls and less time actually spent in python code.
If as you say someone actually observed a performance improvement when
changing from 8k buffers to 4k buffers, it likely has something to do
with python's buffer allocation overhead but even that seems contrary
to my expectation. The referenced article was not available to me so I
was not able to follow and read.
Another possibility is 4k buffers require less fragmentation and is
likely to perform better on lossy connections. Is it possible he/she
was testing on a high lossy connection? In short, performance wise,
TCP stinks on lossy connections.- Hide quoted text -
- Show quoted text -
Thanks for the precious advices.
The discussion I was talking about is this one (sorry for the broken
link, I didn't notice that): http://groups.google.com/group/pyftp...a82b2f41764be#
--- Giampaolo http://code.google.com/p/pyftpdlib/
On Nov 14, 1:58*pm, "Giampaolo Rodola'" <gne...@gmail.c omwrote:
On Nov 14, 5:27*pm, Greg Copeland <gtcopel...@gma il.comwrote:
On Nov 14, 9:56*am, "Giampaolo Rodola'" <gne...@gmail.c omwrote:
Hi,
I'd like to know if there's a way to determine which is the best
buffer size to use when you have to send() and recv() some data over
the network.
I have an FTP server application which, on data channel, uses 8192
bytes as buffer for both incoming and outgoing data.
Some time ago I received a report from a guy [1] who stated that
changing the buffers from 8192 to 4096 results in a drastical speed
improvement.
I tried to make some tests by using different buffer sizes, from 4 Kb
to 256 Kb, but I'm not sure which one to use as default in my
application since I noticed they can vary from different OSes.
Is there a recommended way to determine the best buffer size to use?
Thanks in advance
[1]http://groups.google.c om/group/pyftpdlib/browse_thread/thread/f13a82b...
--- Giampaolohttp://code.google.com/p/pyftpdlib/
As you stated, the answer is obviously OS/stack dependant. Regardless,
I believe you'll likely find the best answer is between 16K-64K. Once
you consider the various TCP stack improvements which are now
available and the rapid increase of available bandwidth, you'll likely
want to use the largest buffers which do not impose scalability issues
for your system/application. Unless you have reason to use a smaller
buffer, use 64K buffers and be done with it. This helps minimize the
number of context switches and helps ensure the stack always has data
to keep pumping.
To look at it another way, using 64k buffers requires 1/8 the number
of system calls and less time actually spent in python code.
If as you say someone actually observed a performance improvement when
changing from 8k buffers to 4k buffers, it likely has something to do
with python's buffer allocation overhead but even that seems contrary
to my expectation. The referenced article was not available to me so I
was not able to follow and read.
Another possibility is 4k buffers require less fragmentation and is
likely to perform better on lossy connections. Is it possible he/she
was testing on a high lossy connection? In short, performance wise,
TCP stinks on lossy connections.- Hide quoted text -
- Show quoted text -
Thanks for the precious advices.
The discussion I was talking about is this one (sorry for the broken
link, I didn't notice that):http://groups.google.com/group/pyftp...thread/f13a82b...
--- Giampaolohttp://code.google.com/p/pyftpdlib/
I read the provided link. There really isn't enough information to
explain what he observed. It is safe to say, his report is contrary to
common performance expectations and my own experience. Since he also
reported large swings in bandwidth far below his potential max, I'm
inclined to say he was suffering from some type of network
abnormality. To be clear, that's just a guess. For all we know some
script kiddie was attempting to scan/hack his system at that given
time - or any number of other variables. One can only be left making
wild assumptions about his operating environment and it's not even
clear if his results are reproducible. Lastly, keep in mind, many
people do not know how to properly benchmark simple applications, let
alone accurately measure bandwidth.
Keep in mind, python can typically saturate a 10Mb link even on fairly
low end systems so it's not likely your application was his problem.
For now, use large buffers unless you can prove otherwise. This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics | |
by: daniel.shaya |
last post by:
I'll try and keep this brief so in a nutshell:
I have large distributed java system running on a Windows 2003 server
(4cpu 8Gb memory).
Periodically the following exceptions occurs in the servers:
java.net.SocketException: No buffer space available (maximum
connections reached?): recv failed
|
by: nephish |
last post by:
lo there,
i have a simple app that connects to a socket to get info from a server
i looks like this
serverhost = 'xxx.xxx.xxx.xxx'
serverport = 9520
aeris_sockobj = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
aeris_sockobj.connect((serverhost,serverport))
|
by: Greg Young |
last post by:
Ok so I think everyone can agree that creating buffers on the fly in
an async socket server is bad ... there is alot of literature
available on the problems this will cause with the heap. I am looking
at a few options to get around this.
1) Have a BufferPool class that hands out ArraySegment<byteportions
of a larger array (large enough that it would be in the LOH). If all
of the array is used create another big segment.
2) Create a...
|
by: semedao |
last post by:
Hi , I am using asyc sockets p2p connection between 2 clients.
when I debug step by step the both sides , i'ts work ok.
when I run it , in somepoint (same location in the code)
when I want to receive 5 bytes buffer , I call the BeginReceive and then
wait on AsyncWaitHandle.WaitOne()
but it is signald imidiatly , and the next call to EndReceive return zero
bytes length , also the buffer is empty.
here is the code:
public static byte...
|
by: ppuniversal |
last post by:
Can anyone tell me how to flush a socket stream in C++.
I am making an Client Server application in C++ and use the send() and recv() functions. i want to flush the socket buffer as i am getting unwanted data when i use recv().
A part of my server code is like this:
int bytesRecv = SOCKET_ERROR;
char recvbuf="";
int ch ;
bytesRecv = recv( clientSocket, recvbuf, 1, 0 );
| | |
by: Aditya |
last post by:
Hi
I am using recv() from socket.h in one of my TCP-client projects.
The problem is that the buffer variable in recv(socketDescriptor,
buffer, flags) points to some stray location and when the incoming
string is filled into it, there are trailing junk characters also.
The code portion refering to it is somewhat like:
int rcvStatus;
char *buffer;
|
by: Aditya |
last post by:
Hi
I am using recv() from socket.h in one of my TCP-client projects.
The problem is that the buffer variable in recv(socketDescriptor,
buffer, flags) points to some stray location and when the incoming
string is filled into it, there are trailing junk characters also.
The code portion refering to it is somewhat like:
int rcvStatus;
char *buffer;
|
by: danfolkes |
last post by:
Hey Everyone, I am trying to send repeated messages from a "Node" to a
"Server". It works the first time I send the from the Node to Server,
but after that it either errors, or does not do anything.
I would love some help, here is the code:
import socket
import thread
import time
|
by: John Salerno |
last post by:
I wrote some pretty basic socket programming again, but I'm still confused about what's happening with the buffer_size variable. Here are the server and client programs:
--------------
from socket import *
host = ''
port = 51567
address = (host, port)
buffer_size = 1024
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| | |
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| | |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |
