473,379 Members | 1,032 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,379 software developers and data experts.

socket.sendall(), non-blocking sockets, and multi-threaded socket sending

My application requires sending a large piece (~2MB) of data to
several devices on a network via TCP sockets. I have experimented with
different methods for doing this and this has raised some questions
about the implementation of Python sockets.

(both methods use blocking sockets)

Method 1: Calls socket.sendall(data) for each device in sequence, all
in a single thread.

Method 2: Each socket has its own thread that calls
socket.send(datachunk) iteratively until all data is sent.

So Method 1 sends all data to device1, then send all data to device2,
etc... until all data is sent to all devices. Method 2 sends chunks of
data to all devices in parallel. What I am seeing is that Method 1 is
faster than Method 2. Given that the bottleneck is the actual sending
of data on the sockets and not some latency in processing the message
at the other end, this result makes sense to me. Method 2 involves
sending the data in smaller chunks, whereas with Method 1 the
chunksize is selected by the TCP/IP stack implementation, which is
probably more efficient. Also, Method 2 entails context switching
between the send threads.

It turns out that using Method 1, all sendall() calls return
immediately. I am assuming at this point that my app is in the hands
of the TCP/IP stack implementation on my machine (win32-XP). Can
anyone explain what happens inside sendall() for Win32?

Also, I would like to get advice and opinions on what is the most
efficient way to broadcast data to several devices on a network.

Thanks,
T
Jul 18 '05 #1
1 6891
[Tim Black]
My application requires sending a large piece (~2MB) of data to
several devices on a network via TCP sockets. I have experimented with
different methods for doing this and this has raised some questions
about the implementation of Python sockets.

(both methods use blocking sockets)

Method 1: Calls socket.sendall(data) for each device in sequence, all
in a single thread.

Method 2: Each socket has its own thread that calls
socket.send(datachunk) iteratively until all data is sent.

So Method 1 sends all data to device1, then send all data to device2,
etc... until all data is sent to all devices. Method 2 sends chunks of
data to all devices in parallel. What I am seeing is that Method 1 is
faster than Method 2. Given that the bottleneck is the actual sending
of data on the sockets and not some latency in processing the message
at the other end, this result makes sense to me. Method 2 involves
sending the data in smaller chunks, whereas with Method 1 the
chunksize is selected by the TCP/IP stack implementation, which is
probably more efficient. Also, Method 2 entails context switching
between the send threads.

It turns out that using Method 1, all sendall() calls return
immediately. I am assuming at this point that my app is in the hands
of the TCP/IP stack implementation on my machine (win32-XP). Can
anyone explain what happens inside sendall() for Win32?

Also, I would like to get advice and opinions on what is the most
efficient way to broadcast data to several devices on a network.


I'd love to have enough time to get into discussing your findings
about sockets. But unfortunately, I don't :-(

But I did want to point out a module that you may not have come across
that makes the job of distributing information peer-to-peer over LANs
very easy: the spread module. I highly recommend that you take a look
over it: it could save you a lot of wheel-reinvention.

http://www.python.org/other/spread/

HTH,

--
alan kennedy
------------------------------------------------------
check http headers here: http://xhaus.com/headers
email alan: http://xhaus.com/contact/alan
Jul 18 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

12
by: lothar | last post by:
re: 4.2.1 Regular Expression Syntax http://docs.python.org/lib/re-syntax.html *?, +?, ?? Adding "?" after the qualifier makes it perform the match in non-greedy or minimal fashion; as few...
5
by: klaus triendl | last post by:
hi, recently i discovered a memory leak in our code; after some investigation i could reduce it to the following problem: return objects of functions are handled as temporary objects, hence...
25
by: Yves Glodt | last post by:
Hello, if I do this: for row in sqlsth: ________pkcolumns.append(row.strip()) ________etc without a prior:
32
by: Adrian Herscu | last post by:
Hi all, In which circumstances it is appropriate to declare methods as non-virtual? Thanx, Adrian.
22
by: Steve - DND | last post by:
We're currently doing some tests to determine the performance of static vs non-static functions, and we're coming up with some odd(in our opinion) results. We used a very simple setup. One class...
14
by: Patrick Kowalzick | last post by:
Dear all, I have an existing piece of code with a struct with some PODs. struct A { int x; int y; };
11
by: ypjofficial | last post by:
Hello All, So far I have been reading that in case of a polymorphic class ( having at least one virtual function in it), the virtual function call get resolved at run time and during that the...
2
by: Ian825 | last post by:
I need help writing a function for a program that is based upon the various operations of a matrix and I keep getting a "non-aggregate type" error. My guess is that I need to dereference my...
399
by: =?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?= | last post by:
PEP 1 specifies that PEP authors need to collect feedback from the community. As the author of PEP 3131, I'd like to encourage comments to the PEP included below, either here (comp.lang.python), or...
13
by: asm23 | last post by:
Hi,I need some help to clarify the warning "initial value of reference to non-const must be an lvalue". I'm searching in this groups to find someone has the same situation like me. I found in...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: ryjfgjl | last post by:
In our work, we often need to import Excel data into databases (such as MySQL, SQL Server, Oracle) for data analysis and processing. Usually, we use database tools like Navicat or the Excel import...
0
by: taylorcarr | last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
0
by: aa123db | last post by:
Variable and constants Use var or let for variables and const fror constants. Var foo ='bar'; Let foo ='bar';const baz ='bar'; Functions function $name$ ($parameters$) { } ...
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.