468,539 Members | 1,607 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 468,539 developers. It's quick & easy.

transfer rate limiting in socket.py

Hi folks,

I have a need in a network data distribution application to send out
data to folks who want it using the protocol of their choice. I´d
like it to support a variety of protocols and I don´t want to
implement any of them :-)
http, ftp (via ftplib) , https (dunno how yet), ssl, ssh, sftp (via
paramiko)

The thing is... I want rate-limiting so that in the case of a failure
of a single client
I don´t penalize the other clients, or if my server (which is acting
as a client pushing to remote servers.) goes down, it doesn´t saturate
the link when it comes back.

So I want to have all the protocols limit the number of bytes they send
per second.
It looks like the easiest way to do this is to dive into socket.py...
and look! it says:

# Wrapper module for _socket, providing some additional facilities
# implemented in Python.

note the ´additional facilities implemented in python´ ...

so we just add logic to:
-- add a ´maxrate´ argument to the constructor and/or an attribute to
modify the setting...
-- tally bytes, and time, and know when we are going ´too fast´
-- when too fast.. in the ´flush´ routine, in the synchronous case,
sleep for the correct time to come back under budget. in the async,
return without writing.
-- do something similar for reading.

Anybody think this would be fun?

Jun 16 '06 #1
3 2296

I looked at twisted briefly. It looks like it is server oriented.
Does it work in for clients initiating connections?
Jean-Paul Calderone wrote:
On 16 Jun 2006 13:53:48 -0700, Peter Silva <pe*********@ec.gc.ca> wrote:
Hi folks,

I have a need in a network data distribution application to send out
data to folks who want it using the protocol of their choice. I´d
like it to support a variety of protocols and I don´t want to
implement any of them :-)
http, ftp (via ftplib) , https (dunno how yet), ssl, ssh, sftp (via
paramiko)

The thing is... I want rate-limiting so that in the case of a failure
of a single client
I don´t penalize the other clients, or if my server (which is acting
as a client pushing to remote servers.) goes down, it doesn´t saturate
the link when it comes back.

So I want to have all the protocols limit the number of bytes they send
per second.
It looks like the easiest way to do this is to dive into socket.py...
and look! it says:

# Wrapper module for _socket, providing some additional facilities
# implemented in Python.

note the ´additional facilities implemented in python´ ...

so we just add logic to:
-- add a ´maxrate´ argument to the constructor and/or an attribute to
modify the setting...
-- tally bytes, and time, and know when we are going ´too fast´
-- when too fast.. in the ´flush´ routine, in the synchronous case,
sleep for the correct time to come back under budget. in the async,
return without writing.
-- do something similar for reading.

Anybody think this would be fun?


Use Twisted instead. It supports every protocol you mentioned, and
rate limiting too.

Jean-Paul


Jun 19 '06 #2
Peter Silva <Pe*********@ec.gc.ca> wrote:
I looked at twisted briefly. It looks like it is server oriented.
Does it work in for clients initiating connections?


Twisted supports clients, servers, and "middleware" (proxies etc) in
equally wonderful and powerful ways.
Alex
Jun 20 '06 #3

Cool! Will check it out...

Alex Martelli wrote:
Peter Silva <Pe*********@ec.gc.ca> wrote:
I looked at twisted briefly. It looks like it is server oriented.
Does it work in for clients initiating connections?


Twisted supports clients, servers, and "middleware" (proxies etc) in
equally wonderful and powerful ways.
Alex


Jun 20 '06 #4

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

11 posts views Thread by Abhishek | last post: by
reply views Thread by Bob | last post: by
10 posts views Thread by David | last post: by
4 posts views Thread by Dimitrios Apostolou | last post: by
reply views Thread by NPC403 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.