transfer rate limiting in socket.py

[Peter Silva]

Hi folks,

I have a need in a network data distribution application to send out
data to folks who want it using the protocol of their choice. I´d
like it to support a variety of protocols and I don´t want to
implement any of them :-)
http, ftp (via ftplib) , https (dunno how yet), ssl, ssh, sftp (via
paramiko)

The thing is... I want rate-limiting so that in the case of a failure
of a single client
I don´t penalize the other clients, or if my server (which is acting
as a client pushing to remote servers.) goes down, it doesn´t saturate
the link when it comes back.

So I want to have all the protocols limit the number of bytes they send
per second.
It looks like the easiest way to do this is to dive into socket.py...
and look! it says:

# Wrapper module for _socket, providing some additional facilities
# implemented in Python.

note the ´additional facilities implemented in python´ ...

so we just add logic to:
-- add a ´maxrate´ argument to the constructor and/or an attribute to
modify the setting...
-- tally bytes, and time, and know when we are going ´too fast´
-- when too fast.. in the ´flush´ routine, in the synchronous case,
sleep for the correct time to come back under budget. in the async,
return without writing.
-- do something similar for reading.

Anybody think this would be fun?

[Peter Silva]

I looked at twisted briefly. It looks like it is server oriented.
Does it work in for clients initiating connections?
Jean-Paul Calderone wrote:

On 16 Jun 2006 13:53:48 -0700, Peter Silva <pe*********@ec.gc.ca> wrote:

Hi folks,

I have a need in a network data distribution application to send out
data to folks who want it using the protocol of their choice. I´d
like it to support a variety of protocols and I don´t want to
implement any of them :-)
http, ftp (via ftplib) , https (dunno how yet), ssl, ssh, sftp (via
paramiko)

The thing is... I want rate-limiting so that in the case of a failure
of a single client
I don´t penalize the other clients, or if my server (which is acting
as a client pushing to remote servers.) goes down, it doesn´t saturate
the link when it comes back.

So I want to have all the protocols limit the number of bytes they send
per second.
It looks like the easiest way to do this is to dive into socket.py...
and look! it says:

# Wrapper module for _socket, providing some additional facilities
# implemented in Python.

note the ´additional facilities implemented in python´ ...

so we just add logic to:
-- add a ´maxrate´ argument to the constructor and/or an attribute to
modify the setting...
-- tally bytes, and time, and know when we are going ´too fast´
-- when too fast.. in the ´flush´ routine, in the synchronous case,
sleep for the correct time to come back under budget. in the async,
return without writing.
-- do something similar for reading.

Anybody think this would be fun?

Use Twisted instead. It supports every protocol you mentioned, and
rate limiting too.

Jean-Paul

[Alex Martelli]

Peter Silva <Pe*********@ec.gc.ca> wrote:

I looked at twisted briefly. It looks like it is server oriented.
Does it work in for clients initiating connections?

Twisted supports clients, servers, and "middleware" (proxies etc) in
equally wonderful and powerful ways.
Alex

[Peter Silva]

Cool! Will check it out...

Alex Martelli wrote:

Peter Silva <Pe*********@ec.gc.ca> wrote:

I looked at twisted briefly. It looks like it is server oriented.
Does it work in for clients initiating connections?

Twisted supports clients, servers, and "middleware" (proxies etc) in
equally wonderful and powerful ways.
Alex