Advice on a TCP passthru daemon for HTTP proxy rotation

[Robin Haswell]

Hey there

Soon we will have many squid proxies on many seperate connections for use
by our services. I want to make them available to users via a single HTTP
proxy - however, I want fine-grained control over how the squid proxies
are selected for each connection. This is so I can collect statistics,
control usage on each proxy, monitor what's going on - etc. However I
don't want to implement the HTTP proxy protocol in Python, and would
much rather let my daemon run as a man-in-the-middle for TCP, similar to
this netcat command:

rob@aranea:~$ mknod backpipe p
rob@aranea:~$ nc -l -p 8080 < backpipe | nc ganesh 8080 backpipe

Basically when my daemon received a connection (call it "c1"), it makes a
connection to one of my squid proxies ("c2"), then all data which gets
read from c1 is written to c2 - all data read from c2 is written to c1.
I'm pretty sure there's an elegant way to do this but I was wondering if
anyone had any input? I've tried GIYF'ing this but it's difficult to
search for :P

Thanks guys

-Rob

[Rob Wolfe]

Robin Haswell wrote:

Hey there

Soon we will have many squid proxies on many seperate connections for use
by our services. I want to make them available to users via a single HTTP
proxy - however, I want fine-grained control over how the squid proxies
are selected for each connection. This is so I can collect statistics,
control usage on each proxy, monitor what's going on - etc. However I
don't want to implement the HTTP proxy protocol in Python, and would
much rather let my daemon run as a man-in-the-middle for TCP, similar to
this netcat command:

rob@aranea:~$ mknod backpipe p
rob@aranea:~$ nc -l -p 8080 < backpipe | nc ganesh 8080 backpipe

Basically when my daemon received a connection (call it "c1"), it makes a
connection to one of my squid proxies ("c2"), then all data which gets
read from c1 is written to c2 - all data read from c2 is written to c1.
I'm pretty sure there's an elegant way to do this but I was wondering if
anyone had any input? I've tried GIYF'ing this but it's difficult to
search for :P

Maybe this is what you're looking for:

http://aspn.activestate.com/ASPN/Coo.../Recipe/483732

Regards,
Rob

[Bryan Olson]

Robin Haswell wrote:

Hey there

Soon we will have many squid proxies on many seperate connections for use
by our services. I want to make them available to users via a single HTTP
proxy - however, I want fine-grained control over how the squid proxies
are selected for each connection. This is so I can collect statistics,
control usage on each proxy, monitor what's going on - etc. However I
don't want to implement the HTTP proxy protocol in Python, and would
much rather let my daemon run as a man-in-the-middle for TCP, similar to
this netcat command:

rob@aranea:~$ mknod backpipe p
rob@aranea:~$ nc -l -p 8080 < backpipe | nc ganesh 8080 backpipe

Basically when my daemon received a connection (call it "c1"), it makes a
connection to one of my squid proxies ("c2"), then all data which gets
read from c1 is written to c2 - all data read from c2 is written to c1.
I'm pretty sure there's an elegant way to do this but I was wondering if
anyone had any input? I've tried GIYF'ing this but it's difficult to
search for :P

Try searching for "port forwarding". Basic port forwarding is
a straightforward sockets programming exercise, but there are
trickier considerations in building it to face the Internet
and front a large number of of servers. Sounds like your TCP
proxy will be a potential single point of failure, and is the
first machine to which users connect. Not the place to try
out new stuff.

You might look into existing proxies and "load balancers"
before coding your own. There's no rule against fronting
Squid proxies with Squid proxies. You may want different
statistics from what they log, but proxying at the TCP
level, you're limited as to what information you can get.

May I ask why you need many Squid proxies? Unless you're
using them to terminating SSL/TLS connections, just one of
them can probably out-run a Python port-forwarder. Squid
usually serves a large percentage of the volume from its
memory cache, while a port forwarder has to read it all
from TCP connections, which is more work.
--
--Bryan