473,289 Members | 1,947 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,289 software developers and data experts.

Proxying downloads

Hello,

more a recipe question. I'm working on a proxy that will download a
file for a client. The thing that doesn't yield problems is:

Alice (Client)
Bob (Client)
Sam (Server)

1 Alice asks Sam for "foobar.iso"
2 Sam can't find "foobar.iso" in "cachedir"
3 Sam requests "foobar.iso" from the uplink
4 Sam now saves each chunk received to "cachedir/foobar.iso"
5 At the same time Sam forwards each chunk to Alice.

But I can't figure out how I would solve the following:

1 Alice asks Sam for "foobar.iso"
2 Sam can't find "foobar.iso" in "cachedir"
3 Sam requests "foobar.iso" from uplink
4 Sam saves and forwards to Alice
5 At about 30 % of the download Bob asks Sam for "foobar.iso"
6 How do I serve Bob now?

Now because the internal link is _a lot_ faster than the uplink Bob
will probably reach the end of (the local) "foobar.iso" before Sam has
received "foobar.iso" in total from uplink. So Bob will end up with a
incomplete file...

How do I solve that. The already downloaded data should of course be
served internally.

The solutions I think of are
* Some kind of subscriber list for the file in question
* That is serve internally and if the state of "foobar.iso" is in
progress switch to receiving chunk directly from Sam as it comes down
the link
* How would I realize this switch from internal serving to pass thru
of chunks?

* Send an acknowledge (lie to the client that we have this file in
the cache) wait until it's finished and then serve the file from the
internal cache)
* This could lead to timeouts for very large files, at least I think so

* Forget about all of it and just pass thru from uplink, with a new
request, as long as files are in progress. This would in the worst
case download the file n times where n is the number of clients.
* I guess that's the easiest one but also the least desirable solution.

I hope I explained my problem somehow understandable.

any hints are welcome
thanks
martin

--
http://noneisyours.marcher.name
http://feeds.feedburner.com/NoneIsYours
Oct 30 '07 #1
2 1149
You use a temp directory to store the file while downloading, then
move it to the cache so the addition of the complete file is atomic.
The file name of the temp file should be checked to validate that you
don't overwrite another process' download.

Currently downloading urls should be registered with the server
process (a simple list or set would work). New requests should be
checked against that; if there is a matching url in there, the process
must wait until that download is finished and that file should be
delivered to both Alice and Bob.

You need to store the local file path and the url it was downloaded
from and checking against that when a request is made; there might be
two foobar.iso files on the Internet or the network, and they may be
different (such as in differently versioned directories).

Oct 30 '07 #2
But I can't figure out how I would solve the following:
>
1 Alice asks Sam for "foobar.iso"
2 Sam can't find "foobar.iso" in "cachedir"
3 Sam requests "foobar.iso" from uplink
4 Sam saves and forwards to Alice
5 At about 30 % of the download Bob asks Sam for "foobar.iso"
6 How do I serve Bob now?
Let every file in your download cache be represented by a Python object.
Instead of streaming the file directly to the clients, you can stream
the objects. The object will know if the file it represents has finished
downloading or not, where the file is located etc. This way you can
also, for the sake of persistence, keep partially downloaded files
separate from the completely downloaded files, as per a previous
suggestion, so that you won't start serving half files after a crash,
and it'll be completely transparent in all code except for your proxy
file objects.

Martin
Oct 30 '07 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
by: Earl Teigrob | last post by:
My company sells software and wants to provide downloadable product. Some of these downloads will be full CD's of over 550M I had everything working fine using Response.filewrite() but when we...
1
by: CodeMonkey | last post by:
Hi all, I have an ASP .NET 2.0 project with the following path on my dev machine: D:\Profiles\MyUsername\My Documents\Visual Studio 2005\Projects \abcWebSite2007\abcWebSite2007\Somefile.aspx ...
3
by: Jeff Dege | last post by:
We have an app that is written in C#/ASP.NET 2.0, and hosted on IIS. Or rather, it's mostly written in ASP.NET, but parts of it are client- side javascript, pulling data from an Apache webserver...
4
by: adam.waterfield | last post by:
I have a script that 'proxys' a .flv file from a .htaccess protected folder. The proxying script is being called by a Flash flv player in the main webpage. This script works wonderfully in...
7
by: =?Utf-8?B?QU9UWCBTYW4gQW50b25pbw==?= | last post by:
Hi, I have been using the code (some of it has been removed for simplicity) below to allow authenticated (using ASP.NET membership database) users to get a file from their archive area. It...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.