Iterated mega file upload - invented wheel wanted

[john_ramsden]

I have to prepare an Apache/PHP based (or perl if easier) web
page that allows a user to upload very large files, possibly
over 10 Gbytes and maybe up to 100 Gbytes.

I'm aware that file upload limits can be increased by config
changes; but I'd rather minimize the need for changes like
this, both for admin reasons and to avoid possible (?) memory problems
or even bugs.

In principle there's a simple solution whereby the user runs
an initial page, in which they input the file name, and the
Submit button then runs JavaScript in the page.

The JavaScript opens the file, positions at a start offset
(initially zero), and reads and uploads up to a fixed length
of the contents, along with a continuation offset (or zero
to indicate the end reached), and the server code re-runs
the script in a loop while this returned offset is non-zero.
Subsequent runs of the script would bypass the data input
and execute the JavaScript directly.

Anyway, the question I have is whether anyone has written
anything similar, ideally with a progress bar?

The only problem I anticipate is possible delays, and even
timeouts, while the code (client or server) positions to
the next offset to read or write (respectively) the next
chunk of a very large, e.g. 50 GByte, file.

Any suggestions gratefully received.

(It's a shame Apache doesn't seem to have an embedded FTP
server in some way. I guess FTP would be the obvious choice,
but the client doesn't wish to have a separate FTP server
running.)
Cheers

John R Ramsden (jramsden at glassouse.com)

[A. Sinan Unur]

jo**********@sagitta-ps.com wrote in news:1109862480.383312.250480
@z14g2000cwz.googlegroups.com:

The JavaScript opens the file, positions at a start offset
(initially zero), and reads and uploads up to a fixed length
of the contents, along with a continuation offset (or zero
to indicate the end reached), and the server code re-runs
the script in a loop while this returned offset is non-zero.
Subsequent runs of the script would bypass the data input
and execute the JavaScript directly.

Do you have a Perl question?

Sinan

[john_ramsden]

A. Sinan Unur wrote:

jo**********@sagitta-ps.com wrote in news:1109862480.383312.250480
@z14g2000cwz.googlegroups.com:

I have to prepare an Apache/PHP based (or perl if easier)
[...]

The JavaScript opens the file, positions at a start offset
(initially zero), and reads and uploads up to a fixed length
of the contents, along with a continuation offset (or zero
to indicate the end reached), and the server code re-runs
the script in a loop while this returned offset is non-zero.
Subsequent runs of the script would bypass the data input
and execute the JavaScript directly.

Do you have a Perl question?

Read the first line, numbnuts: "(or perl if easier)"

[Anno Siegel]

<jo**********@sagitta-ps.com> wrote in comp.lang.perl.misc:

A. Sinan Unur wrote:

jo**********@sagitta-ps.com wrote in news:1109862480.383312.250480
@z14g2000cwz.googlegroups.com: Do you have a Perl question?

Read the first line, numbnuts: "(or perl if easier)"

So long then.

Anno

[Colin McKinnon]

Please don't cross post.

jo**********@sagitta-ps.com wrote:

I have to prepare an Apache/PHP based (or perl if easier) web
page that allows a user to upload very large files, possibly
over 10 Gbytes and maybe up to 100 Gbytes.

Sounds like a recipe for disaster. HTTP wasn't designed with this kind of
thing in mind. You're going to get all sorts of problems. You might be able
to get to work with webDAV, but you'll need to learn a lot more about how
it works.

In principle there's a simple solution whereby the user runs <snip> The JavaScript opens the file, positions at a start offset
(initially zero), and reads and uploads up to a fixed length
of the contents, along with a continuation offset (or zero
to indicate the end reached), and the server code re-runs
the script in a loop while this returned offset is non-zero.
....right, but Javascript doesn't do this. (well, some implementations will,
but you have to jump through hoops to get it to work). If you must run it
over HTTP (and you'll have to split it up), you'll need something other
than a browser at the client end.
(It's a shame Apache doesn't seem to have an embedded FTP
server in some way. I guess FTP would be the obvious choice,
but the client doesn't wish to have a separate FTP server
running.)

NO NO NO.

FTP is an archaic and insecure protocol which is a %^&* to run across NAT
and has a history of broken implementations. I would have thought that the
'Global Leader in Independent Storage Services' (www.glasshouse.com) would
have a bit more nous about network protocols.

C.

[J. Gleixner]

jo**********@sagitta-ps.com wrote:

I have to prepare an Apache/PHP based (or perl if easier) web
page that allows a user to upload very large files, possibly
over 10 Gbytes and maybe up to 100 Gbytes. [...] Anyway, the question I have is whether anyone has written
anything similar, ideally with a progress bar? [...] Any suggestions gratefully received.

Out of the 3 newsgroups you chose, all 3 are off topic. This isn't a
"programming" question.

The wheel is to build a Web page, in HTML, and put it on your Web site
and let the browser do it for you. That's it, problem solved.

Just have a link to the file and the client clicks on the link or does a
right click...Save Link as. That's it. You could use PHP/CGI to build
the list of files, but there's probably no need to call a script to
serve the file, just use HTTP. You don't need to do anything to Apache
or your PHP installation.

Forget about a progress bar. Most (All?) browsers create their own
download manager and they likely have their own progress display. If you
really want to make your own, then an Applet is your best bet.

[Mr. Wilson]

> jo**********@sagitta-ps.com wrote:

...that allows a user to **upload** very large files...

[idiotic non-answer snipped]

Didn't watch enough Sesame Street to learn the difference between up and
down, eh?

Heh, what a maroon.

--Mr. Wilson

[Csaba2000]

This is an interesting question, but I must be missing something:
If I understand you right, you're trying to get a large file from
the client (browser) side to your (the server) side. And you
want the user's browser (possibly with the aid of code you
provide) to cooperate in this endeavor so you can get past
the usual (including timeout) limitations.

But javascript (really the only thing available generically) on
the client side does not have access to the user's disk. So
I am curious to know what I am not understanding here.
If browsers are not involved, then I would turn to cURL
within command line PHP (on the client side) or a similar
variant and have the PHP on the Apache end reassemble
the file. Haven't done it, but it seems straightforward.
Probably would try with a command line version of cURL.

As for progress bar, on the client side, you can use COM
methods on recent Windows systems to put up an IE and
keep changing the IE page as you progress along (see the
php.net/com area for starter examples).

Csaba Gabor from Vienna

<jo**********@sagitta-ps.com> wrote in message
news:11**********************@z14g2000cwz.googlegr oups.com...

I have to prepare an Apache/PHP based (or perl if easier) web
page that allows a user to upload very large files, possibly
over 10 Gbytes and maybe up to 100 Gbytes.

I'm aware that file upload limits can be increased by config
changes; but I'd rather minimize the need for changes like
this, both for admin reasons and to avoid possible (?) memory problems
or even bugs.

In principle there's a simple solution whereby the user runs
an initial page, in which they input the file name, and the
Submit button then runs JavaScript in the page.

The JavaScript opens the file, positions at a start offset
(initially zero), and reads and uploads up to a fixed length
of the contents, along with a continuation offset (or zero
to indicate the end reached), and the server code re-runs
the script in a loop while this returned offset is non-zero.
Subsequent runs of the script would bypass the data input
and execute the JavaScript directly.

Anyway, the question I have is whether anyone has written
anything similar, ideally with a progress bar?

The only problem I anticipate is possible delays, and even
timeouts, while the code (client or server) positions to
the next offset to read or write (respectively) the next
chunk of a very large, e.g. 50 GByte, file.

Any suggestions gratefully received.

(It's a shame Apache doesn't seem to have an embedded FTP
server in some way. I guess FTP would be the obvious choice,
but the client doesn't wish to have a separate FTP server
running.)
Cheers

John R Ramsden (jramsden at glassouse.com)

[juglesh]

<jo**********@sagitta-ps.com> wrote in message
news:11**********************@z14g2000cwz.googlegr oups.com...

I have to prepare an Apache/PHP based (or perl if easier) web
page that allows a user to upload very large files, possibly
over 10 Gbytes and maybe up to 100 Gbytes.

maybe this
http://jupload.biz/

havent tried it for large files (working great so far with smallish images),
but the website says no limit...

[Chung Leong]

"Colin McKinnon" <co**************@andthis.mms3.com> wrote in message
news:d0*******************@news.demon.co.uk...

Please don't cross post.

jo**********@sagitta-ps.com wrote:

I have to prepare an Apache/PHP based (or perl if easier) web
page that allows a user to upload very large files, possibly
over 10 Gbytes and maybe up to 100 Gbytes.

Sounds like a recipe for disaster. HTTP wasn't designed with this kind of
thing in mind. You're going to get all sorts of problems. You might be

able to get to work with webDAV, but you'll need to learn a lot more about how
it works.

Few of today's softwares can handle a file of that size, in fact, as 10G
would overflow an unsigned long.

[john_ramsden]

Mr. Wilson wrote:

jo**********@sagitta-ps.com wrote:

...that allows a user to **upload** very large files...

[idiotic non-answer snipped]

Didn't watch enough Sesame Street to learn the difference
between up and down, eh?

Heh, what a maroon.

--Mr. Wilson

My post was about copying files from a client to a server,
as should have been clear to anyone with a 3-digit IQ who
skimmed it in more than a second.

I bow to your superior knowledge of kids' TV. But Sesame
Street must use a different definition of "upload" to the
one in http://www.vivtek.com/rfc1867.html.

[nospam]

In: <11**********************@z14g2000cwz.googlegroups .com>, jo**********@sagitta-ps.com wrote:

I have to prepare an Apache/PHP based (or perl if easier) web
page that allows a user to upload very large files, possibly
over 10 Gbytes and maybe up to 100 Gbytes.

100G is, well.. large. How large? REALLY large.

Perl on the server side might be your best bet. You can poke around
CGI.pm for the upload stuff (it too has a max size setting, but
you can change it w/out changing your host configuration)

I'd start out with perl in this case, simply because it allows
direct access to the upload data _while_ the upload is taking
place.

Unless you have a v.fast connection, you will still run into
timeouts though.

Your problem is going to be 90% client side. I think you may have
the correct idea as far as chunking it up, but the wrong language.

Java web start or something similiar (which can access client files)
is going to be better. Note that I said "java web start" I didn't
say embedded applet. Two different things. (Applet might be able
to do it, but I don't think I'd try it) either way, a dialog will
be popped up asking your client permission to access the file(s).

Are you sure you have to use HTTP for this? rsync, samba, sftp
will probably be much better tools.

Jamie
--
http://www.geniegate.com Custom web programming
gu******@lnubb.pbz (rot13) User Management Solutions