473,394 Members | 1,267 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,394 software developers and data experts.

Iterated mega file upload - invented wheel wanted


I have to prepare an Apache/PHP based (or perl if easier) web
page that allows a user to upload very large files, possibly
over 10 Gbytes and maybe up to 100 Gbytes.

I'm aware that file upload limits can be increased by config
changes; but I'd rather minimize the need for changes like
this, both for admin reasons and to avoid possible (?) memory problems
or even bugs.

In principle there's a simple solution whereby the user runs
an initial page, in which they input the file name, and the
Submit button then runs JavaScript in the page.

The JavaScript opens the file, positions at a start offset
(initially zero), and reads and uploads up to a fixed length
of the contents, along with a continuation offset (or zero
to indicate the end reached), and the server code re-runs
the script in a loop while this returned offset is non-zero.
Subsequent runs of the script would bypass the data input
and execute the JavaScript directly.

Anyway, the question I have is whether anyone has written
anything similar, ideally with a progress bar?

The only problem I anticipate is possible delays, and even
timeouts, while the code (client or server) positions to
the next offset to read or write (respectively) the next
chunk of a very large, e.g. 50 GByte, file.

Any suggestions gratefully received.

(It's a shame Apache doesn't seem to have an embedded FTP
server in some way. I guess FTP would be the obvious choice,
but the client doesn't wish to have a separate FTP server
running.)
Cheers

John R Ramsden (jramsden at glassouse.com)

Jul 23 '05 #1
10 1494
jo**********@sagitta-ps.com wrote in news:1109862480.383312.250480
@z14g2000cwz.googlegroups.com:
The JavaScript opens the file, positions at a start offset
(initially zero), and reads and uploads up to a fixed length
of the contents, along with a continuation offset (or zero
to indicate the end reached), and the server code re-runs
the script in a loop while this returned offset is non-zero.
Subsequent runs of the script would bypass the data input
and execute the JavaScript directly.


Do you have a Perl question?

Sinan
Jul 23 '05 #2

A. Sinan Unur wrote:

jo**********@sagitta-ps.com wrote in news:1109862480.383312.250480
@z14g2000cwz.googlegroups.com:

I have to prepare an Apache/PHP based (or perl if easier)
[...]

The JavaScript opens the file, positions at a start offset
(initially zero), and reads and uploads up to a fixed length
of the contents, along with a continuation offset (or zero
to indicate the end reached), and the server code re-runs
the script in a loop while this returned offset is non-zero.
Subsequent runs of the script would bypass the data input
and execute the JavaScript directly.


Do you have a Perl question?


Read the first line, numbnuts: "(or perl if easier)"

Jul 23 '05 #3
<jo**********@sagitta-ps.com> wrote in comp.lang.perl.misc:

A. Sinan Unur wrote:

jo**********@sagitta-ps.com wrote in news:1109862480.383312.250480
@z14g2000cwz.googlegroups.com: Do you have a Perl question?


Read the first line, numbnuts: "(or perl if easier)"


So long then.

Anno
Jul 23 '05 #4
On Thu, 03 Mar 2005 07:08:00 -0800, john_ramsden wrote:
(It's a shame Apache doesn't seem to have an embedded FTP server in some
way.


Apache 2.x does, I think. At least the MPM architecture allows for it, I
don't know if one has been written. As has been pointed out, FTP
belongs to the age of a nicer internet, but httpd2 will let you do it if
you really want to ...

R
Jul 23 '05 #5
jo**********@sagitta-ps.com wrote:
I have to prepare an Apache/PHP based (or perl if easier) web
page that allows a user to upload very large files, possibly
over 10 Gbytes and maybe up to 100 Gbytes. [...] Anyway, the question I have is whether anyone has written
anything similar, ideally with a progress bar? [...] Any suggestions gratefully received.


Out of the 3 newsgroups you chose, all 3 are off topic. This isn't a
"programming" question.

The wheel is to build a Web page, in HTML, and put it on your Web site
and let the browser do it for you. That's it, problem solved.

Just have a link to the file and the client clicks on the link or does a
right click...Save Link as. That's it. You could use PHP/CGI to build
the list of files, but there's probably no need to call a script to
serve the file, just use HTTP. You don't need to do anything to Apache
or your PHP installation.

Forget about a progress bar. Most (All?) browsers create their own
download manager and they likely have their own progress display. If you
really want to make your own, then an Applet is your best bet.
Jul 23 '05 #6
> jo**********@sagitta-ps.com wrote:
...that allows a user to **upload** very large files...


[idiotic non-answer snipped]

Didn't watch enough Sesame Street to learn the difference between up and
down, eh?

Heh, what a maroon.

--Mr. Wilson
Jul 23 '05 #7
This is an interesting question, but I must be missing something:
If I understand you right, you're trying to get a large file from
the client (browser) side to your (the server) side. And you
want the user's browser (possibly with the aid of code you
provide) to cooperate in this endeavor so you can get past
the usual (including timeout) limitations.

But javascript (really the only thing available generically) on
the client side does not have access to the user's disk. So
I am curious to know what I am not understanding here.
If browsers are not involved, then I would turn to cURL
within command line PHP (on the client side) or a similar
variant and have the PHP on the Apache end reassemble
the file. Haven't done it, but it seems straightforward.
Probably would try with a command line version of cURL.

As for progress bar, on the client side, you can use COM
methods on recent Windows systems to put up an IE and
keep changing the IE page as you progress along (see the
php.net/com area for starter examples).

Csaba Gabor from Vienna

<jo**********@sagitta-ps.com> wrote in message
news:11**********************@z14g2000cwz.googlegr oups.com...

I have to prepare an Apache/PHP based (or perl if easier) web
page that allows a user to upload very large files, possibly
over 10 Gbytes and maybe up to 100 Gbytes.

I'm aware that file upload limits can be increased by config
changes; but I'd rather minimize the need for changes like
this, both for admin reasons and to avoid possible (?) memory problems
or even bugs.

In principle there's a simple solution whereby the user runs
an initial page, in which they input the file name, and the
Submit button then runs JavaScript in the page.

The JavaScript opens the file, positions at a start offset
(initially zero), and reads and uploads up to a fixed length
of the contents, along with a continuation offset (or zero
to indicate the end reached), and the server code re-runs
the script in a loop while this returned offset is non-zero.
Subsequent runs of the script would bypass the data input
and execute the JavaScript directly.

Anyway, the question I have is whether anyone has written
anything similar, ideally with a progress bar?

The only problem I anticipate is possible delays, and even
timeouts, while the code (client or server) positions to
the next offset to read or write (respectively) the next
chunk of a very large, e.g. 50 GByte, file.

Any suggestions gratefully received.

(It's a shame Apache doesn't seem to have an embedded FTP
server in some way. I guess FTP would be the obvious choice,
but the client doesn't wish to have a separate FTP server
running.)
Cheers

John R Ramsden (jramsden at glassouse.com)

Jul 23 '05 #8

<jo**********@sagitta-ps.com> wrote in message
news:11**********************@z14g2000cwz.googlegr oups.com...

I have to prepare an Apache/PHP based (or perl if easier) web
page that allows a user to upload very large files, possibly
over 10 Gbytes and maybe up to 100 Gbytes.


maybe this
http://jupload.biz/

havent tried it for large files (working great so far with smallish images),
but the website says no limit...

Jul 23 '05 #9

Mr. Wilson wrote:
jo**********@sagitta-ps.com wrote:
...that allows a user to **upload** very large files...


[idiotic non-answer snipped]

Didn't watch enough Sesame Street to learn the difference
between up and down, eh?

Heh, what a maroon.

--Mr. Wilson


My post was about copying files from a client to a server,
as should have been clear to anyone with a 3-digit IQ who
skimmed it in more than a second.

I bow to your superior knowledge of kids' TV. But Sesame
Street must use a different definition of "upload" to the
one in http://www.vivtek.com/rfc1867.html.

Jul 23 '05 #10
In: <11**********************@z14g2000cwz.googlegroups .com>, jo**********@sagitta-ps.com wrote:

I have to prepare an Apache/PHP based (or perl if easier) web
page that allows a user to upload very large files, possibly
over 10 Gbytes and maybe up to 100 Gbytes.


100G is, well.. large. How large? REALLY large.

Perl on the server side might be your best bet. You can poke around
CGI.pm for the upload stuff (it too has a max size setting, but
you can change it w/out changing your host configuration)

I'd start out with perl in this case, simply because it allows
direct access to the upload data _while_ the upload is taking
place.

Unless you have a v.fast connection, you will still run into
timeouts though.

Your problem is going to be 90% client side. I think you may have
the correct idea as far as chunking it up, but the wrong language.

Java web start or something similiar (which can access client files)
is going to be better. Note that I said "java web start" I didn't
say embedded applet. Two different things. (Applet might be able
to do it, but I don't think I'd try it) either way, a dialog will
be popped up asking your client permission to access the file(s).

Are you sure you have to use HTTP for this? rsync, samba, sftp
will probably be much better tools.

Jamie
--
http://www.geniegate.com Custom web programming
gu******@lnubb.pbz (rot13) User Management Solutions
Jul 23 '05 #11

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: matt | last post by:
I have compiled some code, some written by me, some compiled from various sources online, and basically i've got a very simple flat file photo gallery. An upload form, to upload the photos and give...
11
by: john_ramsden | last post by:
I have to prepare an Apache/PHP based (or perl if easier) web page that allows a user to upload very large files, possibly over 10 Gbytes and maybe up to 100 Gbytes. I'm aware that file upload...
7
by: Daniel Vallstrom | last post by:
I am having trouble with floating point addition because of the limited accuracy of floating types. Consider e.g. x0 += f(n) where x0 and f are of some floating type. Sometimes x0 is much larger...
60
by: Julie | last post by:
What is the *fastest* way in .NET to search large on-disk text files (100+ MB) for a given string. The files are unindexed and unsorted, and for the purposes of my immediate requirements, can't...
4
by: Shawn Mesiatowsky | last post by:
I have an html form that accepts a file for uploading, but I wanted to create a program to automate the upload procedure. I beileive you use the Webresponse class, but I was not sure how to use...
4
by: HNguyen | last post by:
Hi, I have a Web application in ASP.NET. My Application allows the users upload files into the server after checking their user names and passwords. For each transaction, the Web program will...
12
by: JMB | last post by:
Hello, I was wondering if anyone knew of any projects extending the inline upload progress bar to utilize an inpage image uploader with bar, without having to refresh or go to a seperate page,...
3
by: adtvff | last post by:
Hi, Given a large ascii file (delimited or fixed width) with one ID field and dimensions/measures fields, sorted by dimensions, I'd like to "flatten" or "rollup" the file by creating new...
2
by: Bjorn Sagbakken | last post by:
I was looking for a solution, without the full page postback, and all the tip was about a hidden iframe and RJS. The RJS was not known to me before, and still I haven't explored this used with...
1
KevinADC
by: KevinADC | last post by:
Note: You may skip to the end of the article if all you want is the perl code. Introduction Many websites have a form or a link you can use to download a file. You click a form button or click...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.