Using AJAX to do inline, inpage file uploads

[JMB]

Hello,

I was wondering if anyone knew of any projects extending the inline upload
progress bar to utilize an inpage image uploader with bar, without having to
refresh or go to a seperate page, nor opening a second box for display of
the progress bar.

I had been using the MegaUpload that was adapted from Raditha's script at
http://www.raditha.com/upload.php . The MegaUpload script I have been using
takes the progress bar inpage, instead of opening a new page/box for the
display of the bar.

Anyhow, here is what I have been wondering. I have projects with large
amounts of 'application' usage and Web2.0 features, I guess you can say.
This includes dynamic form submissions, inline graphical interfaces,
realtime chat, realtime drawing/whiteboarding, etc. The thing I have not
been able to nail is inline progress bar usage with uploaded images/files.
Does anyone know about any projects currently working on this very thing
that I could contribute to? Or, see some demo code to have a better idea.

I am guessing my request to the server should be a put, but then I would
need what from there modifed to handle this? As PHP sux for it, and the
closest I have gotten is using a modified verision of OB_ functions. The
Perl solution seems more feasible, but I would not know where to begin using
the async nature of AJAX, I think sending the file via PUT method, and the
form variables(if any), submitted during this display of the bar from a
seperate thread to the server for storage.

I know the uploader is PHP/Perl, but with it being an AJAX-savy script,
posting here was a starting point.

Thanks in advance for direction to any/all projects, as well as suggestions,
help and comments.

[Sean Berry]

"JMB" <XX****@XXXX.XXX> wrote in message
news:i1%tf.33841$az4.26548@trndny03...

Hello,

I was wondering if anyone knew of any projects extending the inline upload
progress bar to utilize an inpage image uploader with bar, without having
to refresh or go to a seperate page, nor opening a second box for display
of the progress bar.

I had been using the MegaUpload that was adapted from Raditha's script at
http://www.raditha.com/upload.php . The MegaUpload script I have been
using takes the progress bar inpage, instead of opening a new page/box for
the display of the bar.

Anyhow, here is what I have been wondering. I have projects with large
amounts of 'application' usage and Web2.0 features, I guess you can say.
This includes dynamic form submissions, inline graphical interfaces,
realtime chat, realtime drawing/whiteboarding, etc. The thing I have not
been able to nail is inline progress bar usage with uploaded images/files.
Does anyone know about any projects currently working on this very thing
that I could contribute to? Or, see some demo code to have a better idea.

I am guessing my request to the server should be a put, but then I would
need what from there modifed to handle this? As PHP sux for it, and the
closest I have gotten is using a modified verision of OB_ functions. The
Perl solution seems more feasible, but I would not know where to begin
using the async nature of AJAX, I think sending the file via PUT method,
and the form variables(if any), submitted during this display of the bar
from a seperate thread to the server for storage.

I know the uploader is PHP/Perl, but with it being an AJAX-savy script,
posting here was a starting point.

Thanks in advance for direction to any/all projects, as well as
suggestions, help and comments.

I just did something like this a few months ago and would be happy to share
the code. I have it working in both PERL and Python. The PERL is a
modified version of the MegaUpload project and the Python version is my own.
In either case I can send you the source for it. Do you also want the
javascript? That part is pretty straight forward.

Let me know via email and I will pass it along.

[VK]

JMB wrote:

I have projects with large
amounts of 'application' usage and Web2.0 features, I guess you can say.
This includes dynamic form submissions, inline graphical interfaces,
realtime chat, realtime drawing/whiteboarding, etc. The thing I have not
been able to nail is inline progress bar usage with uploaded images/files.
Does anyone know about any projects currently working on this very thing
that I could contribute to? Or, see some demo code to have a better idea.

I am guessing my request to the server should be a put, but then I would
need what from there modifed to handle this? As PHP sux for it, and the
closest I have gotten is using a modified verision of OB_ functions. The
Perl solution seems more feasible, but I would not know where to begin using
the async nature of AJAX, I think sending the file via PUT method, and the
form variables(if any), submitted during this display of the bar from a
seperate thread to the server for storage.

I know the uploader is PHP/Perl, but with it being an AJAX-savy script,
posting here was a starting point.

The question of upload bar is very old (since Netscape 3.0 at least).
It is one of Perpetuum Mobile's of the Web: it cannot be solved in the
given physical conditions but still many people are trying to invent it
instead of reading physics books :-)

The limitation is build into the HTTP upload model. Unlike FTP, HTTP
upload is a start/stop event w/o intermediary states: you have no idea
about uploading data until submission starts and you still have no idea
of the actual data to handle until the upload is done.
The second crutial limitation of HTTP upload is that the data is sent
in BASE64 encoding. It means that the actual submission size is *always
bigger* than the summary of file sizes to upload. The actual growth
depends heavily on the binary structure of files to upload. Roughly it
is no less than +20% and up to +100%. This is why all endless attempts
to make progress bar based on file(s) size + bandwidth speed failed so
miserably.

So in order to have a reliable progress bar for HTTP upload you have to
have *on client-side*:
1) A mechanics to know the exact upload size of files after BASE64
encoded.
2) Per packed-sent notification from HTTP upload module.

Neither of this is possible on the existing browsers. You still *can*
write your own ActiveX / plugin to take over the whole HTTP upload
functionality (or to have an FTP uploader right on the page). Nothing
you can do by just using server-side + slient-side scripting in any
shall perform combination.

P.S. Thermodynamic laws never stopped people from trying to invent a
perpetuum mobile. I'm affraid it is the case with the progress bar
neither. :-)

P.P.S. No current browser supports PUT method except W3C Amaya. Just to
save you from of one of dead branches.

[Duncan Booth]

VK wrote:

The second crutial limitation of HTTP upload is that the data is sent
in BASE64 encoding. It means that the actual submission size is *always
bigger* than the summary of file sizes to upload. The actual growth
depends heavily on the binary structure of files to upload. Roughly it
is no less than +20% and up to +100%. This is why all endless attempts
to make progress bar based on file(s) size + bandwidth speed failed so
miserably.

This is rubbish.

Base64 encoding always increases the size of the data by one third. It does
not depend on the 'binary structure' of the data: quite simply 3 input
bytes are encoded as 4 printable characters.

[Jim Ley]

On 2 Jan 2006 03:01:27 -0800, "VK" <sc**********@yahoo.com> wrote:

So in order to have a reliable progress bar for HTTP upload you have to
have *on client-side*:
1) A mechanics to know the exact upload size of files after BASE64
encoded.
A progress bar could reasonably be done on non-base64 encoded sizes,
the only assumption would then be that the base64 encoding is
approximately linear - that would be a sufficiently accurate approach
for what is already a very rough ready visual indicator.
2) Per packed-sent notification from HTTP upload module.

Or a callback from the server announcing "I've got N bytes", indeed
this may well be more accurate as it counts bytes recieved not sent
(some may have gone missing...)

So all that is really missing from the entire system is the total
upload size. Certainly you need a seperate communication channel from
the server to the client - a script available socket or
xmlhttprequest, or an open iframe - to send the "I've got N bytes"
messages.

It's a relatively simple job in fact to have an "uploaded 10k, 20k,30k
etc." message, indeed I implemented it one about 2001.

Jim.

[VK]

Duncan Booth wrote:

VK wrote:

The second crutial limitation of HTTP upload is that the data is sent
in BASE64 encoding. It means that the actual submission size is *always
bigger* than the summary of file sizes to upload. The actual growth
depends heavily on the binary structure of files to upload. Roughly it
is no less than +20% and up to +100%. This is why all endless attempts
to make progress bar based on file(s) size + bandwidth speed failed so
miserably.

This is rubbish.

Base64 encoding always increases the size of the data by one third. It does
not depend on the 'binary structure' of the data: quite simply 3 input
bytes are encoded as 4 printable characters.

My bad! I was thinking about zipping instead of base64-encoding. The
resulting file size is never less then source size + 30%
At the same time for MIME base64 it is never strictly equal to
binarySize+binarySize/100*30. Because of additional service data is
approx 33%.
On 5-10-more files package this "approx" transforms a strict
calculation into guessing game.
Also there is such things as i) bandwidth fluctuations and ii) thread
priority management where you have no control whatsoever. The latter is
caused by the fact that a script is just another process within the
host process (browser). Periodically system will decide that it has
something more important to do and your script can wait till another
system tick (or two). On a long run it adds up significally.
The only reliable counter would be from a flag "another packet sent"
but you have none of such.

Still no one is forbidden to invent perpetuum mobiles and I expect to
see a lot of "perfectly working models" linked soon.

[Sean Berry]

"JMB" <XX****@XXXX.XXX> wrote in message
news:i1%tf.33841$az4.26548@trndny03...

Hello,

I was wondering if anyone knew of any projects extending the inline upload
progress bar to utilize an inpage image uploader with bar, without having
to refresh or go to a seperate page, nor opening a second box for display
of the progress bar.

I had been using the MegaUpload that was adapted from Raditha's script at
http://www.raditha.com/upload.php . The MegaUpload script I have been
using takes the progress bar inpage, instead of opening a new page/box for
the display of the bar.

Anyhow, here is what I have been wondering. I have projects with large
amounts of 'application' usage and Web2.0 features, I guess you can say.
This includes dynamic form submissions, inline graphical interfaces,
realtime chat, realtime drawing/whiteboarding, etc. The thing I have not
been able to nail is inline progress bar usage with uploaded images/files.
Does anyone know about any projects currently working on this very thing
that I could contribute to? Or, see some demo code to have a better idea.

I am guessing my request to the server should be a put, but then I would
need what from there modifed to handle this? As PHP sux for it, and the
closest I have gotten is using a modified verision of OB_ functions. The
Perl solution seems more feasible, but I would not know where to begin
using the async nature of AJAX, I think sending the file via PUT method,
and the form variables(if any), submitted during this display of the bar
from a seperate thread to the server for storage.

I know the uploader is PHP/Perl, but with it being an AJAX-savy script,
posting here was a starting point.

Thanks in advance for direction to any/all projects, as well as
suggestions, help and comments.

JMB,

The posts above are quite right about Base64 encoding and size fluctuations.
However, in my case I had clients who wanted to have some sort of progress
indicator. When someone has to upload a 50 magabyte file for instance they
want to know that the upload is still in progress after 1 minute.

My program works by using the CONTENT_LENGTH of the request as the total
size. Then, using AJAX, it retrieves the current size of the file on the
server and creates a progress bar based on the that.

Is it 100% accurate? NO!

Does it satisfy the customer by providing them with exactly what they
wanted? YES!

And does it actually work with a fair amount of accuracy? YES!

[JMB]

Hello, and thank you for all of the replies...

However, in the applications I am currently using, I am doing an inpage
upload progress that is about 90% accurate with time and rate of tranfer, so
this is already covered. In fact, a 100k image to a 5mb to(we had reason
to) a 1.8GB file, all displayed near-correct size, time-to-fin and percent
done in all speed cases tried. You just needed a small modifier on readout,
and an incremental for timeFromStart of the upload.

What I am having problem with, is using the XMLHttpRequest object and
asyncronous actions to allow for a file to be placed onto the server, have a
filename callback/src designation to be placed back onto that page(I know
how to do this, so, once again, this is not a problem), but all the while,
maintain the ability of the app to display a progress bar while uploading is
going on in the background(my bane with letting the progress bar do it's
thing).

I know that it can be done, but I hit a wall, and am not having any luck
going around it...seems like my programming attempts on this are trying to
push me through it!:)

Thanks again, and I look forward to any more replies that the group may be
willing to send,
JMB

"VK" <sc**********@yahoo.com> wrote in message
news:11**********************@z14g2000cwz.googlegr oups.com...

Duncan Booth wrote:

VK wrote:

> The second crutial limitation of HTTP upload is that the data is sent
> in BASE64 encoding. It means that the actual submission size is *always
> bigger* than the summary of file sizes to upload. The actual growth
> depends heavily on the binary structure of files to upload. Roughly it
> is no less than +20% and up to +100%. This is why all endless attempts
> to make progress bar based on file(s) size + bandwidth speed failed so
> miserably.
>

This is rubbish.

Base64 encoding always increases the size of the data by one third. It
does
not depend on the 'binary structure' of the data: quite simply 3 input
bytes are encoded as 4 printable characters.

My bad! I was thinking about zipping instead of base64-encoding. The
resulting file size is never less then source size + 30%
At the same time for MIME base64 it is never strictly equal to
binarySize+binarySize/100*30. Because of additional service data is
approx 33%.
On 5-10-more files package this "approx" transforms a strict
calculation into guessing game.
Also there is such things as i) bandwidth fluctuations and ii) thread
priority management where you have no control whatsoever. The latter is
caused by the fact that a script is just another process within the
host process (browser). Periodically system will decide that it has
something more important to do and your script can wait till another
system tick (or two). On a long run it adds up significally.
The only reliable counter would be from a flag "another packet sent"
but you have none of such.

Still no one is forbidden to invent perpetuum mobiles and I expect to
see a lot of "perfectly working models" linked soon.

[VK]

Sean Berry wrote:

The posts above are quite right about Base64 encoding and size fluctuations.
However, in my case I had clients who wanted to have some sort of progress
indicator. When someone has to upload a 50 magabyte file for instance they
want to know that the upload is still in progress after 1 minute.

50Mb over HTTP ?! That's at least 15.5MB of extra base64 trash on each
upload! How about environment protection (not talking about wasted
time) ? ;-)

There are plenty of FTP modules to plug in into your page for such huge
uploads.

And on relatively small uploads the "oopsy miscalculation" can easily
be up to 50% of the predicted time.

But it is indeed possible (and useful) to have an animated "currently
uploading" sign on the page. With a bit of trickery it can be done.

[Sean Berry]

"VK" <sc**********@yahoo.com> wrote in message
news:11**********************@g49g2000cwa.googlegr oups.com...

Sean Berry wrote:

The posts above are quite right about Base64 encoding and size
fluctuations.
However, in my case I had clients who wanted to have some sort of
progress
indicator. When someone has to upload a 50 magabyte file for instance
they
want to know that the upload is still in progress after 1 minute.
50Mb over HTTP ?! That's at least 15.5MB of extra base64 trash on each
upload! How about environment protection (not talking about wasted
time) ? ;-)

You are preaching to the choir with this argument.

The problem is end-users. Our clients for the most part do not deal with
(nor are they themselves) the smartest people on the planet. Heck, most of
them do not have what I would consider the basic aptitude needed for
survival... but they prove me wrong every day.
There are plenty of FTP modules to plug in into your page for such huge
uploads.

In my particular case the upload area is a bit more complex and has users
with different access and abilities. It also needed to have added
functionalites like email notifications for new uploads, description
requirements and a whole bunch of other crap. But, trying to teach my
clients to teach their clients how to successfully upload a file via FTP
wouldn't have been my first choice anyway. And on relatively small uploads the "oopsy miscalculation" can easily
be up to 50% of the predicted time.

But it is indeed possible (and useful) to have an animated "currently
uploading" sign on the page. With a bit of trickery it can be done.

As far as the progress bar, "people" like to have something really fun and
exciting like a progress bar that tells them that something is going on. At
first, I used an animation once the upload started by changing a static
image to an animated gif or something. However, I found that most "people"
didn't understand that when they are uploading a 50 megabyte file it may
take more than ten seconds and became frustrated thinking it "locked up
again". Anyway, what really matters - in my case anyway - is that the end
product does exactly what they, the brilliant people I get to deal with,
want it to do. It gives a visual indication through a recognizable and very
familiar way of how much progress has been made.

SMB

[VK]

Sean Berry wrote:

You are preaching to the choir with this argument.

The problem is end-users. Our clients for the most part do not deal with
(nor are they themselves) the smartest people on the planet. Heck, most of
them do not have what I would consider the basic aptitude needed for
survival... but they prove me wrong every day.

The problem of "sub-zero" users is well known to me either :-)
<http://groups.google.com/group/comp.lang.javascript/browse_frm/thread/355b3f86ed2f11fa/17aa578042f08022>

But it is a wrong thinking IMHO to correlate "primitive user" with
"primitive solution". Mostly it's just opposite: a very "primitive
user" requires a very sophisticated solution - but with a very simple
interface covering all internal complexity. In the above linked case I
had to end up by designing a whole new set of pseudo-inputs paired with
<input type="hidden"... for submission. On a normal run I would try to
avoid such pervertion. :-)

HTTP binary exchange is not a tool to replace FTP. It is merely a
convenience addon to a form for relatively small attachments. Besides
the file growth because of base64 format, it has a bounch of other
implications like buffer mechanics. See for example
<http://support.microsoft.com/default.aspx?scid=kb;en-us;329781> I
don't think that we can talk about usability improvement if
upload/download takes 1hr instead of needed 10 min - but user gets an
animation for the entire hour :-)

There are FTP plugins like say Internet Transfer control from
Microsoft. One can use them to have FTP and emulate as close as needed
(if needed) the usual <input type="file"... interface.

But everyone is the king of his own solution - so it's all just
IMHighlyHO.

[PJ]

I have been using a component from http://www.abcupload.com/ for years.
After creating a unique id that is appended to the action attribute on the
form that does the post for the upload, I can then use this id in get
requests to get informatation on the progress of the upload. I perform
these get requests with XHR requests on the client side and handle the
requests on the server side, where I send the data on the progress of the
upload back in xml. The javascript updates an html progress bar, and byte
count / total bytes of the upload.

This "approx" "working model" handles 80 Gig ( yes, GIG ) uploads as
accurately as my eye is able to discern.

I love people that have looked into a solution, failed to find one, and thus
determine their problem is "impossible" to solve.
"VK" <sc**********@yahoo.com> wrote in message
news:11**********************@z14g2000cwz.googlegr oups.com...

Duncan Booth wrote:

VK wrote:

The second crutial limitation of HTTP upload is that the data is sent
in BASE64 encoding. It means that the actual submission size is *always bigger* than the summary of file sizes to upload. The actual growth
depends heavily on the binary structure of files to upload. Roughly it
is no less than +20% and up to +100%. This is why all endless attempts
to make progress bar based on file(s) size + bandwidth speed failed so
miserably.

This is rubbish.

Base64 encoding always increases the size of the data by one third. It does not depend on the 'binary structure' of the data: quite simply 3 input
bytes are encoded as 4 printable characters.

My bad! I was thinking about zipping instead of base64-encoding. The
resulting file size is never less then source size + 30%
At the same time for MIME base64 it is never strictly equal to
binarySize+binarySize/100*30. Because of additional service data is
approx 33%.
On 5-10-more files package this "approx" transforms a strict
calculation into guessing game.
Also there is such things as i) bandwidth fluctuations and ii) thread
priority management where you have no control whatsoever. The latter is
caused by the fact that a script is just another process within the
host process (browser). Periodically system will decide that it has
something more important to do and your script can wait till another
system tick (or two). On a long run it adds up significally.
The only reliable counter would be from a flag "another packet sent"
but you have none of such.

Still no one is forbidden to invent perpetuum mobiles and I expect to
see a lot of "perfectly working models" linked soon.

[VK]

PJ wrote:

I have been using a component from http://www.abcupload.com/ for years.
After creating a unique id that is appended to the action attribute on the
form that does the post for the upload, I can then use this id in get
requests to get informatation on the progress of the upload. I perform
these get requests with XHR requests on the client side and handle the
requests on the server side, where I send the data on the progress of the
upload back in xml. The javascript updates an html progress bar, and byte
count / total bytes of the upload.

This "approx" "working model" handles 80 Gig ( yes, GIG ) uploads as
accurately as my eye is able to discern.

I love people that have looked into a solution, failed to find one, and thus
determine their problem is "impossible" to solve.

That is a subject substitution. I did not say that there were not ways
to have a status report for upload/download operations in browser. You
see such report in action every time you're downloading something from
the Web.

The problem is not solvable by using only default browser tools (HTML
elements and scripting) and server responces. If there is a need to
have status report at any price, you can do it by extending
browser/server core functionality. And you can do it without privacy
exposure like in the mentioned case. There are plenty of FTP Java
applets and ActiveX controls to embed onto your page.

P.S. 80 Gigs over FTP I hope? (I did not read abcupload.com proposal in
details).
Otherwise it would be an athem to uneffectiveness.