XMLHTTPRequest streaming data

[googlegroups]

I am playing with the XMLHTTPRequest method to perform client/server
transactions. I have it set up right now so that when readyState is 4,
it takes the XML and processes it. This works great until there is alot
of data. In that case, the user will have to wait for the data to come
back which may take a minute or so.

I don't want the user to have to wait. Is it possible for javascript to
periodically (while still receiving more data) stop and display what it
has received thus far? I guess this would be considered a type of
streaming.

In mozilla/firefox, I have read that I can use readyState 3 to run my
callback function every 4096 bytes. I can then take those 4K, parse
them, and then continue on. However I have also read that IE cannot do
this. Since I need this to work in IE, is there a workaround?

Thanks,
Scott

[Matt Kruse]

go**********@scottsavarese.com wrote:

I don't want the user to have to wait. Is it possible for javascript
to periodically (while still receiving more data) stop and display
what it has received thus far?

Not that I know of.
One suggestion - use JSON instead of XML. Or use your own compact data
structure.

XML is verbose, and you may be able to cut down your transmission time if
you use a data format that is more compact.

--
Matt Kruse
http://www.JavascriptToolbox.com
http://www.AjaxToolbox.com

[Martin Honnen]

go**********@scottsavarese.com wrote:

In mozilla/firefox, I have read that I can use readyState 3 to run my
callback function every 4096 bytes. I can then take those 4K, parse
them, and then continue on. However I have also read that IE cannot do
this. Since I need this to work in IE, is there a workaround?

You should be able to access responseText once readyState gives you the
value 3 for interactive.
From the MSXML docs for XMLHTTP:

(3) INTERACTIVE Some data has been received. You can call the
responseBody and responseText properties to get the current partial
results.

--

Martin Honnen
http://JavaScript.FAQTs.com/

[Martin Honnen]

Martin Honnen wrote:

You should be able to access responseText once readyState gives you the
value 3 for interactive.
From the MSXML docs for XMLHTTP:

(3) INTERACTIVE Some data has been received. You can call the
responseBody and responseText properties to get the current partial
results.

I have done some tests here trying to access responseText when
readyState is signalled as 3 but unfortunately MSXML then always gives
an exception that the data necessary is not yet available so in this
case it looks like the docs are promising more than is possible.

--

Martin Honnen
http://JavaScript.FAQTs.com/

[Matt Kruse]

Martin Honnen wrote:

(3) INTERACTIVE Some data has been received. You can call the
responseBody and responseText properties to get the current partial
results.

However, with XML it needs to be well-formed and parsed before being
available, afaik.
You can get to the responseText, but it won't be XML that can be
manipulated.

--
Matt Kruse
http://www.JavascriptToolbox.com
http://www.AjaxToolbox.com

[googlegroups]

Matt Kruse wrote:

Not that I know of.
One suggestion - use JSON instead of XML. Or use your own compact data
structure.

I'll look into JSON. Right now our XML is pretty compact. The structure
is very little overhead as compared to the data itself. I am more
worried by the shear number of data rows we may return than anything
else.

I was playing around with Mozilla. It doesn't seem to hard to keep
track of how much data I've read so far and use substring to extract
the latest. Then parse the latest information and present it to the
user.

However with IE, I can't use the readyState variable. Plus I tried
using a timer (every second) to read the responseText and IE comes back
with an error. Is there anyway to get the partial response from IE or
do I have to wait until it completes? Is there any hack I can do to IE
to get it to work?

Thanks,
Scott

[Jim Ley]

On 22 Jun 2005 12:40:41 -0700, go**********@scottsavarese.com wrote:

However with IE, I can't use the readyState variable. Plus I tried
using a timer (every second) to read the responseText and IE comes back
with an error. Is there anyway to get the partial response from IE or
do I have to wait until it completes? Is there any hack I can do to IE
to get it to work?

No, just use the IFRAME remote scripting method and JSON, it's much
more reliable and cross-platform for achieving what you want, although
it's the server that has to do the breaking up.

Jim.

[googlegroups]

Martin Honnen wrote:

I have done some tests here trying to access responseText when
readyState is signalled as 3 but unfortunately MSXML then always gives
an exception that the data necessary is not yet available so in this
case it looks like the docs are promising more than is possible.

I get the same thing... That is exactly what I am trying to get around.

Scott

[googlegroups]

Jim Ley wrote:

No, just use the IFRAME remote scripting method and JSON, it's much
more reliable and cross-platform for achieving what you want, although
it's the server that has to do the breaking up.

I really like the idea of JSON. Unfortunately in this case, I need my
protocol to be XML. The code can't be language specific. If I was just
using a web browser to obtain the information from the server, then
JSON can be used, and may even be better, however I also have a perl
client side module, and I have a feeling my users are going to want a
Java one as well. How is JSON with security... I run the returned value
in a eval block. Of the examples I've seen everyone seems to trust that
the data returned is valid and not harmful. Can you craft a JSON reply
that does something malicious?

As for the IFRAME piece, I did some research on it... Is there an
example on line of people doing the type of streaming that I am looking
for? Also, it looks like the IFRAME seems to imply a need to use GET as
opposed to POST (as it updates the history). Is that true?

Scott

[Jim Ley]

On 22 Jun 2005 13:37:20 -0700, go**********@scottsavarese.com wrote:

Jim Ley wrote:

No, just use the IFRAME remote scripting method and JSON, it's much
more reliable and cross-platform for achieving what you want, although
it's the server that has to do the breaking up.

I really like the idea of JSON. Unfortunately in this case, I need my
protocol to be XML.

No, you need to provide an XML protocol, that doesn't mean the one you
provide to your website is XML, a server XML->json is entirely
appropriate solution - especially as IFRAME's (which require JSON to
do this) are the only way you're going to get the incremental approach
in IE.

As for a link, no idea, you just need to write out an HTML document
with

<script type="text/javascript">
.... block 1
</script>

<script type="text/javascript">
.... block 2
</script>

<script type="text/javascript">
.... block 3
</script>

etc.

Jim.

[Bruce Stephens]

go**********@scottsavarese.com writes:

[...]

I really like the idea of JSON. Unfortunately in this case, I need
my protocol to be XML. The code can't be language specific. If I was
just using a web browser to obtain the information from the server,
then JSON can be used, and may even be better, however I also have a
perl client side module, and I have a feeling my users are going to
want a Java one as well.
Why is that a problem? JSON has a simple syntax, which ought to be
easy to parse (into any language which has associative arrays, anyway)
by hand even if you can't find an existing implementation.
How is JSON with security... I run the returned value in a eval
block. Of the examples I've seen everyone seems to trust that the
data returned is valid and not harmful. Can you craft a JSON reply
that does something malicious?

Think of the overall solution: the JSON reply can do anything that the
web browser lets it do. But we can get the browser to do all that
anyway, since the Javascript that's evaling the JSON came from the
server in the first place.

How about splitting up the message, so rather than one big message
that you try and read in a stream, you read a number of messages which
you make well-formed in some way? Otherwise whatever solution you
pick, you're going to end up with the client getting a part of a
message, and it's not clear that you can usefully do anything with
that.

[Christopher J. Hahn]

go**********@scottsavarese.com wrote:

Jim Ley wrote:

No, just use the IFRAME remote scripting method and JSON, it's much
more reliable and cross-platform for achieving what you want, although
it's the server that has to do the breaking up.

I really like the idea of JSON. Unfortunately in this case, I need my
protocol to be XML. The code can't be language specific. If I was just
using a web browser to obtain the information from the server, then
JSON can be used, and may even be better, however I also have a perl
client side module, and I have a feeling my users are going to want a
Java one as well. How is JSON with security... I run the returned value
in a eval block. Of the examples I've seen everyone seems to trust that
the data returned is valid and not harmful. Can you craft a JSON reply
that does something malicious?

Just brainstorming here, because this looks interesting.

1. If your audience is so open to many and various clients, how hard
could it be to get them to install another scripting language? If they
already have Perl, maybe ActiveState's PerlScript could do what you're
looking for, here. Theoretically that should be an easy port from the
stand-alone client you already have, to boot.

2. You could also just dump back fully-formed HTML, which is loaded in
a visible frame or iframe. Depending on how you do it (avoid tables),
most browsers will "stream" this to the screen in the way you're
thinking.

3. If you don't mind some ugly scripting on both client- and
server-side, you can have your server-side script break the message
apart so that the client has to request it in (well-formed) segments.
This means that your client-side scripting has to know what to request,
and your server-side obviously has to be able to fulfill those requests
with exactly the expected response. This could also increase your load
time quite a bit, as each segment is a separate HTTP request with all
the overhead that goes with.

4. Return a multipart (MIME) response, with each part being text/html
enclosing segments of your data as JavaScript. Each section of JS code
will execute independently (but asynchronously) of the others, and
because the execution completes before the next HTML body loads, it may
cause the browser to render the changes. I've never done anything like
that before, but it might work. You may also be able to do something
similar with your current XML approach, but I don't know how well (or
if at all) XMLHTTPRequest would handle a multipart response, even if
all the message parts are well-formed XML with the right content-type.

[googlegroups]

Christopher J. Hahn wrote:

Just brainstorming here, because this looks interesting.

1. If your audience is so open to many and various clients, how hard
could it be to get them to install another scripting language? If they
already have Perl, maybe ActiveState's PerlScript could do what you're
looking for, here. Theoretically that should be an easy port from the
stand-alone client you already have, to boot.
Practically impossible. There are hundreds of developers that would
want to use the tool we are creating (both the web side and the client
API piece). Those developers don't have time to learn another language,
and have the tendency to exert pressure on my management to get things
done their way. The API has to be language independent. I don't want to
write it twice, so I would prefer one API for both the web piece and
whatever client side modules get written.
2. You could also just dump back fully-formed HTML, which is loaded in
a visible frame or iframe. Depending on how you do it (avoid tables),
most browsers will "stream" this to the screen in the way you're
thinking.
I will have to think about this. I currently send back some data that
doesn't initially get displayed so i need to figure out a way to embed
it in the data 'tree' in such a way as to make it hidden (css?). It
would really add a lot to the structure of what is returned back though
as I now how to put a lot of stuff in div blocks to create a structure
of sorts. It would also make it harder to manipulate the data (re-sort
on different columns, etc.)
4. Return a multipart (MIME) response, with each part being text/html
enclosing segments of your data as JavaScript. Each section of JS code
will execute independently (but asynchronously) of the others, and
because the execution completes before the next HTML body loads, it may
cause the browser to render the changes. I've never done anything like
that before, but it might work. You may also be able to do something
similar with your current XML approach, but I don't know how well (or
if at all) XMLHTTPRequest would handle a multipart response, even if
all the message parts are well-formed XML with the right content-type.

This URL http://www.xulplanet.com/tutorials/m...serverpush.php (look
at the section on "multipart") suggests that a solution like this might
be possible (in theory I have not tried it out). I could hypothetically
split the data into its rows and send each row as its own part. Using
the onload handler each row would then get handled seperately. There is
however no mention on how IE will handle this.

Thanks,
Scott

[googlegroups]

go**********@scottsavarese.com wrote:

This URL http://www.xulplanet.com/tutorials/m...serverpush.php (look
at the section on "multipart") suggests that a solution like this might
be possible (in theory I have not tried it out). I could hypothetically
split the data into its rows and send each row as its own part. Using
the onload handler each row would then get handled seperately. There is
however no mention on how IE will handle this.

Just tested this... It doesn't work for IE. XMLHTTPRequest in IE does
not support multipart.

Scott

[Christopher J. Hahn]

go**********@scottsavarese.com wrote:

go**********@scottsavarese.com wrote:

This URL http://www.xulplanet.com/tutorials/m...serverpush.php (look
at the section on "multipart") suggests that a solution like this might
be possible (in theory I have not tried it out). I could hypothetically
split the data into its rows and send each row as its own part. Using
the onload handler each row would then get handled seperately. There is
however no mention on how IE will handle this.

Just tested this... It doesn't work for IE. XMLHTTPRequest in IE does
not support multipart.

Scott

Shame about the multipart-- that would've been a cool solution, I
thought.

But as regards extending the browser's language support, I wasn't
thinking of making the API output PerlScript (that would be practically
Satanic for other languages to deal with), but rather using client-side
browser-hosted PerlScript as your parser/DOM manipulator. Perl is an
excellent language for any parsing need, and it may operate
asynchronously enough that you can cause rendering to occur
simultaneously. But I wouldn't know-- I've never touched PerlScript,
just PERL. I know it has support on Windows, but other platforms I am
unsure of. See http://www.activestate.com

If it works out, you could still use XML over HTTP, but using
PerlScript as your parser.

The other option, which I would prefer to that, is this:
Use a hidden iframe to load the data from the server.
The server's response would still be multipart, but XMLHTTPRequest
support for it would no longer be an issue.
The responses should be fully-formed HTML documents, but with nothing
in the body. Instead, it would primarily be outputing JSON (easy to
parse) structures.
A second script tag would also include certain function definitions.
The <body> tag would include an onload to call a loader function in the
parent document (the one that contains the iframe) that actually clones
the JSON structures.

For people who want it to be language-independent:
The HTML is predictable and easy to parse, and the tags should be
considered comments outside of a browser context.
The JSON is simple and should be analogous to data structures available
in most modern languages. Parsing JSON is incredibly simple-- far
simpler in my opinion than, say, XML. It also has the advantage of
extreme ease of transferability to JavaScript (ahem) and Java, and
other languages with similar syntax for declaring literals.

Just strip out the html ( s/<.*?>//g ) and parse the JSON and you've
got your data.

The only issue is the complexity of the multipart.

[googlegroups]

Christopher J. Hahn wrote:

The other option, which I would prefer to that, is this:
Use a hidden iframe to load the data from the server.
The server's response would still be multipart, but XMLHTTPRequest
support for it would no longer be an issue.
The responses should be fully-formed HTML documents, but with nothing
in the body. Instead, it would primarily be outputing JSON (easy to
parse) structures.
A second script tag would also include certain function definitions.
The <body> tag would include an onload to call a loader function in the
parent document (the one that contains the iframe) that actually clones
the JSON structures.

Can the iframes do POST requests? If I only get to manipulate the URL
it calls, it seems that only GET requests would be possible. Also, how
would I do the streaming? Multipart and onload handlers?

Thanks,
Scott

[Jim Ley]

On 24 Jun 2005 07:23:21 -0700, go**********@scottsavarese.com wrote:

Can the iframes do POST requests?
Yes, just submit a form in it.
Also, how
would I do the streaming? Multipart and onload handlers?

No, you'd just but script blocks in and have them call, no event.

Jim.

[Christopher J. Hahn]

go**********@scottsavarese.com wrote:

Christopher J. Hahn wrote:

The other option, which I would prefer to that, is this:
Use a hidden iframe to load the data from the server.
The server's response would still be multipart, but XMLHTTPRequest
support for it would no longer be an issue.
The responses should be fully-formed HTML documents, but with nothing
in the body. Instead, it would primarily be outputing JSON (easy to
parse) structures.
A second script tag would also include certain function definitions.
The <body> tag would include an onload to call a loader function in the
parent document (the one that contains the iframe) that actually clones
the JSON structures.

Can the iframes do POST requests? If I only get to manipulate the URL
it calls, it seems that only GET requests would be possible. Also, how
would I do the streaming? Multipart and onload handlers?

Thanks,
Scott

Just to clarify, in the responses from the server, there should only be
one script "block". The other script tag (technically a block, but
leave it empty) should reference an external .js file.

This way, you can put your JSON data in the one block, and in the empty
one define functions for the parent document to handle the data. Call
the onload handler from the body tag.

The reason you'd do it this way is to satisfy the language independence
requirement. Because all JS unrelated to the JSON structures is defined
or referenced entirely in tag attributes, when you strip the HTML with
that simple s/<.*?>//g regex, you're left with nothing but JSON and
whitespace. No pesky function definitions cluttering up the JSON.

Or you might find it easier to make the server-side logic accept an
optional query-string variable requesting an HTML response, but
defaulting to XML.

Whatever works. I like the multipart idea, though.

[Jim Ley]

On 24 Jun 2005 21:25:34 -0700, "Christopher J. Hahn"
<cj****@gmail.com> wrote:

go**********@scottsavarese.com wrote:

Can the iframes do POST requests? If I only get to manipulate the URL
it calls, it seems that only GET requests would be possible. Also, how
would I do the streaming? Multipart and onload handlers?

Just to clarify, in the responses from the server, there should only be
one script "block". The other script tag (technically a block, but
leave it empty) should reference an external .js file.

This way, you can put your JSON data in the one block, and in the empty
one define functions for the parent document to handle the data. Call
the onload handler from the body tag.

This will not however be "streaming", waiting for onload is a
pointless reduced quality degradation of the page.

Jim.

[Christopher J. Hahn]

Jim Ley wrote:

On 24 Jun 2005 21:25:34 -0700, "Christopher J. Hahn"
<cj****@gmail.com> wrote:

go**********@scottsavarese.com wrote:

Can the iframes do POST requests? If I only get to manipulate the URL
it calls, it seems that only GET requests would be possible. Also, how
would I do the streaming? Multipart and onload handlers?

Just to clarify, in the responses from the server, there should only be
one script "block". The other script tag (technically a block, but
leave it empty) should reference an external .js file.

This way, you can put your JSON data in the one block, and in the empty
one define functions for the parent document to handle the data. Call
the onload handler from the body tag.

This will not however be "streaming", waiting for onload is a
pointless reduced quality degradation of the page.

Jim.

Of course it wouldn't be streaming. Lacking a real and worthwhile
cross-browser way to implement streaming given the requirements of the
project, we're left with emulation.

Whether waiting for onload is pointless is another matter entirely.
Since you've neither asked for an explanation nor given one yourself,
there's little more to say than that.

[Jim Ley]

On 26 Jun 2005 04:57:15 -0700, "Christopher J. Hahn"
<cj****@gmail.com> wrote:

Jim Ley wrote:

>This way, you can put your JSON data in the one block, and in the empty
>one define functions for the parent document to handle the data. Call
>the onload handler from the body tag.
This will not however be "streaming", waiting for onload is a
pointless reduced quality degradation of the page.

Jim.

Of course it wouldn't be streaming. Lacking a real and worthwhile
cross-browser way to implement streaming given the requirements of the
project, we're left with emulation.

but, the example method I've already given in the thread, is a much
better emulation, the problem as posed is that OP does not want to
wait for the entire file to download before anything happens -
completely reasonable. The option I suggested was to write a document
out

<script type="text/javascript">
parent.functionName("donkey");
/* ... */
</script>

<script type="text/javascript">
parent.functionName("chicken");
/* ... */
</script>

<script type="text/javascript">
parent.functionName("Moomin");
/* ... */
</script>

This method has the functions called as the document is loading, the
script executes as the file is loading, not just once onload.
Since you've neither asked for an explanation nor given one yourself,
there's little more to say than that.

I'd already given the example in the thread, maybe you could've
responded to that one if you think it was in some way bad?

Jim.

[Christopher J. Hahn]

Jim Ley wrote:

On 26 Jun 2005 04:57:15 -0700, "Christopher J. Hahn"
<cj****@gmail.com> wrote:

Jim Ley wrote:

>This way, you can put your JSON data in the one block, and in the empty
>one define functions for the parent document to handle the data. Call
>the onload handler from the body tag.

This will not however be "streaming", waiting for onload is a
pointless reduced quality degradation of the page.

Jim.

Of course it wouldn't be streaming. Lacking a real and worthwhile
cross-browser way to implement streaming given the requirements of the
project, we're left with emulation.

but, the example method I've already given in the thread, is a much
better emulation, the problem as posed is that OP does not want to
wait for the entire file to download before anything happens -
completely reasonable. The option I suggested was to write a document
out

<script type="text/javascript">
parent.functionName("donkey");
/* ... */
</script>

<script type="text/javascript">
parent.functionName("chicken");
/* ... */
</script>

<script type="text/javascript">
parent.functionName("Moomin");
/* ... */
</script>

This method has the functions called as the document is loading, the
script executes as the file is loading, not just once onload.

Since you've neither asked for an explanation nor given one yourself,
there's little more to say than that.

I'd already given the example in the thread, maybe you could've
responded to that one if you think it was in some way bad?

Not at all, in and of itself. I thought I was pretty clear that onload
was being used as a component of the solution that the OP and I were
batting around. As a solution itself, your suggestion seems perfectly
reasonable to me. If the function call takes a single argument (the
JSON structure), then parsing the file from other client apps gets only
very slightly more complicated.

So no, I don't think it's in any way bad.

Whether or why the OP couldn't or wouldn't use it isn't entirely clear
to me, but I'm sure he has his reasons. If he couldn't, then onload
seems like a perfectly valid way to deal with multiple partial
responses consisting of JSON within an otherwise barren HTML container.
In the context of a complete response, I agree with you that it'd be
pointless and wouldn't even give the appearance of streaming, but
that's not the context we were discussing.

Please tell me if I've completely misunderstood.

[googlegroups]

Jim Ley wrote:

<script type="text/javascript">
parent.functionName("donkey");
/* ... */
</script>

<script type="text/javascript">
parent.functionName("chicken");
/* ... */
</script>

<script type="text/javascript">
parent.functionName("Moomin");
/* ... */
</script>

This seems like a pretty cool idea. I am going to have to give it a try
for either this project or another one at some point. Thanks for the
information.

Because we wanted to keep server overhead down we implemented a
throttle on the server replies. The throttle returns a couple hundred
rows of data, and then sends a failure message in the XML reply (looks
a lot like soap). If the failure message says that the throttle was
reached, I will simply parse all the data received, then request more
data from the server in a new request. A pretty bad hack I know. The
throttle works for the perl implementation, the javascript
implementation, and other languages as well.

Thanks,
Scott