Large Array Iteration (200000) - any chance for optimization?

[Adam Schaible]

Hello!

My project has the following interface: (screenshot works the best)
http://www.screencast.com/users/asch...9-4a7af883e3c9

Device groups could have 1 device, or 100,000 devices.

I have an asynchronous action when you select a different device group
- it will go to the server, get the applicable devices - remove the
current devices from the list, then re-add all the devices it has
retrieved.

This takes a long time with around 20,000 devices (20-30 seconds).
Here's my code:

//result is an array of strings
function populateDeviceList(result){
var listBox = $('.DeviceList')[0];

//Clear the list
for (var i = listBox.options.length - 1 ; i >= 0 ; i--) {
listBox.options[i] = null;
}

//Repopulate with results
var deviceListFilterBox = $('.DeviceListFilterBox')[0].value;
var len = result.length;
for (var i = 0;i < len;i++){
var thisOne = result[i];
if (thisOne.startsWith(deviceListFilterBox)){
AddItem(listBox, thisOne, thisOne);
}
}
disableSelectedItems($('body'));
}

function AddItem(objListBox, strText, strId, added)
{
var newOption = new Option(strText, strId)
objListBox.options.add(newOption);
}

Any suggestions?

Thanks in advance!

[Peter Michaux]

On Feb 19, 10:15 am, Adam Schaible <[e-mail]:

Hello!

My project has the following interface: (screenshot works the best)http://www.screencast.com/users/asch...media/65d7db9d...

Device groups could have 1 device, or 100,000 devices.

I have an asynchronous action when you select a different device group
- it will go to the server, get the applicable devices - remove the
current devices from the list, then re-add all the devices it has
retrieved.

This takes a long time with around 20,000 devices (20-30 seconds).
Here's my code:

//result is an array of strings
function populateDeviceList(result){
var listBox = $('.DeviceList')[0];

In which library is $()? It is likely true that you can improve the
efficiency here by doing this DOM search manually. If you don't want
to do this manually give the $() some more help. Using a tagName or id
usually helps the most $('ul.DeviceList') or $('#firstDeviceList').

//Clear the list
for (var i = listBox.options.length - 1 ; i >= 0 ; i--) {

What are listBox.options? Is this jQuery and does the jQuery object
have an options array-like property?

listBox.options[i] = null;
}

//Repopulate with results
var deviceListFilterBox = $('.DeviceListFilterBox')[0].value;
var len = result.length;
for (var i = 0;i < len;i++){
var thisOne = result[i];
if (thisOne.startsWith(deviceListFilterBox)){

I don't know what startsWith is or how efficient it is.

AddItem(listBox, thisOne, thisOne);

It is odd to have a function start with a captial letter unless it is
preceded by "new"

}
}
disableSelectedItems($('body'));
}

function AddItem(objListBox, strText, strId, added)
{
var newOption = new Option(strText, strId)
objListBox.options.add(newOption);

}

It is difficult to make suggestions when there is clearly a library
involved and I don't know which one.

Try commenting out the body of the long for-loop and seeing how long
it takes. Then start adding in bits until you find the bit that makes
it take a long time. When you find that bit, optimize it.

Peter

[Adam Schaible]

Here's my code:

* * //Clear the list
* * for (var i = listBox.options.length - 1 ; i >= 0 ; i--) {
* * * * listBox.options[i] = null;
* * }

Why clear the list? Why not just do listBox.options=[];

When I do that, I get a javascript exception: setting a property that
has only a getter.
I did look at simply setting options to it's default state - but so
far the best method I've found is to iterate backwards through the
list.

Thanks for the suggestions so far!

[Joost Diepenmaat]

Adam Schaible <ad***********@gmail.comwrites:

On a side note, does anyone have experience with arrays of this size?
If it's simply not possible, I'd rather move on!

Well, it seems you're adding lots and lots of data to some kind of
option list (/is/ it a <selector just something that vaguely looks
like it?)

Chances are high that the visitor won't be able to view more than about
a 100 items on a single screen. You may be able to take advantage of
that by just rendering the actual viewable data when it scrolls into
view (though it could get ugly & hard if you're using a plain <select>
box), and possibly filling in neighbouring pages in the background.

The big disadvantage is that your users probably won't be able to print
or use CTRL+F or any other built-in search mechanism for items that
aren't rendered yet.
--
Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/

[Adam Schaible]

Joost,

Thanks for the reply!

It is a plain old <selectand I'm adding Options to it.

Around 100 seems to be my convenience limit.

I'm moving towards rendering the entire select element server side and
simply replacing my current one with the new one asynchronously.

Thoughts?

[Joost Diepenmaat]

Adam Schaible <ad***********@gmail.comwrites:

Joost,

Thanks for the reply!

It is a plain old <selectand I'm adding Options to it.

Around 100 seems to be my convenience limit.

I'm moving towards rendering the entire select element server side and
simply replacing my current one with the new one asynchronously.

You mean, using innerHTML? That should be relatively quick (the
insertion part), but it *will* become slow at some point. I think at
20,000 items you may already be approaching the limits of what's
acceptible. Also, transferring all that HTML around may be slow.

Thoughts?

You have a search box above the select, right? You could require a
minimal substring - say 3 characters - before you show anything. That
could bring down the number matches a lot.

Alternatively, just show the first 1000 or so matches (limit the output
at the server side), and show a message that there are more matches but
the user has to either "page" to the next 1000 items or refine the
search. Personally, I'd think this is a good mix of not trying to be too
clever and still giving the user the option of walking & searching
trough all items by hand if (s)he feels like it.

--
Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/

[Adam Schaible]

Ed:

That's a great suggestion - I got it going by doing it all "offline"
but it still took way too long. I think if I run it through the
server, can get a new select element in about 1 second, so I'll do it
through a web service.

[timothytoe]

On Feb 19, 11:34 am, Adam Schaible <[e-mail]:

Here's my code:
//Clear the list
for (var i = listBox.options.length - 1 ; i >= 0 ; i--) {
listBox.options[i] = null;
}

Why clear the list? Why not just do listBox.options=[];

When I do that, I get a javascript exception: setting a property that
has only a getter.
I did look at simply setting options to it's default state - but so
far the best method I've found is to iterate backwards through the
list.

Thanks for the suggestions so far!

Then how about...
listBox.options.length=0;

[David Mark]

On Feb 19, 3:11*pm, Adam Schaible <[e-mail]:

Ed:

That's a great suggestion - I got it going by doing it all "offline"
but it still took way too long. *I think if I run it through the
server, can get a new select element in about 1 second, so I'll do it
through a web service.

Careful. IE fails to set the innerHTML of a select element properly.
You will need to feature test this and use outerHTML to replace the
entire select element when applicable.

[Adam Schaible]

Ok great, thanks for the tip!

[Yanick]

On Feb 19, 1:15 pm, Adam Schaible <[e-mail]:

Hello!

My project has the following interface: (screenshot works the best)http://www.screencast.com/users/asch...media/65d7db9d...

Device groups could have 1 device, or 100,000 devices.

I have an asynchronous action when you select a different device group
- it will go to the server, get the applicable devices - remove the
current devices from the list, then re-add all the devices it has
retrieved.

I don't know what library you are using, but look at the LiveGrid
example here : http://dowdybrown.com/dbprod/rico2/examples/php/ex2.php

This is not a native list control, and ajax calls would need to be re-
written, but the implementation should not be too difficult to write
for your need.