473,324 Members | 1,856 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,324 software developers and data experts.

Filtering a list - efficiently

I have a list of items. Some of the items are another list and that list item
could be another list. Now I want to filter the main list by the inner list
item value based on some criteria...

My fileter criteria is, A and B and C and D and E then get the filtered
item...where A, B , C ,D, E are passed from the frong end.

How can I efficiently filter the main list. Any generic efficient logic so
that I can make it more generic code...

I was thinking if

- I can convert the list into DataView and filter by SQL queries?
- I can conver the list into XML file and use XPATH and XQuery?

Any better thoughts...

The slection of the technology is the key to the performance for our
application

THanks
Jun 27 '08 #1
9 6411
It sounds like you have a tree/hive of data. Could you perhaps throw the
data into a databas with a self-referencing, indexed table and just do
it in the database?

If you want to do it in-memory, then the best approach would probably
depend on how exactly you want to filter it. For example, if it is
equality, you could build a dictionary of values against a (list of)
paths to nodes (or the nodes themselves).

So: what would a typical filter look like?

Marc
Jun 27 '08 #2
Marc,

Thanks for the reply. I am working in SOA based application. My boundaries
are make a request and get a response. DB intereaction is not directly
involved. Its airline reservation system so filters are show me the flights
which are direct, min stops, certain carrier, etc etc. All the filter
criteria is in AND condition.

In a list I get flights which has certain properties and those properties
value I get as a list item.

Our design is like this...We get the response and we will business object
and business object is filling a list...classes are desicgned like that...

Any thoughts,...
"Marc Gravell" wrote:
It sounds like you have a tree/hive of data. Could you perhaps throw the
data into a databas with a self-referencing, indexed table and just do
it in the database?

If you want to do it in-memory, then the best approach would probably
depend on how exactly you want to filter it. For example, if it is
equality, you could build a dictionary of values against a (list of)
paths to nodes (or the nodes themselves).

So: what would a typical filter look like?

Marc
Jun 27 '08 #3
Well, if you are doing repeated AND queries, then one of the easier ways
to build an overall query is using LINQ-to-Objects (no database here) -
something like shown below - but this is still a very basic approach
that doesn't do anything special re indexes. If you need something more
sophisticated you'll need to give a lot of attention to likely queries.

Re the nested data, I don't understand enough from your description to
make much comment. It might be that the below is fine, i.e. if you just
want to check things like (read in context of main body below):

if (via != null)
{
query = query.Where(f =f.Stops.Contains(via));
}

Which again is very simplified; I'd probably suggest a
Dictionary<string, Airportfor things like Stops, so you can say:

string via = "LAX"; // via UI
if (via != null)
{
query = query.Where(f =f.Stops.ContainsKey(via));
}

But again - not a huge amount of context here...

Marc

using System.Linq;
static class Program
{
static void Main()
{
List<Flightflights = new List<Flight>();

var query = flights.AsQueryable();

int? maxStops = 3; // from UI
if (maxStops != null)
{
query = query.Where(f =f.Stops <= maxStops);
}

Carrier carrier = null; // from UI
if (carrier != null)
{
query = query.Where(f =f.Carrier == carrier);
}

// etc
}
}
Jun 27 '08 #4
Marc

THanks for the detailed email. We are right now on VS 2005. So it will take
a while to go with LINQ... but I will create a POC and can push to use the VS
2008 if required...

Thanks
"Marc Gravell" wrote:
Well, if you are doing repeated AND queries, then one of the easier ways
to build an overall query is using LINQ-to-Objects (no database here) -
something like shown below - but this is still a very basic approach
that doesn't do anything special re indexes. If you need something more
sophisticated you'll need to give a lot of attention to likely queries.

Re the nested data, I don't understand enough from your description to
make much comment. It might be that the below is fine, i.e. if you just
want to check things like (read in context of main body below):

if (via != null)
{
query = query.Where(f =f.Stops.Contains(via));
}

Which again is very simplified; I'd probably suggest a
Dictionary<string, Airportfor things like Stops, so you can say:

string via = "LAX"; // via UI
if (via != null)
{
query = query.Where(f =f.Stops.ContainsKey(via));
}

But again - not a huge amount of context here...

Marc

using System.Linq;
static class Program
{
static void Main()
{
List<Flightflights = new List<Flight>();

var query = flights.AsQueryable();

int? maxStops = 3; // from UI
if (maxStops != null)
{
query = query.Where(f =f.Stops <= maxStops);
}

Carrier carrier = null; // from UI
if (carrier != null)
{
query = query.Where(f =f.Carrier == carrier);
}

// etc
}
}
Jun 27 '08 #5
I use a tool named "Enterprise Core Objects" for my business objects /
persistence. One of the features is has is an OCL (object constring
language) parser, so you can execute select queries etc on your object
space. I take stuff like this for granted :-)
--
Pete
=========================================
I use Enterprise Core Objects (Domain driven design)
http://www.capableobjects.com/
=========================================
Jun 27 '08 #6
Well, considering is costs approx the same as Visual Studio (pro), I
think I'll stick to LINQ ;-p

But if you like it...

Marc
Jun 27 '08 #7
Well, considering is costs approx the same as Visual Studio (pro), I think
I'll stick to LINQ ;-p
I've been using it for years, it does a lot more than LINQ. So yes, I do
like it :-)


--
Pete
=========================================
I use Enterprise Core Objects (Domain driven design)
http://www.capableobjects.com/
=========================================
Jun 27 '08 #8
abcd wrote:
Marc

THanks for the detailed email. We are right now on VS 2005. So it will take
a while to go with LINQ... but I will create a POC and can push to use the VS
2008 if required...
The syntax is nicer with extension methods, but Select(), Where(), etc
are not difficult to implement in C#2.

An example (I've not tested or compiled it so I may have done something
stupid) for Select(...) is:

public static class MyLinq
{
public delegate Tresult SelectionDelegate<T, Tresult>(T obj);
public static IEnumerable<TresultSelect<T, Tresult>(IEnumerable<T>
input, SelectionDelegate<T, Tresultfunc)
{
foreach(T item in input)
{
yield return func(item);
}
}
}

Other 'LINQ' functions are similarly trivial.

The great thing about C# 3 is not LINQ in itself, but the fact that I
can now expect other developers to have no problem understanding what
I'm doing (and the better syntax).

Alun Harford
Jun 27 '08 #9
The syntax is nicer with extension methods, but Select(), Where(), etc
are not difficult to implement in C#2.
In fact LINQBridge does all this for you if you want LINQ-to-objects
on 2.0.

Marc
Jun 27 '08 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: Alex Ayzin | last post by:
Hi, I have a problem that might be easy to solve(possibly, I've just overlooked an easy solution). Here we go: I have a dataset with 2 datatables in it. Now, I need to do the following: if...
7
by: Adam Hartshorne | last post by:
As a result of a graphics based algorihtms, I have a list of indices to a set of nodes. I want to efficiently identify any node indices that are stored multiple times in the array and the...
13
by: kevinold | last post by:
Hello everyone, I have a list of about 1600 employees that I'd like to have displayed in a form. I'd like to make the "search" for the user as easy as possible. I ran across this:...
1
by: Elias Farah | last post by:
Hello All, I hope someone can give me (and other keen access enthusiasts) some helpful information to explain how to most efficiently filter Queries & subqueies. Consider this common simple...
19
by: William Wisnieski | last post by:
Hello Everyone, I have a main form with a datasheet subform that I use to query by form. After the user selects two criteria on the main form and clicks the cmdShowResults button on the main...
5
by: sensible | last post by:
Can I solve this problem using Access? If so, will some give this newbie a simple step by step on how to go about it, please....all the way from File/New. In Excel I am using...
7
by: | last post by:
Hello, Does anyone have an idea on how I can filter the data in the gridview control that was returned by an sql query? I have a gridview that works fine when I populate it with data. Now I...
2
by: JUAN ERNESTO FLORES BELTRAN | last post by:
Hi you all, I am developping a python application which connects to a database (postresql) and displays the query results on a treeview. In adittion to displaying the info i do need to implement...
3
by: Harry Haller | last post by:
Hello, I want to implement a generic list which will be used to display 7 columns in a GridView. One should be able to sort, filter and page each of the 7 columns. Ideally the filter should be...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.