473,568 Members | 2,850 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

slow linq query

Eps
Hi there,

I am doing the following, this is a List of audio files.

this.Where(p =p.Album == AnAudioFileObje ct.Album).Selec t(s =>
s.Artist).Disti nct().Count() 1;

The aim is to determine whether AnAudioFileObje ct is from an album that
has various artists on it or just one artist.

If I load several thousand audio files into the list it becomes very
slow, can anyone think of a way I could speed this up ?.

Any help appreciated.

--
Eps
Sep 9 '08 #1
12 3047
How about

if (this.Any(p =p.Album == AnAudioFileObje ct.Album)) { .. }

? :-)

--
With regards
Anders Borum / SphereWorks
Microsoft Certified Professional (.NET MCP)
Sep 9 '08 #2
Eps <ep*@mailinator .comwrote:
I am doing the following, this is a List of audio files.

this.Where(p =p.Album == AnAudioFileObje ct.Album).Selec t(s =>
s.Artist).Disti nct().Count() 1;

The aim is to determine whether AnAudioFileObje ct is from an album that
has various artists on it or just one artist.

If I load several thousand audio files into the list it becomes very
slow, can anyone think of a way I could speed this up ?.

Any help appreciated.
Could you give definite figures for "several thousand" and "very slow"?
All of those should be linear operations as far as I'm aware, so it's
possible something else is going on.

Could you post a short but complete program which demonstrates the
problem?

See http://www.pobox.com/~skeet/csharp/complete.html for details of
what I mean by that.

--
Jon Skeet - <sk***@pobox.co m>
Web site: http://www.pobox.com/~skeet
Blog: http://www.msmvps.com/jon.skeet
C# in Depth: http://csharpindepth.com
Sep 9 '08 #3
Eps
Anders Borum wrote:
How about

if (this.Any(p =p.Album == AnAudioFileObje ct.Album)) { .. }

? :-)
hmmm, do half of it with it linq and the other half programmaticall y ?.

I thought that linq and programmatic iteration were roughly about as
fast as each other, whats the advantage of mixing the two ?.

Just after I posted I realized I could do this...

this.Where(p =p.Album == AnAudioFileObje ct.Album).Take( 2).Select(s =>
s.Artist).Disti nct().Count() 1;

The Take(2) does seem to have a significant impact on the speed, its not
ideal since there are albums with various artists that do feature the
same artist twice (or more times) but for my purposes I think it will be ok.

--
Eps
Sep 9 '08 #4
Eps
Jon Skeet [C# MVP] wrote:
Could you give definite figures for "several thousand" and "very slow"?
All of those should be linear operations as far as I'm aware, so it's
possible something else is going on.

Could you post a short but complete program which demonstrates the
problem?

See http://www.pobox.com/~skeet/csharp/complete.html for details of
what I mean by that.
ok, lets say 6000 audio files, each is an object that will read in data
and metadata directly from the file on the disk, so its IO bound to a
certain extent. In terms of time I can't be exact but the time taken to
process a file jumped from about 30 - 40 seconds to over a couple of
minutes.

I think I kind of have a solution now so I won't bother trying to
replicate the problem in a demonstration program.

Thanks for your help though.

--
Eps
Sep 9 '08 #5
Eps <ep*@mailinator .comwrote:
I am doing the following, this is a List of audio files.

this.Where(p =p.Album == AnAudioFileObje ct.Album).Selec t(s =>
s.Artist).Disti nct().Count() 1;

The aim is to determine whether AnAudioFileObje ct is from an album that
has various artists on it or just one artist.

If I load several thousand audio files into the list it becomes very
slow, can anyone think of a way I could speed this up ?.
I've just thought of something which could theoretically help. Instead
of calling Count(), call Take(1).Any(). That way as soon as Distinct()
yields its second element, you can finish.

That way you can deal with *huge* sequences, or even infinite ones (so
long as they aren't an infinite sequence repeating a single element).
For example:

using System;
using System.Linq;

public static class Test
{
static void Main()
{
var allPositiveInts = Enumerable.Rang e(0, int.MaxValue);
bool quick = allPositiveInts .Distinct().Tak e(1).Any();
Console.WriteLi ne("quick = " + quick);
bool slow = allPositiveInts .Distinct().Cou nt() 1;
Console.WriteLi ne ("slow = " + slow);
}
}

The result is:

quick = True

Unhandled Exception: OutOfMemoryExce ption.

Having said all of this, I strongly suspect that you won't gain much
from this. Try printing out

this.Where(p =p.Album == AnAudioFileObje ct.Album)
.Select(s =s.Artist)
.Count()

just to see how many artist entries we're talking about as the input to
Distinct() to start with.

--
Jon Skeet - <sk***@pobox.co m>
Web site: http://www.pobox.com/~skeet
Blog: http://www.msmvps.com/jon.skeet
C# in Depth: http://csharpindepth.com
Sep 9 '08 #6
Eps <ep*@mailinator .comwrote:
ok, lets say 6000 audio files, each is an object that will read in data
and metadata directly from the file on the disk, so its IO bound to a
certain extent. In terms of time I can't be exact but the time taken to
process a file jumped from about 30 - 40 seconds to over a couple of
minutes.

I think I kind of have a solution now so I won't bother trying to
replicate the problem in a demonstration program.
I strongly suspect the problem wasn't in the code you showed then -
because that part should be very fast. 6000 entries is nothing. I
suspect if you load them all into memory to start with, the query will
execute pretty much instantly.

--
Jon Skeet - <sk***@pobox.co m>
Web site: http://www.pobox.com/~skeet
Blog: http://www.msmvps.com/jon.skeet
C# in Depth: http://csharpindepth.com
Sep 9 '08 #7
Eps <ep*@mailinator .comwrote:
Just after I posted I realized I could do this...

this.Where(p =p.Album == AnAudioFileObje ct.Album).Take( 2).Select(s =>
s.Artist).Disti nct().Count() 1;

The Take(2) does seem to have a significant impact on the speed, its not
ideal since there are albums with various artists that do feature the
same artist twice (or more times) but for my purposes I think it will be ok.
See my other post for an alternative which will be correct but still
fast.

--
Jon Skeet - <sk***@pobox.co m>
Web site: http://www.pobox.com/~skeet
Blog: http://www.msmvps.com/jon.skeet
C# in Depth: http://csharpindepth.com
Sep 9 '08 #8
Eps
Jon Skeet [C# MVP] wrote:
Having said all of this, I strongly suspect that you won't gain much
from this. Try printing out

this.Where(p =p.Album == AnAudioFileObje ct.Album)
.Select(s =s.Artist)
.Count()

just to see how many artist entries we're talking about as the input to
Distinct() to start with.
Hmmm, I haven't had a chance to test this yet but I think I know whats
going on.

Where the album is not set (String.IsNullO rEmpty) I put in "Unknown
Album". Obviously the query considers all these audio files (and there
will be a significant number of them) as a part of the same album.

I could combine your code with mine, do something like....

this.Where(p =p.Album == AnAudioFileObje ct.Album)
.Select(s =s.Artist)
.Distinct().Tak e(10).Any()

This still isn't foolproof, there could be an album with 10 tracks by
one artist and 1 track (or more) by another. But this should be good
enough, I will test it and post the results.

--
Eps
Sep 9 '08 #9
I'm sorry, I misread your original query - sorry about that (I promise I'll
head straight to bed) ;-)

--
With regards
Anders Borum / SphereWorks
Microsoft Certified Professional (.NET MCP)

Sep 9 '08 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
11182
by: szwejk | last post by:
Hi! How to get result od dataTable from Linq query? I have typied DataSet and I want to join couple of tables. And I have a problem with change this result to DataTable type. (I don't want to rewrite everything in foreach) e.g. How can I make DataTable from it? Thx for help! var query =
1
10039
by: silpa | last post by:
Hi, I have an SQL query like this select distinct t1.prodID from Table1 t1 left join Table2 t2 on t2.prodID = t1.prodID left join Table3 t3 on t3.serialno = t2.Id and t3.Qty = 0 where t3.Id is null This query is having 2 left joins.
2
4792
by: Joey | last post by:
I am querying a DataSet with LINQ. I am running into a problem when trying to construct my query because in the "from" clause I do not know the table name (range variable) until runtime. Possible table names might include "SomeTable1" or "SomeTable236" etc... Unfortunately when I try to use something like "from SomeTable +...
2
1392
by: =?Utf-8?B?Tmljaw==?= | last post by:
Hello, I need some assistance with a LINQ query. I've got a simple query: var q = from t in db.Table1 select t; Based on user input I'm adding some where clauses: if (condition1) q = q.Where(t =t.Field1==1);
3
1906
by: Vivien Parlat | last post by:
Hello, I am currently using VB.Net 2008 express. I use linq to perform queries on a database, and I'm using the following link's source to convert those queries into DataTables i can then bind to WinForms' DataGridViews: http://blogs.msdn.com/aconrad/archive/2007/09/07/science-project.aspx The point is, the properties names extracted...
1
8668
by: alex21 | last post by:
Ok i am trying to use a Linq query to access a dictionary. public static Dictionary<string, Client> Clients = new Dictionary<string, Client>();Using this Linq query: IEnumerable<Staff> loginquery = from staff in Database.Staff where staff.Value.Passcode == txt_password.Text select staff;but i get the error Error 2 Cannot implicitly...
2
6424
by: scott1010 | last post by:
Hello I am having problems with a Linq query. I need to return an ID field which is of type GUID (c.Id) along with other fields of type String. I have tried both anonymous types and strongly typed objects but continue to encounter the same error: Method 1: Anonymously Typed var results = (from c in uow.Contacts orderby c.LastName...
0
942
by: James Folkerman | last post by:
Hi. I have SQL Server database that is connected to my WPF desktop-based application via ADO.NET Entity Framework. Now I need get content of one of the table via LINQ query and show it in DataGrid control. Also, I need to let user to edit or delete data from DataGrid and update my database according this change. Here is a LINQ query: ...
1
2799
by: kgkgkg | last post by:
Hey everyone, I've dealt with some simple LINQ queries before, but am not sure how to tackle this one... I've got a project I'm working on where I'm dealing with two classes I've created, 'Package' and 'Product' (along with other classes). I've also created a Job class which houses a List<Package> and inside the Package class it houses a...
0
7604
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
8117
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7660
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
7962
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
0
6275
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
1
5498
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
5217
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3651
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
1
2101
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.