By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
425,925 Members | 732 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 425,925 IT Pros & Developers. It's quick & easy.

Slow Sql Query

P: 12
Hi there, I'm new to this forum, but I've come here quite a bit to find solutions to problems other people have encountered... I'm currently trying to query a database up to 5000 times as fast as possible to retrieve certain pricing data for a list of companies. I've run into a few queries that seem to be taking much longer then others and ultimately slow down my function call to a point where it is no longer useful. For instance... I have 2 different catalogues for different sources of data... one holds information regarding pricing, the other holds information regarding the trading volume.... the pricing query executes for all 5000 companies in less then 10 seconds, but the trading volume takes upwards of 5 minutes. There are 2 queries for each catalogue that seem to be running slow... I've tried to eliminate computational items from the queries to speed things up, but this still does not help. Here are 2 of the slow queries(as stored procedures):

Expand|Select|Wrap|Line Numbers
  1. ALTER PROCEDURE dbo.GetQuarterlyEPS
  2.     (
  3.     @Gvkey int,
  4.     @StartDate smalldatetime,
  5.     @EndDate smalldatetime
  6.     )
  7. AS
  8.     SELECT     gvkey, datadate, epspxq
  9.     FROM       co_ifndq
  10.     WHERE     (gvkey = @Gvkey) AND consol = 'C' AND datadate <= @StartDate AND datadate > @EndDate
  11.     ORDER BY datadate DESC
  12.     RETURN
  13.  
  14. ALTER PROCEDURE dbo.GetMeanEstimate
  15.     (
  16.     @ibtic nchar(3),
  17.     @StartFirstDate smalldatetime,
  18.     @EndFirstDate smalldatetime,
  19.     @StartSecondDate smalldatetime,
  20.     @EndSecondDate smalldatetime
  21.     )
  22. AS
  23.  
  24.     SELECT IBESStatisticalPeriod, MeanEstimate 
  25.     FROM USSummaryStatisticsAdjusted
  26.     WHERE IBESTicker = @ibtic 
  27.     AND 
  28.         (
  29.             (IBESStatisticalPeriod <= @StartFirstDate
  30.             AND IBESStatisticalPeriod > @EndFirstDate) 
  31.         OR 
  32.             (IBESStatisticalPeriod <= @StartSecondDate 
  33.             AND IBESStatisticalPeriod > @EndSecondDate)
  34.         ) 
  35.     AND ForecastPeriodIndicator = 1
  36.     ORDER BY IBESStatisticalPeriod DESC
  37.     RETURN
  38.  
  39.  
I'm not an SQL guy... my forte is much more C++/C#, so please keep that in mind if you have a solution or any ideas to help speed things up.

The tables have upwards of 100,000 entries each and in some cases more then 300 columns. Would doing a subquery for the columns I need and then filtering the results speed things up?

Thanks! - Will
Feb 1 '08 #1
Share this Question
Share on Google+
4 Replies


ck9663
Expert 2.5K+
P: 2,878
Hi there, I'm new to this forum, but I've come here quite a bit to find solutions to problems other people have encountered... I'm currently trying to query a database up to 5000 times as fast as possible to retrieve certain pricing data for a list of companies. I've run into a few queries that seem to be taking much longer then others and ultimately slow down my function call to a point where it is no longer useful. For instance... I have 2 different catalogues for different sources of data... one holds information regarding pricing, the other holds information regarding the trading volume.... the pricing query executes for all 5000 companies in less then 10 seconds, but the trading volume takes upwards of 5 minutes. There are 2 queries for each catalogue that seem to be running slow... I've tried to eliminate computational items from the queries to speed things up, but this still does not help. Here are 2 of the slow queries(as stored procedures):

Expand|Select|Wrap|Line Numbers
  1. ALTER PROCEDURE dbo.GetQuarterlyEPS
  2.     (
  3.     @Gvkey int,
  4.     @StartDate smalldatetime,
  5.     @EndDate smalldatetime
  6.     )
  7. AS
  8.     SELECT     gvkey, datadate, epspxq
  9.     FROM       co_ifndq
  10.     WHERE     (gvkey = @Gvkey) AND consol = 'C' AND datadate <= @StartDate AND datadate > @EndDate
  11.     ORDER BY datadate DESC
  12.     RETURN
  13.  
  14. ALTER PROCEDURE dbo.GetMeanEstimate
  15.     (
  16.     @ibtic nchar(3),
  17.     @StartFirstDate smalldatetime,
  18.     @EndFirstDate smalldatetime,
  19.     @StartSecondDate smalldatetime,
  20.     @EndSecondDate smalldatetime
  21.     )
  22. AS
  23.  
  24.     SELECT IBESStatisticalPeriod, MeanEstimate 
  25.     FROM USSummaryStatisticsAdjusted
  26.     WHERE IBESTicker = @ibtic 
  27.     AND 
  28.         (
  29.             (IBESStatisticalPeriod <= @StartFirstDate
  30.             AND IBESStatisticalPeriod > @EndFirstDate) 
  31.         OR 
  32.             (IBESStatisticalPeriod <= @StartSecondDate 
  33.             AND IBESStatisticalPeriod > @EndSecondDate)
  34.         ) 
  35.     AND ForecastPeriodIndicator = 1
  36.     ORDER BY IBESStatisticalPeriod DESC
  37.     RETURN
  38.  
  39.  
I'm not an SQL guy... my forte is much more C++/C#, so please keep that in mind if you have a solution or any ideas to help speed things up.

The tables have upwards of 100,000 entries each and in some cases more then 300 columns. Would doing a subquery for the columns I need and then filtering the results speed things up?

Thanks! - Will
am not much of a sql optimizer...but i would assume am the one creating your application i would do the following:
1. create index with those searched columns as keys. if your table has only 1 index, it's recommended to be clustered. it does not have to unique.
2. use BETWEEN to check those dates
3. conditions are processed from left to right. in an AND condition, only one has to be false to make the entire condition false. so re-arrange your conditions properly based on what your data would most probably show.

-- CK
Feb 1 '08 #2

Delerna
Expert 100+
P: 1,134
Agree with all that CK says.
Indexes will have the biggest impact on speeding up your queries.
I have seen indexes take 10 minute queries down to seconds

Looking at your queries those 2 date fields look like good candidates
but think carefully about the fields you choose to index.
Indexes speed reads of your data but they also slowdown writes.
This is because each time a record is written to, any an all indexs
also have to be updated. If you have too many indexes all your reads
will be superfast but the writes can become unuseable.
Feb 1 '08 #3

P: 12
Thanks for the response guys... I've managed to speed up 2 of my 4 slow queries with the keys for one of the databases, but the queries that access the tables with 300+ columns are still performing very slow. They're already keyed with 6 different columns and I'm querying using 3 in the where clauses. I'm assuming I shouldn't index them if they are already keyed.

Here are the remaining Queries:

Expand|Select|Wrap|Line Numbers
  1. ALTER PROCEDURE dbo.Get5YearQuarterlyEPS
  2.     (
  3.     @Gvkey int,
  4.     @Date smalldatetime,
  5.     @EndDate smalldatetime
  6.     )
  7. AS
  8.     SELECT     gvkey, datadate, epspxq
  9.     FROM       co_ifndq
  10.     WHERE     (gvkey = @Gvkey) AND (datadate BETWEEN @Date AND @EndDate) AND consol = 'C' 
  11.     ORDER BY datadate DESC
  12.     RETURN
  13.  
  14.  
  15. ALTER PROCEDURE dbo.GetQuarterlyEPS
  16.     (
  17.     @Gvkey int,
  18.     @StartDate smalldatetime,
  19.     @EndDate smalldatetime
  20.     )
  21. AS
  22.     SELECT     gvkey, datadate, epspxq
  23.     FROM       co_ifndq
  24.     WHERE     (gvkey = @Gvkey) AND datadate BETWEEN @StartDate AND @EndDate AND consol = 'C' 
  25.     ORDER BY datadate DESC
  26.     RETURN
  27.  
Without these queries, 1 iteration of data collection takes about 3-4 seconds... with them, it takes about 3 minutes. If I change the order by to match the same as the indecies, would that possibly speed things up?
Feb 4 '08 #4

P: 12
Ahhhh never mind. Problem Solved. I created indecies as well as keys and everything spead up. Thanks so much for the help, the suggestions were excactly what I needed. Cheers - Will
Feb 4 '08 #5

Post your reply

Sign in to post your reply or Sign up for a free account.