Slow Sql Query

[bfoo75]

Hi there, I'm new to this forum, but I've come here quite a bit to find solutions to problems other people have encountered... I'm currently trying to query a database up to 5000 times as fast as possible to retrieve certain pricing data for a list of companies. I've run into a few queries that seem to be taking much longer then others and ultimately slow down my function call to a point where it is no longer useful. For instance... I have 2 different catalogues for different sources of data... one holds information regarding pricing, the other holds information regarding the trading volume.... the pricing query executes for all 5000 companies in less then 10 seconds, but the trading volume takes upwards of 5 minutes. There are 2 queries for each catalogue that seem to be running slow... I've tried to eliminate computational items from the queries to speed things up, but this still does not help. Here are 2 of the slow queries(as stored procedures):

ALTER PROCEDURE dbo.GetQuarterlyEPS
    (
    @Gvkey int,
    @StartDate smalldatetime,
    @EndDate smalldatetime
    )
AS
    SELECT     gvkey, datadate, epspxq
    FROM       co_ifndq
    WHERE     (gvkey = @Gvkey) AND consol = 'C' AND datadate <= @StartDate AND datadate > @EndDate
    ORDER BY datadate DESC
    RETURN
 
ALTER PROCEDURE dbo.GetMeanEstimate
    (
    @ibtic nchar(3),
    @StartFirstDate smalldatetime,
    @EndFirstDate smalldatetime,
    @StartSecondDate smalldatetime,
    @EndSecondDate smalldatetime
    )
AS
 
    SELECT IBESStatisticalPeriod, MeanEstimate 
    FROM USSummaryStatisticsAdjusted
    WHERE IBESTicker = @ibtic 
    AND 
        (
            (IBESStatisticalPeriod <= @StartFirstDate
            AND IBESStatisticalPeriod > @EndFirstDate) 
        OR 
            (IBESStatisticalPeriod <= @StartSecondDate 
            AND IBESStatisticalPeriod > @EndSecondDate)
        ) 
    AND ForecastPeriodIndicator = 1
    ORDER BY IBESStatisticalPeriod DESC
    RETURN
 
 

I'm not an SQL guy... my forte is much more C++/C#, so please keep that in mind if you have a solution or any ideas to help speed things up.

The tables have upwards of 100,000 entries each and in some cases more then 300 columns. Would doing a subquery for the columns I need and then filtering the results speed things up?

Thanks! - Will

[ck9663]

Hi there, I'm new to this forum, but I've come here quite a bit to find solutions to problems other people have encountered... I'm currently trying to query a database up to 5000 times as fast as possible to retrieve certain pricing data for a list of companies. I've run into a few queries that seem to be taking much longer then others and ultimately slow down my function call to a point where it is no longer useful. For instance... I have 2 different catalogues for different sources of data... one holds information regarding pricing, the other holds information regarding the trading volume.... the pricing query executes for all 5000 companies in less then 10 seconds, but the trading volume takes upwards of 5 minutes. There are 2 queries for each catalogue that seem to be running slow... I've tried to eliminate computational items from the queries to speed things up, but this still does not help. Here are 2 of the slow queries(as stored procedures):

ALTER PROCEDURE dbo.GetQuarterlyEPS
    (
    @Gvkey int,
    @StartDate smalldatetime,
    @EndDate smalldatetime
    )
AS
    SELECT     gvkey, datadate, epspxq
    FROM       co_ifndq
    WHERE     (gvkey = @Gvkey) AND consol = 'C' AND datadate <= @StartDate AND datadate > @EndDate
    ORDER BY datadate DESC
    RETURN
 
ALTER PROCEDURE dbo.GetMeanEstimate
    (
    @ibtic nchar(3),
    @StartFirstDate smalldatetime,
    @EndFirstDate smalldatetime,
    @StartSecondDate smalldatetime,
    @EndSecondDate smalldatetime
    )
AS
 
    SELECT IBESStatisticalPeriod, MeanEstimate 
    FROM USSummaryStatisticsAdjusted
    WHERE IBESTicker = @ibtic 
    AND 
        (
            (IBESStatisticalPeriod <= @StartFirstDate
            AND IBESStatisticalPeriod > @EndFirstDate) 
        OR 
            (IBESStatisticalPeriod <= @StartSecondDate 
            AND IBESStatisticalPeriod > @EndSecondDate)
        ) 
    AND ForecastPeriodIndicator = 1
    ORDER BY IBESStatisticalPeriod DESC
    RETURN
 
 

I'm not an SQL guy... my forte is much more C++/C#, so please keep that in mind if you have a solution or any ideas to help speed things up.

The tables have upwards of 100,000 entries each and in some cases more then 300 columns. Would doing a subquery for the columns I need and then filtering the results speed things up?

Thanks! - Will

am not much of a sql optimizer...but i would assume am the one creating your application i would do the following:
1. create index with those searched columns as keys. if your table has only 1 index, it's recommended to be clustered. it does not have to unique.
2. use BETWEEN to check those dates
3. conditions are processed from left to right. in an AND condition, only one has to be false to make the entire condition false. so re-arrange your conditions properly based on what your data would most probably show.

-- CK

[Delerna]

Agree with all that CK says.
Indexes will have the biggest impact on speeding up your queries.
I have seen indexes take 10 minute queries down to seconds

Looking at your queries those 2 date fields look like good candidates
but think carefully about the fields you choose to index.
Indexes speed reads of your data but they also slowdown writes.
This is because each time a record is written to, any an all indexs
also have to be updated. If you have too many indexes all your reads
will be superfast but the writes can become unuseable.

[bfoo75]

Thanks for the response guys... I've managed to speed up 2 of my 4 slow queries with the keys for one of the databases, but the queries that access the tables with 300+ columns are still performing very slow. They're already keyed with 6 different columns and I'm querying using 3 in the where clauses. I'm assuming I shouldn't index them if they are already keyed.

Here are the remaining Queries:

ALTER PROCEDURE dbo.Get5YearQuarterlyEPS
    (
    @Gvkey int,
    @Date smalldatetime,
    @EndDate smalldatetime
    )
AS
    SELECT     gvkey, datadate, epspxq
    FROM       co_ifndq
    WHERE     (gvkey = @Gvkey) AND (datadate BETWEEN @Date AND @EndDate) AND consol = 'C' 
    ORDER BY datadate DESC
    RETURN
 
 
ALTER PROCEDURE dbo.GetQuarterlyEPS
    (
    @Gvkey int,
    @StartDate smalldatetime,
    @EndDate smalldatetime
    )
AS
    SELECT     gvkey, datadate, epspxq
    FROM       co_ifndq
    WHERE     (gvkey = @Gvkey) AND datadate BETWEEN @StartDate AND @EndDate AND consol = 'C' 
    ORDER BY datadate DESC
    RETURN
 

Without these queries, 1 iteration of data collection takes about 3-4 seconds... with them, it takes about 3 minutes. If I change the order by to match the same as the indecies, would that possibly speed things up?

[bfoo75]

Ahhhh never mind. Problem Solved. I created indecies as well as keys and everything spead up. Thanks so much for the help, the suggestions were excactly what I needed. Cheers - Will