473,326 Members | 2,111 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

Performance problem with correlated sub-query

I have created a web app that stores and displays all the messages from
my database maintenance jobs that run each night. The web app uses Java
servlets and has PostgreSQL 7.0 as the back end.



When the user requests the first page, he gets a list of all the servers
with maintenance records in the database, and a drop down list of all
the dates of maintenance records. If the user chooses a date first, then
the app uses a prepared statement with the date contained in a
parameter, and this executes very quickly - no problems.



However, if the web page user does not choose a date, then the app uses
a correlated sub-query to grab only the current (latest) day's
maintenance records. The query that is executed is:



select servername, databasename, message from messages o where
o.date_of_msg =

(select max(date_of_msg) from messages i where i.servername
= o.servername);



And this is a dog. It takes 15 - 20 minutes to execute the query (there
are about 200,000 rows in the table). I have an index on (servername,
date_of_msg), but it doesn't seem to be used in this query.



Is there a way to improve the performance on this query?



Thanks,



Steve Howard




This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message. Any disclosure, copying, or distribution of this message, or the taking of any action based on it, is strictly prohibited.

Nov 23 '05 #1
8 2034
Howard, Steven (US - Tulsa) wrote:
select servername, databasename, message from messages o where
o.date_of_msg = (select max(date_of_msg) from messages i where
i.servername = o.servername);

And this is a dog. It takes 15 – 20 minutes to execute the
query (there are about 200,000 rows in the table). I have an
index on (servername, date_of_msg), but it doesn’t seem to
be used in this query.


Just off the top of my head:

SELECT servername, databasename, message
FROM messages o
WHERE o.date_of_msg = (
SELECT date_of_msg
FROM messages i
WHERE i.servername = o.servername
ORDER BY date_of_msg
LIMIT 1
);
HTH,

Mike Mascari

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

Nov 23 '05 #2
Howard, Steven (US - Tulsa) wrote:
select servername, databasename, message from messages o where
o.date_of_msg = (select max(date_of_msg) from messages i where
i.servername = o.servername);

And this is a dog. It takes 15 – 20 minutes to execute the
query (there are about 200,000 rows in the table). I have an
index on (servername, date_of_msg), but it doesn’t seem to
be used in this query.


Just off the top of my head:

SELECT servername, databasename, message
FROM messages o
WHERE o.date_of_msg = (
SELECT date_of_msg
FROM messages i
WHERE i.servername = o.servername
ORDER BY date_of_msg
LIMIT 1
);
HTH,

Mike Mascari

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

Nov 23 '05 #3
On Thu, 29 Apr 2004, Howard, Steven (US - Tulsa) wrote:
I have created a web app that stores and displays all the messages from
my database maintenance jobs that run each night. The web app uses Java
servlets and has PostgreSQL 7.0 as the back end.
Step 1 is upgrade. ;)
However, if the web page user does not choose a date, then the app uses
a correlated sub-query to grab only the current (latest) day's
maintenance records. The query that is executed is:

select servername, databasename, message from messages o where
o.date_of_msg =

(select max(date_of_msg) from messages i where i.servername
= o.servername);
This is likely to be running the subquery once for each row in messages,
and probably not going to use an index in the inner either. The former
might be optimized by recent versions.

Changing the inner query to something like:
(select date_of_msg from messages i where i.servername=o.servername
order by date_of_msg desc limit 1)

or changing it to use a subselect in from (something like):
from messages o, (select servername, max(date_of_msg) from messages) i
where o.servername=i.servername

might both help, but I'm not sure either will work on 7.0.
And this is a dog. It takes 15 - 20 minutes to execute the query (there
are about 200,000 rows in the table). I have an index on (servername,
date_of_msg), but it doesn't seem to be used in this query.


You might wish to play around with changing the indexes and the order of
the columns in the multicolumn index as well.

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

Nov 23 '05 #4
On Thu, 29 Apr 2004, Howard, Steven (US - Tulsa) wrote:
I have created a web app that stores and displays all the messages from
my database maintenance jobs that run each night. The web app uses Java
servlets and has PostgreSQL 7.0 as the back end.
Step 1 is upgrade. ;)
However, if the web page user does not choose a date, then the app uses
a correlated sub-query to grab only the current (latest) day's
maintenance records. The query that is executed is:

select servername, databasename, message from messages o where
o.date_of_msg =

(select max(date_of_msg) from messages i where i.servername
= o.servername);
This is likely to be running the subquery once for each row in messages,
and probably not going to use an index in the inner either. The former
might be optimized by recent versions.

Changing the inner query to something like:
(select date_of_msg from messages i where i.servername=o.servername
order by date_of_msg desc limit 1)

or changing it to use a subselect in from (something like):
from messages o, (select servername, max(date_of_msg) from messages) i
where o.servername=i.servername

might both help, but I'm not sure either will work on 7.0.
And this is a dog. It takes 15 - 20 minutes to execute the query (there
are about 200,000 rows in the table). I have an index on (servername,
date_of_msg), but it doesn't seem to be used in this query.


You might wish to play around with changing the indexes and the order of
the columns in the multicolumn index as well.

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

Nov 23 '05 #5

On 29/04/2004 14:34 "Howard, Steven (US - Tulsa)" wrote:
I have created a web app that stores and displays all the messages from
my database maintenance jobs that run each night. The web app uses Java
servlets and has PostgreSQL 7.0 as the back end.
7.0? That's positively ancient!
When the user requests the first page, he gets a list of all the servers
with maintenance records in the database, and a drop down list of all
the dates of maintenance records. If the user chooses a date first, then
the app uses a prepared statement with the date contained in a
parameter, and this executes very quickly - no problems.

However, if the web page user does not choose a date, then the app uses
a correlated sub-query to grab only the current (latest) day's
maintenance records. The query that is executed is:

select servername, databasename, message from messages o where
o.date_of_msg =

(select max(date_of_msg) from messages i where i.servername
= o.servername);

And this is a dog. It takes 15 - 20 minutes to execute the query (there
are about 200,000 rows in the table). I have an index on (servername,
date_of_msg), but it doesn't seem to be used in this query.
PG doesn't use indexes for things like count(), max, min()...

You can avoid using max() by something like

select my_date from my_table order by my_date desc limit 1;

which will use the index.

Is there a way to improve the performance on this query?


In addition to the above, I'd strongly recommend upgrading to 7.4 to take
advantage of the last ~4 years of continuous improvements.

--
Paul Thomas
+------------------------------+---------------------------------------------+
| Thomas Micro Systems Limited | Software Solutions for
Business |
| Computer Consultants |
http://www.thomas-micro-systems-ltd.co.uk |
+------------------------------+---------------------------------------------+
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

Nov 23 '05 #6

On 29/04/2004 14:34 "Howard, Steven (US - Tulsa)" wrote:
I have created a web app that stores and displays all the messages from
my database maintenance jobs that run each night. The web app uses Java
servlets and has PostgreSQL 7.0 as the back end.
7.0? That's positively ancient!
When the user requests the first page, he gets a list of all the servers
with maintenance records in the database, and a drop down list of all
the dates of maintenance records. If the user chooses a date first, then
the app uses a prepared statement with the date contained in a
parameter, and this executes very quickly - no problems.

However, if the web page user does not choose a date, then the app uses
a correlated sub-query to grab only the current (latest) day's
maintenance records. The query that is executed is:

select servername, databasename, message from messages o where
o.date_of_msg =

(select max(date_of_msg) from messages i where i.servername
= o.servername);

And this is a dog. It takes 15 - 20 minutes to execute the query (there
are about 200,000 rows in the table). I have an index on (servername,
date_of_msg), but it doesn't seem to be used in this query.
PG doesn't use indexes for things like count(), max, min()...

You can avoid using max() by something like

select my_date from my_table order by my_date desc limit 1;

which will use the index.

Is there a way to improve the performance on this query?


In addition to the above, I'd strongly recommend upgrading to 7.4 to take
advantage of the last ~4 years of continuous improvements.

--
Paul Thomas
+------------------------------+---------------------------------------------+
| Thomas Micro Systems Limited | Software Solutions for
Business |
| Computer Consultants |
http://www.thomas-micro-systems-ltd.co.uk |
+------------------------------+---------------------------------------------+
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

Nov 23 '05 #7
Howard, Steven (US - Tulsa) wrote:
I have created a web app that stores and displays all the messages from
my database maintenance jobs that run each night. The web app uses Java
servlets and has PostgreSQL 7.0 as the back end.

When the user requests the first page, he gets a list of all the servers
with maintenance records in the database, and a drop down list of all
the dates of maintenance records. If the user chooses a date first, then
the app uses a prepared statement with the date contained in a
parameter, and this executes very quickly – no problems.

However, if the web page user does not choose a date, then the app uses
a correlated sub-query to grab only the current (latest) day’s
maintenance records. The query that is executed is:

select servername, databasename, message from messages o where
o.date_of_msg =

(select max(date_of_msg) from messages i where i.servername
= o.servername);

And this is a dog. It takes 15 – 20 minutes to execute the query (there
are about 200,000 rows in the table). I have an index on (servername,
date_of_msg), but it doesn’t seem to be used in this query.


Few basic checks..
- What does explain analyze says for the slow query?
- Have you vacuumed and analyzed recently?
- Have you done basic optimisations from default state? Check
http://www.varlena.com/varlena/Gener...bits/perf.html and
http://www.varlena.com/varlena/Gener...ed_conf_e.html

And 7.0 is way too old. If you can afford to upgrade, upgrade to 7.4.2.

HTH

Shridhar
---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

Nov 23 '05 #8
Howard, Steven (US - Tulsa) wrote:
I have created a web app that stores and displays all the messages from
my database maintenance jobs that run each night. The web app uses Java
servlets and has PostgreSQL 7.0 as the back end.

When the user requests the first page, he gets a list of all the servers
with maintenance records in the database, and a drop down list of all
the dates of maintenance records. If the user chooses a date first, then
the app uses a prepared statement with the date contained in a
parameter, and this executes very quickly – no problems.

However, if the web page user does not choose a date, then the app uses
a correlated sub-query to grab only the current (latest) day’s
maintenance records. The query that is executed is:

select servername, databasename, message from messages o where
o.date_of_msg =

(select max(date_of_msg) from messages i where i.servername
= o.servername);

And this is a dog. It takes 15 – 20 minutes to execute the query (there
are about 200,000 rows in the table). I have an index on (servername,
date_of_msg), but it doesn’t seem to be used in this query.


Few basic checks..
- What does explain analyze says for the slow query?
- Have you vacuumed and analyzed recently?
- Have you done basic optimisations from default state? Check
http://www.varlena.com/varlena/Gener...bits/perf.html and
http://www.varlena.com/varlena/Gener...ed_conf_e.html

And 7.0 is way too old. If you can afford to upgrade, upgrade to 7.4.2.

HTH

Shridhar
---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

Nov 23 '05 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: ajjvn | last post by:
I 'inherited' a group of SQL Server server class machines. They are true server technology but the disk sub-systems are lacking. There is one hot-swap backplane that all the drives share (with...
5
by: rc | last post by:
Hi We have a SQL server on Win2k. the physical size of the db is about 40G and the main table has approx 65m rows in it. At the moment the entire database is on one data file. The entire server...
4
by: Alex Callea | last post by:
Hi there, We have a web application handling thousands of requests per seconds reading sql server data which is heavily updated. We are generally experiencing no performance problems. On some...
13
by: Kurt Schroeder | last post by:
does compiling the code behind source using studio give and performance boost? this is a rephrase of an a question i posted yesterday. To be honist VS.net is becomming a pain to work with for...
4
by: James Radke | last post by:
Hello, I am creating an owner draw listbox for a windows application. It is all working, except the performance is significantly slower than the standard listbox. Basically what I have done is...
2
by: Jonesgj | last post by:
Hi, I have a test box which I would like to monitor CPU usage and run queue during the day. I don't want to buy any 3rd party tool, if I can do it easily, as I only need to monitor the box's...
19
by: Tom Jastrzebski | last post by:
Hello, I was just testing VB.Net on Framework.Net 2.0 performance when I run into the this problem. This trivial code attached below executed hundreds, if not thousand times faster in VB 6.0...
48
by: Alex Chudnovsky | last post by:
I have come across with what appears to be a significant performance bug in ..NET 2.0 ArrayList.Sort method when compared with Array.Sort on the same data. Same data on the same CPU gets sorted a...
4
by: muzu1232004 | last post by:
Can anyone explain me when we use correlated subqueries rather than nested subqueries. Do all the correlated subqueries can be written in nested subqueries form as well ? What are the major...
10
by: Rafael Cunha de Almeida | last post by:
Hi, I've found several sites on google telling me that I shouldn't use rand() % range+1 and I should, instead, use something like: lowest+int(range*rand()/(RAND_MAX + 1.0))
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.