473,405 Members | 2,279 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,405 software developers and data experts.

Optimize query: time of "single * IN(many)" > time of "many * IN(single)"

Hello!

Can anyone help me out with the following situation:
(a) a single query with 550 id's in the IN-clause resulting into 800+
seconds;
(b) 550 queries with a single id in the IN-clause resulting into overall
time of <60 seconds;
The table consists of 950.000 records, and the resultset consists of 205.000
records.
Why is there such an extreme difference in time?
And is there a way to reduce the difference in time?

More information about the situation is below.

Thank you for your help and time! =)

Postgres-version
7.3.1

The query is like:
SELECT a_id, b_id, score, c_id, d_id
FROM tbl_scores
WHERE a_id IN(...)
UNION
SELECT a_id, b_id, score, c_id, d_id
FROM tbl_scores_alike
WHERE a_id IN(...)

Definition of tables is like:
___Fields
CREATE TABLE public.tbl_scores (
id int4 DEFAULT nextval('"tbl_scores_id_seq"'::text) NOT NULL,
a_id int4,
b_id int4,
score int4, d_id int8,
CONSTRAINT tbl_scores_pkey UNIQUE (id),
dc date DEFAULT now(),
c_id int4,
INITIALLY IMMEDIATE
) WITHOUT OIDS;
___Index
CREATE INDEX tbl_scores_idx10 ON tbl_scores USING btree (a_id, b_id,
score, c_id, d_id);

Things that I tried to reduce the time of situation (a) - single * IN(many):
* vacuum of the database; hardly any improvement.
* selecting a single field in the resultset (a_id) instead of all fields;
hardly any improvement.
* only querying one table, skipping the UNION; hardly any improvement;
* ... what would you try?

__________________________________________________ _______________
MSN Zoeken helpt je om de gekste dingen te vinden! http://search.msn.nl
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

Nov 12 '05 #1
3 2236
Firstly you should always provide an explain from your query before
posting to this list. I think the problem is that in <7.4 PG did not use
indexes for IN queries. This has been fixed in 7.4.

On Thu, 2004-01-08 at 05:44, Paul Janssen wrote:
Hello!

Can anyone help me out with the following situation:
(a) a single query with 550 id's in the IN-clause resulting into 800+
seconds;
(b) 550 queries with a single id in the IN-clause resulting into overall
time of <60 seconds;
The table consists of 950.000 records, and the resultset consists of 205.000
records.
>> Why is there such an extreme difference in time?
>> And is there a way to reduce the difference in time?

More information about the situation is below.

Thank you for your help and time! =)

Postgres-version
7.3.1

The query is like:
SELECT a_id, b_id, score, c_id, d_id
FROM tbl_scores
WHERE a_id IN(...)
UNION
SELECT a_id, b_id, score, c_id, d_id
FROM tbl_scores_alike
WHERE a_id IN(...)

Definition of tables is like:
___Fields
CREATE TABLE public.tbl_scores (
id int4 DEFAULT nextval('"tbl_scores_id_seq"'::text) NOT NULL,
a_id int4,
b_id int4,
score int4, d_id int8,
CONSTRAINT tbl_scores_pkey UNIQUE (id),
dc date DEFAULT now(),
c_id int4,
INITIALLY IMMEDIATE
) WITHOUT OIDS;
___Index
CREATE INDEX tbl_scores_idx10 ON tbl_scores USING btree (a_id, b_id,
score, c_id, d_id);

Things that I tried to reduce the time of situation (a) - single * IN(many):
* vacuum of the database; hardly any improvement.
* selecting a single field in the resultset (a_id) instead of all fields;
hardly any improvement.
* only querying one table, skipping the UNION; hardly any improvement;
* ... what would you try?

__________________________________________________ _______________
MSN Zoeken helpt je om de gekste dingen te vinden! http://search.msn.nl
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ma*******@postgresql.org so that your
message can get through to the mailing list cleanly

Nov 12 '05 #2
On Thu, Jan 08, 2004 at 06:28:14AM -0500, Dave Smith wrote:
Firstly you should always provide an explain from your query before
posting to this list.
You mean "while posting", because he can't possible provide the explain
before having the means to do so, can he? :-)
I think the problem is that in <7.4 PG did not use
indexes for IN queries. This has been fixed in 7.4.


I think what was done is to optimize queries like IN (SELECT ...) but
there has not been improvement for IN (1,2,3, ... 550) like he
appears to be doing.

Maybe something to try is putting the IDs in a (temp?) table and using
the first form.

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"Cada quien es cada cual y baja las escaleras como quiere" (JMSerrat)

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ma*******@postgresql.org so that your
message can get through to the mailing list cleanly

Nov 12 '05 #3
"Paul Janssen" <po**********@hotmail.com> writes:
Can anyone help me out with the following situation:
(a) a single query with 550 id's in the IN-clause resulting into 800+
seconds;
(b) 550 queries with a single id in the IN-clause resulting into overall
time of <60 seconds;
The table consists of 950.000 records, and the resultset consists of 205.000
records. Why is there such an extreme difference in time?


Most likely the planner is opting not to use an indexscan in the first
case. Could we see EXPLAIN ANALYZE results for both cases? Also, try
"SET enable_seqscan TO OFF" and then repeat EXPLAIN ANALYZE for case (a).

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

Nov 12 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

68
by: Marco Bubke | last post by:
Hi I have read some mail on the dev mailing list about PEP 318 and find the new Syntax really ugly. def foo(x, y): pass I call this foo(1, 2), this isn't really intuitive to me! Also I...
0
by: Anthony Baxter | last post by:
To go along with the 2.4a3 release, here's an updated version of the decorator PEP. It describes the state of decorators as they are in 2.4a3. PEP: 318 Title: Decorators for Functions and...
0
by: Sean | last post by:
I received this error when a trigger attempted to perform an update to its own table with a where clause that did not guarantee a single row. I believe this results in the iterative firing of this...
23
by: raj | last post by:
I used to remember why c++ needed both ? Could somebody help me here ? For example class A{ f(); }; A* aa;
5
by: Mateusz Loskot | last post by:
Hi, I'd like to ask how XML parsers should handle attributes which consists of &quot; entity as value. I know XML allows to use both: single and double quotes as attribute value terminator. That's...
1
by: laredotornado | last post by:
Hi, I'm using PHP 4.4.4 on Apache 2 on Fedora Core 5. PHP was installed using Apache's apxs and the php library was installed to /usr/local/php. However, when I set my "error_reporting"...
2
by: Bob | last post by:
I'm running sql server ver 7.0 SP4. I have an access project (.adp) that runs a view which is nothing more than a select statement. Access locks up solid when I try to run this query - with NO...
169
by: JohnQ | last post by:
(The "C++ Grammer" thread in comp.lang.c++.moderated prompted this post). It would be more than a little bit nice if C++ was much "cleaner" (less complex) so that it wasn't a major world wide...
2
by: Chris Thomasson | last post by:
I was wondering if the 'SLINK_*' and 'SLIST_*' macros, which implement a simple singly-linked list, will produce _any_ possible undefined behavior: ____________________________ #include...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
0
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.