searching archives should be a weeeee bit faster ...

Marc G. Fournier

I spent this aft enabling debugging on the postmaster side, and determined
that there was a critical index missing on one of the tables ... there was
no index on the url.rec_id field, so a query that looks like:

SELECT rec_id, site_id, pop_rank FROM url WHERE rec_id IN
('31356','31364 ','32786','3278 7','32857','328 58','32871','32 872','32873','3 2874','32877',' 32878','32891', '32892','32894' ,'32922','32923 ','32928','3292 9','33003','330 04','33030','33 090','33091','3 5
146','37026','3 7028','39730',' 44317','44339', '44348','44359' ,'44413','44420 ','44438','4531 4','45322','458 61','47914','48 756','49587','5 0564','50567',' 51222','52330', '53029','53550' ,'53581','53898 '
,'53901','53932 ','54235','5494 4','54951','549 55','54965','54 973','55041','5 5295','55370',' 55374','55609', '55885','55889' ,'55898','55903 ','55905','5590 6','55907','564 42','57144','57 228','57230','5 7
233','58497','5 8499','58512',' 58519','58540', '58581','58585' ,'59281','59865 ','59874','6047 6','60478','604 82','60486','60 590','60827','6 1465','61531',' 61778','62272', '62585','62602' ,'62609','62731 '
,'63933','66733 ','66740','6674 3','66747','667 58','66760','66 763','66765','6 6768','66774',' 66777','68100', '68191','68195' ,'68212','68213 ','68257','6826 6','68288','682 95','68300','68 315','68332','6 8
335','68349','6 8354','69422',' 69435','69446', '69447','69987' ,'69991','69995 ','70003','7000 7','70010','700 84','70089','70 095','70468','7 0692','70699',' 70739','71022', '74531','74810' ,'77576','77736 '
,'78346','78608 ','79208','7929 1','79312','793 49','80034','80 038','82203','8 2852','84155',' 84456','85945', '86709','87055' ,'87061','87065 ','87078','8740 6','87413','875 18','89229','89 740','91262','9 4
205','94209','9 4215','94217',' 94940','96242', '96295','96303' ,'97442','97827 ','97833','9785 4','98262','988 45','98846','98 847','98848','9 8849','98850',' 98851','98852', '98854','98855' ,'98856','98857 '
,'98858','98859 ','98860','9923 9','100360','10 0407','100459', '100731','10084 0','102020','10 2026','103782', '106017','10852 3','109645','10 9654','109667', '109670','11121 3','111232','11 1233','111349', '
111351','111356 ','111620','116 673','116677',' 116734','117709 ','117733','118 075','118103',' 122444','126754 ','127945','127 949','128132',' 131062','131066 ','131067','131 068','131079',' 131084','131085 '
,'131094','1312 21','131222','1 31223','131225' ,'131226','1312 27','134426','1 35465','135466' ,'135468','1354 70','137034','1 37035','137536' ,'142057') ORDER BY rec_id;

was doing a sequence scan over 230k+ records in the URL file each time ...
created the index, and now I can actually get results ~32sec for the
single string (mvcc) that I've been using for testing, instead of it never
seeming to come back ...

hopefully that was the major part of it ...
---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

Nov 11 '05 #1

Subscribe Reply

1181

Sean Chittenden

Is "index" a stop word in the archives? It doesn't seem to return any
results when typing in "index" with anything (ex: index correlation).
FWIW, just a data point. -sc

--
Sean Chittenden

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

Nov 11 '05 #2

Marc G. Fournier

yes it is/was ...I've just removed it from the list of stop words, but am
not going to force a re-index of 250k URLs ;( So, any new index'ng will
pull in index as a valid search word, and when the current URLs
expire/re-index,they will get pulled in then also ...

On Thu, 4 Sep 2003, Sean Chittenden wrote:

Is "index" a stop word in the archives? It doesn't seem to return any
results when typing in "index" with anything (ex: index correlation).
FWIW, just a data point. -sc

--
Sean Chittenden

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ma*******@postg resql.org

Nov 11 '05 #3

Sean Chittenden

> yes it is/was ...I've just removed it from the list of stop words,

but am not going to force a re-index of 250k URLs ;( So, any new
index'ng will pull in index as a valid search word, and when the
current URLs expire/re-index,they will get pulled in then also ...

Hrm... if I search for "correlatio n," I get some results. If I search
for "index," I don't get anything (expected), but if I search for
"index correlation," I get nothing. Why isn't it returning the same
results as when I searched for "correlatio n?"

Just a scratch that I finally had to scratch and ask about...

-sc

--
Sean Chittenden

---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

Nov 11 '05 #4

Marc G. Fournier

On Thu, 4 Sep 2003, Sean Chittenden wrote:

yes it is/was ...I've just removed it from the list of stop words,
but am not going to force a re-index of 250k URLs ;( So, any new
index'ng will pull in index as a valid search word, and when the
current URLs expire/re-index,they will get pulled in then also ...

Hrm... if I search for "correlatio n," I get some results. If I search
for "index," I don't get anything (expected), but if I search for
"index correlation," I get nothing. Why isn't it returning the same
results as when I searched for "correlatio n?"

Just a scratch that I finally had to scratch and ask about...

my understanding of how it searches is it defaults to an 'and' ... so it
would have to find all docs that have both index (0) *and* correlation (n)
.... which would equal zero results ...
---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

Nov 11 '05 #5

Similar topics

1438

Searching Archives

by: Ernest Pianalto | last post by:

Hi, I just joined this list and would like to search the archives before I ask a = question. I there a way to search. Thanks=20 -- MySQL General Mailing List

MySQL Database

1758

FULLTEXT Searching

by: Ralph Guzman | last post by:

I have to do a catalog search through multiple tables and columns for product model number, description, and name. I realize that doing pattern matching with multiple LIKE statements is slow so I found that FULLTEXT searches is a better alternative. I have added a FULLTEXT index to the tables I'm searching, but I get an unkown error when I run my query: SELECT p2c.categories_id, p.products_id, pd.products_name, p.products_quantity,...

MySQL Database

1114

Searching on Two Keys with OR?

by: Joshua Spoerri | last post by:

Which version is targetted for optimization of OR searching on two keys, that is, "select * from sometable where f1 = 123 or f2 = 123", as described in http://www.mysql.com/doc/en/Searching_on_two_keys.html ? Thanks -- MySQL General Mailing List

MySQL Database

2478

Searching a table

by: Geoff Jones | last post by:

Hiya I have a DataTable containing thousands of records. Each record has a primary key field called "ID" and another field called "PRODUCT" I want to retrieve the rows that satisy the following criteria: I have a list of about 100 numbers which correspond to the ID field and also another 40 say numbers corresponding to the numbers in the PRODUCT field. I want to show the rows that correspond to both these criteria.

Visual Basic .NET

7020

case insensitive sorting & searching in oracle 10g

by: David Garamond | last post by:

in oracle 10g, you can issue: ALTER SESSION SET NLS_COMP = ansi; ALTER SESSION SET NLS_SORT = binary_ci; do you think this is an elegant solution for case insensitive sorting & searching? is there interest in seeing this in postgres? -- dave

PostgreSQL Database

1442

Searching python-list and MySQL

by: Richard Brosnahan | last post by:

I hate asking questions that have probably been answered before, but I have not found a way to conveniently search the archives of this mailing list. So... Can someone tell me how to search this mailing list, short of downloading every month's archive and searching manually? Or, can someone tell me how to easily interface with MySQL with

Python

1868

Searching 18,000 strings of an average size of 10Kb

by: Allan Ebdrup | last post by:

What would be the fastest way to search 18,000 strings of an average size of 10Kb, I can have all the strings in memory, should I simply do a instr on all of the strings? Or is there a faster way? I would like to have a kind of search like google where you can enter several words to search for, guess that calls for a regular expression "word1|word2|word3|...". Is there any kind of indexing tools available for this kind of thing, I have my...

C# / C Sharp

841

Searching this forum

by: =?Utf-8?B?Qm9iQWNoZ2lsbA==?= | last post by:

How can I search the archives of this forum? I know there are some old posts that are not showing up in my search. Thanks! Bob

Visual Basic .NET

1593

searching for article...

by: moswald | last post by:

I've been searching for a specific article for awhile now. About two years ago (maybe longer) there was an article in the C/C++ User's Journal that compared the performances of various stl containers. Unfortunately, while I remember the topic, I don't remember much else, so if anyone is inclined to help me find it, they'll have to do some detective work. I remember it was co-written by two people, one of which I *think* is Herb Sutter...

C / C++

8395

What is ONU?

by: marktang | last post by:

ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...

General

8826

Problem With Comparison Operator <=> in G++

by: Oralloy | last post by:

Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...

C / C++

8503

The easy way to turn off automatic updates for Windows 10/11

by: Hystou | last post by:

Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...

Windows Server

8605

Discussion: How does Zigbee compare with other wireless protocols in smart home applications?

by: tracyyun | last post by:

Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...

General

7330

AI Job Threat for Devs

by: agi2029 | last post by:

Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...

Career Advice

5632

Couldn’t get equations in html when convert word .docx file to html file in C#.

by: conductexam | last post by:

I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...

C# / C Sharp

4155

Trying to create a lan-to-lan vpn between two differents networks

by: TSSRALBI | last post by:

Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...

Networking - Hardware / Configuration

4306

Windows Forms - .Net 8.0

by: adsilva | last post by:

A Windows Forms form does not have the event Unload, like VB6. What one acts like?

Visual Basic .NET

2726

transfer the data from one system to another through ip address

by: 6302768590 | last post by:

Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

C# / C Sharp