How should I set up this database?

[Ondernemer]

Imagine 25.000.000 records in a database with a table like this:

TABLE SUBJECTS

table_id (INT)
subject (VARCHAR(200))
date_added (TIMESTAMP)

The SUBJECT field is a long string. Something like these:

Example 1:
"Johnny posts 'PHP Examples' version 2'

Example 2:
"Christmas Melodies from Japan (2001)"

QUESTION:

The user should be able to search the database on the SUBJECT field and
should be able to use partial keywords like:

Keyword examples:

- lodies (finds Melodies)
- Examp (finds Examples)
- John (finds Johnny)

Right now, using WHERE LIKE takes 40 seconds since it checks every single
record. I can't use indexes either. "Of course" you might say.

But what I want to know is... what is the workaround for this? How can I
create a database which allows people to do wilcard searches while keeping
my queries lightning fast?

Other websites and applications have this... so there must be a brilliant
"trick" or workaround for this.

Any ideas?

[Aggro]

Ondernemer wrote:

Other websites and applications have this... so there must be a brilliant
"trick" or workaround for this.

Basic idea is to create a table where is a list of all words used in
your text. Then create another table which links these words to the
table where the actual texts are located. After searching correct words
from the word table, rest of the queries can use indexes so it should be
fast. Even if you have to check all rows in the word table, you will
still propably have less than 25 000 000 rows there.

I just wrote some time ago a pretty good example about this subject:

http://groups.google.com/groups?selm...0read3.inet.fi

I should warn you that I have not studied this method much, so there
might be better ways to implement the same thing. But the idea behind
the method should be clear enough.

[Ondernemer]

"Aggro" wrote:

http://groups.google.com/groups?selm...0read3.inet.fi

Brilliant. I actually was thinking the same thing, but was afraid I was
thinking in an amateur way and there was a more efficient command/way of
doing this. :-)

Thanks m8.

[Rich R]

"Aggro" <sp**********@yahoo.com> wrote in message
news:2E***************@read3.inet.fi...

Ondernemer wrote:

Other websites and applications have this... so there must be a brilliant "trick" or workaround for this.

Basic idea is to create a table where is a list of all words used in
your text. Then create another table which links these words to the
table where the actual texts are located. After searching correct words
from the word table, rest of the queries can use indexes so it should be
fast. Even if you have to check all rows in the word table, you will
still propably have less than 25 000 000 rows there.

I just wrote some time ago a pretty good example about this subject:

http://groups.google.com/groups?selm...0read3.inet.fi

I should warn you that I have not studied this method much, so there
might be better ways to implement the same thing. But the idea behind
the method should be clear enough.

"And" will not work using a table of keywords. "Or" will work but you will
get everything. The best way to do this is to create a column with all
keywords (this must be of type MyIsam) then do a full text search.

Rich

[Norman Peelman]

"Ondernemer" <no**@email.com> wrote in message
news:10*************@corp.supernews.com...

Imagine 25.000.000 records in a database with a table like this:

TABLE SUBJECTS

table_id (INT)
subject (VARCHAR(200))
date_added (TIMESTAMP)

The SUBJECT field is a long string. Something like these:

Example 1:
"Johnny posts 'PHP Examples' version 2'

Example 2:
"Christmas Melodies from Japan (2001)"

QUESTION:

The user should be able to search the database on the SUBJECT field and
should be able to use partial keywords like:

Keyword examples:

- lodies (finds Melodies)
- Examp (finds Examples)
- John (finds Johnny)

Right now, using WHERE LIKE takes 40 seconds since it checks every single
record. I can't use indexes either. "Of course" you might say.

But what I want to know is... what is the workaround for this? How can I
create a database which allows people to do wilcard searches while keeping
my queries lightning fast?

Other websites and applications have this... so there must be a brilliant
"trick" or workaround for this.

Any ideas?

One quick idea would be to place a LIMIT on the returned rows (like
search engines). Look up the use of LIMIT in the mysql docs...
http://dev.mysql.com/doc/mysql/en/SELECT.html

From the page:

The LIMIT clause can be used to constrain the number of rows returned by the
SELECT statement. LIMIT takes one or two numeric arguments, which must be
integer constants. With two arguments, the first argument specifies the
offset of the first row to return, and the second specifies the maximum
number of rows to return. The offset of the initial row is 0 (not 1):

mysql> SELECT * FROM table LIMIT 5,10; # Retrieve rows 6-15
For compatibility with PostgreSQL, MySQL also supports the LIMIT row_count
OFFSET offset syntax. To retrieve all rows from a certain offset up to the
end of the result set, you can use some large number for the second
parameter. This statement retrieves all rows from the 96th row to the last:

mysql> SELECT * FROM table LIMIT 95,18446744073709551615;
With one argument, the value specifies the number of rows to return from the
beginning of the result set:

mysql> SELECT * FROM table LIMIT 5; # Retrieve first 5 rows
--- used from the MySQL web site---