473,322 Members | 1,719 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,322 software developers and data experts.

Select Query

43
Hi,
Expand|Select|Wrap|Line Numbers
  1. select c19,name,c5,count(*) as count,sum(c13) as cost  from TableA where c1 like '%' and c5 like '%' and  name like 'bravo' and c19  between  '2009-01-01 00:00:00' and '2012-01-01 00:00:00' group by date(c19),name,c5 limit 10000 offset 0 
i am using mysql version 4.1 and there is a table having 2.5 million
records , the table structure is given below.. if i run above select
query time taken is 19.33 sec
I have some Questions:
1. Is this acceptable time in terms of mysql? (For me its not)
2. How can i reduce this time so that to increase performance?

I have tried multicolumn indexing but it didnt helped.currently i am using
indexing over name . The time is proportional to number of records fetched .

Explain results:

----------------------------------------------------------------------------------------------------------------------------------------------------
id | select_type | table | type | possible_keys | key | key_len | ref | rows |Extra
---------------------------------------------------------------------------------------------------------------------------------------------------
1 | SIMPLE | TableA | range| name | name | 15 | NULL | 256903 | Using where; Using temporary; Using filesort

----------------------------------------------------------------------------------------------------------------------------------------------------
Table structure:

Expand|Select|Wrap|Line Numbers
  1. CREATE TABLE TableA (
  2.   msg_id varchar(20) NOT NULL default '',
  3.   name varchar(15) NOT NULL default '',
  4.   c1 varchar(50) NOT NULL default '',
  5.   c2 varchar(15) default NULL,
  6.   c3 varchar(15) default NULL,
  7.   c4 tinyint(3) unsigned NOT NULL default '0',
  8.   c5 tinyint(3) unsigned NOT NULL default '0',
  9.   c6 varchar(15) default NULL,
  10.   c7 tinyint(3) unsigned NOT NULL default '0',
  11.   c8 tinyint(3) unsigned NOT NULL default '0',
  12.   c9 tinyint(3) unsigned NOT NULL default '0',
  13.   c10 tinyint(3) unsigned NOT NULL default '0',
  14.   c11 tinyint(3) unsigned NOT NULL default '0',
  15.   c12 tinyint(3) unsigned NOT NULL default '0',
  16.   c13 decimal(12,5) NOT NULL default '0.00000',
  17.   c14 varchar(15) NOT NULL default '',
  18.   c15 int(5) unsigned default '0',
  19.   c16 int(5) unsigned default '0',
  20.   c17 varchar(20) NOT NULL default '',
  21.   c18 tinyint(3) unsigned default '0',
  22.   c19 varchar(50) NOT NULL default '0000-00-00 00:00:00',
  23.   c20 datetime NOT NULL default '0000-00-00 00:00:00',
  24.   c21 varchar(15) default 'PENDING',
  25.   c22 int(3) unsigned default '0',
  26.   c23 varchar(5) NOT NULL default 'false',
  27.   PRIMARY KEY  (msg_id),
  28.   KEY uname (username)
  29.  
  30. ) ENGINE=MyISAM DEFAULT CHARSET=latin1;
  31.  
  32.  
please suggest me how i can reduce the query time.
Thanks In Advance.....
Mar 4 '09 #1
6 2025
mwasif
802 Expert 512MB
I can not see any index. c19 should be of type datetime not varchar. Index on c19 will be helpful.
Mar 4 '09 #2
bravo
43
Thanks for ur quick response... sorry my mistake .. i changed the fields name before submitting post... the c19 is not varchar its same as c20 and the index is on name not username... i have tried to use c19 as index already but same result.
Mar 4 '09 #3
bravo
43
Now i dropped index name and added new index on c19 ... now the same query giving the result in 39 sec approx.. time got doubled now.... it didnt worked...any thing else i can try? Also have one question .. what should be the acceptable time for a query like above in same conditions...in mysql?
Mar 4 '09 #4
chaarmann
785 Expert 512MB
@bravo
Multicolumn index has no effect, because your query only references to one index! It would make sense and speed up the query if you rewrite your query to reference all the indexed columns.

How to speed up performance:
1.) rewrite your query:
You shouldn't use "like" everywhere! "like" is very slow. Instead "c1 like '%'" write "c1 is not null". Instead "name like 'bravo' " write "name='bravo'". I hope you have an index on "name" column! You can delete "name" in your group-by statement. Put the most restricting index in front, that means place "name='bravo'" right after the where-clause.
2.) use a shadow table: insert a trigger that writes to a second, new table (the shadow table) at every table update. The trigger should already include the sum and group-by clause. The shadow table should only have the needed columns "c19,name,c5,count,cost", so that it is very small and can be memory-mapped. Of course the shadow table should also have indexed columns (for example on "name")
Then you inquire only the shadow table to get results back within a few milliseconds. It's so fast because the shadow table only contains a few hundred entries in opposition to million entries in the original table.
Mar 4 '09 #5
chaarmann
785 Expert 512MB
I just noticed that you are using mySql 4.1, and it doesn't yet have triggers.

So upgrade to mySql5.0 or later first.
Or produce your shadow-table in another way (for example in a nightly batch-job, or as an additional SQL in your source code at every place where you modify the big table)
Mar 4 '09 #6
chaarmann
785 Expert 512MB
@bravo
Less than 1 second if a user (for example on the webpage) waits for the result.
You can always get it that fast by writing a good inquiry and having a good database design. I know that sounds not possible, but I have doen it many times. Just imagine how fast google delivers the search results from billion of records (stored webpages)!

good inquiry:
the user only needs to see a few data to make a decision, so don't return unnecessary data that you don't display or the user doesn't need to see. It's probably already enough for the user to see only name and date. Then he can click on a record from the list to see its additional info (all the cn, n=1..23 columns)

good database design:
Don't put all your data inside one big table! Split it up into 2 tables, one with the most used data, the other with the seldom used. That means make one with columns c19,name,c5,count,cost only and reference (foreign key) to a second table that holds all the other cn, n=1..23 columns
Mar 4 '09 #7

Sign in to post your reply or Sign up for a free account.

Similar topics

1
by: Phil Powell | last post by:
Here is the scope of what I need to do; want: enrollment_year allowed (even if null) all of ica criteria:
21
by: John Fabiani | last post by:
Hi, I'm a newbie and I'm attempting to learn howto create a select statement. When I use >>> string1='18 Tadlock Place' >>> cursor.execute("SELECT * FROM mytest where address = %s",string1) All...
4
by: jimh | last post by:
I'm not a SQL expert. I want to be able to write a stored procedure that will return 'people who bought this product also bought this...'. I have a user table that links to a transaction table...
3
by: Tcs | last post by:
My backend is DB2 on our AS/400. While I do HAVE DB2 PE for my PC, I haven't loaded it yet. I'm still using MS Access. And no, I don't believe this is an Access question. (But who knows? I...
1
by: Andrew McNab | last post by:
Hi folks, I have a problem with an MS Access SQL query which is being used in an Access Report, and am wondering if anyone can help. Basically, my query (shown below) gets some records from a...
2
by: Pavel Stehule | last post by:
Hello, Pg make query 1. and 2. very fast (use index), but for query 3. dont use index. I can solve its using select union, but I readed so pg 7.5 don't problem with OR operator. I use cvs pg. I...
2
by: marco | last post by:
Dear List, as it seems, MS SQL as used in Access does not allow a select INTO within a UNION query. Also, it seems that a UNION query can not be used as a subquery. Maybe my (simplified)...
6
by: jjturon | last post by:
Can anyone help me?? I am trying to pass a Select Query variable to a table using Dlookup and return the value to same select query but to another field. Ex. SalesManID ...
0
by: djflow | last post by:
Hi! II was wondering if you can help me with SQL query.. Below 7 separated select query works fine(only when they are retrieved separately) But I want to combined them together and so that i...
22
by: Rickster66 | last post by:
As Instructed this is a new thread regarding my original post: "Select Only 10 Columns Going Back" I'm sorry for the late response. I've been gathering up information and carefully with as much...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.