473,770 Members | 1,642 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Erratic performance for a specific query in MySQL

I have two job queues doing complimentary work and writing their output
once a minute into a file. The files for each minute have to processed
in pairs - one from the first job queue and one from the second and the
results are merged and delivered to the final client. The filename and
other details of these files go in to a MySQL database - a different
table for each job queue.
The merging process has to read the two files for that minute from the
two tables joining on a file id. Here is MySQL schema I am using:

CREATE TABLE `Job_Queue_1` (
`FileID` varchar(64) NOT NULL default '',
`File_Path` text NOT NULL,
`File_TimeStamp ` bigint(12) NOT NULL default '0',
`Process_Status ` char(1) NOT NULL default '',
PRIMARY KEY (`FileID`),
KEY `Process_Status ` (`Process_Statu s`,`File_TimeSt amp`)
) ENGINE=InnoDB
CREATE TABLE `Job_Queue_2` (
`FileID` varchar(64) NOT NULL default '',
`File_Path` text NOT NULL,
`File_TimeStamp ` bigint(12) NOT NULL default '0',
`Process_Status ` char(1) NOT NULL default '',
PRIMARY KEY (`FileID`),
KEY `Process_Status ` (`Process_Statu s`,`File_TimeSt amp`)
) ENGINE=InnoDB

File_TimeStamp is an integer representing the time as YYYYMMDDhhmm at a
minute granularity. Process_Status is 'N' for a new file and 'F' for
that which has already been merged with its pair. FileID is the join
condition.

The query used by the application to pick up a pair of files is:

select a.FileID, a.File_Path, b.FileID, b.File_Path
from Job_Queue_1 a, Job_Queue_2 b
where a.FileID = b.FileID
and a.Process_Statu s = 'N' and b.Process_Statu s = 'N'
order by a.File_TimeStam p limit 1 ;

This runs very well for upto two million rows in each table. But I was
doing some stress testing with 4 million rows in each table, 2.5 with
Process_Status as 'F' and 1.5 with Process_Status as 'N'. The query was
hitting the database almost 50 times each second. I noticed that for 5
minutes the query would be very fast - about 20 milliseconds per query
and the explain statement on MySQL showed "Using where". And randomly
for 5 minutes the same query would drastically degrade in performance
and take about 30 seconds and the explain statement would show "Using
where, using temporary, using filesort". And then it would go back to
being fast.

Any clue of what might be happening here? The server is running on
RHEL3 with dual AMD CPUs and and the innodb_buffer_p ool_size is 1.8 GB.

Thanks in advance.

Apr 23 '06 #1
0 1200

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
4682
by: sandy | last post by:
Hi All, I am a newbie to MySQL and Python. At the first place, I would like to know what are the general performance issues (if any) of using MySQL with Python. By performance, I wanted to know how will the speed be, what is the memory overhead involved, etc during database specific operations (retrieval, update, insert, etc) when MySQL is used with Python.
3
949
by: Andy Tran | last post by:
I built a system using mysql innodb to archive SMS messages but the innodb databases are not keeping up with the number of SMS messages coming in. I'm looking for performance of 200 msgs/sec where 1 msg is 1 database row. I'm running on Red Linux: 2.4.20-8bigmem #1 SMP Thu Mar 13 17:32:29 EST 2003 i686 i686 i386 GNU/Linux The machine has dual CPU and 2G of RAM.
0
2488
by: Rajesh Kapur | last post by:
I have a master slave configuration on linux machines running MySQL 4.0.21. Once every hour, a process deletes about 9000 rows and re-inserts fresh data on the master. The master process completes processing under a minute. The updates arrive on the slave within seconds. However, the slave takes about 15-20 minutes to post the same updates. Both master and slave have InnoDB tables. The master and the slave are almost identical in capacity...
11
6975
by: Dave [Hawk-Systems] | last post by:
have the table "numbercheck" Attribute | Type | Modifier -----------+------------+---------- svcnumber | integer | not null svcqual | varchar(9) | svcequip | char(1) | svctroub | varchar(6) | svcrate | varchar(4) | svcclass | char(1) | trailer | varchar(3) |
57
25525
by: Bing Wu | last post by:
Hi all, I am running a database containing large datasets: frames: 20 thousand rows, coordinates: 170 million row. The database has been implemented with: IBM DB2 v8.1
0
3227
by: ZMan | last post by:
Scenario: This is about debugging server side scripts that make calls to middle-tier business DLLs. The server side scripts are legacy ASP 3.0 pages, and the DLLs are managed DLLs converted/developed with VB.NET. What I want from debugging is to be able to step into the methods in the DLLs called from ASP scripts using Visual Studio .NET. Background: For typical script debugging issues, you can read and follow the two documents on...
1
2394
by: murray | last post by:
Large table performance problem with MYSQL performance with ASP.NET I am doing some benchmarking of an application using MYSQL under Windows and I confused about the results. Environment Emachine 6417 with amd3200+ 64 bit processor 3.3GB ram mysql 4.1 with odbc 3.51 MYISAM windows 2003 server std edition
0
1348
by: OctoTools | last post by:
Large table performance problem with MYSQL performance with ASP.NET I am doing some benchmarking of an application using MYSQL under Windows and I confused about the results. Environment Emachine 6417 with amd3200+ 64 bit processor 3.3GB ram mysql 4.1 with odbc 3.51 MYISAM windows 2003 server std edition
0
2166
by: jllanten | last post by:
I will appreciate any help you can provide me. In the company where i work we have a project which creates about 4-5M records daily of stats. We're currently storing this data in a db named 'summary' which contains all the stats data for up to 15 days. After that we begin to move the data in a daily basis to a single 'archived like' unique table in another fast performance mysql instance. Why we do this ? to respect the 80%-20% rule: 80% of...
0
9595
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9432
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10059
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
9873
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
8891
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
5313
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5454
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3974
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
3578
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.