Hi,
I am using MySQL for a table which will have 100M+ records, avg length
of records being 130 bytes.
When the number of records reach approx. 25M (and the file size close
to 4GB), the rate of inserts falls drastically from 800 per second to
30-40 per second.
Details:
* MySQL 3.23.58 on Fedora Core 3
* Table has 4 indices.
* I have got rid of 4GB file size problem with MAX_ROWS=100000 0000.
* File system : ext3 on single disk. ext3 could create 10G file
without much trouble. So I am convinced that ext3 is not the
bottleneck.
* Tried using InnoDB engine but it also doesn't meet the requirements.
Requirements of database:
* A single table in the database with 100M+ rows, each of size 130
bytes (approx).
* 500-600 inserts per second.
* 200 selects and 200 updates per second. (These statements will
affect only one row)
* 3-5 select statements per minute which can return 10k to 500k
records.
* No foreign keys/ACID transaction requirements.
* Fast recovery in case of crash.
Questions:
* Does MySQL performance drops when the table grows beyond 4G?
* Which are most important server variables which need to fine tuned?
Currently I am using only key_buffer = 512M.
Answers/advice/opinions/pointers much appreciated.
Thanks,
--shashi 2 2323
Shashikant Kore wrote: Hi,
I am using MySQL for a table which will have 100M+ records, avg length of records being 130 bytes. When the number of records reach approx. 25M (and the file size close to 4GB), the rate of inserts falls drastically from 800 per second to 30-40 per second.
You might try using MERGE tables to spread the data over multiple
physical files that are each less than 4G. See: http://dev.mysql.com/doc/mysql/en/me...ge-engine.html
Details: * MySQL 3.23.58 on Fedora Core 3
Why such an old version of MySQL? Have you tried using a more recent
version to see if it makes a difference?
* I have got rid of 4GB file size problem with MAX_ROWS=100000 0000. * File system : ext3 on single disk. ext3 could create 10G file without much trouble. So I am convinced that ext3 is not the bottleneck.
Here's a posting from a person who says that switching to xfs or
reiserfs, instead of ext3 filesystem helped get better performance from
MySQL: http://lists.mysql.com/benchmarks/126
Also, have you read the following web page: http://dev.mysql.com/doc/mysql/en/insert-speed.html
There are several good suggestions for speeding up inserts, suchas using
LOAD DATA INFILE.
Regards,
Bill K.
Thanks for the response, Bill.
Bill Karwin wrote: You might try using MERGE tables to spread the data over multiple physical files that are each less than 4G. See: http://dev.mysql.com/doc/mysql/en/me...ge-engine.html
MERGE tables proved to be of great help. The inserts are now faster. I
am not really looking into option of LOAD DATA INFILE as the inserts
are not batched. But if inserts get slower in future, I would
definitely look into this matter. Details: * MySQL 3.23.58 on Fedora Core 3
Why such an old version of MySQL? Have you tried using a more recent version to see if it makes a difference?
I cannot say confidently that performance has improved a lot. I need to
go through the change log to figure out the new tunable parameters in
4.1 which will improve the performance. * I have got rid of 4GB file size problem with MAX_ROWS=100000 0000. * File system : ext3 on single disk. ext3 could create 10G file without much trouble. So I am convinced that ext3 is not the bottleneck.
Here's a posting from a person who says that switching to xfs or reiserfs, instead of ext3 filesystem helped get better performance from MySQL: http://lists.mysql.com/benchmarks/126
Will look into the option of using XFS/JFS.
Also, have you read the following web page: http://dev.mysql.com/doc/mysql/en/insert-speed.html There are several good suggestions for speeding up inserts, suchas using LOAD DATA INFILE.
Yeah, I had scanned through this. Since I run very basic queries, most
of the suggestions were not applicable.
Meanwhile, I also explored option of using BDB engine, but it didn't
work out.
If anyone has worked with MERGE tables, please give me some insights
into improving its performance. The manual doesn't speak much on this.
Thanks,
--shashi
Regards, Bill K. This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Matt Liverance |
last post by:
I REALLY dont want to switch to oracle :( but I cant get these tables
working any faster.
I've got 2 dedicated servers, each with a slave, all run 32gig 15k rpm
raid 5 on u320 perc raid cards, dell 2600/4600's with single channel
backplanes (new ones will have dual channel)
All have 2 gig of ram, but I've never seen mysql use more than 300mb
of ram.
|
by: wlcna |
last post by:
mysql v4.0.16:
I had been using mysql with innodb and thought that was fine, until i used
it for something requiring a few - perhaps slightly involved - joins, and
have now seen the performance become totally unacceptable.
I have a query that takes over 35 seconds using mysql and innodb, for
reasons that are completely a mystery to me, in a result set consisting of
only a handful of items.
|
by: Michi |
last post by:
I was wondering what the best solution is for making large numbers of
TEXT (or BLOB?) fields searchable. For example, if I have a forum,
what is the best way to be able to search for specific words? How
about exact phrases?
I saw a solution where all words are preindexed in a "dictionary" like
table and then another table stores the word matches. That seems
really fast, but it has two major problems: 1) it can't do exact
matches, and 2)...
|
by: DJJ |
last post by:
I am using the MySQL ODBC 3.51 driver to link three relatively small MySQL
tables to a Microsoft Access 2003 database. I am finding that the data from
the MySQL tables takes a hell of a long time to load making any kind linkage
with my Access data virtually useless.
I have the MySQL driver setup in as a USER DSN. The MySQL data is sitting
out on a server and the Access database is running locally. The network
connection is very...
|
by: Mairhtin O'Feannag |
last post by:
Hello,
I have a client (customer) who asked the question : "Why would I buy and
use UDB, when MySql is free?"
I had to say I was stunned. I have no experience with MySql, so I was
left sort of stammering and sputtering, and managed to pull out something
I heard a couple of years back - that there was no real transaction
safety in MySql. In flight transactions could be lost.
| |
by: jlee |
last post by:
I'm pretty much a newbie on mysql, and I need some help.
I am running mysql Ver 12.22 Distrib 4.0.24, for portbld-freebsd5.4
(i386) on a server hosting an active website.
The site's developer uses his own php shopping cart to receive customer
orders.
The configuration was done via cPanel with no external modifications -
which produced no protests when built, ran and connected with no
|
by: Good Man |
last post by:
Hi there
I've noticed some very weird things happening with my current MySQL setup
on my XP Laptop, a development machine.
For a while, I have been trying to get the MySQL cache to work. Despite
entering the required lines to "my.ini" (the new my.cnf) through notepad
AND MySQL Administrator, the cache does not work.
So, today I took a peek at the 'Health' tab in MySQL Administrator.
|
by: daniel |
last post by:
I have the following scenario.
A mysql database running 3 databases. It is version 5.0.27 on Windows
XP Prof.. All innodb databases. The one database is particularly large
(7.8GB of data)...pretty much held in 1 table....there are probabably
30 tables in the rest of the databases....combined they probably take
up 200MB. The machine is pretty well spec'ed AMD X2 4600+, 2GB RAM,
SATA RAID1. Normally the services that use the databases are...
|
by: Coldfire |
last post by:
Since i cannot show the differences in a two-column like table. I am first putting
MS SQL Server 2005 and then MySQL 5.x.
MS SQL Server 2005
Brief Overview
- SQL Server is a full-fledged database system developed specifically for large enterprise databases. All advanced features of a relational database are fully implemented.
- Once you purchase the product, you are only limited to the Sybase-derived engine.
|
by: Atli |
last post by:
This is an easy to digest 12 step guide on basics of using MySQL. It's a great refresher for those who need it and it work's great for first time MySQL users.
Anyone should be able to get through this without much trouble. Programming knowledge is not required.
Index
What is SQL?
Why MySQL?
Installing MySQL.
Using the MySQL command line interface
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
| |
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
| |
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
|
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |