I have a test database that I have built in a 3 partition (and 3 node)
environment. I have defined all the tables so they have the same
partition key. The tables (7 of them) form a hierarchical
arrangement. The data is all bogus, so I using the generate_unique
function to come up with a partitioning key for each record. This is
all done through a stored proc with will insert about 90 records in
the heirarchy.
The behavior that I am seeing is when the proc generates a
partitioning key that will put (all) the data on the local partition
it runs quickly, (.2 second or less), when the partitioning key
resolves to another partition it takes 2 to 4 seconds. This is true
no matter which node I actually run the stored procedure on.
I know that there will be internodal communication and inserting the
data on one of those nodes will be slower than on the local partition,
but this seems excessive. Has anyone dealt with this kind of problem?
I have exhausted my ideas of where to look to address this
communication issue.
Any ideas?
thanks
Jack 8 3701
"Jack" <jb****@marketr on.com> wrote in message
news:13******** *************** ***@posting.goo gle.com... I have a test database that I have built in a 3 partition (and 3 node) environment. I have defined all the tables so they have the same partition key. The tables (7 of them) form a hierarchical arrangement. The data is all bogus, so I using the generate_unique function to come up with a partitioning key for each record. This is all done through a stored proc with will insert about 90 records in the heirarchy.
The behavior that I am seeing is when the proc generates a partitioning key that will put (all) the data on the local partition it runs quickly, (.2 second or less), when the partitioning key resolves to another partition it takes 2 to 4 seconds. This is true no matter which node I actually run the stored procedure on.
I know that there will be internodal communication and inserting the data on one of those nodes will be slower than on the local partition, but this seems excessive. Has anyone dealt with this kind of problem? I have exhausted my ideas of where to look to address this communication issue.
Any ideas?
thanks
Jack
Can you explain exactly how many inserts are being done and what the syntax
is?
Can you explain your network configuration (the hardware used to link the
nodes together)?
Insert into a partitioned db will run slower than a single-partition db.
But not so slow like your testing result. I will not suggest define the
partitioning key on the function.
BTW, if you only want to improve INSERT performance, you can check with
Buffered Insert and local bypass feature.
Jack wrote: I have a test database that I have built in a 3 partition (and 3 node) environment. I have defined all the tables so they have the same partition key. The tables (7 of them) form a hierarchical arrangement. The data is all bogus, so I using the generate_unique function to come up with a partitioning key for each record. This is all done through a stored proc with will insert about 90 records in the heirarchy.
The behavior that I am seeing is when the proc generates a partitioning key that will put (all) the data on the local partition it runs quickly, (.2 second or less), when the partitioning key resolves to another partition it takes 2 to 4 seconds. This is true no matter which node I actually run the stored procedure on.
I know that there will be internodal communication and inserting the data on one of those nodes will be slower than on the local partition, but this seems excessive. Has anyone dealt with this kind of problem? I have exhausted my ideas of where to look to address this communication issue.
Any ideas?
thanks
Jack
I've looked at ISERT BUF, it appears to have no impact on my tests.
Local Bypass is really not an option because the partition key is
created at run time in the Stored Procedure.
The database is seven tables in a hiearchical arrangement. All tables
are partitioned on the key of the parent table (T1). The structure
looks something like this:
T1
--------------
T2 T3
-----------
T4 T5
-----
T6 T7
The SP just does a series of inserts with do loops
The keys for each table is generated in the SP via a Generate_Unique
function.
The 3 nodes are dual IA32 boxes running RH AS 2.1. They are connected
via a public 100 mb network and a private 1gb network. The private
network is defined as the switching network for DB2. This private
network is only for DB2 cluster. If have verified by FTP tests that
the networks are running at expected speeds.
I believe the issue tied to the performance of communication between
the nodes.
thanks
Jack
Fan Ruo Xin <fa*****@sbcglo bal.net> wrote in message news:<3F******* ********@sbcglo bal.net>... Insert into a partitioned db will run slower than a single-partition db. But not so slow like your testing result. I will not suggest define the partitioning key on the function. BTW, if you only want to improve INSERT performance, you can check with Buffered Insert and local bypass feature.
Jack wrote:
I have a test database that I have built in a 3 partition (and 3 node) environment. I have defined all the tables so they have the same partition key. The tables (7 of them) form a hierarchical arrangement. The data is all bogus, so I using the generate_unique function to come up with a partitioning key for each record. This is all done through a stored proc with will insert about 90 records in the heirarchy.
The behavior that I am seeing is when the proc generates a partitioning key that will put (all) the data on the local partition it runs quickly, (.2 second or less), when the partitioning key resolves to another partition it takes 2 to 4 seconds. This is true no matter which node I actually run the stored procedure on.
I know that there will be internodal communication and inserting the data on one of those nodes will be slower than on the local partition, but this seems excessive. Has anyone dealt with this kind of problem? I have exhausted my ideas of where to look to address this communication issue.
Any ideas?
thanks
Jack
Hi. jb****@marketro n.com (Jack) wrote in message news:<13******* *************** ****@posting.go ogle.com>... I've looked at ISERT BUF, it appears to have no impact on my tests. Local Bypass is really not an option because the partition key is created at run time in the Stored Procedure.
There's a statement in HTML document that came with DB2 V7.1:
Local bypass is enabled automatically whenever possible, but you can
increase its use by routing transactions to the partition containing
the data for that transactions.
Does this statement imply that there's another secret DB2 configuration
which is described in somewhere deep in the manual and only known to
someone who's read through each and every manual page :), or does it
simply mean that local bypass is only available when the target partition
happens to reside in the same node as the controlling partition?
Any ideas?
Thanks in advance. y0******@yahoo. co.jp (YONETANI Tomokazu) wrote in message news:<b2******* *************** ****@posting.go ogle.com>... Hi.
jb****@marketro n.com (Jack) wrote in message news:<13******* *************** ****@posting.go ogle.com>... I've looked at ISERT BUF, it appears to have no impact on my tests. Local Bypass is really not an option because the partition key is created at run time in the Stored Procedure.
There's a statement in HTML document that came with DB2 V7.1: Local bypass is enabled automatically whenever possible, but you can increase its use by routing transactions to the partition containing the data for that transactions.
Does this statement imply that there's another secret DB2 configuration which is described in somewhere deep in the manual and only known to someone who's read through each and every manual page :), or does it simply mean that local bypass is only available when the target partition happens to reside in the same node as the controlling partition? Any ideas?
Sorry, the previous message was unclear about what "statement" I was
referring to: my question was how "Local bypass is enabled automatically
whenever possible", in other words, how or whether it's possible to
exploit local bypass feature without intervention of user code(by calling
sqlugrpn() API to determine the appropriate node, for example).
Jack,
One thought, are you commiting each insert, or all as a single transaction?
- I'm thinking the commit overhead between partitions could
be causing the large difference?
Paul.
YONETANI Tomokazu wrote: y0******@yahoo. co.jp (YONETANI Tomokazu) wrote in message news:<b2******* *************** ****@posting.go ogle.com>... Hi.
jb****@marketro n.com (Jack) wrote in message news:<13******* *************** ****@posting.go ogle.com>... I've looked at ISERT BUF, it appears to have no impact on my tests. Local Bypass is really not an option because the partition key is created at run time in the Stored Procedure.
There's a statement in HTML document that came with DB2 V7.1: Local bypass is enabled automatically whenever possible, but you can increase its use by routing transactions to the partition containing the data for that transactions.
Does this statement imply that there's another secret DB2 configuration which is described in somewhere deep in the manual and only known to someone who's read through each and every manual page :), or does it simply mean that local bypass is only available when the target partition happens to reside in the same node as the controlling partition? Any ideas?
Sorry, the previous message was unclear about what "statement" I was referring to: my question was how "Local bypass is enabled automatically whenever possible", in other words, how or whether it's possible to exploit local bypass feature without intervention of user code(by calling sqlugrpn() API to determine the appropriate node, for example).
Local Bypass is very helpful for those short transaction applications.
It is in user code, which should figure out which target partition the data should send to. When the data you need to
insert will be on the coordinator partition, then you don't need to do any configuration. If not, you can use db2 API
help you to find out.
For the OLTP system, another thing is also very important - don't make one db partition to become the workload
bottleneck.
Regards,
FRX
This doesn't really surprise me in the least bit. Do a couple of
searches for inserts and partitions and read a couple of my posts. In
our env we went from 4000 rows/sec Insert in a non partitioned table to
400 rows/sec when partitioned.
The only way around this is utilize buffered inserts which was mentioned
or build your application smart enough to load directly to each
partition inside of your application.
Check this link out for more information on buffered inserts.
m.com/dmdd/library/techarticle/0204pooloth/0204pooloth.htm l?Open&ca=daw-
ad-dr
Spencer
--
Posted via http://dbforums.com This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Eric |
last post by:
I'm trying to insert data for a real-time application that demands
less than 3 second receipt-to-commit time, has a relatively large
(500,000 records/hr) data rate, and requires access times on the order
of 10-15 seconds. We're storing about 24 hours worth of data on a
rolling basis, so I've partitioned the table by hour. Three of the
columns are (separately) indexed.
I know I can do this by piping data through sqlldr into the live...
|
by: SM |
last post by:
/*
problem: Trying to get partitioned views to "prune" unneeded
partitions from
select statements against the partitioned view. There are 5
partitioned
tables. Each with a check constraint based on a range of formula_id
column.
Test: Run this script to create the 5 partitioned tables and the
partitioned view. Then
|
by: Jay |
last post by:
Hi I have a huge table with over 100million records and on regular
basis ineed to delete nearly a million records and insert a million
records.
Currently I delete indexes before going through the process and
recreate the indexes which takes a very very long time.
IS there a way to disable indexes and re enable them after doing
insert and delete by reindexing or anything of that sort?
OR
Is there an approach to append two tables with...
|
by: jafastinger |
last post by:
In our shop we have a 28,000,000 row insert that gets reloaded every
night. I am not looking to change this process. I would like to know
why in UDB Vs 7.2 we were getting an insert rate of 3300 rows per
second. We migrated to version 8.1 of UDB (non-partitioned) and now
the rate decreased to ~1100 rows per second. The only thing we changed
is the database version.
On, a test box I did some testing.
The source in UDB version 7.2 fixpack...
|
by: simon |
last post by:
Hi
Hopefully someone could point me in the right direction on this one.
INFRASTRUCTURE
DB2 v7 on MVS OS390
SCENARIO
We are currently loading large volumes of data (eg 20MM rows) per day
into a partitioned DB2 database. In the installed version of DB2
there are only upto 240 (or so) partitions, so we have to double up
| |
by: shsandeep |
last post by:
The ETL application loaded around 3000 rows in 14 seconds in a Development
database while it took 2 hours to load in a UAT database.
UAT db is partitioned.
Dev db is not partitioned.
the application looks for existing rows in the table...if they already
exist then it updates otherwise inserts them.
The table is pretty large, around 6.5 million rows.
|
by: eavery |
last post by:
Does anyone know of any documentation on the performance of partition
merge/split? Does the merge or split of a partition cause any locking
on the partitioned table? If you were merging or splitting a large
volume of data rebalancing your partitioned table would you
potentially lock users out?
|
by: eavery |
last post by:
Does anyone know of any documentation on the performance of partition
merge/split? Does the merge or split of a partition cause any locking
on the partitioned table? If you were merging or splitting a large
volume of data rebalancing your partitioned table would you
potentially lock users out?
|
by: eeriehunk |
last post by:
Hi All,
Is it possible to create a partitioned index on a table which is not partitioned? If so what is such a partition called and please explain?
I have done some research on partitions and index partitions and this is where I hit the wall the 2nd time. As per my study there are 3 types of indexed partitions: Global index (which is on an entire partitioned table) then there is Local Index (on just the partition of the table) and then there...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| |
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |