Greetings,
Could someone confirm if the following reasoning is correct (64-bit DB2
on 64-bit AIX v5.3):
=============== =============== =============== =============== ======
For each connection to DB2 db, AIX will reserve a 64GB shared memory
segment for buffer caching.
So if there are connections to 3 different DBs, AIX will reserve a
total of 192 GB shared memory segment and one could quickly run out of
real memory. eg AIX with 5 GB of real memory and 6GB of paging space.
Excessive paging is the result of this.
=============== =============== =============== =============== ======
Many thanks! 6 8121 ne*****@hotmail .com wrote: Greetings,
Could someone confirm if the following reasoning is correct (64-bit DB2 on 64-bit AIX v5.3):
=============== =============== =============== =============== ====== For each connection to DB2 db, AIX will reserve a 64GB shared memory segment for buffer caching.
So if there are connections to 3 different DBs, AIX will reserve a total of 192 GB shared memory segment and one could quickly run out of real memory. eg AIX with 5 GB of real memory and 6GB of paging space.
Excessive paging is the result of this.
I have a hard time believing this.
I do recall issues on AIX 32bit on the _intance_ level where if someone
installs several instances of DB2 they can run out of segments.
Cheers
Serge
--
Serge Rielau
DB2 Solutions Development
IBM Toronto Lab
Serge Rielau wrote: ne*****@hotmail .com wrote: Greetings,
Could someone confirm if the following reasoning is correct (64-bit DB2 on 64-bit AIX v5.3):
=============== =============== =============== =============== ====== For each connection to DB2 db, AIX will reserve a 64GB shared memory segment for buffer caching.
So if there are connections to 3 different DBs, AIX will reserve a total of 192 GB shared memory segment and one could quickly run out of real memory. eg AIX with 5 GB of real memory and 6GB of paging space.
Excessive paging is the result of this. I have a hard time believing this. I do recall issues on AIX 32bit on the _intance_ level where if someone installs several instances of DB2 they can run out of segments.
Cheers Serge -- Serge Rielau DB2 Solutions Development IBM Toronto Lab
AIX has a late swap space allocation policy, so shared memory that has
never been accessed does not contribute to pageing, and does not occupy
real memory. Even though each DB segment will be 64GB, DB2 will only
use however much it is configured to (i.e. look at the size of your
bufferpools, locklist, shared sort heap, package cache, etc). The
excess will only be used if you try to dynamically allocate a new
bufferpool, extend an existing bufferpool, increase the package cache,
etc.
If you want to see how much of the database shared segment DB2 is
configured to use, then connect to your database, and run the
following:
db2 get db cfg show detail | grep DATABASE_MEMORY
The in-memory value will show the maximum number of 4K pages DB2 is
configured to use. DB2 will likely be consuming even less real memory
+ pageing space than this number, as a portion of those 4K pages are
not accessed until they are needed (i.e. memory for the utility heap is
not consumed until you start running backups/loads/restores/etc).
Cheers,
Liam
Thanks Serge/Liam.
I agree with you.
However, what would likely contribute to the massive allocation of
shared memory segments? Our Sys admin saw the allocation comes down as
soon as they turn db2_pinned_bp on.
As we never set this variable on for other AIX servers, I tend to
believe the over-allocated shared memory segments is caused by
something else. Excessive buffering for file system may be one of them.
Any suggestions?
Thanks!
Liam Finnie wrote: Serge Rielau wrote: ne*****@hotmail .com wrote: Greetings,
Could someone confirm if the following reasoning is correct (64-bit DB2 on 64-bit AIX v5.3):
=============== =============== =============== =============== ====== For each connection to DB2 db, AIX will reserve a 64GB shared memory segment for buffer caching.
So if there are connections to 3 different DBs, AIX will reserve a total of 192 GB shared memory segment and one could quickly run out of real memory. eg AIX with 5 GB of real memory and 6GB of paging space.
Excessive paging is the result of this. I have a hard time believing this. I do recall issues on AIX 32bit on the _intance_ level where if someone installs several instances of DB2 they can run out of segments.
Cheers Serge -- Serge Rielau DB2 Solutions Development IBM Toronto Lab
AIX has a late swap space allocation policy, so shared memory that has never been accessed does not contribute to pageing, and does not occupy real memory. Even though each DB segment will be 64GB, DB2 will only use however much it is configured to (i.e. look at the size of your bufferpools, locklist, shared sort heap, package cache, etc). The excess will only be used if you try to dynamically allocate a new bufferpool, extend an existing bufferpool, increase the package cache, etc.
If you want to see how much of the database shared segment DB2 is configured to use, then connect to your database, and run the following: db2 get db cfg show detail | grep DATABASE_MEMORY
The in-memory value will show the maximum number of 4K pages DB2 is configured to use. DB2 will likely be consuming even less real memory + pageing space than this number, as a portion of those 4K pages are not accessed until they are needed (i.e. memory for the utility heap is not consumed until you start running backups/loads/restores/etc).
Cheers, Liam
Hello,
As soon as you set DB2_PINNED_BP to true, or enable large pages for the
database shared memory set, DB2 will no longer attempt to allocate the
huge 64GB segment. The reason is that in these cases we attempt to
physically pin the entire segment. Pinning a 64GB segment on a system
that doesn't have more than 64GB of physical RAM is not desirable :-)
Therefore, we only allocate how much we are currently configured to
use. This, of course, limits your ability to dynamically grow your
bufferpools, locklist, package cache, etc.
Pinned and large-page memory should be used with caution - you need to
really know what type of workload you'll have, and carefully provision
memory on the machine. Otherwise, if you're not careful, pinning
memory could actually *cause* pageing of hot data (if some infrequently
used memory is pinned, and the system is low on memory, it will have to
start pageing more frequently used memory).
If you haven't changed your default file cache settings, you should
probably investigate that. The default file cache configuration is not
likely suitable when running a database server, since database servers
maintain their own cache of file system data (i.e. the DB2
bufferpools). Having a large file cache ends up giving you
double-caching - the OS caches the page in its file cache, and DB2
caches the same page in its bufferpools. You can run 'vmstat -v', and
look at the minperm, maxperm, and numperm settings. numperm will tell
you the percentage of memory currently used for the file system cache.
If it's lower than maxperm, or minperm is set fairly high, then that
may end up causing pageing - the OS might page out working pages to
maintain a large file cache.
Two things to consider if your system is using a large amount of file
cache:
1) Lower minperm/maxperm. This will leave more room for working
memory, which will usually benefit database servers. This could also
reduce pageing, since the system may then favor evicting clean file
cache pages - this is very cheap since the page can just be discarded
without writing any data back to disk. When evicting working memory,
the OS must write the data to swap space, which is very slow.
2) Look into Direct IO/Concurrent IO. This will avoid double-buffering
of data, since DB2 will avoid using the OS file cache to read in data
pages to the bufferpool.
You should probably consider doing both. Lowering maxperm will favor
keeping working pages in memory, which should help reduce pageing.
Enabling Direct IO/Concurrent IO will reduce file cache usage for data
that DB2 will be cacheing in its bufferpools, leaving the file cache
free to cache other data.
Cheers,
Liam
Liam,
Thank you so much for the in-depth explanation.
Any reasons the system will favor evicting CLEAN file cache pages after
lowering minperm/maxperm?
Thanks again!
Liam Finnie wrote: Hello,
As soon as you set DB2_PINNED_BP to true, or enable large pages for the database shared memory set, DB2 will no longer attempt to allocate the huge 64GB segment. The reason is that in these cases we attempt to physically pin the entire segment. Pinning a 64GB segment on a system that doesn't have more than 64GB of physical RAM is not desirable :-) Therefore, we only allocate how much we are currently configured to use. This, of course, limits your ability to dynamically grow your bufferpools, locklist, package cache, etc.
Pinned and large-page memory should be used with caution - you need to really know what type of workload you'll have, and carefully provision memory on the machine. Otherwise, if you're not careful, pinning memory could actually *cause* pageing of hot data (if some infrequently used memory is pinned, and the system is low on memory, it will have to start pageing more frequently used memory).
If you haven't changed your default file cache settings, you should probably investigate that. The default file cache configuration is not likely suitable when running a database server, since database servers maintain their own cache of file system data (i.e. the DB2 bufferpools). Having a large file cache ends up giving you double-caching - the OS caches the page in its file cache, and DB2 caches the same page in its bufferpools. You can run 'vmstat -v', and look at the minperm, maxperm, and numperm settings. numperm will tell you the percentage of memory currently used for the file system cache. If it's lower than maxperm, or minperm is set fairly high, then that may end up causing pageing - the OS might page out working pages to maintain a large file cache.
Two things to consider if your system is using a large amount of file cache: 1) Lower minperm/maxperm. This will leave more room for working memory, which will usually benefit database servers. This could also reduce pageing, since the system may then favor evicting clean file cache pages - this is very cheap since the page can just be discarded without writing any data back to disk. When evicting working memory, the OS must write the data to swap space, which is very slow.
2) Look into Direct IO/Concurrent IO. This will avoid double-buffering of data, since DB2 will avoid using the OS file cache to read in data pages to the bufferpool.
You should probably consider doing both. Lowering maxperm will favor keeping working pages in memory, which should help reduce pageing. Enabling Direct IO/Concurrent IO will reduce file cache usage for data that DB2 will be cacheing in its bufferpools, leaving the file cache free to cache other data.
Cheers, Liam
Hello,
That's starting to get deep into VMM layer of the OS. If numperm is
still above minperm, my assumption would be that clean file cache pages
(as long as they're not too hot) will likely be favored for eviction,
since the page-out cost is zero (i.e. no data has to be written out to
disk, since the on-disk copy of the file already exists, so the memory
page can simply be discarded). Pageing out a working memory page, or a
dirty file cache page, will require a physical I/O before that memory
page can be re-used by someone else.
I believe AIX keeps stats on re-pageing rates (i.e. if the OS pages out
or discards a particular page, but finds that it needs to read it back
in again shortly afterwards) for both working and file cache pages.
These re-pageing rates, along with minperm/maxperm, help determine
whether file cache pages or working pages get evicted.
Cheers,
Liam. This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: ck388 |
last post by:
For some reason when I enable the callback feature of the gridview I
still get a page refresh, that is it seems like there is a postback
that occurs, not a callback which is just supposed to update not the
whole page, but a portion of the page.
Strangely enough the URL below
http://beta.asp.net/QUICKSTARTV20/aspnet/doc/ctrlref/data/gridview.aspx
(VB GridView Paging and Sorting Callbacks example)
|
by: Shawn |
last post by:
Hi.
I already have a datagrid where I'm using paging. I have a stored procedure
that fills a temp table with 200-500 rows and then sends back 10 records at
the time. When I go to page 2 the SP fills the temp table again and returns
rows 10-19. The temp table is dropped after each call to the SP, so it has
to be created and filled every time the user changes page in the datagrid.
My question is this: Would it be more efficient to...
|
by: asad |
last post by:
Hello friends, i am designing a ASP.NET page where i want to use custom
paging bcoz data is too heavy so pls tell me how can i use custom paging in
ASP.NET
Thanks
|
by: tom fredriksen |
last post by:
Hi
I was doing a simple test of the speed of a "maths" operation and when I
tested it I found that removing the loop that initialises the data array
for the operation caused the whole program to spend twice the time to
complete. If the loop is included it takes about 7.48 seconds to
complete, but when removed it takes about 11.48 seconds.
Does anybody have a suggestion as to why this is so and whether I can
trust the results of the...
|
by: farhad13841384 |
last post by:
Hi , I Hope You fine.
I have some problem with this code for paging in asp.net
this bottom code work correctly without any error but when
I try to place separate code in .VB file then error is begin and occured .I want to separate this code and compiling .vb code using VBC.exe later .(bin/paging.dll)
when do it like me so you retrive only < Previous Page Next Page > in your web browser and you don't retrive list Of data in your web browser....
| |
by: anonieko |
last post by:
This approach I found very efficient and FAST when compared to the
rowcount, or Subquery Approaches.
This is before the advent of a ranking function from DB such as
ROW_NUMBER() in SQL Server 2005 and the likes of it. So
This one works with SQL2000
What do you think?
|
by: rn5a |
last post by:
In a shopping cart app, a ASPX page retrieves the order details &
personal details of a user from a MS-Access database table depending
upon the username of the user.
The order details of a particular order (like ProductID, Name,
Description, Quantity etc.) are displayed in one DataGrid where as the
personal details of the buyer corresponding to this order (like Name,
E-Mail, Shipping & Billing Address etc.) are displayed in another...
|
by: Ronald S. Cook |
last post by:
I was told that if calling lots of records from the database (let's say
100,000), that the GridView's paging feature would automatically "handle"
everything.
But the 100,000 records are still coming to the client, right? I mean, the
paging feature isn't somehow making calls to the database for 25 records at
a time or anything like that is it?
I remember in the past having to write nasty stored procedures that took in
|
by: Donald Adams |
last post by:
Hi,
I will have both web and win clients and would like to page my data. I
could not find out how the datagrid control does it's paging though I did
find some sample code that says they do it this way, but I can't see these
methods as public.
BookmarksDataSetTableAdapters.BookmarksTableAdapter bookTA = new
BookmarksDataSetTableAdapters.BookmarksTableAdapter();
BookmarkList1.DataSource = bookTA.GetAllBookmarksWrtUser(
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
| |
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |