473,699 Members | 3,282 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

db2 File system caching ..DIO or CIO??

Raj
What is the purpose of file system caching while creating a tablespace?
Memory on the test server gets used up pretty quickly after a user
executes a complex query(database is already activated), after some
investgation i found out that most of it being consumed by filesystem
caching... thanks to Liam and Phil Sherman for their valuable
suggestions. Is it safe to turn off filesystem caching on every
tablespaceon the server(i.e. DIO) ?? or can we set DB2_MMPAP_READ= ON
and DB2_MMAP_WRITE= ON ?? what is the down side of it?? We do have a big
enough bufferpool...

---------------------------------------------------------------------------------------------------------------------------------------
vmstat -v
2097152 memory pages
1982466 lruable pages
862 free pages
1 memory pools
130485 pinned pages
80.1 maxpin percentage
20.0 minperm percentage
80.0 maxperm percentage
37.7 numperm percentage
747627 file pages
0.0 compressed percentage
0 compressed pages
0.0 numclient percentage
80.0 maxclient percentage
0 client pages
0 remote pageouts scheduled
0 pending disk I/Os blocked with no pbuf
1199107 paging space I/Os blocked with no psbuf
1744312 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
0 external pager filesystem I/Os blocked with no
fsbuf
---------------------------------------------------------------------------------------------------------------------------------------
Thanks,
Raj

Mar 31 '06 #1
5 7851
File systme caching has nothing to do with a database. It is an
operating system mechanism to improve I/O performance when working with
files. Operating systems attempt to improve I/O performance by caching
frequently used data (ie. a directory) in memory to avoid the physical
I/O delays when it's necessary to READ data. On a dedicated database
system, this acts in a similar fashion to a processor's level 1 and
level 2 caches where the buffer pool is level 1 and the file system
cache is level 2. (Note that the cache proportions here are much
different than hardware caches.)

Real memory larger than program needs, if not used, is a wasted
resource. Mature, well designed, operating systems such as AIX, UNIX,
LINUX and others use this "excess" memory to support a large file system
cache. When program memory requirements rise, then the cache is shrunk
as needed. As requirements fall, the cache is expanded keeping memory
"in use".

Vmstat only provides you with part of the information you need to
determine if your database system is taking advantage of the operating
system's file caching mechanism. You need database statistics showing
the percentage of requests being handled by the buffer pool and the I/O
performance when the data being read is not in the buffer pool. Don't
forget that UDB uses two I/O mechansims, one for individual rows,
another for scans. Descriptions of the statistics and the formulas to
compute I/O performance are available in the manuals.

Your vmstat output indicates 2g memory on the system with about 35% of
it currently being used for caching the file system. If this system is
only a database servier, then your "big enough" buffer pool could be
expanded quite a bit. A competent DBA, with a good knowledge of the
application requirements can probably find a better use for the memory
than the file system's caching mechanism can.

I'd get a good baseline of buffer pool performance under a number of
different workloads and start altering your tablespaces, one by one, to
not use file system caching. Carefully measure buffer pool performance
with each one you change to verify that you are maintaining your buffer
pool performance. Don't be surprised if you see little change in the
vmstat file pages because the operating system will fill them with other
file data. Also keep an eye on paging - it it starts rising, you'll need
to investigate its causes.

Philip Sherman


Raj wrote:
What is the purpose of file system caching while creating a tablespace?
Memory on the test server gets used up pretty quickly after a user
executes a complex query(database is already activated), after some
investgation i found out that most of it being consumed by filesystem
caching... thanks to Liam and Phil Sherman for their valuable
suggestions. Is it safe to turn off filesystem caching on every
tablespaceon the server(i.e. DIO) ?? or can we set DB2_MMPAP_READ= ON
and DB2_MMAP_WRITE= ON ?? what is the down side of it?? We do have a big
enough bufferpool...

---------------------------------------------------------------------------------------------------------------------------------------
vmstat -v
2097152 memory pages
1982466 lruable pages
862 free pages
1 memory pools
130485 pinned pages
80.1 maxpin percentage
20.0 minperm percentage
80.0 maxperm percentage
37.7 numperm percentage
747627 file pages
0.0 compressed percentage
0 compressed pages
0.0 numclient percentage
80.0 maxclient percentage
0 client pages
0 remote pageouts scheduled
0 pending disk I/Os blocked with no pbuf
1199107 paging space I/Os blocked with no psbuf
1744312 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
0 external pager filesystem I/Os blocked with no
fsbuf
---------------------------------------------------------------------------------------------------------------------------------------
Thanks,
Raj

Mar 31 '06 #2
Raj
Thank you so much for your time and the detailed explanation ...The mem
allocation on our server is --> out of 8gb mem, 2.7 gb is allocated
to bufferpools and 4.2G is used by db2, when we run complex sql's the
free memory (from nmon/vmstat) reduces and paging happens (pi & po
vmstat )...

Mar 31 '06 #3
Ian

The only point I would add to Phil's comments: The file system cache
effectively creates double-buffering if you are using SMS or DMS file
containers. This degrades performance somewhat, and is why solutions
like concurrent IO or direct IO are helpful in database systems.

Also: On AIX (since that's what you show in vmstat -v output), you
can alter the behavior of the file system cache with the vmo command
and the minperm/maxperm parameters.
Apr 2 '06 #4
"Ian" <ia*****@mobile audio.com> wrote in message
news:44******** @newsfeed.slurp .net...

The only point I would add to Phil's comments: The file system cache
effectively creates double-buffering if you are using SMS or DMS file
containers. This degrades performance somewhat, and is why solutions
like concurrent IO or direct IO are helpful in database systems.

Also: On AIX (since that's what you show in vmstat -v output), you
can alter the behavior of the file system cache with the vmo command
and the minperm/maxperm parameters.


What about caching and pre-fetching built into some disk subsystems like
NAS?
Apr 2 '06 #5
Ian
Mark A wrote:
"Ian" <ia*****@mobile audio.com> wrote in message
news:44******** @newsfeed.slurp .net...
The only point I would add to Phil's comments: The file system cache
effectively creates double-buffering if you are using SMS or DMS file
containers. This degrades performance somewhat, and is why solutions
like concurrent IO or direct IO are helpful in database systems.

Also: On AIX (since that's what you show in vmstat -v output), you
can alter the behavior of the file system cache with the vmo command
and the minperm/maxperm parameters.


What about caching and pre-fetching built into some disk subsystems like
NAS?


What about them?

Caches at the disk subsystem level are a totally different beast from
the OS file system cache. Their effectiveness depends on a large number
of variables, such as workload (for the entire disk subsystem, not just
for the specific database), RAID and LUN design, and implementation at
both the OS and database levels.

Apr 2 '06 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

14
2207
by: ajikoe | last post by:
Hello, If I use os.remove(fileName), does it always assure that the code will move to the next code only if the fileName is deleted completely? Pujo
7
3369
by: clusardi2k | last post by:
Hello, I have a shared drive on SGI, Linux, and Windows. A second call to fopen doesn't create the file if it has been deleted. I would like to use fopen for its pointer return value to solve this. What is the best way to fix this problem?
13
4309
by: Sky Sigal | last post by:
I have created an IHttpHandler that waits for uploads as attachments for a webmail interface, and saves it to a directory that is defined in config.xml. My question is the following: assuming that this is suppossed to end up as a component for others to use, and therefore I do NOT have access to their global.cs::Session_End() how do I cleanup files that were uploaded -- but obviously left stranded when the users aborted/gave up writting...
1
5574
by: MarkWH | last post by:
I am writing a simple reverse proxy using .Net. The basic design is to capture all traffic (from all URL's) in an HttpHandler and the respond accordingly. The problem is that I when a URL is received that contains a "`" character I get the exception listed above; and the call stack listed below. Questions; 1) Where can I find more documentation on the FileChangesMonitor class?
5
483
by: Joe | last post by:
I'm getting the following error when trying to call a page on a secure server. I'm not doing any impersonations or file access of any kind. The page is using PayPal and I'm wondering if PayPal has something to do with this. We had to register an api certificate on the server using WinHttpCertCfg. Maybe this is doing something? Access denied to 'D:\WWWRoot\sslserver'. Failed to start monitoring file changes. Description: An unhandled...
17
4886
by: shineofleo | last post by:
Here is the situation: I wrote a VB programm, which stores all the information in a single Access database file using jet engine. It worked well, however one of my customs reported that there was some problems with this programm. I checked, the log files showed that the database was corrupted. The customer told me that there no 'illegal' operation such as pull out the plug, or kill the programm via task manager... So is there any...
5
1547
by: Russell Warren | last post by:
I've got a case where I'm seeing text files that are either all null characters, or are trailed with nulls due to interrupted file access resulting from an electrical power interruption on the WinXP pc. In tracking it down, it seems that what is being interrupted is either os.remove(), or os.rename(). Has anyone seen this behaviour, or have any clue what is going on? On first pass I would think that both of those calls are single step...
17
2724
by: NeoAlchemy | last post by:
I am starting to find more web pages that are using a query parameters after the JavaScript file. Example can be found at www.opensourcefood.com. Within the source you'll see: <script src="/shared/scripts/common.js?revision=1.6" type="text/javascript">. I am trying to see if there is any big deal to this or a best practice that is starting to creep up in the JavaScript community. If this is used only as a way to distinguish what...
15
5270
by: lxyone | last post by:
Using a flat file containing table names, fields, values whats the best way of creating html pages? I want control over the html pages ie 1. layout 2. what data to show 3. what controls to show - text boxes, input boxes, buttons, hyperlinks ie the usual. The data is not obtained directly from a database.
0
8623
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9187
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9053
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
8894
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7776
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
4390
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4636
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3071
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2360
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.