DB2 (v9fp2/linux) has 4G limit for DEVICE containers ?!

[Konstantin Andreev]

Hello.

I'm experiencing very disagreeable problem: DB2 refuses tablespace containers, which are DEVICE's bigger than 4G.

So, If I below 4G limit:

| # db2 create tablespace TS1 pagesize 4096 managed by database using ( device '/dev/dtas1' 1048575 )
|
| DB20000I The SQL command completed successfully.

But If I add just 1 (one) page to come over 4G:

| # db2 create tablespace TS2 pagesize 4096 managed by database using ( device '/dev/dtas2' 1048576 )
|
| DB21034E The command was processed as an SQL statement because it was not a valid Command Line Processor command. During SQL processing it returned:
| SQL1422N The size of the container is invalid. SQLSTATE=54039

Additionally, "db2diag.log" contains messages like:

| ADM6040E Invalid container size specified for container "/dev/dtas2". The number of pages specified for use is too large.
| ZRC=0x80020046=-2147352506=SQLB_BAD_CONT_SZ "bad container size" DATA #1 : String, 34 bytes totalPages = 1048576, contSize = 0

At the same time DB2 smoothly creates FILE containers >4G. I have tried 20G and 30G.

Is this limitation (4G for DEVICE'es) by design, or something wrong with my system ?
I'm running Linux kernel "2.6.21.1", glibc "2.5". libaio disabled in DB2.
--
Konstantin Andreev.

[Jan M. Nelken]

Konstantin Andreev wrote:

| # db2 create tablespace TS1 pagesize 4096 managed by database using (
device '/dev/dtas1' 1048575 )
|
| DB20000I The SQL command completed successfully.

But If I add just 1 (one) page to come over 4G:

| # db2 create tablespace TS2 pagesize 4096 managed by database using (
device '/dev/dtas2' 1048576 )
|
| DB21034E The command was processed as an SQL statement because it was
not a valid Command Line Processor command. During SQL processing it returned:
| SQL1422N The size of the container is invalid. SQLSTATE=54039

Additionally, "db2diag.log" contains messages like:

| ADM6040E Invalid container size specified for container "/dev/dtas2".
The number of pages specified for use is too large.
| ZRC=0x80020046=-2147352506=SQLB_BAD_CONT_SZ "bad container size" DATA
#1 : String, 34 bytes totalPages = 1048576, contSize = 0

What is device size of /dev/dtas2?

Did you verify size as documented in:
http://publib.boulder.ibm.com/infoce...c/t0004971.htm

What is output of db2level command and whichdistro command?
Jan M. Nelken

[Konstantin Andreev]

Jan M. Nelken wrote:

>| # db2 create tablespace TS1 pagesize 4096 managed by database using ( device '/dev/dtas1' 1048575 )
| DB20000I The SQL command completed successfully.

| # db2 create tablespace TS2 pagesize 4096 managed by database using ( device '/dev/dtas2' 1048576 )
| SQL1422N The size of the container is invalid. SQLSTATE=54039

Additionally, "db2diag.log" contains messages like:

| ADM6040E Invalid container size specified for container "/dev/dtas2". The number of pages specified for use is too large.
| ZRC=0x80020046=-2147352506=SQLB_BAD_CONT_SZ "bad container size"
DATA #1 : String, 34 bytes totalPages = 1048576, contSize = 0

What is device size of /dev/dtas2?

The sizes of the devices above are exactly as indicated to DB2, i.e.
/dev/dtas2 = 1048576 of 4k pages = ( 1048576 * 8 = 8388608 blocks of 512b )

I verified it in a number of ways:

| # blockdev --getsz /dev/dtas2
| 8388608
|
| # dd if=/dev/dtas2 of=/dev/null bs=512
| 8388608+0 records in
| 8388608+0 records out
| 4294967296 bytes (4,3 GB) copied, 90,7024 s, 47,4 MB/s

Did you verify size as documented in:
http://publib.boulder.ibm.com/infoce...c/t0004971.htm

Certainly. I have extensively practiced in DEVICE containers creation ( but below 4G ) to be sure I do all right.

Don't be confused by these suspicious sizes: 1048575 (4G - 4K) and 1048576 (4G exactly). These devices are really specially crafted (for this demonstration) slices (of bigger disk) made by 'device mapper'. To be exact:

| # echo "0 8388600 linear /dev/sdb 166136392" | dmsetup create dtas1
| # echo "0 8388608 linear /dev/sda 166136392" | dmsetup create dtas2

But limitation of 4G is not specific for 'device-mapper' devices. This limitation stays if I supply DB2 with, say, conventional primary partition /dev/sdc1.

The most curious that DB2 only sees only lowest 32 bits of size of any DEVICE container. Say, If you have device of 40.1G, the maximum container you could acquire from DB2 is... **** 0.1G ****.

What is output of db2level command and whichdistro command?

| # db2level
| DB21085I Instance "dbi" uses "32" bits and DB2 code release "SQL09012" with level identifier "01030107".
| Informational tokens are "DB2 v9.1.0.2", "s070210", "MI00183", and Fix Pack "2".
| Product is installed at "/db2/sys".

| # whichdist
| ------------------Distribution Information
| /etc/gentoo-release:
| Gentoo Base System release 1.12.9
| ------------------Kernel Information
| Node name: avalon
| Kernel Level: 2.6.21.1-av11
| Kernel Build Date: #6 SMP PREEMPT Thu May 10 22:12:16 MSD 2007
| Machine Type: i686
| ------------------libc.so Information
| GLIBC version: 2.5
| ------------------libstdc++ Information
| DB2 v9.x (i686): 32-bit libstdc++.so.5 exists as /usr/lib/libstdc++-v3/libstdc++.so.5.0.6

I had assumed that some incompatibility exists between DB2 and this system's kernel. This day I had tested DB2 against official (recommended by IBM) RHEL 5 latest stock kernel: "kernel-2.6.18-8.1.4.el5.i686.rpm", and it hadn't helped. The same 4G limit on DEVICE container.

Too bad...
--
Konstantin Andreev.

[Haider Rizvi]

Konstantin, a similar problem was found with CentOS and it will be
fixed (worked around) in the next fixpack of DB2 9. So if you can wait
until then. I am told the problem is with a bug in ioctl
(blkgetsize64) implementation.

A work around for you is to move to 64-bit kernel.

The list of validated linux distros is kept here:
http://www-306.ibm.com/software/data...inux/validate/

Regards,
- Haider

Konstantin Andreev <ci************@datatech.ruwrites:

Jan M. Nelken wrote:

>>| # db2 create tablespace TS1 pagesize 4096 managed by database
| using ( device '/dev/dtas1' 1048575 ) DB20000I The SQL
| command completed successfully.

| # db2 create tablespace TS2 pagesize 4096 managed by database
| using ( device '/dev/dtas2' 1048576 ) SQL1422N The size of
| the container is invalid. SQLSTATE=54039

Additionally, "db2diag.log" contains messages like:

| ADM6040E Invalid container size specified for container
| "/dev/dtas2". The number of pages specified for use is too
| large. ZRC=0x80020046=-2147352506=SQLB_BAD_CONT_SZ "bad
| container size"
DATA #1 : String, 34 bytes totalPages = 1048576, contSize = 0

>What is device size of /dev/dtas2?

The sizes of the devices above are exactly as indicated to DB2,
i.e. /dev/dtas2 = 1048576 of 4k pages = ( 1048576 * 8 = 8388608
blocks of 512b )

I verified it in a number of ways:

| # blockdev --getsz /dev/dtas2 8388608
|
| # dd if=/dev/dtas2 of=/dev/null bs=512 8388608+0 records in
| 8388608+0 records out 4294967296 bytes (4,3 GB) copied, 90,7024
| s, 47,4 MB/s

>Did you verify size as documented in:
http://publib.boulder.ibm.com/infoce...c/t0004971.htm

Certainly. I have extensively practiced in DEVICE containers
creation ( but below 4G ) to be sure I do all right.

Don't be confused by these suspicious sizes: 1048575 (4G - 4K) and
1048576 (4G exactly). These devices are really specially crafted
(for this demonstration) slices (of bigger disk) made by 'device
mapper'. To be exact:

| # echo "0 8388600 linear /dev/sdb 166136392" | dmsetup create
| dtas1 # echo "0 8388608 linear /dev/sda 166136392" | dmsetup
| create dtas2

But limitation of 4G is not specific for 'device-mapper'
devices. This limitation stays if I supply DB2 with, say,
conventional primary partition /dev/sdc1.

The most curious that DB2 only sees only lowest 32 bits of size of
any DEVICE container. Say, If you have device of 40.1G, the
maximum container you could acquire from DB2 is... **** 0.1G ****.

>What is output of db2level command and whichdistro command?

| # db2level DB21085I Instance "dbi" uses "32" bits and DB2 code
| release "SQL09012" with level identifier "01030107".
| Informational tokens are "DB2 v9.1.0.2", "s070210", "MI00183",
| and Fix Pack "2". Product is installed at "/db2/sys".

| # whichdist ------------------Distribution Information
| /etc/gentoo-release: Gentoo Base System release 1.12.9
| ------------------Kernel Information Node name: avalon Kernel
| Level: 2.6.21.1-av11 Kernel Build Date: #6 SMP PREEMPT Thu May
| 10 22:12:16 MSD 2007 Machine Type: i686
| ------------------libc.so Information GLIBC version: 2.5
| ------------------libstdc++ Information DB2 v9.x (i686): 32-bit
| libstdc++.so.5 exists as
| /usr/lib/libstdc++-v3/libstdc++.so.5.0.6

I had assumed that some incompatibility exists between DB2 and
this system's kernel. This day I had tested DB2 against official
(recommended by IBM) RHEL 5 latest stock kernel:
"kernel-2.6.18-8.1.4.el5.i686.rpm", and it hadn't helped. The same
4G limit on DEVICE container.

Too bad...
--
Konstantin Andreev.

--
Haider

[Konstantin Andreev]

Haider Rizvi wrote:

A similar problem was found with CentOS and it will be fixed (worked around) in the next fixpack of DB2 9. So if you can wait until then. I am told the problem is with a bug in ioctl (blkgetsize64) implementation.

Wrong! Somebody is trying to avoid the responsibility and lay the blame on the kernel developers. But for now it's almost clear this is a bug in DB2. DB2 truncates container size to the lowest 32 bits.
Here are the evidences.

1) Let's try to create container with size: (4G exactly + 4k), i.e 1048577 of 4k pages:

| # db2 create tablespace TS2 pagesize 4096 managed by database using ( device '/dev/dtas2' 1048577 )

2) Let's spy, what is doing 'db2agent' upon this application request ( in my experiment the 'db2agent' in question has pid 5597 ).

| # strace -p 5597 -o db2agnt_trace

Skipping the trace garbage, I see only few system calls concerning '/dev/dtas2':

| access("/dev/dtas2", F_OK) = 0
| open("/dev/dtas2", O_RDWR|O_CREAT|O_DIRECT|O_LARGEFILE, 0600) = 5
| fstat64(5, {st_mode=S_IFBLK|0600, st_rdev=makedev(254, 60), ...}) = 0
| ioctl(5, 0x80041272, 0xbfc440ec) = 0

The ioctl() call above is just you are telling about, and must return the device size in bytes.

3) Simultaniously this entry appears in db2diag.log :

| 2007-05-30-18.38.24.545782+240 I141154G491 LEVEL: Error
| PID : 5597 TID : 2988168880 PROC : db2agent (T8)
| INSTANCE: dbi NODE : 000 DB : T8
| APPHDL : 0-7 APPID: *LOCAL.dbi.070530143323
| AUTHID : ROOT
| FUNCTION: DB2 UDB, buffer pool services, sqlbDMSAddContainerRequest, probe:104
| MESSAGE : ZRC=0x80020046=-2147352506=SQLB_BAD_CONT_SZ "bad container size"
| DATA #1 : String, 34 bytes
| totalPages = 1048577, contSize = 1

Note the text "contSize = 1". That's how the DB2 sees the size of the contaner : JUST ONE PAGE, whereas the real device size is one page over 4G.

4) It's not hard to write small C program to make 2 system calls:

| int fd = open( "/dev/dtas2", O_RDWR|O_CREAT|O_DIRECT|O_LARGEFILE|O_NONBLOCK, 0600);
| ioctl( fd, 0x80041272, & bs ); // bs is 64-bit wide unsigned int.

The ioctl() above returns the CORRECT VALUE: 0x100001000 = 1048577 * 4096.

So, the BUG exists, but not in the ioctl() implementation, but in DB2.
--
Konstantin Andreev.

PS: It's worth to mention that using O_CREAT flag on device open is rather silly.

[Jan M. Nelken]

Konstantin Andreev wrote:

Wrong! Somebody is trying to avoid the responsibility and lay the blame
on the kernel developers. But for now it's almost clear this is a bug in
DB2. DB2 truncates container size to the lowest 32 bits.

Calm down, Kostia!

what Haider was telling - and I can confirm - is that we have an issue
within DB2 and that is being fixed in FP3 - soon to be released.

For now on only workaround is to use 64-bit environment.
Jan M. Nelken

[Ray]

On May 30, 5:32 pm, "Jan M. Nelken" <[e-mail]>
wrote:

what Haider was telling - and I can confirm - is that we have an issue
within DB2 and that is being fixed in FP3 - soon to be released.

"...the problem is with a bug in ioctl(blkgetsize64)
implementation..."

I can understand the reaction. It may have been what Haider meant but
it wasn't what he wrote. What was written could cause the wrong set of
developers a lot of needless heartburn if many people took that
literally and started banging on their OS vendor.

So maybe "implementation" should have been "return value handling."

[Haider Rizvi]

Ray <mc*****@gmail.comwrites:

On May 30, 5:32 pm, "Jan M. Nelken" <[e-mail]>
wrote:

>what Haider was telling - and I can confirm - is that we have an
issue within DB2 and that is being fixed in FP3 - soon to be
released.

"...the problem is with a bug in ioctl(blkgetsize64)
implementation..."

I can understand the reaction. It may have been what Haider meant
but it wasn't what he wrote. What was written could cause the
wrong set of developers a lot of needless heartburn if many people
took that literally and started banging on their OS vendor.

So maybe "implementation" should have been "return value
handling."

Whoa, I missed all the action, too busy today.

Others have reiterated that the problem is with a bug in ioctl
implementation, and we have worked around that problem in DB2
code. That's what will be available in FP3.

Regards,
--
Haider

[Konstantin Andreev]

Haider Rizvi wrote:

>So maybe "implementation" should have been "return value handling."

Others have reiterated that the problem is with a bug in ioctl implementation

In my yesterday letter I disproved this statement by direct evidences. Where are your reasons to insist on your compromised statement ? Could you point to an error in my deduction ?
--
Konstantin Andreev.