Weird Error Message

[Michel Esber]

Hello,

Environment is: DB2 V8 FP10 Linux Kernel 2.6

My db2diag.log is full of the following lines:

2006-02-01-15.56.26.557312-120 E405226240G308 LEVEL: Error (OS)
PID : 31005 TID : 4143875776 PROC : db2pclnr
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloLioInit, probe:10
CALLED : OS, -, io_queue_init OSERR: EAGAIN (11)

I have been unable to find a meaning for this. Can anyone help ? Seems
like a SO error, but where?

Thanks in advance,

[Serge Rielau]

Michel Esber wrote:

Hello,

Environment is: DB2 V8 FP10 Linux Kernel 2.6

My db2diag.log is full of the following lines:

2006-02-01-15.56.26.557312-120 E405226240G308 LEVEL: Error (OS)
PID : 31005 TID : 4143875776 PROC : db2pclnr
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloLioInit, probe:10
CALLED : OS, -, io_queue_init OSERR: EAGAIN (11)

I have been unable to find a meaning for this. Can anyone help ? Seems
like a SO error, but where?

Thanks in advance,

Which DIAGLEVEL are you running?
The error is being logged by the "page cleaner", that's the process
writes dirty pages back to disk.

Cheers
Serge
--
Serge Rielau
DB2 Solutions Development
DB2 UDB for Linux, Unix, Windows
IBM Toronto Lab

[Michel Esber]

Serge,

Running DIAGLEVEL 3. Here is some additional info:

DB2LINUXAIO=TRUE
DB2_SCATTERED_IO=ON
DB2COMM=tcpip
DB2AUTOSTART=YES

Thanks,

[Knut Stolze]

Michel Esber wrote:

Hello,

Environment is: DB2 V8 FP10 Linux Kernel 2.6

My db2diag.log is full of the following lines:

2006-02-01-15.56.26.557312-120 E405226240G308 LEVEL: Error (OS)
PID : 31005 TID : 4143875776 PROC : db2pclnr
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloLioInit, probe:10
CALLED : OS, -, io_queue_init OSERR: EAGAIN (11)

I have been unable to find a meaning for this. Can anyone help ? Seems
like a SO error, but where?

This says that you have asynchronous I/O enabled, DB2 tries to detect
whether AIO is available for your underlying file system. For that, an
initialization is attempted by calling "io_queue_init" in the aio library
(libaio). The call to that function returns EAGAIN, probably because you
don't have an AIO-capable file system?

--
Knut Stolze
DB2 Information Integration Development
IBM Germany

[Serge Rielau]

Michel Esber wrote:

Running DIAGLEVEL 3. Here is some additional info:

That should be fine. Just wanted to make sure you are not running level
4 which sends DB2 onto the diagnostic couch
How often do you get these?

I'll ask backstage.. no promisses...
Cheers
Serge

--
Serge Rielau
DB2 Solutions Development
DB2 UDB for Linux, Unix, Windows
IBM Toronto Lab

[Michel Esber]

I am getting around +- 70 events like that per minute.

db2diag.log gets really big and we have to rotate it quite frequently.

Thanks Serge,

[Knut Stolze]

Michel Esber wrote:

I am getting around +- 70 events like that per minute.

db2diag.log gets really big and we have to rotate it quite frequently.

Uugh. That's ugly. Have you checked if you can actually use AIO?

--
Knut Stolze
DB2 Information Integration Development
IBM Germany

[Michel Esber]

The FS is AIO capable. In fact, I have another server, with the same
environment, but without any errors.

I was reading some man pages and it seems like EAGAIN indicates lack of
resources to complete the asynchronous request. Now I have to figure
out which resource is this ... (or which limit should be increased)

Thanks,

[Serge Rielau]

Michel Esber wrote:

The FS is AIO capable. In fact, I have another server, with the same
environment, but without any errors.

I was reading some man pages and it seems like EAGAIN indicates lack of
resources to complete the asynchronous request. Now I have to figure
out which resource is this ... (or which limit should be increased)

Thanks,

Comments from backstage:
check if libaio is installed by doing rpm -qa | grep aio . If it is
installed then please let us know the version number of the libaio package.

Cheers
Serge
--
Serge Rielau
DB2 Solutions Development
DB2 UDB for Linux, Unix, Windows
IBM Toronto Lab

[Michel Esber]

libaio-0.3.103-3
libaio-devel-0.3.103-3

One new info: seems like this box has been upgraded from a Linux RedHat
2.1 to RH 4.0 Update 2. All filesystems are ext3 though. Could this be
a possible reason?

My other servers (where I have no problems) had a clean installation of
RH 4.0.

Thanks

[Serge Rielau]

Michel Esber wrote:

libaio-0.3.103-3
libaio-devel-0.3.103-3

One new info: seems like this box has been upgraded from a Linux RedHat
2.1 to RH 4.0 Update 2. All filesystems are ext3 though. Could this be
a possible reason?

My other servers (where I have no problems) had a clean installation of
RH 4.0.

Thanks

More:
What does uname -a say ?
Are you using a 32-bit or 64-bit instance ?

It seems there is a recently discovered issue with AIO on RH4 on 32bit.

Cheers
Serge
--
Serge Rielau
DB2 Solutions Development
DB2 UDB for Linux, Unix, Windows
IBM Toronto Lab

[Michel Esber]

[smichel@autctx02rjomk ~]$ uname -a
Linux autctx02rjomk 2.6.9-22.0.1.ELhugemem #1 SMP Tue Oct 18 18:49:54
EDT 2005 i686 i686 i386 GNU/Linux

Any link to the bug is appreciated. I am also experiencing severe
IOWAIT times. VMSTAT shows 80% of time spent waiting for IO. Not sure
if this could cause the problem.

Thanks,

Michel Esber wrote:

libaio-0.3.103-3
libaio-devel-0.3.103-3

One new info: seems like this box has been upgraded from a Linux RedHat
2.1 to RH 4.0 Update 2. All filesystems are ext3 though. Could this be
a possible reason?

My other servers (where I have no problems) had a clean installation of
RH 4.0.

Thanks

More:
What does uname -a say ?
Are you using a 32-bit or 64-bit instance ?

It seems there is a recently discovered issue with AIO on RH4 on 32bit.

Cheers
Serge
--
Serge Rielau
DB2 Solutions Development
DB2 UDB for Linux, Unix, Windows
IBM Toronto Lab

[Michel Esber]

Ooops I forgot to mention. 32-bit instance.

Thanks,

[Serge Rielau]

Michel Esber wrote:

Ooops I forgot to mention. 32-bit instance.

Thanks,

The known issues don't apply to this situation.

Here is what the Linux Technology Center says:
" Maybe /proc/sys/fs/aio-max-nr, the system-wide maximum number of aio
requests, is too small. What is it set to? Can you try bumping it
up? A related value is /proc/sys/fs/aio-nr, the current system-wide
number of aio requests. The maxevents parm of io_queue_init() is
added to aio-nr and the sum must not exceed aio-max-nr; otherwise
-EAGAIN would be returned. "

Another possible cause could be running out of kernel memory. Please
provide /proc/slabinfo and /proc/meminfo at time of failure as well as
/var/log/messages.

The fact that the machine was upgraded from RH 2.1 to RHEL 4 could have
something to do with it. Red Hat recommends fresh installs.

BTW, I think this is turning out to be a bit of an expedition. Can you
open a PMR? It will capture the resolution for future requests and also
make sure the problem is tracked beyond my in-box....

Cheers
Serge
--
Serge Rielau
DB2 Solutions Development
DB2 UDB for Linux, Unix, Windows
IBM Toronto Lab