KAIO: error in kaio_WRITE, kaiocbp = 0xfb84a1d0, errno = 0

Hi all, IDS 10.00.FC5 OS: Red Hat Enterprise Linux AS release 4 (Nahant Update 4 (2.6.9-42.ELsmp with Hyper threading, 64 bit) HDR secondary has crashed a few times with the following errors coming up in the online.log: 13:36:21 KAIO: error in kaio_WRITE, kaiocbp = 0xfb84a1d0, errno = 0 13:36:21 fildes = 264 (gfd 7), buf = 0x0xfb857000, nbytes = 8192, offset = 276310016 < thousand of repeated lines with different values and then finally... > 13:36:47 KAIO: out of OS resources, errno = 11, pid = 9298 13:37:03 KAIO: out of OS resources, errno = 11, pid = 9298 < and then crash > The HDR secondary is identical to the HDR Primary with the only difference being that it's using local disks and the Primary is using an HP SAN. Both instances are using KAIO with block devices (supported since kernel 2.6). The kernel parameters have been set as recommended in the machine notes. After the first crash we increased the fs.aio-max-nr kernel parameter from it's default value of 64K to 128K and bounced the instance. I have no measures from fs.aio-nr from the time of the crashes to see if it ever grew at all. There's also nothing in the OS logs to indicate a problem. I've come across Oracle setup documentation for Linux where they recommend setting fs.aio-max-nr to 1M but I haven't found any info regarding this for IDS. Does anyone have any experience with this? The only instance-wide changes done previous to the first crash were the following: RESIDENT from 1 to 2 SHMVIRTSIZE from 2048000 to 2560000 STACKSIZE from 32 to 128 RA_THRESHOLD from 16 to 4 BUFFERPOOL from size=2K,buffers=1500000,lrus=17,lru_min_dirty=50.000000,lru_max_dirty=60.000000 to size=2K,buffers=1500000,lrus=127,lru_min_dirty=50.000000,lru_max_dirty=60.000000 The box has 8 CPUs (Hyper Threading enabled), 10GB of RAM (5.6GB used by instance) and it hasn't been seen to be swapping. Any suggestions? Cheers, RoB