DIRECT I/O (KAIO) Disabled in RHEL 9

Installing Informix 14.10.FC10 in a new linux RHEL 9 server. KAIO is disabled and I don't know the reason: rpm -qa | grep libaio libaio-0.3.111-13.el9.x86_64 ls -l $INFORMIXDIR/bin/oninit -rwsr-sr--. 1 root informix 28388480 Mar 8 01:38 /home/informix/pdbwic2016_dev/bin/oninit onstat -g cfg DIRECT_IO IBM Informix Dynamic Server Version 14.10.FC10AEE -- On-Line -- Up 00:00:12 -- 16570916 Kbytes 2023-03-22 18:32:32name current value DIRECT_IO 0x5mount | grep nvme /dev/md0 on /nvme0 type ext2 (rw,relatime,stripe=24) Also tested with XFS fileystem And the message in online.log 18:17:56 IBM Informix Dynamic Server Version 14.10.FC10AEE 18:17:56 Performance Advisory: Currently IBM Informix Dynamic Server cannotsupport KAIO on the device containing '/INFORMIXDEV/pdbwic2016_dev/rootdbs' 18:17:56 Results: Direct and concurrent I/O are disabled for this chunk. 18:17:56 Action: Standard AIO (possibly buffered) will be used for I/Oto this chunk. 18:17:56 IBM Informix Dynamic Server Initialized -- Shared Memory Initialized.In theory RHEL 9 is supported by 14.10.FC10 (I derected some curses library links are detected to dbaccess to work) ------------------------------ Vicente Salvador Cubedo ------------------------------

Trying to create root dbx chunk directly on a device, shows a better understanding message: 18:41:19 IBM Informix Dynamic Server Version 14.10.FC10AEE 18:41:19 Performance Advisory: Currently IBM Informix Dynamic Server cannotsupport KAIO on the device containing '/dev/md0' Its block size (4096) is larger than 1024. 18:41:19 Results: Direct and concurrent I/O are disabled for this chunk. 18:41:19 Action: Standard AIO (possibly buffered) will be used for I/Oto this chunk. So, it seams than informis only allows to use DIRECT/IO y block size is lower than 1024 bytes. My disks have sector size of 4096 bytes. Is there a workaround? ------------------------------ Vicente Salvador Cubedo ------------------------------

Vicente: There are several things that can prevent a chunk from participating in DIRECT_IO: Cooked devices do not all participate. Some filesystems do not support direct_io There may not be sufficient KAIO resources allocated in the OS. Check with: cat /proc/sys/fs/aio-max-nr You want to see about 10,000 per CPU VP. You can change the setting by echoing the new value to it as root: sudo echo 129734 >/proc/sys/fs/aio-max-nr What does this chunk look like? ls -l -L /INFORMIXDEV/pdbwic2016_dev/rootdbs Art ------------------------------ Art S. Kagel, President and Principal Consultant ASK Database Management Corp. www.askdbmgt.com ------------------------------

It's not related to the server. It's related to the filesystem or underlaying characteristics. In same server, placing the rootdbs cooked file in one filesystem or other makes KAIO to work or not to work. In this filesystem, KAIO works # fdisk -l /dev/sda Disk /dev/sda: 19.64 TiB, 21598316789760 bytes, 42184212480 sectors Disk model: PERC H755 Front Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 262144 bytes / 1048576 bytes Disklabel type: gpt Disk identifier: 650B3DF9-63AC-4B78-9129-01DEBC46A3A6 Device Start End Sectors Size Type /dev/sda1 2048 42184210431 42184208384 19.6T Linux filesystem # xfs_info /dev/sda1 meta-data=/dev/sda1 isize=512 agcount=32, agsize=164782016 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 bigtime=1 inobtcount=1 data = bsize=4096 blocks=5273024512, imaxpct=5 = sunit=64 swidth=256 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=521728, version=2 = sectsz=512 sunit=64 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 In this filesystem KAIO does not work # fdisk -l /dev/md0 Disk /dev/md0: 5.24 TiB, 5760724500480 bytes, 1406426880 sectors Units: sectors of 1 * 4096 = 4096 bytes Sector size (logical/physical): 4096 bytes / 4096 bytes I/O size (minimum/optimal): 32768 bytes / 98304 bytes # xfs_info /dev/md0 meta-data=/dev/md0 isize=512 agcount=32, agsize=43950832 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 bigtime=1 inobtcount=1 data = bsize=4096 blocks=1406426624, imaxpct=5 = sunit=8 swidth=24 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=521728, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Only diference between filesystems is sector size. But this comes from undelaying hardware (disk characteristics). Can this be considered a bug or there are some technical restrictions preventing to use disk with sector size 4096 for Informix? ------------------------------ Vicente Salvador Cubedo ------------------------------

Vicente: Interesting. I have never seen this one before. Definitely worth a PMR to see what IBM thinks of it. Art ------------------------------ Art S. Kagel, President and Principal Consultant ASK Database Management Corp. www.askdbmgt.com ------------------------------

The direct I/O on Linux requires alignment on the block boundary, so for the devices where block size exceeds 512 (or, maybe, 1024) bytes (for example advanced format devices with no 512 byte emulation) Informix will disable direct I/O. Since it's an "md" device - I wonder if it's possible to recreate it with 1k or 512 byte block ? You can check block size for devices by something like: lsblk -o NAME,PHY-SEC,LOG-SEC and/or blockdev --getss /dev/xxx blockdev --getpbsz /dev/xxx blockdev --getbsz /dev/xxx I think the "important" one is the logical block (sector) size (getss), but I may be mistaken. ------------------------------ Vladimir Kolobrodov ------------------------------

Doesn't /dev/md0 SUGGEST some form of RAID? ------------------------------ Jon Ritson ------------------------------

Hi Vladimir, I've made al kind of testing and I identified the DIRECT I/O disabled by Informix in related to logical disk sector size, no matter if this disks are joined in a RAID or accessed directly or the filesystem used: ext2, ext3 or xfs If the hard disk has a logical sector size of 4096 bytes Informix disables Direct I/O and if the hard disk has a logical sector size of 512 bytes, Direct I/O works. So next question is, Is this related to some hardcoded value inside Informix only working with 512 bytes blocs. Modern Linux allows to use Direct I/O with other sector sizes. Software should align to the size of the disk sector, but it is possible. https://access.redhat.com/articles/3911611 In resume: O_DIRECT requires that I/O occur in multiples of 512 bytes and from memory aligned on a 512-byte boundary because O_DIRECT performs direct memory access (DMA) straight to the backing store , bypassing any intermediate buffers. Performing unaligned transfers would require "fixing up" the I/O by explicitly aligning the user-space buffer and zeroing out the slack (the space between the end of the buffer and the next 512 byte multiple), obviating the benefits of O_DIRECT . Here's another way to look at it: The "beauty" of O_DIRECT (such as it is) is that it cuts out the VM from the I/O process. No copying from user to kernel-space, no page cache, no pages period. But this means all the little things that the kernel handles for you-alignment being the biggest-you, the user, now need to handle. The underlying backing store expects everything in sectors, so you need to talk in sectors, too. That is why the magic number isn't always 512 bytes. It is the sector size of the underlying block device. This number is almost always 512 bytes for a standard hard drive, but it can vary. For example, ISO 9660, the standard format of CD-ROMs, sports 2048 byte sectors. You can use the BLKSSZGET ioctl to obtain a backing store's sector size. Portable code should use the value returned from this ioctl and not hard-code 512 bytes. So, I think there is maybe an issue in Informix just using a hardcoded 512 value and disabling Direct I/O if sector size is not 512 instead of reading the logical sector size and using it. Regards, ------------------------------ Vicente Salvador Cubedo ------------------------------

You might be able to validate this with "losetup" ... (at your own risk / adventure as most likely a destructive exercise!): Set up a loop device: losetup [-o offset] [--sizelimit size] [--sector-size size] [-Pr] [--show] -f loopdev file losetup --sectorsize 512 --show -f loopdev /dev/md0 Then use the loopdev device for an xfs filesystem etc. etc. I have NOT done this :D ------------------------------ Jon Ritson ------------------------------

Hi Vicente, > ... related to logical disk sector size, no matter if ... Not entirely true - on Linux you can create a loopback device on top of the 4k device, the loop device will have 512 byte block and the direct I/O will "work". Performance will be awful, of course. I am not sure if different (smaller) block size can be forced for the "md" device and/or the file system when the physical storage has 4k logical block, and even if yes - what would be the performance impact. (with the loopback - impact is considerable) > ... Is this related to some hardcoded value inside Informix ... It's a bit more complex than that. On (x86/AMD) Linux - the Informix base page size is 2k, so, in theory, the minimum I/O could be 2k aligned, however - I think that your suspicion is correct and any logical block size over 1k (1024 bytes) will result in Informix turning direct I/O off. Where 1k is the hardcoded value. On (x86/AMD) Linux it would be somewhat difficult to enable direct I/O on 4k devices for the rootdbs and logs because the base page size (minimum I/O) will be less that that. However, in my opinion, it should be possible to enable direct I/O for chinks with custom page sizes which are equal or multiples of the device page size. Until recently it was not a (big) problem - since most "advanced format devices" (and SSDs) provided 512 byte block emulation, for some SSDs you can actually control the (reported) logical block size with vendor tools. I think having support case about this issue may prompt development to take another look at the direct I/O on Linux. ------------------------------ Vladimir Kolobrodov ------------------------------

I suspect a code change is required 4K Sector Support had to be added to DB2 - https://www.ibm.com/docs/en/db2/11.5?topic=115-performance-enhancements - Even with 4K Sector Support in DB2 it appears Direct I/O is not supported: https://www.ibm.com/docs/en/db2/11.5?topic=variables-performance#r0005665__P_DB2_4K_DEVICE_SUPPORT The use of database managed (DMS) table spaces that are configured for direct disk (raw) access is not supported. The use of database managed (DMS) table spaces that are configured for direct disk (raw) access is not supported. The use of system managed (SMS) table spaces that are configured with NO FILESYSTEM CACHING is not supported. ------------------------------ David Williams ------------------------------

Also note: APAR IT43456: KAIO NOT SUPPORTED WITH SECTOR_SIZES GREATER THAN 1KB ------------------------------ David Williams ------------------------------

Hi Vicente, Have you checked if this problem still exists in version 14.10.13(14.10.FC13) on linux RHEL 9 server? Regarding fix list: APAR IT43456: KAIO NOT SUPPORTED WITH SECTOR_SIZES GREATER THAN 1KB was fixed. Regarding docs(new features): KAIO support for large block format devices: Direct I/O is enhanced to support devices with 2k and 4k block sizes. ... Device Block Sizes on Linux: Prior to 14.10.xC13, Direct I/O on Linux was supported only on storage devices with a native or emulated 512-byte block size. Starting from 14.10.xC13, Direct I/O is also supported on devices with 2k and 4k block sizes. Or someone else.. Thanks, ------------------------------ Boycho Velkov ------------------------------

FC13 just released a few days ago. Still not verified but I'll do soon. Keep you informed Vicente

Hi Boycho, Despite my last post about kio class not present. It was my fault. Currently tested with dbspaces with 8k and 4k page sizes and it works like a charm on NVMe disk with 4k sectors and RAID 1 using mdadm. All my previous founding about new messages alerting the reason for not enabling KAIO is very nice. AIO I/O vps: class/vp/id s io/s totalops dskread dskwrite dskcopy wakeups io/wup errors tempops fifo 7 0 i 0.0 0 0 0 0 1 0.0 0 0 kio -1 0 i 5.4 86 78 8 0 169 0.5 0 0 msc 6 0 i 0.4 7 0 0 0 4 1.8 0 7 aio 5 0 i 44.2 707 622 11 0 687 1.0 0 515 aio 9 1 i 2.1 33 29 3 0 31 1.1 0 29 aio 10 2 i 1.2 19 15 3 0 25 0.8 0 15 aio 11 3 i 1.1 17 14 2 0 20 0.9 0 14 aio 12 4 i 0.9 15 12 2 0 14 1.1 0 12 aio 13 5 i 0.6 10 5 4 0 15 0.7 0 5 aio 14 6 i 0.3 5 2 2 0 8 0.6 0 2 aio 15 7 i 0.1 1 0 1 0 6 0.2 0 0 pio 4 0 i 0.1 2 0 2 0 3 0.7 0 2 lio 3 0 i 0.7 11 0 11 0 12 0.9 0 11 onstat -d IBM Informix Dynamic Server Version 14.10.FC13W1DE -- On-Line -- Up 00:05:33 -- 376364 Kbytes 2026-02-01 15:41:55Dbspaces address number flags fchunk nchunks pgsize flags owner name 45165028 1 0x4030001 1 1 2048 N BA informix rootdbs 45166b90 2 0x5030001 2 1 2048 N PBA informix plog 452a6850 3 0x4020001 3 1 2048 N BA informix llog 452a6aa8 4 0x20001 4 1 4096 N BA informix datadbs1 452a6d00 5 0x4020001 5 1 2048 N BA informix datadbs2 4599d028 6 0x4020001 6 1 2048 N BA informix datadbs3 4599d280 7 0x4020001 7 1 8192 N BA informix data8dbs1 4599d4d8 8 0x4020001 8 1 8192 N BA informix data8dbs2 4599d730 9 0x4020001 9 1 8192 N BA informix data8dbs3 4599d988 10 0x2001 10 1 8192 N TBA informix tmpdbspace 4599dbe0 11 0x4028001 11 1 2048 N SBA informix sbspace1 4599e028 12 0xa001 12 1 2048 N UBA informix tmpsbspace 12 active, 2047 maximum Chunks address chunk/dbs offset size free bpages flags pathname 45165280 1 1 0 150000 138189 PO-B-- /home/informix/ol_informix1410/dbspaces/rootdbs 4599e280 2 2 0 32768 0 PO-BE- /home/informix/ol_informix1410/dbspaces/ol_informix1410_plog_p_1 4599f028 3 3 0 32768 15 PO-BE- /home/informix/ol_informix1410/dbspaces/ol_informix1410_llog_p_1 459a0028 4 4 0 16384 15337 PO-B-D /home/informix/ol_informix1410/dbspaces/ol_informix1410_datadbs1_p_1 45ad3028 5 5 0 32768 32715 PO-BE- /home/informix/ol_informix1410/dbspaces/ol_informix1410_datadbs2_p_1 45ad4028 6 6 0 32768 32715 PO-BE- /home/informix/ol_informix1410/dbspaces/ol_informix1410_datadbs3_p_1 45ad5028 7 7 0 8192 8139 PO-BED /home/informix/ol_informix1410/dbspaces/ol_informix1410_data8dbs1_p_1 45ad6028 8 8 0 8192 8139 PO-BED /home/informix/ol_informix1410/dbspaces/ol_informix1410_data8dbs2_p_1 45ad7028 9 9 0 8192 8139 PO-BED /home/informix/ol_informix1410/dbspaces/ol_informix1410_data8dbs3_p_1 45ad8028 10 10 0 8192 8139 PO-BE- /home/informix/ol_informix1410/dbspaces/ol_informix1410_tmpdbspace_p_1 45ad9028 11 11 0 32768 23802 23802 POSB-- /home/informix/ol_informix1410/dbspaces/ol_informix1410_sbspace1_p_1 Metadata 1745 8466 1745 45ada028 12 12 0 32768 23802 23802 POSB-- /home/informix/ol_informix1410/dbspaces/ol_informix1410_tmpsbspace_p_1 Metadata 1745 8466 1745 12 active, 32766 maximum ------------------------------ Vicente Salvador Cubedo Software Architect DEISTER, S.A. Barcelona +34 932063298 ------------------------------

Hi Vicente, So, you have 1 KAIO for all chunks with page size 4K/8K(because of VPCLASS cpu,num=1) and one AIO for every chunk with page size = 2K. Informix will use AIO for every chunk with page size smaller than 4K. Thanks a lot for info. Cheers, ------------------------------ Boycho Velkov ------------------------------

No. If DIRECT_IO is set then all chunks built on filesystem files, regardless of page size, are handled by the KAIO threads, one in each CPU VP. It's just that the example has only one CPU VP. The problem, and the fix, are about file on filesystems that have a block size larger than 2k. Those had previously not been handled by KAIO threads, but are in the latest update. It has nothing to do with the dbspace's pagesize. ------------------------------ Art S. Kagel, President and Principal Consultant ASK Database Management Corp. www.askdbmgt.com ------------------------------

Hi Art, I meant that in the output from the "onstat -d" command for the cooked files that make up the dbspaces with 2K page, the D flag is missing in the 6th position. Yes, it depends on the value of the DIRECT_IO parameter. Similar message may appear in the "online.log" file: Performance Advisory: Currently IBM Informix Dynamic Server cannot support KAIO on the device containing '/home/informix/ol_informix1410/dbspaces/ol_informix1410_datadbs2_p_1' Its Sector Size (4096) and the page size (2048) are incompatible. Anyway. I prefer to use raw devices for chunks and when I see more than 1 AIO VP I feel uncomfortable ;-) I was just asking for information about this feature in 14.10.FC13. Thanks again to Vicente for the information. Have a nice day, ------------------------------ Boycho Velkov ------------------------------

Yes. Endeed KAIO class vcpu is related to the number of VPCLASS of type cpu you have. In my case just for testing VPCLASS cpu,num=1 Let me say that Informix will use AIO for every chunk which its size is not multiple of disk sector size (even if mapped to a mdadm RAID). So, for 4k sector sizes, only 4k, 8k, 12k, 16k page sizes will use KAIO other sizes will use AIO Informix will use AIO for every dbspace with page sizes not multiple of disk sector sizes ------------------------------ Vicente Salvador Cubedo Software Architect DEISTER, S.A. Barcelona +34 932063298 ------------------------------

Thanks Vicente, Have a nice day, ------------------------------ Boycho Velkov ------------------------------

Boycho: Just a note on your comment that "when I see more than 1 AIO VP I feel uncomfortable". Note that the AIO VPs handle all flat file IO that is not under DIRECT_IO. That includes: The message log The console log Audit files af file output SET EXPLAIN output and a handful of less common IO So, you almost always need at least two AIO VPs, but the metric to watch is the io/wkup column in the onstat -g iov report. If at least one AIO VP has io/wkup less than 1.0 you have enough. If most are less than 1.0, you have more than you need. If they are all at 1.0 or above, then you need more to avoid any flat file IOs having to wait to be serviced.Typical production instances will usually do well with between four and six AIO VPs. In Vicente's test instance there were WAY to many AIO VPs. Art ------------------------------ Art S. Kagel, President and Principal Consultant ASK Database Management Corp. www.askdbmgt.com ------------------------------