KAIO not enabling on a device with 4096B block size

Hello everyone, It seems others have run into this issue recently (this year) and now I am running into it. KAIO is not enabling on a chunk that sits on a device with a 4096 B block size. 14.10.FC9WE running on Ubuntu LTS 22.04. root@informix03:/# blockdev --getbsz /dev/sdb1 4096 (virtual server SSD disk in some cloud) 2023-09-14 18:27:07.931 IBM Informix Dynamic Server Version 14.10.FC9WE 2023-09-14 18:27:07.932 Performance Advisory: Currently IBM Informix Dynamic Server cannot support KAIO on the device containing '/ilink/prod/root' (this is on /dev/sdb1, cooked chunk, on an ext2 partition) 2023-09-14 18:27:07.932 Results: Direct and concurrent I/O are disabled for this chunk. 2023-09-14 18:27:07.932 Action: Standard AIO (possibly buffered) will be used for I/O to this chunk. I suspect there is no fix to this in the software yet (14.10.FC10 ?). If you ran into this, what did you do? Thank you, Hal Hal Maner M Systems International, Inc. hmaner@msystemsintl.com www.msystemsintl.com

Hal, I am not sure this is related to the block size of the underlying disk, but there is some kind of defect from my point of view. I can reproduce this when the device has ext2 filesystem, also on ext4 09:32:20 IBM Informix Dynamic Server Version 14.10.FC10IE 09:32:20 Performance Advisory: Currently IBM Informix Dynamic Server cannotsupport KAIO on the device containing '/srv/dbstest/rootdbs' 09:32:20 Results: Direct and concurrent I/O are disabled for this chunk. 09:32:20 Action: Standard AIO (possibly buffered) will be used for I/Oto this chunk. BUT if I directly use the block device as rootdbs (so not a file), this does not happen. 09:39:08 IBM Informix Dynamic Server Version 14.10.FC10IE 09:39:08 VP 1 is requesting 16384 kaio event resources. 09:39:27 IBM Informix Dynamic Server Initialized -- Complete Disk Initialized.# ls -lsinsgesamt 0 0 lrwxrwxrwx 1 root root 24 Sep 16 09:38 rootdbs -> /dev/mapper/vg01-dbstest # blockdev --getbsz /dev/mapper/vg01-dbstest 4096 (this is a lvm mapped device on a NVME which has also 4k block size). So there is an issue with the filesystem approach rather than the block size from my point of view. I believe only IBM support can answer this. I tested this in Ubuntu 20.04 and got the same results when initializing a rootdbs on a ext4 fs. (same message as above that KAIO cannot be supported). Not sure if a different OS is behaving correctly. Best,

I was curious and installed a 14.10 Dev version in a fresh Fedora 38 instance. Same error occurred there when initializing the rootdbs. So no Ubuntu issue. Kernel 6.2.9 in the Fedora, 6.2.0 on my Ubuntu 22, 5.15 in my Ubuntu 20 test. Time to file a bug I would say.

Interestingly, this seems to happen only at initialization time. When restarting the instance, kaio is started without any error in the logs. Maybe that's why I did not recognize it before. Also onstat -g iov shows kio vps correctly (not after oninit -i, but after restart). Directliy after oninit -i: onstat -g iov IBM Informix Dynamic Server Version 14.10.FC10IE -- On-Line -- Up 00:00:32 -- 1227304 Kbytes 2023-09-16 13:08:16 AIO I/O vps:class/vp/id s io/s totalops dskread dskwrite dskcopy wakeups io/wup errors tempops fifo 7 0 i 0.0 0 0 0 0 1 0.0 0 0 msc 6 0 i 0.0 1 0 0 0 2 0.5 0 1 aio 5 0 s 289.3 8967 3287 5530 0 8588 1.0 0 1427 pio 4 0 i 0.2 5 0 5 0 6 0.8 0 5 lio 3 0 i 64.9 2011 0 2011 0 2012 1.0 0 2011 After a restart (KAIOON=1000): 13:08:32 Event notification facility epoll enabled. 13:08:32 IBM Informix Dynamic Server Version 14.10.FC10IE 13:08:32 VP 8 is requesting 1000 kaio event resources. 13:08:32 IBM Informix Dynamic Server Initialized -- Shared Memory Initialized.onstat -g iov IBM Informix Dynamic Server Version 14.10.FC10IE -- On-Line -- Up 00:00:16 -- 1227304 Kbytes 2023-09-16 13:08:42 AIO I/O vps:class/vp/id s io/s totalops dskread dskwrite dskcopy wakeups io/wup errors tempops fifo 7 0 i 0.0 0 0 0 0 1 0.0 0 0 kio -1 0 i 92.5 1480 313 1167 0 2644 0.6 0 0 kio -1 1 i 35.3 565 312 253 0 1087 0.5 0 0 msc 6 0 i 0.2 3 0 0 0 4 0.8 0 3 aio 5 0 i 98.9 1583 1194 226 0 1584 1.0 0 0 pio 4 0 i 0.0 0 0 0 0 1 0.0 0 0 lio 3 0 i 0.0 0 0 0 0 1 0.0 0 0 Weird behaviour.

Marcus, Thank you very much for testing this and your feedback. I was hopeful that all would be well with KAIO after a restart like your case but unfortunately the same messages in online.log persist after a restart in my case and my onstat -g iov output is the same as before restart and I see no kio in that output. Hal Maner

Hmm, this is strange. Did you check all the prerequisites? Libaio needs to be installed, direct_io setting should be 1, listeners should have the NET type, and aio-max-nr needs to be bigger than KAIOON value. Can you test with ext4 or with direct device links? Sorry, no better ideas.... Best, Marcus Gesendet von Outlook für Android

Yes, all prerequisites are there. No, I need to keep the file system and the chunks as is. Thank you, Hal

This is already discussed in this forum. At this moment KAIO is disabled with sector sizes larger than 1k This is due to standard page size of Informix in Linux is 2k and page size should be aligned to disk sector sizes. There is a PMR request to enable KAIO on dbspaces with sector sizes multiple of sector sizes (e.g. 4k, 8k, 12k or 16k) Vicente Salvador

Vicente, thank you. it does not sound like it, but, Is this a relatively new "bug" or weakness with IDS 14? I ask because I did not run into such a problem with the predecessor of this server that I had setup back in 2020 using 12.10.FC4 on Linux. I checked and the device block size is also 4096 there yet KAIO has been running with basically the same configuration and onstat -g iov shows kio. Hal

Vicente, can you then explain to me why this is working on my system when database is restarted ? On initial allocation of the rootdbs with oninit -i, this does not work, resulting in the discussed warning. When restarting, KAIO is established without warning and I can see it in onstat -g iov. For Hal, this does not seem to work, maybe because the filesystem does not start a 4k boundary ? I am running the filesystem on a lvm based subpartition, formatted with ext2. Maybe my lvm setup is aligned to a 4k page boundary and works because of this ? Informix is setup with 2k pages here (standard, new installation). (file descriptor is shown in fdinfo as 0150002) 040000 O_DIRECT 010000 O_DSYNC 02 RDWR 0100000 O_LARGEFILE Immediately after initialization, the mode is 0110002. (O_DIRECT is missing) Thank you for your opinon.

You can find a good answer from Vladimir in this post: https://community.ibm.com/community/user/datamanagement/discussion/direct-io-kaio-disabled-in-rhel-9 Vicente Salvador

Hi Vicente, I read the posting and found that it is related to hardware rather than filesystem type. My device shows a 512bytes block size, which means that KAIO should be working. My question was, why does it not work at initialisation time ? I am getting the mentioned warning in the online.log, and after a restart the KAIO is working correctly. lsblk -o NAME,PHY-SEC,LOG-SEC /dev/nvme0n1p1 NAME PHY-SEC LOG-SEC nvme0n1p1 512 512 For Hal, this is not working at all, probably because the underlying disk is a 4k block size media. Maybe he can check the media with lsblk in order to see what type of hardware is installed. I remember now that our admins tried to get hardware with smaller block sizes for the IDS systems. So this is understandable, knowing there is an outstanding APAR on this issue. On my machine, it is working in general, but not at initialization time (which is not a real issue probably, because typically we are restarting the engine some times until it gets productive, but we could maybe forget about it, resulting in a problematic instance until we are restarting).