HOW-TO setup block devices with O_DIRECT on RHEL5

I was looking for the information on how setup and use Informix block devices on Red Hat Enterprise Linux 5, since that OS version discontinues raw devices. I have found no document that entirely fulfilled my needs, so I've investigated by myself and figured out the solution. Since it can be of interest for others as well, I am posting my findings here - hope it can be useful. Background - what are raw devices and block devices? * RAW DEVICE (or CHARACTER DEVICE) = devices through which data is transmitted one character at a time, using unbuffered input and output routines; such devices are managed by using direct access and bypassing the OS layer and its cache; raw devices are not mounted and don't have any filesystem on them. * BLOCK DEVICE = devices through which data is transmitted in the form of blocks, the most significant difference between block and character devices is that block devices use buffered input and output routines; normally a block device is a mounted partition with a filesystem. With IDS 10 and any Linux kernel greater or equal to 2.6 (e.g. RHEL4 onwards), it is (automatically) possible to use block devices with O_DIRECT access method, which makes block devices to be used as if they were raw devices (unbuffered access via KAIO - Kernel Asynchronous I/O). Source: http://www-128.ibm.com/developerworks/db2/library/techarticle/dm-0503szabo/ Here is how I managed to correctly configure block devices to be used for Informix on RHEL5: 1) If the Informix block devices are currently mounted, do 'df -k' and redirect the output to a file, to create a permanent record of the correspondence between the physical device (e.g. /dev/cciss/cXdXpX or /dev/mdX or whatever else, depending if you are using HW or SW mirroring or nothing) and the partition name. If they were not mounted, it is assumed you know which physical devices you have allocated for Informix during the disks partitioning. 2) If mounted, 'umount' the Informix block devices comment their corresponding entries in /etc/fstab to prevent them to be mounted when server is rebooted. 3) For easier usability of Informix block devices, create symbolic links under /etc/udev/devices, e.g. ln -s /dev/cciss/c1d1p8 /etc/udev/devices/online_logdbs ln -s /dev/cciss/c1d0p2 /etc/udev/devices/online_rootdbs01 ln -s /dev/cciss/c1d1p5 /etc/udev/devices/online_rootdbs02 ln -s /dev/cciss/c1d1p9 /etc/udev/devices/online_tempdbs the above setting makes the symbolic links (which by default will be created under /dev once 'udev' service is [re]started) to be persistent across reboots. 4) Set persistent ownership and permissions for the Informix physical devices, by adding the proper entries to /etc/udev/rules.d/50-udev.rules, e.g.: ---------------------------------------------------------- # Informix block devices configuration KERNEL=="cciss/c1d1p8", NAME="%k", OWNER="informix", GROUP="informix", MODE="0660", OPTIONS="last_rule" KERNEL=="cciss/c1d0p2", NAME="%k", OWNER="informix", GROUP="informix", MODE="0660", OPTIONS="last_rule" KERNEL=="cciss/c1d1p5", NAME="%k", OWNER="informix", GROUP="informix", MODE="0660", OPTIONS="last_rule" KERNEL=="cciss/c1d1p9", NAME="%k", OWNER="informix", GROUP="informix", MODE="0660", OPTIONS="last_rule" # End of Informix block devices configuration ---------------------------------------------------------- 5) [re]start udev (or reboot the machine) ---------------------------------------------------------- /sbin/start_udev ---------------------------------------------------------- 6) check result: ---------------------------------------------------------- [root@zeus dev]# ls -la /dev/online_* lrwxrwxrwx 1 root root 17 May 3 17:49 /dev/online_logdbs -> /dev/cciss/c1d1p8 lrwxrwxrwx 1 root root 17 May 3 17:49 /dev/online_rootdbs01 -> /dev/cciss/c1d0p2 lrwxrwxrwx 1 root root 17 May 3 17:49 /dev/online_rootdbs02 -> /dev/cciss/c1d1p5 lrwxrwxrwx 1 root root 17 May 3 17:49 /dev/online_tempdbs -> /dev/cciss/c1d1p9 [root@zeus dev]# ls -lL /dev/online_* brw-rw---- 1 informix informix 105, 24 May 3 16:02 /dev/online_logdbs brw-rw---- 1 informix informix 105, 2 May 3 16:02 /dev/online_rootdbs01 brw-rw---- 1 informix informix 105, 21 May 3 16:02 /dev/online_rootdbs02 brw-rw---- 1 informix informix 105, 25 May 3 16:02 /dev/online_tempdbs ---------------------------------------------------------- Note the "b" prefix for the block devices. Should work, at least it does for me. Cheers Rupan3rd (from rainy Italy) P.S. IDS 10 is not yet officially certified for RHEL5 but AFAICS it seems to work flawlessly.

Great write-up. I always appreciate it when someone submits summaries like yours for the benefit of the group. Thanks. Someday I'm going to Italy; rain or shine! Mike Badar ESRI ArcSDE for Informix UNIX administrator 1 International Ct. Broomfield, CO 80021 mbadar@esri.com > -----Original Message----- > From: informix-list-bounces@iiug.org > [mailto:informix-list-bounces@iiug.org] On Behalf Of Rupan3rd > Sent: Friday, May 04, 2007 5:03 AM > To: informix-list@iiug.org > Subject: HOW-TO setup block devices with O_DIRECT on RHEL5 > > I was looking for the information on how setup and use > Informix block devices on Red Hat Enterprise Linux 5, since > that OS version discontinues raw devices. > > I have found no document that entirely fulfilled my needs, so > I've investigated by myself and figured out the solution. > Since it can be of interest for others as well, I am posting > my findings here - hope it can be useful. > > Background - what are raw devices and block devices? > > * RAW DEVICE (or CHARACTER DEVICE) = devices through which > data is transmitted > one character at a time, using unbuffered input and output > routines; such > devices are managed by using direct access and bypassing > the OS layer and > its cache; raw devices are not mounted and don't have any > filesystem on them. > > * BLOCK DEVICE = devices through which data is transmitted in > the form of > blocks, the most significant difference between block and > character devices > is that block devices use buffered input and output > routines; normally a block > device is a mounted partition with a filesystem. > > With IDS 10 and any Linux kernel greater or equal to 2.6 > (e.g. RHEL4 onwards), it is (automatically) possible to use > block devices with O_DIRECT access method, which makes block > devices to be used as if they were raw devices (unbuffered > access via KAIO - Kernel Asynchronous I/O). Source: > http://www-128.ibm.com/developerworks/db2/library/techarticle/ > dm-0503szabo/ > > > Here is how I managed to correctly configure block devices to > be used for Informix on RHEL5: > > 1) If the Informix block devices are currently mounted, do > 'df -k' and redirect > the output to a file, to create a permanent record of the > correspondence > between the physical device (e.g. /dev/cciss/cXdXpX or > /dev/mdX or whatever > else, depending if you are using HW or SW mirroring or > nothing) and the > partition name. If they were not mounted, it is assumed > you know which > physical devices you have allocated for Informix during the disks > partitioning. > > 2) If mounted, 'umount' the Informix block devices comment > their corresponding > entries in /etc/fstab to prevent them to be mounted when > server is rebooted. > > 3) For easier usability of Informix block devices, create > symbolic links under > /etc/udev/devices, e.g. > > ln -s /dev/cciss/c1d1p8 /etc/udev/devices/online_logdbs > ln -s /dev/cciss/c1d0p2 /etc/udev/devices/online_rootdbs01 > ln -s /dev/cciss/c1d1p5 /etc/udev/devices/online_rootdbs02 > ln -s /dev/cciss/c1d1p9 /etc/udev/devices/online_tempdbs > > the above setting makes the symbolic links (which by > default will be created > under /dev once 'udev' service is [re]started) to be > persistent across > reboots. > > > 4) Set persistent ownership and permissions for the Informix > physical devices, > by adding the proper entries to > /etc/udev/rules.d/50-udev.rules, e.g.: > ---------------------------------------------------------- > # Informix block devices configuration > KERNEL=="cciss/c1d1p8", NAME="%k", OWNER="informix", > GROUP="informix", > MODE="0660", OPTIONS="last_rule" > KERNEL=="cciss/c1d0p2", NAME="%k", OWNER="informix", > GROUP="informix", > MODE="0660", OPTIONS="last_rule" > KERNEL=="cciss/c1d1p5", NAME="%k", OWNER="informix", > GROUP="informix", > MODE="0660", OPTIONS="last_rule" > KERNEL=="cciss/c1d1p9", NAME="%k", OWNER="informix", > GROUP="informix", > MODE="0660", OPTIONS="last_rule" > # End of Informix block devices configuration > ---------------------------------------------------------- > > 5) [re]start udev (or reboot the machine) > ---------------------------------------------------------- > /sbin/start_udev > ---------------------------------------------------------- > > 6) check result: > ---------------------------------------------------------- > [root@zeus dev]# ls -la /dev/online_* > lrwxrwxrwx 1 root root 17 May 3 17:49 /dev/online_logdbs -> > /dev/cciss/c1d1p8 lrwxrwxrwx 1 root root 17 May 3 17:49 > /dev/online_rootdbs01 -> /dev/cciss/c1d0p2 lrwxrwxrwx 1 root > root 17 May 3 17:49 /dev/online_rootdbs02 -> > /dev/cciss/c1d1p5 lrwxrwxrwx 1 root root 17 May 3 17:49 > /dev/online_tempdbs -> /dev/cciss/c1d1p9 [root@zeus dev]# ls > -lL /dev/online_* > brw-rw---- 1 informix informix 105, 24 May 3 16:02 /dev/online_logdbs > brw-rw---- 1 informix informix 105, 2 May 3 16:02 > /dev/online_rootdbs01 > brw-rw---- 1 informix informix 105, 21 May 3 16:02 > /dev/online_rootdbs02 > brw-rw---- 1 informix informix 105, 25 May 3 16:02 > /dev/online_tempdbs > ---------------------------------------------------------- > > Note the "b" prefix for the block devices. > > Should work, at least it does for me. > > Cheers > Rupan3rd (from rainy Italy) > > > P.S. IDS 10 is not yet officially certified for RHEL5 but > AFAICS it seems to work flawlessly. > _______________________________________________ > Informix-list mailing list > Informix-list@iiug.org > http://www.iiug.org/mailman/listinfo/informix-list > >