Cooked vs RAW

I was reading the "Fixed / Know defects" for IDS 9.4.UD4 for Linux and found the following defect: bug_number 162482 description NEED TO UPDATE ADMIN GUIDE PAGE 1-9 AND 9-7 AND PERFORMANCE GUIDE PAGE 5-6 TO STATE THAT NEWER OS DISK CACHE MECHANISMS PROVIDE BETTER COOKED VS RAW product_code ONLINE component_code RSAM Can anyone shed some more light on this? If the disk case mechanisms are improving cooked why don't they also improve raw performance as well? -Peter ---------- CONFIDENTIALITY NOTICE: This e-mail message, including any attachments,is for the sole use of the intended recipient(s), even if addressed incorrectly, and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy or delete all copies of the original message and all attachments, including deletion from the trash or equivalent folder. Thank you.

They're apparently talking about the lightweight filesystems which use an improved OS level cache that is much faster than traditional FS's, rather than any hardware caching that would also apply to RAW devices. The performance hit from these filesystems is typically about 5% on top of the RAW -versus- COOKED device file hit of 15-25%. Using such light FS drops the range of additional overhead beyond RAW devices from 25-40% down to 20-30%. A big improvement, and worth considering if administrative simplicity is your goal, but not nearly enough if performance is primary. Personally I don't understand why anyone considers RAW devices an administrative problem. To me they're simpler. One less layer of software to administer and worry about. Art S. Kagel ----- Original Message ----- From: Peter J Dia.... At: 9/14 8:06 > > I was reading the "Fixed / Know defects" for IDS 9.4.UD4 for Linux and > found the following defect: > > bug_number 162482 > description NEED TO UPDATE ADMIN GUIDE PAGE 1-9 AND 9-7 AND PERFORMANCE > GUIDE PAGE 5-6 TO STATE THAT NEWER OS DISK CACHE MECHANISMS PROVIDE BETTER > COOKED VS RAW > product_code ONLINE > component_code RSAM > > > > Can anyone shed some more light on this? If the disk case mechanisms > are improving cooked why don't they also improve raw performance as well? > > -Peter > > > > ---------- > CONFIDENTIALITY NOTICE: This e-mail message, including any attachments,is for > the sole use of the intended recipient(s), even if addressed incorrectly, and > may contain confidential and privileged information. Any unauthorized review, > use, disclosure or distribution is prohibited. If you are not the intended > recipient, please contact the sender by reply e-mail and destroy or delete all > copies of the original message and all attachments, including deletion from the > trash or equivalent folder. Thank you.

> <pdiazdeleon@infinityhealthcare.com> ----- > > Date: Tue, 14 Sep 2004 08:00:08 -0400 (EDT) > From: "Peter J Dia...." <pdiazdeleon@infinityhealthcare.com> > To: ids@iiug.org > Subject: Cooked vs RAW [3416] > > > I was reading the "Fixed / Know defects" for IDS 9.4.UD4 for > Linux and > found the following defect: > > bug_number 162482 > description NEED TO UPDATE ADMIN GUIDE PAGE 1-9 AND 9-7 > AND PERFORMANCE GUIDE PAGE 5-6 TO STATE THAT NEWER OS DISK > CACHE MECHANISMS PROVIDE BETTER COOKED VS RAW > product_code ONLINE > component_code RSAM > > > > Can anyone shed some more light on this? If the disk case mechanisms > are improving cooked why don't they also improve raw > performance as well? Comment from number one son (who was the LVM guru) "People often ask about fragmentation of logical volumes, my answer is always that the extent size (ie. the smallest chunk of an LV that we manipulate) is so large (typically 4-16M) that the performance degradation is tiny, since your files are usually so much smaller than that. Linux has a couple of caches, the page cache which is indexed by a (device, file, offset) tuple, and a buffer cache which is just (device, offset). The page cache is a _lot_ bigger than the buffer cache. I understood that DB companies recommend the file system (cooked) approach because it's more flexible (you can grow the fs) and the performance is similar. Yes, LVM would let you resize raw partitions, assuming the database can also deal with resized partitions." -- Mark Thornber ========================= E M Thornber CEng MIEE Enchanted Systems Limited Software Toolsmiths +44 (0) 1503 272097

Hi, in general "Informix"-jargon, "cooked" refers to chunks in the OS's filesystem. This means that for I/O to the filesystem the OS will do some buffering. How much buffering there is may be configurable (i.e. how much memory is used for the buffers). On the IDS side this kind of I/O is handled by "AIO"-threads which are running on AIO-VPs. "raw" on the other hand refers to chunks on raw devices and utilizing KAIO (kernel asynchronous I/O). This means that the OS will not do any buffering. Instead, IDS writes directly to the device, but through special kernel routines, i.e. the I/O is done "in the kernel" (rather than by extra processes). Linux is an exception (so far), as for quite some time KAIO was not supported. Therefore IDS also did not support KAIO on Linux. In newer kernels (2.6.x and higher), KAIO is mostly supported. Therefore the next major release of IDS (9.50.UCx) will support KAIO on Linux also. Now, whether one or the other is faster is a difficult question, that cannot really be answered generally. If you want to find out for your particular environment (OS version, hardware, application, etc.) you probably have to do some testing. There have been advances in the implementation of filesystems, and even new filesystems have been implemented. With these the buffering efficiency may be so good, that you may have higher throughput to/from chunks on cooked files than to/from chunks on raw devices. Obviously, only chunks on cooked files will benefit from these enhancements, chunks on raw devices get no benefit as the I/O on the latter is done "without" the filesystem. On the other hand, even with improved filesystems, to do the I/O you need AIO-threads (i.e. you need to configure sufficient AIO-VPs via the onconfig parameter NUMAIOVP). This means you may have a lot of processes just for this I/O to chunks on cooked files. And these processes need additional resources (like memory allocated by the filesystem/OS for the I/O buffering). By switching to KAIO on raw devices you do not need those many AIO-threads (one should be enough to handle I/O to the message log file, etc., so you set NUMAIOVPS to 1). Thus you will have less processes running on your system using less memory (no memory for the OS-buffering needed). This may free up memory for IDS itself (e.g. the buffer pool). There have been tests where this additional memory as shared memory for IDS facilitated a significant increase in throughput. So IDS was able to handle more client requests with KAIO on raw devices than using cooked files. Just as well all this depends on your system, disk I/O rates, number of CPUs, application, etc. So it really is difficult to give a general answer as to what you should use. Regards, Martin -- Martin Fuerderer IBM Informix Development Munich Data Management Solutions forum.subscriber@iiug.org wrote on 14.09.2004 14:00:08: > > I was reading the "Fixed / Know defects" for IDS 9.4.UD4 for Linux and > found the following defect: > > bug_number 162482 > description NEED TO UPDATE ADMIN GUIDE PAGE 1-9 AND 9-7 AND PERFORMANCE > GUIDE PAGE 5-6 TO STATE THAT NEWER OS DISK CACHE MECHANISMS > PROVIDE BETTER COOKED VS RAW > product_code ONLINE > component_code RSAM > > Can anyone shed some more light on this? If the disk case mechanisms > are improving cooked why don't they also improve raw performance as well? > > -Peter

From: Martin Fuer.... <MARTINFU@de.ibm.com> At: 9/14 11:32 Wrote: Martin I'm just going to comment on one thing: > in general "Informix"-jargon, "cooked" refers to chunks in the OS's > filesystem. This means that for I/O to the filesystem the OS will do Actually there are two types of 'COOKED' chunks. Those built, as you suggest, using filesystem files and those build using the BLOCK device driver file representing a disk or virtual partition. For clarity let's call the former COOKED files and the latter COOKED devices. RAW devices, which you discuss below, are represented by the CHARACTER device driver file representing a disk or virtual partition. The difference between the two types of COOKED chunks is this: - Cooked devices suffer from some performance slow down compared to RAW devices caused by having to copy the application page buffer to the device driver's page cache and wait (because IDS opens COOKED chunks in O_SYNC mode which causes the write() system call to wait for writes to be confirmed to disk before returning). - Cooked files suffer from all of the performance slowdown of a cooked device, plus the additional overhead of the filesystem, any filesystem caching, and possible file fragmentation. All of these overheads are variable and addressed by more modern and lightweight filesystems, but they still exist. For example, to write to a page in cooked file a traditional UNIX filesytem, the page's byte address has to be translated into a physical disk address which entails walking up to 4 levels of linked inode entries depending on how far into the file the page lives. > some buffering. How much buffering there is may be configurable > (i.e. how much memory is used for the buffers). On the IDS side > this kind of I/O is handled by "AIO"-threads which are running on > AIO-VPs. All valid points. > "raw" on the other hand refers to chunks on raw devices and utilizing > KAIO (kernel asynchronous I/O). This means that the OS will not do > any buffering. Instead, IDS writes directly to the device, but through ... And there's the place where RAW gains most. Yes, KAIO is more efficient than AIO VPs, however, on systems I have used where there is no KAIO (Linux 2.4 and DGUX on M88K and Intel Pentium) RAW devices performed >35% faster than traditional filesystem space and about 20% faster than light filesystems I've been convinced to try suing AIO VPs all around. Add on the reduced overhead and performance gains inside the engine that KAIO represents and I see no reason to use COOKED devices or COOKED files. Your point about testing is well taken. Any DBA worth his walking papers is going to test all this on his/her own hardware and software and not trust any of our advice anyway. Art S. Kagel

When my latest project was scheduled to go to SAN space as cooked files, I was skeptical about not having raw partitions. However, as it was explained to me by the engineering group at both the company and EMC, the FAST SAN devices are comparable to RAW disks when it comes to performance. There was even a document co-written by Informix and EMC that documented that fact (though I do not remember its name at the moment). Needless to say, the engineers got the last word on the project and the instance (around 75 GB) is size, is within flat files on a EMC Symmetric SAN Array. (For Disaster Recovery, business continuous volumes (BCVs) are used to copy the database (and the application files) from their Production server to a Disaster Recovery server.) Take care. Clifton "Martin Fuer...." <MARTINFU@de.ibm.com> wrote: Hi, in general "Informix"-jargon, "cooked" refers to chunks in the OS's filesystem. This means that for I/O to the filesystem the OS will do some buffering. How much buffering there is may be configurable (i.e. how much memory is used for the buffers). On the IDS side this kind of I/O is handled by "AIO"-threads which are running on AIO-VPs. "raw" on the other hand refers to chunks on raw devices and utilizing KAIO (kernel asynchronous I/O). This means that the OS will not do any buffering. Instead, IDS writes directly to the device, but through special kernel routines, i.e. the I/O is done "in the kernel" (rather than by extra processes). Linux is an exception (so far), as for quite some time KAIO was not supported. Therefore IDS also did not support KAIO on Linux. In newer kernels (2.6.x and higher), KAIO is mostly supported. Therefore the next major release of IDS (9.50.UCx) will support KAIO on Linux also. Now, whether one or the other is faster is a difficult question, that cannot really be answered generally. If you want to find out for your particular environment (OS version, hardware, application, etc.) you probably have to do some testing. There have been advances in the implementation of filesystems, and even new filesystems have been implemented. With these the buffering efficiency may be so good, that you may have higher throughput to/from chunks on cooked files than to/from chunks on raw devices. Obviously, only chunks on cooked files will benefit from these enhancements, chunks on raw devices get no benefit as the I/O on the latter is done "without" the filesystem. On the other hand, even with improved filesystems, to do the I/O you need AIO-threads (i.e. you need to configure sufficient AIO-VPs via the onconfig parameter NUMAIOVP). This means you may have a lot of processes just for this I/O to chunks on cooked files. And these processes need additional resources (like memory allocated by the filesystem/OS for the I/O buffering). By switching to KAIO on raw devices you do not need those many AIO-threads (one should be enough to handle I/O to the message log file, etc., so you set NUMAIOVPS to 1). Thus you will have less processes running on your system using less memory (no memory for the OS-buffering needed). This may free up memory for IDS itself (e.g. the buffer pool). There have been tests where this additional memory as shared memory for IDS facilitated a significant increase in throughput. So IDS was able to handle more client requests with KAIO on raw devices than using cooked files. Just as well all this depends on your system, disk I/O rates, number of CPUs, application, etc. So it really is difficult to give a general answer as to what you should use. Regards, Martin -- Martin Fuerderer IBM Informix Development Munich Data Management Solutions forum.subscriber@iiug.org wrote on 14.09.2004 14:00:08: > > I was reading the "Fixed / Know defects" for IDS 9.4.UD4 for Linux and > found the following defect: > > bug_number 162482 > description NEED TO UPDATE ADMIN GUIDE PAGE 1-9 AND 9-7 AND PERFORMANCE > GUIDE PAGE 5-6 TO STATE THAT NEWER OS DISK CACHE MECHANISMS > PROVIDE BETTER COOKED VS RAW > product_code ONLINE > component_code RSAM > > Can anyone shed some more light on this? If the disk case mechanisms > are improving cooked why don't they also improve raw performance as well? > > -Peter