Re: Cooked files vs. raw partitions

So far on this thread, raw seems to be the overwhelming choice, but I would like to hear some feedback based on my specific configuration. We're in the process of bringing up a new production P750 with 32 cores against a Hitachi HUS-VM with 54TB of flash storage connected over 8 8Gb Fiber Channel ports to the P750 and we've been able to generate over 400k 4k IOPS with a 50/50 read/write IO mixture on cooked files using "Direct IO". For those of you running P series hardware from IBM, it appears that the sweet spot on the new Power 7+ processors seems to be at 4 IO threads per core so I've been running 128 IO threads for my tests. One issue that I've found in using flash storage is that Informix seems more optimized for spinning disk in it's IO behavior (considering flash is fairly new that makes sense I guess). The documentation that I've been able to find about Informix IO has stated the Informix default IO requests to storage are 64k at a time (or 16 x OS pages size); I'm hoping to find a way to scale back the IO request size to 4k to better leverage flash architecture, but even at 64k page sizes I'm seeing 250k IOPS against cooked files using "Direct IO". I've also tried "Concurrent IO" and while the description of concurrent IO sounds like it should be faster than direct IO; in my tests concurrent IO has been about 5% slower than direct IO. At this point, 400k IOPS against cooked files using Direct IO on AIX 7.1 with a 32 core P750 seem pretty good. Even if raw is a little fast, I'm not sure that it's worth managing hundreds of chunks on raw devices for the sake of a few percent faster. I'm planning to do a few raw tests on Monday, so I'll post again when I see those numbers.

More testing results: What I think I'm seeing is that cooked chunks max out at about 4k IOPS using 4 threads per chunk. Adding more than 4 threads per chunk for a cooked chunks yields no substantial gain in performance. It appears to me that the limiting factor in performance in cooked chunks isn't performance per thread, but more scale-ability of threads per chunk; because a raw chunk can scale nicely up to about 24 threads yielding about 70k IOPS raw chunks get much better performance when IO threads are available and called to service IO needs. Depending on IOPS available on your storage and the number of productive IO threads that you can define on your system would probably determine whether raw could achieve a worthwhile benefit in an environment. On my soon to be deprecated IBM DS5300 storage where I had about 20k IOPS total, I never saw a significant increase in performance with raw over cooked. But now that we're moving into the flash storage world, the number of IOPS available seems to be making raw very appealing.

Anthony, Thank you for sharing your testing resolts. Dan -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of ANTHONY LANDRY Sent: Monday, October 28, 2013 2:21 PM To: ids@iiug.org Subject: Re: Cooked files vs. raw partitions [31832] More testing results: What I think I'm seeing is that cooked chunks max out at about 4k IOPS using 4 threads per chunk. Adding more than 4 threads per chunk for a cooked chunks yields no substantial gain in performance. It appears to me that the limiting factor in performance in cooked chunks isn't performance per thread, but more scale-ability of threads per chunk; because a raw chunk can scale nicely up to about 24 threads yielding about 70k IOPS raw chunks get much better performance when IO threads are available and called to service IO needs. Depending on IOPS available on your storage and the number of productive IO threads that you can define on your system would probably determine whether raw could achieve a worthwhile benefit in an environment. On my soon to be deprecated IBM DS5300 storage where I had about 20k IOPS total, I never saw a significant increase in performance with raw over cooked. But now that we're moving into the flash storage world, the number of IOPS available seems to be making raw very appealing. ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.