Esoteric Disk Issues
A shop moving IDS 10 from Solaris 8 to Solaris 10 found that using ~1000 small (2GB) ZFS-based LUNs over an HP SAN made the server take ~16 hours to boot, and asked whether to use big ZFS LUNs with chunk offsets, SVM raw devices, or cooked UFS/ZFS. Replies suggested far fewer, much larger LUNs, warned that fast cooked I/O on Solaris may require extra licensed software (e.g. Veritas CIO), and estimated cooked/filesystem chunks run roughly 5-8% slower than raw with DIRECT_IO and 15-25% without. Resolution: they dropped ZFS and went with Sun Volume Manager raw devices, with Art Kagel and Solaris Internals links explaining why ZFS suits filesystems, not databases.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Informix v10.00.UC8 Solaris 10 We are migrating from Sun 8 to Sun 10 and having issues with disk. We initially used ~1000 two gig "luns" via zfs that were links to devices or pools within our HP SAN. With this config, booting our Unix box took ~16 hours - no typo. We are now looking at other options. These are our options: 1) Use giant ZFS luns with Informix chunk offsets, instead of a lot of smaller luns. 2) Use SVM (Sun Volume Manager) with raw devices. We understand that SVM is being phased out though and support may be discontinued. 3) Raw UFS 4) Cooked UFS 5) Cooked ZFS How much slower is cooked than raw - really? I know that Sun now has direct i/o mount options and that Informix 11 has config parameters that allow kaio to be used with cooked files. Would we die from performance lags if we were on cooked files (direct i/o mounted) using aio vps while we moved to Informix 11? I have heard that ZFS is not suggested for databases and I am cool with avoiding it. We are a quasi DW environment with no OLTP use. Ideas? MM
2GB LUNs? How about rebuilding with 10 or 50 or 100GB LUNs? Is there a reason you're sticking with 2GB? Bob ----- Original Message ----- From: "MIKE MAGIE" <jmmagie@yahoo.com> To: ids@iiug.org Sent: Tuesday, March 31, 2009 11:18:02 AM GMT -05:00 US/Canada Eastern Subject: Esoteric Disk Issues [15379] Informix v10.00.UC8 Solaris 10 We are migrating from Sun 8 to Sun 10 and having issues with disk. We initially used ~1000 two gig "luns" via zfs that were links to devices or pools within our HP SAN. With this config, booting our Unix box took ~16 hours - no typo. We are now looking at other options. These are our options: 1) Use giant ZFS luns with Informix chunk offsets, instead of a lot of smaller luns. 2) Use SVM (Sun Volume Manager) with raw devices. We understand that SVM is being phased out though and support may be discontinued. 3) Raw UFS 4) Cooked UFS 5) Cooked ZFS How much slower is cooked than raw - really? I know that Sun now has direct i/o mount options and that Informix 11 has config parameters that allow kaio to be used with cooked files. Would we die from performance lags if we were on cooked files (direct i/o mounted) using aio vps while we moved to Informix 11? I have heard that ZFS is not suggested for databases and I am cool with avoiding it. We are a quasi DW environment with no OLTP use. Ideas? MM
Mike, One thing you may also need to check is the cost of the Sun software to get the fast i/o on cooked chunks. At my last job we had to buy the "go faster i/o" software when we migrated Informix/raw to DB2/cooked. The "go faster i/o" software was not cheap. NJ -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of MIKE MAGIE Sent: Tuesday, March 31, 2009 10:18 AM To: ids@iiug.org Subject: Esoteric Disk Issues [15379] Informix v10.00.UC8 Solaris 10 We are migrating from Sun 8 to Sun 10 and having issues with disk. We initially used ~1000 two gig "luns" via zfs that were links to devices or pools within our HP SAN. With this config, booting our Unix box took ~16 hours - no typo. We are now looking at other options. These are our options: 1) Use giant ZFS luns with Informix chunk offsets, instead of a lot of smaller luns. 2) Use SVM (Sun Volume Manager) with raw devices. We understand that SVM is being phased out though and support may be discontinued. 3) Raw UFS 4) Cooked UFS 5) Cooked ZFS How much slower is cooked than raw - really? I know that Sun now has direct i/o mount options and that Informix 11 has config parameters that allow kaio to be used with cooked files. Would we die from performance lags if we were on cooked files (direct i/o mounted) using aio vps while we moved to Informix 11? I have heard that ZFS is not suggested for databases and I am cool with avoiding it. We are a quasi DW environment with no OLTP use. Ideas? MM ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum.
Can you elaborate on this NJ? From what I understood if we had a UFS mounted file system for example, and when the file system was mounted we used the "forcedirectio" option, we would bypass the file system's buffer cache, closer mimicking raw disk i/o behavior. MM
Hi Mike, I will get in contact with my old sys admins who should be able to describe what we did more accurately. Thanks, NJ -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of MIKE MAGIE Sent: Tuesday, March 31, 2009 2:31 PM To: ids@iiug.org Subject: Re: RE: Esoteric Disk Issues [15385] Can you elaborate on this NJ? From what I understood if we had a UFS mounted file system for example, and when the file system was mounted we used the "forcedirectio" option, we would bypass the file system's buffer cache, closer mimicking raw disk i/o behavior. MM ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum.
Hi Mike, Me again... My old unix admin (he's younger than me, but from my last job...therefore 'old' :-)) anyway, my old unix admin said the following about the Sun "quick i/o" software I spoke of: --------------- It is part of the Veritas Storage Foundation for DB2 Enterprise product, and the exact "option" is Concurrent I/O (cio) on Veritas File System (VxFS). With Solaris 10, you can also mention that ZFS could be an option. It's supposed to be 25% faster than running UFS file system. I am very interested in testing DB2 performance on ZFS and comparing to VxFS with CIO enabled. There are some big money savings to be had if ZFS is comparable in performance. ------------- Hopefully that is helpful to you even though it's what we had to do when we (at old job) went from Informix/raw/solaris9(or10) to DB2/cooked/solaris10. We were initially told that we should not have to purchase extra software to run DB2/cooked/solaris10, but we found out the hard way when it was a dog. As you see from my old admin, he is interested in ZFS, but your original post said it's not suggested for databases?... If you'd like to discuss this off-forum, I can loop my old admin in... can we contact you at your yahoo addr? (jmmagie at yahoo) Thanks, NJ -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of MIKE MAGIE Sent: Tuesday, March 31, 2009 2:31 PM To: ids@iiug.org Subject: Re: RE: Esoteric Disk Issues [15385] Can you elaborate on this NJ? From what I understood if we had a UFS mounted file system for example, and when the file system was mounted we used the "forcedirectio" option, we would bypass the file system's buffer cache, closer mimicking raw disk i/o behavior. MM -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of MIKE MAGIE Sent: Tuesday, March 31, 2009 10:18 AM To: ids@iiug.org Subject: Esoteric Disk Issues [15379] Informix v10.00.UC8 Solaris 10 We are migrating from Sun 8 to Sun 10 and having issues with disk. We initially used ~1000 two gig "luns" via zfs that were links to devices or pools within our HP SAN. With this config, booting our Unix box took ~16 hours - no typo. We are now looking at other options. These are our options: 1) Use giant ZFS luns with Informix chunk offsets, instead of a lot of smaller luns. 2) Use SVM (Sun Volume Manager) with raw devices. We understand that SVM is being phased out though and support may be discontinued. 3) Raw UFS 4) Cooked UFS 5) Cooked ZFS How much slower is cooked than raw - really? I know that Sun now has direct i/o mount options and that Informix 11 has config parameters that allow kaio to be used with cooked files. Would we die from performance lags if we were on cooked files (direct i/o mounted) using aio vps while we moved to Informix 11? I have heard that ZFS is not suggested for databases and I am cool with avoiding it. We are a quasi DW environment with no OLTP use. Ideas? MM
MIKE MAGIE wrote: > Informix v10.00.UC8 > Solaris 10 > > We are migrating from Sun 8 to Sun 10 and having issues with disk. We > initially used ~1000 two gig "luns" via zfs that were links to devices or > pools within our HP SAN. With this config, booting our Unix box took ~16 hours > - no typo. We are now looking at other options. These are our options: > > 1) Use giant ZFS luns with Informix chunk offsets, instead of a lot of smaller > luns. > > 2) Use SVM (Sun Volume Manager) with raw devices. We understand that SVM is > being phased out though and support may be discontinued. > > 3) Raw UFS > > 4) Cooked UFS > > 5) Cooked ZFS > > How much slower is cooked than raw - really? 5-15% with DIRECT_IO. -- Cheers, Obnoxio The Clown http://obotheclown.blogspot.com -- This message has been scanned for viruses and dangerous content by OpenProtect(http://www.openprotect.com), and is believed to be clean.
Cooked devices are ~4-5% slower than RAW devices with DIRECT_IO and 5-10% slower without DIRECT_IO. Cooked files (or filesystem chunks) are an additional 1-3% slower than COOKED devices with DIRECT_IO and 10-15% than COOKED device chunks without DIRECT_IO. That makes the total penalty for using filesystem chunks betwwn 5 and 8% with DIRECT_IO and 15-25% without it. That does not count the extra overhead you will experience if you are using a costly filesystem such as most journaled FSes and remote mirrored FSes. These numbers are based on testing I did over 10 years ago but they have been substantiated by others experience many times since. Art Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Apr 2, 2009 at 11:12 AM, Obnoxio The Clown <obnoxio@serendipita.com>wrote: > MIKE MAGIE wrote: > > Informix v10.00.UC8 > > Solaris 10 > > > > We are migrating from Sun 8 to Sun 10 and having issues with disk. We > > initially used ~1000 two gig "luns" via zfs that were links to devices or > > pools within our HP SAN. With this config, booting our Unix box took ~16 > hours > > - no typo. We are now looking at other options. These are our options: > > > > 1) Use giant ZFS luns with Informix chunk offsets, instead of a lot of > smaller > > luns. > > > > 2) Use SVM (Sun Volume Manager) with raw devices. We understand that SVM > is > > being phased out though and support may be discontinued. > > > > 3) Raw UFS > > > > 4) Cooked UFS > > > > 5) Cooked ZFS > > > > How much slower is cooked than raw - really? > > 5-15% with DIRECT_IO. > > -- > Cheers, > Obnoxio The Clown > > http://obotheclown.blogspot.com > > -- > This message has been scanned for viruses and > dangerous content by OpenProtect(http://www.openprotect.com), and is > believed to be clean. > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --0016e6441e28847b4704669798c3
Calm heads prevailed and we ended up scrapping ZFS and going with the good old Sun Volume Manager with beautifully raw devices... Thanks for all of the input - if any of you are considering using ZFS with Informix - rethink it... MM
Hi Mike, You initially had said not to use ZFS with any database (or rather, you didn't list a specific database so 'all' might be implied)... is it just Informix that has a problem with ZFS? If you have links to articles about it being problematic with all databases, please post/send the links. Thanks, NJ -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of MIKE MAGIE Sent: Friday, April 03, 2009 10:36 AM To: ids@iiug.org Subject: Re: Esoteric Disk Issues [15413] Calm heads prevailed and we ended up scrapping ZFS and going with the good old Sun Volume Manager with beautifully raw devices... Thanks for all of the input - if any of you are considering using ZFS with Informix - rethink it... MM ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum.
Found this: http://www.solarisinternals.com/wiki/index.php/ZFS_for_Databases http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide thanks. -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Norma Jean Sebastian Sent: Friday, April 03, 2009 2:08 PM To: ids@iiug.org Subject: RE: Esoteric Disk Issues [15421] Hi Mike, You initially had said not to use ZFS with any database (or rather, you didn't list a specific database so 'all' might be implied)... is it just Informix that has a problem with ZFS? If you have links to articles about it being problematic with all databases, please post/send the links. Thanks, NJ -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of MIKE MAGIE Sent: Friday, April 03, 2009 10:36 AM To: ids@iiug.org Subject: Re: Esoteric Disk Issues [15413] Calm heads prevailed and we ended up scrapping ZFS and going with the good old Sun Volume Manager with beautifully raw devices... Thanks for all of the input - if any of you are considering using ZFS with Informix - rethink it... MM ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum. ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum.
What's wrong with ZFS for databases is that it was designed for filesystems. From the OpenSolaris ZFS pages ( http://www.opensolaris.org/os/community/zfs/whatis/): ZFS presents a pooled storage model that completely eliminates the concept of volumes and the associated problems of partitions, provisioning, wasted bandwidth and stranded storage. Thousands of file systems can draw from a common storage pool, each one consuming only as much space as it actually needs. The combined I/O bandwidth of all devices in the pool is available to all filesystems at all times. All operations are copy-on-write transactions, so the on-disk state is always valid. There is no need to fsck(1M) a ZFS file system, ever. Every block is checksummed to prevent silent data corruption, and the data is self-healing in replicated (mirrored or RAID) configurations. If one copy is damaged, ZFS detects it and uses another copy to repair it. ZFS introduces a new data replication model called RAID-Z. It is similar to RAID-5 but uses variable stripe width to eliminate the RAID-5 write hole (stripe corruption due to loss of power between data and parity updates). All RAID-Z writes are full-stripe writes. There's no read-modify-write tax, no write hole, and the best part no need for NVRAM in hardware. ZFS loves cheap disks. But cheap disks can fail, so ZFS provides disk scrubbing. Like ECC memory scrubbing, the idea is to read all data to detect latent errors while they're still correctable. A scrub traverses the entire storage pool to read every copy of every block, validate it against its 256-bit checksum, and repair it if necessary. All this happens while the storage pool is live and in use. Problem #1: Note the 'shared bandwidth' . Your database chunks are competing for bandwidth with every other ZFS filesystem on the local net. Problem #2: Copy-on-write means that data pages are always being moved from location to location every time they are written to. Later the ZFS has to go back and remove the previous copy of the page. More wasted bandwidth and IO capacity. Problem #3: ZFS uses RAID-Z which is just a fancy RAID6 ie RAID5 but with two parity disks. Yes it can use RAID1, but what SA is going to pass on configuring the hot new RAIDZ? (Yes, ZFS is proactive about repairing bad pages if the filesystem is protected by RAID5, RAIDZ, or RAID1. I do like that part, but does it work over the long haul?) Problem #4: ZFS loves cheap (read 'slow') disks. What sales person could pass up on selling you the cheapest and hence the slowest disks around? Problem #5: Every block read and every block write has to use up CPU bandwidth calculating per block checksums. Do I really have to go on? Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Apr 3, 2009 at 3:07 PM, Norma Jean Sebastian < nsebastian@flinnsci.com> wrote: > Hi Mike, > You initially had said not to use ZFS with any database (or rather, you > didn't list a specific database so 'all' might be implied)... > is it just Informix that has a problem with ZFS? > > If you have links to articles about it being problematic with all > databases, please post/send the links. > Thanks, > NJ > > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > MIKE MAGIE > Sent: Friday, April 03, 2009 10:36 AM > To: ids@iiug.org > Subject: Re: Esoteric Disk Issues [15413] > > Calm heads prevailed and we ended up scrapping ZFS and going with the > good old > Sun Volume Manager with beautifully raw devices... > > Thanks for all of the input - if any of you are considering using ZFS > with > Informix - rethink it... > > MM > > ************************************************************************ > ******* > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --0016368e1f214cc23c0466ac4ba1
These are my favorites from NJ's Best Practices link: As of July 2007, the following features might impact database performance: - ZFS issues up to 35 concurrent I/Os to each top-level device and this can lead to inflated service time. - ZFS does some low-level prefetches of up to 64K for each input block, which can cause saturation of storage channels. See bug 6437054<http://bugs.opensolaris.org/view_bug.do?bug_id=6437054>and this blog<http://blogs.sun.com/erickustarz/entry/vdev_cache_improvements_to_help>;. Art Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Apr 3, 2009 at 4:05 PM, Norma Jean Sebastian < nsebastian@flinnsci.com> wrote: > Found this: > http://www.solarisinternals.com/wiki/index.php/ZFS_for_Databases > http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide > thanks. > > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > Norma Jean Sebastian > Sent: Friday, April 03, 2009 2:08 PM > To: ids@iiug.org > Subject: RE: Esoteric Disk Issues [15421] > > Hi Mike, > You initially had said not to use ZFS with any database (or rather, you > didn't list a specific database so 'all' might be implied)... > is it just Informix that has a problem with ZFS? > > If you have links to articles about it being problematic with all > databases, please post/send the links. > Thanks, > NJ > > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > MIKE MAGIE > Sent: Friday, April 03, 2009 10:36 AM > To: ids@iiug.org > Subject: Re: Esoteric Disk Issues [15413] > > Calm heads prevailed and we ended up scrapping ZFS and going with the > good old > Sun Volume Manager with beautifully raw devices... > > Thanks for all of the input - if any of you are considering using ZFS > with > Informix - rethink it... > > MM > > ************************************************************************ > > ******* > Forum Note: Use "Reply" to post a response in the discussion forum. > > ************************************************************************ > ******* > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --0016368e24ce13a4e40466ac6478