Re: General RAID5 question - dangers?
From: Edgie <per2edge@newsguy.com> > >We currently have our data on an EMC disk array RAID1 and plan to migrate >data >to an Hitachi XP256 configured as RAID5 with 7.31.UC6A-1 and HP-UX 10.20. >With >EMC, I used Informix fragmentation to balance data across buses and cards. >I'm not that smart about RAID details, but looks like I've seen some posts >warning about dangers of RAID5. >Could someone please elaborate on these dangers, if indeed there are any? >Or maybe point me to some url's where I can educate myself about pro's and >con's >of RAID5, especially with Informix? To paraphrase Art: "There are two problems with RAID5. The first is performance which is the one most people notice and if you can live with write throughput that is 50% of the equivalent RAID0 stripe set, then that is fine. The performance hit is caused because RAID5 only reads the one drive containing the requested sector leaving the other drives free to return other sectors from different stripe blocks. This is the reason that RAID5 is preferred to RAID3 or RAID4 for file systems: this feature improves small random read performance. However, since the parity and the balance of the stripe block were not read, if you rewrite the block (which databases do far more frequently than file systems) the other drives must all be read and a new parity calculated and then both the modified block and the parity block must be written back to disk. This READ-WRITE-READ-WRITE for each modified block is the reason RAID5 is so poor in terms of write throughput. Large RAID controller caches and on-controller firmware level RAID implementations alleviate the problem somewhat but not completely and write performance still hovers at around half what a pure stripe (RAID0) would get. The second problem is a fundamental problem with the design of RAID5 that various implementers have tried to correct with varying levels of success. The problem is that if a drive fails slowly over time, known as partial media failure, where periodically a sector or two goes bad, this is not detected by RAID5's parity and so is propagated to the parity when that sector is rewritten which means that if another drive fails catastrophically its data will be rebuilt utilizing damaged parity resulting in two sectors with garbage. Now this may not even be noticed for a long time as modern SCSI drives automatically re-map bad sectors to a set of sectors set aside for the purpose but the corrected error is not reported to the operating system or the administrators. Over time if the drive is going it will run out of re-map sectors and will have to begin returning data reconstructed from the drive's own Error Correcting Code (ECC). Eventually the damage will exceed the ECC's ability to rebuild a single bit error per byte and will return garbage. RAID3 and RAID4 are superior in both areas. In both all drives are read for any block which improves sequential read performance (Informix Read Ahead depends on sequential read performance) over RAID5 and parity can be (and in most implementations, is) checked at read time so that partial media failure problems can be detected. Write performance is approximately the same as RAID0 for large writes or smaller stripe block sizes. One problem with early implementations of RAID3/4 was slow parity checking since it has to be calculated for every read and every write. Modern controller based RAID systems use the on-board processor on the SCSI controller to perform the parity checks without impacting system performance by tying up a system CPU to check and produce parity. These RAID levels require the exact same number of drives as RAID5. RAID10 provides the best protection and performance with read performance exceeding any other RAID level (since both drives of a mirrored pair can be reading different sectors on parallel) and write performance is closest to pure striping. Indeed in a hardware/firmware implemented RAID10 array with on-board cache apparent write throughput can exceed RAID0 for brief periods due to the two drives of each pair being written to independently though the gain is not sustainable over time. A third problem with all RAID3/4/5 from which RAID10 does not suffer is multiple drive failure. (Have you ever received a batch of 200 bad drives? One client has!) If one drive in a RAID3/4/5 array fails catastrophically you are at risk for complete data loss if any of the remaining 4 (or more) drives should fail before the original failed drive can be replaced and rebuilt. With RAID10, since it is made up as a stripe set of N mirrored pairs, when a drive fails you are only at risk for complete data loss if that one drives particular mirror partner should fail. Make each mirrored pair from drives selected from different manufacturer's lots and the probability of this happening become vanishing small. Fourth problem: during drive rebuild RAID3/4/5 (and RAID01 mirrored stripe sets) performance of the array during the rebuild can degrade by as much as 80%! Some RAID systems let you tune the relative priority of rebuild versus production to reduce the performance hit to as low as about 40% degradation but this will increase the recovery time increasing the number of production requests that are degraded and increasing the risk of the previous problem with a second drive failure. RAID10, since only one drive is involved in mirror recovery, the array's performance (for a four drive array) is degraded only a maximum of 80% for reads and writes against the failed pair and only slightly (due to controller traffic) for accesses to the other drives, on average, since the one pair comprises only 20% of accesses, performance is affected no more than 16% during recovery and the risk of catastrophic data loss is reduced." On a personal note, I have seen a RAID5 system experience gradual parity failure and then let go, and it was extremely ugly. ________________________________________________________________________ Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com
In article <8hl5k8$rgg$1@news.xmission.com>, Obnoxio The Clown <obnoxio@hotmail.com> writes > >To paraphrase Art: > >"There are two problems with RAID5. The first is performance which is the >one most people notice and if you can live with write throughput that is 50% >of the equivalent RAID0 stripe set, then that is fine. The performance hit ... >On a personal note, I have seen a RAID5 system experience gradual parity >failure and then let go, and it was extremely ugly. >________________________________________________________________________ >Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com > Great answer, one for the FAQ. -- David Williams
Whoa, looks like I don't have to post this summer's annual tirade! Thanks Obnoxio! NO RAID5 NO RAID5 NO RAID5 NO RAID5 NO RAID5 NO RAID5 NO RAID5 Art S. Kagel Obnoxio The Clown wrote: > > From: Edgie <per2edge@newsguy.com> > > > >We currently have our data on an EMC disk array RAID1 and plan to migrate > >data > >to an Hitachi XP256 configured as RAID5 with 7.31.UC6A-1 and HP-UX 10.20. > >With > >EMC, I used Informix fragmentation to balance data across buses and cards. > >I'm not that smart about RAID details, but looks like I've seen some posts > >warning about dangers of RAID5. > >Could someone please elaborate on these dangers, if indeed there are any? > >Or maybe point me to some url's where I can educate myself about pro's and > >con's > >of RAID5, especially with Informix? > > To paraphrase Art: > > "There are two problems with RAID5. The first is performance which is the > one most people notice and if you can live with write throughput that is 50% > of the equivalent RAID0 stripe set, then that is fine. The performance hit > is caused because RAID5 only reads the one drive containing the requested > sector leaving the other drives free to return other sectors from different > stripe blocks. This is the reason that RAID5 is preferred to RAID3 or RAID4 > for file systems: this feature improves small random read performance. > However, since the parity and the balance of the stripe block were not read, > if you rewrite the block (which databases do far more frequently than file > systems) the other drives must all be read and a new parity calculated and > then both the modified block and the parity block must be written back to > disk. This READ-WRITE-READ-WRITE for each modified block is the reason RAID5 > is so poor in terms of write throughput. Large RAID controller caches and > on-controller firmware level RAID implementations alleviate the problem > somewhat but not completely and write performance still hovers at around > half what a pure stripe (RAID0) would get. > > The second problem is a fundamental problem with the design of RAID5 that > various implementers have tried to correct with varying levels of success. > The problem is that if a drive fails slowly over time, known as partial > media failure, where periodically a sector or two goes bad, this is not > detected by RAID5's parity and so is propagated to the parity when that > sector is rewritten which means that if another drive fails catastrophically > its data will be rebuilt utilizing damaged parity resulting in two sectors > with garbage. Now this may not even be noticed for a long time as modern > SCSI drives automatically re-map bad sectors to a set of sectors set aside > for the purpose but the corrected error is not reported to the operating > system or the administrators. Over time if the drive is going it will run > out of re-map sectors and will have to begin returning data reconstructed > from the drive's own Error Correcting Code (ECC). Eventually the damage will > exceed the ECC's ability to rebuild a single bit error per byte and will > return garbage. > > RAID3 and RAID4 are superior in both areas. In both all drives are read for > any block which improves sequential read performance (Informix Read Ahead > depends on sequential read performance) over RAID5 and parity can be (and in > most implementations, is) checked at read time so that partial media failure > problems can be detected. Write performance is approximately the same as > RAID0 for large writes or smaller stripe block sizes. One problem with early > implementations of RAID3/4 was slow parity checking since it has to be > calculated for every read and every write. Modern controller based RAID > systems use the on-board processor on the SCSI controller to perform the > parity checks without impacting system performance by tying up a system CPU > to check and produce parity. These RAID levels require the exact same number > of drives as RAID5. > > RAID10 provides the best protection and performance with read performance > exceeding any other RAID level (since both drives of a mirrored pair can be > reading different sectors on parallel) and write performance is closest to > pure striping. Indeed in a hardware/firmware implemented RAID10 array with > on-board cache apparent write throughput can exceed RAID0 for brief periods > due to the two drives of each pair being written to independently though the > gain is not sustainable over time. > > A third problem with all RAID3/4/5 from which RAID10 does not suffer is > multiple drive failure. (Have you ever received a batch of 200 bad drives? > One client has!) If one drive in a RAID3/4/5 array fails catastrophically > you are at risk for complete data loss if any of the remaining 4 (or more) > drives should fail before the original failed drive can be replaced and > rebuilt. With RAID10, since it is made up as a stripe set of N mirrored > pairs, when a drive fails you are only at risk for complete data loss if > that one drives particular mirror partner should fail. Make each mirrored > pair from drives selected from different manufacturer's lots and the > probability of this happening become vanishing small. > > Fourth problem: during drive rebuild RAID3/4/5 (and RAID01 mirrored stripe > sets) performance of the array during the rebuild can degrade by as much as > 80%! Some RAID systems let you tune the relative priority of rebuild versus > production to reduce the performance hit to as low as about 40% degradation > but this will increase the recovery time increasing the number of production > requests that are degraded and increasing the risk of the previous problem > with a second drive failure. RAID10, since only one drive is involved in > mirror recovery, the array's performance (for a four drive array) is > degraded only a maximum of 80% for reads and writes against the failed pair > and only slightly (due to controller traffic) for accesses to the other > drives, on average, since the one pair comprises only 20% of accesses, > performance is affected no more than 16% during recovery and the risk of > catastrophic data loss is reduced." > > On a personal note, I have seen a RAID5 system experience gradual parity > failure and then let go, and it was extremely ugly. > ________________________________________________________________________ > Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com
In article <393FE9AE.36BDF8C6@bloomberg.net>, Art S. Kagel <kagel@bloomberg.net> writes >Whoa, looks like I don't have to post this summer's annual tirade! Thanks >Obnoxio! > > NO RAID5 NO RAID5 NO RAID5 NO RAID5 NO RAID5 NO RAID5 NO RAID5 > >Art S. Kagel > >Obnoxio The Clown wrote: >> >> From: Edgie <per2edge@newsguy.com> >> > >> >We currently have our data on an EMC disk array RAID1 and plan to migrate >> >data >> >to an Hitachi XP256 configured as RAID5 with 7.31.UC6A-1 and HP-UX 10.20. >> >With >> >EMC, I used Informix fragmentation to balance data across buses and cards. >> >I'm not that smart about RAID details, but looks like I've seen some posts >> >warning about dangers of RAID5. >> >Could someone please elaborate on these dangers, if indeed there are any? >> >Or maybe point me to some url's where I can educate myself about pro's and >> >con's >> >of RAID5, especially with Informix? >> >> To paraphrase Art: Many thanks for posting this which is the best discussion of the various RAID options I've seen so far. It gives the clear explanation of why RAID10 is just about the holy grail. <snip> <snip> -- Surfer!