Large chunks versus small chunks.
Question: for performance (not admin convenience), is it better to use a few large chunks or many smaller ones? Answers: Art Kagel explained the engine assigns one flush thread per chunk, so spreading dirty pages over more chunks makes checkpoints flush faster; on pre-11 versions (the poster was on 10.00.FC8/Solaris) long checkpoints also block transactions, while 11.x's non-blocking checkpoints make duration mostly a recovery-time concern. Jonathan Leffler favoured large chunks for modern disks and easier administration, and Andrew Clarke suggested roughly 4-8 busy chunks per spindle, noting large arrays behaved differently. No single definitive answer, just trade-offs.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
All, I would like gauge opinion on whether or not it is better, worse or does not matter if you use fewer larger chunks or more smaller chunks. I am trying to find out if you have any experiences or better still evidence that chunk size matters. I not over concerned about the amount of administration it involves. I am more interested in the technical/performance aspect. Thanks Andy G. _________________________________________________________________ New Windows 7: Find the right PC for you. Learn more. http://www.microsoft.com/uk/windows/buy/
NOTE TO ALL POSTERS: PLEASE, POST YOUR VERSION AND PLATFORM INFORMATION. Whether you think so it not, more often than not that information will impact the answers you receive here. Here's my take. There are trade offs, as with everything else we do unfortunately. At checkpoint or chunk flush time the engine assigned a flush thread to each chunk. That means that if the dirty pages are spread across more chunks they will tend to be flushed more quickly making your checkpoints faster. If you are running IDS 11.xx this only impacts recovery time and, less so, data safety. If you are running any earlier release of IDS, it will also impact the lenght of time during a checkpoint that tranasactions are restricted from entering critical sections and so will be blocked. See, a version dependency ;-) Art Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Nov 6, 2009 at 8:36 AM, Andrew Grantham <agrantha@hotmail.com>wrote: > All, > > I would like gauge opinion on whether or not it is better, worse or does > not > matter if you use fewer larger chunks or more smaller chunks. I am trying > to > find out if you have any experiences or better still evidence that chunk > size > matters. I not over concerned about the amount of administration it > involves. > I am more interested in the technical/performance aspect. > > Thanks > > Andy G. > > _________________________________________________________________ > New Windows 7: Find the right PC for you. Learn more. > http://www.microsoft.com/uk/windows/buy/ > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --0015173ff15ecf1e050477b444bf
Art, The version is 10.00.FC8 and Sun Solaris. Sorry for the omission. Regards Andy G. > To: ids@iiug.org > From: art.kagel@gmail.com > Subject: Re: Large chunks versus small chunks. [17973] > Date: Fri, 6 Nov 2009 09:00:43 -0500 > > NOTE TO ALL POSTERS: PLEASE, POST YOUR VERSION AND PLATFORM INFORMATION. > Whether you think so it not, more often than not that information will > impact the answers you receive here. > > Here's my take. There are trade offs, as with everything else we do > unfortunately. At checkpoint or chunk flush time the engine assigned a > flush thread to each chunk. That means that if the dirty pages are spread > across more chunks they will tend to be flushed more quickly making your > checkpoints faster. If you are running IDS 11.xx this only impacts recovery > time and, less so, data safety. If you are running any earlier release of > IDS, it will also impact the lenght of time during a checkpoint that > tranasactions are restricted from entering critical sections and so will be > blocked. See, a version dependency ;-) > > Art > > Art S. Kagel > Oninit (www.oninit.com) > IIUG Board of Directors (art@iiug.org) > > Disclaimer: Please keep in mind that my own opinions are my own opinions and > do not reflect on my employer, Oninit, the IIUG, nor any other organization > with which I am associated either explicitly or implicitly. Neither do > those opinions reflect those of other individuals affiliated with any entity > with which I am affiliated nor those of the entities themselves. > > On Fri, Nov 6, 2009 at 8:36 AM, Andrew Grantham <agrantha@hotmail.com>wrote: > > > All, > > > > I would like gauge opinion on whether or not it is better, worse or does > > not > > matter if you use fewer larger chunks or more smaller chunks. I am trying > > to > > find out if you have any experiences or better still evidence that chunk > > size > > matters. I not over concerned about the amount of administration it > > involves. > > I am more interested in the technical/performance aspect. > > > > Thanks > > > > Andy G. > > > > _________________________________________________________________ > > New Windows 7: Find the right PC for you. Learn more. > > http://www.microsoft.com/uk/windows/buy/ > > > > > > > > > ******************************************************************************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > --0015173ff15ecf1e050477b444bf > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > _________________________________________________________________ New Windows 7: Find the right PC for you. Learn more. http://www.microsoft.com/uk/windows/buy/
> > On Fri, Nov 6, 2009 at 8:36 AM, Andrew Grantham <agrantha@hotmail.com > >wrote: > > > I would like gauge opinion on whether or not it is better, worse or > does not > > > matter if you use fewer larger chunks or more smaller chunks. I am > trying to > > > find out if you have any experiences or better still evidence that > chunk size > > > matters. I not over concerned about the amount of administration > it involves. > > > I am more interested in the technical/performance aspect. > > > From: art.kagel@gmail.com > > Date: Fri, 6 Nov 2009 09:00:43 -0500 > > > > NOTE TO ALL POSTERS: PLEASE, POST YOUR VERSION AND PLATFORM > INFORMATION. > > Whether you think so it not, more often than not that information will > > impact the answers you receive here. > > > > Here's my take. There are trade offs, as with everything else we do > > unfortunately. At checkpoint or chunk flush time the engine assigned a > > flush thread to each chunk. That means that if the dirty pages are spread > > across more chunks they will tend to be flushed more quickly making your > > checkpoints faster. If you are running IDS 11.xx this only impacts > recovery > > time and, less so, data safety. If you are running any earlier release of > > IDS, it will also impact the lenght of time during a checkpoint that > > tranasactions are restricted from entering critical sections and so will > be > > blocked. See, a version dependency ;-) > On Fri, Nov 6, 2009 at 06:56, Andrew Grantham <agrantha@hotmail.com> wrote: > > The version is 10.00.FC8 and Sun Solaris. Sorry for the omission. > If you have more than 2 GB of disk, it is usually better to create large chunks. It is a number of years since the mainstream disks were as small as 2 GB - current sizes are more like 100-500 GB and larger are available. For convenience of administration, I would strongly recommend using large chunks. Not necessarily a single large chunk - but certainly use large chunks for most purposes. -- Jonathan Leffler #include <disclaimer.h> Email: jleffler@earthlink.net, jleffler@us.ibm.com Guardian of DBD::Informix v2008.0513 -- http://dbi.perl.org/ "Blessed are we who can laugh at ourselves, for we shall never cease to be amused." NB: Please do not use this email for correspondence. I don't necessarily read it every week, even. Mike Ditka <http://www.brainyquote.com/quotes/authors/m/mike_ditka.html>; - "If God had wanted man to play soccer, he wouldn't have given us arms." --000e0cd150304493fa0477b76bfa
We are on 10 and moving to 11 soon - why would checkpoint duration not be affected by having fewer larger chunks than many smaller ones on 11.latest? School me - I have read next to nothing about 11.5. Figured we'd just blindly upgrade and see what happens... (kidding- kind of). MM
11 has non-blocking checkpoints, so the checkpoint duration should not = be an issue. = From: "MIKE MAGIE" <jmmagie@yahoo.com> = = To: ids@iiug.org = = Date: 11/06/2009 01:11 PM = = Subject: Re: Large chunks versus small chunks. [17984] = = Sent by: ids-bounces@iiug.org = = We are on 10 and moving to 11 soon - why would checkpoint duration not = be affected by having fewer larger chunks than many smaller ones on 11.lat= est? School me - I have read next to nothing about 11.5. Figured we'd just blindly upgrade and see what happens... (kidding- kind of). MM ***********************************************************************= ******** Forum Note: Use "Reply" to post a response in the discussion forum. =
11 is just as affected as 7, 9 & 10 as to fewer chunks causing longer checkpoints, however, since 11.xx has non-blocking checkpoints, the checkpoint duration no longer affects performance as it did in earlier releases. Art Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Nov 6, 2009 at 2:09 PM, MIKE MAGIE <jmmagie@yahoo.com> wrote: > We are on 10 and moving to 11 soon - why would checkpoint duration not be > affected by having fewer larger chunks than many smaller ones on 11.latest? > > School me - I have read next to nothing about 11.5. Figured we'd just > blindly > upgrade and see what happens... (kidding- kind of). > > MM > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --0015174760421562e70477b8d2b0
> All, > > I would like gauge opinion on whether or not it is better, worse or does > not matter if you use fewer larger chunks or more smaller chunks. I am > trying to find out if you have any experiences or better still evidence > that chunk size matters. I not over concerned about the amount of > administration it involves. I am more interested in the > technical/performance aspect. > I've always pushed for roughly 4-8 busy chunks per disk spindle; there are tests you can do which show the performance of a disk when it has more than one thread accessing it at the same time. Total throughput tends to climb upto a max of from 5-8 threads, then crashes back quickly to the baseline of 1 thread if you let the number busy chunks per spindle get too high. This was for ordinary disks on typical systems; ATA or SCSI when I was testing. I confess to not having tested disks in the last few years. Also when I tested bad-ass disk arrays like Clariion, they had a uniform flat throughput that was identical from 1 to N active threads - where N was as many as I could be assed to test. So there's something magical in some large arrays. This may apply to SANs but I've never tested. Also a busy chunk is counted as one that is holding physical or logical logs, or will be read on a regular basis, and probably will be written to on a regular basis. You can discount boring chunks if they have boring data that won't often be accessed, but note: if you have too many boring that come awake at the same time, the peformance would tank for the duration. I'll see if I can rustle up the disk testing tools because it would be interesting to see the results from contemporary machines or arrays.