Re: chekcpoints, BUFFER settings, and optimization

David Kosenko wrote: > > "Art S. Kagel" <kagel@bloomberg.com> wrote: > +Dave I have to disagree. I agree that additional cleaners are of > +limited use during normal LRU writes but they are needed for fastest > +checkpointing. Since at checkpoint time each chunk is assigned to > +another cleaner thread until the threads are exhausted, and since that > +one thread, as has already been pointed out, is only scheduling the > +actual I/Os with either the AIO VPs or KAIO VPs, that thread will block > +on each I/O that it schedules and single thread writes to your disks. > +You must have multiple cleaners, even with a single CPU VP, since the > +other cleaners can wake and schedule the I/Os that they are responsible > +for while the first cleaner is blocked waiting for I/O service. > > The cleaner threads do not block on the i/o calls. There are NO blocking > calls whatsoever in the CPU VPs. > > Besides, if a cleaner thread "blocked", it would not be yielding the cpu vp, in > which case having more cleaner threads than cpu vps would still not make any > sense. I'm with Dave Williams on this one. I was not clear. Perhaps 'blocked' is not the correct word here. My point is that the single cleaner thread has to wait for the issued I/O to complete. It does relinquish the CPU VP to other threads so that other work is not blocked, but, only one disk is cleaned at a time. On any significant system this is takes too long. I am not constrained to a single CPU VP but the sheer size of our systems provide similar behavioral situations, observe. We have 32 SLOW CPUs, 4 reserved for system processes and our proprietary IPC processes, with 28 CPU VPs. KAIO on DG M88k is slow so we still use AIO VPs. Our typical server has either 200,000 or 300,000 buffers in 128 LRUs we run ~120 AIO VPs (different machines have different peak load needs). We must keep our checkpoints < 8 seconds or our data feed applications fall behind real time performance requirements. To do this we keep MIN_DIRTY = 0 and MAX_DIRTY = (1 or 2). I have tried cleaner values <= the number of CPU VPs and checkpoint times are ALWAYS too long. We have found that having 2-2.5 cleaner threads per CPU VP makes certain that a cleaner is always ready for service when a VP has cycles available and checkpoints are minimized. Because of the large number of LRUs we use (32 are too few as contention for buffers increases and I have made clear my belief that there is a bug in the LRU access hashing with 64 LRUs, someone else reported similar problems to me with 96 LRUs, so since 128 works...) my advice to use 1.5 cleaners per LRU is overkill for us but for 32 LRUs or less I believe that that figure is probably correct. At any rate it is simple enough to test. Just choose a configuration (1 cleaner or N cleaners) load some data and read the checkpoint times from the log then change the value and try again. Let me modify my recommendation slightly. I keep forgetting most installations do not have 258 chunks. So I recommend 1.5 * min(#chunks,#LRUs) as the number of cleaners keep the 2-2.5 * NUMCPUs idea for multiple CPUs in mind and constrained by a maximum useful value of about 64 for multiple CPU systems. I have not worked with single CPU installations enough to guess at a maximum value but I STRONGLY suspect that it is MUCH greater than one (1)! I know that this sound complex, but the configuration that I use is the result of months of testing and the recommendations are an attempt to extrapolate a formula from these results. If someone will try to apply my formula and Dave's recommendation to a and single CPU machine and come up specific empirical recommendations I would like to see the outcome. That would be the best way to settle this debate. Art S. Kagel