Re: KAIO vps, and bufwaits

Even LRUS need love... How many CLEANERS and LRUS per CPU VP are you running? I've been using the "rule" of CLEANERS = LRUS = CPU VPs. What you're saying about the hashing makes good sense, I'm just trying to find a starting point for our machines (one has high BUFWAITs > 18%). Thanks, Doug B. <<snip>> > >When a thread needs a buffer to update and the page is not already in >the buffer pool the thread uses a random hash to select an LRU to get >the buffer from. If this LRU is not locked by another process doing >the same thing the thread acquires the lock on that LRU queue takes >the Least Recently Used clean buffer, moves it to the LRU queue's >dirty list, locks the buffer page and releases the lock on the LRU. >If the LRU is already locked it spins (BUFWAIT) for a while trying a >few more times to get a lock on that LRU if not it rehashes to another >queue. If there are many updaters and few LRUs it is likely that that >LRU will also be locked already and so our poor thread spins again and >so on. Eventually it gets a lock on sone LRU and continues. If there >are more LRUs then the probablility of finding a free one improves >greatly and BUFWAITS go down and throughput goes up. > >If the page to be updated is already in the cache but has not yet been >modified the number of LRUs is still important as the size of the buffer >cache increases. More cache means it is more likely that a page that is >about to be modified is already in the clean cache (or the dirty cache for >that matter but that is not affected by the number of LRUs). With more >LRUs it is less likely that some other process is in the process of moving >some other already cached clean page to the dirty list from the same LRU >that contains the page that your thread needs. Again this reduces >BUFWAITS and improves throughput. > >The bug I suspect is that for certain numbers of LRUs the rehash >algorithm causes many threads to rehash to the same sequence of LRUs >so that the acquisition of LRUs is a race condition and single >threaded. Here one thread at a time gets an LRU and drops out of the >race the symptoms are that updates/inserts/deletes do not scale (ie >adding another updater does not improve throughput significantly) and >BUFWAITS skyrocket. We were seeing both of these symptoms. > >With 128 LRUs (the maximum) we can have 100 update server tasks running >constantly serving update requests from clients and still get near linear >gains from adding up to 40 data sync or cleanup tasks concurrent with >normal production load. > >An additional benefit of many LRUs is that it helps to keep more buffers >flushed to disk by the LRU_MIN_DIRTY/LRU_MAX_DIRTY parameters. Because of >the hashing used to select LRUs the LRUs fill unevenly. More LRUs fill >less evenly than fewer LRUs meaning fewer dirty buffers at CHECKPOINT time >and faster checkpoints. Also more LRUs can take advantage of more CLEANERS >for faster checkpoints. > >Hope this explains my love of LRUs. > >Art S. Kagel, kagel@bloomberg.com >