Pagecleaners timing out (was Re: (no subject))
Posted in 1998
"Julia Barbeito Rodr'guez" <sessolucions@interbook.net> offerred:
:Hello,
:
:I am having a problem with the page cleaners. When I run tbstat -F
:(OnLine v. 5.10) I get that 2 of my four page cleaners are in an "E"
:state. I try many things, but the only solution that works is to take it
:offline, and
:online; and sometimes I even have to reboot the machine.
A page cleaner with a status of E has exited. This occurs when the page cleaner
attempts to perform a write, and does not communicate back to the main daemon
process that it has completed the write within 2 minutes (i.e. the main daemon
has waited for 2 minutes for the page cleaner to report back that the write was
successful). From what I have seen in the past, this is most commonly caused by
the LRU queues being too dirty at checkpoint time. Since a pagecleaner is
assigned to a chunk at checkpoint time, it has to locate all the pages in that
chunk and then sort them by location and finally write them all out to disk. If
you have a large buffer pool, and only a few chunks, the amount of writing
necessary at the time of the checkpoint could well exceed 2 minutes for any
single chunk. If you combine this with a chunk located on a disk with an i/o
bottleneck, you have a recipe for pagecleaner timeouts.
The best solution I can recommend is to tune your LRU_MAX_DIRTY and
LRU_MIN_DIRTY parameters down, possiblly significantly. I don't recall offhand
anymore what the default values were in 5.0 (run onstat -R to see what they are
set to), but I seem to recall them being pretty high. It is not at all unusual
to have these set as low as 2/1 (respectively) for high throughput OLTP systems.
This allows a lot of the i/o to be done between checkpoints, so the checkpoints
themselves are as short as possible. You might start out reducing their value
to 75% of their current setting, then monitor the system for cleaner timeouts.
If you still see them, reduce it further. You should probably also monitor your
i/o throughput (sar or iostat can give you this) to see if any of your chunks
are on disks with a lot of contention, or otherwise slow i/o.
Dave
**********
"Everybody plays the fool, sometimes."