Re: Long checkpoint woe

Make sure the KAIO is working. onstat -g ath | grep -i kaio will show if kaio is working, you shouldhave I kaio thread for each NUMCPU defined in your onconfig. Does onstat -g ioq show any values in the "len" and "maxlen" for any gfd's? Post it if unsure. Post onstat -d and ls -al for the symlinks for your devices. Are you sure you are pointing to the correct raw device?

"Roy Mercer" <roy.mercer@gmail.com> wrote in message news:1143638317.695988.98890@z34g2000cwc.googlegroups.com... > Make sure the KAIO is working. > onstat -g ath | grep -i kaio will show if kaio is working, you should> have I kaio thread for each NUMCPU defined in your onconfig. > > Does onstat -g ioq show any values in the "len" and "maxlen" for any > gfd's? Post it if unsure. > > Post onstat -d and ls -al for the symlinks for your devices. > Are you sure you are pointing to the correct raw device? > $ onstat -g ioq IBM Informix Dynamic Server Version 10.00.FC3 -- On-Line (Prim) -- Up 3 days 12:56:42 -- 3312640 Kbytes AIO I/O queues:q name/id len maxlen totalops dskread dskwrite dskcopy sqli_dbg 0 0 0 0 0 0 0 kio 0 0 17 2449452 2331509 117943 0 kio 1 0 16 1268293 1165337 102956 0 adt 0 0 0 0 0 0 0 msc 0 0 2 24205 0 0 0 aio 0 0 0 0 0 0 0 pio 0 0 0 0 0 0 0 lio 0 0 0 0 0 0 0 gfd 3 0 0 0 0 0 0 gfd 4 0 0 0 0 0 0 gfd 5 0 0 0 0 0 0 gfd 6 0 0 0 0 0 0 gfd 7 0 0 0 0 0 0 gfd 8 0 0 0 0 0 0 gfd 9 0 0 0 0 0 0 gfd 10 0 0 0 0 0 0 gfd 11 0 0 0 0 0 0 gfd 12 0 0 0 0 0 0 gfd 13 0 0 0 0 0 0 gfd 14 0 0 0 0 0 0 gfd 15 0 0 0 0 0 0 gfd 16 0 0 0 0 0 0 $ ls -l /opt/informix/dbspaces1total 20012 lrwxrwxrwx 1 informix informix 16 Aug 11 2005 capsdbs_1 -> /dev/md/rdsk/d67 lrwxrwxrwx 1 informix informix 16 Aug 11 2005 capsdbs_2 -> /dev/md/rdsk/d68 lrwxrwxrwx 1 root other 16 Mar 25 20:55 capsdbs2_1 -> /dev/md/rdsk/d70 lrwxrwxrwx 1 informix informix 16 Aug 11 2005 capsdbs_3 -> /dev/md/rdsk/d69 lrwxrwxrwx 1 root other 16 Mar 25 20:55 capsdbs3_1 -> /dev/md/rdsk/d71 lrwxrwxrwx 1 root other 16 Mar 25 20:55 capsdbs4_1 -> /dev/md/rdsk/d72 lrwxrwxrwx 1 root other 16 Mar 25 20:56 capsdbs5_1 -> /dev/md/rdsk/d73 lrwxrwxrwx 1 informix informix 16 Aug 11 2005 llogdbs_1 -> /dev/md/rdsk/d63 -rw-r--r-- 1 informix informix 0 Sep 14 2005 ltapedev lrwxrwxrwx 1 informix informix 16 Aug 11 2005 physdbs_1 -> /dev/md/rdsk/d62 lrwxrwxrwx 1 informix informix 16 Aug 11 2005 rootdbs_1 -> /dev/md/rdsk/d61 lrwxrwxrwx 1 informix informix 31 Oct 19 21:33 tapedev -> /opt/informix/backup/data/1.dmp lrwxrwxrwx 1 informix informix 16 Aug 11 2005 tempdbs1_1 -> /dev/md/rdsk/d64 lrwxrwxrwx 1 informix informix 16 Aug 11 2005 tempdbs2_1 -> /dev/md/rdsk/d65 lrwxrwxrwx 1 informix informix 16 Aug 11 2005 tempdbs3_1 -> /dev/md/rdsk/d66

KAIO is on and you are pointing to raw disks. This leaves the storage device as the only bottle neck. The dd times indicate there is a problem with the disk I/O. The Informix engine seems to be OK. You will have to get someone that understands your VM and SAN to make sure it is configured correctly. Thanks

Hey, I just noticed this is part of HDR. The checkpoint has to wait for the secondary server to write to disk. This could be your bottleneck also. Make sure the network connection between the two server is good. Are the disks the same on the secondary? Take the secondary server offline and then see how long your checkpoints are on the primary. FYI

Disks def. look slow but have you seen this ? bug_number 170919 description CHECKPOINT DURATION INCREASES LINEARLY DEPENDING ON NUMBER OF BUFFERS CONFIGURED DUE TO INSPECTION OF ALL BUFFERS product_code ONLINE component_code RSAM Do you have alot of buffers configured ? Haven't had a chance to test this... it was fixed in 9.4 FC7 and I think 10 FC4 ( you'll need to check this ). So possibly not what you are seeing but if its fixed in 10FC4 then its well worth testing

"Roy Mercer" <roy.mercer@gmail.com> wrote in message news:1143647081.810907.208860@u72g2000cwu.googlegroups.com... > Hey, I just noticed this is part of HDR. > The checkpoint has to wait for the secondary server to write to disk. > This could be your bottleneck also. Make sure the network connection > between the two server is good. > Are the disks the same on the secondary? > > Take the secondary server offline and then see how long your > checkpoints are on the primary. It makes little or no difference. The dd which you all reckon is slow is slow on the secondary too. I have been testing on standard (ie non-HDR) servers onm the secondary and they are slow too, eg 1700s checkpoint to write out 1.25 m dirty pages during an ALTER FRAGMENT ... INIT. cheers Neil