Re: IDS9.4/AIX5.2 performace degradation

On Thu, 22 Jul 2004 08:00:15 -0400, Dalibor Krleza wrote: > Hello everybody! > > Well, we are experiencing following problems: 1) Before migration to new > system (server and operating system) we had installed IDS 7.31 database on > Caldera operating system. > > 2) We migrated on IDS9.4 FC4, operating system AIX 5.2 on IBM pSeries server > (p630). > > All configuration setting for database were copied to a new server. > > In meanwhile we are constantly experiencing significant performace > degradation, like 4x slower than before. > Any ideas?? Several. Like Neil and Keith I'm going to recommend that you start with running the recommended suite of UPDATE STATISTICS commands as outlined in the Performance Guide. If you've already updated stats, please post the command(s) or a sample of the commands you used. If you want help tuning the beast post the following (not as attachments!): System description: # CPUs, disk farm (RAID levels, # spindles, local or SAN, etc.), contents of you ONCONFIG file, elapsed time since onstat -z was last run (or since startup if never since) output from: onstat -p onstat -P onstat -d onstat -D onstat -g glo onstat -g iov onstat -g iof onstat -F onstat -R onstat -l onstat -T onstat -g dsc onstat -g dic onstat -g prc onstat -g env onstat -g seg Art S. Kagel

OK Art, I will post these lists. Hope you'll find something. UPDATE STATISTICS - done. After US recreated all indexes, same thing. Everyone else, sorry for long post! I'll try to be short as possible by kicking all remarks out of lists. ***ONCONFIG*** ROOTNAME rootdbs ROOTPATH /dev/rlvinfx01 ROOTOFFSET 16 ROOTSIZE 30000MIRROR 0 MIRRORPATH MIRROROFFSET 0PHYSDBS fizldbs PHYSFILE 48000 LOGFILES 150 LOGSIZE 2000MSGPATH /informix/online.log CONSOLE /informix/console.log ALARMPROGRAM /informix/etc/no_log.sh TBLSPACE_STATS 1 TAPEDEV /dev/rmt1 TAPEBLK 256 TAPESIZE 209715200 LTAPEDEV /dev/null LTAPEBLK 32 LTAPESIZE 8388608STAGEBLOB SERVERNUM 0 DBSERVERNAME kov_dbs DBSERVERALIASES DEADLOCK_TIMEOUT 60 RESIDENT 0 MULTIPROCESSOR 1 NUMCPUVPS 2 SINGLE_CPU_VP 0 NOAGE 0 AFF_SPROC 0 AFF_NPROCS 0 LOCKS 200000 BUFFERS 200000 NUMAIOVPS PHYSBUFF 32 LOGBUFF 32 CLEANERS 1 SHMBASE 0x700000000000000 SHMVIRTSIZE 8000 SHMADD 8192 SHMTOTAL 0 CKPTINTVL 300 LRUS 8 LRU_MAX_DIRTY 2 LRU_MIN_DIRTY 1 TXTIMEOUT 0x12c STACKSIZE 64 DYNAMIC_LOGS 2 LTXHWM 70 LTXEHWM 80 OFF_RECVRY_THREADS 10 ON_RECVRY_THREADS 1 DRINTERVAL 30 DRTIMEOUT 30 DRLOSTFOUND /usr/informix/etc/dr.lostfound CDR_EVALTHREADS 1,2 CDR_DSLOCKWAIT 5 CDR_QUEUEMEM 4096 CDR_NIFCOMPRESS 0 CDR_SERIAL 0,0 CDR_DBSPACE CDR_QHDR_DBSPACE CDR_QDATA_SBSPACE CDR_MAX_DYNAMIC_LOGS 0 BAR_ACT_LOG /usr/informix/bar_act.log BAR_DEBUG_LOG /usr/informix/bar_dbug.log BAR_MAX_BACKUP 0 BAR_RETRY 1 BAR_NB_XPORT_COUNT 10 BAR_XFER_BUF_SIZE 31 RESTARTABLE_RESTORE on BAR_PROGRESS_FREQ 0ISM_DATA_POOL ISMData ISM_LOG_POOL ISMLogs RA_PAGES RA_THRESHOLDDBSPACETEMP tmp1,tmp2,tmp3 DUMPDIR /tmp DUMPSHMEM 0 DUMPGCORE 0 DUMPCORE 0 DUMPCNT 1 FILLFACTOR 90 USEOSTIME 0 MAX_PDQPRIORITY 100 DS_MAX_QUERIES DS_TOTAL_MEMORY DS_MAX_SCANS 1048576 DATASKIP off OPTCOMPIND 2 DIRECTIVES 1 ONDBSPACEDOWN 2OPCACHEMAX 0 HETERO_COMMIT 0 SBSPACENAME SYSSBSPACENAME BLOCKTIMEOUT 3600SYSALARMPROGRAM /informix/etc/evidence.sh OPT_GOAL -1 ALLOW_NEWLINE 0 JVPJAVAHOME /usr/informix/extend/krakatoa/jreJVPHOME /usr/informix/extend/krakatoa JVPPROPFILE /usr/informix/extend/krakatoa/.jvpprops JDKVERSION 1.3 JVPJAVALIB /bin JVPJAVAVM hpi:jvm:java:net:zip:jpeg JVPCLASSPATH /usr/informix/extend/krakatoa/krakatoa.jar:/usr/informix/extend/krakatoa/jdbc.jar ***ONSTAT -p*** IBM Informix Dynamic Server Version 9.40.FC4 -- On-Line -- Up 06:13:43 -- 943248 Kbytes Profile dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached 6561 7206 411016 98.40 29 376 244 88.11 isamtot open start read write rewrite delete commit rollbk 490089 21666 23427 319744 33 1 0 2 0 gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs 0 0 0 0 0 0 0 ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes 0 0 0 8.08 4.43 0 0 bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans 261 0 301728 0 0 0 16 13 ixda-RA idx-RA da-RA RA-pgsused lchwaits 433 0 6 436 16 ***ONSTAT -P*** IBM Informix Dynamic Server Version 9.40.FC4 -- On-Line -- Up 06:44:14 -- 943248 Kbytespartnum total btree data other dirty 0 11 0 7 4 4 1048578 2 1 1 0 0 4194305 1 0 0 1 1 4195019 1 0 0 1 1 4195020 1 0 0 1 1 4195058 1 0 0 1 0 4195062 1 0 0 1 0 4195065 1 0 0 1 0 4195066 1 0 0 1 0 4195067 1 0 0 1 0 4195068 1 0 0 1 0 4195069 1 0 0 1 0 4195072 1 0 0 1 0 4195073 1 0 0 1 0 4195074 1 0 0 1 0 4195075 1 0 0 1 0 4195076 1 0 0 1 0 4195077 1 0 0 1 0 4195078 1 0 0 1 0 4195079 1 0 0 1 0 4195080 1 0 0 1 0 4195081 1 0 0 1 0 4195082 1 0 0 1 0 4195083 1 0 0 1 0 4195084 1 0 0 1 0 4195085 1 0 0 1 0 4195086 1 0 0 1 0 4195087 1 0 0 1 0 4195088 1 0 0 1 0 4195089 1 0 0 1 0 4195090 1 0 0 1 0 4195091 1 0 0 1 0 4195092 1 0 0 1 0 4195093 1 0 0 1 0 4195094 1 0 0 1 0 4195095 1 0 0 1 0 4195096 1 0 0 1 0 4195097 1 0 0 1 0 4195098 1 0 0 1 0 4195099 1 0 0 1 0 4195100 1 0 0 1 0 4195101 1 0 0 1 0 4195102 1 0 0 1 0 4195103 1 0 0 1 0 4195104 1 0 0 1 0 4195105 3 0 2 1 0 4195106 1 0 0 1 0 4195107 1 0 0 1 0 4195108 1 0 0 1 0 4195109 1 0 0 1 0 4195110 1 0 0 1 0 4195111 1 0 0 1 0 4195112 1 0 0 1 0 4195113 1 0 0 1 0 4195114 1 0 0 1 1 4195115 1 0 0 1 1 4195116 1 0 0 1 1 4195117 1 0 0 1 1 4195118 1 0 0 1 1 4195119 1 0 0 1 1 4195120 1 0 0 1 1 4195121 1 0 0 1 1 4195122 4 1 1 2 4 4195123 1 0 0 1 1 4195124 1 0 0 1 1 4195125 1 0

Dalibor Krleza wrote: I may have missed it in your long post, but I didn't see anything about disc layouts or number of CPUs. Also I may have missed it as well but I couldn't see any output from "onstat -g iof" or "onstat -g iov". I have commented on your onconfig anyway. > OK Art, I will post these lists. Hope you'll find something. > UPDATE STATISTICS - done. After US recreated all indexes, same thing.> > Everyone else, sorry for long post! I'll try to be short as possible by kicking all remarks out of lists. > > ***ONCONFIG*** > > ROOTNAME rootdbs > ROOTPATH /dev/rlvinfx01 > ROOTOFFSET 16 > ROOTSIZE 30000> MIRROR 0 > MIRRORPATH > MIRROROFFSET 0> PHYSDBS fizldbs > PHYSFILE 48000 > LOGFILES 150 > LOGSIZE 2000 Personally I think your log size is very small although others may have more of an idea about this than me. > MSGPATH /informix/online.log > CONSOLE /informix/console.log > ALARMPROGRAM /informix/etc/no_log.sh > TBLSPACE_STATS 1 Try setting this to 0 unless you need the stats. > TAPEDEV /dev/rmt1 > TAPEBLK 256 > TAPESIZE 209715200 > LTAPEDEV /dev/null > LTAPEBLK 32 > LTAPESIZE 8388608> STAGEBLOB > SERVERNUM 0 > DBSERVERNAME kov_dbs > DBSERVERALIASES > DEADLOCK_TIMEOUT 60 > RESIDENT 0 Try setting to 1 or -1. > MULTIPROCESSOR 1 > NUMCPUVPS 2 Comment this out and use VPCLASS - see 9.40 performance guide. > SINGLE_CPU_VP 0 > NOAGE 0 > AFF_SPROC 0 > AFF_NPROCS 0 Comment out NOAGE, AFF_* and use VPCLASS - see 9.40 performance guide. > LOCKS 200000 > BUFFERS 200000 > NUMAIOVPS Comment this out and use VPCLASS - see 9.40 performance guide. > PHYSBUFF 32 > LOGBUFF 32 > CLEANERS 1 How many discs are used by IDS for data? Generally speaking Ttis parameter should be set to the number of discs you use. Any RAID discs that appear as one disc to the OS count as 1. The 9.40 performance guide has more information. > SHMBASE 0x700000000000000 > SHMVIRTSIZE 8000 You have a lot of shared memory segments. Try increasing this to more like 80000, rather than 8000. Check then how many segments you have with onstat -g seg. Ideally you want just one but shared memory should not bebigger than it needs to be. > SHMADD 8192 > SHMTOTAL 0 > CKPTINTVL 300 > LRUS 8 Read up on LRUS in the 9.40 performance guide. > LRU_MAX_DIRTY 2 > LRU_MIN_DIRTY 1 You may get more throughput if you set these values higher as long as checkpoint times do not get too high. > TXTIMEOUT 0x12c > STACKSIZE 64 > DYNAMIC_LOGS 2 > LTXHWM 70 > LTXEHWM 80 > OFF_RECVRY_THREADS 10 > ON_RECVRY_THREADS 1 > DRINTERVAL 30 > DRTIMEOUT 30 > DRLOSTFOUND /usr/informix/etc/dr.lostfound > CDR_EVALTHREADS 1,2 > CDR_DSLOCKWAIT 5 > CDR_QUEUEMEM 4096 > CDR_NIFCOMPRESS 0 > CDR_SERIAL 0,0 > CDR_DBSPACE > CDR_QHDR_DBSPACE > CDR_QDATA_SBSPACE > CDR_MAX_DYNAMIC_LOGS 0 > BAR_ACT_LOG /usr/informix/bar_act.log > BAR_DEBUG_LOG /usr/informix/bar_dbug.log > BAR_MAX_BACKUP 0 > BAR_RETRY 1 > BAR_NB_XPORT_COUNT 10 > BAR_XFER_BUF_SIZE 31 > RESTARTABLE_RESTORE on > BAR_PROGRESS_FREQ 0> ISM_DATA_POOL ISMData > ISM_LOG_POOL ISMLogs > RA_PAGES > RA_THRESHOLD> DBSPACETEMP tmp1,tmp2,tmp3 > DUMPDIR /tmp > DUMPSHMEM 0 > DUMPGCORE 0 > DUMPCORE 0 > DUMPCNT 1 > FILLFACTOR 90 > USEOSTIME 0 > MAX_PDQPRIORITY 100 > DS_MAX_QUERIES > DS_TOTAL_MEMORY > DS_MAX_SCANS 1048576 > DATASKIP off > OPTCOMPIND 2 Try 0 and see if this helps. It may or may not. > DIRECTIVES 1 > ONDBSPACEDOWN 2> OPCACHEMAX 0 > HETERO_COMMIT 0 > SBSPACENAME > SYSSBSPACENAME > BLOCKTIMEOUT 3600> SYSALARMPROGRAM /informix/etc/evidence.sh > OPT_GOAL -1 > ALLOW_NEWLINE 0 > JVPJAVAHOME /usr/informix/extend/krakatoa/jre> JVPHOME /usr/informix/extend/krakatoa > JVPPROPFILE /usr/informix/extend/krakatoa/.jvpprops > JDKVERSION 1.3 > JVPJAVALIB /bin > JVPJAVAVM hpi:jvm:java:net:zip:jpeg > JVPCLASSPATH /usr/informix/extend/krakatoa/krakatoa.jar:/usr/informix/extend/krakatoa/jdbc.jar

Ups, sorry.. here it is ***ONSTAT -g iov*** IBM Informix Dynamic Server Version 9.40.FC4 -- On-Line -- Up 06:16:20 -- 943248 Kbytes AIO I/O vps:class/vp s io/s totalops dskread dskwrite dskcopy wakeups io/wup errors kio 0 i 125.3 21921 21000 921 0 63403 0.3 0 kio 1 i 267.7 46843 46291 552 0 90272 0.5 0 msc 0 i 0.1 18 0 0 0 18 1.0 0 aio 0 i 0.2 27 9 0 0 27 1.0 0 aio 1 i 0.0 0 0 0 0 0 0.0 0 aio 2 i 0.0 0 0 0 0 0 0.0 0 aio 3 i 0.0 0 0 0 0 0 0.0 0 aio 4 i 0.0 0 0 0 0 0 0.0 0 aio 5 i 0.0 0 0 0 0 0 0.0 0 aio 6 i 0.0 0 0 0 0 0 0.0 0 aio 7 i 0.0 0 0 0 0 0 0.0 0 aio 8 i 0.0 0 0 0 0 0 0.0 0 aio 9 i 0.0 0 0 0 0 0 0.0 0 aio 10 i 0.0 0 0 0 0 0 0.0 0 aio 11 i 0.0 0 0 0 0 0 0.0 0 aio 12 i 0.0 0 0 0 0 0 0.0 0 aio 13 i 0.0 0 0 0 0 0 0.0 0 aio 14 i 0.0 0 0 0 0 0 0.0 0 aio 15 i 0.0 0 0 0 0 0 0.0 0 aio 16 i 0.0 0 0 0 0 0 0.0 0 aio 17 i 0.0 0 0 0 0 0 0.0 0 aio 18 i 0.0 0 0 0 0 0 0.0 0 aio 19 i 0.0 0 0 0 0 0 0.0 0 aio 20 i 0.0 0 0 0 0 0 0.0 0 aio 21 i 0.0 0 0 0 0 0 0.0 0 aio 22 i 0.0 0 0 0 0 0 0.0 0 aio 23 i 0.0 0 0 0 0 0 0.0 0 pio 0 i 0.0 0 0 0 0 0 0.0 0 lio 0 i 0.0 0 0 0 0 0 0.0 0 ***ONSTAT -g iof*** IBM Informix Dynamic Server Version 9.40.FC4 -- On-Line -- Up 06:16:34 -- 943248 Kbytes AIO global files: gfd pathname totalops dskread dskwrite io/s 3 /dev/rlvinfx01 8 4 4 0.0 4 /dev/rlvinfx03 42 0 42 0.2 5 /dev/rlvinfx05 771 0 771 4.1 6 /dev/rlvinfx01 194 46 148 1.0 7 /dev/rlvinfx03 260 100 160 1.4 8 /dev/rlvinfx05 212 51 161 1.1 9 /dev/rlvinfx01 2332 2126 206 12.3 10 /dev/rlvinfx03 67252 67169 83 355.8 11 /dev/rlvinfx05 1893 1853 40 10.0 12 /dev/rlvinfx02 892 822 70 4.7 13 /dev/rlvinfx04 0 0 0 0.0 14 /dev/rlvinfx06 0 0 0 0.0

Dalibor Krleza wrote: > Ups, sorry.. here it is > ***ONSTAT -g iof*** > > IBM Informix Dynamic Server Version 9.40.FC4 -- On-Line -- Up 06:16:34 -- 943248 Kbytes> > AIO global files: > gfd pathname totalops dskread dskwrite io/s > 3 /dev/rlvinfx01 8 4 4 0.0 > 4 /dev/rlvinfx03 42 0 42 0.2 > 5 /dev/rlvinfx05 771 0 771 4.1 > 6 /dev/rlvinfx01 194 46 148 1.0 > 7 /dev/rlvinfx03 260 100 160 1.4 > 8 /dev/rlvinfx05 212 51 161 1.1 > 9 /dev/rlvinfx01 2332 2126 206 12.3 > 10 /dev/rlvinfx03 67252 67169 83 355.8 > 11 /dev/rlvinfx05 1893 1853 40 10.0 > 12 /dev/rlvinfx02 892 822 70 4.7 > 13 /dev/rlvinfx04 0 0 0 0.0 > 14 /dev/rlvinfx06 0 0 0 0.0 That looks like quite an impressive amount of IO and all on one raw device? You seem to be using some sort of logical volume manager which I can't help you with, having never got my hands on such juicy kit. Have you tried reading the performance guide for 9.40 and applying the changes recommended earlier to your set up? Ben.

On Fri, 23 Jul 2004 06:06:52 -0400, Ben Thompson wrote: Dalibor, I did not see the post that Ben is replying to at all, I am commenting on what's below, but if you send me the full posting with the SYSTEM DESCRIPTION and the onstat's I requested, via email I'll look it all over. Ben's comments are mostly well taken. See below. Art S. Kagel > Dalibor Krleza wrote: > > I may have missed it in your long post, but I didn't see anything about disc > layouts or number of CPUs. Also I may have missed it as well but I couldn't > see any output from "onstat -g iof" or "onstat -g iov". I have commented on > your onconfig anyway. > >> OK Art, I will post these lists. Hope you'll find something. UPDATE >> STATISTICS - done. After US recreated all indexes, same thing. >> >> Everyone else, sorry for long post! I'll try to be short as possible by >> kicking all remarks out of lists. >> >> ***ONCONFIG*** >> >> ROOTNAME rootdbs >> ROOTPATH /dev/rlvinfx01 >> ROOTOFFSET 16 >> ROOTSIZE 30000>> MIRROR 0 >> MIRRORPATH >> MIRROROFFSET 0>> PHYSDBS fizldbs >> PHYSFILE 48000 >> LOGFILES 150 >> LOGSIZE 2000> > Personally I think your log size is very small although others may have more > of an idea about this than me. > >> MSGPATH /informix/online.log >> CONSOLE /informix/console.log >> ALARMPROGRAM /informix/etc/no_log.sh TBLSPACE_STATS 1 > > Try setting this to 0 unless you need the stats. > >> TAPEDEV /dev/rmt1 >> TAPEBLK 256 >> TAPESIZE 209715200 >> LTAPEDEV /dev/null >> LTAPEBLK 32 >> LTAPESIZE 8388608>> STAGEBLOB >> SERVERNUM 0 I do not like using SERVERNUM zero. It works but almost always causes problems when you try to bring up a second server. FWIW. >> DBSERVERNAME kov_dbs >> DBSERVERALIASES >> DEADLOCK_TIMEOUT 60 >> RESIDENT 0 RESIDENT 1 can be a big performance improvement! > Try setting to 1 or -1. > >> MULTIPROCESSOR 1 >> NUMCPUVPS 2> > Comment this out and use VPCLASS - see 9.40 performance guide. > >> SINGLE_CPU_VP 0 >> NOAGE 0 I don't remember if there is any issue with using NOAGE on AIX, but this can also improve performance if the OS is aggressive about lowering the priority of long running processes. Solaris and especially HPUX are very aggressive, IB AIX is less so, but it can still help, especially if you notice that performance is better for a day or so after you bounce the engine. >> AFF_SPROC 0 >> AFF_NPROCS 0> > Comment out NOAGE, AFF_* and use VPCLASS - see 9.40 performance guide. > >> LOCKS 200000 >> BUFFERS 200000 >> NUMAIOVPS Looking at the onstat -g iov output in your other post I see that only one AIO VP is normally used. Since you have KAIO enabled and it would seem that all chunks are RAW devices, you do not need all 24 AIO VPs that are being automatically allocated for you because you did not set a value here. Set this to 4 to 6 (better take Ben's advice and use a VPCLASS....AIO instead to set the number of AIO VPs). > Comment this out and use VPCLASS - see 9.40 performance guide. > >> PHYSBUFF 32 >> LOGBUFF 32 >> CLEANERS 1 This may be the reason why you are seeing so many CHUNK writes and FGwrites. See below for my recommendation for LRUS and set CLEANERS >= LRUS to improve LRU cleaning (and as Ben says if you have many disks set it >= numdisks or numchunks to improve checkpoint performance. > How many discs are used by IDS for data? Generally speaking Ttis parameter > should be set to the number of discs you use. Any RAID discs that appear as > one disc to the OS count as 1. The 9.40 performance guide has more > information. > >> SHMBASE 0x700000000000000 >> SHMVIRTSIZE 8000 I did not see the onstat -g seg output, but assuming Ben is correct, it would be VERY good to fold the extra virtual segments into the size of the initial segment as AIX does not do well with multiple segments. Also, IMS, AIX has some kind of minimal allocation so that segments < 256MB are rounded up internally, it's a waste to make smaller ones. > You have a lot of shared memory segments. Try increasing this to more like > 80000, rather than 8000. Check then how many segments you have with onstat > -g seg. Ideally you want just one but shared memory should not be bigger > than it needs to be. > >> SHMADD 8192 >> SHMTOTAL 0 >> CKPTINTVL 300 >> LRUS 8 Given the BR of 9.51 that I calculated from your original post, I'd increase this to at least 16 (avoid 32, 64, and 96 these specific values have caused performance degradation in previous versions and I do not know if it's been fixed since Informix & IBM have never acknowledged the bug). Don't forget to match or exceed the new LRUS value when setting CLEANERS. > > Read up on LRUS in the 9.40 performance guide. > >> LRU_MAX_DIRTY 2 >> LRU_MIN_DIRTY 1 With values this low you should not be seeing mostly chunk writes and FG writes as you are seeing. The only reason I can see would be the single CLEANER or it may be that you just do not have enough BUFFERS configured. If I get to see onstat -P and onstat -p output and time since stats were zero'd I can better determine if you need more buffers. > You may get more throughput if you set these values higher as long as > checkpoint times do not get too high. > >> TXTIMEOUT 0x12c >> STACKSIZE 64 >> DYNAMIC_LOGS 2 >> LTXHWM 70 >> LTXEHWM 80 >> OFF_RECVRY_THREADS 10 >> ON_RECVRY_THREADS 1 >> DRINTERVAL 30 >> DRTIMEOUT 30 >> DRLOSTFOUND /usr/informix/etc/dr.lostfound CDR_EVALTHREADS 1,2 >> CDR_DSLOCKWAIT 5 >> CDR_QUEUEMEM 4096 >> CDR_NIFCOMPRESS 0 >> CDR_SERIAL 0,0 >> CDR_DBSPACE >> CDR_QHDR_DBSPACE >> CDR_QDATA_SBSPACE >> CDR_MAX_DYNAMIC_LOGS 0 >> BAR_ACT_LOG /usr/informix/bar_act.log BAR_DEBUG_LOG >> /usr/informix/bar_dbug.log BAR_MAX_BACKUP 0 BAR_RETRY 1 >> BAR_NB_XPORT_COUNT 10 >> BAR_XFER_BUF_SIZE 31 >> RESTARTABLE_RESTORE on >> BAR_PROGRESS_FREQ 0>> ISM_DATA_POOL ISMData >> ISM_LOG_POOL ISMLogs >> RA_PAGES >> RA_THRESHOLD You are using defaults, and since your RAU is 99.99% it's not a real worry, but, the defaults may be stressing your IO systems and delaying writes, especially given your VERY LOW read cache percentage (62%). Read cache should ideally be in the high 90% range. >> DBSPACETEMP tmp1,tmp2,tmp3 >> DUMPDIR /tmp >> DUMPSHMEM 0 >> DUMPGCORE 0 >> DUMPCORE 0 >> DUMPCNT 1 >> FILLFACTOR 90 >> USEOSTIME 0 >> MAX_PDQPRIORITY 1002 >> DS_MAX_QUERIES >> DS_TOTAL_MEMORY >> DS_MAX_SCANS 1048576 >> DATASKIP off >> OPTCOMPIND 2> > Try 0 and see if this helps. It may or may not. > >> DIRECTIVES 1 >> ONDBSPACEDOWN 2