High CPU IO Wait

Hi, We are having recently having some CPU IO Wait issues. The CPU IO Wait value stays arount 15% . I've also noticed that CPU Eff is very low: IBM Informix Dynamic Server Version 12.10.FC7 -- On-Line -- Up 231 days 02:25:26 -- 5343628 Kbytes MT global info: sessions threads vps lngspins time 55 112 31 38 260806 sched calls thread switches yield 0 yield n yield forever total: 3209079298 3953281725 271323002 213364796 510946501 per sec: 4345841 12278 4334915 542 4195 Virtual processor summary: class vps usercpu syscpu total cpu 10 148357.28 11552.33 159909.61 aio 5 6.29 87.65 93.94 lio 1 0.24 0.51 0.75 pio 1 0.40 0.73 1.13 adm 1 5.10 5.30 10.40 soc 1 1833.92 3178.71 5012.63 msc 1 0.06 0.05 0.11 crypto 10 2555.36 196.35 2751.71 fifo 1 0.24 0.58 0.82 total 31 152758.89 15022.21 167781.10 Individual virtual processors: vp pid class usercpu syscpu total Thread Eff 1 12026 cpu 30532.56 2848.88 33381.44 61215.75 54% 2 12027 adm 5.10 5.30 10.40 0.00 0% 3 12028 lio 0.24 0.51 0.75 0.00 0% 4 12029 pio 0.40 0.73 1.13 0.00 0% 5 12030 aio 4.85 83.89 88.74 431.46 20% 6 12031 msc 0.06 0.05 0.11 0.61 18% 7 12032 fifo 0.24 0.58 0.82 0.00 0% 8 12033 cpu 23771.06 2078.80 25849.86 50232.95 51% 9 12034 cpu 20413.37 1491.44 21904.81 43167.55 50% 10 12035 cpu 15906.21 1280.29 17186.50 33191.59 51% 11 12036 cpu 12555.32 972.54 13527.86 26927.19 50% 12 12037 cpu 11260.30 719.93 11980.23 22081.00 54% 13 12038 cpu 9577.72 604.43 10182.15 18237.37 55% 14 12039 cpu 8613.38 502.04 9115.42 16041.61 56% 15 12040 cpu 8653.01 506.41 9159.42 17372.45 52% 16 12041 cpu 7074.35 547.57 7621.92 13583.73 56% 17 12042 crypto 2552.29 190.75 2743.04 2743.04 100% 18 12043 crypto 0.32 0.63 0.95 0.00 0% 19 12044 crypto 0.42 0.66 1.08 0.00 0% 20 12045 crypto 0.29 0.58 0.87 0.00 0% 21 12046 crypto 0.33 0.60 0.93 0.00 0% 22 12047 crypto 0.26 0.60 0.86 0.00 0% 23 12048 crypto 0.40 0.70 1.10 0.00 0% 24 12049 crypto 0.39 0.72 1.11 0.00 0% 25 12050 crypto 0.36 0.62 0.98 0.00 0% 26 12051 crypto 0.30 0.49 0.79 0.00 0% 27 12052 soc 1833.92 3178.71 5012.63 NA NA 28 12053 aio 0.35 0.90 1.25 2.21 56% 29 12054 aio 0.34 1.04 1.38 1.38 100% 30 14967 aio 0.36 0.97 1.33 1.33 100% 31 32478 aio 0.39 0.85 1.24 1.24 100% tot 152758.89 15022.21 167781.10 Always when there is a such high CPU IO Wait I can see lots of KAIO threads in active threads: Running threads: tid tcb rstcb prty status vp-class name 8 4aec4800 0 1 running 27soc* soctcppoll 30 4b31b8b0 0 3 running 1cpu* kaio 55 4b58dd28 0 3 running 9cpu* kaio 68 4b803178 0 3 running 10cpu* kaio 69 4b8034b8 0 3 running 11cpu* kaio 70 4b989608 0 3 running 12cpu* kaio 71 4b8037f8 0 3 running 13cpu* kaio 98 4bb73028 0 3 running 14cpu* kaio Thank you.

##onstat -c ROOTNAME rootdbs ROOTPATH /opt/ibm/informix/storage/chunks/rootdbs ROOTOFFSET 0 ROOTSIZE 2097152MIRROR 0 MIRRORPATH $INFORMIXDIR/tmp/demo_on.root_mirror MIRROROFFSET 0 PHYSFILE 2048000 PLOG_OVERFLOW_PATH $INFORMIXDIR/tmp PHYSBUFF 128 LOGFILES 101 LOGSIZE 300000 DYNAMIC_LOGS 2 LOGBUFF 64 LTXHWM 70 LTXEHWM 80 MSGPATH $INFORMIXDIR/tmp/online.log CONSOLE $INFORMIXDIR/tmp/online.con TBLTBLFIRST 0 TBLTBLNEXT 0 TBLSPACE_STATS 1DBSPACETEMP tempdbs1, tempdbs2, tempdbs3 SBSPACETEMP SBSPACENAME sbspace SYSSBSPACENAME sbspace ONDBSPACEDOWN 2 SERVERNUM 2 DBSERVERNAME infx_prod DBSERVERALIASES infx_db,lo_infx_tseries,ol_informix1210_json_01 FULL_DISK_INIT 0 NETTYPE soctcp,1,64,NET LISTEN_TIMEOUT 60 MAX_INCOMPLETE_CONNECTIONS 1024 FASTPOLL 1 NUMFDSERVERS 4 NS_CACHE host=900,service=900,user=900,group=900 MULTIPROCESSOR 1 VPCLASS cpu,num=10,aff=(0-9),noage VP_MEMORY_CACHE_KB 2048 SINGLE_CPU_VP 0 AUTO_TUNE 1 CLEANERS 8 DIRECT_IO 1 LOCKS 200000 DEF_TABLE_LOCKMODE page RESIDENT 0 SHMBASE 0x44000000L SHMVIRTSIZE 5120000 SHMADD 51200 EXTSHMADD 51200 SHMTOTAL 0 SHMVIRT_ALLOCSEG 0,3 SHMNOACCESS CKPTINTVL 300 RTO_SERVER_RESTART 0 BLOCKTIMEOUT 3600 CONVERSION_GUARD 2 RESTORE_POINT_DIR $INFORMIXDIR/tmp TXTIMEOUT 300 DEADLOCK_TIMEOUT 60 HETERO_COMMIT 0 TAPEDEV /ifxdb/archive TAPEBLK 32 TAPESIZE 0 LTAPEDEV /ifxdb/bcklog LTAPEBLK 32 LTAPESIZE 999999 BAR_ACT_LOG $INFORMIXDIR/tmp/bar_act.log BAR_DEBUG_LOG $INFORMIXDIR/tmp/bar_dbug.log BAR_DEBUG 0 BAR_MAX_BACKUP 0 BAR_MAX_RESTORE BAR_RETRY 1 BAR_NB_XPORT_COUNT 20 BAR_XFER_BUF_SIZE 128 RESTARTABLE_RESTORE ON BAR_PROGRESS_FREQ 0 BAR_BSALIB_PATH /opt/ibm/informix/lib/libbsapsm.so BACKUP_FILTER /usr/bin/pigz -7c RESTORE_FILTER /usr/bin/unpigz BAR_PERFORMANCE 0 BAR_CKPTSEC_TIMEOUT 15 PSM_DBS_POOL DBSPOOL PSM_LOG_POOL LOGPOOL DD_HASHSIZE 31 DD_HASHMAX 10 DS_HASHSIZE 31 DS_POOLSIZE 127 PC_HASHSIZE 31 PC_POOLSIZE 127 PRELOAD_DLL_FILE STMT_CACHE 0 STMT_CACHE_HITS 0 STMT_CACHE_SIZE 512 STMT_CACHE_NOLIMIT 0 STMT_CACHE_NUMPOOL 1 USEOSTIME 0 STACKSIZE 64 ALLOW_NEWLINE 0 USELASTCOMMITTED "NONE" FILLFACTOR 90 MAX_FILL_DATA_PAGES 0 BTSCANNER num=1,threshold=5000,rangesize=-1,alice=6,compression=default ONLIDX_MAXMEM 5120 MAX_PDQPRIORITY 100 DS_MAX_QUERIES DS_TOTAL_MEMORY 250000 DS_MAX_SCANS 1048576 DS_NONPDQ_QUERY_MEM 25000 DATASKIP OPTCOMPIND 2 DIRECTIVES 1 EXT_DIRECTIVES 0 OPT_GOAL -1 IFX_FOLDVIEW 1 STATCHANGE 10 USTLOW_SAMPLE 1 BATCHEDREAD_TABLE 1 BATCHEDREAD_INDEX 1 EXPLAIN_STAT 1 IFX_EXTEND_ROLE 1 SECURITY_LOCALCONNECTION UNSECURE_ONSTAT ADMIN_USER_MODE_WITH_DBSA ADMIN_MODE_USERS SSL_KEYSTORE_LABEL TLS_VERSION PLCY_POOLSIZE 127 PLCY_HASHSIZE 31 USRC_POOLSIZE 127 USRC_HASHSIZE 31 SQL_LOGICAL_CHAR OFF SEQ_CACHE_SIZE 10 ENCRYPT_HDR ENCRYPT_SMX ENCRYPT_CDR 0 ENCRYPT_CIPHERS ENCRYPT_MAC ENCRYPT_MACFILE ENCRYPT_SWITCH CDR_EVALTHREADS 1,2 CDR_DSLOCKWAIT 5 CDR_QUEUEMEM 4096 CDR_NIFCOMPRESS 0 CDR_SERIAL 0 CDR_DBSPACE CDR_QHDR_DBSPACE CDR_QDATA_SBSPACE CDR_SUPPRESS_ATSRISWARN CDR_DELAY_PURGE_DTC 0 CDR_LOG_LAG_ACTION ddrblock CDR_LOG_STAGING_MAXSIZE 0 CDR_MAX_DYNAMIC_LOGS 0GRIDCOPY_DIR $INFORMIXDIR CDR_TSINSTANCEID 0 CDR_MAX_FLUSH_SIZE 50 CDR_AUTO_DISCOVER 0 CDR_MEM 0 SHARD_MEM 0 SHARD_ID 0 DRAUTO 0 DRINTERVAL 0 HDR_TXN_SCOPE NEAR_SYNC DRTIMEOUT 30 HA_ALIAS HA_FOC_ORDER SDS,HDR,RSSDRLOSTFOUND $INFORMIXDIR/etc/dr.lostfound DRIDXAUTO 0 LOG_INDEX_BUILDS SDS_ENABLE SDS_TIMEOUT 20 SDS_TEMPDBS SDS_PAGING SDS_LOGCHECK 10 SDS_ALTERNATE NONE SDS_FLOW_CONTROL 0 UPDATABLE_SECONDARY 0 FAILOVER_CALLBACK FAILOVER_TX_TIMEOUT 0 TEMPTAB_NOLOG 0 DELAY_APPLY 0 STOP_APPLY 0 LOG_STAGING_DIR RSS_FLOW_CONTROL 0 SMX_NUMPIPES 1 ENABLE_SNAPSHOT_COPY 0 SMX_COMPRESS 0 SMX_PING_INTERVAL 10 SMX_PING_RETRY 6 CLUSTER_TXN_SCOPE SERVER ON_RECVRY_THREADS 1 OFF_RECVRY_THREADS 10 DUMPDIR $INFORMIXDIR/tmp DUMPSHMEM 1 DUMPGCORE 0 DUMPCORE 0 DUMPCNT 1ALARMPROGRAM $INFORMIXDIR/etc/alarmprogram.sh ALRM_ALL_EVENTS 0 STORAGE_FULL_ALARM 600,3SYSALARMPROGRAM $INFORMIXDIR/etc/evidence.sh RAS_PLOG_SPEED 55196 RAS_LLOG_SPEED 975 EILSEQ_COMPAT_MODE 0 QSTATS 0 WSTATS 0 USERMAPPING OFF SP_AUTOEXPAND 1 SP_THRESHOLD 0 SP_WAITTIME 30 AUTOLOCATE 0 DEFAULTESCCHAR \\\\ MQSERVER MQCHLLIB MQCHLTAB REMOTE_SERVER_CFG hosts.equiv REMOTE_USERS_CFG hosts.equiv S6_USE_REMOTE_SERVER_CFG 1 LOW_MEMORY_RESERVE 0 LOW_MEMORY_MGR 0 JVPPROPFILE $INFORMIXDIR/extend/krakatoa/.jvpprops JVPLOGFILE $INFORMIXDIR/tmp/jvp.log JVPARGS -Dcom.ibm.tools.attach.enable=no JVPCLASSPATH $INFORMIXDIR/extend/krakatoa/krakatoa.jar:$INFORMIXDIR/extend/krakatoa/jdbc.jar BUFFERPOOL default,buffers=10000,lrus=8,lru_min_dirty=50,lru_max_dirty=60.5 BUFFERPOOL size=2K,buffers=50000,lrus=8,lru_min_dirty=50,lru_max_dirty=60 BUFFERPOOL size=8K,buffers=10000,lrus=8,lru_min_dirty=50,lru_max_dirty=60.5

Hello. Your engine configuration is pretty standard, so you are with a Ferrari, driving on a slow speed lane. You must tune it as soon as possible. Pretty sure you are missing buffers, so I would suggest you to tune then first if all. Several other parameters could bring you a great performance benefit. Hope it helps. Best regards. Alexandre Marini

Arba, are there other applications running on the database server ? it appears that your database buffers are set way too low (80 meg & 100 meg) so you probably are experiencing buffer waits and are turning them over many times per hour. Mark