High CPU IO Wait
Hi,
We are having recently having some CPU IO Wait issues. The CPU IO Wait value
stays arount 15% .
I've also noticed that CPU Eff is very low:
IBM Informix Dynamic Server Version 12.10.FC7 -- On-Line -- Up 231 days
02:25:26 -- 5343628 Kbytes
MT global info:
sessions threads vps lngspins time
55 112 31 38 260806
sched calls thread switches yield 0 yield n yield forever
total: 3209079298 3953281725 271323002 213364796 510946501
per sec: 4345841 12278 4334915 542 4195
Virtual processor summary:
class vps usercpu syscpu total
cpu 10 148357.28 11552.33 159909.61
aio 5 6.29 87.65 93.94
lio 1 0.24 0.51 0.75
pio 1 0.40 0.73 1.13
adm 1 5.10 5.30 10.40
soc 1 1833.92 3178.71 5012.63
msc 1 0.06 0.05 0.11
crypto 10 2555.36 196.35 2751.71
fifo 1 0.24 0.58 0.82
total 31 152758.89 15022.21 167781.10
Individual virtual processors:
vp pid class usercpu syscpu total Thread Eff
1 12026 cpu 30532.56 2848.88 33381.44 61215.75 54%
2 12027 adm 5.10 5.30 10.40 0.00 0%
3 12028 lio 0.24 0.51 0.75 0.00 0%
4 12029 pio 0.40 0.73 1.13 0.00 0%
5 12030 aio 4.85 83.89 88.74 431.46 20%
6 12031 msc 0.06 0.05 0.11 0.61 18%
7 12032 fifo 0.24 0.58 0.82 0.00 0%
8 12033 cpu 23771.06 2078.80 25849.86 50232.95 51%
9 12034 cpu 20413.37 1491.44 21904.81 43167.55 50%
10 12035 cpu 15906.21 1280.29 17186.50 33191.59 51%
11 12036 cpu 12555.32 972.54 13527.86 26927.19 50%
12 12037 cpu 11260.30 719.93 11980.23 22081.00 54%
13 12038 cpu 9577.72 604.43 10182.15 18237.37 55%
14 12039 cpu 8613.38 502.04 9115.42 16041.61 56%
15 12040 cpu 8653.01 506.41 9159.42 17372.45 52%
16 12041 cpu 7074.35 547.57 7621.92 13583.73 56%
17 12042 crypto 2552.29 190.75 2743.04 2743.04 100%
18 12043 crypto 0.32 0.63 0.95 0.00 0%
19 12044 crypto 0.42 0.66 1.08 0.00 0%
20 12045 crypto 0.29 0.58 0.87 0.00 0%
21 12046 crypto 0.33 0.60 0.93 0.00 0%
22 12047 crypto 0.26 0.60 0.86 0.00 0%
23 12048 crypto 0.40 0.70 1.10 0.00 0%
24 12049 crypto 0.39 0.72 1.11 0.00 0%
25 12050 crypto 0.36 0.62 0.98 0.00 0%
26 12051 crypto 0.30 0.49 0.79 0.00 0%
27 12052 soc 1833.92 3178.71 5012.63 NA NA
28 12053 aio 0.35 0.90 1.25 2.21 56%
29 12054 aio 0.34 1.04 1.38 1.38 100%
30 14967 aio 0.36 0.97 1.33 1.33 100%
31 32478 aio 0.39 0.85 1.24 1.24 100%
tot 152758.89 15022.21 167781.10
Always when there is a such high CPU IO Wait I can see lots of KAIO threads in
active threads:
Running threads:
tid tcb rstcb prty status vp-class name
8 4aec4800 0 1 running 27soc* soctcppoll
30 4b31b8b0 0 3 running 1cpu* kaio
55 4b58dd28 0 3 running 9cpu* kaio
68 4b803178 0 3 running 10cpu* kaio
69 4b8034b8 0 3 running 11cpu* kaio
70 4b989608 0 3 running 12cpu* kaio
71 4b8037f8 0 3 running 13cpu* kaio
98 4bb73028 0 3 running 14cpu* kaio
Thank you.
##onstat -c
ROOTNAME rootdbs
ROOTPATH /opt/ibm/informix/storage/chunks/rootdbs
ROOTOFFSET 0
ROOTSIZE 2097152MIRROR 0
MIRRORPATH $INFORMIXDIR/tmp/demo_on.root_mirror
MIRROROFFSET 0
PHYSFILE 2048000
PLOG_OVERFLOW_PATH $INFORMIXDIR/tmp
PHYSBUFF 128
LOGFILES 101
LOGSIZE 300000
DYNAMIC_LOGS 2
LOGBUFF 64
LTXHWM 70
LTXEHWM 80
MSGPATH $INFORMIXDIR/tmp/online.log
CONSOLE $INFORMIXDIR/tmp/online.con
TBLTBLFIRST 0
TBLTBLNEXT 0
TBLSPACE_STATS 1DBSPACETEMP tempdbs1, tempdbs2, tempdbs3
SBSPACETEMP
SBSPACENAME sbspace
SYSSBSPACENAME sbspace
ONDBSPACEDOWN 2
SERVERNUM 2
DBSERVERNAME infx_prod
DBSERVERALIASES infx_db,lo_infx_tseries,ol_informix1210_json_01
FULL_DISK_INIT 0
NETTYPE soctcp,1,64,NET
LISTEN_TIMEOUT 60
MAX_INCOMPLETE_CONNECTIONS 1024
FASTPOLL 1
NUMFDSERVERS 4
NS_CACHE host=900,service=900,user=900,group=900
MULTIPROCESSOR 1
VPCLASS cpu,num=10,aff=(0-9),noage
VP_MEMORY_CACHE_KB 2048
SINGLE_CPU_VP 0
AUTO_TUNE 1
CLEANERS 8
DIRECT_IO 1
LOCKS 200000
DEF_TABLE_LOCKMODE page
RESIDENT 0
SHMBASE 0x44000000L
SHMVIRTSIZE 5120000
SHMADD 51200
EXTSHMADD 51200
SHMTOTAL 0
SHMVIRT_ALLOCSEG 0,3
SHMNOACCESS
CKPTINTVL 300
RTO_SERVER_RESTART 0
BLOCKTIMEOUT 3600
CONVERSION_GUARD 2
RESTORE_POINT_DIR $INFORMIXDIR/tmp
TXTIMEOUT 300
DEADLOCK_TIMEOUT 60
HETERO_COMMIT 0
TAPEDEV /ifxdb/archive
TAPEBLK 32
TAPESIZE 0
LTAPEDEV /ifxdb/bcklog
LTAPEBLK 32
LTAPESIZE 999999
BAR_ACT_LOG $INFORMIXDIR/tmp/bar_act.log
BAR_DEBUG_LOG $INFORMIXDIR/tmp/bar_dbug.log
BAR_DEBUG 0
BAR_MAX_BACKUP 0
BAR_MAX_RESTORE
BAR_RETRY 1
BAR_NB_XPORT_COUNT 20
BAR_XFER_BUF_SIZE 128
RESTARTABLE_RESTORE ON
BAR_PROGRESS_FREQ 0
BAR_BSALIB_PATH /opt/ibm/informix/lib/libbsapsm.so
BACKUP_FILTER /usr/bin/pigz -7c
RESTORE_FILTER /usr/bin/unpigz
BAR_PERFORMANCE 0
BAR_CKPTSEC_TIMEOUT 15
PSM_DBS_POOL DBSPOOL
PSM_LOG_POOL LOGPOOL
DD_HASHSIZE 31
DD_HASHMAX 10
DS_HASHSIZE 31
DS_POOLSIZE 127
PC_HASHSIZE 31
PC_POOLSIZE 127
PRELOAD_DLL_FILE
STMT_CACHE 0
STMT_CACHE_HITS 0
STMT_CACHE_SIZE 512
STMT_CACHE_NOLIMIT 0
STMT_CACHE_NUMPOOL 1
USEOSTIME 0
STACKSIZE 64
ALLOW_NEWLINE 0
USELASTCOMMITTED "NONE"
FILLFACTOR 90
MAX_FILL_DATA_PAGES 0
BTSCANNER num=1,threshold=5000,rangesize=-1,alice=6,compression=default
ONLIDX_MAXMEM 5120
MAX_PDQPRIORITY 100
DS_MAX_QUERIES
DS_TOTAL_MEMORY 250000
DS_MAX_SCANS 1048576
DS_NONPDQ_QUERY_MEM 25000
DATASKIP
OPTCOMPIND 2
DIRECTIVES 1
EXT_DIRECTIVES 0
OPT_GOAL -1
IFX_FOLDVIEW 1
STATCHANGE 10
USTLOW_SAMPLE 1
BATCHEDREAD_TABLE 1
BATCHEDREAD_INDEX 1
EXPLAIN_STAT 1
IFX_EXTEND_ROLE 1
SECURITY_LOCALCONNECTION
UNSECURE_ONSTAT
ADMIN_USER_MODE_WITH_DBSA
ADMIN_MODE_USERS
SSL_KEYSTORE_LABEL
TLS_VERSION
PLCY_POOLSIZE 127
PLCY_HASHSIZE 31
USRC_POOLSIZE 127
USRC_HASHSIZE 31
SQL_LOGICAL_CHAR OFF
SEQ_CACHE_SIZE 10
ENCRYPT_HDR
ENCRYPT_SMX
ENCRYPT_CDR 0
ENCRYPT_CIPHERS
ENCRYPT_MAC
ENCRYPT_MACFILE
ENCRYPT_SWITCH
CDR_EVALTHREADS 1,2
CDR_DSLOCKWAIT 5
CDR_QUEUEMEM 4096
CDR_NIFCOMPRESS 0
CDR_SERIAL 0
CDR_DBSPACE
CDR_QHDR_DBSPACE
CDR_QDATA_SBSPACE
CDR_SUPPRESS_ATSRISWARN
CDR_DELAY_PURGE_DTC 0
CDR_LOG_LAG_ACTION ddrblock
CDR_LOG_STAGING_MAXSIZE 0
CDR_MAX_DYNAMIC_LOGS 0GRIDCOPY_DIR $INFORMIXDIR
CDR_TSINSTANCEID 0
CDR_MAX_FLUSH_SIZE 50
CDR_AUTO_DISCOVER 0
CDR_MEM 0
SHARD_MEM 0
SHARD_ID 0
DRAUTO 0
DRINTERVAL 0
HDR_TXN_SCOPE NEAR_SYNC
DRTIMEOUT 30
HA_ALIAS
HA_FOC_ORDER SDS,HDR,RSSDRLOSTFOUND $INFORMIXDIR/etc/dr.lostfound
DRIDXAUTO 0
LOG_INDEX_BUILDS
SDS_ENABLE
SDS_TIMEOUT 20
SDS_TEMPDBS
SDS_PAGING
SDS_LOGCHECK 10
SDS_ALTERNATE NONE
SDS_FLOW_CONTROL 0
UPDATABLE_SECONDARY 0
FAILOVER_CALLBACK
FAILOVER_TX_TIMEOUT 0
TEMPTAB_NOLOG 0
DELAY_APPLY 0
STOP_APPLY 0
LOG_STAGING_DIR
RSS_FLOW_CONTROL 0
SMX_NUMPIPES 1
ENABLE_SNAPSHOT_COPY 0
SMX_COMPRESS 0
SMX_PING_INTERVAL 10
SMX_PING_RETRY 6
CLUSTER_TXN_SCOPE SERVER
ON_RECVRY_THREADS 1
OFF_RECVRY_THREADS 10
DUMPDIR $INFORMIXDIR/tmp
DUMPSHMEM 1
DUMPGCORE 0
DUMPCORE 0
DUMPCNT 1ALARMPROGRAM $INFORMIXDIR/etc/alarmprogram.sh
ALRM_ALL_EVENTS 0
STORAGE_FULL_ALARM 600,3SYSALARMPROGRAM $INFORMIXDIR/etc/evidence.sh
RAS_PLOG_SPEED 55196
RAS_LLOG_SPEED 975
EILSEQ_COMPAT_MODE 0
QSTATS 0
WSTATS 0
USERMAPPING OFF
SP_AUTOEXPAND 1
SP_THRESHOLD 0
SP_WAITTIME 30
AUTOLOCATE 0
DEFAULTESCCHAR \\\\
MQSERVER
MQCHLLIB
MQCHLTAB
REMOTE_SERVER_CFG hosts.equiv
REMOTE_USERS_CFG hosts.equiv
S6_USE_REMOTE_SERVER_CFG 1
LOW_MEMORY_RESERVE 0
LOW_MEMORY_MGR 0
JVPPROPFILE $INFORMIXDIR/extend/krakatoa/.jvpprops
JVPLOGFILE $INFORMIXDIR/tmp/jvp.log
JVPARGS -Dcom.ibm.tools.attach.enable=no
JVPCLASSPATH
$INFORMIXDIR/extend/krakatoa/krakatoa.jar:$INFORMIXDIR/extend/krakatoa/jdbc.jar
BUFFERPOOL default,buffers=10000,lrus=8,lru_min_dirty=50,lru_max_dirty=60.5
BUFFERPOOL size=2K,buffers=50000,lrus=8,lru_min_dirty=50,lru_max_dirty=60
BUFFERPOOL size=8K,buffers=10000,lrus=8,lru_min_dirty=50,lru_max_dirty=60.5
Hello. Your engine configuration is pretty standard, so you are with a Ferrari, driving on a slow speed lane. You must tune it as soon as possible. Pretty sure you are missing buffers, so I would suggest you to tune then first if all. Several other parameters could bring you a great performance benefit. Hope it helps. Best regards. Alexandre Marini
Arba, are there other applications running on the database server ? it appears that your database buffers are set way too low (80 meg & 100 meg) so you probably are experiencing buffer waits and are turning them over many times per hour. Mark