DBAmon - Engine monitor
Folks,
Some time ago I wrote a monitor to track engine performance and tell the DBA
of problems. Mike Alder has since made several enhancements to it. This is
an update of the one I posted previously (probably about a year ago). I
asked for and received permission to make this public as payment for all
the help I've received from the net. Please note it makes extensive use of
awk which may not work the same way on other platforms. Also note that if
it blows up your data - that's your problem not mine. No warranties expressed
or implied.
As always - any comments or suggested improvements would be welcome.
Synopsis:
DBAmon - Database engine monitor
Rating: B+
Author: JParker/Mike Alder
Tested under: HP - Online 5.0
Pro: has saved our bacon a number of times.
Con: is a resource hog, it dumps a memory image (later cleaned up) which can
be rather large.
DBAmon reads a file (online.engines) to determine what engines are running
on a platform and then performs a tbstat against each engine and performs a
number of checks to ensure:
a) that the engine is running
b) that the IStar deamon is running (configurable)
c) checks installation permissions (uses ckfls)
d) check performance against user-defined thresholds
e) check logfiles for errors
f) check chunk status != PD or MD
g) check log files backup (only applicable when performed to disk)
h) check for resource hogs or DOA users
i) is a tbtape -c running (configurable)
j) check for free system memory
k) check for disabled page cleaners
l) check for free dbspace
m) check disk fragmentation
n) check for logfile errors
o) check Online reserved pages
Results are mailed to the $DBA.
--- cut here --------------------------------------------------------------
# This is a shell archive. Remove anything before this line,
# then unpack it by saving it in a file and typing "sh file".
#
# Wrapped by /Informix <informix@hpbs2651> on Fri Mar 4 18:05:39 1994
#
# This archive contains:
# DBAmon online.engines ckfls
#
LANG=""; export LANG
PATH=/bin:/usr/bin:$PATH; export PATH
echo x - DBAmon
cat >DBAmon <<'@EOF'
#!/bin/ksh
############################################################################
#
# Shell to monitor DB Engine performance.
#
# Jack Parker, HP Boise Surface Mount Center 12/92
#
# Appended,Modified and Maint - Contact Mike Alder Ext. 5245
#
# Maintained some more JParker 2/26/94
# (learned a bit since '92) - shortened somewhat, but didn't go
# overboard.
#
############################################################################
ENGINE_FILE="/users/informix/online.engines"
# see enclosed sample online.engines.
PAGE_SIZE=4098 # this can be found using monitor->Virtual mem
# lower right hand corner
OUT_FILE=/tmp/DBAmon.tmp # where to write core image of engine
# (can be big)
DBA=address@location # who to tell of problems
alert_cmd=mailx # how to tell them.
############################################################################
SYSTEM=`uname -n`
# Monitor thresholds - tell DBA after this level is passed.
# note these are PER DAY figures.
############################################################################
set_var()
{
o_buff_th=50 # Over buffer
o_user_th=10 # Over users
o_lock_th=10 # Over locks
o_tbls_th=10 # Over Tables (too many requests for tblspace)
o_buf_wt_th=20 # buffer waits
o_lck_wt_th=10 # lock waits
o_doa_lck_th=10 # Deadlocks
o_dltout_th=10 # Deadlock timeouts
o_lchwait_th=20 # latch waits
o_ckp_wt_th=20 # checkpoint waits
# note these are NOT per day figures.
###########################################################################
rcache_th=95 # read cache %
wcache_th=82 # write cache %
pcnt_th=20 # disk full at 20 percent free
lgf_th=66 # log files are full - 66 %
b_m_th=80 # buffers are too small
b_l_th=70 # buffers are too big
lg_full_th=75 # Logical log device is full
mfr_th=10000 # how much memory should be free in "kbytes"
lst_arch_th=1 # how many days ago should archive have been?
}
############################################################################
# tbtape -c running?
# doesn't always give right numbers
# but only prints when there is a problem
# we don't use tbtape -c, so little opporunity
# to test.
chk_tbtape()
{
no_tps=`grep -v "^#" $ENGINE_FILE |cut -d":" -f 4|grep -c y`
norun=`ps -e | grep -c "tbtape -c"`
if [ -z "$norun" ]
then norun=0
fi
if [ -z "$no_tps" ]
then no_tps=0
fi
let no_tps="$no_tps * 2"
if [ $no_tps -ne $norun ]
then
echo "\\nOnly $norun of the $no_tps continuous backups are running!"
fi
}
###############################################################################
# get list of engines from system file
# format is: (user maintained)
#Engine_Name:config_file:server_no:tbtape-c?:arch_level:grps
get_engines()
{
engines=`grep -v "^#" $ENGINE_FILE |cut -d":" -f 1`
}
###############################################################################
# Check to see if engine is on line.
# how many days has engine been up?
# reset thresholds for that number
# watch out for bogus values...
set_thrs()
{
up=`tbstat -o /dev/null | grep -i "on-line" | awk '{print $8}'`
if [ -z "$up" ]
then
echo " $engine is OFF line - no further stats will be taken"
RETURN_FLAG=1
return
fi
dys_up=`echo $up | grep -c ":"`
if [ -z "$dys_up" ]
then
echo variable dys_up was not set properly
return
fi
if [ $dys_up -ne 0 ]
then
up=1
fi
if [ $up = "--" ]
then
up=1
fi
let buff_th=o_buff_th*$up # Over buffer
let user_th=o_user_th*$up # Over users
let lock_th=o_lock_th*$up # Over locks
let tbls_th=o_tbls_th*$up # Over Tables (too many rqst for tblspc)
let buf_wt_th=o_buf_wt_th*$up # buffer waits
let lck_wt_th=o_lck_wt_th*$up # lock waits
let doa_lck_th=o_doa_lck_th*$up # Deadlocks
let dltout_th=o_dltout_th*$up # Deadlock timeouts
let lchwait_th=o_lchwait_th*$up # latch waits
let ckp_wt_th=o_ckp_wt_th*$up # checkpoint waits
}
###############################################################################
# check permissions on distribution files - only run on Sundays
###############################################################################
chk_perm()
{
# Verify correct installation permissions on all files at low peak
ck_day=`date +%a`
if [ -z "$ck_day" ]
then
echo "ck_day is not set "
return
fi
if [ $ck_day = "Sun" ]
then
/users/informix/bin/ckfls onlinefiles >> /tmp/DBA.warning
/users/informix/bin/ckfls starfiles >> /tmp/DBA.warning@@