Re: Why are my boxes sooo slow sometimes? (w/ detailed 'Monitor' output)
In article <546jgf$icp@hawaii.conterra.com>, SRC <src@conterra.com> writes >ross@charlie.ece.sc.edu >Richard Ross writes: > > I've got a few RISC 6k (520) running AIX 3.2.5 networked >together. At night at ~7pm things slow to a crawl. I get >NFS server 'image' not responding on a machine w/ a few of >'images' drives mounted (tabletop). Also 'ls' seems >to work fine but when I use 'ls -al' it take several minutes >to come back.(what is the difference?). This happens on 'tabletop' >(the main box) and 'image' (legacy names). > Pings from image to tabletop comes back in 1ms no problem. > What could make this happen like clockwork when there is >no big cron job occuring? An informix backup is running but >has been going for several hours w/ no noticable drag on the >system and when it finishes I still have the problem. > What else should I be looking at? Sounds like a network problem to me try running netstat -i on both machines and look for Ierrs/Oerrs columns. Any number a few percent indicates a network problem. >Below is detailed output from the 'Monitor' utility > >(caution: use of wide margins important below) > >Using 'monitor' (downloaded software) I find: > >1 Users: 1 of 11 active 11 remote 502:02 sleep time >CPU: Sys 7.7% Wait 18.0% User 1.5% Idle 72.8% Refresh: 10.00 s >0% 25% 50% 75% 100% >=====WWWWWWWWWWWW> > It's not lack of CPU power >Runnable (Swap-in) processes 0.10 (0.00) load average: 0.28, 0.22, 0.26 > >Memory Real Virtual Paging (4kB) Process events File/TTY-IO >free 161 MB 360 MB 0.1 pgfaults 187 pswitch 3 iget >files 126 MB 0.1 pgin 143 syscall 1 namei >total 384 MB 480 MB 1.5 pgout 38 read 0 dirblk >IO (kB/s) read write busy% 0.1 pgsin 34 write 480815 readch >hdisk0 0.0 0.4 0 0.0 pgsout 0 fork 479077 writech It's not paging/lack of memory >hdisk1 0.0 0.4 0 0 exec 0 ttyrawch >hdisk2 0.0 0.0 0 0 rcvint 0 ttycanch >hdisk3 467.2 0.0 18 0 xmtint 26 ttyoutch Possibly disk I/O is the NFS mount point on hdisk3? >hdisk4 0.0 5.2 3 0 mdmint >hdisk5 0.0 0.0 >hdisk6 0.4 0.0 0 Netw read write kB/s > lo0 0.0 0.0 > sl1 0.0 0.0 > en0 0.0 0.0 > en1 0.4 0.9 > en2 0.0 0.0 > sl17 0.0 0.0 It's not network overload! >My top ten processes are: >root 514 46.2 0.0 12 8 - R Oct 09 5237:53 kproc What is kproc? >lisah 161607 7.3 1.0 420 1736 pts/29 S 19:43:47 3:14 sqlturbo lisah 4 This could be worrying - possibly this sqlturbo is scanning a database table on hdisk3 - I'd check what the user is doing and get the developer of the associated .4ge to 'set explain on' and produce a trace of the sql being executed. Get the developer to check no database indicies are missing. >informix 256226 5.2 0.0 252 536 - R 17:25:11 9:34 /u/informix/bin/ >lisah 224582 1.5 1.0 440 1264 pts/29 S 19:43:47 0:39 newclaim.4ge >root 21759 1.5 0.0 1652 12 - S Oct 09 171:40 nsrmmd -n 1 What is nsrmmd? >root 20655 0.8 0.0 168 64 - S Oct 09 89:14 /usr/etc/rpc.loc >root 0 0.5 0.0 8 8 - S Oct 09 57:47 swapper >root 276692 0.5 0.0 204 280 - S 19:10:31 0:21 telnetd >root 771 0.4 0.0 16 16 - S Oct 09 48:43 kproc >root 3251 0.4 0.0 1672 1104 - S Oct 09 46:26 /usr/bin/nsrd >number of sqlturbos: 12 > >The informix job is the backup. hdisk3 access goes to zero when the backup >is finished but the slowness persists. > Any ideas or direction very appreciated! >Richard Ross >ross@charlie.ece.sc.edu > E-mail me if this doesn't help - like Stefan I'm very interested in performance tuning. -- David Williams