Why are my boxes sooo slow sometimes? (w/ detailed 'Monitor' output)

ross@charlie.ece.sc.edu Richard Ross writes: I've got a few RISC 6k (520) running AIX 3.2.5 networked together. At night at ~7pm things slow to a crawl. I get NFS server 'image' not responding on a machine w/ a few of 'images' drives mounted (tabletop). Also 'ls' seems to work fine but when I use 'ls -al' it take several minutes to come back.(what is the difference?). This happens on 'tabletop' (the main box) and 'image' (legacy names). Pings from image to tabletop comes back in 1ms no problem. What could make this happen like clockwork when there is no big cron job occuring? An informix backup is running but has been going for several hours w/ no noticable drag on the system and when it finishes I still have the problem. What else should I be looking at? Below is detailed output from the 'Monitor' utility (caution: use of wide margins important below) Using 'monitor' (downloaded software) I find: 1 Users: 1 of 11 active 11 remote 502:02 sleep time CPU: Sys 7.7% Wait 18.0% User 1.5% Idle 72.8% Refresh: 10.00 s 0% 25% 50% 75% 100% =====WWWWWWWWWWWW> Runnable (Swap-in) processes 0.10 (0.00) load average: 0.28, 0.22, 0.26 Memory Real Virtual Paging (4kB) Process events File/TTY-IO free 161 MB 360 MB 0.1 pgfaults 187 pswitch 3 iget files 126 MB 0.1 pgin 143 syscall 1 namei total 384 MB 480 MB 1.5 pgout 38 read 0 dirblk IO (kB/s) read write busy% 0.1 pgsin 34 write 480815 readch hdisk0 0.0 0.4 0 0.0 pgsout 0 fork 479077 writech hdisk1 0.0 0.4 0 0 exec 0 ttyrawch hdisk2 0.0 0.0 0 0 rcvint 0 ttycanch hdisk3 467.2 0.0 18 0 xmtint 26 ttyoutch hdisk4 0.0 5.2 3 0 mdmint hdisk5 0.0 0.0 0 hdisk6 0.4 0.0 0 Netw read write kB/s lo0 0.0 0.0 sl1 0.0 0.0 en0 0.0 0.0 en1 0.4 0.9 en2 0.0 0.0 sl17 0.0 0.0 My top ten processes are: root 514 46.2 0.0 12 8 - R Oct 09 5237:53 kproc lisah 161607 7.3 1.0 420 1736 pts/29 S 19:43:47 3:14 sqlturbo lisah 4 informix 256226 5.2 0.0 252 536 - R 17:25:11 9:34 /u/informix/bin/ lisah 224582 1.5 1.0 440 1264 pts/29 S 19:43:47 0:39 newclaim.4ge root 21759 1.5 0.0 1652 12 - S Oct 09 171:40 nsrmmd -n 1 root 20655 0.8 0.0 168 64 - S Oct 09 89:14 /usr/etc/rpc.loc root 0 0.5 0.0 8 8 - S Oct 09 57:47 swapper root 276692 0.5 0.0 204 280 - S 19:10:31 0:21 telnetd root 771 0.4 0.0 16 16 - S Oct 09 48:43 kproc root 3251 0.4 0.0 1672 1104 - S Oct 09 46:26 /usr/bin/nsrd number of sqlturbos: 12 The informix job is the backup. hdisk3 access goes to zero when the backup is finished but the slowness persists. Any ideas or direction very appreciated! Richard Ross ross@charlie.ece.sc.edu