IDS performance problem
Hi,
Any help will be appreciated!
I'm dealing with an IDS 9.21.FC3 on HPUX 11.11 system, and two weeks ago the system
began to run soooo slowly on some peak hours. Of course my friend is aware that he
needs to upgrade, but that's not a possibility right now.
After 2 days of intense monitoring we saw that IDS is not able to take more than 30%
of the CPU (10 physical CPUS), and there are no disk issues (no more than 3-5% wio
as showed by sar -u).
I truly believe that the system is quite well configured, but...
What I've observed is that this is the system with most latch and buffer waits I've
ever seen, and the ready queue is 20 threads long in bad times, 90 threads long in
worst cases. Yes, we tried up to 20 CPUVPs with no results - no more than 30% real
CPU use. And this afternoon I've repeatedly seen this:
onstat -g wmx
Informix Dynamic Server 2000 Version 9.21.FC3 -- On-Line -- Up 6 days 20:53:27
-- 3967268 Kbytes
Mutexes with waiters:
mid addr name holder lkcnt waiter waittime
4179 c0000000d3e515e8 nsf.lock 178903 0 178727 11
178919 2
178878 17
178897 28
178336 5
178908 2
178887 5
178866 5
178914 6
178913 5
146757 1
178915 5
178673 1
178921 9
178399 11
168900 3
178534 1
178920 2
178901 8
177881 4
178402 3
178868 11
177845 16
178885 36
178882 8
19 0
178879 2
178924 0
46 0
18 0
178865 11
178918 2
4264 c0000000d4655358 hash PC_ID -1 0 178404 0
onstat -g wmx
Informix Dynamic Server 2000 Version 9.21.FC3 -- On-Line -- Up 6 days 21:01:42
-- 4172068 Kbytes
Mutexes with waiters:
mid addr name holder lkcnt waiter waittime
4179 c0000000d3e515e8 nsf.lock 179073 0 179123 10
179167 1
27 1
179166 4
178337 4
179174 3
178970 2
179173 1
179177 4
179081 13
179138 1
179064 2
179150 1
179141 1
179102 5
179114 6
179186 1
154345 1
179172 1
178901 1
179125 3
179176 8
178935 0
178912 8
46 0
179187 0
179183 4
179175 1
28 0
178923 5
47 0
179096 0
179098 0
179182 0
179142 0
4258 c0000000d4655028 hash PC_ID 178432 1 178867 0
178956 0
178387 0
4259 c0000000d46550b0 hash PC_ID 178902 1 179144 0
178698 0
178951 0
4268 c0000000d4655578 hash PC_ID 177879 1 179121 0
178985 0
4269 c0000000d4655600 hash PC_ID 178898 1 177319 0
and this:
onstat -g spi
Spin locks with waits:
Num Waits Num Loops Avg Loop/Wait Name
461 5736195 12442.94 mtcb vproc_sync_lock
1204 22413 18.62 mtcb sleeping_lock
3 10 3.33 mtcb tcb_queue_lock
29099 561272 19.29 mtcb mutex_list_lock
47 1167 24.83 mtcb cond_list_lock
634 30765 48.53 mtcb notify_lock
190 7806 41.08 class cl_lock, class = 0
178839 6820049 38.14 vproc vp_lock, id = 1
180588 7060191 39.10 vproc vp_lock, id = 3
160169 6678508 41.70 vproc vp_lock, id = 4
141997 6210570 43.74 vproc vp_lock, id = 5
117829 5194097 44.08 vproc vp_lock, id = 6
95753 4379701 45.74 vproc vp_lock, id = 7
73398 3546255 48.32 vproc vp_lock, id = 8
69551 3438488 49