Informix use of CPU's question

I'm curious why I see such uneven distribution of CPU time across the AIX PID's that are used by Informix. This is an eight processor RS/6000 and I would have thought that with 150+ sql users beating against it that each PID would have handled a little more of the workload, if nothing else, doing KAIO. The onconfig file has NUMCPUVPS set to 8. Any comments? I'll submit more info if required. By chance is there an AIX parameter that limits the number of CPU's. ================== monitor output ===================== court1 # ps -ef | grep oninit root 11356 14454 0 Sep 19 - 0:26 /opt/informix/bin/oninit root 14454 1 1 Sep 19 - 167:52 /opt/informix/bin/oninit root 16258 14454 8 Sep 19 - 112:26 /opt/informix/bin/oninit root 16770 14454 1 Sep 19 - 49:24 /opt/informix/bin/oninit root 17028 14454 0 Sep 19 - 18:27 /opt/informix/bin/oninit root 17286 14454 0 Sep 19 - 8:40 /opt/informix/bin/oninit root 17544 14454 0 Sep 19 - 3:13 /opt/informix/bin/oninit root 17802 14454 0 Sep 19 - 1:18 /opt/informix/bin/oninit root 18060 14454 0 Sep 19 - 0:52 /opt/informix/bin/oninit root 18318 16258 0 Sep 19 - 0:03 /opt/informix/bin/oninit root 18576 14454 0 Sep 19 - 0:03 /opt/informix/bin/oninit root 18834 14454 0 Sep 19 - 0:03 /opt/informix/bin/oninit root 19092 14454 0 Sep 19 - 0:11 /opt/informix/bin/oninit root 19350 14454 0 Sep 19 - 0:03 /opt/informix/bin/oninit root 19608 14454 0 Sep 19 - 0:55 /opt/informix/bin/oninit informix 20046 11162 1 07:50:59 pts/1 0:00 grep oninit onstat -g sch VP Scheduler Statistics: vp pid class semops busy waits spins/wait 1 14454 cpu 64 106 991 2 11356 adm 0 0 0 3 16258 cpu 93052 310318 1000 4 16770 cpu 215368 402425 1000 5 17028 cpu 349113 714393 1000 6 17286 cpu 189857 376467 999 7 17544 cpu 52466 151072 999 8 17802 cpu 16531 79294 998 9 18060 cpu 8742 65209 998 10 18318 lio 3 0 0 11 18576 pio 2 0 0 12 18834 aio 73 0 0 13 19092 msc 3129 0 0 14 19350 aio 19 0 0 15 19608 shm 2 2 1000 court1 # onstat -g iov AIO I/O vps: class/vp s io/s totalops dskread dskwrite dskcopy wakeups io/wup errors kio 0 i 0.0 0 0 0 0 0 0.0 0 kio 1 i 3.0 461802 443420 18382 0 947347 0.5 0 kio 2 i 0.0 0 0 0 0 0 0.0 0 kio 3 i 7.2 1118012 1101912 16100 0 2317228 0.5 0 kio 4 i 3.3 513626 496202 17424 0 1115425 0.5 0 kio 5 i 1.9 293857 277264 16593 0 617253 0.5 0 kio 6 i 0.4 58240 42787 15453 0 140670 0.4 0 kio 7 i 0.9 133960 119942 14018 0 299521 0.4 0 kio 8 i 0.2 23873 9130 14743 0 73703 0.3 0 kio 9 i 0.1 21405 7232 14173 0 65664 0.3 0 msc 0 i 0.0 3132 0 0 0 3133 1.0 0 aio 0 i 0.0 88 8 16 0 71 1.2 0 aio 1 i 0.0 25 3 12 0 18 1.4 0 pio 0 i 0.0 0 0 0 0 1 0.0 0 lio 0 i 0.0 1 0 1 0 2 0.5 0

OK, here is what I see. You have a shared memory NET VP and no TLI NET VPs so I assume that your onconfig (PLEASE post ONCONFIG files for ALL performance related questions) has the following two lines: NETTYPE ipcshm,1,150,NET NETTYPE tlitcp,1,150,CPU or something VERY similar. This is a main cause of: o CPU VP #1 having so little CPU time used. o All other CPU VPs skewed with unequal workload. If you place shared memory listeners in NET VPs they have to poll shared memory constantly which uses resources or has to sleep which slows response time. If you place TCP listeners in CPU VP, the CPU VP cannot very well block on a listen() system call, because it has to be able to do work for existing sessions/threads when there are no pending connection requests. So it also has to poll which takes it away from processing user requests so the other CPU VPs without listeners tend to take up the slack getting most of the work. Since there is only one TCP listener it cannot take the incomming work on itself because it has to check for more pending requests so it passes them off to the first CPU VP which is not busy which is one of the next few, the others just stand at the back of the line jumping up and down going: "Use me! Use me!". A better configuration would be: NETTYPE ipcshm,8,100,CPU # ALL CPU VPs listen for shared memory NETTYPE tlitcp,4,100,NET # 2-4 NET VPS listen for TCP conn. is good In this way ALL of the CPU VPS will be listening on shared memory connections. Since shared memory has to be polled and the CPU VPs have to poll to be able to do work in between, this is a good match and each CPU VP will get a more equal share of the work load. A few NET VPs should be able to handle the incoming connection requests from the network and since the NET VPs can block on a TCP listen() call they will not be using any resources when no requests are pending. There will still be some imbalance but you will notice a distinct improvement both in the stats and in user response time. Art S. Kagel FProse wrote: > > I'm curious why I see such uneven distribution of CPU time across the > AIX PID's that are used by Informix. > > This is an eight processor RS/6000 and I would have thought that with > 150+ sql users beating against it that each PID would have handled a > little more of the workload, if nothing else, doing KAIO. > The onconfig file has NUMCPUVPS set to 8. > > Any comments? I'll submit more info if required. > > By chance is there an AIX parameter that limits the number of CPU's. > > ================== monitor output ===================== > > court1 # ps -ef | grep oninit > root 11356 14454 0 Sep 19 - 0:26 > /opt/informix/bin/oninit > root 14454 1 1 Sep 19 - 167:52 > /opt/informix/bin/oninit > root 16258 14454 8 Sep 19 - 112:26 > /opt/informix/bin/oninit > root 16770 14454 1 Sep 19 - 49:24 > /opt/informix/bin/oninit > root 17028 14454 0 Sep 19 - 18:27 > /opt/informix/bin/oninit > root 17286 14454 0 Sep 19 - 8:40 > /opt/informix/bin/oninit > root 17544 14454 0 Sep 19 - 3:13 > /opt/informix/bin/oninit > root 17802 14454 0 Sep 19 - 1:18 > /opt/informix/bin/oninit > root 18060 14454 0 Sep 19 - 0:52 > /opt/informix/bin/oninit > root 18318 16258 0 Sep 19 - 0:03 > /opt/informix/bin/oninit > root 18576 14454 0 Sep 19 - 0:03 > /opt/informix/bin/oninit > root 18834 14454 0 Sep 19 - 0:03 > /opt/informix/bin/oninit > root 19092 14454 0 Sep 19 - 0:11 > /opt/informix/bin/oninit > root 19350 14454 0 Sep 19 - 0:03 > /opt/informix/bin/oninit > root 19608 14454 0 Sep 19 - 0:55 > /opt/informix/bin/oninit > informix 20046 11162 1 07:50:59 pts/1 0:00 grep oninit > > onstat -g sch> > VP Scheduler Statistics: > vp pid class semops busy waits spins/wait > 1 14454 cpu 64 106 991 > 2 11356 adm 0 0 0 > 3 16258 cpu 93052 310318 1000 > 4 16770 cpu 215368 402425 1000 > 5 17028 cpu 349113 714393 1000 > 6 17286 cpu 189857 376467 999 > 7 17544 cpu 52466 151072 999 > 8 17802 cpu 16531 79294 998 > 9 18060 cpu 8742 65209 998 > 10 18318 lio 3 0 0 > 11 18576 pio 2 0 0 > 12 18834 aio 73 0 0 > 13 19092 msc 3129 0 0 > 14 19350 aio 19 0 0 > 15 19608 shm 2 2 1000 > > court1 # onstat -g iov > > AIO I/O vps: > class/vp s io/s totalops dskread dskwrite dskcopy wakeups io/wup > errors > kio 0 i 0.0 0 0 0 0 0 0.0 > 0 > kio 1 i 3.0 461802 443420 18382 0 947347 0.5 > 0 > kio 2 i 0.0 0 0 0 0 0 0.0 > 0 > kio 3 i 7.2 1118012 1101912 16100 0 2317228 0.5 > 0 > kio 4 i 3.3 513626 496202 17424 0 1115425 0.5 > 0 > kio 5 i 1.9 293857 277264 16593 0 617253 0.5 > 0 > kio 6 i 0.4 58240 42787 15453 0 140670 0.4 > 0 > kio 7 i 0.9 133960 119942 14018 0 299521 0.4 > 0 > kio 8 i 0.2 23873 9130 14743 0 73703 0.3 > 0 > kio 9 i 0.1 21405 7232 14173 0 65664 0.3 > 0 > msc 0 i 0.0 3132 0 0 0 3133 1.0 > 0 > aio 0 i 0.0 88 8 16 0 71 1.2 > 0 > aio 1 i 0.0 25 3 12 0 18 1.4 > 0 > pio 0 i 0.0 0 0 0 0 1 0.0 > 0 > lio 0 i 0.0 1 0 1 0 2 0.5 > 0