Re: MSC virtual-processor class too loaded aft....

Hi, first of all, thank you very much John and Jacques for your answers. The IDS engine allocated another vp msc (I supose due to the high load of the first one). I have tried without success (at the moment) to catch operations that msc vp is doing with the following command: for num in $(seq 1 10000); do onstat -g ioq msc; done |egrep -v "^$|IBM Informix Dynamic Server Version|request list for queue msc" About problems with some network service, I will check now. Thanks a lot, Marc On Tue, Feb 24, 2009 at 6:02 PM, John Miller iii <miller3@us.ibm.com> wrote: > Marc: > > As you might guess the misc vp does many things, but it does > run allot of network authentication calls. I would check things > like NIS, dns,... I am guessing you will find some network > service is not responding timely. > > I see you have allocated a second misc vp. You are executed > about 450 misc vp calls a second. I would try to print out the > operations on the misc vp work queue by using onstat -g ioq msc > These commands are short lived you will have to run it several > times to catch an operation. > > John F. Miller III > STSM, Support Architect > miller3@us.ibm.com > 503-578-5645 > IBM Informix Dynamic Server (IDS) > > ids-bounces@iiug.org wrote on 02/24/2009 04:59:41 AM: > > > Hi, > > > > After migration our DWH from IDS 10FC9 to IDS 11.5FC3 (Under SLES 10 SP2) > I > > found that msc vp class (Miscelaneous System Call) are too loaded, as you > > > can see in the following onstats: > > > > Can anybody tell me the reason, or can point me to the correct > documentation > > to understand this behaviour? > > > > Many thanks, > > > > Marc > > > > informix@XXXX:~> onstat -g glo > > > > IBM Informix Dynamic Server Version 11.50.FC3 -- On-Line -- Up > > 00:54:20 -- 27766080 Kbytes> > > > MT global info: > > sessions threads vps lngspins > > 23 82 26 18 > > > > sched calls thread switches yield 0 yield n yield forever > > total: 29871180 9510392 20382994 94292 3373998 > > per sec: 9524 9517 6 64 294 > > > > Virtual processor summary: > > class vps usercpu syscpu total > > cpu 16 268.73 75.04 343.77 > > aio 2 0.05 0.03 0.08 > > shm 1 0.14 0.22 0.36 > > lio 1 0.00 0.00 0.00 > > pio 1 0.00 0.00 0.00 > > adm 1 0.00 0.00 0.00 > > soc 2 6.88 12.86 19.74 > > msc 2 2633.20 159.34 2792.54 > > total 26 2909.00 247.49 3156.49 > > > > Individual virtual processors: > > vp pid class usercpu syscpu total Thread Eff > > 1 28739 cpu 106.76 32.19 138.95 176.17 78% > > 2 28740 adm 0.00 0.00 0.00 0.00 0% > > 3 28741 cpu 89.20 23.93 113.13 143.93 78% > > 4 28742 cpu 32.07 7.99 40.06 43.20 92% > > 5 28743 cpu 14.70 4.04 18.74 18.74 100% > > 6 28744 cpu 4.61 1.24 5.85 5.85 99% > > 7 28745 cpu 9.69 1.57 11.26 11.26 100% > > 8 28746 cpu 3.94 1.10 5.04 5.20 97% > > 9 28747 cpu 5.18 1.93 7.11 7.11 100% > > 10 28748 cpu 0.27 0.14 0.41 1.27 32% > > 11 28749 cpu 2.22 0.87 3.09 3.09 100% > > 12 28750 cpu 0.06 0.02 0.08 0.32 25% > > 13 28751 cpu 0.01 0.01 0.02 0.03 58% > > 14 28752 cpu 0.02 0.00 0.02 0.03 68% > > 15 28753 cpu 0.00 0.00 0.00 0.01 0% > > 16 28754 cpu 0.00 0.00 0.00 0.01 0% > > 17 28755 cpu 0.00 0.01 0.01 0.01 100% > > 18 28756 lio 0.00 0.00 0.00 0.00 0% > > 19 28757 pio 0.00 0.00 0.00 0.00 0% > > 20 28758 aio 0.05 0.03 0.08 1.47 5% > > 21 28759 msc 2207.70 124.59 2332.29 2332.85 99% > > 22 28760 aio 0.00 0.00 0.00 0.00 0% > > 23 28761 shm 0.14 0.22 0.36 NA NA > > 24 28762 soc 1.84 3.05 4.89 NA NA > > 25 28763 soc 5.04 9.81 14.85 NA NA > > 26 29360 msc 425.50 34.75 460.25 460.25 100% > > > > tot 2909.00 247.49 3156.49 > > > > informix@XXXXX:~> onstat -g iov > > > > IBM Informix Dynamic Server Version 11.50.FC3 -- On-Line -- Up > > 00:54:41 -- 27766080 Kbytes > > > > AIO I/O vps:> > class/vp s io/s totalops dskread dskwrite dskcopy wakeups io/wup > > errors tempops > > kio 0 i 24.2 79484 75367 4117 0 161052 > > 0.5 0 0 > > kio 1 i 26.0 85230 79404 5826 0 173237 > > 0.5 0 0 > > kio 2 i 0.3 865 146 719 0 1047 > > 0.8 0 0 > > kio 3 i 0.6 1862 1546 316 0 3715 > > 0.5 0 0 > > kio 4 i 0.0 2 0 2 0 4 > > 0.5 0 0 > > kio 5 i 0.2 735 315 420 0 1511 > > 0.5 0 0 > > kio 6 i 5.0 16427 15282 1145 0 31824 > > 0.5 0 0 > > kio 7 i 0.4 1285 501 784 0 1842 > > 0.7 0 0 > > kio 8 i 0.2 795 478 317 0 1576 > > 0.5 0 0 > > kio 9 i 0.0 1 1 0 0 2 > > 0.5 0 0 > > kio 10 i 0.0 141 83 58 0 282 > > 0.5 0 0 > > kio 11 i 0.2 620 562 58 0 1305 > > 0.5 0 0 > > msc 0 i 368.1 1207506 0 0 0 1061718 > > 1.1 0 1207506 > > msc 1 i 80.3 263317 0 0 0 255838 > > 1.0 0 263317 > > aio 0 i 2.8 9115 75 8513 0 9116 > > 1.0 0 9 > > aio 1 i 0.0 0 0 0 0 1 > > 0.0 0 0 > > pio 0 i 0.0 0 0 0 0 1 > > 0.0 0 0 > > lio 0 i 0.0 0 0 0 0 1 > > 0.0 0 0 > > > > --000e0cd28ea6a3e9ad0463a9b080 > > > > > > > > > ******************************************************************************* > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --000e0cd250166f24d00463ad65ae

Do you have clients connect to the engine and not do any work, or just disconnect? It just strikes me as very odd that the server has only been up 54 minutes, but 1 of the msc vp has consumed ~2200 seconds of user cpu time, while your cpu vps have done practialy nothing. For that to happen it seems like you would have to have clients connecting to the engine and then just disconnect which would cause the msc vp to go through it's work of authenticating the connection, but then have new connection disconnect without doing anything. On my 11.50 instance 1 new network connection client caused my msc vp to do 5 msc vps ops as shown by onstat -g ioq. I would guess that may somewhat vary based on how your clients have to authenticate. Also for it to be able to do the ~400ish ops per second, it seems like whatever op it is doing wouldn't be very time consuming, but it's getting bombed with a lot of op requests. If your system has a pstack type command that you could run against the pid of the msc vp, you might run that a couple times in a row to get the stack trace for the msc vp, and if it was always the same that might be some sort of indications of what it is doing that might be consuming the user cpu time. I guess it might also be interesting to look at your onstat -g ntd to see if you do have any client type that has a really high number of accepted count which might be the client type that's causing the work the msc vp is doing.

Hi, Thank you Jacques for your answers. Their are very useful for me to grow my knowledge of this great DB Engine (I am a Unix/Linux sysadmin that is begining to learn IDS) At the time that I made the onstat -g iov ... there were one or two users connected to the engine. Fortunately, the msc vp high usage is over. I don't know (at the moment) the reason, but I think that problem is related with the following: After migrating the server (DSS) from 10FC9 to 11.5FC3 we have a performance issue with a process that loads data from our OLTP server ( SLES 10 SP2 IDS 10FC9) to our DSS (DWH) server (SLES 10 SP2 IDS 11.5FC3, which has this msc load problems). This process has a function that makes some selects via synonims against to our OLTP server, after some work we found that the select via synonims was the clue of this poor performance. Then, we stop using this synonim inside the procedure, the performance issue stopped and since this time, our msc vp are not high loaded. I open a PMR on IBM support about this problem. I don't know if I am on the right way, but at the moment I don't have any other clue. With IDS 10FC9 on our DSS server we never had this kind of problem. Tomorrow, I will try to reproduce this problem in our development eviroment. Excused my poor english :) Regards, Marc On Tue, Feb 24, 2009 at 7:45 PM, JACQUES RENAUT <jrenaut@us.ibm.com> wrote: > Do you have clients connect to the engine and not do any work, or just > disconnect? It just strikes me as very odd that the server has only been up > 54 > minutes, but 1 of the msc vp has consumed ~2200 seconds of user cpu time, > while your cpu vps have done practialy nothing. For that to happen it seems > like you would have to have clients connecting to the engine and then just > disconnect which would cause the msc vp to go through it's work of > authenticating the connection, but then have new connection disconnect > without > doing anything. On my 11.50 instance 1 new network connection client caused > my > msc vp to do 5 msc vps ops as shown by onstat -g ioq. I would guess that > may > somewhat vary based on how your clients have to authenticate. > > Also for it to be able to do the ~400ish ops per second, it seems like > whatever op it is doing wouldn't be very time consuming, but it's getting > bombed with a lot of op requests. If your system has a pstack type command > that you could run against the pid of the msc vp, you might run that a > couple > times in a row to get the stack trace for the msc vp, and if it was always > the > same that might be some sort of indications of what it is doing that might > be > consuming the user cpu time. > > I guess it might also be interesting to look at your onstat -g ntd to see > if > you do have any client type that has a really high number of accepted count > which might be the client type that's causing the work the msc vp is doing. > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --000e0cd25cfaf15c2e0463af16eb