Seeing one oninit working hard while others are no

I think we are running into a condition where only one virtual processor is trying to do a majority of the work on a remote server via synonyms. When you look at all the oninit modules running on the remote server, it looks like only one is really taxed: ps -ef |grep oninit hds 303146 1007698 0 11:27:08 pts/0 0:00 grep oninit informix 332022 1327236 3 03:11:49 - 4:37 oninit -s informix 339984 1327236 0 03:11:50 - 0:01 oninit -s informix 573444 1327236 0 03:11:53 - 0:00 oninit -s informix 663766 1327236 0 03:11:54 - 0:57 oninit -s informix 729146 1327236 0 03:11:55 - 0:00 oninit -s informix 938176 1327236 0 03:11:51 - 0:15 oninit -s informix 962686 1327236 84 03:11:52 - 164:32 oninit -s informix 1032288 1327236 0 03:11:54 - 0:27 oninit -s informix 1101972 1327236 0 03:11:49 - 0:01 oninit -s informix 1327236 1499276 0 03:11:49 - 0:01 oninit -s informix 1380472 1327236 0 03:11:53 - 0:00 oninit -s informix 1388604 1327236 0 03:11:54 - 0:23 oninit -s informix 1400916 1327236 0 03:11:53 - 0:02 oninit -s informix 1454178 1327236 0 03:11:54 - 0:21 oninit -s informix 1499276 1 4 03:11:48 - 8:35 oninit -s from 11.5 Running Poll Threads on CPU or Network Virtual Processors Poll threads can run either on CPU virtual processors or on network virtual processors. In general, and particularly on a single-processor computer, poll threads run more efficiently on CPU virtual processors. This might not be true, however, on a multiprocessor computer with a large number of remote clients. The NETTYPE parameter has an optional entry, called vp class, that allows you to specify either CPU or NET, for CPU or network virtual-processor classes, respectively. If you do not specify a virtual processor class for the interface/protocol combination (poll threads) associated with the DBSERVERNAME variable, the class defaults to CPU. The database server assumes that the interface/protocol combination associated with DBSERVERNAME is the primary interface/protocol combination and that it should be the most efficient. For other interface/protocol combinations, if no vp class is specified, the default is NET. While the database server is in online mode, you cannot drop a CPU virtual processor that is running a poll thread. Note: You should carefully distinguish between poll threads for network connections and poll threads for shared memory connections, which should run one per CPU virtual processor. TCP connections should only be in network virtual processors, and you should only have the minimum needed to maintain responsiveness. Shared memory connections should only be in CPU virtual processors and should run in every CPU virtual processor. I revise my suggestion for crusher (goal is to handle most connections possible) How many CPUs are assigned to Informix on Crusher? VPCLASS cpu,num=2,noage so I'll say 2 physical 4 virtual? With that in mind: NETTYPE ipcshm,4,50,CPU <- have shared memory polling on every virtual cpu NETTYPE soctcp,3,350,NET <- LISTEN_TIMEOUT 60 MAX_INCOMPLETE_CONNECTIONS 1024 FASTPOLL 1 Excerpt from Informix group: Also I notice that your shared memory NETTYPE only specifies one listener poll thread, shared memory listeners should always be run in ALL CPU VPs for best responsiveness and to keep CPU VP#1 (which handles the first poll thread) from running flat out and taking 90% of the work for itself. What do you think? This is IDS 11.50.UC3 remote server and 11.50.FC3 database server on OS AIX 6.1. The remote server has synonyms pointing to the database server database tables. We are using COOKED files also. Thanks, ******************************************************************* Ernie Knox IT Database Administrator Specialist Sears Holdings 3333 Beverly Rd., B4-266A Hoffman Estates, IL. 60179 Office: (847) 286-5735 Email: Ernest.Knox@searshc.com Blackberry: 2244650553@messaging.sprintpcs.com <mailto:2244650553@messaging.sprintpcs.com> Page via Skytel: 2244650553@sprint.skytel.com <mailto:2244650553@sprint.skytel.com> Informix or MySQL Primary: 9110210@skytel.com <mailto:9110210@skytel.com> Informix or MySQL Secondary: 7276872@skytel.com <mailto:7276872@skytel.com> " Yes we can make a Change! " " It's always a great day to watch Sports - GO LIONS, TIGERS, and BEARS! " " Lets not forget - GO Pistons and Red Wings! " GSU *******************************************************************

It'd make more sense to look at onstat -g glo than ps -ef. That way w= e can see the function of each of the oninit processes to determine if anything is really out of line. M.P. = From: "Knox, Ernest" <Ernest.Knox@searshc.com> = = To: ids@iiug.org = = Date: 10/08/2009 02:57 PM = = Subject: Seeing one oninit working hard while others ar.... [1744= 5] = Sent by: ids-bounces@iiug.org = = I think we are running into a condition where only one virtual processo= r is trying to do a majority of the work on a remote server via synonyms.= When you look at all the oninit modules running on the remote server, i= t looks like only one is really taxed: ps -ef |grep oninit hds 303146 1007698 0 11:27:08 pts/0 0:00 grep oninit informix 332022 1327236 3 03:11:49 - 4:37 oninit -s informix 339984 1327236 0 03:11:50 - 0:01 oninit -s informix 573444 1327236 0 03:11:53 - 0:00 oninit -s informix 663766 1327236 0 03:11:54 - 0:57 oninit -s informix 729146 1327236 0 03:11:55 - 0:00 oninit -s informix 938176 1327236 0 03:11:51 - 0:15 oninit -s informix 962686 1327236 84 03:11:52 - 164:32 oninit -s informix 1032288 1327236 0 03:11:54 - 0:27 oninit -s informix 1101972 1327236 0 03:11:49 - 0:01 oninit -s informix 1327236 1499276 0 03:11:49 - 0:01 oninit -s informix 1380472 1327236 0 03:11:53 - 0:00 oninit -s informix 1388604 1327236 0 03:11:54 - 0:23 oninit -s informix 1400916 1327236 0 03:11:53 - 0:02 oninit -s informix 1454178 1327236 0 03:11:54 - 0:21 oninit -s informix 1499276 1 4 03:11:48 - 8:35 oninit -s from 11.5 Running Poll Threads on CPU or Network Virtual Processors Poll threads can run either on CPU virtual processors or on network virtual processors. In general, and particularly on a single-processor computer, poll threads run more efficiently on CPU virtual processors. This might not be true, however, on a multiprocessor computer with a large number of remote clients. The NETTYPE parameter has an optional entry, called vp class, that allows you to specify either CPU or NET, for CPU or network virtual-processor classes, respectively. If you do not specify a virtual processor class for the interface/protocol combination (poll threads) associated with the DBSERVERNAME variable, the class defaults to CPU. The database server assumes that the interface/protocol combination associated with DBSERVERNAME is the primary interface/protocol combination and that it should be the most efficient. For other interface/protocol combinations, if no vp class is specified,= the default is NET. While the database server is in online mode, you cannot drop a CPU virtual processor that is running a poll thread. Note: You should carefully distinguish between poll threads for network= connections and poll threads for shared memory connections, which shoul= d run one per CPU virtual processor. TCP connections should only be in network virtual processors, and you should only have the minimum needed= to maintain responsiveness. Shared memory connections should only be in= CPU virtual processors and should run in every CPU virtual processor. I revise my suggestion for crusher (goal is to handle most connections possible) How many CPUs are assigned to Informix on Crusher? VPCLASS cpu,num=3D2,noage so I'll say 2 physical 4 virtual? With that in mind: NETTYPE ipcshm,4,50,CPU <- have shared memory polling on every virtual cpu NETTYPE soctcp,3,350,NET <- LISTEN_TIMEOUT 60 MAX_INCOMPLETE_CONNECTIONS 1024 FASTPOLL 1 Excerpt from Informix group: Also I notice that your shared memory NETTYPE only specifies one listener poll thread, shared memory listeners should always be run in ALL CPU VPs for= best responsiveness and to keep CPU VP#1 (which handles the first poll thread) from running flat out and taking 90% of the work for itself. What do you think? This is IDS 11.50.UC3 remote server and 11.50.FC3 database server on OS= AIX 6.1. The remote server has synonyms pointing to the database server database= tables. We are using COOKED files also. Thanks, ******************************************************************* Ernie Knox IT Database Administrator Specialist Sears Holdings 3333 Beverly Rd., B4-266A Hoffman Estates, IL. 60179 Office: (847) 286-5735 Email: Ernest.Knox@searshc.com Blackberry: 2244650553@messaging.sprintpcs.com <mailto:2244650553@messaging.sprintpcs.com> Page via Skytel: 2244650553@sprint.skytel.com <mailto:2244650553@sprint.skytel.com> Informix or MySQL Primary: 9110210@skytel.com <mailto:9110210@skytel.com> Informix or MySQL Secondary: 7276872@skytel.com <mailto:7276872@skytel.com> " Yes we can make a Change! " " It's always a great day to watch Sports - GO LIONS, TIGERS, and BEARS= ! " " Lets not forget - GO Pistons and Red Wings! " GSU ******************************************************************* ***********************************************************************= ******** Forum Note: Use "Reply" to post a response in the discussion forum. =

Database server: Database:/home/informix>onstat -g glo IBM Informix Dynamic Server Version 11.50.FC3 -- On-Line -- Up 4 days 09:51:55 -- 6104464 Kbytes MT global info: sessions threads vps lngspins 692 850 22 117863 sched calls thread switches yield 0 yield n yield forever total: 2489299885 572251415 1929513329 50426116 88396984 per sec: 4847 2841 2026 158 358 Virtual processor summary: class vps usercpu syscpu total cpu 4 126472.60 3243.03 129715.63 aio 9 1221.85 2199.69 3421.54 shm 1 6.02 7.43 13.45 lio 1 109.82 493.09 602.91 pio 1 3.77 34.09 37.86 adm 1 38.85 14.77 53.62 soc 4 974.19 2377.86 3352.05 msc 1 4365.69 398.30 4763.99 total 22 133192.79 8768.26 141961.05 Individual virtual processors: vp pid class usercpu syscpu total Thread Eff 1 581758 cpu 37824.67 1386.72 39211.39 39211.39 100% 2 491564 adm 38.85 14.77 53.62 0.00 0% 3 749710 cpu 52742.08 1066.24 53808.32 54099.88 99% 4 1110090 cpu 13775.60 515.50 14291.10 86223.55 16% 5 1601776 cpu 22130.25 274.57 22404.82 23063.49 97% 6 1495118 lio 109.82 493.09 602.91 35616.46 1% 7 1675312 pio 3.77 34.09 37.86 1556.62 2% 8 602174 aio 357.42 1066.91 1424.33 53515.96 2% 9 1171642 msc 4365.69 398.30 4763.99 4887.14 97% 10 733362 aio 194.82 664.36 859.18 47093.70 1% 11 905420 aio 114.99 118.42 233.41 44258.21 0% 12 901344 aio 108.95 87.28 196.23 43906.64 0% 13 1450166 aio 104.93 75.29 180.22 43706.33 0% 14 1102038 aio 105.22 70.60 175.82 43414.00 0% 15 1388554 shm 6.02 7.43 13.45 NA NA 16 946322 soc 211.77 490.53 702.30 NA NA 17 540754 soc 269.73 669.01 938.74 NA NA 18 1654948 soc 264.59 657.44 922.03 NA NA 19 1667226 soc 228.10 560.88 788.98 NA NA 20 508154 aio 99.85 56.50 156.35 36702.92 0% 21 1650716 aio 82.14 38.90 121.04 26227.22 0% 22 1224704 aio 53.53 21.43 74.96 15691.52 0% tot 133192.79 8768.26 141961.05 Remote Server: Remote:/usr/informix>onstat -g glo IBM Informix Dynamic Server Version 11.50.UC3 -- On-Line -- Up 03:06:40 -- 1242192 Kbytes MT global info: sessions threads vps lngspins 116 185 16 2911 sched calls thread switches yield 0 yield n yield forever total: 36843457 12620339 24225482 558900 2206927 per sec: 1196 488 708 52 147 Virtual processor summary: class vps usercpu syscpu total cpu 3 427.30 210.58 637.88 aio 4 5.86 12.28 18.14 shm 1 0.03 0.07 0.10 lio 1 0.36 0.83 1.19 pio 1 0.35 0.84 1.19 adm 1 0.41 0.28 0.69 soc 4 24.11 60.51 84.62 msc 1 7832.11 280.25 8112.36 total 16 8290.53 565.64 8856.17 Individual virtual processors: vp pid class usercpu syscpu total Thread Eff 1 663652 cpu 239.61 123.81 363.42 363.42 100% 2 1380526 adm 0.41 0.28 0.69 0.00 0% 3 938224 cpu 168.60 79.53 248.13 283.67 87% 4 1478902 lio 0.36 0.83 1.19 1.19 100% 5 1286202 pio 0.35 0.84 1.19 1.19 100% 6 913522 aio 3.81 9.32 13.13 562.13 2% 7 340018 msc 7832.11 280.25 8112.36 10577.03 76% 8 311550 aio 1.34 1.52 2.86 79.80 3% 9 430252 aio 0.40 0.75 1.15 9.49 12% 10 1155232 aio 0.31 0.69 1.00 4.69 21% 11 1032332 soc 5.27 12.14 17.41 NA NA 12 1347764 soc 5.67 13.77 19.44 NA NA 13 712894 soc 4.88 11.96 16.84 NA NA 14 1019938 soc 8.29 22.64 30.93 NA NA 15 1261766 shm 0.03 0.07 0.10 NA NA 16 1544216 cpu 19.09 7.24 26.33 26.33 100% tot 8290.53 565.64 8856.17 Thanks, ******************************************************************* Ernie Knox IT Database Administrator Specialist Sears Holdings 3333 Beverly Rd., B4-266A Hoffman Estates, IL. 60179 Office: (847) 286-5735 Email: Ernest.Knox@searshc.com Blackberry: 2244650553@messaging.sprintpcs.com Page via Skytel: 2244650553@sprint.skytel.com Informix or MySQL Primary: 9110210@skytel.com Informix or MySQL Secondary: 7276872@skytel.com " Yes we can make a Change! " " It's always a great day to watch Sports - GO LIONS, TIGERS, and BEARS! " " Lets not forget - GO Pistons and Red Wings! " GSU ******************************************************************* -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Madison Pruet Sent: Thursday, October 08, 2009 3:01 PM To: ids@iiug.org Subject: Re: Seeing one oninit working hard while other.... [17446] It'd make more sense to look at onstat -g glo than ps -ef. That way w= e can see the function of each of the oninit processes to determine if anything is really out of line. M.P. = From: "Knox, Ernest" <Ernest.Knox@searshc.com> = = To: ids@iiug.org = = Date: 10/08/2009 02:57 PM = = Subject: Seeing one oninit working hard while others ar.... [1744= 5] = Sent by: ids-bounces@iiug.org = = I think we are running into a condition where only one virtual processo= r is trying to do a majority of the work on a remote server via synonyms.= When you look at all the oninit modules running on the remote server, i= t looks like only one is really taxed: ps -ef |grep oninit hds 303146 1007698 0 11:27:08 pts/0 0:00 grep oninit informix 332022 1327236 3 03:11:49 - 4:37 oninit -s informix 339984 1327236 0 03:11:50 - 0:01 oninit -s informix 573444 1327236 0 03:11:53 - 0:00 oninit -s informix 663766 1327236 0 03:11:54 - 0:57 oninit -s informix 729146 1327236 0 03:11:55 - 0:00 oninit -s informix 938176 1327236 0 03:11:51 - 0:15 oninit -s informix 962686 1327236 84 03:11:52 - 164:32 oninit -s informix 1032288 1327236 0 03:11:54 - 0:27 oninit -s informix 1101972 1327236 0 03:11:49 - 0:01 oninit -s informix 1327236 1499276 0 03:11:49 - 0:01 oninit -s informix 1380472 1327236 0 03:11:53 - 0:00 oninit -s informix 1388604 1327236 0 03:11:54 - 0:23 oninit -s informix 1400916 1327236 0 03:11:53 - 0:02 oninit -s informix 1454178 1327236 0 03:11:54 - 0:21 oninit -s informix 1499276 1 4 03:11:48 - 8:35 oninit -s from 11.5 Running Poll Threads on CPU or Network Virtual Processors Poll threads can run either on CPU virtual processors or on network virtual processors. In general, and particularly on a single-processor computer, poll threads run more efficiently on CPU virtual processors. This might not be true, however, on a multiprocessor computer with a large number of remote clients. The NETTYPE parameter has an optional entry, called vp class, that allows you to specify either CPU or NET, for CPU or network virtual-processor classes, respectively. If you do not specify a virtual processor class for the interface/protocol combination (poll threads) associated with the DBSERVERNAME variable, the class defaults to CPU. The database server assumes that the interface/protocol combination associated with DBSERVERNAME is the primary interface/protocol combination and that it should be the most efficient. For other interface/protocol combinations, if no vp class is specified,= the default is NET. While the database server is in online mode, you cannot drop a CPU virtual processor that is run

It's normal to see the CPUVPS doing the bulk of the work. The other VP= S are special purpose VPS and are only used to do things such as IO, netw= ork stuff, etc. Also it is normal to see the first couple of CPUVPS doing more work tha= n the later. It is not efficient to assign a thread to a VP if that VP i= s currently sleeping. The later CPUVPS will tend to do work only when th= ere is something in the ready queue to be worked on and the earlier CPUVPS = are already busy. So to see the first couple of CPUVPS being more busy tha= n the later ones is not unexpected. = From: "Knox, Ernest" <Ernest.Knox@searshc.com> = = To:) ids@iiug.org = = Date: 10/08/2009 03:58 PM = = Subject: RE: Seeing one oninit working hard while other.... [1744= 7] = Sent by: ids-bounces@iiug.org = = Database server: Database:/home/informix>onstat -g glo IBM Informix Dynamic Server Version 11.50.FC3 -- On-Line -- Up 4 days 09:51:55 -- 6104464 Kbytes MT global info: sessions threads vps lngspins 692 850 22 117863 sched calls thread switches yield 0 yield n yield forever total: 2489299885 572251415 1929513329 50426116 88396984 per sec: 4847 2841 2026 158 358 Virtual processor summary: class vps usercpu syscpu total cpu 4 126472.60 3243.03 129715.63 aio 9 1221.85 2199.69 3421.54 shm 1 6.02 7.43 13.45 lio 1 109.82 493.09 602.91 pio 1 3.77 34.09 37.86 adm 1 38.85 14.77 53.62 soc 4 974.19 2377.86 3352.05 msc 1 4365.69 398.30 4763.99 total 22 133192.79 8768.26 141961.05 Individual virtual processors: vp pid class usercpu syscpu total Thread Eff 1 581758 cpu 37824.67 1386.72 39211.39 39211.39 100% 2 491564 adm 38.85 14.77 53.62 0.00 0% 3 749710 cpu 52742.08 1066.24 53808.32 54099.88 99% 4 1110090 cpu 13775.60 515.50 14291.10 86223.55 16% 5 1601776 cpu 22130.25 274.57 22404.82 23063.49 97% 6 1495118 lio 109.82 493.09 602.91 35616.46 1% 7 1675312 pio 3.77 34.09 37.86 1556.62 2% 8 602174 aio 357.42 1066.91 1424.33 53515.96 2% 9 1171642 msc 4365.69 398.30 4763.99 4887.14 97% 10 733362 aio 194.82 664.36 859.18 47093.70 1% 11 905420 aio 114.99 118.42 233.41 44258.21 0% 12 901344 aio 108.95 87.28 196.23 43906.64 0% 13 1450166 aio 104.93 75.29 180.22 43706.33 0% 14 1102038 aio 105.22 70.60 175.82 43414.00 0% 15 1388554 shm 6.02 7.43 13.45 NA NA 16 946322 soc 211.77 490.53 702.30 NA NA 17 540754 soc 269.73 669.01 938.74 NA NA 18 1654948 soc 264.59 657.44 922.03 NA NA 19 1667226 soc 228.10 560.88 788.98 NA NA 20 508154 aio 99.85 56.50 156.35 36702.92 0% 21 1650716 aio 82.14 38.90 121.04 26227.22 0% 22 1224704 aio 53.53 21.43 74.96 15691.52 0% tot 133192.79 8768.26 141961.05 Remote Server: Remote:/usr/informix>onstat -g glo IBM Informix Dynamic Server Version 11.50.UC3 -- On-Line -- Up 03:06:40 -- 1242192 Kbytes MT global info: sessions threads vps lngspins 116 185 16 2911 sched calls thread switches yield 0 yield n yield forever total: 36843457 12620339 24225482 558900 2206927 per sec: 1196 488 708 52 147 Virtual processor summary: class vps usercpu syscpu total cpu 3 427.30 210.58 637.88 aio 4 5.86 12.28 18.14 shm 1 0.03 0.07 0.10 lio 1 0.36 0.83 1.19 pio 1 0.35 0.84 1.19 adm 1 0.41 0.28 0.69 soc 4 24.11 60.51 84.62 msc 1 7832.11 280.25 8112.36 total 16 8290.53 565.64 8856.17 Individual virtual processors: vp pid class usercpu syscpu total Thread Eff 1 663652 cpu 239.61 123.81 363.42 363.42 100% 2 1380526 adm 0.41 0.28 0.69 0.00 0% 3 938224 cpu 168.60 79.53 248.13 283.67 87% 4 1478902 lio 0.36 0.83 1.19 1.19 100% 5 1286202 pio 0.35 0.84 1.19 1.19 100% 6 913522 aio 3.81 9.32 13.13 562.13 2% 7 340018 msc 7832.11 280.25 8112.36 10577.03 76% 8 311550 aio 1.34 1.52 2.86 79.80 3% 9 430252 aio 0.40 0.75 1.15 9.49 12% 10 1155232 aio 0.31 0.69 1.00 4.69 21% 11 1032332 soc 5.27 12.14 17.41 NA NA 12 1347764 soc 5.67 13.77 19.44 NA NA 13 712894 soc 4.88 11.96 16.84 NA NA 14 1019938 soc 8.29 22.64 30.93 NA NA 15 1261766 shm 0.03 0.07 0.10 NA NA 16 1544216 cpu 19.09 7.24 26.33 26.33 100% tot 8290.53 565.64 8856.17 Thanks, ******************************************************************* Ernie Knox IT Database Administrator Specialist Sears Holdings 3333 Beverly Rd., B4-266A Hoffman Estates, IL. 60179 Office: (847) 286-5735 Email: Ernest.Knox@searshc.com Blackberry: 2244650553@messaging.sprintpcs.com Page via Skytel: 2244650553@sprint.skytel.com Informix or MySQL Primary: 9110210@skytel.com Informix or MySQL Secondary: 7276872@skytel.com " Yes we can make a Change! " " It's always a great day to watch Sports - GO LIONS, TIGERS, and BEARS= ! " " Lets not forget - GO Pistons and Red Wings! " GSU ******************************************************************* -----Original Message----- From: ids-bounces@iiug.org [?mailto:ids-bounces@iiug.org:] On Behalf Of= Madison Pruet Sent: Thursday, October 08, 2009 3:01 PM To: ids@iiug.org Subject: Re: Seeing one oninit working hard while other.... [17446] It'd make more sense to look at onstat -g glo than ps -ef. That way w=3D= e can see the function of each of the oninit processes to determine if anything is really out of line. M.P. =3D From: "Knox, Ernest" <Ernest.Knox@searshc.com> =3D =3D To: ids@iiug.org =3D =3D Date: 10/08/2009 02:57 PM =3D =3D Subject: Seeing one oninit working hard while others ar.... [1744=3D 5] =3D Sent by: ids-bounces@iiug.org =3D =3D I think we are running into a condition where only one virtual processo= =3D r is trying to do a majority of the work on a remote server via synonyms.= =3D When you look at all the oninit modules running on the remote server, i= =3D t looks like only one is really taxed: ps -ef |grep oninit hds 303146 1007698 0 11:27:08 pts/0 0:00 grep oninit informix 332022 1327236 3 03:11:49 - 4:37 oninit -s informix 339984 1327236 0 03:11:50 - 0:01 oninit -s informix 573444 1327236 0 03:11:53 - 0:00 oninit -s informix 663766 1327236 0 03:11:54 - 0:57 oninit -s informix 729146 1327236 0 03:11:55 - 0:00 oninit -s informix 938176 1327236 0 03:11:51 - 0:15 oninit -s informix 962686 1327236 84 03:11:52 - 164:32 oninit -s informix 1032288 1327236 0 03:11:54 - 0:27 oninit -s informix 1101972 1327236 0 03:11:49 - 0:01 oninit -s informix 1327236 1499276 0 03:11:49 - 0:01 oninit -s informix 1380472 1327236 0 03:11:53 - 0:00 oninit -s informix 1388604 1327236 0 03:11:54 - 0:23 oninit -s informix 1400916 1327236 0 03:11:53 - 0:02 oninit -s informix 1454178 1327236 0 03:11:54 - 0:21 oninit -s informix 1499276 1 4 03:11:48 - 8:35 oninit -s from 11.5 Running Poll Threads on CPU or Network Virtual Processors Poll threads can run either on CPU virtual processors

One thing that strikes me as very very odd is on what you are calling the remote server, the msc vp has used ~ 90% of the total cpu time for that instance. That seems very abnormal. Normally, the highest usages vps would be cpu vps. The msc vp is normally used to authenticate incoming connections. When you just posed the ps output, was the 1 oninit you were concerned with how much cpu time it was getting the msc vp (you can compare the ps output obviously with the pid from the onstat -g glo)? If so, there would appear to be some sort of issue here. You might want to try and get some stack traces off the msc vp to see what it seems to be spending it's time doing and then see if that leads you to some configuration issue or the like with how you are authenticating users on that machine. Jacques Renaut IBM Informix Advanced Support APD team You wrote: Remote Server: Remote:/usr/informix>onstat -g glo IBM Informix Dynamic Server Version 11.50.UC3 -- On-Line -- Up 03:06:40 -- 1242192 Kbytes MT global info: sessions threads vps lngspins 116 185 16 2911 sched calls thread switches yield 0 yield n yield forever total: 36843457 12620339 24225482 558900 2206927 per sec: 1196 488 708 52 147 Virtual processor summary: class vps usercpu syscpu total cpu 3 427.30 210.58 637.88 aio 4 5.86 12.28 18.14 shm 1 0.03 0.07 0.10 lio 1 0.36 0.83 1.19 pio 1 0.35 0.84 1.19 adm 1 0.41 0.28 0.69 soc 4 24.11 60.51 84.62 msc 1 7832.11 280.25 8112.36 total 16 8290.53 565.64 8856.17 Individual virtual processors: vp pid class usercpu syscpu total Thread Eff 1 663652 cpu 239.61 123.81 363.42 363.42 100% 2 1380526 adm 0.41 0.28 0.69 0.00 0% 3 938224 cpu 168.60 79.53 248.13 283.67 87% 4 1478902 lio 0.36 0.83 1.19 1.19 100% 5 1286202 pio 0.35 0.84 1.19 1.19 100% 6 913522 aio 3.81 9.32 13.13 562.13 2% 7 340018 msc 7832.11 280.25 8112.36 10577.03 76% 8 311550 aio 1.34 1.52 2.86 79.80 3% 9 430252 aio 0.40 0.75 1.15 9.49 12% 10 1155232 aio 0.31 0.69 1.00 4.69 21% 11 1032332 soc 5.27 12.14 17.41 NA NA 12 1347764 soc 5.67 13.77 19.44 NA NA 13 712894 soc 4.88 11.96 16.84 NA NA 14 1019938 soc 8.29 22.64 30.93 NA NA 15 1261766 shm 0.03 0.07 0.10 NA NA 16 1544216 cpu 19.09 7.24 26.33 26.33 100% tot 8290.53 565.64 8856.17

He probably has SQL capture running. Art Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Oct 9, 2009 at 10:35 AM, JACQUES RENAUT <jrenaut@us.ibm.com> wrote: > One thing that strikes me as very very odd is on what you are calling the > remote server, the msc vp has used ~ 90% of the total cpu time for that > instance. That seems very abnormal. Normally, the highest usages vps would > be > cpu vps. The msc vp is normally used to authenticate incoming connections. > When you just posed the ps output, was the 1 oninit you were concerned with > how much cpu time it was getting the msc vp (you can compare the ps output > obviously with the pid from the onstat -g glo)? If so, there would appear > to > be some sort of issue here. You might want to try and get some stack traces > off the msc vp to see what it seems to be spending it's time doing and then > see if that leads you to some configuration issue or the like with how you > are > authenticating users on that machine. > > Jacques Renaut > IBM Informix Advanced Support > APD team > > You wrote: > Remote Server: > > Remote:/usr/informix>onstat -g glo > > IBM Informix Dynamic Server Version 11.50.UC3 -- On-Line -- Up > 03:06:40 -- 1242192 Kbytes> > MT global info: > sessions threads vps lngspins > 116 185 16 2911 > > sched calls thread switches yield 0 yield n yield > forever > total: 36843457 12620339 24225482 558900 2206927 > per sec: 1196 488 708 52 147 > > Virtual processor summary: > class vps usercpu syscpu total > cpu 3 427.30 210.58 637.88 > aio 4 5.86 12.28 18.14 > shm 1 0.03 0.07 0.10 > lio 1 0.36 0.83 1.19 > pio 1 0.35 0.84 1.19 > adm 1 0.41 0.28 0.69 > soc 4 24.11 60.51 84.62 > msc 1 7832.11 280.25 8112.36 > total 16 8290.53 565.64 8856.17 > > Individual virtual processors: > vp pid class usercpu syscpu total Thread Eff > 1 663652 cpu 239.61 123.81 363.42 363.42 100% > 2 1380526 adm 0.41 0.28 0.69 0.00 0% > 3 938224 cpu 168.60 79.53 248.13 283.67 87% > 4 1478902 lio 0.36 0.83 1.19 1.19 100% > 5 1286202 pio 0.35 0.84 1.19 1.19 100% > 6 913522 aio 3.81 9.32 13.13 562.13 2% > 7 340018 msc 7832.11 280.25 8112.36 10577.03 76% > 8 311550 aio 1.34 1.52 2.86 79.80 3% > 9 430252 aio 0.40 0.75 1.15 9.49 12% > 10 1155232 aio 0.31 0.69 1.00 4.69 21% > 11 1032332 soc 5.27 12.14 17.41 NA NA > 12 1347764 soc 5.67 13.77 19.44 NA NA > 13 712894 soc 4.88 11.96 16.84 NA NA > 14 1019938 soc 8.29 22.64 30.93 NA NA > 15 1261766 shm 0.03 0.07 0.10 NA NA > 16 1544216 cpu 19.09 7.24 26.33 26.33 100% > > tot 8290.53 565.64 8856.17 > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --0015174bdcc809678c0475b11b37

We didn't plan on turning that on. I will check, but could that take up resources and cause issues? Thanks, ******************************************************************* Ernie Knox IT Database Administrator Specialist Sears Holdings 3333 Beverly Rd., B4-266A Hoffman Estates, IL. 60179 Office: (847) 286-5735 Email: Ernest.Knox@searshc.com Blackberry: 2244650553@messaging.sprintpcs.com Page via Skytel: 2244650553@sprint.skytel.com Informix or MySQL Primary: 9110210@skytel.com Informix or MySQL Secondary: 7276872@skytel.com " Yes we can make a Change! " " It's always a great day to watch Sports - GO LIONS, TIGERS, and BEARS! " " Lets not forget - GO Pistons and Red Wings! " GSU ******************************************************************* -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Art Kagel Sent: Sunday, October 11, 2009 6:24 PM To: ids@iiug.org Subject: Re: RE: Seeing one oninit working hard while o.... [17476] He probably has SQL capture running. Art Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Oct 9, 2009 at 10:35 AM, JACQUES RENAUT <jrenaut@us.ibm.com> wrote: > One thing that strikes me as very very odd is on what you are calling the > remote server, the msc vp has used ~ 90% of the total cpu time for that > instance. That seems very abnormal. Normally, the highest usages vps would > be > cpu vps. The msc vp is normally used to authenticate incoming connections. > When you just posed the ps output, was the 1 oninit you were concerned with > how much cpu time it was getting the msc vp (you can compare the ps output > obviously with the pid from the onstat -g glo)? If so, there would appear > to > be some sort of issue here. You might want to try and get some stack traces > off the msc vp to see what it seems to be spending it's time doing and then > see if that leads you to some configuration issue or the like with how you > are > authenticating users on that machine. > > Jacques Renaut > IBM Informix Advanced Support > APD team > > You wrote: > Remote Server: > > Remote:/usr/informix>onstat -g glo > > IBM Informix Dynamic Server Version 11.50.UC3 -- On-Line -- Up > 03:06:40 -- 1242192 Kbytes> > MT global info: > sessions threads vps lngspins > 116 185 16 2911 > > sched calls thread switches yield 0 yield n yield > forever > total: 36843457 12620339 24225482 558900 2206927 > per sec: 1196 488 708 52 147 > > Virtual processor summary: > class vps usercpu syscpu total > cpu 3 427.30 210.58 637.88 > aio 4 5.86 12.28 18.14 > shm 1 0.03 0.07 0.10 > lio 1 0.36 0.83 1.19 > pio 1 0.35 0.84 1.19 > adm 1 0.41 0.28 0.69 > soc 4 24.11 60.51 84.62 > msc 1 7832.11 280.25 8112.36 > total 16 8290.53 565.64 8856.17 > > Individual virtual processors: > vp pid class usercpu syscpu total Thread Eff > 1 663652 cpu 239.61 123.81 363.42 363.42 100% > 2 1380526 adm 0.41 0.28 0.69 0.00 0% > 3 938224 cpu 168.60 79.53 248.13 283.67 87% > 4 1478902 lio 0.36 0.83 1.19 1.19 100% > 5 1286202 pio 0.35 0.84 1.19 1.19 100% > 6 913522 aio 3.81 9.32 13.13 562.13 2% > 7 340018 msc 7832.11 280.25 8112.36 10577.03 76% > 8 311550 aio 1.34 1.52 2.86 79.80 3% > 9 430252 aio 0.40 0.75 1.15 9.49 12% > 10 1155232 aio 0.31 0.69 1.00 4.69 21% > 11 1032332 soc 5.27 12.14 17.41 NA NA > 12 1347764 soc 5.67 13.77 19.44 NA NA > 13 712894 soc 4.88 11.96 16.84 NA NA > 14 1019938 soc 8.29 22.64 30.93 NA NA > 15 1261766 shm 0.03 0.07 0.10 NA NA > 16 1544216 cpu 19.09 7.24 26.33 26.33 100% > > tot 8290.53 565.64 8856.17 > > > > ************************************************************************ ******* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --0015174bdcc809678c0475b11b37 ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum.

Where is the capture parameter? I really only see the EXPLAIN_STAT enable. Is that it? If I turn it off, will that possibly improve anything? # SQL Statement Cache Configuration Parameters ################################################################### # STMT_CACHE - Controls SQL statement caching. Acceptable # values are: # 0 Disabled # 1 Enabled at the session level # 2 All statements are cached # STMT_CACHE_HITS - The number of times an SQL statement must be # executed before becoming fully cached. # 0 indicates that all statements are # fully cached the first time. # STMT_CACHE_SIZE - The size, in KB, of the SQL statement cache # STMT_CACHE_NOLIMIT - Controls additional memory consumption. # Acceptable values are: # 0 Limit memory to STMT_CACHE_SIZE # 1 Obtain as much memory, temporarily, as needed # STMT_CACHE_NUMPOOL - The number of pools for the SQL statement # cache. Acceptable value is a positive # integer between 1 and 256, inclusive. ################################################################### STMT_CACHE 0 STMT_CACHE_HITS 0 STMT_CACHE_SIZE 512 STMT_CACHE_NOLIMIT 0 STMT_CACHE_NUMPOOL 1 ################################################################### # Operating System Session-Related Configuration Parameters ################################################################### # USEOSTIME - The precision of SQL statement timing. # Accepted values are 0 (precision to seconds) # and 1 (precision to subseconds). Subsecond # precision can degrade performance. # STACKSIZE - The size, in KB, for a session stack # ALLOW_NEWLINE - Controls whether embedded new line characters # in string literals are allowed in SQL # statements. Acceptable values are 1 (allowed) # and any number other than 1 (not allowed). # USELASTCOMMITTED - Controls the committed read isolation level. # Acceptable values are: # - NONE Waits on a lock # - DIRTY READ Uses the last committed value in # place of a dirty read # - COMMITTED READ Uses the last committed value # in place of a committed read # - ALL Uses the last committed value in place # of all isolation levels that support the last # committed option ################################################################### USEOSTIME 0 STACKSIZE 64 ALLOW_NEWLINE 0 USELASTCOMMITTED NONE ################################################################### # Optimizer Configuration Parameters ################################################################### # OPTCOMPIND - Controls how the optimizer determines the best # query path. Acceptable values are: # 0 Nested loop joins are preferred # 1 If isolation level is repeatable read, # works the same as 0, otherwise works same as 2 # 2 Optimizer decisions are based on cost only # DIRECTIVES - Specifies whether optimizer directives are # enabled (1) or disabled (0). Default is 1. # EXT_DIRECTIVES - Controls the use of external SQL directives. # Acceptable values are: # 0 Disabled # 1 Enabled if the IFX_EXTDIRECTIVES environment # variable is enabled # 2 Enabled even if the IFX_EXTDIRECTIVES # environment is not set # OPT_GOAL - Controls how the optimizer should optimize for # fastest retrieval. Acceptable values are: # -1 All rows in a query # 0 The first rows in a query # IFX_FOLDVIEW - Enables (1) or disables (0) folding views that # have multiple tables or a UNION ALL clause. # Disabled by default. # AUTO_REPREPARE - Enables (1) or disables (0) automatically # re-optimizing stored procedures and re-preparing # prepared statements when tables that are referenced # by them change. Minimizes the occurrence of the # -710 error. #################################################################### OPTCOMPIND 2 DIRECTIVES 1 EXT_DIRECTIVES 0 OPT_GOAL -1 IFX_FOLDVIEW 0AUTO_REPREPARE 1 ################################################################### # SQL Tracing and EXPLAIN Plan Configuration Parameters ################################################################### # EXPLAIN_STAT - Enables (1) or disables (0) including the Query # Statistics section in the EXPLAIN output file # SQLTRACE - Configures SQL tracing. The format is: # SQLTRACE level=(low|med|high),ntraces=<#>,size=<#>, # mode=(global|user) ################################################################### EXPLAIN_STAT 1 #SQLTRACE level=low,ntraces=1000,size=2,mode=global Thanks, ******************************************************************* Ernie Knox IT Database Administrator Specialist Sears Holdings 3333 Beverly Rd., B4-266A Hoffman Estates, IL. 60179 Office: (847) 286-5735 Email: Ernest.Knox@searshc.com Blackberry: 2244650553@messaging.sprintpcs.com Page via Skytel: 2244650553@sprint.skytel.com Informix or MySQL Primary: 9110210@skytel.com Informix or MySQL Secondary: 7276872@skytel.com " Yes we can make a Change! " " It's always a great day to watch Sports - GO LIONS, TIGERS, and BEARS! " " Lets not forget - GO Pistons and Red Wings! " GSU ******************************************************************* -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Knox, Ernest Sent: Sunday, October 11, 2009 6:27 PM To: ids@iiug.org Subject: RE: RE: Seeing one oninit working hard while o.... [17477] We didn't plan on turning that on. I will check, but could that take up resources and cause issues? Thanks, ******************************************************************* Ernie Knox IT Database Administrator Specialist Sears Holdings 3333 Beverly Rd., B4-266A Hoffman Estates, IL. 60179 Office: (847) 286-5735 Email: Ernest.Knox@searshc.com Blackberry: 2244650553@messaging.sprintpcs.com Page via Skytel: 2244650553@sprint.skytel.com Informix or MySQL Primary: 9110210@skytel.com Informix or MySQL Secondary: 7276872@skytel.com " Yes we can make a Change! " " It's always a great day to watch Sports - GO LIONS, TIGERS, and BEARS! " " Lets not forget - GO Pistons and Red Wings! " GSU ******************************************************************* -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Art Kagel Sent: Sunday, October 11, 2009 6:24 PM To: ids@iiug.org Subject: Re: RE: Seeing one oninit working hard while o.... [17476] He probably has SQL capture running. Art Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Oct 9,

No the parameter is SQLTRACE which appears to be off, at least the config line you posted is commented out: #SQLTRACE level=low,ntraces=1000,size=2,mode=global Art Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Sun, Oct 11, 2009 at 7:42 PM, Knox, Ernest <Ernest.Knox@searshc.com>wrote: > Where is the capture parameter? I really only see the EXPLAIN_STAT > enable. Is that it? If I turn it off, will that possibly improve > anything? > > # SQL Statement Cache Configuration Parameters > ################################################################### > # STMT_CACHE - Controls SQL statement caching. Acceptable > # values are: > # 0 Disabled > # 1 Enabled at the session level > # 2 All statements are cached > # STMT_CACHE_HITS - The number of times an SQL statement must be > # executed before becoming fully cached. > # 0 indicates that all statements are > # fully cached the first time. > # STMT_CACHE_SIZE - The size, in KB, of the SQL statement cache > # STMT_CACHE_NOLIMIT - Controls additional memory consumption. > # Acceptable values are: > # 0 Limit memory to STMT_CACHE_SIZE > # 1 Obtain as much memory, temporarily, as needed > # STMT_CACHE_NUMPOOL - The number of pools for the SQL statement > # cache. Acceptable value is a positive > # integer between 1 and 256, inclusive. > ################################################################### > > STMT_CACHE 0 > STMT_CACHE_HITS 0 > STMT_CACHE_SIZE 512 > STMT_CACHE_NOLIMIT 0 > STMT_CACHE_NUMPOOL 1> > ################################################################### > # Operating System Session-Related Configuration Parameters > ################################################################### > # USEOSTIME - The precision of SQL statement timing. > # Accepted values are 0 (precision to seconds) > # and 1 (precision to subseconds). Subsecond > # precision can degrade performance. > # STACKSIZE - The size, in KB, for a session stack > # ALLOW_NEWLINE - Controls whether embedded new line characters > # in string literals are allowed in SQL > # statements. Acceptable values are 1 (allowed) > # and any number other than 1 (not allowed). > # USELASTCOMMITTED - Controls the committed read isolation level. > # Acceptable values are: > # - NONE Waits on a lock > # - DIRTY READ Uses the last committed value in > # place of a dirty read > # - COMMITTED READ Uses the last committed value > # in place of a committed read > # - ALL Uses the last committed value in place > # of all isolation levels that support the last > # committed option > ################################################################### > > USEOSTIME 0 > STACKSIZE 64 > ALLOW_NEWLINE 0 > USELASTCOMMITTED NONE> > ################################################################### > # Optimizer Configuration Parameters > ################################################################### > # OPTCOMPIND - Controls how the optimizer determines the best > # query path. Acceptable values are: > # 0 Nested loop joins are preferred > # 1 If isolation level is repeatable read, > # works the same as 0, otherwise works same as 2 > # 2 Optimizer decisions are based on cost only > # DIRECTIVES - Specifies whether optimizer directives are > # enabled (1) or disabled (0). Default is 1. > # EXT_DIRECTIVES - Controls the use of external SQL directives. > # Acceptable values are: > # 0 Disabled > # 1 Enabled if the IFX_EXTDIRECTIVES environment > # variable is enabled > # 2 Enabled even if the IFX_EXTDIRECTIVES > # environment is not set > # OPT_GOAL - Controls how the optimizer should optimize for > # fastest retrieval. Acceptable values are: > # -1 All rows in a query > # 0 The first rows in a query > # IFX_FOLDVIEW - Enables (1) or disables (0) folding views that > # have multiple tables or a UNION ALL clause. > # Disabled by default. > # AUTO_REPREPARE - Enables (1) or disables (0) automatically > # re-optimizing stored procedures and re-preparing > # prepared statements when tables that are referenced > # by them change. Minimizes the occurrence of the > # -710 error. > #################################################################### > > OPTCOMPIND 2 > DIRECTIVES 1 > EXT_DIRECTIVES 0 > OPT_GOAL -1 > IFX_FOLDVIEW 0> AUTO_REPREPARE 1 > > ################################################################### > # SQL Tracing and EXPLAIN Plan Configuration Parameters > ################################################################### > # EXPLAIN_STAT - Enables (1) or disables (0) including the Query > # Statistics section in the EXPLAIN output file > # SQLTRACE - Configures SQL tracing. The format is: > # SQLTRACE level=(low|med|high),ntraces=<#>,size=<#>, > # mode=(global|user) > ################################################################### > > EXPLAIN_STAT 1 > #SQLTRACE level=low,ntraces=1000,size=2,mode=global> > Thanks, > > ******************************************************************* > > Ernie Knox > > IT Database Administrator Specialist > > Sears Holdings > > 3333 Beverly Rd., B4-266A > > Hoffman Estates, IL. 60179 > > Office: (847) 286-5735 > > Email: Ernest.Knox@searshc.com > > Blackberry: 2244650553@messaging.sprintpcs.com > > Page via Skytel: 2244650553@sprint.skytel.com > > Informix or MySQL Primary: 9110210@skytel.com > > Informix or MySQL Secondary: 7276872@skytel.com > > " Yes we can make a Change! " > > " It's always a great day to watch Sports - GO LIONS, TIGERS, and BEARS! > " > > " Lets not forget - GO Pistons and Red Wings! " > > GSU > > ******************************************************************* > > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > Knox, Ernest > Sent: Sunday, October 11, 2009 6:27 PM > To: ids@iiug.org > Subject: RE: RE: Seeing one oninit working hard while o.... [17477] > > We didn't plan on turning that on. I will check, but could that take up > resources and cause issues? > > Thanks, > > ******************************************************************* > > Ernie Knox > > IT Database Administrator Specialist > > Sears Holdings > > 3333 Beverly Rd., B4-266A > > Hoffman Estates, IL. 60179 > > Office: (847) 286-5735 > > Email: Ernest.Knox@searshc.com > > Blackberry: 2244650553@messaging.sprintpcs.com > > Page via Skytel: 2244650553@sprint.skytel.com > > Informix or MySQL Primary: 9110210@skytel.com > > Informix or MySQL Secondary: 7276872@skytel.com > > " Yes we can make a Change! " > > " It's