CPU load balancing - Tru64

Another Unix question (also on Tru64): I used the "cpuinfo" command to monitor each CPU individualy. But the picture I get does not make 100% sense to me. Let's say there are 10 cpu's. What I am seeing is, 3 of those cpu's are very busy, and the other 7 are idle. Should Unix not be load balancing (context switching) ? Because this now means, when I run a sar, I see 85% idle time, which means in total the server is not very busy, but 3 of the cpu's are running flat out. This would cause problems, would it not ? Dirk ________________________________ NOTE: This e-mail message is subject to the MTN Group disclaimer see http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx

I'm not sure if I'm missing some peace of context (I noticed there are other messages from you). In any case, that could happen for example if you had only 3 CPUVPs on an Informix instance. The software would not be able to saturate more than 3 CPUs. I think you would see that the set of 3 would not necessarily be the same 3 (e.g. cpu0, cpu1, cpu2) every time. But this could depend on CPU affinity. When you have free CPU and things are slow you should always consider other sources of contention (like I/O, swap etc.) Regarding your questions on another post: - Be careful when looking at each process memory because: - The processes that use shared memory cannot be added (since the memory they are referencing is shared among them). I assume this is clear, but I've seen too many people making this basic mistake - The RSS can (and probably will) include memory that is shared among the processes. I'm not talking about "shared memory" segments but things like , code segments etc. - On Tru64 there is a parameter called something like "swap eager" (see /etc/sysconfigtab) that defines how it uses swap. If it's set to "1" every memory allocation will mean the same ammount of swap reservation. If your swap space is low you can consume it in "reserverd". This may never be used but if the swap space is exhausted (even for reserved space) the OS may start killing processes. - I've seen performance problems that were attributed to advfs issues, although I never had a proper explanation. But it would happen when you have large file systems full of small files (many, many thousands or millions). I've seen machines that have slow response, although the CPU usage is low - Having too many processes (thounsands) apparently also can cause problems. There are settings that change the process table sizes, but I never changed them. Not sure if it helps. This translated in slow response and commands like "ps" took very long time and consumed too many CPUs If you want to share your concerns, maybe we can be more useful. If you already did it, I probably missed it, so I apologize. Regards. On Wed, Aug 31, 2011 at 10:16 AM, Dirk Cornel.... <moolma_dc@mtn.co.za>wrote: > Another Unix question (also on Tru64): > > I used the "cpuinfo" command to monitor each CPU individualy. But the > picture > I get does not make 100% sense to me. > > Let's say there are 10 cpu's. What I am seeing is, 3 of those cpu's are > very > busy, and the other 7 are idle. Should Unix not be load balancing (context > switching) ? > > Because this now means, when I run a sar, I see 85% idle time, which means > in > total the server is not very busy, but 3 of the cpu's are running flat out. > > This would cause problems, would it not ? > > Dirk > > ________________________________ > NOTE: This e-mail message is subject to the MTN Group disclaimer see > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --0016364d1f8d45020104abc9ff72

" I've seen machines that have slow response, although the CPU usage is low" Yes, this is what we are experiencing. So my first check were the cpu's. I also looked at I/O (the I/O was good, no problems there), and the memory was good (no swapping taking place on the server). So now I start to wonder, is this an inherent problem on Tru64 perhaps ? .... the cpu's not load balancing properly ? There could be many valid explanations. This server does not only run Informix. Maybe a process binds to a cpu, and cannot be split further, into multiple threads for multiple cpu's to be utilised. > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > Fernando Nunes > Sent: Wednesday, 31 August 2011 11:46 AM > To: ids@iiug.org > Subject: Re: CPU load balancing - Tru64 [24780] > > I'm not sure if I'm missing some peace of context (I noticed there are > other > messages from you). > In any case, that could happen for example if you had only 3 CPUVPs on > an > Informix instance. > The software would not be able to saturate more than 3 CPUs. I think > you > would see that the set of 3 would not necessarily be the same 3 (e.g. > cpu0, > cpu1, cpu2) every time. But this could depend on CPU affinity. > > When you have free CPU and things are slow you should always consider > other > sources of contention (like I/O, swap etc.) > > Regarding your questions on another post: > > - Be careful when looking at each process memory because: > - The processes that use shared memory cannot be added (since the > memory > they are referencing is shared among them). I assume this is clear, but > I've > seen too many people making this basic mistake > - The RSS can (and probably will) include memory that is shared among > the > processes. I'm not talking about "shared memory" segments but things > like , > code segments etc. > - On Tru64 there is a parameter called something like "swap eager" (see > /etc/sysconfigtab) that defines how it uses swap. If it's set to "1" > every > memory allocation will mean the same ammount of swap reservation. If > your > swap space is low you can consume it in "reserverd". This may never be > used > but if the swap space is exhausted (even for reserved space) the OS may > start killing processes. > - I've seen performance problems that were attributed to advfs issues, > although I never had a proper explanation. But it would happen when you > have > large file systems full of small files (many, many thousands or > millions). > I've seen machines that have slow response, although the CPU usage is > low > - Having too many processes (thounsands) apparently also can cause > problems. There are settings that change the process table sizes, but I > never changed them. Not sure if it helps. This translated in slow > response > and commands like "ps" took very long time and consumed too many CPUs > > If you want to share your concerns, maybe we can be more useful. If you > already did it, I probably missed it, so I apologize. > > Regards. > On Wed, Aug 31, 2011 at 10:16 AM, Dirk Cornel.... > <moolma_dc@mtn.co.za>wrote: > > > Another Unix question (also on Tru64): > > > > I used the "cpuinfo" command to monitor each CPU individualy. But the > > picture > > I get does not make 100% sense to me. > > > > Let's say there are 10 cpu's. What I am seeing is, 3 of those cpu's > are > > very > > busy, and the other 7 are idle. Should Unix not be load balancing > (context > > switching) ? > > > > Because this now means, when I run a sar, I see 85% idle time, which > means > > in > > total the server is not very busy, but 3 of the cpu's are running > flat out. > > > > This would cause problems, would it not ? > > > > Dirk > > > > ________________________________ > > NOTE: This e-mail message is subject to the MTN Group disclaimer see > > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > > > > > > *********************************************************************** > ******** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > -- > Fernando Nunes > Portugal > > http://informix-technology.blogspot.com > My email works... but I don't check it frequently... > > --0016364d1f8d45020104abc9ff72 > > > *********************************************************************** > ******** > Forum Note: Use "Reply" to post a response in the discussion forum. NOTE: This e-mail message is subject to the MTN Group disclaimer see http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx

"- Be careful when looking at each process memory because" I can still use this to get a general overall view, correct ? It should be accurate enough to still show me which applications are more memory intensive than others, even if not 100% accurate. > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > Fernando Nunes > Sent: Wednesday, 31 August 2011 11:46 AM > To: ids@iiug.org > Subject: Re: CPU load balancing - Tru64 [24780] > > I'm not sure if I'm missing some peace of context (I noticed there are > other > messages from you). > In any case, that could happen for example if you had only 3 CPUVPs on > an > Informix instance. > The software would not be able to saturate more than 3 CPUs. I think > you > would see that the set of 3 would not necessarily be the same 3 (e.g. > cpu0, > cpu1, cpu2) every time. But this could depend on CPU affinity. > > When you have free CPU and things are slow you should always consider > other > sources of contention (like I/O, swap etc.) > > Regarding your questions on another post: > > - Be careful when looking at each process memory because: > - The processes that use shared memory cannot be added (since the > memory > they are referencing is shared among them). I assume this is clear, but > I've > seen too many people making this basic mistake > - The RSS can (and probably will) include memory that is shared among > the > processes. I'm not talking about "shared memory" segments but things > like , > code segments etc. > - On Tru64 there is a parameter called something like "swap eager" (see > /etc/sysconfigtab) that defines how it uses swap. If it's set to "1" > every > memory allocation will mean the same ammount of swap reservation. If > your > swap space is low you can consume it in "reserverd". This may never be > used > but if the swap space is exhausted (even for reserved space) the OS may > start killing processes. > - I've seen performance problems that were attributed to advfs issues, > although I never had a proper explanation. But it would happen when you > have > large file systems full of small files (many, many thousands or > millions). > I've seen machines that have slow response, although the CPU usage is > low > - Having too many processes (thounsands) apparently also can cause > problems. There are settings that change the process table sizes, but I > never changed them. Not sure if it helps. This translated in slow > response > and commands like "ps" took very long time and consumed too many CPUs > > If you want to share your concerns, maybe we can be more useful. If you > already did it, I probably missed it, so I apologize. > > Regards. > On Wed, Aug 31, 2011 at 10:16 AM, Dirk Cornel.... > <moolma_dc@mtn.co.za>wrote: > > > Another Unix question (also on Tru64): > > > > I used the "cpuinfo" command to monitor each CPU individualy. But the > > picture > > I get does not make 100% sense to me. > > > > Let's say there are 10 cpu's. What I am seeing is, 3 of those cpu's > are > > very > > busy, and the other 7 are idle. Should Unix not be load balancing > (context > > switching) ? > > > > Because this now means, when I run a sar, I see 85% idle time, which > means > > in > > total the server is not very busy, but 3 of the cpu's are running > flat out. > > > > This would cause problems, would it not ? > > > > Dirk > > > > ________________________________ > > NOTE: This e-mail message is subject to the MTN Group disclaimer see > > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > > > > > > *********************************************************************** > ******** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > -- > Fernando Nunes > Portugal > > http://informix-technology.blogspot.com > My email works... but I don't check it frequently... > > --0016364d1f8d45020104abc9ff72 > > > *********************************************************************** > ******** > Forum Note: Use "Reply" to post a response in the discussion forum. NOTE: This e-mail message is subject to the MTN Group disclaimer see http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx

On 31/08/2011 10:56, Dirk Cornel.... wrote: > " I've seen machines that have slow response, although the CPU usage is low" > > Yes, this is what we are experiencing. So my first check were the cpu's. I > also looked at I/O (the I/O was good, no problems there), and the memory was > good (no swapping taking place on the server). > > So now I start to wonder, is this an inherent problem on Tru64 perhaps ? .... > the cpu's not load balancing properly ? > > There could be many valid explanations. This server does not only run > Informix. Maybe a process binds to a cpu, and cannot be split further, into > multiple threads for multiple cpu's to be utilised. What does onstat -C show? -- Cheers, Obnoxio The Clown http://obotheclown.blogspot.com I will now proceed to pleasure myself with this fish.

> -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > Obnoxio The Clown > Sent: Wednesday, 31 August 2011 12:02 PM > To: ids@iiug.org > Subject: Re: CPU load balancing - Tru64 [24783] > > On 31/08/2011 10:56, Dirk Cornel.... wrote: > > " I've seen machines that have slow response, although the CPU usage > is low" > > > > Yes, this is what we are experiencing. So my first check were the > cpu's. I > > also looked at I/O (the I/O was good, no problems there), and the > memory was > > good (no swapping taking place on the server). > > > > So now I start to wonder, is this an inherent problem on Tru64 > perhaps ? > ..... > > the cpu's not load balancing properly ? > > > > There could be many valid explanations. This server does not only run > > Informix. Maybe a process binds to a cpu, and cannot be split > further, into > > multiple threads for multiple cpu's to be utilised. > > What does onstat -C show? > > -- > Cheers, > Obnoxio The Clown The problem is, this is an Apps server - it only runs 4ge's, and Java applications - no database. And all the other servers are fine. So I think the problem is limited to this server. More frustrating is, the server would show 85% plus idle time, but they are complaining it is slow. We reboot only this server, and everything is fine again. This is why I started digging into processes now, seeing which are using the most memory on this server, etc. The CPU scenario I described, is also strange to me. What if the process of the user that is complaining, is running on one of the cpu's that are busy, while the server in general still has idle time ? NOTE: This e-mail message is subject to the MTN Group disclaimer see http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx

On 31/08/2011 11:17, Dirk Cornel.... wrote: >> -----Original Message----- >> From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of >> Obnoxio The Clown >> Sent: Wednesday, 31 August 2011 12:02 PM >> To: ids@iiug.org >> Subject: Re: CPU load balancing - Tru64 [24783] >> >> On 31/08/2011 10:56, Dirk Cornel.... wrote: >>> " I've seen machines that have slow response, although the CPU usage >> is low" >>> >>> Yes, this is what we are experiencing. So my first check were the >> cpu's. I >>> also looked at I/O (the I/O was good, no problems there), and the >> memory was >>> good (no swapping taking place on the server). >>> >>> So now I start to wonder, is this an inherent problem on Tru64 >> perhaps ? >> ..... >>> the cpu's not load balancing properly ? >>> >>> There could be many valid explanations. This server does not only run >>> Informix. Maybe a process binds to a cpu, and cannot be split >> further, into >>> multiple threads for multiple cpu's to be utilised. >> >> What does onstat -C show? >> >> -- >> Cheers, >> Obnoxio The Clown > > The problem is, this is an Apps server - it only runs 4ge's, and Java > applications - no database. And all the other servers are fine. So I think the > problem is limited to this server. > > More frustrating is, the server would show 85% plus idle time, but they are > complaining it is slow. We reboot only this server, and everything is fine > again. > This is why I started digging into processes now, seeing which are using the > most memory on this server, etc. > > The CPU scenario I described, is also strange to me. What if the process of > the user that is complaining, is running on one of the cpu's that are busy, > while the server in general still has idle time ? My best guess (and it is just a guess) is that the server has priority aging, in other words, the older a process is, the less CPU time it gets. So even if the box isn't doing much else, those processes that actually are doing work will slide down the scale. Look at the nice values on processes before and after a reboot, and look into how to disable aging in the kernel. -- Cheers, Obnoxio The Clown http://obotheclown.blogspot.com I will now proceed to pleasure myself with this fish.

> -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > Obnoxio The Clown > Sent: Wednesday, 31 August 2011 12:24 PM > To: ids@iiug.org > Subject: Re: CPU load balancing - Tru64 [24785] > > On 31/08/2011 11:17, Dirk Cornel.... wrote: > >> -----Original Message----- > >> From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf > Of > >> Obnoxio The Clown > >> Sent: Wednesday, 31 August 2011 12:02 PM > >> To: ids@iiug.org > >> Subject: Re: CPU load balancing - Tru64 [24783] > >> > >> On 31/08/2011 10:56, Dirk Cornel.... wrote: > >>> " I've seen machines that have slow response, although the CPU > usage > >> is low" > >>> > >>> Yes, this is what we are experiencing. So my first check were the > >> cpu's. I > >>> also looked at I/O (the I/O was good, no problems there), and the > >> memory was > >>> good (no swapping taking place on the server). > >>> > >>> So now I start to wonder, is this an inherent problem on Tru64 > >> perhaps ? > >> ..... > >>> the cpu's not load balancing properly ? > >>> > >>> There could be many valid explanations. This server does not only > run > >>> Informix. Maybe a process binds to a cpu, and cannot be split > >> further, into > >>> multiple threads for multiple cpu's to be utilised. > >> > >> What does onstat -C show? > >> > >> -- > >> Cheers, > >> Obnoxio The Clown > > > > The problem is, this is an Apps server - it only runs 4ge's, and Java > > applications - no database. And all the other servers are fine. So I > think > the > > problem is limited to this server. > > > > More frustrating is, the server would show 85% plus idle time, but > they are > > complaining it is slow. We reboot only this server, and everything is > fine > > again. > > This is why I started digging into processes now, seeing which are > using the > > most memory on this server, etc. > > > > The CPU scenario I described, is also strange to me. What if the > process of > > the user that is complaining, is running on one of the cpu's that are > busy, > > while the server in general still has idle time ? > My best guess (and it is just a guess) is that the server has priority > aging, in other words, the older a process is, the less CPU time it > gets. So even if the box isn't doing much else, those processes that > actually are doing work will slide down the scale. Look at the nice > values on processes before and after a reboot, and look into how to > disable aging in the kernel. > > -- > Cheers, > Obnoxio The Clown Good idea, thanks OB. Dirk NOTE: This e-mail message is subject to the MTN Group disclaimer see http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx

But are your 4GE's "long runners"? I never found this to be a problem with oninit on Tru64. But maybe I was lucky. On Wed, Aug 31, 2011 at 11:26 AM, Dirk Cornel.... <moolma_dc@mtn.co.za>wrote: > > -----Original Message----- > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > > Obnoxio The Clown > > Sent: Wednesday, 31 August 2011 12:24 PM > > To: ids@iiug.org > > Subject: Re: CPU load balancing - Tru64 [24785] > > > > On 31/08/2011 11:17, Dirk Cornel.... wrote: > > >> -----Original Message----- > > >> From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf > > Of > > >> Obnoxio The Clown > > >> Sent: Wednesday, 31 August 2011 12:02 PM > > >> To: ids@iiug.org > > >> Subject: Re: CPU load balancing - Tru64 [24783] > > >> > > >> On 31/08/2011 10:56, Dirk Cornel.... wrote: > > >>> " I've seen machines that have slow response, although the CPU > > usage > > >> is low" > > >>> > > >>> Yes, this is what we are experiencing. So my first check were the > > >> cpu's. I > > >>> also looked at I/O (the I/O was good, no problems there), and the > > >> memory was > > >>> good (no swapping taking place on the server). > > >>> > > >>> So now I start to wonder, is this an inherent problem on Tru64 > > >> perhaps ? > > >> ..... > > >>> the cpu's not load balancing properly ? > > >>> > > >>> There could be many valid explanations. This server does not only > > run > > >>> Informix. Maybe a process binds to a cpu, and cannot be split > > >> further, into > > >>> multiple threads for multiple cpu's to be utilised. > > >> > > >> What does onstat -C show? > > >> > > >> -- > > >> Cheers, > > >> Obnoxio The Clown > > > > > > The problem is, this is an Apps server - it only runs 4ge's, and Java > > > applications - no database. And all the other servers are fine. So I > > think > > the > > > problem is limited to this server. > > > > > > More frustrating is, the server would show 85% plus idle time, but > > they are > > > complaining it is slow. We reboot only this server, and everything is > > fine > > > again. > > > This is why I started digging into processes now, seeing which are > > using the > > > most memory on this server, etc. > > > > > > The CPU scenario I described, is also strange to me. What if the > > process of > > > the user that is complaining, is running on one of the cpu's that are > > busy, > > > while the server in general still has idle time ? > > > My best guess (and it is just a guess) is that the server has priority > > aging, in other words, the older a process is, the less CPU time it > > gets. So even if the box isn't doing much else, those processes that > > actually are doing work will slide down the scale. Look at the nice > > values on processes before and after a reboot, and look into how to > > disable aging in the kernel. > > > > -- > > Cheers, > > Obnoxio The Clown > > Good idea, thanks OB. > > Dirk > > NOTE: This e-mail message is subject to the MTN Group disclaimer see > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --002215b02e3e0118ef04abcb761f

Fernando's point is that you cannot add up the memory usage of the Informix oninit processes (or other processes that share memory with each other) to determine if you are overusing memory because that counts the.shared segments mulitiple times. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Wed, Aug 31, 2011 at 5:59 AM, Dirk Cornel.... <moolma_dc@mtn.co.za>wrote: > "- Be careful when looking at each process memory because" > > I can still use this to get a general overall view, correct ? It should be > accurate enough to still show me which applications are more memory > intensive > than others, even if not 100% accurate. > > > -----Original Message----- > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > > Fernando Nunes > > Sent: Wednesday, 31 August 2011 11:46 AM > > To: ids@iiug.org > > Subject: Re: CPU load balancing - Tru64 [24780] > > > > I'm not sure if I'm missing some peace of context (I noticed there are > > other > > messages from you). > > In any case, that could happen for example if you had only 3 CPUVPs on > > an > > Informix instance. > > The software would not be able to saturate more than 3 CPUs. I think > > you > > would see that the set of 3 would not necessarily be the same 3 (e.g. > > cpu0, > > cpu1, cpu2) every time. But this could depend on CPU affinity. > > > > When you have free CPU and things are slow you should always consider > > other > > sources of contention (like I/O, swap etc.) > > > > Regarding your questions on another post: > > > > - Be careful when looking at each process memory because: > > - The processes that use shared memory cannot be added (since the > > memory > > they are referencing is shared among them). I assume this is clear, but > > I've > > seen too many people making this basic mistake > > - The RSS can (and probably will) include memory that is shared among > > the > > processes. I'm not talking about "shared memory" segments but things > > like , > > code segments etc. > > - On Tru64 there is a parameter called something like "swap eager" (see > > /etc/sysconfigtab) that defines how it uses swap. If it's set to "1" > > every > > memory allocation will mean the same ammount of swap reservation. If > > your > > swap space is low you can consume it in "reserverd". This may never be > > used > > but if the swap space is exhausted (even for reserved space) the OS may > > start killing processes. > > - I've seen performance problems that were attributed to advfs issues, > > although I never had a proper explanation. But it would happen when you > > have > > large file systems full of small files (many, many thousands or > > millions). > > I've seen machines that have slow response, although the CPU usage is > > low > > - Having too many processes (thounsands) apparently also can cause > > problems. There are settings that change the process table sizes, but I > > never changed them. Not sure if it helps. This translated in slow > > response > > and commands like "ps" took very long time and consumed too many CPUs > > > > If you want to share your concerns, maybe we can be more useful. If you > > already did it, I probably missed it, so I apologize. > > > > Regards. > > > On Wed, Aug 31, 2011 at 10:16 AM, Dirk Cornel.... > > <moolma_dc@mtn.co.za>wrote: > > > > > Another Unix question (also on Tru64): > > > > > > I used the "cpuinfo" command to monitor each CPU individualy. But the > > > picture > > > I get does not make 100% sense to me. > > > > > > Let's say there are 10 cpu's. What I am seeing is, 3 of those cpu's > > are > > > very > > > busy, and the other 7 are idle. Should Unix not be load balancing > > (context > > > switching) ? > > > > > > Because this now means, when I run a sar, I see 85% idle time, which > > means > > > in > > > total the server is not very busy, but 3 of the cpu's are running > > flat out. > > > > > > This would cause problems, would it not ? > > > > > > Dirk > > > > > > ________________________________ > > > NOTE: This e-mail message is subject to the MTN Group disclaimer see > > > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > > > > > > > > > > > *********************************************************************** > > ******** > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > > > > -- > > Fernando Nunes > > Portugal > > > > http://informix-technology.blogspot.com > > My email works... but I don't check it frequently... > > > > --0016364d1f8d45020104abc9ff72 > > > > > > *********************************************************************** > > ******** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > NOTE: This e-mail message is subject to the MTN Group disclaimer see > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --002215974c5ea7e80a04abcd063f

This sounds like a process aging problem. TRU64 may be aggressive about reducing the run time priority of long running processes. Your app server applications sound like they start up and keep running forever which would bring them to the attention of the OS's priority aging algorithms. One of the results of this aging is that long running processes get fewer time slices that are longer then younger processes. This will manifest as a process that seems to stay on a single CPU/core for a long time before migrating to another CPU. This will definitely result in pegging that CPU to near 100% if the process does not have to wait for IO very often. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Wed, Aug 31, 2011 at 6:17 AM, Dirk Cornel.... <moolma_dc@mtn.co.za>wrote: > > -----Original Message----- > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > > Obnoxio The Clown > > Sent: Wednesday, 31 August 2011 12:02 PM > > To: ids@iiug.org > > Subject: Re: CPU load balancing - Tru64 [24783] > > > > On 31/08/2011 10:56, Dirk Cornel.... wrote: > > > " I've seen machines that have slow response, although the CPU usage > > is low" > > > > > > Yes, this is what we are experiencing. So my first check were the > > cpu's. I > > > also looked at I/O (the I/O was good, no problems there), and the > > memory was > > > good (no swapping taking place on the server). > > > > > > So now I start to wonder, is this an inherent problem on Tru64 > > perhaps ? > > ..... > > > the cpu's not load balancing properly ? > > > > > > There could be many valid explanations. This server does not only run > > > Informix. Maybe a process binds to a cpu, and cannot be split > > further, into > > > multiple threads for multiple cpu's to be utilised. > > > > What does onstat -C show? > > > > -- > > Cheers, > > Obnoxio The Clown > > The problem is, this is an Apps server - it only runs 4ge's, and Java > applications - no database. And all the other servers are fine. So I think > the > problem is limited to this server. > > More frustrating is, the server would show 85% plus idle time, but they are > complaining it is slow. We reboot only this server, and everything is fine > again. > This is why I started digging into processes now, seeing which are using > the > most memory on this server, etc. > > The CPU scenario I described, is also strange to me. What if the process of > the user that is complaining, is running on one of the cpu's that are busy, > while the server in general still has idle time ? > > NOTE: This e-mail message is subject to the MTN Group disclaimer see > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --002215974c5e4e6cc304abcd4d35

On 31/08/2011 11:21, noreply@corsicaferries.com wrote: > Bonjour, > je serai absent du 22/08/2011 au 07/09/2011 > je vous repondrai des que possible. > > cordialement, > > Franck ABELLI I am going to hunt you down and kill you. You fucking cunt. -- Cheers, Obnoxio The Clown http://obotheclown.blogspot.com I will now proceed to pleasure myself with this fish.

Take the fish with you. ;-) j. On Aug 31, 2011, at 3:58 PM, Obnoxio The Clown wrote: > On 31/08/2011 11:21, noreply@corsicaferries.com wrote:=20 >> Bonjour,=20 >> je serai absent du 22/08/2011 au 07/09/2011=20 >> je vous repondrai des que possible.=20 >>=20 >> cordialement,=20 >>=20 >> Franck ABELLI=20 >=20 > I am going to hunt you down and kill you. You fucking cunt.=20 >=20 > --=20 > Cheers,=20 > Obnoxio The Clown=20 >=20 > http://obotheclown.blogspot.com=20 > I will now proceed to pleasure myself with this fish.=20 >=20 >=20 > = **************************************************************************= *****=20 > Forum Note: Use "Reply" to post a response in the discussion forum.=20= >=20

Thanks Art. So how would I determine the applications using the most memory ? > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > Art Kagel > Sent: Wednesday, 31 August 2011 03:23 PM > To: ids@iiug.org > Subject: Re: CPU load balancing - Tru64 [24788] > > Fernando's point is that you cannot add up the memory usage of the > Informix > oninit processes (or other processes that share memory with each other) > to > determine if you are overusing memory because that counts the.shared > segments mulitiple times. > > Art > > Art S. Kagel > Advanced DataTools (www.advancedatatools.com) > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own > opinions and > do not reflect on my employer, Advanced DataTools, the IIUG, nor any > other > organization with which I am associated either explicitly, implicitly, > or by > inference. Neither do those opinions reflect those of other individuals > affiliated with any entity with which I am affiliated nor those of the > entities themselves. > On Wed, Aug 31, 2011 at 5:59 AM, Dirk Cornel.... > <moolma_dc@mtn.co.za>wrote: > > > "- Be careful when looking at each process memory because" > > > > I can still use this to get a general overall view, correct ? It > should be > > accurate enough to still show me which applications are more memory > > intensive > > than others, even if not 100% accurate. > > > > > -----Original Message----- > > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf > Of > > > Fernando Nunes > > > Sent: Wednesday, 31 August 2011 11:46 AM > > > To: ids@iiug.org > > > Subject: Re: CPU load balancing - Tru64 [24780] > > > > > > I'm not sure if I'm missing some peace of context (I noticed there > are > > > other > > > messages from you). > > > In any case, that could happen for example if you had only 3 CPUVPs > on > > > an > > > Informix instance. > > > The software would not be able to saturate more than 3 CPUs. I > think > > > you > > > would see that the set of 3 would not necessarily be the same 3 > (e.g. > > > cpu0, > > > cpu1, cpu2) every time. But this could depend on CPU affinity. > > > > > > When you have free CPU and things are slow you should always > consider > > > other > > > sources of contention (like I/O, swap etc.) > > > > > > Regarding your questions on another post: > > > > > > - Be careful when looking at each process memory because: > > > - The processes that use shared memory cannot be added (since the > > > memory > > > they are referencing is shared among them). I assume this is clear, > but > > > I've > > > seen too many people making this basic mistake > > > - The RSS can (and probably will) include memory that is shared > among > > > the > > > processes. I'm not talking about "shared memory" segments but > things > > > like , > > > code segments etc. > > > - On Tru64 there is a parameter called something like "swap eager" > (see > > > /etc/sysconfigtab) that defines how it uses swap. If it's set to > "1" > > > every > > > memory allocation will mean the same ammount of swap reservation. > If > > > your > > > swap space is low you can consume it in "reserverd". This may never > be > > > used > > > but if the swap space is exhausted (even for reserved space) the OS > may > > > start killing processes. > > > - I've seen performance problems that were attributed to advfs > issues, > > > although I never had a proper explanation. But it would happen when > you > > > have > > > large file systems full of small files (many, many thousands or > > > millions). > > > I've seen machines that have slow response, although the CPU usage > is > > > low > > > - Having too many processes (thounsands) apparently also can cause > > > problems. There are settings that change the process table sizes, > but I > > > never changed them. Not sure if it helps. This translated in slow > > > response > > > and commands like "ps" took very long time and consumed too many > CPUs > > > > > > If you want to share your concerns, maybe we can be more useful. If > you > > > already did it, I probably missed it, so I apologize. > > > > > > Regards. > > > > > On Wed, Aug 31, 2011 at 10:16 AM, Dirk Cornel.... > > > <moolma_dc@mtn.co.za>wrote: > > > > > > > Another Unix question (also on Tru64): > > > > > > > > I used the "cpuinfo" command to monitor each CPU individualy. But > the > > > > picture > > > > I get does not make 100% sense to me. > > > > > > > > Let's say there are 10 cpu's. What I am seeing is, 3 of those > cpu's > > > are > > > > very > > > > busy, and the other 7 are idle. Should Unix not be load balancing > > > (context > > > > switching) ? > > > > > > > > Because this now means, when I run a sar, I see 85% idle time, > which > > > means > > > > in > > > > total the server is not very busy, but 3 of the cpu's are running > > > flat out. > > > > > > > > This would cause problems, would it not ? > > > > > > > > Dirk > > > > > > > > ________________________________ > > > > NOTE: This e-mail message is subject to the MTN Group disclaimer > see > > > > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > > > > > > > > > > > > > > > > > *********************************************************************** > > > ******** > > > > Forum Note: Use "Reply" to post a response in the discussion > forum. > > > > > > > > > > > > > > -- > > > Fernando Nunes > > > Portugal > > > > > > http://informix-technology.blogspot.com > > > My email works... but I don't check it frequently... > > > > > > --0016364d1f8d45020104abc9ff72 > > > > > > > > > > *********************************************************************** > > > ******** > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > NOTE: This e-mail message is subject to the MTN Group disclaimer see > > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > > > > > > *********************************************************************** > ******** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > --002215974c5ea7e80a04abcd063f > > > *********************************************************************** > ******** > Forum Note: Use "Reply" to post a response in the discussion forum. NOTE: This e-mail message is subject to the MTN Group disclaimer see http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx

I have some doubts - let me explain. The server in general runs ok, until it is time for our .... let's call them, Invoice Runs. So we have a team of people who do this. They start an invoice run, and after 2 or 3 hours they phone us and complain that the server is slow. So the processes they started, are new (fresh) processes (4ge's). Could it still be ageing (on a Unix level perhaps) - something more technical I am missing with regards to ageing ? Dirk > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > Art Kagel > Sent: Wednesday, 31 August 2011 03:43 PM > To: ids@iiug.org > Subject: Re: CPU load balancing - Tru64 [24789] > > This sounds like a process aging problem. TRU64 may be aggressive about > reducing the run time priority of long running processes. Your app > server > applications sound like they start up and keep running forever which > would > bring them to the attention of the OS's priority aging algorithms. One > of > the results of this aging is that long running processes get fewer time > slices that are longer then younger processes. This will manifest as a > process that seems to stay on a single CPU/core for a long time before > migrating to another CPU. This will definitely result in pegging that > CPU > to near 100% if the process does not have to wait for IO very often. > > Art > > Art S. Kagel > Advanced DataTools (www.advancedatatools.com) > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own > opinions and > do not reflect on my employer, Advanced DataTools, the IIUG, nor any > other > organization with which I am associated either explicitly, implicitly, > or by > inference. Neither do those opinions reflect those of other individuals > affiliated with any entity with which I am affiliated nor those of the > entities themselves. > On Wed, Aug 31, 2011 at 6:17 AM, Dirk Cornel.... > <moolma_dc@mtn.co.za>wrote: > > > > -----Original Message----- > > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf > Of > > > Obnoxio The Clown > > > Sent: Wednesday, 31 August 2011 12:02 PM > > > To: ids@iiug.org > > > Subject: Re: CPU load balancing - Tru64 [24783] > > > > > > On 31/08/2011 10:56, Dirk Cornel.... wrote: > > > > " I've seen machines that have slow response, although the CPU > usage > > > is low" > > > > > > > > Yes, this is what we are experiencing. So my first check were the > > > cpu's. I > > > > also looked at I/O (the I/O was good, no problems there), and the > > > memory was > > > > good (no swapping taking place on the server). > > > > > > > > So now I start to wonder, is this an inherent problem on Tru64 > > > perhaps ? > > > ..... > > > > the cpu's not load balancing properly ? > > > > > > > > There could be many valid explanations. This server does not only > run > > > > Informix. Maybe a process binds to a cpu, and cannot be split > > > further, into > > > > multiple threads for multiple cpu's to be utilised. > > > > > > What does onstat -C show? > > > > > > -- > > > Cheers, > > > Obnoxio The Clown > > > > The problem is, this is an Apps server - it only runs 4ge's, and Java > > applications - no database. And all the other servers are fine. So I > think > > the > > problem is limited to this server. > > > > More frustrating is, the server would show 85% plus idle time, but > they are > > complaining it is slow. We reboot only this server, and everything is > fine > > again. > > This is why I started digging into processes now, seeing which are > using > > the > > most memory on this server, etc. > > > > The CPU scenario I described, is also strange to me. What if the > process of > > the user that is complaining, is running on one of the cpu's that are > busy, > > while the server in general still has idle time ? > > > > NOTE: This e-mail message is subject to the MTN Group disclaimer see > > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > > > > > > *********************************************************************** > ******** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > --002215974c5e4e6cc304abcd4d35 > > > *********************************************************************** > ******** > Forum Note: Use "Reply" to post a response in the discussion forum. NOTE: This e-mail message is subject to the MTN Group disclaimer see http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx

I know I've been off this list for quite some time, so the rules might have changed, but ...... do you have to be so obscene ? Dirk > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > Obnoxio The Clown > Sent: Wednesday, 31 August 2011 09:59 PM > To: ids@iiug.org > Subject: Re: Re: CPU load balancing - Tru64 [24806] > > On 31/08/2011 11:21, noreply@corsicaferries.com wrote: > > Bonjour, > > je serai absent du 22/08/2011 au 07/09/2011 > > je vous repondrai des que possible. > > > > cordialement, > > > > Franck ABELLI > > I am going to hunt you down and kill you. You fucking cunt. > > -- > Cheers, > Obnoxio The Clown > > http://obotheclown.blogspot.com > I will now proceed to pleasure myself with this fish. > > > *********************************************************************** > ******** > Forum Note: Use "Reply" to post a response in the discussion forum. NOTE: This e-mail message is subject to the MTN Group disclaimer see http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx

Sorry, I forgot to add: they complain the server is slow, we reboot the server, they restart the invoice run, and then everything is ok again. > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > Dirk Cornel.... > Sent: Thursday, 01 September 2011 11:26 AM > To: ids@iiug.org > Subject: RE: CPU load balancing - Tru64 [24817] > > I have some doubts - let me explain. The server in general runs ok, > until it > is time for our .... let's call them, Invoice Runs. > > So we have a team of people who do this. They start an invoice run, and > after > 2 or 3 hours they phone us and complain that the server is slow. So the > processes they started, are new (fresh) processes (4ge's). > > Could it still be ageing (on a Unix level perhaps) - something more > technical > I am missing with regards to ageing ? > > Dirk > > -----Original Message----- > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > > Art Kagel > > Sent: Wednesday, 31 August 2011 03:43 PM > > To: ids@iiug.org > > Subject: Re: CPU load balancing - Tru64 [24789] > > > > This sounds like a process aging problem. TRU64 may be aggressive > about > > reducing the run time priority of long running processes. Your app > > server > > applications sound like they start up and keep running forever which > > would > > bring them to the attention of the OS's priority aging algorithms. > One > > of > > the results of this aging is that long running processes get fewer > time > > slices that are longer then younger processes. This will manifest as > a > > process that seems to stay on a single CPU/core for a long time > before > > migrating to another CPU. This will definitely result in pegging that > > CPU > > to near 100% if the process does not have to wait for IO very often. > > > > Art > > > > Art S. Kagel > > Advanced DataTools (www.advancedatatools.com) > > Blog: http://informix-myview.blogspot.com/ > > > > Disclaimer: Please keep in mind that my own opinions are my own > > opinions and > > do not reflect on my employer, Advanced DataTools, the IIUG, nor any > > other > > organization with which I am associated either explicitly, > implicitly, > > or by > > inference. Neither do those opinions reflect those of other > individuals > > affiliated with any entity with which I am affiliated nor those of > the > > entities themselves. > > > On Wed, Aug 31, 2011 at 6:17 AM, Dirk Cornel.... > > <moolma_dc@mtn.co.za>wrote: > > > > > > -----Original Message----- > > > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On > Behalf > > Of > > > > Obnoxio The Clown > > > > Sent: Wednesday, 31 August 2011 12:02 PM > > > > To: ids@iiug.org > > > > Subject: Re: CPU load balancing - Tru64 [24783] > > > > > > > > On 31/08/2011 10:56, Dirk Cornel.... wrote: > > > > > " I've seen machines that have slow response, although the CPU > > usage > > > > is low" > > > > > > > > > > Yes, this is what we are experiencing. So my first check were > the > > > > cpu's. I > > > > > also looked at I/O (the I/O was good, no problems there), and > the > > > > memory was > > > > > good (no swapping taking place on the server). > > > > > > > > > > So now I start to wonder, is this an inherent problem on Tru64 > > > > perhaps ? > > > > ..... > > > > > the cpu's not load balancing properly ? > > > > > > > > > > There could be many valid explanations. This server does not > only > > run > > > > > Informix. Maybe a process binds to a cpu, and cannot be split > > > > further, into > > > > > multiple threads for multiple cpu's to be utilised. > > > > > > > > What does onstat -C show? > > > > > > > > -- > > > > Cheers, > > > > Obnoxio The Clown > > > > > > The problem is, this is an Apps server - it only runs 4ge's, and > Java > > > applications - no database. And all the other servers are fine. So > I > > think > > > the > > > problem is limited to this server. > > > > > > More frustrating is, the server would show 85% plus idle time, but > > they are > > > complaining it is slow. We reboot only this server, and everything > is > > fine > > > again. > > > This is why I started digging into processes now, seeing which are > > using > > > the > > > most memory on this server, etc. > > > > > > The CPU scenario I described, is also strange to me. What if the > > process of > > > the user that is complaining, is running on one of the cpu's that > are > > busy, > > > while the server in general still has idle time ? > > > > > > NOTE: This e-mail message is subject to the MTN Group disclaimer > see > > > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > > > > > > > > > > > > *********************************************************************** > > ******** > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > > > > --002215974c5e4e6cc304abcd4d35 > > > > > > > *********************************************************************** > > ******** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > NOTE: This e-mail message is subject to the MTN Group disclaimer see > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > *********************************************************************** > ******** > Forum Note: Use "Reply" to post a response in the discussion forum. NOTE: This e-mail message is subject to the MTN Group disclaimer see http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx

On 01/09/2011 10:32, Dirk Cornel.... wrote: > I know I've been off this list for quite some time, so the rules might have > changed, but ...... do you have to be so obscene ? The rules haven't changed and yes, I do. :o) -- Cheers, Obnoxio The Clown http://obotheclown.blogspot.com I will now proceed to pleasure myself with this fish.

Questions: 1- Are these processes interactive? 2- Why they say it's slow? If it's a GUI program are they seeing slow response? If it's a batch program do they have any progress indication? 3- At that time does the server have free resources (CPU, memory, I/O)? Assuming yes, can't they be confusing "server slow" with "slow database response"? 4- I suppose the database is not local, correct? Regards. On Thu, Sep 1, 2011 at 10:26 AM, Dirk Cornel.... <moolma_dc@mtn.co.za>wrote: > I have some doubts - let me explain. The server in general runs ok, until > it > is time for our .... let's call them, Invoice Runs. > > So we have a team of people who do this. They start an invoice run, and > after > 2 or 3 hours they phone us and complain that the server is slow. So the > processes they started, are new (fresh) processes (4ge's). > > Could it still be ageing (on a Unix level perhaps) - something more > technical > I am missing with regards to ageing ? > > Dirk > > > -----Original Message----- > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > > Art Kagel > > Sent: Wednesday, 31 August 2011 03:43 PM > > To: ids@iiug.org > > Subject: Re: CPU load balancing - Tru64 [24789] > > > > This sounds like a process aging problem. TRU64 may be aggressive about > > reducing the run time priority of long running processes. Your app > > server > > applications sound like they start up and keep running forever which > > would > > bring them to the attention of the OS's priority aging algorithms. One > > of > > the results of this aging is that long running processes get fewer time > > slices that are longer then younger processes. This will manifest as a > > process that seems to stay on a single CPU/core for a long time before > > migrating to another CPU. This will definitely result in pegging that > > CPU > > to near 100% if the process does not have to wait for IO very often. > > > > Art > > > > Art S. Kagel > > Advanced DataTools (www.advancedatatools.com) > > Blog: http://informix-myview.blogspot.com/ > > > > Disclaimer: Please keep in mind that my own opinions are my own > > opinions and > > do not reflect on my employer, Advanced DataTools, the IIUG, nor any > > other > > organization with which I am associated either explicitly, implicitly, > > or by > > inference. Neither do those opinions reflect those of other individuals > > affiliated with any entity with which I am affiliated nor those of the > > entities themselves. > > > On Wed, Aug 31, 2011 at 6:17 AM, Dirk Cornel.... > > <moolma_dc@mtn.co.za>wrote: > > > > > > -----Original Message----- > > > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf > > Of > > > > Obnoxio The Clown > > > > Sent: Wednesday, 31 August 2011 12:02 PM > > > > To: ids@iiug.org > > > > Subject: Re: CPU load balancing - Tru64 [24783] > > > > > > > > On 31/08/2011 10:56, Dirk Cornel.... wrote: > > > > > " I've seen machines that have slow response, although the CPU > > usage > > > > is low" > > > > > > > > > > Yes, this is what we are experiencing. So my first check were the > > > > cpu's. I > > > > > also looked at I/O (the I/O was good, no problems there), and the > > > > memory was > > > > > good (no swapping taking place on the server). > > > > > > > > > > So now I start to wonder, is this an inherent problem on Tru64 > > > > perhaps ? > > > > ..... > > > > > the cpu's not load balancing properly ? > > > > > > > > > > There could be many valid explanations. This server does not only > > run > > > > > Informix. Maybe a process binds to a cpu, and cannot be split > > > > further, into > > > > > multiple threads for multiple cpu's to be utilised. > > > > > > > > What does onstat -C show? > > > > > > > > -- > > > > Cheers, > > > > Obnoxio The Clown > > > > > > The problem is, this is an Apps server - it only runs 4ge's, and Java > > > applications - no database. And all the other servers are fine. So I > > think > > > the > > > problem is limited to this server. > > > > > > More frustrating is, the server would show 85% plus idle time, but > > they are > > > complaining it is slow. We reboot only this server, and everything is > > fine > > > again. > > > This is why I started digging into processes now, seeing which are > > using > > > the > > > most memory on this server, etc. > > > > > > The CPU scenario I described, is also strange to me. What if the > > process of > > > the user that is complaining, is running on one of the cpu's that are > > busy, > > > while the server in general still has idle time ? > > > > > > NOTE: This e-mail message is subject to the MTN Group disclaimer see > > > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > > > > > > > > > > > *********************************************************************** > > ******** > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > > > > --002215974c5e4e6cc304abcd4d35 > > > > > > *********************************************************************** > > ******** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > NOTE: This e-mail message is subject to the MTN Group disclaimer see > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --002215b034662db86904abdefc1e

There are the rules... and then there is OTC... He sure doesn't have to be obscene, but if he wasn't someone would ask "who are you and what have you done with OTC?!". Note however that usually he is obscene in extreme situations... You may notice that in other situations he can actually be helpful. I'd say many times he verbalizes what's on other people minds.... But these people have some filters. His filters are not working properly Regards On Thu, Sep 1, 2011 at 10:32 AM, Dirk Cornel.... <moolma_dc@mtn.co.za>wrote: > I know I've been off this list for quite some time, so the rules might have > changed, but ...... do you have to be so obscene ? > > Dirk > > > -----Original Message----- > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > > Obnoxio The Clown > > Sent: Wednesday, 31 August 2011 09:59 PM > > To: ids@iiug.org > > Subject: Re: Re: CPU load balancing - Tru64 [24806] > > > > On 31/08/2011 11:21, noreply@corsicaferries.com wrote: > > > Bonjour, > > > je serai absent du 22/08/2011 au 07/09/2011 > > > je vous repondrai des que possible. > > > > > > cordialement, > > > > > > Franck ABELLI > > > > I am going to hunt you down and kill you. You fucking cunt. > > > > -- > > Cheers, > > Obnoxio The Clown > > > > http://obotheclown.blogspot.com > > I will now proceed to pleasure myself with this fish. > > > > > > *********************************************************************** > > ******** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > NOTE: This e-mail message is subject to the MTN Group disclaimer see > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --20cf3002555ac0d0c204abdf0b54

Correct, the database is remote, on another server. We have other application servers also connecting to the same database, and those applications seem to be fine when this team starts complaining about this specific server. I am not sure about the front end (screen), but they monitor by number of records (customers) processed. This process has existed for years, so they go on their averages for the previous runs, and this is how they identify that the server becomes slow (by comparing to previous runs). PS. All written in 4GL (it is not GUI). It runs in batch form, in the sense of it will process a range of customers in the database (generate invoices). Yes, we log in, and from our point of view the server is ok in terms of memory, io, cpu ...... for example 85% or more idle time, etc. But they complain that it is slow. The problem seems to be localised to this server. > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > Fernando Nunes > Sent: Thursday, 01 September 2011 12:49 PM > To: ids@iiug.org > Subject: Re: CPU load balancing - Tru64 [24821] > > Questions: > 1- Are these processes interactive? > 2- Why they say it's slow? If it's a GUI program are they seeing slow > response? If it's a batch program do they have any progress indication? > 3- At that time does the server have free resources (CPU, memory, I/O)? > Assuming yes, can't they be confusing "server slow" with "slow database > response"? > 4- I suppose the database is not local, correct? > > Regards. > On Thu, Sep 1, 2011 at 10:26 AM, Dirk Cornel.... > <moolma_dc@mtn.co.za>wrote: > > > I have some doubts - let me explain. The server in general runs ok, > until > > it > > is time for our .... let's call them, Invoice Runs. > > > > So we have a team of people who do this. They start an invoice run, > and > > after > > 2 or 3 hours they phone us and complain that the server is slow. So > the > > processes they started, are new (fresh) processes (4ge's). > > > > Could it still be ageing (on a Unix level perhaps) - something more > > technical > > I am missing with regards to ageing ? > > > > Dirk > > > > > -----Original Message----- > > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf > Of > > > Art Kagel > > > Sent: Wednesday, 31 August 2011 03:43 PM > > > To: ids@iiug.org > > > Subject: Re: CPU load balancing - Tru64 [24789] > > > > > > This sounds like a process aging problem. TRU64 may be aggressive > about > > > reducing the run time priority of long running processes. Your app > > > server > > > applications sound like they start up and keep running forever > which > > > would > > > bring them to the attention of the OS's priority aging algorithms. > One > > > of > > > the results of this aging is that long running processes get fewer > time > > > slices that are longer then younger processes. This will manifest > as a > > > process that seems to stay on a single CPU/core for a long time > before > > > migrating to another CPU. This will definitely result in pegging > that > > > CPU > > > to near 100% if the process does not have to wait for IO very > often. > > > > > > Art > > > > > > Art S. Kagel > > > Advanced DataTools (www.advancedatatools.com) > > > Blog: http://informix-myview.blogspot.com/ > > > > > > Disclaimer: Please keep in mind that my own opinions are my own > > > opinions and > > > do not reflect on my employer, Advanced DataTools, the IIUG, nor > any > > > other > > > organization with which I am associated either explicitly, > implicitly, > > > or by > > > inference. Neither do those opinions reflect those of other > individuals > > > affiliated with any entity with which I am affiliated nor those of > the > > > entities themselves. > > > > > On Wed, Aug 31, 2011 at 6:17 AM, Dirk Cornel.... > > > <moolma_dc@mtn.co.za>wrote: > > > > > > > > -----Original Message----- > > > > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On > Behalf > > > Of > > > > > Obnoxio The Clown > > > > > Sent: Wednesday, 31 August 2011 12:02 PM > > > > > To: ids@iiug.org > > > > > Subject: Re: CPU load balancing - Tru64 [24783] > > > > > > > > > > On 31/08/2011 10:56, Dirk Cornel.... wrote: > > > > > > " I've seen machines that have slow response, although the > CPU > > > usage > > > > > is low" > > > > > > > > > > > > Yes, this is what we are experiencing. So my first check were > the > > > > > cpu's. I > > > > > > also looked at I/O (the I/O was good, no problems there), and > the > > > > > memory was > > > > > > good (no swapping taking place on the server). > > > > > > > > > > > > So now I start to wonder, is this an inherent problem on > Tru64 > > > > > perhaps ? > > > > > ..... > > > > > > the cpu's not load balancing properly ? > > > > > > > > > > > > There could be many valid explanations. This server does not > only > > > run > > > > > > Informix. Maybe a process binds to a cpu, and cannot be split > > > > > further, into > > > > > > multiple threads for multiple cpu's to be utilised. > > > > > > > > > > What does onstat -C show? > > > > > > > > > > -- > > > > > Cheers, > > > > > Obnoxio The Clown > > > > > > > > The problem is, this is an Apps server - it only runs 4ge's, and > Java > > > > applications - no database. And all the other servers are fine. > So I > > > think > > > > the > > > > problem is limited to this server. > > > > > > > > More frustrating is, the server would show 85% plus idle time, > but > > > they are > > > > complaining it is slow. We reboot only this server, and > everything is > > > fine > > > > again. > > > > This is why I started digging into processes now, seeing which > are > > > using > > > > the > > > > most memory on this server, etc. > > > > > > > > The CPU scenario I described, is also strange to me. What if the > > > process of > > > > the user that is complaining, is running on one of the cpu's that > are > > > busy, > > > > while the server in general still has idle time ? > > > > > > > > NOTE: This e-mail message is subject to the MTN Group disclaimer > see > > > > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > > > > > > > > > > > > > > > > > *********************************************************************** > > > ******** > > > > Forum Note: Use "Reply" to post a response in the discussion > forum. > > > > > > > > > > > > > > --002215974c5e4e6cc304abcd4d35 > > > > > > > > > > *********************************************************************** > > > ******** > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > NOTE: This e-mail message is subject to the MTN Group disclaimer see > > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > > > > > > ***********************************************************************@@

They aer fresh, but the Informix oninit processes are not fresh. Do you have NOAGE set in the ONCONFIG file for the engine? Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Sep 1, 2011 at 5:26 AM, Dirk Cornel.... <moolma_dc@mtn.co.za> wrote: > I have some doubts - let me explain. The server in general runs ok, until > it > is time for our .... let's call them, Invoice Runs. > > So we have a team of people who do this. They start an invoice run, and > after > 2 or 3 hours they phone us and complain that the server is slow. So the > processes they started, are new (fresh) processes (4ge's). > > Could it still be ageing (on a Unix level perhaps) - something more > technical > I am missing with regards to ageing ? > > Dirk > > > -----Original Message----- > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > > Art Kagel > > Sent: Wednesday, 31 August 2011 03:43 PM > > To: ids@iiug.org > > Subject: Re: CPU load balancing - Tru64 [24789] > > > > This sounds like a process aging problem. TRU64 may be aggressive about > > reducing the run time priority of long running processes. Your app > > server > > applications sound like they start up and keep running forever which > > would > > bring them to the attention of the OS's priority aging algorithms. One > > of > > the results of this aging is that long running processes get fewer time > > slices that are longer then younger processes. This will manifest as a > > process that seems to stay on a single CPU/core for a long time before > > migrating to another CPU. This will definitely result in pegging that > > CPU > > to near 100% if the process does not have to wait for IO very often. > > > > Art > > > > Art S. Kagel > > Advanced DataTools (www.advancedatatools.com) > > Blog: http://informix-myview.blogspot.com/ > > > > Disclaimer: Please keep in mind that my own opinions are my own > > opinions and > > do not reflect on my employer, Advanced DataTools, the IIUG, nor any > > other > > organization with which I am associated either explicitly, implicitly, > > or by > > inference. Neither do those opinions reflect those of other individuals > > affiliated with any entity with which I am affiliated nor those of the > > entities themselves. > > > On Wed, Aug 31, 2011 at 6:17 AM, Dirk Cornel.... > > <moolma_dc@mtn.co.za>wrote: > > > > > > -----Original Message----- > > > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf > > Of > > > > Obnoxio The Clown > > > > Sent: Wednesday, 31 August 2011 12:02 PM > > > > To: ids@iiug.org > > > > Subject: Re: CPU load balancing - Tru64 [24783] > > > > > > > > On 31/08/2011 10:56, Dirk Cornel.... wrote: > > > > > " I've seen machines that have slow response, although the CPU > > usage > > > > is low" > > > > > > > > > > Yes, this is what we are experiencing. So my first check were the > > > > cpu's. I > > > > > also looked at I/O (the I/O was good, no problems there), and the > > > > memory was > > > > > good (no swapping taking place on the server). > > > > > > > > > > So now I start to wonder, is this an inherent problem on Tru64 > > > > perhaps ? > > > > ..... > > > > > the cpu's not load balancing properly ? > > > > > > > > > > There could be many valid explanations. This server does not only > > run > > > > > Informix. Maybe a process binds to a cpu, and cannot be split > > > > further, into > > > > > multiple threads for multiple cpu's to be utilised. > > > > > > > > What does onstat -C show? > > > > > > > > -- > > > > Cheers, > > > > Obnoxio The Clown > > > > > > The problem is, this is an Apps server - it only runs 4ge's, and Java > > > applications - no database. And all the other servers are fine. So I > > think > > > the > > > problem is limited to this server. > > > > > > More frustrating is, the server would show 85% plus idle time, but > > they are > > > complaining it is slow. We reboot only this server, and everything is > > fine > > > again. > > > This is why I started digging into processes now, seeing which are > > using > > > the > > > most memory on this server, etc. > > > > > > The CPU scenario I described, is also strange to me. What if the > > process of > > > the user that is complaining, is running on one of the cpu's that are > > busy, > > > while the server in general still has idle time ? > > > > > > NOTE: This e-mail message is subject to the MTN Group disclaimer see > > > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > > > > > > > > > > > *********************************************************************** > > ******** > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > > > > --002215974c5e4e6cc304abcd4d35 > > > > > > *********************************************************************** > > ******** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > NOTE: This e-mail message is subject to the MTN Group disclaimer see > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --90e6ba6e8e8a2f456704abe5f16e

Right, rebooting the server makes the oninit processes "young" again so not aged! Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Sep 1, 2011 at 5:34 AM, Dirk Cornel.... <moolma_dc@mtn.co.za> wrote: > Sorry, I forgot to add: they complain the server is slow, we reboot the > server, they restart the invoice run, and then everything is ok again. > > > -----Original Message----- > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > > Dirk Cornel.... > > Sent: Thursday, 01 September 2011 11:26 AM > > To: ids@iiug.org > > Subject: RE: CPU load balancing - Tru64 [24817] > > > > I have some doubts - let me explain. The server in general runs ok, > > until it > > is time for our .... let's call them, Invoice Runs. > > > > So we have a team of people who do this. They start an invoice run, and > > after > > 2 or 3 hours they phone us and complain that the server is slow. So the > > processes they started, are new (fresh) processes (4ge's). > > > > Could it still be ageing (on a Unix level perhaps) - something more > > technical > > I am missing with regards to ageing ? > > > > Dirk > > > > -----Original Message----- > > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > > > Art Kagel > > > Sent: Wednesday, 31 August 2011 03:43 PM > > > To: ids@iiug.org > > > Subject: Re: CPU load balancing - Tru64 [24789] > > > > > > This sounds like a process aging problem. TRU64 may be aggressive > > about > > > reducing the run time priority of long running processes. Your app > > > server > > > applications sound like they start up and keep running forever which > > > would > > > bring them to the attention of the OS's priority aging algorithms. > > One > > > of > > > the results of this aging is that long running processes get fewer > > time > > > slices that are longer then younger processes. This will manifest as > > a > > > process that seems to stay on a single CPU/core for a long time > > before > > > migrating to another CPU. This will definitely result in pegging that > > > CPU > > > to near 100% if the process does not have to wait for IO very often. > > > > > > Art > > > > > > Art S. Kagel > > > Advanced DataTools (www.advancedatatools.com) > > > Blog: http://informix-myview.blogspot.com/ > > > > > > Disclaimer: Please keep in mind that my own opinions are my own > > > opinions and > > > do not reflect on my employer, Advanced DataTools, the IIUG, nor any > > > other > > > organization with which I am associated either explicitly, > > implicitly, > > > or by > > > inference. Neither do those opinions reflect those of other > > individuals > > > affiliated with any entity with which I am affiliated nor those of > > the > > > entities themselves. > > > > > On Wed, Aug 31, 2011 at 6:17 AM, Dirk Cornel.... > > > <moolma_dc@mtn.co.za>wrote: > > > > > > > > -----Original Message----- > > > > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On > > Behalf > > > Of > > > > > Obnoxio The Clown > > > > > Sent: Wednesday, 31 August 2011 12:02 PM > > > > > To: ids@iiug.org > > > > > Subject: Re: CPU load balancing - Tru64 [24783] > > > > > > > > > > On 31/08/2011 10:56, Dirk Cornel.... wrote: > > > > > > " I've seen machines that have slow response, although the CPU > > > usage > > > > > is low" > > > > > > > > > > > > Yes, this is what we are experiencing. So my first check were > > the > > > > > cpu's. I > > > > > > also looked at I/O (the I/O was good, no problems there), and > > the > > > > > memory was > > > > > > good (no swapping taking place on the server). > > > > > > > > > > > > So now I start to wonder, is this an inherent problem on Tru64 > > > > > perhaps ? > > > > > ..... > > > > > > the cpu's not load balancing properly ? > > > > > > > > > > > > There could be many valid explanations. This server does not > > only > > > run > > > > > > Informix. Maybe a process binds to a cpu, and cannot be split > > > > > further, into > > > > > > multiple threads for multiple cpu's to be utilised. > > > > > > > > > > What does onstat -C show? > > > > > > > > > > -- > > > > > Cheers, > > > > > Obnoxio The Clown > > > > > > > > The problem is, this is an Apps server - it only runs 4ge's, and > > Java > > > > applications - no database. And all the other servers are fine. So > > I > > > think > > > > the > > > > problem is limited to this server. > > > > > > > > More frustrating is, the server would show 85% plus idle time, but > > > they are > > > > complaining it is slow. We reboot only this server, and everything > > is > > > fine > > > > again. > > > > This is why I started digging into processes now, seeing which are > > > using > > > > the > > > > most memory on this server, etc. > > > > > > > > The CPU scenario I described, is also strange to me. What if the > > > process of > > > > the user that is complaining, is running on one of the cpu's that > > are > > > busy, > > > > while the server in general still has idle time ? > > > > > > > > NOTE: This e-mail message is subject to the MTN Group disclaimer > > see > > > > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > > > > > > > > > > > > > > > > > > *********************************************************************** > > > ******** > > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > > > > > > > > --002215974c5e4e6cc304abcd4d35 > > > > > > > > > > > *********************************************************************** > > > ******** > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > NOTE: This e-mail message is subject to the MTN Group disclaimer see > > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > > > *********************************************************************** > > ******** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > NOTE: This e-mail message is subject to the MTN Group disclaimer see > http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --90e6ba21241d6b32c204abe5f53e