CPUVP overloaded but physical CPU only 50% used

Has anyone set NUMCPUVP to be more than the number of physical CPUs ? and has anyone seen similar problems. We have a problem that our informix instance gets overloaded (30+ active queries for minutes) with user complaining of timed out jobs, but our total CPU's utilization (sar) stays below 50% and I note that we have lots of ready threads. I've set NUMCPUS to 7 which is text book -1 the number of physical CPUs (8). Could it be that in the new super-fast CPU clock speeds that the engine's internal architecture needs to be rethought, perhaps threads are yielding too quickly and there is a problem with the mechanism of thread context switching ? Versions: 9.30.FC2R1 AIX 5.1 System Model: IBM,7040-681 Memory Size: 32768 MB Number Of Processors: 8 Processor Clock Speed: 1300 MHz CPU Type: 64-bit Geoff Poole Verizon Florida

Since you can add and remove virtual CPU's with the onmode command, you could try it and see if the physical CPU goes up and the number of ready threads goes down. We have a Sun V880 with 4 900 MHz CPU's and run with the number of virtual CPU's equal to the number of physical CPU's. -----Original Message----- From: geoff.poole.... [mailto:geoff.poole@verizon.com] Sent: Wednesday, July 09, 2003 2:41 PM To: ids@iiug.org Subject: CPUVP overloaded but physical CPU only 50% used [1524] Has anyone set NUMCPUVP to be more than the number of physical CPUs ? and has anyone seen similar problems. We have a problem that our informix instance gets overloaded (30+ active queries for minutes) with user complaining of timed out jobs, but our total CPU's utilization (sar) stays below 50% and I note that we have lots of ready threads. I've set NUMCPUS to 7 which is text book -1 the number of physical CPUs (8). Could it be that in the new super-fast CPU clock speeds that the engine's internal architecture needs to be rethought, perhaps threads are yielding too quickly and there is a problem with the mechanism of thread context switching ? Versions: 9.30.FC2R1 AIX 5.1 System Model: IBM,7040-681 Memory Size: 32768 MB Number Of Processors: 8 Processor Clock Speed: 1300 MHz CPU Type: 64-bit Geoff Poole Verizon Florida

We need to look at other things besides cpu utilization to address performance issues. Have you checked to see if the ready queues have anything on them? If the threads are waiting on some non-cpu resource, then having excess cpu power is not going to help. If the contention resource is cpu power, then I would expect to see threads always in the ready queue (onstat -g rea). If you don't see threads there, then I'd search other resources to see where the contention really is. M.Pruet "geoff.poole...." <geoff.poole@veri To: ids@iiug.org zon.com> cc: Sent by: Subject: CPUVP overloaded but physical CPU only 50% used [1524] forum.subscriber@ iiug.org 07/09/2003 02:40 PM Has anyone set NUMCPUVP to be more than the number of physical CPUs ? and has anyone seen similar problems. We have a problem that our informix instance gets overloaded (30+ active queries for minutes) with user complaining of timed out jobs, but our total CPU's utilization (sar) stays below 50% and I note that we have lots of ready threads. I've set NUMCPUS to 7 which is text book -1 the number of physical CPUs (8). Could it be that in the new super-fast CPU clock speeds that the engine's internal architecture needs to be rethought, perhaps threads are yielding too quickly and there is a problem with the mechanism of thread context switching ? Versions: 9.30.FC2R1 AIX 5.1 System Model: IBM,7040-681 Memory Size: 32768 MB Number Of Processors: 8 Processor Clock Speed: 1300 MHz CPU Type: 64-bit Geoff Poole Verizon Florida

Let me add on to this thread... Yes - we have clients that will set NUMCPUVPS greater than physical processors. Never been a recommendation from IFMX, but in reality, for some clients it has worked. As mentioned, you can add/drop cpuvps w/ onmode, so experimenting with this is easy (all things equal). But as Madison pointed out, adding more CPUVPS to the engine when physical cpu utilization is at 1/2 typically won't help. You're simply adding more processes (oninit) to the unix box, so you may see more cpu usage, but it's unlikely. I would agree to that more digging needs to be done to see where the contention really lies. I/O? Disk/chunk contention? You can see alot from "onstat -g rea". And "-g iof". Make sure though you check the unix side with respect to i/o..."-g iof" could show you very even accesses across the chunks (which are typically logical names/links), but when looking at the physical disk (iostat), everything could be piled on one or two devices. I would also want to check locking contention (onstat -p...ovlock; deadlck...) - if there are only 30 queries concurrent, on a box this size, that's not really that many . It doesn't sound like a performance problem of being out of horsepower (due to the phys cpus low usage) - it's a perceived performance problem based on some other contention(s). HTH - Mark Mark Scranton Principal Consultant/Teacher IBM Denver IBM Software Group - Data Management Office: 303-773-5067 Cell: 303-929-0914 email: mscranto@us.ibm.com Madison Pruet/Dallas/IBM@ To: ids@iiug.org IBMUS cc: Sent by: Subject: Re: CPUVP overloaded but physical CPU only 50% used [1526] forum.subscriber@ iiug.org 07/09/2003 03:26 PM We need to look at other things besides cpu utilization to address performance issues. Have you checked to see if the ready queues have anything on them? If the threads are waiting on some non-cpu resource, then having excess cpu power is not going to help. If the contention resource is cpu power, then I would expect to see threads always in the ready queue (onstat -g rea). If you don't see threads there, then I'd search other resources to see where the contention really is. M.Pruet "geoff.poole...." <geoff.poole@veri To: ids@iiug.org zon.com > cc: Sent by: Subject: CPUVP overloaded but physical CPU only 50% used [1524] forum.subscriber@ iiug.org 07/09/2003 02:40 PM Has anyone set NUMCPUVP to be more than the number of physical CPUs ? and has anyone seen similar problems. We have a problem that our informix instance gets overloaded (30+ active queries for minutes) with user complaining of timed out jobs, but our total CPU's utilization (sar) stays below 50% and I note that we have lots of ready threads. I've set NUMCPUS to 7 which is text book -1 the number of physical CPUs (8). Could it be that in the new super-fast CPU clock speeds that the engine's internal architecture needs to be rethought, perhaps threads are yielding too quickly and there is a problem with the mechanism of thread context switching ? Versions: 9.30.FC2R1 AIX 5.1 System Model: IBM,7040-681 Memory Size: 32768 MB Number Of Processors: 8 Processor Clock Speed: 1300 MHz CPU Type: 64-bit Geoff Poole Verizon Florida

If onstat -g ath shows a running thread for every vp (at least most ofthe time) and lots of ready threads and you have low sar cpu usage, there might be indeed a performance problem in the server that needs fixing. As Jay Aymond already suggested it would be interesting so see if doubling the number of cpu vps increases the sar cpu usage (just use onmode -p to add vps dynamically). If it does, it points into the samedirection and would be a workaround for the problem. If this is really your situation, you should contact Informix support and get it analyzed. Maybe you can find out more about the threads that are running most of the time. What are they doing (which sql stmt)? It's likely that the problem is tied to some specific user activity. You might be running into some already known bug. Michael geoff.poole.... wrote: > Has anyone set NUMCPUVP to be more than the number of physical CPUs ? and > has anyone seen similar problems. > > We have a problem that our informix instance gets overloaded (30+ active > queries for minutes) with user complaining of timed out jobs, but our total > CPU's utilization (sar) stays below 50% and I note that we have lots of > ready threads. I've set NUMCPUS to 7 which is text book -1 the number of > physical CPUs (8). > > Could it be that in the new super-fast CPU clock speeds that the engine's > internal architecture needs to be rethought, perhaps threads are yielding > too quickly and there is a problem with the mechanism of thread context > switching ? > > Versions: 9.30.FC2R1 AIX 5.1 > System Model: IBM,7040-681 > Memory Size: 32768 MB > Number Of Processors: 8 > Processor Clock Speed: 1300 MHz > CPU Type: 64-bit > > Geoff Poole > Verizon Florida > > > > -- === Michael Mueller ================== Tel. + 49 8171 63600 Fax. + 49 8171 63615 Web: http://www.mm.kay-mueller.de http://www.planets.kay-mueller.de ======================================

Geoff -- We run on Sun boxes so I have no idea if you are seeing the same problem, but we had a similar problem with our runtime installations -- 4 CPU box and the load never got much above .8 and throughput was not as good as we would have liked. We had two problems -- one was using cooked disk space, but the real impediment was a problem with physical log space not being big enough to accomodate all of the connections; making this much larger seems to have yielded an improvement -- CPU loads now range between 1 and 2 with normal traffic and throughput is better. We currently have NUMCPUS set to 3 and have some virtual CPUs running and the server itself seems to be behaving much better. Occasionally the logs would reveal this specific issue but not always -- usually only when we faced a sudden spike in connections (from 80-100 to more than 400 say). Greg Wiliamson DBA GlobeXplorer LLC gsw@globexplorer.com ********************************************************************** SUBJECT: CPUVP overloaded but physical CPU only 50% used [1524] URL: http://www.iiug.org/forums/ids/?rev=1524 AUTHOR: geoff.poole.... DATE: Wednesday, 9 July 2003, at 3:40 p.m. Has anyone set NUMCPUVP to be more than the number of physical CPUs ? and has anyone seen similar problems. We have a problem that our informix instance gets overloaded (30+ active queries for minutes) with user complaining of timed out jobs, but our total CPU's utilization (sar) stays below 50% and I note that we have lots of ready threads. I've set NUMCPUS to 7 which is text book -1 the number of physical CPUs (8). Could it be that in the new super-fast CPU clock speeds that the engine's internal architecture needs to be rethought, perhaps threads are yielding too quickly and there is a problem with the mechanism of thread context switching ? Versions: 9.30.FC2R1 AIX 5.1 System Model: IBM,7040-681 Memory Size: 32768 MB Number Of Processors: 8 Processor Clock Speed: 1300 MHz CPU Type: 64-bit Geoff Poole Verizon Florida **********************************************************************