New IDS / performance & stability

hello, Informix users for 20 years since version 5, the entire IT infrastructure was built around this powerful and robust product. Based on a set of AIX servers with 8TB databases Informix handle about 10,000 SQL / seconds. The successive inplace upgrades have always gone well until the 11.70 release. From 1 year now, we are trying to maintain acceptable levels of service by adding more and more hardware (SAN + CPU + RAM). Moving from 11.50 to 11.70 we have tripled the memory (SHMEM + Buffer) with a result of 20% less performances. Upgrades xC4 then xC5, xC7, xC8W1 (and many special builds) are now source, each time, of many bugs. We opened 24 PMRs on past 12 months; a lot of them corrected now. 4 new PMRs with the last xC8W1 upgrade and go back to xC7 ... The Informix Support suggests not to upgrade to 12.10 in order to avoid more serious problems. I was a convinced DBA but ... we are very worried about the future. Probably because of restrictions in testing staffs and the focus on new functionalities (AGILE development), Informix from version to version loses its reliability. What are your feedback in 11.70 and also 12.10, do you have similar problems ? Sam

Sam: Do you have VP_MEMORY_CACHE_VP set? Is it set to a number only or is it a number comma STATIC? There was a change in the behavior of this feature between 11.50 and 11.70 which can cause the server to add many virtual segments and grab memory constantly adversely affecting performance. In later releases of 11.70 (v 11.70.xC7+ IB) the option to revert to the original behavior was introduced with the optional DYNAMIC (new behavior - default) and STATIC (original behavior) specifications. You would set it like this: VP_MEMORY_CACHE_KB 4096,STATIC This may be one reason that you have experienced negative performance since upgrading from 11.50 to 11.70. Art Art S. Kagel, President and Principal Consultant ASK Database Management Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Wed, Oct 1, 2014 at 11:18 AM, Deillon Samuel <deillon@ofac.ch> wrote: > hello, > > Informix users for 20 years since version 5, the entire IT infrastructure > was > built around this powerful and robust product. > Based on a set of AIX servers with 8TB databases Informix handle about > 10,000 > SQL / seconds. > The successive inplace upgrades have always gone well until the 11.70 > release. > > >From 1 year now, we are trying to maintain acceptable levels of service by > adding more and more hardware (SAN + CPU + RAM). > Moving from 11.50 to 11.70 we have tripled the memory (SHMEM + Buffer) > with a > result of 20% less performances. > Upgrades xC4 then xC5, xC7, xC8W1 (and many special builds) are now source, > each time, of many bugs. We opened 24 PMRs on past 12 months; a lot of them > corrected now. 4 new PMRs with the last xC8W1 upgrade and go back to xC7 > ... > The Informix Support suggests not to upgrade to 12.10 in order to avoid > more > serious problems. > > I was a convinced DBA but ... we are very worried about the future. > Probably > because of restrictions in testing staffs and the focus on new > functionalities > (AGILE development), Informix from version to version loses its > reliability. > > What are your feedback in 11.70 and also 12.10, do you have similar > problems ? > > Sam > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a113340ca4e8ba505045e1fce

This is a very good point, and at least as of 11.70.FC8, was not well documented. As soon as we upgraded one of our production servers to IDS11.70.FC8, we started seeing multiple segments being allocated and performance was very degraded. IBM support discovered that VP_MEMORY_CACHE_KB was the issue, and since we have changed its value to include the "STATIC" setting, performance has been fine. Pam ________________________________________ From: ids-bounces@iiug.org [ids-bounces@iiug.org] On Behalf Of Art Kagel [art.kagel@gmail.com] Sent: Wednesday, October 01, 2014 11:26 AM To: ids@iiug.org Subject: Re: New IDS / performance & stability [33884] Sam: Do you have VP_MEMORY_CACHE_VP set? Is it set to a number only or is it a number comma STATIC? There was a change in the behavior of this feature between 11.50 and 11.70 which can cause the server to add many virtual segments and grab memory constantly adversely affecting performance. In later releases of 11.70 (v 11.70.xC7+ IB) the option to revert to the original behavior was introduced with the optional DYNAMIC (new behavior - default) and STATIC (original behavior) specifications. You would set it like this: VP_MEMORY_CACHE_KB 4096,STATIC This may be one reason that you have experienced negative performance since upgrading from 11.50 to 11.70. Art Art S. Kagel, President and Principal Consultant ASK Database Management Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Wed, Oct 1, 2014 at 11:18 AM, Deillon Samuel <deillon@ofac.ch> wrote: > hello, > > Informix users for 20 years since version 5, the entire IT infrastructure > was > built around this powerful and robust product. > Based on a set of AIX servers with 8TB databases Informix handle about > 10,000 > SQL / seconds. > The successive inplace upgrades have always gone well until the 11.70 > release. > > >From 1 year now, we are trying to maintain acceptable levels of service by > adding more and more hardware (SAN + CPU + RAM). > Moving from 11.50 to 11.70 we have tripled the memory (SHMEM + Buffer) > with a > result of 20% less performances. > Upgrades xC4 then xC5, xC7, xC8W1 (and many special builds) are now source, > each time, of many bugs. We opened 24 PMRs on past 12 months; a lot of them > corrected now. 4 new PMRs with the last xC8W1 upgrade and go back to xC7 > ... > The Informix Support suggests not to upgrade to 12.10 in order to avoid > more > serious problems. > > I was a convinced DBA but ... we are very worried about the future. > Probably > because of restrictions in testing staffs and the focus on new > functionalities > (AGILE development), Informix from version to version loses its > reliability. > > What are your feedback in 11.70 and also 12.10, do you have similar > problems ? > > Sam > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a113340ca4e8ba505045e1fce ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Hi Art, I think your versioning is a bit wrong here. I blogged about this issue late last year: http://informixdba.wordpress.com/2013/11/11/monitoring-virtual-segment-usage-and -the-cpu-vp-caches/ I think the 11.70 behaviour was the same as 11.50 initially and the new behaviour appeared in 11.70.FC7W1 as a bug fix for a performance issue reported by a customer. This is why the new feature appeared in a W1 release which is meant to be fixes only. The STATIC and DYNAMIC behaviour was put in after a PMR I raised and I forget which versions have it exactly but I think it's in 11.70.FC8. The defect number is IC95684. Something that may make the grabbing of memory hard to see or seem inexplicable is that the memory usage of VP memory caches is not shown in "onstat -g mem" or any sysmaster view, e.g. syspoollst. It has to be monitored separately by "onstat -g vpcache". Ben.

No, it's only in the 12.10 documentation (http://www-01.ibm.com/support/knowledgecenter/#!/SSGU8G_12.1.0/com.ibm.adref.do c/ids_adr_0188.htm) but it applies to 11.70.FC8.

Thanks for the clarification Ben. Art Art S. Kagel, President and Principal Consultant ASK Database Management Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Wed, Oct 1, 2014 at 12:13 PM, BENJAMIN THOMPSON < benjamin.thompson@bskyb.com> wrote: > Hi Art, > > I think your versioning is a bit wrong here. > > I blogged about this issue late last year: > > > http://informixdba.wordpress.com/2013/11/11/monitoring-virtual-segment-usage-and -the-cpu-vp-caches/ > > I think the 11.70 behaviour was the same as 11.50 initially and the new > behaviour appeared in 11.70.FC7W1 as a bug fix for a performance issue > reported by a customer. This is why the new feature appeared in a W1 > release > which is meant to be fixes only. The STATIC and DYNAMIC behaviour was put > in > after a PMR I raised and I forget which versions have it exactly but I > think > it's in 11.70.FC8. The defect number is IC95684. > > Something that may make the grabbing of memory hard to see or seem > inexplicable is that the memory usage of VP memory caches is not shown in > "onstat -g mem" or any sysmaster view, e.g. syspoollst. It has to be > monitored > separately by "onstat -g vpcache". > > Ben. > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a11c34e482d8bca05045ef6d7

Not sure if this will help out, but you can monitor the vpcache by sysmaster. If you look at sysvplst and check the columns which start with vp_cache you will see information on how much memory and how efficient the vpcache is. John F. Miller III STSM, Lead Architect miller3@us.ibm.com ids-bounces@iiug.org wrote on 10/01/2014 09:13:22 AM: > From: "BENJAMIN THOMPSON" <benjamin.thompson@bskyb.com> > To: ids@iiug.org > Date: 10/01/2014 09:14 AM > Subject: Re: New IDS / performance & stability [33886] > Sent by: ids-bounces@iiug.org > > Hi Art, > > I think your versioning is a bit wrong here. > > I blogged about this issue late last year: > > http://informixdba.wordpress.com/2013/11/11/monitoring-virtual- > segment-usage-and-the-cpu-vp-caches/ > > I think the 11.70 behaviour was the same as 11.50 initially and the new > behaviour appeared in 11.70.FC7W1 as a bug fix for a performance issue > reported by a customer. This is why the new feature appeared in a W1 release > which is meant to be fixes only. The STATIC and DYNAMIC behaviour was put in > after a PMR I raised and I forget which versions have it exactly but I think > it's in 11.70.FC8. The defect number is IC95684. > > Something that may make the grabbing of memory hard to see or seem > inexplicable is that the memory usage of VP memory caches is not shown in > "onstat -g mem" or any sysmaster view, e.g. syspoollst. It has to bemonitored > separately by "onstat -g vpcache". > > Ben. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >

To answer your points more generally than in the other replies, I also run some systems as busy as yours. You can probably infer this from the fact we ran into the VP_MEMORY_CACHE_KB issue, which will only be seen on a very busy system. I am disappointed to hear the FC8W1 fix pack hasn't work for you and that you were forced to back out. We haven't needed to invest in better CPUs or a SAN as a result of a 11.70 upgrade. The VP_MEMORY_CACHE_KB issue gobbles RAM as you may have seen but 11.70 is otherwise ok in this regard for us. Generally, 11.70 performance hasn't been an issue and there are some scalability improvements in 11.70 we needed (NUMFDSERVERS, index page limit vastly increased). We've had a small number of serious reliability issues with 11.70 but these were dealt with by IBM and fixes delivered, like your experience. One of these was actually down to an ancient bug, caused by a failure to latch a memory area while reading from it. We now run reliably with no serious defects awaiting a fix. Is 11.70 any less reliable than any release that went before? Generally I am pretty satisfied with it and IBM's response to any problems. You maybe have to be more careful when moving between fix packs and you might be better advised getting specific fixes for your issues back ported. More features is likely to lead to more defects. I certainly didn't think 11.50 was defect free and I recall running into a pretty disastrous memory leak bug in 11.50.FC2 or FC3 on site somewhere a few years ago for which I am famous. Ok, these were early fix packs and FC9W2 was solid but even so. To give a historical perspective, I also have in my mind that 9.21 and 11.10 were not the finest releases. Ben.

Thanks John.