Increasing shared memory

Hi All, We have Informix 11.10.FC2 on SUSE Linux 10,RAM 24GB with shared memory of 19190788 Kbytes. The bufferpool is as follows: BUFFERPOOL default,buffers=102400,lrus=32,lru_min_dirty=1.000000,lru_max_dirty=2.000000 BUFFERPOOL size=2K,buffers=204800,lrus=32,lru_min_dirty=1.000000,lru_max_dirty=2.000000 BUFFERPOOL size=8K,buffers=1700000,lrus=128,lru_min_dirty=1.000000,lru_max_dirty=2.000000 We added 24GB more RAM to system and tried doubling the shared memory due to performance issues by setting the buffers as follows: BUFFERPOOL default,buffers=204800,lrus=32,lru_min_dirty=1.000000,lru_max_dirty=2.000000 BUFFERPOOL size=2K,buffers=409600,lrus=32,lru_min_dirty=1.000000,lru_max_dirty=2.000000 BUFFERPOOL size=8K,buffers=3400000,lrus=128,lru_min_dirty=1.000000,lru_max_dirty=2.000000 Instead of improving, the server perfromance has degraded. The processes are taking longer time. Also the IDS shuts downs in between with the following error: Fatal error in ADM VP at mt.c:13388Unexpected virtual processor termination, pid = 1825, exit = 0x9 PANIC: Attempting to bring system down We had the followng advisory in the online.log 09:45:31 WARNING! Logical log layout may cause Dynamic Server to get into a locked state. Recommended smallest logical log size is 16 times maximum concurrent user threads. 09:59:56 Performance Advisory: Logical log file size might be too small for a checkpoint to complete. 09:59:56 Results: The size of individual logical log files is too small for the current workload, resulting in each log file filling very quickly. If log files fill in less than 30 seconds, the checkpoint might remain blocked because the last log file fills during the time needed to perform the checkpoint. 09:59:56 Action: Increase the size of the individual logical log files so that it takes at least 30 seconds to fill each one. Look at the online log to determine how quickly the log files are filling, and then increase the size of the files proportionately. 10:53:13 Performance Advisory: Based on the current workload, the physical log might be too small to accommodate the time it takes to flush the buffer pool. 10:53:13 Results: The server might block transactions during checkpoints. 10:53:13 Action: If transactions are blocked during the checkpoint, increase the size of the physical log to at least 739760 KB. Also when checked the swap memory was completely utilised. So we had to rollback the buffer changes and get it back to the old setting of the buffer. I would like to know the following after increasing the shared memory: 1) If the shared memory is to be increased, what are the other parameters (LGOBUFF, PHYBUFF)to increase along with buffers, lrus, lru_min_dirty, lru_max_dirty? 2) What are areas we need to look into apart from increasing the shared memory? What are the utilies (eg. onstat -g con) that will help us analyze the performance? Do let me know if any details are required. thanks in advance. Regards, Sushant

See some notes below: Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Feb 17, 2012 at 12:42 AM, SUSHANT KODE <ksushants@gmail.com> wrote: > Hi All, > > We have Informix 11.10.FC2 on SUSE Linux 10,RAM 24GB with shared memory of > 19190788 Kbytes. The bufferpool is as follows: > BUFFERPOOL> > default,buffers=102400,lrus=32,lru_min_dirty=1.000000,lru_max_dirty=2.000000 > BUFFERPOOL> > size=2K,buffers=204800,lrus=32,lru_min_dirty=1.000000,lru_max_dirty=2.000000 > BUFFERPOOL> > size=8K,buffers=1700000,lrus=128,lru_min_dirty=1.000000,lru_max_dirty=2.000000 > > We added 24GB more RAM to system and tried doubling the shared memory due > to > performance issues by setting the buffers as follows: > BUFFERPOOL> > default,buffers=204800,lrus=32,lru_min_dirty=1.000000,lru_max_dirty=2.000000 > Why double the 'default' setting? This is ONLY used to create new buffer pools if you create a new dbspace with a pagesize that does not already have a BUFFERPOOL setting. > BUFFERPOOL> > size=2K,buffers=409600,lrus=32,lru_min_dirty=1.000000,lru_max_dirty=2.000000 > BUFFERPOOL> > size=8K,buffers=3400000,lrus=128,lru_min_dirty=1.000000,lru_max_dirty=2.000000 > You only have 409,600 2K buffers but 3,400,000 8K buffers. Is this realistic? Do you really have 64 times as much data in your 8K dbspaces as you do in 2K dbspaces? Also, you increased your shared memory by 14GB by increasing the buffers like this. Do you even have 28GB of data on disk? > > Instead of improving, the server perfromance has degraded. The processes > are > taking longer time. Also the IDS shuts downs in between with the following > error: > Fatal error in ADM VP at mt.c:13388> Unexpected virtual processor termination, pid = 1825, exit = 0x9 > PANIC: Attempting to bring system down > Probably an old bug. Informix 11.10.FC2 is older and an early version of 11.10 at that. You should upgrade to 11.50.FC9 or 11.70.FC4 or later. > > We had the followng advisory in the online.log > 09:45:31 WARNING! Logical log layout may cause Dynamic Server to get into > > a locked state. Recommended smallest logical log size > This is telling you do that you need to recreate your logical logs with a larger size for each log. > > is 16 times maximum concurrent user threads. > > 09:59:56 Performance Advisory: Logical log file size might be too small > for a > > checkpoint to complete. > 09:59:56 Results: The size of individual logical log files is too small for > > the current workload, resulting in each log file filling very > > quickly. If log files fill in less than 30 seconds, the checkpoint > > might remain blocked because the last log file fills during the time > > needed to perform the checkpoint. > 09:59:56 Action: Increase the size of the individual logical log files so > > that it takes at least 30 seconds to fill each one. Look at the > > online log to determine how quickly the log files are filling, and > > then increase the size of the files proportionately. > > 10:53:13 Performance Advisory: Based on the current workload, the physical > log > might be too small to > accommodate the time it takes to flush the buffer pool. > 10:53:13 Results: The server might block transactions during checkpoints. > 10:53:13 Action: If transactions are blocked during the checkpoint, > increase > the size of the > physical log to at least 739760 KB. > This one is often spurious. Watch the server shortly before a checkpoint during peak update load and see what percent of the physical log is full (see the onstat -F report). If it is close to 75% then this message is valid and you probably should increase the physical log size as recommended. If, however, the physical log is mostly empty at checkpoint time then the message is spurious. The algorithm that the engine is using to produce that message was brand new in 11.10 and is still evolving even in 11.70. > > Also when checked the swap memory was completely utilised. So we had to > rollback the buffer changes and get it back to the old setting of the > buffer. > Linux follows the BSD memory model and copies everything from memory to swap immediately when it is first loaded into memory. That means that you need at least 3x the amount of memory for swap space. Did you double the swap when you doubled main memory? Also, you should be running the engine with the RESIDENT setting in the ONCONFIG file set to 1 or -1 so that all shared memory segments used by Informix are non-swappable. > > I would like to know the following after increasing the shared memory: > 1) If the shared memory is to be increased, what are the other parameters > (LGOBUFF, PHYBUFF)to increase along with buffers, lrus, lru_min_dirty, > lru_max_dirty? > You are running with the lru_min/max_dirty settings very aggressive. In 11.xx we can tune these down from 1 & 2, respectively, that we would normally use in Informix 10.00 and earlier to higher values like 50 & 20 that allow the engine to do its writes more efficiently since checkpoints no longer block user threads. > 2) What are areas we need to look into apart from increasing the shared > memory? What are the utilies (eg. onstat -g con) that will help us analyze > the > performance? > > Do let me know if any details are required. > thanks in advance. > > Regards, > Sushant > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --e89a8f3ba6813065d004b926e8ce

Older versions of Unix-type operating systems (such as Sun OS and Ultrix) demanded a swap space of two to three times that of physical memory. Modern implementations (such as Linux) don't require that much, but they can use it if you configure it. A good rule of thumb is as follows: 1) for a desktop system, use a swap space of double system memory, as it will allow you to run a large number of applications (many of which may will be idle and easily swapped), making more RAM available for the active applications; 2) for a server, have a smaller amount of swap available (say half of physical memory) so that you have some flexibility for swapping when needed, but monitor the amount of swap space used and upgrade your RAM if necessary; 3) for older desktop machines (with say only 128MB), use as much swap space as you can spare, even up to 1GB. The Linux 2.6 kernel added a new kernel parameter called swappiness to let administrators tweak the way Linux swaps. It is a number from 0 to 100. In essence, higher values lead to more pages being swapped, and lower values lead to more applications being kept in memory, even if they are idle. Kernel maintainer Andrew Morton has said that he runs his desktop machines with a swappiness of 100, stating that "My point is that decreasing the tendency of the kernel to swap stuff out is wrong. You really don't want hundreds of megabytes of BloatyApp's untouched memory floating about in the machine. Get it out on the disk, use the memory for something useful." One downside to Morton's idea is that if memory is swapped out too quickly then application response time drops, because when the application's window is clicked the system has to swap the application back into memory, which will make it feel slow. Cheers Paul > See some notes below: > > Art > > Art S. Kagel > Advanced DataTools (www.advancedatatools.com) > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > other organization with which I am associated either explicitly, > implicitly, or by inference. Neither do those opinions reflect those of > other individuals affiliated with any entity with which I am affiliated > nor > those of the entities themselves. > > On Fri, Feb 17, 2012 at 12:42 AM, SUSHANT KODE <ksushants@gmail.com> > wrote: > >> Hi All, >> >> We have Informix 11.10.FC2 on SUSE Linux 10,RAM 24GB with shared memory >> of >> 19190788 Kbytes. The bufferpool is as follows: >> BUFFERPOOL>> >> default,buffers=102400,lrus=32,lru_min_dirty=1.000000,lru_max_dirty=2.000000 >> BUFFERPOOL>> >> size=2K,buffers=204800,lrus=32,lru_min_dirty=1.000000,lru_max_dirty=2.000000 >> BUFFERPOOL>> >> > size=8K,buffers=1700000,lrus=128,lru_min_dirty=1.000000,lru_max_dirty=2.000000 >> >> We added 24GB more RAM to system and tried doubling the shared memory >> due >> to >> performance issues by setting the buffers as follows: >> BUFFERPOOL>> >> default,buffers=204800,lrus=32,lru_min_dirty=1.000000,lru_max_dirty=2.000000 >> > > Why double the 'default' setting? This is ONLY used to create new buffer > pools if you create a new dbspace with a pagesize that does not already > have a BUFFERPOOL setting. > >> BUFFERPOOL>> >> size=2K,buffers=409600,lrus=32,lru_min_dirty=1.000000,lru_max_dirty=2.000000 >> BUFFERPOOL>> >> > size=8K,buffers=3400000,lrus=128,lru_min_dirty=1.000000,lru_max_dirty=2.000000 >> > > You only have 409,600 2K buffers but 3,400,000 8K buffers. Is this > realistic? Do you really have 64 times as much data in your 8K dbspaces as > you do in 2K dbspaces? Also, you increased your shared memory by 14GB by > increasing the buffers like this. Do you even have 28GB of data on disk? > >> >> Instead of improving, the server perfromance has degraded. The processes >> are >> taking longer time. Also the IDS shuts downs in between with the >> following >> error: >> Fatal error in ADM VP at mt.c:13388>> Unexpected virtual processor termination, pid = 1825, exit = 0x9 >> PANIC: Attempting to bring system down >> > > Probably an old bug. Informix 11.10.FC2 is older and an early version of > 11.10 at that. You should upgrade to 11.50.FC9 or 11.70.FC4 or later. > >> >> We had the followng advisory in the online.log >> 09:45:31 WARNING! Logical log layout may cause Dynamic Server to get >> into >> >> a locked state. Recommended smallest logical log size >> > > This is telling you do that you need to recreate your logical logs with a > larger size for each log. > >> >> is 16 times maximum concurrent user threads. >> >> 09:59:56 Performance Advisory: Logical log file size might be too small >> for a >> >> checkpoint to complete. >> 09:59:56 Results: The size of individual logical log files is too small >> for >> >> the current workload, resulting in each log file filling very >> >> quickly. If log files fill in less than 30 seconds, the checkpoint >> >> might remain blocked because the last log file fills during the time >> >> needed to perform the checkpoint. >> 09:59:56 Action: Increase the size of the individual logical log files >> so >> >> that it takes at least 30 seconds to fill each one. Look at the >> >> online log to determine how quickly the log files are filling, and >> >> then increase the size of the files proportionately. >> >> 10:53:13 Performance Advisory: Based on the current workload, the >> physical >> log >> might be too small to >> accommodate the time it takes to flush the buffer pool. >> 10:53:13 Results: The server might block transactions during >> checkpoints. >> 10:53:13 Action: If transactions are blocked during the checkpoint, >> increase >> the size of the >> physical log to at least 739760 KB. >> > > This one is often spurious. Watch the server shortly before a checkpoint > during peak update load and see what percent of the physical log is full > (see the onstat -F report). If it is close to 75% then this message is > valid and you probably should increase the physical log size as > recommended. If, however, the physical log is mostly empty at checkpoint > time then the message is spurious. The algorithm that the engine is using > to produce that message was brand new in 11.10 and is still evolving even > in 11.70. > >> >> Also when checked the swap memory was completely utilised. So we had to >> rollback the buffer changes and get it back to the old setting of the >> buffer. >> > > Linux follows the BSD memory model and copies everything from memory to > swap immediately when it is first loaded into memory. That means that you > need at least 3x the amount of memory for swap space. Did you double the > swap when you doubled main memory? Also, you should be running the engine > with the RESIDENT setting in the ONCONFIG file set to 1 or -1 so that all > shared memory segments used by Informix are non-swappable. > >> >> I would like to know the following after increasing the shared me

Hi, >Instead of improving, the server performance has degraded. The processes are >taking longer time. Also the IDS shuts downs in between with the following >error: >Fatal error in ADM VP at mt.c:13388>Unexpected virtual processor termination, pid = 1825, exit = 0x9 >PANIC: Attempting to bring system down In other Linux implementations I have worked with the above error is indicative the OOM ( out of Memory ) has come to life and killed the ADM VP with a signal 9 ( exit = 0x9 ) Check you message log to see if that has happened. ADM VP is often the victim as it does not reference the shared memory often but is attached to it and remains idle much of the time it is running. This causes the OOM calculation, on who to kill, to score this process very high. It would be nice to see some utilization of the buffer pools ( onstat -g buf ) as it is possible you have a great deal too many 8K buffers ( they can only be used by 8K dbspaces did you change all dbspace to 8k except root and logical logs ? ) George. From: "SUSHANT KODE" <ksushants@gmail.com> To: ids@iiug.org Date: 02/16/2012 11:42 PM Subject: Increasing shared memory [26263] Sent by: ids-bounces@iiug.org Hi All, We have Informix 11.10.FC2 on SUSE Linux 10,RAM 24GB with shared memory of 19190788 Kbytes. The bufferpool is as follows: BUFFERPOOL default,buffers=102400,lrus=32,lru_min_dirty=1.000000,lru_max_dirty=2.000000 BUFFERPOOL size=2K,buffers=204800,lrus=32,lru_min_dirty=1.000000,lru_max_dirty=2.000000 BUFFERPOOL size=8K,buffers=1700000,lrus=128,lru_min_dirty=1.000000,lru_max_dirty=2.000000 We added 24GB more RAM to system and tried doubling the shared memory due to performance issues by setting the buffers as follows: BUFFERPOOL default,buffers=204800,lrus=32,lru_min_dirty=1.000000,lru_max_dirty=2.000000 BUFFERPOOL size=2K,buffers=409600,lrus=32,lru_min_dirty=1.000000,lru_max_dirty=2.000000 BUFFERPOOL size=8K,buffers=3400000,lrus=128,lru_min_dirty=1.000000,lru_max_dirty=2.000000 Instead of improving, the server perfromance has degraded. The processes are taking longer time. Also the IDS shuts downs in between with the following error: Fatal error in ADM VP at mt.c:13388Unexpected virtual processor termination, pid = 1825, exit = 0x9 PANIC: Attempting to bring system down We had the followng advisory in the online.log 09:45:31 WARNING! Logical log layout may cause Dynamic Server to get into a locked state. Recommended smallest logical log size is 16 times maximum concurrent user threads. 09:59:56 Performance Advisory: Logical log file size might be too small for a checkpoint to complete. 09:59:56 Results: The size of individual logical log files is too small for the current workload, resulting in each log file filling very quickly. If log files fill in less than 30 seconds, the checkpoint might remain blocked because the last log file fills during the time needed to perform the checkpoint. 09:59:56 Action: Increase the size of the individual logical log files so that it takes at least 30 seconds to fill each one. Look at the online log to determine how quickly the log files are filling, and then increase the size of the files proportionately. 10:53:13 Performance Advisory: Based on the current workload, the physical log might be too small to accommodate the time it takes to flush the buffer pool. 10:53:13 Results: The server might block transactions during checkpoints. 10:53:13 Action: If transactions are blocked during the checkpoint, increase the size of the physical log to at least 739760 KB. Also when checked the swap memory was completely utilised. So we had to rollback the buffer changes and get it back to the old setting of the buffer. I would like to know the following after increasing the shared memory: 1) If the shared memory is to be increased, what are the other parameters (LGOBUFF, PHYBUFF)to increase along with buffers, lrus, lru_min_dirty, lru_max_dirty? 2) What are areas we need to look into apart from increasing the shared memory? What are the utilies (eg. onstat -g con) that will help us analyze the performance? Do let me know if any details are required. thanks in advance. Regards, Sushant ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.