Memory issue on Informix primary server

Dear All, We have upgrade our Informix server from 11.7 to Version 14.10.FC6X3 on Sun OS 5.11. After that We are facing multiple memory segment issue on Primary server, but everything is normal on Secondary server. Before Upgrade there was only on Virtual memory segment. I have checked all informix resource thru onstat -g osi, top , CPU, Memory all. mostly all looks same. I've attached a word file for some Informix command output for both Informix Primary and Secondary server. Can someone check and help me to solve the issue ? what could the best option apart from increasing the shared memory. Thanks Amit Patel ------------------------------ AMIT PATEL ------------------------------ #Informix

Amit: All of those additional virtual segments are in use. I think that the only thing you can do is merge them all into the initial SHMVIRTSIZE and increase the SHMADD size by 10X or more. Solaris is REALLY slow to allocate shared memory, so the fewer segments the better. Art ------------------------------ Art S. Kagel, President and Principal Consultant ASK Database Management Corp. www.askdbmgt.com ------------------------------

Thanks Art, But total of virtual segment is 518617300992 Bytes means ( 518 GB ) , which looks much to replace with current total shared memory. SHMVIRTSIZE 31457280 (31GB)and as per "onstat -g osi" system memory is System memory 781568 MB. So is there any other resource we can change Art? Or we can check to restart to production server to check if resource can be cleared. Thanks Amit Patel ------------------------------ AMIT PATEL ------------------------------

Looking at the onstat -g seg output, currently those additonal virtual segments appears to be mostly unused. It is likely that there were sessions that were using a large amount of shared memory and triggered the engine to dynamically allocate additonal shared memory segments at one time. You can find out the timing on when these virtual segments were added from online.log, and try to find out if there are any special task/job running around the time. As the session/pool that was using the large amount memory has disconnected from the server, the onstat output would not show anything useful. Running onmode -F might be able to free up virtual segments if there are no other resources using it, or restaring the engine will be able to release the additional virtual segments. Segment Summary: id key addr size ovhd class blkused blkfree 6 52574801 10a000000 1140850688 13811032 R* 276082 2446 7 52574802 14e000000 32212254720 377489352 V* 2153941 5710379 8 52574803 8ce000000 6375342080 1 B* 1556480 0 9 52574804 a4a000000 33554432 1 B* 8192 0 10 52574805 a4c000000 3145728 38232 M 539 229 11 52574806 a4c300000 3145728 38232 M 537 231 12 52574807 a4d000000 3221225472 37750584 V* 10461 775971 13 52574808 b0d000000 3221225472 37750584 V* 10153 776279 14 52574809 bcd000000 3221225472 37750584 V* 9956 776476 15 5257480a c8d000000 3221225472 37750584 V* 9775 776657 16 5257480b d4d000000 3221225472 37750584 V* 9817 776615 17 5257480c e0d000000 3221225472 37750584 V* 10195 776237 ------------------------------ Casey Tan ------------------------------

hi, with 14.10 you have more tasks running in the scheduler than with 11.70, may be you haven't launch it in version 11.70. Jacques ------------------------------ jack rage ------------------------------

Amit: Casey's right, it looks like most of the memory allocated is unused. I had missed that. Here's what I think happened. As I said Solaris is VERY SLOW to allocate shared memory, so when a session needed more memory the engine requested another 3GB segment. It took so long that another session wanted more memory before the segment was completed which just slowed the process even more, and another session, and another, etc. until 500GB had been allocated. What you can try is to increase SHMVIRTSIZE by 6 to 12 GB up from 32GB to say 44 GB and see how that goes. If it is not enough and the engine is still adding segments, then bump it again until that stops happening. The only way to stop the engine from creating new shared memory segments altogether would be to cap the maximum allocation with the SHMTOTAL parameter. But I don't think that is needed. Art ------------------------------ Art S. Kagel, President and Principal Consultant ASK Database Management Corp. www.askdbmgt.com ------------------------------

I think there's a memory leak If you do not control, you will eventually run out of physical memory of the operating system I have encountered a similar case before informix 12.10fc4+redhat 6.5 However, we found that it was caused by an application. When the application started to connect to the database, segment V began to add madly ------------------------------ ZhiWei Cui GBASE ------------------------------

Hi, You can either run "onstat -g mem" every few minutes from cron to hope to catch it. Or you can update the case statement in the alarmprogram so that when an extra shared memory segment is added it runs "onstat -g mem" to a file to you can see what is using the memory. You can also run "onstat -g ses" from cron every few minutes (10? 15?) to try to catch sessions which are using a lot of memory. If you find a session using a lot of memory then "onstat -g pqs" to see prepared statements and what the session is doing. David. ------------------------------ David Williams ------------------------------

I encourage anyone who needs a window into steady memory growth to check out onstat -g top, which is only available in 14.10: https://www.ibm.com/docs/en/informix-servers/14.10?topic=ogmo-onstat-g-top-command-print-top-consumers-resources General syntax: onstat -g top <keyword pair> <max consumers> <interval> <repeats> Example: onstat -g top mempool gro 10 30 0The above command will display the 10 memory pools that are growing the most, over 30 second intervals, repeating indefinitely. onstat -g top sessmem gro 5 20 3The above command will display the 5 sessions whose memory usage is growing the most, over 20 second intervals, repeating 3 times. This command will automatically sample, slice, and dice onstat data in all kinds of ways depending on which keyword pair you use. -jc ------------------------------ John Lengyel ------------------------------