shmget: [ENOSPC][28]: key 525f4802: The server could not allocate shared memory because an operating system limit was reached

Hey gurus, What do I need to increase to allow the database the ability to add additional virtual memory segments? HCL person could not answer my question when I opened the case and I have dozens of servers getting the same message. -- Thanks, Kate Tomchik She/Her/Hers Principal Systems Engineer, Database Solutions The Home Depot Cell: 678-427-4914 (Text preferred) For non-urgent requests create a TICKET: https://servicecatalog.apps.homedepot.com/home/catalog/Databases/databasesupport Choose experience: IT-09034-Database Solutions Operations INTERNAL USE #Informix

Kate: Probably it will be the shmseg setting. Art ------------------------------ Art S. Kagel, President and Principal Consultant ASK Database Management Corp. www.askdbmgt.com ------------------------------

Oops, that got away from me. It could be shmseg or shmmni. Art ------------------------------ Art S. Kagel, President and Principal Consultant ASK Database Management Corp. www.askdbmgt.com ------------------------------

How depends on the OS, check the releases notes -- Cheers Paul Paul Watson Oninit LLC +1-913-387-7529 Oninit is a Registered Trademark of Oninit LLC

Hi Kate, You are likely bumping into one of the kernel limits, like SHMMAX or something. I don't know what OS you are using, but try running lsipc and see if that shows that you are close to any of the limits. iscs -l (that's an "L") will also show your shared memory limits. Do you have a lot of virtual memory segments that have been added dynamically (onstat -g seg)? If so, then onmode -F may buy you some time. ------------------------------ Mike Walker xDB Systems, Inc www.xdbsystems.com ------------------------------

Redhat 6 and I have already confirmed we have all the Kernel paramaters set as the release notes say. -- Thanks, Kate Tomchik She/Her/Hers Principal Systems Engineer, Database Solutions The Home Depot Cell: 678-427-4914 (Text preferred) For non-urgent requests create a TICKET: https://servicecatalog.apps.homedepot.com/home/catalog/Databases/databasesupport Choose experience: IT-09034-Database Solutions Operations

Share memory is running crazy /home/informix> onstat -g seg IBM Informix Dynamic Server Version 12.10.FC13 -- On-Line -- Up 3 days 06:55:28 -- 302260 KbytesSegment Summary: id key addr size ovhd class blkused blkfree 1015815 525f4801 44000000 13893632 601096 R 3392 0 1048584 525f4802 44d40000 144678912 1697112 V 35152 170 1081353 525f4803 4d73a000 11632640 1 B 2840 0 1114122 525f4804 4e252000 21307392 1 B 5202 0 1146891 525f4805 4f6a4000 561152 7848 M 136 1 1179660 525f4806 4f72d000 16777216 198072 V 3641 455 1212429 525f4807 5072d000 16777216 198072 V 3280 816 1245198 525f4808 5172d000 16777216 198072 V 2912 1184 1277967 525f4809 5272d000 16777216 198072 V 3512 584 1310736 525f480a 5372d000 16777216 198072 V 1415 2681 1343505 525f480b 5472d000 16777216 198072 V 2937 1159 1376274 525f480c 5572d000 16777216 198072 V 174 3922 Total: - - 309514240 - - 64593 10972 (* segment locked in memory) No reserve memory is allocated >lsipc No such file or directory >iscs -l No su file or directory -- Thanks, Kate Tomchik She/Her/Hers Principal Systems Engineer, Database Solutions The Home Depot Cell: 678-427-4914 (Text preferred) For non-urgent requests create a TICKET: https://servicecatalog.apps.homedepot.com/home/catalog/Databases/databasesupport Choose experience: IT-09034-Database Solutions Operations

Kate: The release note settings for kernel parameters are minimums and are simply values that allowed the Informix techs to start and test the engine. You may certainly need higher settings for some. When you have many instances on a host, shmmninis one of those. Art ------------------------------ Art S. Kagel, President and Principal Consultant ASK Database Management Corp. www.askdbmgt.com ------------------------------

run the command $ sudo sysctl -a | grep shmkernel.shmmax = 68719476736 kernel.shmall = 4294967296 kernel.shmmni = 4096 vm.hugetlb_shm_group = 0 Where SHMMNI ??? This parameter sets the system wide maximum number of shared memory segments. SHMALL ??? This parameter sets the total amount of shared memory pages that can be used system wide. Hence, SHMALL should always be at least ceiling(shmmax/PAGE_SIZE). SHMMAX ??? This parameter defines the maximum size in bytes of a single shared memory segment that a Linux?? process can allocate in its virtual address space. -- Cheers Paul Paul Watson Oninit LLC +1-913-387-7529 Oninit is a Registered Trademark of Oninit LLC

That doesn't seem too crazy, and the memory footprint is still quite small, but it may still be more than normal so may be hitting OS limits...especially if you have multiple instances on the server. I still think that's the likely problem that you are hitting, but the underlying problem may be something consuming more memory than usual. Sorry, I mistyped one of those commands...should be ipcs -l (that's an "L") You may want to check the Informix log to see when the memory segments were added and whether that corresponds to the failure to allocate shared memory. Also check to see if the lock table has been dynamically expanded as that can be responsible for adding more memory segments (onstat -k | tail). ------------------------------ Mike Walker xDB Systems, Inc www.xdbsystems.com ------------------------------

Hi Kate, A few weeks ago, our main engine became to get into the same problem here , IFX 12.15FC15 / RH 7.4. Here, at least three times the instance crashed with AF and memory dump because they aren't able to allocate more memory. However, our configuration is set the same for years and never get any like this. Yesterday, again we get into the same problem, luckily, the instance not crashed. I was able to collect few information to the HCL support where I have my own suspicious the engine isn't freeing the virtual segments, where they have something pointing to it, but no session or memory pool is really using them... then they became to allocating more and more as the workload of the moment requires until they crash... and the "onmode -F" doesn't free them. If I have any news about my case, if became detected some defect, I will share it here. ------------------------------ Cesar Martins ------------------------------