shmget ENOSPC Informix 11.70FC7 on Ubuntu 64 bit

Hello, sometimes connections to our IBM Informix Dynamic Server Version 11.70.FC7 hosted on Ubuntu 12.04.4 LTS (64 bit) in a virtual machine (8GB RAM) become very slow and clients get error messages like: -243 Could not position within a table ... The whole server seems to be affected. In online.log are many messages like this: shmget: [ENOSPC][28]: key 52564864: The server could not allocate shared memory because an operating system limit was reached I did not find much about this. There have been bugs in 11.70 causing ENOSPC errors, but they are supposed to have been fixed in 11.70.xC5. I found this thread: https://groups.google.com/forum/#!msg/comp.databases.informix/Q4emLX1Q_cI/j67-NE ZvpuAJ However, shmmni is 4096 per default and I did not find shmseg in the kernel settings or any infomation on how to set it on Ubuntu. If I remember correctly setting a fixed value for SHMTOTAL did not help. I'd be very happy about any hints on how to resolve this and find the cause. Thank you very much in advance! Best regards, Peter Seifert

Try setting the memlock kernel parameter to unlimited. I have seen this problem when memlock is set to the default. Art Art S. Kagel, Principal Consultant ASK Database Management Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, May 15, 2014 at 5:44 AM, JAN-PETER SEIFERT <seifert@his.de> wrote: > Hello, > > sometimes connections to our IBM Informix Dynamic Server Version 11.70.FC7 > hosted on Ubuntu 12.04.4 LTS (64 bit) in a virtual machine (8GB RAM) become > very slow and clients get error messages like: > -243 Could not position within a table ... > > The whole server seems to be affected. > > In online.log are many messages like this: > shmget: [ENOSPC][28]: key 52564864: The server could not allocate shared > memory because an operating system limit was reached > > I did not find much about this. There have been bugs in 11.70 causing > ENOSPC > errors, but they are supposed to have been fixed in 11.70.xC5. > > I found this thread: > > > https://groups.google.com/forum/#!msg/comp.databases.informix/Q4emLX1Q_cI/j67-NE ZvpuAJ > > However, shmmni is 4096 per default and I did not find shmseg in the kernel > settings or any infomation on how to set it on Ubuntu. > > If I remember correctly setting a fixed value for SHMTOTAL did not help. > > I'd be very happy about any hints on how to resolve this and find the > cause. > > Thank you very much in advance! > > Best regards, > > Peter Seifert > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a11c3b5dce1facf04f96df0e8

Hi, "shmget: [ENOSPC][28]: key 52564864: The server could not allocate shared memory because an operating system limit was reached" So, what can we infer from this? 1. The number of shared memory segments allocated to your instance has reached: 0x52564864 - 0x52564801 = 0x63, so 0x64 segments in total = 100. Have a look at onstat -g seg and ipcs -ma. You probably need to really think about setting "good values" for SHMVIRTSIZE along with SHMADD (or google stuff on it). A starting point would be to see what all the "V"'s add up to from onstat -g seg and set SHMVIRTSIZE to that. 2. The error was ENOSPC - I would suggest you look swap configuration. How much swap is allocated to this host? How much memory is IBM Informix running in? (onstat - will show you as well as onstat -g seg giving you a detailed breakdown).What is "RESIDENT" set to in the $ONCONFIG? Have you really configured this instance to "work within resources that are available"? 3. http://www.linuxquestions.org/questions/linux-newbie-8/shmmax-in-ubuntu-860206/ ... SHMMAX is probably the kernel parameter you are looking for.

Hello Art, thank you very much for your fast reply! The memlock default for Ubuntu seems to be 64 according to ulimit -l. I've set it to unlimited ( for informix and root ) as explained here: http://askubuntu.com/questions/396429/changing-ulimit-on-ubuntu-12-04-never-work s Seems to work for every new connection to the host now. To make sure I'll reboot the host at the next opportunity and check again. Best regards, Peter

Hello, thank you very much for your fast reply! > Have a look at onstat -g seg and ipcs -ma. > A starting point would be to see what all the "V"'s add up to from onstat -g seg and set SHMVIRTSIZE to that. Currently those are set in the onconfig file to: SHMVIRTSIZE 262144 SHMADD 32768 onstat -g seg shows 95 virtual segments that have been added: ----------------------------------------------------------------------------- IBM Informix Dynamic Server Version 11.70.FC7 -- On-Line -- Up 21:53:00 --4042644 Kbytes Segment Summary: id key addr size ovhd class blkused blkfree 0 52564801 44000000 682450944 8430968 R 166613 1 32769 52564802 6cad6000 268435456 3147384 V 64909 627 65538 52564803 7cad6000 1110016 14328 M 270 1 98307 52564804 7cbe5000 33554432 394728 V 8192 0 131076 52564805 7ebe5000 33554432 394728 V 8192 0 . . . 3113055 52564860 134be5000 33554432 394728 V 8192 0 3145824 52564861 136be5000 33554432 394728 V 8191 1 3178593 52564862 138be5000 33554432 394728 V 543 7649 Total: - - 4139667456 - - 1002373 8288 ----------------------------------------------------------------------------- Should I really set SHMVIRTSIZE to 3375104 then? I wonder what is adding so much virtual segments automatically around 3 PM after each server restart. Today they have been added from 14:56:36 until 14:59:58. According to OAT these tasks could be responsible: ---------------------------------------------------------------------- mon_users 3672 0.02 100.22 2014-05-15 14:56:32 Success mon_low_storage 14557 0.00 23.09 2014-05-15 14:56:32 Success check_backup 616 0.02 16.69 2014-05-15 14:56:32 Success Alert Cleanup 616 0.03 22.75 2014-05-15 14:56:32 Success Job Results Cleanup 616 0.01 7.41 2014-05-15 14:56:32 Success mon_memory_system 7336 0.01 94.41 2014-05-15 14:56:34 Success mon_checkpoint 14660 0.01 219.92 2014-05-15 14:56:34 Success mon_table_profile 1229 1.49 1833.08 2014-05-15 14:56:34 Success mon_profile 3672 0.01 61.38 2014-05-15 14:56:34 Success mon_command_history 616 0.00 1.29 2014-05-15 14:56:34 Success mon_vps 3672 0.00 22.42 2014-05-15 14:56:34 Success mon_table_names 616 32.60 20083.29 2014-05-15 14:57:28 Success Low Memory Reconfig 14660 0.00 72.84 2014-05-15 15:03:25 Success ---------------------------------------------------------------------- I guess it must be 'Low Memory Reconfig', but then 'Last Run Time' means the last time the task has been finished running? > 2. The error was ENOSPC - I would suggest you look swap configuration. > How much swap is allocated to this host? swapon -s: Filename Type Size Used Priority /dev/sda1 partition 7811068 188 -1 > What is "RESIDENT" set to in the $ONCONFIG? RESIDENT 0 I'll have a look whether Ubuntu supports forcing residency. > Have you really configured this instance to "work within resources that are available"? I guess not really, because of the error. After the problem appeared the first time I set SHMTOTAL to a fixed value and it didn't help. So it's currently back at 0. I guess it was set too low. The current official system requirements aren't that high though: http://www-01.ibm.com/support/docview.wss?rs=630&uid=swg27013343 > 3. http://www.linuxquestions.org/questions/linux-newbie-8/shmmax-in-ubuntu-860206/ ... SHMMAX is probably the kernel parameter you are looking for. It's currently set to 4187361280 - but considering that Informix is using 4042644 Kbytes according to 'onstat -' already 6281041920 might be better? Thank you very much! Best regards, Peter Seifert

Peter: One thing to note is that in 11.70 IBM changed the way that the VP_MEMORY_CACHE_KB works. In 11.50 the value configured for this parameter is fixed and permanent or static. If a VP needed more memory than the VPCACHE setting it would get that from the central memory pool and release the excess when it was finished with it. In 11.70 the value became a starting point with the engine free to increase the maximum amount cached in each CPU VP dynamically. Unfortunately once memory is added to the CPU VPCACHE it is NEVER released back to the central memory pool so if queries need memory for sorting or grouping etc. then engine may have to add more virtual segments. So, it may be that the memory segments being added are needed permanently for large complex queries, but it may just be the new VPCACHE behavior. In 11.70.xC8+ and 12.10 the VP_CACHE_MEMORY_KB was extended to take a second argument of either ,STATIC or ,DYNAMIC (so VP_CACHE_MEMORY_KB 1024,STATIC for example). The latter is the new default behavior and ,STATIC restored the older v11.50 behavior. So you may want to upgrade so you can set the cache back to STATIC behaviors. Once test to see if it may be the VPCACHE is to try to run onmode -F and see if any of the used memory in the virtual segments is freed. If the memory was being used for large queries that have completed, the -F will gather all free bits and pieces of virtual memory together and all the free memory together and free any unneeded virtual segments. If the memory has been assigned to the VPCACHE then it isn't free and no segments will be released. Art Art S. Kagel, Principal Consultant ASK Database Management Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, May 15, 2014 at 10:52 AM, JAN-PETER SEIFERT <seifert@his.de> wrote: > Hello, > > thank you very much for your fast reply! > > > Have a look at onstat -g seg and ipcs -ma. > > > A starting point would be to see what all the "V"'s add up to from > onstat -g> seg and set SHMVIRTSIZE to that. > > Currently those are set in the onconfig file to: > SHMVIRTSIZE 262144 > SHMADD 32768> > onstat -g seg shows 95 virtual segments that have been added:> > ----------------------------------------------------------------------------- > IBM Informix Dynamic Server Version 11.70.FC7 -- On-Line -- Up 21:53:00 --> 4042644 Kbytes > > Segment Summary: > id key addr size ovhd class blkused blkfree > 0 52564801 44000000 682450944 8430968 R 166613 1 > 32769 52564802 6cad6000 268435456 3147384 V 64909 627 > 65538 52564803 7cad6000 1110016 14328 M 270 1 > 98307 52564804 7cbe5000 33554432 394728 V 8192 0 > 131076 52564805 7ebe5000 33554432 394728 V 8192 0 > .. > .. > .. > 3113055 52564860 134be5000 33554432 394728 V 8192 0 > 3145824 52564861 136be5000 33554432 394728 V 8191 1 > 3178593 52564862 138be5000 33554432 394728 V 543 7649 > Total: - - 4139667456 - - 1002373 8288 > > ----------------------------------------------------------------------------- > Should I really set SHMVIRTSIZE to 3375104 then? > > I wonder what is adding so much virtual segments automatically around 3 PM > after each server restart. Today they have been added from 14:56:36 until > 14:59:58. > > According to OAT these tasks could be responsible: > ---------------------------------------------------------------------- > mon_users 3672 0.02 100.22 2014-05-15 14:56:32 Success > mon_low_storage 14557 0.00 23.09 2014-05-15 14:56:32 Success > check_backup 616 0.02 16.69 2014-05-15 14:56:32 Success > Alert Cleanup 616 0.03 22.75 2014-05-15 14:56:32 Success > Job Results Cleanup 616 0.01 7.41 2014-05-15 14:56:32 Success > mon_memory_system 7336 0.01 94.41 2014-05-15 14:56:34 Success > mon_checkpoint 14660 0.01 219.92 2014-05-15 14:56:34 Success > mon_table_profile 1229 1.49 1833.08 2014-05-15 14:56:34 Success > mon_profile 3672 0.01 61.38 2014-05-15 14:56:34 Success > mon_command_history 616 0.00 1.29 2014-05-15 14:56:34 Success > mon_vps 3672 0.00 22.42 2014-05-15 14:56:34 Success > mon_table_names 616 32.60 20083.29 2014-05-15 14:57:28 Success > Low Memory Reconfig 14660 0.00 72.84 2014-05-15 15:03:25 Success > ---------------------------------------------------------------------- > > I guess it must be 'Low Memory Reconfig', but then 'Last Run Time' means > the > last time the task has been finished running? > > > 2. The error was ENOSPC - I would suggest you look swap configuration. > > > How much swap is allocated to this host? > > swapon -s: > Filename Type Size Used Priority > /dev/sda1 partition 7811068 188 -1 > > > What is "RESIDENT" set to in the $ONCONFIG? > RESIDENT 0> > I'll have a look whether Ubuntu supports forcing residency. > > > Have you really configured this instance to "work within resources that > are > available"? > > I guess not really, because of the error. After the problem appeared the > first > time I set SHMTOTAL to a fixed value and it didn't help. So it's currently > back at 0. I guess it was set too low. The current official system > requirements aren't that high though: > http://www-01.ibm.com/support/docview.wss?rs=630&uid=swg27013343 > > > 3. > > http://www.linuxquestions.org/questions/linux-newbie-8/shmmax-in-ubuntu-860206/ > .... SHMMAX is probably the kernel parameter you are looking for. > > It's currently set to 4187361280 - but considering that Informix is using > 4042644 Kbytes according to 'onstat -' already 6281041920 might be better? > > Thank you very much! > > Best regards, > > Peter Seifert > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a11340950fa2f6904f971dba4

Hello Art, thank you very very much for this valuable information/explanation! An 'onmode -F' after 'Low Memory Reconfig'(?) has added new virtual shared memory segments dynamically hardly frees memory resources and no segments are released again. I guess the new default VPCACHE behavior has been introduced to improve performance on NUMA systems though. So taking advantage of this might be better. I wonder whether system settings concerning NUMA should be optimized for Informix - e.g. kernel.sched_migration_cost to prevent process migration between cores? Best regards, Peter

Not really. Numa performance is most strongly improved by using affinity to keep VPs from migrating from one processor group to another to avoid having to reload level 3 cache memory. The new behavior, like several other changes in 11.70, was introduced to improve performance for complex analytical queries. It does that by reducing contention for the central memory pool's single memory latch. Unfortunately, like most of those other changes, it hurts OLTP performance. Art Art S. Kagel, Principal Consultant ASK Database Management Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, May 16, 2014 at 4:46 AM, JAN-PETER SEIFERT <seifert@his.de> wrote: > Hello Art, > > thank you very very much for this valuable information/explanation! > An 'onmode -F' after 'Low Memory Reconfig'(?) has added new virtual shared > memory segments dynamically hardly frees memory resources and no segments > are > released again. I guess the new default VPCACHE behavior has been > introduced > to improve performance on NUMA systems though. So taking advantage of this > might be better. I wonder whether system settings concerning NUMA should be > optimized for Informix - e.g. kernel.sched_migration_cost to prevent > process > migration between cores? > > Best regards, > > Peter > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a11c352da330bbf04f981f477

Hello Art, thanks a lot again! I love getting background information so your blog is a very interesting read for me (filesystem etc.). Best regards, Peter