RESIDENT onconfig Parameter and Multiple Instances
Richard ran two Informix instances on CentOS 6.3, each with 8GB and RESIDENT=-1, but the second instance failed with shmget ENOMEM. Fernando and Art suggested OS limits on locked memory (RLIMIT_MEMLOCK, ulimit, /etc/security/limits.conf, PAM), but raising those didn't help. Richard found the real cause: Linux HugePages. He had configured only 4096 pages of 2MB (8GB), enough for one instance; raising it to 8192 (16GB) let both instances start with memory locked.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Good Morning All! Long-time reader, first-time poster. I'm working on setting up two instances of Informix on a single server (32 gigs of ram, 32 cores, but since I'm running the Growth Edition of Informix, I can only use 16 gigs of the RAM...). I was planning on allocating 8gb of RAM to each instance, and enabling the RESIDENT parameter to lock that RAM in Informix. The first instance is successful with that, 8 gigs allocated to it and held by Informix (RESIDENT set to -1), but the second instance will not let me do this. The only way I can get the second instance to start is if I leave RESIDENT disabled on it. So currently I have the first instance and second instance both running, but only the first instance has the RAM locked by Informix using the RESIDENT parameter. I've found no documentation saying this isn't allowed for multiple instances, so I was wondering if any of you had any input on the matter? Thank you so much for your help. Richard
What is the error? And what is the operating system? Some operating systems (if not all) have configuration parameters to define how much memory you can request to "lock". AIX is one example. Regards. On Tue, Jul 24, 2012 at 3:52 PM, RICHARD GUFFEY <reguffey@gmail.com> wrote: > Good Morning All! > > Long-time reader, first-time poster. I'm working on setting up two > instances > of Informix on a single server (32 gigs of ram, 32 cores, but since I'm > running the Growth Edition of Informix, I can only use 16 gigs of the > RAM...). > I was planning on allocating 8gb of RAM to each instance, and enabling the > RESIDENT parameter to lock that RAM in Informix. The first instance is > successful with that, 8 gigs allocated to it and held by Informix (RESIDENT > set to -1), but the second instance will not let me do this. The only way I > can get the second instance to start is if I leave RESIDENT disabled on > it. So > currently I have the first instance and second instance both running, but > only > the first instance has the RAM locked by Informix using the RESIDENT > parameter. I've found no documentation saying this isn't allowed for > multiple > instances, so I was wondering if any of you had any input on the matter? > Thank > you so much for your help. > > Richard > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --20cf303b3f51fb286104c594b74f
Usually this is a problem with an OS parameter preventing that much locked memory. Under the 2.4 through 2.6.8 kernel a user cannot lock more than 50% of physical memory. Under the 2.6.9 and later kernels there is an RLIMIT_MEMLOCK setting per user session that controls this. There are also kernel parameter settings controlling this system wide and per-session. Not a Linux Admin, so... Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Tue, Jul 24, 2012 at 10:52 AM, RICHARD GUFFEY <reguffey@gmail.com> wrote: > Good Morning All! > > Long-time reader, first-time poster. I'm working on setting up two > instances > of Informix on a single server (32 gigs of ram, 32 cores, but since I'm > running the Growth Edition of Informix, I can only use 16 gigs of the > RAM...). > I was planning on allocating 8gb of RAM to each instance, and enabling the > RESIDENT parameter to lock that RAM in Informix. The first instance is > successful with that, 8 gigs allocated to it and held by Informix (RESIDENT > set to -1), but the second instance will not let me do this. The only way I > can get the second instance to start is if I leave RESIDENT disabled on > it. So > currently I have the first instance and second instance both running, but > only > the first instance has the RAM locked by Informix using the RESIDENT > parameter. I've found no documentation saying this isn't allowed for > multiple > instances, so I was wondering if any of you had any input on the matter? > Thank > you so much for your help. > > Richard > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --bcaec51a758e7d43c604c595b0a2
My uname -a output is this: Linux server_name 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux I'm running centOS 6.3. The error in the online.log is the following: shmget: [ENOMEM][12]: key 52584801: The server ran out of memory during an attempt to allocate a new shared memory segment. I'm looking into the RLIMIT_MEMLOCK kernel parameter Art had mentioned...
Yes... I didn't notice the OS in the first email. I think Art suggestion should work (as usual) If not, please tell us. Regards On Tue, Jul 24, 2012 at 8:25 PM, RICHARD GUFFEY <reguffey@gmail.com> wrote: > My uname -a output is this: > Linux server_name 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012 > x86_64 x86_64 x86_64 GNU/Linux > > I'm running centOS 6.3. > > The error in the online.log is the following: > shmget: [ENOMEM][12]: key 52584801: The server ran out of memory during an > attempt to allocate a new shared memory segment. > > I'm looking into the RLIMIT_MEMLOCK kernel parameter Art had mentioned... > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --20cf303b3f51232ed604c5999557
I did a bit of search and noticed that apparently on Linux this can be changed with ulimit (although there should be some more permanent way to settle it). I find it a bit strange that I never hit this one before... (on Linux). So, assuming you're just making some tests you could see the current value using ulimit -aH and/or ulimit -a and then try to ajust it in the two instances (split the value in half a decrement it a bit because of possible roundups) (if you don't want to bother with the above, just take the current value for the first instance and use it for calculations) If both instances start with RESIDENT setup you found the cause and can work on the solution. Regards On Tue, Jul 24, 2012 at 9:55 PM, Fernando Nunes <domusonline@gmail.com>wrote: > Yes... I didn't notice the OS in the first email. > I think Art suggestion should work (as usual) > If not, please tell us. > Regards > > > > On Tue, Jul 24, 2012 at 8:25 PM, RICHARD GUFFEY <reguffey@gmail.com>wrote: > >> My uname -a output is this: >> Linux server_name 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC >> 2012 >> x86_64 x86_64 x86_64 GNU/Linux >> >> I'm running centOS 6.3. >> >> The error in the online.log is the following: >> shmget: [ENOMEM][12]: key 52584801: The server ran out of memory during an >> attempt to allocate a new shared memory segment. >> >> I'm looking into the RLIMIT_MEMLOCK kernel parameter Art had mentioned... >> >> >> >> ******************************************************************************* >> Forum Note: Use "Reply" to post a response in the discussion forum. >> >> > > > -- > Fernando Nunes > Portugal > > http://informix-technology.blogspot.com > My email works... but I don't check it frequently... > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --00235452ea583e2c9a04c599c4d3
I've been doing some research as well, and discovered these parameters can be changed in /etc/security/limits.conf file... I've changed the memlock, rss, and as services values to unlimited, but am still having the same issue...out of curiosity, what is the overhead of Informix, aside from the memory you allocate in the onconfig parameters? If I set the SHMTOTAL to 8gb, that's a combination of the virtual and resident memory segments for that instance of Informix, but is there other memory overhead aside from that for the instance? How much...? Is it proportional to the amount of shared memory allocated?
Hmmm.... That file accordingly to some sources is used by the pam_limit module. Do you have it setup on your PAM stack? Did you check the effects of those changes with ulimit -a and ulimit -aH on a newly created session? Did you set up hard or soft limits? For which user/group/wildcard? Could you post the configuration lines you added, the PAM configuration stack and the output of ulimit -a and ulimit -aH? There should be no overhead... SHMTOTAL as the name implies should be the sum of all the shared memory segments. Individual processes will need their own private memory, but that's not enforced by SHMTOTAL. Regards On Tue, Jul 24, 2012 at 10:43 PM, RICHARD GUFFEY <reguffey@gmail.com> wrote: > I've been doing some research as well, and discovered these parameters can > be > changed in /etc/security/limits.conf file... I've changed the memlock, rss, > and as services values to unlimited, but am still having the same > issue...out > of curiosity, what is the overhead of Informix, aside from the memory you > allocate in the onconfig parameters? If I set the SHMTOTAL to 8gb, that's a > combination of the virtual and resident memory segments for that instance > of > Informix, but is there other memory overhead aside from that for the > instance? > How much...? Is it proportional to the amount of shared memory allocated? > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --00248c6a66d67096de04c59a6b59
Solved! As it turns out, at least on the Growth Edition of Informix, when it is installed, it sets the maximum memory allowed to be locked by user informix to 16 gigs. So making changes to my limits.conf file was useless. My problem was in the use of Hugepages. If you use Hugepages, you must be sure you have enough of them to cover the amount of memory you'll be allocating. When I set mine up, I had picked an arbitrary number to set the value to based on some documentation I had been reading in my reasearch (4096), which with a 2048 kB Hugepage size (found out by cat /proc/meminfo), only gave me 8 gigs worth of RAM in Hugepages. So once I tried to start a second instance of Informix, and it tried to grab Hugepages for memory allocation, it was out of them, and thus the error. All I had to do to fix it was give the OS enough Hugepages to cover the RAM I'd be allocating, so I set it up to 8192, giving me 16 gigs of RAM in Hugepages. Now I can start both instances and both of them have 8 gigs of RAM locked using the RESIDENT parameter in onconfig. Fernando, Art, thank you both for your input, and your time. I look forward to becoming a more active member in this forum...hopefully more in the advisory sense than the needing help sense!