out of virtual shared memory
On IDS 11.50FC6/Solaris, a production instance's virtual shared memory grew rapidly until it hit SHMMAX, logging "out of virtual shared memory", with many sessions in lock wait; the offending session was never identified and the instance was restarted. Respondents suggested the likely cause was runaway lock allocation (locks now cost ~100 bytes each, so lock-table expansion drives big memory growth) and advised checking the online log for lock-table expansions and sessions holding many locks. For alerting, they noted ALARMPROGRAM may not fire on segment allocation, so a script that tails online.log and pattern-matches messages (invoking the alarm program with a custom class ID) can generate such alerts. Side discussion covered wishing for a per-session lock limit and using RAW tables for batch loads. No confirmed root cause or fix was recorded.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
We are running IDS 11.5FC6 on Solaris. Today memory consumption started increasing on our production database and within few minutes it reached the shmmax limit. We found message in online log "out of virtual shared memory", so we restarted the database instance. There were a lot of sessions in lock-wait state. We did not found the culprit session. Can we set alarmprogram to log the information on event when a memory segment is requested/added to IDS instance. What could be the possible reasons of rapid memory consumption while instance in running in stable state for a long time. What are the events in alramprogram.sh for "memory segment request" and "further lock allocation requests". Thanks and regards, Kamran
On Thu, Dec 23, 2010 at 9:45 PM, KAMRAN HAQ <khaq@i2cinc.com> wrote: > We are running IDS 11.5FC6 on Solaris. Today memory consumption started > increasing on our production database and within few minutes it reached the > shmmax limit. We found message in online log > "out of virtual shared memory", so we restarted the database instance. > There were a lot of sessions in lock-wait state. We did not found the > culprit > session. > Can we set alarmprogram to log the information on event when a memory > segment > is requested/added to IDS instance. > What could be the possible reasons of rapid memory consumption while > instance > in running in stable state for a long time. > What are the events in alramprogram.sh for "memory segment request" and > "further lock allocation requests". > Thanks and regards, > Kamran > > shared memory allocation + lots of sessions in lock wait ~= increase in number of locks. "~=" is my way of writing "usually equals".... You should check that if you gathered some info... But if it happened you should see messages in the online log saying that the lock table is being expanded. Locks in recent versions, contrary to older ones tend to consume a reasonable large number of bytes (I believe the manual states around 100 and it used to be 4 in old versions). What this means is that the lock table increase can have a significant impact on memory allocations. And If I'm allowed to say this in public, IBM keeps insensitive to the fact that we don't have a per session limit of locks.... (I've already said it in private more than once). I believe this is a "small" feature with a large (positive) impact on customers.... Maybe customers out there can raise your voice if you agree with me. Regarding the alarmprogram, I can't give you a definitive answer but I believe the alarmprogram is not called when a new shared memory segment is allocated. Having said this, let me share an email I received a couple of days ago: ----------------------- start here -------------------------------------------------------------------------------- ------------------------------------------ This is an automatic generated mail created by the alarm.sh script of the Informix instance XPTO at host XPTO_HOST ------------------------------------------------------------------ Date: Tue Dec 21 12:58:36 PWT 2010 Severity: 3 Class Id: 905 Class desc: Dynamically allocated new memory segment Class msg: Dynamically allocated new memory segment Specific msg: 12/21/10 12:58:31 Dynamically allocated new virtual shared memory segment (size 262144KB) Aditional information: -------------------- Ends here---------------------------------------------------------------------------- --------------------------------------------------------- The class ID (905) is not a standard one. I have a script that monitors the online.log, checks each new line against a configuration script (which includes a string to do the match, a severity and a custom class id) and then if it matches calls the alarm program with the appropriate arguments. So, assuming it does not call automatically, there's a way to do it. I can give more details, or share the script if needed. In any case, verify the information you collected and see if any session had a large number of locks. Regards. -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --000e0cd1fa5894949604981d1a39
If you write your ALARM program (or shell script) to trap even the MESSAGE types, you can try to pattern match for those online log messages. I've done it for HDR connecting and failing messages before since HDR breaking causes an alarm but the resyncing does not. Bob ----- Original Message ----- From: "Fernando Nunes" <domusonline@gmail.com> To: ids@iiug.org Sent: Thursday, December 23, 2010 7:31:21 PM Subject: Re: out of virtual shared memory [22299] On Thu, Dec 23, 2010 at 9:45 PM, KAMRAN HAQ <khaq@i2cinc.com> wrote: > We are running IDS 11.5FC6 on Solaris. Today memory consumption started > increasing on our production database and within few minutes it reached the > shmmax limit. We found message in online log > "out of virtual shared memory", so we restarted the database instance. > There were a lot of sessions in lock-wait state. We did not found the > culprit > session. > Can we set alarmprogram to log the information on event when a memory > segment > is requested/added to IDS instance. > What could be the possible reasons of rapid memory consumption while > instance > in running in stable state for a long time. > What are the events in alramprogram.sh for "memory segment request" and > "further lock allocation requests". > Thanks and regards, > Kamran > > shared memory allocation + lots of sessions in lock wait ~= increase in number of locks. "~=" is my way of writing "usually equals".... You should check that if you gathered some info... But if it happened you should see messages in the online log saying that the lock table is being expanded. Locks in recent versions, contrary to older ones tend to consume a reasonable large number of bytes (I believe the manual states around 100 and it used to be 4 in old versions). What this means is that the lock table increase can have a significant impact on memory allocations. And If I'm allowed to say this in public, IBM keeps insensitive to the fact that we don't have a per session limit of locks.... (I've already said it in private more than once). I believe this is a "small" feature with a large (positive) impact on customers.... Maybe customers out there can raise your voice if you agree with me. Regarding the alarmprogram, I can't give you a definitive answer but I believe the alarmprogram is not called when a new shared memory segment is allocated. Having said this, let me share an email I received a couple of days ago: ----------------------- start here -------------------------------------------------------------------------------- ------------------------------------------ This is an automatic generated mail created by the alarm.sh script of the Informix instance XPTO at host XPTO_HOST ------------------------------------------------------------------ Date: Tue Dec 21 12:58:36 PWT 2010 Severity: 3 Class Id: 905 Class desc: Dynamically allocated new memory segment Class msg: Dynamically allocated new memory segment Specific msg: 12/21/10 12:58:31 Dynamically allocated new virtual shared memory segment (size 262144KB) Aditional information: -------------------- Ends here---------------------------------------------------------------------------- --------------------------------------------------------- The class ID (905) is not a standard one. I have a script that monitors the online.log, checks each new line against a configuration script (which includes a string to do the match, a severity and a custom class id) and then if it matches calls the alarm program with the appropriate arguments. So, assuming it does not call automatically, there's a way to do it. I can give more details, or share the script if needed. In any case, verify the information you collected and see if any session had a large number of locks. Regards. -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --000e0cd1fa5894949604981d1a39 ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.
Thanks guys for suggesting the way to get event alarms. Fernando! Your suggestion can help in pure OLTP systems. But don't you think that if we put a max locks limit per session on a hybrid (OLTP+OLAP) systems, it may lead to complex issues and might be it degrade the IDS performance as it has to do extra management?
On Fri, Dec 24, 2010 at 10:43 AM, KAMRAN HAQ <khaq@i2cinc.com> wrote: > Thanks guys for suggesting the way to get event alarms. > Fernando! Your suggestion can help in pure OLTP systems. But don't you > think > that if we put a max locks limit per session on a hybrid (OLTP+OLAP) > systems, > it may lead to complex issues and might be it degrade the IDS performance > as > it has to do extra management? > Hello... No... I don't agree with that... I believe it will take one machine code instruction to check that against a session limit for each lock request. It's largely irrelevant. And when you setup the max locks per instance you're thinking that it should be enough for all the sessions. If someone makes a mistake, it can consume all the locks and that yes, will have a large impact on the instance... If you're working with large loads you don't want to have locks for each row anyway... So you'll use raw tables (if no MACH 11 involved), or you'll lock the whole table etc... Regards. -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --00151748dfea0b3d5d04982a558b
Or you will alter the tables involved to RAW before the batch run and alter them back to STANDARD after the batch job completes. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) IIUG Board of Directors (art@iiug.org) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Dec 24, 2010 at 11:18 AM, Fernando Nunes <domusonline@gmail.com>wrote: > On Fri, Dec 24, 2010 at 10:43 AM, KAMRAN HAQ <khaq@i2cinc.com> wrote: > > > Thanks guys for suggesting the way to get event alarms. > > Fernando! Your suggestion can help in pure OLTP systems. But don't you > > think > > that if we put a max locks limit per session on a hybrid (OLTP+OLAP) > > systems, > > it may lead to complex issues and might be it degrade the IDS performance > > as > > it has to do extra management? > > > > Hello... No... I don't agree with that... > I believe it will take one machine code instruction to check that against a > session limit for each lock request. It's largely irrelevant. > And when you setup the max locks per instance you're thinking that it > should > be enough for all the sessions. If someone makes a mistake, it can consume > all the locks and that yes, will have a large impact on the instance... > If you're working with large loads you don't want to have locks for each > row > anyway... So you'll use raw tables (if no MACH 11 involved), or you'll lock > the whole table etc... > Regards. > -- > Fernando Nunes > Portugal > > http://informix-technology.blogspot.com > My email works... but I don't check it frequently... > > --00151748dfea0b3d5d04982a558b > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --20cf30433efe00c8a104982ab781