Informix virtual shared memory
A 9.40.FC4 server on HP-UX 11i kept adding virtual shared memory segments without ever releasing them, eventually crashing with "shmget: [ENOMEM][12] ... out of shared memory"; the site was bouncing the instance every 4-5 days. shmmax (8GB) was ruled out as the cause. Suggestions included using onstat -a / -g stm / -g ses to spot a looping or repeated SQL (big sorts/group-bys, or cursors opened but never freed) whose usage pattern had changed with data volume. One poster identified it as the known 9.40 memory-leak defect IC54412, fixable only via IBM support or an upgrade to IDS 10+; the original poster never confirmed a resolution.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
We are running Informix 9.40.FC4 on HPUX-11i. For the past few weeks,
we are experiencing this problem. The informix server keeps
continuosuly allocating memory and never releases it back and
allocates many virtual shared memory segments and finally runs out of
shared memory and crashes with
" shmget: [ENOMEM][12]: key 5257480f: out of shared memory, check
system max shared memory segment size"
We are now monitoring the memory usage and when it's very low we are
bouncing the server and we have to bounce the server every 4 to 5
days.
The applications have not changed for a very long time and the only
change is in the data volume. We are monitoring the memory used per
session but did not find any thing unusual. These applications are
daemons and are always running.
We have the following settings for shared memory parameters.
SHMBASE 0x0 # Shared memory base address
SHMVIRTSIZE 1048576 # initial virtual shared memory segmentsize
SHMADD 100000 # Size of new shared memory segments
(Kbytes)
SHMTOTAL 0 # Total shared memory (Kbytes).
0=>unlimited
Has any one run into this problem. Any help would be appreciated.
Thanks
"Pankaja Kadakuntla" <pkadakuntla@gmail.com> wrote in message
news:231ce9b9-16c8-4caf-9456-032bbf33abce@h30g2000vbr.googlegroups.com...
> We are running Informix 9.40.FC4 on HPUX-11i. For the past few weeks,
> we are experiencing this problem. The informix server keeps
> continuosuly allocating memory and never releases it back and
> allocates many virtual shared memory segments and finally runs out of
> shared memory and crashes with
> " shmget: [ENOMEM][12]: key 5257480f: out of shared memory, check
> system max shared memory segment size"
> We are now monitoring the memory usage and when it's very low we are
> bouncing the server and we have to bounce the server every 4 to 5
> days.
> The applications have not changed for a very long time and the only
> change is in the data volume. We are monitoring the memory used per
> session but did not find any thing unusual. These applications are
> daemons and are always running.
> We have the following settings for shared memory parameters.
> SHMBASE 0x0 # Shared memory base address
> SHMVIRTSIZE 1048576 # initial virtual shared memory segment> size
> SHMADD 100000 # Size of new shared memory segments
> (Kbytes)
> SHMTOTAL 0 # Total shared memory (Kbytes).
> 0=>unlimited>
> Has any one run into this problem. Any help would be appreciated.
>
> Thanks
And what is the system max shared memory segment size? (You can find its
value in SAM).
On Jul 30, 6:07 pm, "Neil Truby" <neil.tr...@ardenta.com> wrote:
> "Pankaja Kadakuntla" <pkadakun...@gmail.com> wrote in message
>
> news:231ce9b9-16c8-4caf-9456-032bbf33abce@h30g2000vbr.googlegroups.com...
>
>
>
>
>
> > We are running Informix 9.40.FC4 on HPUX-11i. For the past few weeks,
> > we are experiencing this problem. The informix server keeps
> > continuosuly allocating memory and never releases it back and
> > allocates many virtual shared memory segments and finally runs out of
> > shared memory and crashes with
> > " shmget: [ENOMEM][12]: key 5257480f: out of shared memory, check
> > system max shared memory segment size"
> > We are now monitoring the memory usage and when it's very low we are
> > bouncing the server and we have to bounce the server every 4 to 5
> > days.
> > The applications have not changed for a very long time and the only
> > change is in the data volume. We are monitoring the memory used per
> > session but did not find any thing unusual. These applications are
> > daemons and are always running.
> > We have the following settings for shared memory parameters.
> > SHMBASE 0x0 # Shared memory base address
> > SHMVIRTSIZE 1048576 # initial virtual shared memory segment> > size
> > SHMADD 100000 # Size of new shared memory segments
> > (Kbytes)
> > SHMTOTAL 0 # Total shared memory (Kbytes).
> > 0=>unlimited>
> > Has any one run into this problem. Any help would be appreciated.
>
> > Thanks
>
> And what is the system max shared memory segment size? (You can find its
> value in SAM).- Hide quoted text -
>
> - Show quoted text -
Its's 8GB(shmmax 8589934592)
"Pankaja Kadakuntla" <pkadakuntla@gmail.com> wrote in message news:f9925b69-1d99-4ea2-8ac9-345c37a7e3c2@a13g2000yqc.googlegroups.com... On Jul 30, 6:07 pm, "Neil Truby" <neil.tr...@ardenta.com> wrote: > "Pankaja Kadakuntla" <pkadakun...@gmail.com> wrote in message > > news:231ce9b9-16c8-4caf-9456-032bbf33abce@h30g2000vbr.googlegroups.com... > > We are running Informix 9.40.FC4 on HPUX-11i. For the past few weeks, > > we are experiencing this problem. The informix server keeps > > continuosuly allocating memory and never releases it back and > > allocates many virtual shared memory segments and finally runs out of > > shared memory and crashes with > > " shmget: [ENOMEM][12]: key 5257480f: out of shared memory, check > > system max shared memory segment size" >> And what is the system max shared memory segment size? (You can find its > value in SAM).- Hide quoted text - > > - Show quoted text - > Its's 8GB(shmmax 8589934592) Ok well that's higher than the release note value, which is 4294967296 (9.40FC3), so you seem to be OK there.
Neil Truby wrote: > > "Pankaja Kadakuntla" <pkadakuntla@gmail.com> wrote in message > news:f9925b69-1d99-4ea2-8ac9-345c37a7e3c2@a13g2000yqc.googlegroups.com... > On Jul 30, 6:07 pm, "Neil Truby" <neil.tr...@ardenta.com> wrote: >> "Pankaja Kadakuntla" <pkadakun...@gmail.com> wrote in message >> >> news:231ce9b9-16c8-4caf-9456-032bbf33abce@h30g2000vbr.googlegroups.com... > >> > We are running Informix 9.40.FC4 on HPUX-11i. For the past few weeks, >> > we are experiencing this problem. The informix server keeps >> > continuosuly allocating memory and never releases it back and >> > allocates many virtual shared memory segments and finally runs out of >> > shared memory and crashes with >> > " shmget: [ENOMEM][12]: key 5257480f: out of shared memory, check >> > system max shared memory segment size" >>> And what is the system max shared memory segment size? (You can find its >> value in SAM).- Hide quoted text - >> >> - Show quoted text - > >> Its's 8GB(shmmax 8589934592) > > Ok well that's higher than the release note value, which is 4294967296 > (9.40FC3), so you seem to be OK there. Does it allocate them all in one go or does it just gradually grow (i.e. adds a segment a day, or adds 4 in 1 hour?)
We get this occasionally but we managed to narrow it down eventually.
You probably have a query or queries that are performing large sorts
or group-by operations; you need to jump into action as soon as it
starts happening by taking an "onstat -a" and searching through the
(large!) output until you come across the session breakdown (search
for lines beginning "session <nnnnnnn>") - I think this is the output
from "onstat -g stm".
Under here all the running sessions are listed, and in one of them you
will almost certainly find a single sql that is repeated many many
times, each repetition of the query usies a finite amount of virtual
memory and causing the problem - in our case it was an app connecting
via jdbc that was looping.
IDS is notoriously bad at releasing shared memory (even using onmode -
F), which is a persistent nuisance to many of us.
You need to find out where the repeating SQL is coming from by
tracking back the session number to a process using onstat -g ses and
beat the developer up!
Good luck
Malc
Pankaja Kadakuntla wrote:
> We are running Informix 9.40.FC4 on HPUX-11i. For the past few weeks,
> we are experiencing this problem. The informix server keeps
> continuosuly allocating memory and never releases it back and
> allocates many virtual shared memory segments and finally runs out of
> shared memory and crashes with
> " shmget: [ENOMEM][12]: key 5257480f: out of shared memory, check
> system max shared memory segment size"
> We are now monitoring the memory usage and when it's very low we are
> bouncing the server and we have to bounce the server every 4 to 5
> days.
> The applications have not changed for a very long time and the only
> change is in the data volume. We are monitoring the memory used per
> session but did not find any thing unusual. These applications are
> daemons and are always running.
> We have the following settings for shared memory parameters.
> SHMBASE 0x0 # Shared memory base address
> SHMVIRTSIZE 1048576 # initial virtual shared memory segment> size
> SHMADD 100000 # Size of new shared memory segments
> (Kbytes)
> SHMTOTAL 0 # Total shared memory (Kbytes).
> 0=>unlimited>
> Has any one run into this problem. Any help would be appreciated.
I experienced something like this at a client way back on OnLine days.
A monthly invoicing program which had run for years with no problems
suddenly started having its backend run out of memory. The underlying
problem was that cursors were being opened and closed but not freed.
The one causing the problem was buried deep in an inner loop. The usage
pattern had changed: one contract had gone from having a few large
invoices per month to many small ones and this caused that particular
cursor to be called more often. We then discovered that another site
using the same package was running daily invoices as a workaround to the
same problem. We had to go back to the application vendor & explain to
them how to write efficient 4GL.
In your case I'd suggest that you combine SGBoyUK's suggestion with
trying to find whether the pattern of use of your apps has changed even
if the apps themselves haven't.
--
Ian
Hotmail is for spammers. Real mail address is igoddard
at nildram co uk
> You need to find out where the repeating SQL is coming from by
> tracking back the session number to a process using onstat -g ses and
> beat the developer up!
Hope its not something I wrote many years ago
On 1 Aug., 03:30, wcottishpoet <drybur...@yahoo.com> wrote:
> > You need to find out where the repeating SQL is coming from by
> > tracking back the session number to a process using onstat -g ses and
> > beat the developer up!
>
> Hope its not something I wrote many years ago
Hi,
Last year i have the same problem with hp-ux11i and IDS 9.40FC9
description for IC54412:
http://www-01.ibm.com/support/docview.wss?rs=0&context=SSGU8G&dc=DB500&q1=Memory+Leak&uid=swg1IC54412&loc=en_US&cs=utf-8&cc=us&lang=en
the only way to fix this problem is to talk with your IBM/Informix
Support or upgrade to IDS 10 or higher
also be aware, that IDS 9.40 is not longer supported
regards
Manfred