Panic analysis
Frank's IDS 12.10.FC4 on Linux crashed after a storm of 4 GB virtual shared-memory segment allocations, ending with a fatal error in the ADM VP (mt.c:14304) and a VP killed by signal 9. Replies identified the kill as the Linux OOM killer reacting to runaway memory growth, caused by known IDS bugs. Jeff Filippi suggested upgrading to 12.10.FC5 or setting IFX_NOPENPS=4096 to cap open-table growth; Art Kagel noted several related bugs (all ports) triggered by sysmaster queries scanning partition header pages (sysextents, systabnames, sysptnhdr, sysactptnhdr), which allocate per-session cache pages. Workaround: avoid such sysmaster scans until patched.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Folks,
IDS 12.10 FC4, Linux 6.
The following log is the only info we have about the crash ( no other
info dumped )
You can see it was struggling to add lot memories segments before
Panic ....
The Panic was a also VERY curious one with some unexpected process
terminated ...
What are your comments ? What might be the culprit ? ....
Thanks
Frank
.........................................
21:10:26 Dynamically allocated new virtual shared memory segment (size
2097152KB)
21:10:26 Memory sizes:resident:114076 KB, virtual:84996324 KB, no SHMTOTAL
limit
21:10:28 Dynamically allocated new virtual shared memory segment (size
4194304KB)
21:10:28 Memory sizes:resident:114076 KB, virtual:89190628 KB, no SHMTOTAL
limit
21:10:30 Dynamically allocated new virtual shared memory segment (size
4194304KB)
21:10:30 Memory sizes:resident:114076 KB, virtual:93384932 KB, no SHMTOTAL
limit
21:10:33 Dynamically allocated new virtual shared memory segment (size
4194304KB)
21:10:33 Memory sizes:resident:114076 KB, virtual:97579236 KB, no SHMTOTAL
limit
21:10:36 Logical Log 279454 Complete, timestamp: 0x5e24c0a1.
21:10:36 Dynamically allocated new virtual shared memory segment (size
4194304KB)
21:10:36 Memory sizes:resident:114076 KB, virtual:101773540 KB, no
SHMTOTAL limit21:10:38 Dynamically allocated new virtual shared memory segment (size
4194304KB)
21:10:38 Memory sizes:resident:114076 KB, virtual:105967844 KB, no
SHMTOTAL limit21:10:41 Dynamically allocated new virtual shared memory segment (size
4194304KB)
21:10:41 Memory sizes:resident:114076 KB, virtual:110162148 KB, no
SHMTOTAL limit21:10:44 Dynamically allocated new virtual shared memory segment (size
4194304KB)
21:10:44 Memory sizes:resident:114076 KB, virtual:114356452 KB, no
SHMTOTAL limit21:10:46 Dynamically allocated new virtual shared memory segment (size
4194304KB)
21:10:46 Memory sizes:resident:114076 KB, virtual:118550756 KB, no
SHMTOTAL limit21:10:49 Dynamically allocated new virtual shared memory segment (size
4194304KB)
21:10:49 Memory sizes:resident:114076 KB, virtual:122745060 KB, no
SHMTOTAL limit21:10:52 Dynamically allocated new virtual shared memory segment (size
4194304KB)
21:10:52 Memory sizes:resident:114076 KB, virtual:126939364 KB, no
SHMTOTAL limit21:10:55 Dynamically allocated new virtual shared memory segment (size
4194304KB)
21:10:55 Memory sizes:resident:114076 KB, virtual:131133668 KB, no
SHMTOTAL limit21:12:29 Dynamically allocated new virtual shared memory segment (size
4194304KB)
21:12:29 Memory sizes:resident:114076 KB, virtual:135327972 KB, no
SHMTOTAL limit21:12:51 Dynamically allocated new virtual shared memory segment (size
4194304KB)
21:12:51 Memory sizes:resident:114076 KB, virtual:139522276 KB, no
SHMTOTAL limit21:12:56 Fatal error in ADM VP at mt.c:14304
21:12:56 Unexpected virtual processor termination: pid = 9812, signal 9
received.
21:12:57 PANIC: Attempting to bring system down
--001a11c119e20474cb0518e09ffe
Its the OOM=20
Linux has an Out Of Memory tool.
Signal 9 is the clue. Informix never issues a signal 8, or kill -9, in its c=
ode.
Basically you Os OOM manager felt too much memory consumed and it killed the=
culprit=20
Sent from my iPhone
> On Jun 19, 2015, at 10:31 AM, FRANK <yunyaoqu@gmail.com> wrote:
>=20
> Folks,=20
>=20
> IDS 12.10 FC4, Linux 6.=20
>=20
> The following log is the only info we have about the crash ( no other=20
> info dumped )=20
> You can see it was struggling to add lot memories segments before=20
> Panic ....=20
> The Panic was a also VERY curious one with some unexpected process=20
> terminated ...=20
>=20
> What are your comments ? What might be the culprit ? ....=20
>=20
> Thanks=20
> Frank=20
>=20
> ..........................................=20
> 21:10:26 Dynamically allocated new virtual shared memory segment (size=20
> 2097152KB)=20
> 21:10:26 Memory sizes:resident:114076 KB, virtual:84996324 KB, no SHMTOTAL=
=20
> limit=20
> 21:10:28 Dynamically allocated new virtual shared memory segment (size=20
> 4194304KB)=20
> 21:10:28 Memory sizes:resident:114076 KB, virtual:89190628 KB, no SHMTOTAL=
=20
> limit=20
> 21:10:30 Dynamically allocated new virtual shared memory segment (size=20
> 4194304KB)=20
> 21:10:30 Memory sizes:resident:114076 KB, virtual:93384932 KB, no SHMTOTAL=
=20
> limit=20
> 21:10:33 Dynamically allocated new virtual shared memory segment (size=20
> 4194304KB)=20
> 21:10:33 Memory sizes:resident:114076 KB, virtual:97579236 KB, no SHMTOTAL=
=20
> limit=20
> 21:10:36 Logical Log 279454 Complete, timestamp: 0x5e24c0a1.=20
> 21:10:36 Dynamically allocated new virtual shared memory segment (size=20
> 4194304KB)=20
> 21:10:36 Memory sizes:resident:114076 KB, virtual:101773540 KB, no=20
> SHMTOTAL limit=20> 21:10:38 Dynamically allocated new virtual shared memory segment (size=20
> 4194304KB)=20
> 21:10:38 Memory sizes:resident:114076 KB, virtual:105967844 KB, no=20
> SHMTOTAL limit=20> 21:10:41 Dynamically allocated new virtual shared memory segment (size=20
> 4194304KB)=20
> 21:10:41 Memory sizes:resident:114076 KB, virtual:110162148 KB, no=20
> SHMTOTAL limit=20> 21:10:44 Dynamically allocated new virtual shared memory segment (size=20
> 4194304KB)=20
> 21:10:44 Memory sizes:resident:114076 KB, virtual:114356452 KB, no=20
> SHMTOTAL limit=20> 21:10:46 Dynamically allocated new virtual shared memory segment (size=20
> 4194304KB)=20
> 21:10:46 Memory sizes:resident:114076 KB, virtual:118550756 KB, no=20
> SHMTOTAL limit=20> 21:10:49 Dynamically allocated new virtual shared memory segment (size=20
> 4194304KB)=20
> 21:10:49 Memory sizes:resident:114076 KB, virtual:122745060 KB, no=20
> SHMTOTAL limit=20> 21:10:52 Dynamically allocated new virtual shared memory segment (size=20
> 4194304KB)=20
> 21:10:52 Memory sizes:resident:114076 KB, virtual:126939364 KB, no=20
> SHMTOTAL limit=20> 21:10:55 Dynamically allocated new virtual shared memory segment (size=20
> 4194304KB)=20
> 21:10:55 Memory sizes:resident:114076 KB, virtual:131133668 KB, no=20
> SHMTOTAL limit=20> 21:12:29 Dynamically allocated new virtual shared memory segment (size=20
> 4194304KB)=20
> 21:12:29 Memory sizes:resident:114076 KB, virtual:135327972 KB, no=20
> SHMTOTAL limit=20> 21:12:51 Dynamically allocated new virtual shared memory segment (size=20
> 4194304KB)=20
> 21:12:51 Memory sizes:resident:114076 KB, virtual:139522276 KB, no=20
> SHMTOTAL limit=20
> 21:12:56 Fatal error in ADM VP at mt.c:14304=20> 21:12:56 Unexpected virtual processor termination: pid =3D 9812, signal 9=20=
> received.=20
> 21:12:57 PANIC: Attempting to bring system down=20
>=20
> --001a11c119e20474cb0518e09ffe=20
>=20
>=20
> **************************************************************************=
*****=20
> Forum Note: Use "Reply" to post a response in the discussion forum.=20
>=20
I had some customers have the same thing happen with 12.10FC4, there is a bug. Either upgrade to 12.10FC5 or try setting the environment variable "IFX_NOPENPS=4096;export IFX_NOPENPS" This helped the issue with my customers. Thanks, Jeff -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of FRANK Sent: Friday, June 19, 2015 10:32 AM To: ids@iiug.org Subject: Panic analysis [35298] Folks, IDS 12.10 FC4, Linux 6. The following log is the only info we have about the crash ( no other info dumped ) You can see it was struggling to add lot memories segments before Panic .... The Panic was a also VERY curious one with some unexpected process terminated ... What are your comments ? What might be the culprit ? .... Thanks Frank .......................................... 21:10:26 Dynamically allocated new virtual shared memory segment (size 2097152KB) 21:10:26 Memory sizes:resident:114076 KB, virtual:84996324 KB, no SHMTOTAL limit 21:10:28 Dynamically allocated new virtual shared memory segment (size 4194304KB) 21:10:28 Memory sizes:resident:114076 KB, virtual:89190628 KB, no SHMTOTAL limit 21:10:30 Dynamically allocated new virtual shared memory segment (size 4194304KB) 21:10:30 Memory sizes:resident:114076 KB, virtual:93384932 KB, no SHMTOTAL limit 21:10:33 Dynamically allocated new virtual shared memory segment (size 4194304KB) 21:10:33 Memory sizes:resident:114076 KB, virtual:97579236 KB, no SHMTOTAL limit 21:10:36 Logical Log 279454 Complete, timestamp: 0x5e24c0a1. 21:10:36 Dynamically allocated new virtual shared memory segment (size 4194304KB) 21:10:36 Memory sizes:resident:114076 KB, virtual:101773540 KB, no SHMTOTAL limit 21:10:38 Dynamically allocated new virtual shared memory segment (size 4194304KB) 21:10:38 Memory sizes:resident:114076 KB, virtual:105967844 KB, no SHMTOTAL limit 21:10:41 Dynamically allocated new virtual shared memory segment (size 4194304KB) 21:10:41 Memory sizes:resident:114076 KB, virtual:110162148 KB, no SHMTOTAL limit 21:10:44 Dynamically allocated new virtual shared memory segment (size 4194304KB) 21:10:44 Memory sizes:resident:114076 KB, virtual:114356452 KB, no SHMTOTAL limit 21:10:46 Dynamically allocated new virtual shared memory segment (size 4194304KB) 21:10:46 Memory sizes:resident:114076 KB, virtual:118550756 KB, no SHMTOTAL limit 21:10:49 Dynamically allocated new virtual shared memory segment (size 4194304KB) 21:10:49 Memory sizes:resident:114076 KB, virtual:122745060 KB, no SHMTOTAL limit 21:10:52 Dynamically allocated new virtual shared memory segment (size 4194304KB) 21:10:52 Memory sizes:resident:114076 KB, virtual:126939364 KB, no SHMTOTAL limit 21:10:55 Dynamically allocated new virtual shared memory segment (size 4194304KB) 21:10:55 Memory sizes:resident:114076 KB, virtual:131133668 KB, no SHMTOTAL limit 21:12:29 Dynamically allocated new virtual shared memory segment (size 4194304KB) 21:12:29 Memory sizes:resident:114076 KB, virtual:135327972 KB, no SHMTOTAL limit 21:12:51 Dynamically allocated new virtual shared memory segment (size 4194304KB) 21:12:51 Memory sizes:resident:114076 KB, virtual:139522276 KB, no SHMTOTAL limit 21:12:56 Fatal error in ADM VP at mt.c:14304 21:12:56 Unexpected virtual processor termination: pid = 9812, signal 9 received. 21:12:57 PANIC: Attempting to bring system down --001a11c119e20474cb0518e09ffe **************************************************************************** *** Forum Note: Use "Reply" to post a response in the discussion forum.
There are actually three bugs that cause memory allocation storms, the one Jeff mentioned and two others that mostly affect servers with many thousands of partitions (tables, indexes, partitions of tables or indexes) and occur when you try to access sysmaster tables that query the partition header pages (so systabnames, sysptnhdr, sysactptnhdr, sysextents, etc.). They are all fixed in v12.10.FC5 and the latest fixpacks for v11.70 and 11.50. Ran into all three at around the same time this past fall. Temporary work around: avoid querying sysmaster data about tables. Dbschema, myschema, dostats, some OAT sensors, and other utilities are some things that can trigger this depending on what options you pass in. Art Art Art S. Kagel, President and Principal Consultant ASK Database Management www.askdbmgt.com Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Jun 19, 2015 at 10:47 AM, Jeff Filippi <iiug@itdataconsulting.com> wrote: > I had some customers have the same thing happen with 12.10FC4, there is a > bug. > > Either upgrade to 12.10FC5 or try setting the environment variable > "IFX_NOPENPS=4096;export IFX_NOPENPS" > > This helped the issue with my customers. > > Thanks, Jeff > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > FRANK > Sent: Friday, June 19, 2015 10:32 AM > To: ids@iiug.org > Subject: Panic analysis [35298] > > Folks, > > IDS 12.10 FC4, Linux 6. > > The following log is the only info we have about the crash ( no other info > dumped ) You can see it was struggling to add lot memories segments before > Panic .... > The Panic was a also VERY curious one with some unexpected process > terminated ... > > What are your comments ? What might be the culprit ? .... > > Thanks > Frank > > ........................................... > 21:10:26 Dynamically allocated new virtual shared memory segment (size > 2097152KB) > 21:10:26 Memory sizes:resident:114076 KB, virtual:84996324 KB, no SHMTOTAL > limit > 21:10:28 Dynamically allocated new virtual shared memory segment (size > 4194304KB) > 21:10:28 Memory sizes:resident:114076 KB, virtual:89190628 KB, no SHMTOTAL > limit > 21:10:30 Dynamically allocated new virtual shared memory segment (size > 4194304KB) > 21:10:30 Memory sizes:resident:114076 KB, virtual:93384932 KB, no SHMTOTAL > limit > 21:10:33 Dynamically allocated new virtual shared memory segment (size > 4194304KB) > 21:10:33 Memory sizes:resident:114076 KB, virtual:97579236 KB, no SHMTOTAL > limit > 21:10:36 Logical Log 279454 Complete, timestamp: 0x5e24c0a1. > 21:10:36 Dynamically allocated new virtual shared memory segment (size > 4194304KB) > 21:10:36 Memory sizes:resident:114076 KB, virtual:101773540 KB, no SHMTOTAL > limit > 21:10:38 Dynamically allocated new virtual shared memory segment (size > 4194304KB) > 21:10:38 Memory sizes:resident:114076 KB, virtual:105967844 KB, no SHMTOTAL > limit > 21:10:41 Dynamically allocated new virtual shared memory segment (size > 4194304KB) > 21:10:41 Memory sizes:resident:114076 KB, virtual:110162148 KB, no SHMTOTAL > limit > 21:10:44 Dynamically allocated new virtual shared memory segment (size > 4194304KB) > 21:10:44 Memory sizes:resident:114076 KB, virtual:114356452 KB, no SHMTOTAL > limit > 21:10:46 Dynamically allocated new virtual shared memory segment (size > 4194304KB) > 21:10:46 Memory sizes:resident:114076 KB, virtual:118550756 KB, no SHMTOTAL > limit > 21:10:49 Dynamically allocated new virtual shared memory segment (size > 4194304KB) > 21:10:49 Memory sizes:resident:114076 KB, virtual:122745060 KB, no SHMTOTAL > limit > 21:10:52 Dynamically allocated new virtual shared memory segment (size > 4194304KB) > 21:10:52 Memory sizes:resident:114076 KB, virtual:126939364 KB, no SHMTOTAL > limit > 21:10:55 Dynamically allocated new virtual shared memory segment (size > 4194304KB) > 21:10:55 Memory sizes:resident:114076 KB, virtual:131133668 KB, no SHMTOTAL > limit > 21:12:29 Dynamically allocated new virtual shared memory segment (size > 4194304KB) > 21:12:29 Memory sizes:resident:114076 KB, virtual:135327972 KB, no SHMTOTAL > limit > 21:12:51 Dynamically allocated new virtual shared memory segment (size > 4194304KB) > 21:12:51 Memory sizes:resident:114076 KB, virtual:139522276 KB, no SHMTOTAL > limit > 21:12:56 Fatal error in ADM VP at mt.c:14304 > 21:12:56 Unexpected virtual processor termination: pid = 9812, signal 9 > received. > 21:12:57 PANIC: Attempting to bring system down > > --001a11c119e20474cb0518e09ffe > > > **************************************************************************** > *** > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a1140727a36dabc0518e1d019
Art / all - Are the bugs you mention specific to Linux or all IDS pre-12.1xC5 ports? Thanks - Mark The Mark Scranton Group www.markscranton.com
All ports. Again, they have the most serious effect when the server has many many thousands of partitions in it. Also a similar fourth bug affecting extensive external table use causing the server to run out of file handles. Art On Jun 22, 2015 8:36 AM, "MARK SCRANTON" <mark@markscranton.com> wrote: > Art / all - > > Are the bugs you mention specific to Linux or all IDS pre-12.1xC5 ports? > > Thanks - > Mark > The Mark Scranton Group > www.markscranton.com > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a113321eeee046f05191ae553
Thank Art, Jeff and Mark for the info!
We do not do partition/fragmentation.
We are heavily using the following sysmaster tables to monitor space usage
daily,
sysdbspaces
syschunks
sysextents
Seems difficult to avoid using them.
By the way, What does IFX_NOPENPS=4096 mean? Any possible side effect?
Thanks
Frank
On Fri, Jun 19, 2015 at 12:57 PM, Art Kagel <art.kagel@gmail.com> wrote:
> There are actually three bugs that cause memory allocation storms, the one
> Jeff mentioned and two others that mostly affect servers with many
> thousands of partitions (tables, indexes, partitions of tables or indexes)
> and occur when you try to access sysmaster tables that query the partition
> header pages (so systabnames, sysptnhdr, sysactptnhdr, sysextents, etc.).
> They are all fixed in v12.10.FC5 and the latest fixpacks for v11.70 and
> 11.50. Ran into all three at around the same time this past fall.
>
> Temporary work around: avoid querying sysmaster data about tables.
> Dbschema, myschema, dostats, some OAT sensors, and other utilities are some
> things that can trigger this depending on what options you pass in.
>
> Art
>
> Art
>
> Art S. Kagel, President and Principal Consultant
> ASK Database Management
> www.askdbmgt.com
>
> Blog: http://informix-myview.blogspot.com/
>
> Disclaimer: Please keep in mind that my own opinions are my own opinions
> and do not reflect on the IIUG, nor any other organization with which I am
> associated either explicitly, implicitly, or by inference. Neither do
> those opinions reflect those of other individuals affiliated with any
> entity with which I am affiliated nor those of the entities themselves.
>
> On Fri, Jun 19, 2015 at 10:47 AM, Jeff Filippi <iiug@itdataconsulting.com>
> wrote:
>
> > I had some customers have the same thing happen with 12.10FC4, there is a
> > bug.
> >
> > Either upgrade to 12.10FC5 or try setting the environment variable
> > "IFX_NOPENPS=4096;export IFX_NOPENPS"
> >
> > This helped the issue with my customers.
> >
> > Thanks, Jeff
> > -----Original Message-----
> > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of
> > FRANK
> > Sent: Friday, June 19, 2015 10:32 AM
> > To: ids@iiug.org
> > Subject: Panic analysis [35298]
> >
> > Folks,
> >
> > IDS 12.10 FC4, Linux 6.
> >
> > The following log is the only info we have about the crash ( no other
> info
> > dumped ) You can see it was struggling to add lot memories segments
> before
> > Panic ....
> > The Panic was a also VERY curious one with some unexpected process
> > terminated ...
> >
> > What are your comments ? What might be the culprit ? ....
> >
> > Thanks
> > Frank
> >
> > ...........................................
> > 21:10:26 Dynamically allocated new virtual shared memory segment (size
> > 2097152KB)
> > 21:10:26 Memory sizes:resident:114076 KB, virtual:84996324 KB, no
> SHMTOTAL> > limit
> > 21:10:28 Dynamically allocated new virtual shared memory segment (size
> > 4194304KB)
> > 21:10:28 Memory sizes:resident:114076 KB, virtual:89190628 KB, no
> SHMTOTAL> > limit
> > 21:10:30 Dynamically allocated new virtual shared memory segment (size
> > 4194304KB)
> > 21:10:30 Memory sizes:resident:114076 KB, virtual:93384932 KB, no
> SHMTOTAL> > limit
> > 21:10:33 Dynamically allocated new virtual shared memory segment (size
> > 4194304KB)
> > 21:10:33 Memory sizes:resident:114076 KB, virtual:97579236 KB, no
> SHMTOTAL> > limit
> > 21:10:36 Logical Log 279454 Complete, timestamp: 0x5e24c0a1.
> > 21:10:36 Dynamically allocated new virtual shared memory segment (size
> > 4194304KB)
> > 21:10:36 Memory sizes:resident:114076 KB, virtual:101773540 KB, no
> SHMTOTAL> > limit
> > 21:10:38 Dynamically allocated new virtual shared memory segment (size
> > 4194304KB)
> > 21:10:38 Memory sizes:resident:114076 KB, virtual:105967844 KB, no
> SHMTOTAL> > limit
> > 21:10:41 Dynamically allocated new virtual shared memory segment (size
> > 4194304KB)
> > 21:10:41 Memory sizes:resident:114076 KB, virtual:110162148 KB, no
> SHMTOTAL> > limit
> > 21:10:44 Dynamically allocated new virtual shared memory segment (size
> > 4194304KB)
> > 21:10:44 Memory sizes:resident:114076 KB, virtual:114356452 KB, no
> SHMTOTAL> > limit
> > 21:10:46 Dynamically allocated new virtual shared memory segment (size
> > 4194304KB)
> > 21:10:46 Memory sizes:resident:114076 KB, virtual:118550756 KB, no
> SHMTOTAL> > limit
> > 21:10:49 Dynamically allocated new virtual shared memory segment (size
> > 4194304KB)
> > 21:10:49 Memory sizes:resident:114076 KB, virtual:122745060 KB, no
> SHMTOTAL> > limit
> > 21:10:52 Dynamically allocated new virtual shared memory segment (size
> > 4194304KB)
> > 21:10:52 Memory sizes:resident:114076 KB, virtual:126939364 KB, no
> SHMTOTAL> > limit
> > 21:10:55 Dynamically allocated new virtual shared memory segment (size
> > 4194304KB)
> > 21:10:55 Memory sizes:resident:114076 KB, virtual:131133668 KB, no
> SHMTOTAL> > limit
> > 21:12:29 Dynamically allocated new virtual shared memory segment (size
> > 4194304KB)
> > 21:12:29 Memory sizes:resident:114076 KB, virtual:135327972 KB, no
> SHMTOTAL> > limit
> > 21:12:51 Dynamically allocated new virtual shared memory segment (size
> > 4194304KB)
> > 21:12:51 Memory sizes:resident:114076 KB, virtual:139522276 KB, no
> SHMTOTAL> > limit
> > 21:12:56 Fatal error in ADM VP at mt.c:14304
> > 21:12:56 Unexpected virtual processor termination: pid = 9812, signal 9
> > received.
> > 21:12:57 PANIC: Attempting to bring system down
> >
> > --001a11c119e20474cb0518e09ffe
> >
> >
> >
> ****************************************************************************
> > ***
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
> >
> >
>
>
*******************************************************************************
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
>
> --001a1140727a36dabc0518e1d019
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--001a1135d7966824820519444fa6
Setting the IFX_NOPENPS environment variable is to enforce a limit on the
growth of the opentable usage.
I do not know of any side effects.
Thanks, Jeff
-----Original Message-----
From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of FRANK
Sent: Wednesday, June 24, 2015 9:28 AM
To: ids@iiug.org
Subject: Re: Panic analysis [35310]
Thank Art, Jeff and Mark for the info!
We do not do partition/fragmentation.
We are heavily using the following sysmaster tables to monitor space usage
daily, sysdbspaces syschunks sysextents
Seems difficult to avoid using them.
By the way, What does IFX_NOPENPS=4096 mean? Any possible side effect?
Thanks
Frank
On Fri, Jun 19, 2015 at 12:57 PM, Art Kagel <art.kagel@gmail.com> wrote:
> There are actually three bugs that cause memory allocation storms, the
> one Jeff mentioned and two others that mostly affect servers with many
> thousands of partitions (tables, indexes, partitions of tables or
> indexes) and occur when you try to access sysmaster tables that query
> the partition header pages (so systabnames, sysptnhdr, sysactptnhdr,
sysextents, etc.).
> They are all fixed in v12.10.FC5 and the latest fixpacks for v11.70
> and 11.50. Ran into all three at around the same time this past fall.
>
> Temporary work around: avoid querying sysmaster data about tables.
> Dbschema, myschema, dostats, some OAT sensors, and other utilities are
> some things that can trigger this depending on what options you pass in.
>
> Art
>
> Art
>
> Art S. Kagel, President and Principal Consultant ASK Database
> Management www.askdbmgt.com
>
> Blog: http://informix-myview.blogspot.com/
>
> Disclaimer: Please keep in mind that my own opinions are my own
> opinions and do not reflect on the IIUG, nor any other organization
> with which I am associated either explicitly, implicitly, or by
> inference. Neither do those opinions reflect those of other
> individuals affiliated with any entity with which I am affiliated nor
those of the entities themselves.
>
> On Fri, Jun 19, 2015 at 10:47 AM, Jeff Filippi
> <iiug@itdataconsulting.com>
> wrote:
>
> > I had some customers have the same thing happen with 12.10FC4, there
> > is a bug.
> >
> > Either upgrade to 12.10FC5 or try setting the environment variable
> > "IFX_NOPENPS=4096;export IFX_NOPENPS"
> >
> > This helped the issue with my customers.
> >
> > Thanks, Jeff
> > -----Original Message-----
> > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf
> > Of FRANK
> > Sent: Friday, June 19, 2015 10:32 AM
> > To: ids@iiug.org
> > Subject: Panic analysis [35298]
> >
> > Folks,
> >
> > IDS 12.10 FC4, Linux 6.
> >
> > The following log is the only info we have about the crash ( no
> > other
> info
> > dumped ) You can see it was struggling to add lot memories segments
> before
> > Panic ....
> > The Panic was a also VERY curious one with some unexpected process
> > terminated ...
> >
> > What are your comments ? What might be the culprit ? ....
> >
> > Thanks
> > Frank
> >
> > ...........................................
> > 21:10:26 Dynamically allocated new virtual shared memory segment
> > (size
> > 2097152KB)
> > 21:10:26 Memory sizes:resident:114076 KB, virtual:84996324 KB, no
> SHMTOTAL> > limit
> > 21:10:28 Dynamically allocated new virtual shared memory segment
> > (size
> > 4194304KB)
> > 21:10:28 Memory sizes:resident:114076 KB, virtual:89190628 KB, no
> SHMTOTAL> > limit
> > 21:10:30 Dynamically allocated new virtual shared memory segment
> > (size
> > 4194304KB)
> > 21:10:30 Memory sizes:resident:114076 KB, virtual:93384932 KB, no
> SHMTOTAL> > limit
> > 21:10:33 Dynamically allocated new virtual shared memory segment
> > (size
> > 4194304KB)
> > 21:10:33 Memory sizes:resident:114076 KB, virtual:97579236 KB, no
> SHMTOTAL> > limit
> > 21:10:36 Logical Log 279454 Complete, timestamp: 0x5e24c0a1.
> > 21:10:36 Dynamically allocated new virtual shared memory segment
> > (size
> > 4194304KB)
> > 21:10:36 Memory sizes:resident:114076 KB, virtual:101773540 KB, no
> SHMTOTAL> > limit
> > 21:10:38 Dynamically allocated new virtual shared memory segment
> > (size
> > 4194304KB)
> > 21:10:38 Memory sizes:resident:114076 KB, virtual:105967844 KB, no
> SHMTOTAL> > limit
> > 21:10:41 Dynamically allocated new virtual shared memory segment
> > (size
> > 4194304KB)
> > 21:10:41 Memory sizes:resident:114076 KB, virtual:110162148 KB, no
> SHMTOTAL> > limit
> > 21:10:44 Dynamically allocated new virtual shared memory segment
> > (size
> > 4194304KB)
> > 21:10:44 Memory sizes:resident:114076 KB, virtual:114356452 KB, no
> SHMTOTAL> > limit
> > 21:10:46 Dynamically allocated new virtual shared memory segment
> > (size
> > 4194304KB)
> > 21:10:46 Memory sizes:resident:114076 KB, virtual:118550756 KB, no
> SHMTOTAL> > limit
> > 21:10:49 Dynamically allocated new virtual shared memory segment
> > (size
> > 4194304KB)
> > 21:10:49 Memory sizes:resident:114076 KB, virtual:122745060 KB, no
> SHMTOTAL> > limit
> > 21:10:52 Dynamically allocated new virtual shared memory segment
> > (size
> > 4194304KB)
> > 21:10:52 Memory sizes:resident:114076 KB, virtual:126939364 KB, no
> SHMTOTAL> > limit
> > 21:10:55 Dynamically allocated new virtual shared memory segment
> > (size
> > 4194304KB)
> > 21:10:55 Memory sizes:resident:114076 KB, virtual:131133668 KB, no
> SHMTOTAL> > limit
> > 21:12:29 Dynamically allocated new virtual shared memory segment
> > (size
> > 4194304KB)
> > 21:12:29 Memory sizes:resident:114076 KB, virtual:135327972 KB, no
> SHMTOTAL> > limit
> > 21:12:51 Dynamically allocated new virtual shared memory segment
> > (size
> > 4194304KB)
> > 21:12:51 Memory sizes:resident:114076 KB, virtual:139522276 KB, no
> SHMTOTAL> > limit
> > 21:12:56 Fatal error in ADM VP at mt.c:14304
> > 21:12:56 Unexpected virtual processor termination: pid = 9812,
> > signal 9 received.
> > 21:12:57 PANIC: Attempting to bring system down
> >
> > --001a11c119e20474cb0518e09ffe
> >
> >
> >
> **********************************************************************
> ******
> > ***
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
> >
> >
>
>
****************************************************************************
***
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
>
> --001a1140727a36dabc0518e1d019
>
>
>
>
****************************************************************************
***
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--001a1135d7966824820519444fa6
****************************************************************************
***
Forum Note: Use "Reply" to post a response in the discussion forum.
The tables in sysmaster that MIGHT cause a memory allocation storm are:
sysextents, systabnames, sysptnhdr (this one when searched or queried with
a filter other than partnum), & sysactptnhdr (which is the same disk access
as sysptnhdr). Where you search by partnum memory allocation isn't an
issue because only a single partition header page is read. It is queries
that have to scan all header pages that cause a problem. It allocates
private query level cache pages for every partition page that is touched
and they are not released until the querying session goes away. If your
partition count is six or seven digits (yes I have clients with millions of
objects in their servers) this will naturally require GBs of memory.
Art
Art S. Kagel, President and Principal Consultant
ASK Database Management
www.askdbmgt.com
Blog: http://informix-myview.blogspot.com/
Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on the IIUG, nor any other organization with which I am
associated either explicitly, implicitly, or by inference. Neither do
those opinions reflect those of other individuals affiliated with any
entity with which I am affiliated nor those of the entities themselves.
On Wed, Jun 24, 2015 at 7:27 AM, FRANK <yunyaoqu@gmail.com> wrote:
> Thank Art, Jeff and Mark for the info!
>
> We do not do partition/fragmentation.
> We are heavily using the following sysmaster tables to monitor space usage
> daily,
> sysdbspaces
> syschunks
> sysextents
>
> Seems difficult to avoid using them.
>
> By the way, What does IFX_NOPENPS=4096 mean? Any possible side effect?
>
> Thanks
> Frank
>
> On Fri, Jun 19, 2015 at 12:57 PM, Art Kagel <art.kagel@gmail.com> wrote:
>
> > There are actually three bugs that cause memory allocation storms, the
> one
> > Jeff mentioned and two others that mostly affect servers with many
> > thousands of partitions (tables, indexes, partitions of tables or
> indexes)
> > and occur when you try to access sysmaster tables that query the
> partition
> > header pages (so systabnames, sysptnhdr, sysactptnhdr, sysextents, etc.).
> > They are all fixed in v12.10.FC5 and the latest fixpacks for v11.70 and
> > 11.50. Ran into all three at around the same time this past fall.
> >
> > Temporary work around: avoid querying sysmaster data about tables.
> > Dbschema, myschema, dostats, some OAT sensors, and other utilities are
> some
> > things that can trigger this depending on what options you pass in.
> >
> > Art
> >
> > Art
> >
> > Art S. Kagel, President and Principal Consultant
> > ASK Database Management
> > www.askdbmgt.com
> >
> > Blog: http://informix-myview.blogspot.com/
> >
> > Disclaimer: Please keep in mind that my own opinions are my own opinions
> > and do not reflect on the IIUG, nor any other organization with which I
> am
> > associated either explicitly, implicitly, or by inference. Neither do
> > those opinions reflect those of other individuals affiliated with any
> > entity with which I am affiliated nor those of the entities themselves.
> >
> > On Fri, Jun 19, 2015 at 10:47 AM, Jeff Filippi <
> iiug@itdataconsulting.com>
> > wrote:
> >
> > > I had some customers have the same thing happen with 12.10FC4, there
> is a
> > > bug.
> > >
> > > Either upgrade to 12.10FC5 or try setting the environment variable
> > > "IFX_NOPENPS=4096;export IFX_NOPENPS"
> > >
> > > This helped the issue with my customers.
> > >
> > > Thanks, Jeff
> > > -----Original Message-----
> > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of
> > > FRANK
> > > Sent: Friday, June 19, 2015 10:32 AM
> > > To: ids@iiug.org
> > > Subject: Panic analysis [35298]
> > >
> > > Folks,
> > >
> > > IDS 12.10 FC4, Linux 6.
> > >
> > > The following log is the only info we have about the crash ( no other
> > info
> > > dumped ) You can see it was struggling to add lot memories segments
> > before
> > > Panic ....
> > > The Panic was a also VERY curious one with some unexpected process
> > > terminated ...
> > >
> > > What are your comments ? What might be the culprit ? ....
> > >
> > > Thanks
> > > Frank
> > >
> > > ...........................................
> > > 21:10:26 Dynamically allocated new virtual shared memory segment (size
> > > 2097152KB)
> > > 21:10:26 Memory sizes:resident:114076 KB, virtual:84996324 KB, no
> > SHMTOTAL> > > limit
> > > 21:10:28 Dynamically allocated new virtual shared memory segment (size
> > > 4194304KB)
> > > 21:10:28 Memory sizes:resident:114076 KB, virtual:89190628 KB, no
> > SHMTOTAL> > > limit
> > > 21:10:30 Dynamically allocated new virtual shared memory segment (size
> > > 4194304KB)
> > > 21:10:30 Memory sizes:resident:114076 KB, virtual:93384932 KB, no
> > SHMTOTAL> > > limit
> > > 21:10:33 Dynamically allocated new virtual shared memory segment (size
> > > 4194304KB)
> > > 21:10:33 Memory sizes:resident:114076 KB, virtual:97579236 KB, no
> > SHMTOTAL> > > limit
> > > 21:10:36 Logical Log 279454 Complete, timestamp: 0x5e24c0a1.
> > > 21:10:36 Dynamically allocated new virtual shared memory segment (size
> > > 4194304KB)
> > > 21:10:36 Memory sizes:resident:114076 KB, virtual:101773540 KB, no
> > SHMTOTAL> > > limit
> > > 21:10:38 Dynamically allocated new virtual shared memory segment (size
> > > 4194304KB)
> > > 21:10:38 Memory sizes:resident:114076 KB, virtual:105967844 KB, no
> > SHMTOTAL> > > limit
> > > 21:10:41 Dynamically allocated new virtual shared memory segment (size
> > > 4194304KB)
> > > 21:10:41 Memory sizes:resident:114076 KB, virtual:110162148 KB, no
> > SHMTOTAL> > > limit
> > > 21:10:44 Dynamically allocated new virtual shared memory segment (size
> > > 4194304KB)
> > > 21:10:44 Memory sizes:resident:114076 KB, virtual:114356452 KB, no
> > SHMTOTAL> > > limit
> > > 21:10:46 Dynamically allocated new virtual shared memory segment (size
> > > 4194304KB)
> > > 21:10:46 Memory sizes:resident:114076 KB, virtual:118550756 KB, no
> > SHMTOTAL> > > limit
> > > 21:10:49 Dynamically allocated new virtual shared memory segment (size
> > > 4194304KB)
> > > 21:10:49 Memory sizes:resident:114076 KB, virtual:122745060 KB, no
> > SHMTOTAL> > > limit
> > > 21:10:52 Dynamically allocated new virtual shared memory segment (size
> > > 4194304KB)
> > > 21:10:52 Memory sizes:resident:114076 KB, virtual:126939364 KB, no
> > SHMTOTAL> > > limit
> > > 21:10:55 Dynamically allocated new virtual shared memory segment (size
> > > 4194304KB)
> > > 21:10:55 Memory sizes:resident:114076 KB, virtual:131133668 KB, no
> > SHMTOTAL> > > limit
> > > 21:12:29 Dynamically allocated new virtual shared memory segment (size
> > > 4194304KB)
> > > 21:12:29 Memory sizes:resident:114076 KB, virtual:135327972 KB, no
> > SHMTOTAL> > > limit
> > > 21:12:51 Dynamically allocated new virtual shared memory segment (size
> > > 4194304KB)
> > > 21:12:51 Memory sizes:resident:114076 KB, virtual:139522276 KB, no
> > SHMTOTAL> > > limit
> > > 21:12:56 Fatal error i