problem with IDS11.7 + RHE:5.8 + 512GB RAM
A user running IDS 11.70 (64-bit) on RHEL 5.8 with 512GB RAM reported that the engine started fine with buffers set to ~80% of memory, but utilities like onstat/oncheck intermittently failed with OS error 22 when attaching to shared memory; they worked around it by cutting memory to 256GB. Suggestions: verify actual /proc/sys/kernel/shmmax and shmall values, check the oninit process's /proc maps for insufficient virtual address space between SHMBASE and the shared libraries (move SHMBASE above them, e.g. 0x10000000000L), confirm the exact version via 'oninit -version' since some limit memory, set vm.nr_hugepages at the kernel level as well as in Informix, and open a PMR with IBM. No confirmed resolution is recorded.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Hi We have HPDL380 with 512GB RAM with 2 MSA (every has 25 hard 300GB) on every msa we define 1 space and the others 24 Hards have been raid 1+0 We install IDS1.7 as DB. But it seems it cann't support this amount of memory and when we have a huge transaction it doesn't support . Can any one guide me to solve this problem ? -Regards Sophia
What messages are you seeing in the log when you have the large transaction
problem? Is it something like "long transaction rollback"?
That means that you do not have enough logical log space to support such a
large single transaction. Either add more logical logs (see the onparams
utility) or break up the transaction into smaller ones.
As far as memory support, did you install a 32bit memory model version of
Informix 11.70 or a 64bit version? You can tell the memory model from the
full version number which is something like 11.70.xC# where 'x' is one of
'T', 'U', 'F', or 'H':
- 'T' - Windows 32bit
- 'U' - Linux/Unix/MacOS 32bit
- 'H' - For HPUX - 32bit memory model running as a 64bit executable
- 'F' - Windows/Linux/Unix/MacOS 64bit
The 32bit versions of Informix support up to from 2.1GB to 3.4GB of memory
depending on the operating system's limits.
The 64bit versions of Informix support up to about 4TB of main memory usage.
All versions and editions of Informix (except the space limited Developer
Edition) support up to 128PB of disk storage.
Art
Art S. Kagel
Advanced DataTools (www.advancedatatools.com)
Blog: http://informix-myview.blogspot.com/
Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on my employer, Advanced DataTools, the IIUG, nor any
other organization with which I am associated either explicitly,
implicitly, or by inference. Neither do those opinions reflect those of
other individuals affiliated with any entity with which I am affiliated nor
those of the entities themselves.
On Tue, Feb 12, 2013 at 1:19 AM, Alikhani Sophia
<S_Alikhani@isc.iranet.net>wrote:
> Hi
> We have HPDL380 with 512GB RAM with 2 MSA (every has 25 hard 300GB) on
> every
> msa we define 1 space and the others 24 Hards have been raid 1+0
> We install IDS1.7 as DB.
> But it seems it cann't support this amount of memory and when we have a
> huge
> transaction it doesn't support .
> Can any one guide me to solve this problem ?
>
> -Regards
> Sophia
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--f46d0401fd91354e9004d5813b6e
I explain the problem as below:
We have an OS : RHEL5.8 64 bit with IDS11.7 64bit with 512GB of RAM .
The RHEL5.8 Technical document says that it can support 512GB of RAM on this
distribution.
When we install IDS11.7 on theis OS we set these two parameters on sysctl.conf
Kernel.shmall= 500GB
Kernel.shmmax = 500GB
When IDS runs and oninit , it is ok without any problem and also we set the
buffers on onconfig to 80% of memeory.
But when we run some command as like as oncheck , onstat , ....
It seems that Informix has a pointer for every command that point to some
address of memory and sometimes when we run these commands we get the error
number 22 that related to OS that it seems it can't find the correct address
of memory , so at this point we reduce the memory to 256GB but I want to know
is there any configuration on os parameters or Informix parameters to solve
this problem ?
-----Original Message-----
From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Art Kagel
Sent: Tuesday, February 12, 2013 10:06 AM
To: ids@iiug.org
Subject: Re: problem with IDS11.7 + RHE:5.8 + 512GB RAM [29497]
What messages are you seeing in the log when you have the large transaction
problem? Is it something like "long transaction rollback"?
That means that you do not have enough logical log space to support such a
large single transaction. Either add more logical logs (see the onparams
utility) or break up the transaction into smaller ones.
As far as memory support, did you install a 32bit memory model version of
Informix 11.70 or a 64bit version? You can tell the memory model from the full
version number which is something like 11.70.xC# where 'x' is one of 'T', 'U',
'F', or 'H':
- 'T' - Windows 32bit
- 'U' - Linux/Unix/MacOS 32bit
- 'H' - For HPUX - 32bit memory model running as a 64bit executable
- 'F' - Windows/Linux/Unix/MacOS 64bit
The 32bit versions of Informix support up to from 2.1GB to 3.4GB of memory
depending on the operating system's limits.
The 64bit versions of Informix support up to about 4TB of main memory usage.
All versions and editions of Informix (except the space limited Developer
Edition) support up to 128PB of disk storage.
Art
Art S. Kagel
Advanced DataTools (www.advancedatatools.com)
Blog: http://informix-myview.blogspot.com/
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Advanced DataTools, the IIUG, nor any other
organization with which I am associated either explicitly, implicitly, or by
inference. Neither do those opinions reflect those of other individuals
affiliated with any entity with which I am affiliated nor those of the
entities themselves.
On Tue, Feb 12, 2013 at 1:19 AM, Alikhani Sophia
<S_Alikhani@isc.iranet.net>wrote:
> Hi
> We have HPDL380 with 512GB RAM with 2 MSA (every has 25 hard 300GB) on
> every msa we define 1 space and the others 24 Hards have been raid 1+0
> We install IDS1.7 as DB.
> But it seems it cann't support this amount of memory and when we have
> a huge transaction it doesn't support .
> Can any one guide me to solve this problem ?
>
> -Regards
> Sophia
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--f46d0401fd91354e9004d5813b6e
*******************************************************************************
Forum Note: Use "Reply" to post a response in the discussion forum.
Open a case with IBM Support.
Art
Art S. Kagel
Advanced DataTools (www.advancedatatools.com)
Blog: http://informix-myview.blogspot.com/
Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on my employer, Advanced DataTools, the IIUG, nor any
other organization with which I am associated either explicitly,
implicitly, or by inference. Neither do those opinions reflect those of
other individuals affiliated with any entity with which I am affiliated nor
those of the entities themselves.
On Tue, Feb 12, 2013 at 2:25 AM, Alikhani Sophia
<S_Alikhani@isc.iranet.net>wrote:
> I explain the problem as below:
> We have an OS : RHEL5.8 64 bit with IDS11.7 64bit with 512GB of RAM .
> The RHEL5.8 Technical document says that it can support 512GB of RAM on
> this
> distribution.
> When we install IDS11.7 on theis OS we set these two parameters on
> sysctl.conf
>
> Kernel.shmall= 500GB
> Kernel.shmmax = 500GB
>
> When IDS runs and oninit , it is ok without any problem and also we set the
> buffers on onconfig to 80% of memeory.
> But when we run some command as like as oncheck , onstat , ....
> It seems that Informix has a pointer for every command that point to some
> address of memory and sometimes when we run these commands we get the error
> number 22 that related to OS that it seems it can't find the correct
> address
> of memory , so at this point we reduce the memory to 256GB but I want to
> know
> is there any configuration on os parameters or Informix parameters to solve
> this problem ?
> -----Original Message-----
> From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Art
> Kagel
> Sent: Tuesday, February 12, 2013 10:06 AM
> To: ids@iiug.org
> Subject: Re: problem with IDS11.7 + RHE:5.8 + 512GB RAM [29497]
>
> What messages are you seeing in the log when you have the large transaction
> problem? Is it something like "long transaction rollback"?
>
> That means that you do not have enough logical log space to support such a
> large single transaction. Either add more logical logs (see the onparams
> utility) or break up the transaction into smaller ones.
>
> As far as memory support, did you install a 32bit memory model version of
> Informix 11.70 or a 64bit version? You can tell the memory model from the
> full
> version number which is something like 11.70.xC# where 'x' is one of 'T',
> 'U',
> 'F', or 'H':
>
> - 'T' - Windows 32bit
>
> - 'U' - Linux/Unix/MacOS 32bit
>
> - 'H' - For HPUX - 32bit memory model running as a 64bit executable
>
> - 'F' - Windows/Linux/Unix/MacOS 64bit
>
> The 32bit versions of Informix support up to from 2.1GB to 3.4GB of memory
> depending on the operating system's limits.
> The 64bit versions of Informix support up to about 4TB of main memory
> usage.
>
> All versions and editions of Informix (except the space limited Developer
> Edition) support up to 128PB of disk storage.
>
> Art
>
> Art S. Kagel
> Advanced DataTools (www.advancedatatools.com)
> Blog: http://informix-myview.blogspot.com/
>
> Disclaimer: Please keep in mind that my own opinions are my own opinions
> and
> do not reflect on my employer, Advanced DataTools, the IIUG, nor any other
> organization with which I am associated either explicitly, implicitly, or
> by
> inference. Neither do those opinions reflect those of other individuals
> affiliated with any entity with which I am affiliated nor those of the
> entities themselves.
>
> On Tue, Feb 12, 2013 at 1:19 AM, Alikhani Sophia
> <S_Alikhani@isc.iranet.net>wrote:
>
> > Hi
> > We have HPDL380 with 512GB RAM with 2 MSA (every has 25 hard 300GB) on
> > every msa we define 1 space and the others 24 Hards have been raid 1+0
> > We install IDS1.7 as DB.
> > But it seems it cann't support this amount of memory and when we have
> > a huge transaction it doesn't support .
> > Can any one guide me to solve this problem ?
> >
> > -Regards
> > Sophia
> >
> >
> >
> >
>
>
>
*******************************************************************************
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
>
> --f46d0401fd91354e9004d5813b6e
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--bcaec555549441ab3e04d584f5f0
I could be wrong - I have a low attention span - but you still don't seem to have answered Art's question about exactly what version of Informix you are using. Have you enabled Informix large pages support? regards Neil
On Tuesday, February 12, 2013 08:25:15 am you wrote:
> I explain the problem as below:
> We have an OS : RHEL5.8 64 bit with IDS11.7 64bit with 512GB of RAM .
> The RHEL5.8 Technical document says that it can support 512GB of RAM on this
> distribution.
> When we install IDS11.7 on theis OS we set these two parameters on
sysctl.conf
>
> Kernel.shmall= 500GB
> Kernel.shmmax = 500GB
Please check real settings of the parameters in proc filesystem to get 500GB
shared memory:
cat /proc/sys/kernel/shmmax
# Controls the maximum shared segment size, in bytes
should be >= 536870912000
calculation: 500 GB = 536870912000 bytes
cat /proc/sys/kernel/shmall
# Controls the maximum number of shared memory segments, in pages
should be >= 131072000
calculation: 536870912000 bytes / 4096 = 131072000
In addition its possible there is not enough space between Informix SHMBASE
and libraries loaded in virtual address space. For example:
cat /proc/<pid of oninit cpuvp>/maps
...
44000000-1149c2000 rwxs 00000000 00:09 1405714439
/SYSV52b94801 (deleted)
1149c2000-11acc2000 rwxs 00000000 00:09 1405747208
/SYSV52b94802 (deleted)
3fad400000-3fad41c000 r-xp 00000000 08:01 823692
/lib64/ld-2.5.so
...
SHMBASE = 0x44000000shared libraries address = 0x3fad400000
0x3fad400000 - 0x44000000 = 272348741632 bytes = 261MB
In this case you could try to move the Informix SHMBASE above the shared
libraries, for example:
SHMBASE = 0x10000000000L
cat /proc/<pid of oninit cpuvp>/maps
...
3fc1c16000-3fc1c18000 rwxp 00016000 08:01 819276
/lib64/libaudit.so.0.0.0
10000000000-100d09c2000 rwxs 00000000 00:09 1406304263
/SYSV52b94801 (deleted)
100d09c2000-100d6cc2000 rwxs 00000000 00:09 1406337032
/SYSV52b94802 (deleted)
2ba52d7bd000-2ba52d7bf000 rwxp 2ba52d7bd000 00:00 0
...
On this example there is enough space now between
0x10000000000 and 0x2ba52d7bd000
Exact addresses depend on your system configuration.
Thanks,
Andreas
>
> When IDS runs and oninit , it is ok without any problem and also we set the
> buffers on onconfig to 80% of memeory.
> But when we run some command as like as oncheck , onstat , ....
> It seems that Informix has a pointer for every command that point to some
> address of memory and sometimes when we run these commands we get the error
> number 22 that related to OS that it seems it can't find the correct address
> of memory , so at this point we reduce the memory to 256GB but I want to
know
> is there any configuration on os parameters or Informix parameters to solve
> this problem ?
> -----Original Message-----
> From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Art
> Kagel
> Sent: Tuesday, February 12, 2013 10:06 AM
> To: ids@iiug.org
> Subject: Re: problem with IDS11.7 + RHE:5.8 + 512GB RAM [29497]
>
> What messages are you seeing in the log when you have the large transaction
> problem? Is it something like "long transaction rollback"?
>
> That means that you do not have enough logical log space to support such a
> large single transaction. Either add more logical logs (see the onparams
> utility) or break up the transaction into smaller ones.
>
> As far as memory support, did you install a 32bit memory model version of
> Informix 11.70 or a 64bit version? You can tell the memory model from the
full
> version number which is something like 11.70.xC# where 'x' is one of 'T',
'U',
> 'F', or 'H':
>
> - 'T' - Windows 32bit
>
> - 'U' - Linux/Unix/MacOS 32bit
>
> - 'H' - For HPUX - 32bit memory model running as a 64bit executable
>
> - 'F' - Windows/Linux/Unix/MacOS 64bit
>
> The 32bit versions of Informix support up to from 2.1GB to 3.4GB of memory
> depending on the operating system's limits.
> The 64bit versions of Informix support up to about 4TB of main memory usage.
>
> All versions and editions of Informix (except the space limited Developer
> Edition) support up to 128PB of disk storage.
>
> Art
>
> Art S. Kagel
> Advanced DataTools (www.advancedatatools.com)
> Blog: http://informix-myview.blogspot.com/
>
> Disclaimer: Please keep in mind that my own opinions are my own opinions and
> do not reflect on my employer, Advanced DataTools, the IIUG, nor any other
> organization with which I am associated either explicitly, implicitly, or by
> inference. Neither do those opinions reflect those of other individuals
> affiliated with any entity with which I am affiliated nor those of the
> entities themselves.
>
> On Tue, Feb 12, 2013 at 1:19 AM, Alikhani Sophia
> <S_Alikhani@isc.iranet.net>wrote:
>
> > Hi
> > We have HPDL380 with 512GB RAM with 2 MSA (every has 25 hard 300GB) on
> > every msa we define 1 space and the others 24 Hards have been raid 1+0
> > We install IDS1.7 as DB.
> > But it seems it cann't support this amount of memory and when we have
> > a huge transaction it doesn't support .
> > Can any one guide me to solve this problem ?
> >
> > -Regards
> > Sophia
> >
> >
> >
> >
>
>
*******************************************************************************
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
>
> --f46d0401fd91354e9004d5813b6e
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--
Andreas Breitfeld; Informix Development Munich
IBM Deutschland Research & Development GmbH; Vorsitzende des Aufsichtsrats:
Martina Koederitz; Geschaeftsfuehrung: Dirk Wittkopp; Sitz der Gesellschaft:
Boeblingen; Registergericht: Amtsgericht Stuttgart, HRB 243294
Yes we enabled hugepage support on Informix 11.7 -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of NEIL TRUBY Sent: Tuesday, February 12, 2013 3:12 PM To: ids@iiug.org Subject: Re: RE: problem with IDS11.7 + RHE:5.8 + 512GB RAM [29501] I could be wrong - I have a low attention span - but you still don't seem to have answered Art's question about exactly what version of Informix you are using. Have you enabled Informix large pages support? regards Neil ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.
What exact version are you using. Some versions limit the amount of memory
that can be utilized.
To get the exact version run "oninit -version"
John F. Miller III
ids-bounces@iiug.org wrote on 02/12/2013 08:49:20 AM:
> From: "Alikhani Sophia" <S_Alikhani@ISC.IRANET.NET>
> To: ids@iiug.org,
> Date: 02/12/2013 08:50 AM
> Subject: RE: RE: problem with IDS11.7 + RHE:5.8 + 512GB RAM [29506]
> Sent by: ids-bounces@iiug.org
>
> Yes we enabled hugepage support on Informix 11.7
>
> -----Original Message-----
> From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of
NEIL
> TRUBY
> Sent: Tuesday, February 12, 2013 3:12 PM
> To: ids@iiug.org
> Subject: Re: RE: problem with IDS11.7 + RHE:5.8 + 512GB RAM [29501]
>
> I could be wrong - I have a low attention span - but you still don't seem
to
> have answered Art's question about exactly what version of Informix you
are
> using.
>
> Have you enabled Informix large pages support?
>
> regards
> Neil
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
With a little of delay on my comment... just for the record. On OpenSuse 12.2 where the linux kernel is version 3.4 to able to use hugepages you need to change the vm.nr_hugepages kernel parameter. Enable the parameter inside of Informix only , isn't enough. The same configuration appear in RedHat 5 . Please, check if this have any relation with your problem. https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/5/html/T uning_and_Optimizing_Red_Hat_Enterprise_Linux_for_Oracle_9i_and_10g_Databases/se ct-Oracle_9i_and_10g_Tuning_Guide-Large_Memory_Optimization_Big_Pages_and_Huge_P ages-Configuring_Huge_Pages_in_Red_Hat_Enterprise_Linux_4_or_5.html On 12/2/2013 14:49, Alikhani Sophia wrote: > Yes we enabled hugepage support on Informix 11.7 > > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of NEIL > TRUBY > Sent: Tuesday, February 12, 2013 3:12 PM > To: ids@iiug.org > Subject: Re: RE: problem with IDS11.7 + RHE:5.8 + 512GB RAM [29501] > > I could be wrong - I have a low attention span - but you still don't seem to > have answered Art's question about exactly what version of Informix you are > using. > > Have you enabled Informix large pages support? > > regards > Neil > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >