Second instance IDS issue

Hi everyone, My compagny have a SOLARIS 10 (8/11 SPARC 64) server with IDS 11.7 with two database instances at startup. We have an issue when the server boot. The first instance start correctly. But, we have an issue for the second instance with the shared memory. No matter the order, the second instance has failed due shared memory issue. onstat - (on fail instance): informix:XXXXX_8:/opt/IBM/informix11.7 > onstat - shared memory not initialized for INFORMIXSERVER 'XXXXX_8' informix:XXXXX_8:/opt/IBM/informix11.7 > ipcs -a (line of failed instance shared memory): T ID KEY MODE OWNER GROUP CREATOR CGROUP m 4 0x52574801 --rw-rw---- root informix root informix NATTCH SEGSZ CPID LPID ATIME DTIME CTIME 0 216006656 902 1471 14:42:55 14:42:55 14:36:17 There is no PID using this allocated shared memory. i did a truss of "oninit -v". LOG : shmget(1381451777, 3452960768, 0660|IPC_CREAT|IPC_EXCL) Err#22 EINVAL shmget(1381451777, 1727004672, 0660|IPC_CREAT|IPC_EXCL) Err#22 EINVAL shmget(1381451777, 864026624, 0660|IPC_CREAT|IPC_EXCL) Err#22 EINVAL shmget(1381451777, 432013312, 0660|IPC_CREAT|IPC_EXCL) Err#22 EINVAL shmget(1381451777, 216006656, 0660|IPC_CREAT|IPC_EXCL) = 4 shmat(4, 0x10A000000, 0) = 0x10A000000 shmget(1381451778, 216006656, 0660|IPC_CREAT|IPC_EXCL) Err#22 EINVAL Now it is the weird moment ! If we delete the symlink of Informix in /etc/rc2.d and execute it manually later, it works for each instances. I checked all IDS parameters(onconfig) and system parameters(/etc/project & /etc/system) but nothing seems wrong. Kind regards, Yann

Yann Do you have differing server numbers in the two onconfig files ? Are the server identifiers in onconfig, sqlhosts and /etc/services all separate between the two instances but matching within each instance ? Do you have sufficient physical memory to hold both instances, and with enough left over for the O/S and any other applications? What are the full error messages from the failed instance ? Is the script in rc2.d correctly resetting the environment between starting the two instances ?? Keith On 23 September 2014 13:49, VALLETTE YANN <yannvallette@gmail.com> wrote: > Hi everyone, > > My compagny have a SOLARIS 10 (8/11 SPARC 64) server with IDS 11.7 with two > database instances at startup. > > We have an issue when the server boot. The first instance start correctly. > But, we have an issue for the second instance with the shared memory. No > matter the order, the second instance has failed due shared memory issue. > > onstat - (on fail instance): > > informix:XXXXX_8:/opt/IBM/informix11.7 > onstat - > shared memory not initialized for INFORMIXSERVER 'XXXXX_8' > informix:XXXXX_8:/opt/IBM/informix11.7 > > > ipcs -a (line of failed instance shared memory): > T ID KEY MODE OWNER GROUP CREATOR CGROUP > m 4 0x52574801 --rw-rw---- root informix root informix > > NATTCH SEGSZ CPID LPID ATIME DTIME CTIME > > 0 216006656 902 1471 14:42:55 14:42:55 14:36:17 > > There is no PID using this allocated shared memory. > > i did a truss of "oninit -v". > > LOG : > shmget(1381451777, 3452960768, 0660|IPC_CREAT|IPC_EXCL) Err#22 EINVAL > shmget(1381451777, 1727004672, 0660|IPC_CREAT|IPC_EXCL) Err#22 EINVAL > shmget(1381451777, 864026624, 0660|IPC_CREAT|IPC_EXCL) Err#22 EINVAL > shmget(1381451777, 432013312, 0660|IPC_CREAT|IPC_EXCL) Err#22 EINVAL > shmget(1381451777, 216006656, 0660|IPC_CREAT|IPC_EXCL) = 4 > shmat(4, 0x10A000000, 0) = 0x10A000000 > shmget(1381451778, 216006656, 0660|IPC_CREAT|IPC_EXCL) Err#22 EINVAL > > Now it is the weird moment ! If we delete the symlink of Informix in > /etc/rc2.d and execute it manually later, it works for each instances. > > I checked all IDS parameters(onconfig) and system parameters(/etc/project & > /etc/system) but nothing seems wrong. > > Kind regards, > > Yann > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --047d7bdca31875510f0503bb40e4

Hi Keith, First, thanks for your reply. Do you have differing server numbers in the two onconfig files ? SERVERNUM 0 # First instance SERVERNUM 1 # Second instance Are the server identifiers in onconfig, sqlhosts and /etc/services all separate between the two instances but matching within each instance ? /etc/services : # Informix srv2_24tcp 1536/tcp # HDR instance 24 srv2_8tcp 1537/tcp # HDR instance 8 srv1_8tcp 1539/tcp # instance 8 srv1_24tcp 1540/tcp # instance 24 sqlhosts : srv2_24tcp ontlitcp srv2 srv2_24tcp srv2_24 onipcshm srv2 shm00 srv2_8tcp ontlitcp srv2 srv2_8tcp srv2_8 onipcshm srv2 shm01 # HDR Replication srv1_8tcp ontlitcp srv1 srv2_8tcp srv1_24tcp ontlitcp srv1 srv2_24tcp # Client access s_24 ontlitcp srv2 s_24tcp s_8 ontlitcp srv2 s_8tcp Onconfig first instance: SERVERNUM 0 # Unique id corresponding to a Dynamic Server instance DBSERVERNAME sbase8_24 # Name of default database server DBSERVERALIASES sbase8_24tcp,s_24 # List of alternate dbservernames Onconfig second instance: SERVERNUM 1 # Unique id corresponding to a Dynamic Server instance DBSERVERNAME sbase8_8 # Name of default database server DBSERVERALIASES sbase8_8tcp,s_8 # List of alternate dbservernames Do you have sufficient physical memory to hold both instances, and with enough left over for the O/S and any other applications? We have 16Gb of RAM for this server. In /etc/project, i put this parameter : pinformix:111::informix::project.max-sem-ids=(privileged,128,deny);project.max-s hm-ids=(privileged,128,deny);project.max-shm-memory(privileged,4294967296,deny) So each informix instance will use 4Gb of RAM max if i understand correctly. In the specs for Informix, it tells 1/4 of physical memory for this value. What are the full error messages from the failed instance ? from onlineX.log : 11:26:50 IBM Informix Dynamic Server Started. 11:26:50 Warning: The IBM IDS Growth Edition license restriction limits 11:26:50 the total shared memory size for this server to 16777216 KB. 11:26:50 The maximum allowable shared memory was reset to this size to start the database server. 11:26:50 shmget: [EINVAL][22]: key 52574802: invalid segment size 11:26:50 mt_shm_init: can't create resident segment Is the script in rc2.d correctly resetting the environment between starting the two instances ?? rc2.d/S99informix : start ) . $INFORMIXDIR/env24.sh echo "Demarrage du serveur Informix : >$INFORMIXSERVER<" oninit sleep 15 onstat - sleep 15 . $INFORMIXDIR/env8.sh echo "Demarrage du serveur Informix : >$INFORMIXSERVER<" oninit sleep 60 onstat - ;; env8.sh : ONCONFIG=onconfig.sbase8.8 INFORMIXSERVER=sbase8_8 export ONCONFIG INFORMIXSERVER DBDATE=DMY4/export DBDATE Kind regards, Yann

Hello Keith, If you edit /etc/project file manually on SUN OS, Kernel resources would not be allocated maybe. If so, try it using next command : Step 1 : Add project for user informix Projadd -U Informix -G Informix Informix Step 2 : Define or replace params for the project <informix> projmod -s -K "project.max-sem-ids=(priv,128,deny)" informix projmod -s -K "process.max-sem-nsems=(priv,512,deny)" informix projmod -s -K "project.max-shm-memory=(priv,4TB,deny)" informix projmod -s -K "project.max-shm-ids=(priv,128,deny)" informix you can do to define for "default" project like Step 2. And then redo oninit. Good luck to you! Kind regards, Jeong-seop Oh ========================== CEO of UbicoIT Co., Ltd. ========================== -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of VALLETTE YANN Sent: Tuesday, September 23, 2014 10:52 PM To: ids@iiug.org Subject: Re: Second instance IDS issue [33854] Hi Keith, First, thanks for your reply. Do you have differing server numbers in the two onconfig files ? SERVERNUM 0 # First instance SERVERNUM 1 # Second instance Are the server identifiers in onconfig, sqlhosts and /etc/services all separate between the two instances but matching within each instance ? /etc/services : # Informix srv2_24tcp 1536/tcp # HDR instance 24 srv2_8tcp 1537/tcp # HDR instance 8 srv1_8tcp 1539/tcp # instance 8 srv1_24tcp 1540/tcp # instance 24 sqlhosts : srv2_24tcp ontlitcp srv2 srv2_24tcp srv2_24 onipcshm srv2 shm00 srv2_8tcp ontlitcp srv2 srv2_8tcp srv2_8 onipcshm srv2 shm01 # HDR Replication srv1_8tcp ontlitcp srv1 srv2_8tcp srv1_24tcp ontlitcp srv1 srv2_24tcp # Client access s_24 ontlitcp srv2 s_24tcp s_8 ontlitcp srv2 s_8tcp Onconfig first instance: SERVERNUM 0 # Unique id corresponding to a Dynamic Server instance DBSERVERNAME sbase8_24 # Name of default database server DBSERVERALIASESsbase8_24tcp,s_24 # List of alternate dbservernames Onconfig second instance: SERVERNUM 1 # Unique id corresponding to a Dynamic Server instance DBSERVERNAME sbase8_8 # Name of default database server DBSERVERALIASESsbase8_8tcp,s_8 # List of alternate dbservernames Do you have sufficient physical memory to hold both instances, and with enough left over for the O/S and any other applications? We have 16Gb of RAM for this server. In /etc/project, i put this parameter : pinformix:111::informix::project.max-sem- ids=(privileged,128,deny);project.max-shm-ids=(privileged,128,deny);project. max-shm-memory(privileged,4294967296,deny) So each informix instance will use 4Gb of RAM max if i understand correctly. In the specs for Informix, it tells 1/4 of physical memory for this value. What are the full error messages from the failed instance ? from onlineX.log : 11:26:50 IBM Informix Dynamic Server Started. 11:26:50 Warning: The IBM IDS Growth Edition license restriction limits 11:26:50 the total shared memory size for this server to 16777216 KB. 11:26:50 The maximum allowable shared memory was reset to this size to start the database server. 11:26:50 shmget: [EINVAL][22]: key 52574802: invalid segment size 11:26:50 mt_shm_init: can't create resident segment Is the script in rc2.d correctly resetting the environment between starting the two instances ?? rc2.d/S99informix : start ) .. $INFORMIXDIR/env24.sh echo "Demarrage du serveur Informix : >$INFORMIXSERVER<" oninit sleep 15 onstat - sleep 15 .. $INFORMIXDIR/env8.sh echo "Demarrage du serveur Informix : >$INFORMIXSERVER<" oninit sleep 60 onstat - ;; env8.sh : ONCONFIG=onconfig.sbase8.8 INFORMIXSERVER=sbase8_8 export ONCONFIG INFORMIXSERVER DBDATE=DMY4/export DBDATE Kind regards, Yann **************************************************************************** *** Forum Note: Use "Reply" to post a response in the discussion forum.

Yann How much memory do you think you are trying to allocate to each instance ? What is SHMVIRTSIZE ? How many BUFFERS have you got configured and at what page size, I wonder if you are trying to assign more memory than is being made available ?? Your SERVERNAME/ALIAS - sqlhosts - /etc/services do not seem correct but I cannot unpick them just at the moment. You said in the original post that you could start both instances together from the command line ? How did you switch environments in that case ? Did you use the env8.sh script? Do you have a similar env24.sh script and what happens if you try reversing the order of starting in the rc2.d script ?? Keith On 23 September 2014 14:51, VALLETTE YANN <yannvallette@gmail.com> wrote: > Hi Keith, > > First, thanks for your reply. > > Do you have differing server numbers in the two onconfig files ? > SERVERNUM 0 # First instance > SERVERNUM 1 # Second instance> > Are the server identifiers in onconfig, sqlhosts and /etc/services all > separate between the two instances but matching within each instance ? > > /etc/services : > > # Informix > srv2_24tcp 1536/tcp # HDR instance 24 > srv2_8tcp 1537/tcp # HDR instance 8 > srv1_8tcp 1539/tcp # instance 8 > srv1_24tcp 1540/tcp # instance 24 > > sqlhosts : > srv2_24tcp ontlitcp srv2 srv2_24tcp > srv2_24 onipcshm srv2 shm00 > srv2_8tcp ontlitcp srv2 srv2_8tcp > srv2_8 onipcshm srv2 shm01> > # HDR Replication > srv1_8tcp ontlitcp srv1 srv2_8tcp > srv1_24tcp ontlitcp srv1 srv2_24tcp> > # Client access > s_24 ontlitcp srv2 s_24tcp > s_8 ontlitcp srv2 s_8tcp> > Onconfig first instance: > > SERVERNUM 0 # Unique id corresponding to a Dynamic Server instance > DBSERVERNAME sbase8_24 # Name of default database server > DBSERVERALIASES sbase8_24tcp,s_24 # List of alternate dbservernames> > Onconfig second instance: > > SERVERNUM 1 # Unique id corresponding to a Dynamic Server instance > DBSERVERNAME sbase8_8 # Name of default database server > DBSERVERALIASES sbase8_8tcp,s_8 # List of alternate dbservernames> > Do you have sufficient physical memory to hold both instances, and with > enough left over for the O/S and any other applications? > > We have 16Gb of RAM for this server. > > In /etc/project, i put this parameter : > > > pinformix:111::informix::project.max-sem-ids=(privileged,128,deny);project.max-s hm-ids=(privileged,128,deny);project.max-shm-memory(privileged,4294967296,deny) > > So each informix instance will use 4Gb of RAM max if i understand > correctly. > > In the specs for Informix, it tells 1/4 of physical memory for this value. > > What are the full error messages from the failed instance ? > > from onlineX.log : > 11:26:50 IBM Informix Dynamic Server Started. > 11:26:50 Warning: The IBM IDS Growth Edition license restriction limits > 11:26:50 the total shared memory size for this server to 16777216 KB. > 11:26:50 The maximum allowable shared memory was reset to this size to > start > the database server. > 11:26:50 shmget: [EINVAL][22]: key 52574802: invalid segment size > 11:26:50 mt_shm_init: can't create resident segment > > Is the script in rc2.d correctly resetting the environment between starting > the two instances ?? > > rc2.d/S99informix : > > start ) > > .. $INFORMIXDIR/env24.sh > > echo "Demarrage du serveur Informix : >$INFORMIXSERVER<" > > oninit> > sleep 15 > > onstat - > > sleep 15 > > .. $INFORMIXDIR/env8.sh > > echo "Demarrage du serveur Informix : >$INFORMIXSERVER<" > > oninit> > sleep 60 > > onstat - > > ;; > > env8.sh : > > ONCONFIG=onconfig.sbase8.8 > INFORMIXSERVER=sbase8_8 > export ONCONFIG INFORMIXSERVER > DBDATE=DMY4/> export DBDATE > > Kind regards, > > Yann > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a1134bad0e54ae40503be368c

Hi Keith, First... I made a mistake on the previous answer. > env8.sh : > > ONCONFIG=onconfig.svr2.8 > INFORMIXSERVER=svr2_8 > export ONCONFIG INFORMIXSERVER > DBDATE=DMY4/> export DBDATE The env24 is similar. How much memory do you think you are trying to allocate to each instance ? 4Gb if i correctly read the /etc/project? What is SHMVIRTSIZE ? RESIDENT 0 SHMBASE 0x10A000000L SHMVIRTSIZE 200000 <==== SHMADD 32768 EXTSHMADD 8192 SHMTOTAL 0 SHMVIRT_ALLOCSEG 0,3 SHMNOACCESS How many BUFFERS have you got configured and at what page size, I wonder if you are trying to assign more memory than is being made available ?? BUFFERPOOL default,buffers=1500000,lrus=256,lru_min_dirty=1.00,lru_max_dirty=2.00 BUFFERPOOL size=2K,buffers=1500000,lrus=256,lru_min_dirty=1.00,lru_max_dirty=2.00 Your SERVERNAME/ALIAS - sqlhosts - /etc/services do not seem correct but I cannot unpick them just at the moment. Could you tell me why do you think it's wrong? You said in the original post that you could start both instances together from the command line ? How did you switch environments in that case ? Did you use the env8.sh script? If i execute the script (/etc/init.d/informix start) after server boot both instances works. Do you have a similar env24.sh script and what happens if you try reversing the order of starting in the rc2.d script ?? If i switch boot order of instance in rc2.d script (/etc/init.d/informix) the first one works but the second failed due to shared memory issue. No matter which one is first ... That's my problem, i don't know why both instance couldn't start in rc2.d while manually it works ...

Yann SERVERNAME/ALIAS - sqlhosts - /etc/services must all be OK if both instances will start manually, disregard my comment. When you run /etc/init.d/infomrix start manually are yo running as root or informix or/and is there an su - informix in the script ? Reversing out all my previous comments and thoughts !! this looks more like an environment issue. Can you insert "echo `env` > /tmp/env24.env" before the relevant oninit and similar before the 8 oninit. Bounce the server to get the environments at boot, save the files and then stop and start the instances manually and compare the environment files to see what differs. The issue may be related to the subdivision of the server. This is something with which I am not familiar, you may need to reach out to someone with more Solaris experiance. Keith On 24 September 2014 07:31, VALLETTE YANN <yannvallette@gmail.com> wrote: > Hi Keith, > > First... I made a mistake on the previous answer. > > > env8.sh : > > > > ONCONFIG=onconfig.svr2.8 > > INFORMIXSERVER=svr2_8 > > export ONCONFIG INFORMIXSERVER > > DBDATE=DMY4/> > export DBDATE > > The env24 is similar. > > How much memory do you think you are trying to allocate to each instance ? > > 4Gb if i correctly read the /etc/project? > > What is SHMVIRTSIZE ? > > RESIDENT 0 > SHMBASE 0x10A000000L > SHMVIRTSIZE 200000 <==== > SHMADD 32768 > EXTSHMADD 8192 > SHMTOTAL 0 > SHMVIRT_ALLOCSEG 0,3 > SHMNOACCESS> > How many BUFFERS have you got configured and at what page size, I wonder if > you are trying to assign more memory than is being made available ?? > > BUFFERPOOL > default,buffers=1500000,lrus=256,lru_min_dirty=1.00,lru_max_dirty=2.00 > BUFFERPOOL > size=2K,buffers=1500000,lrus=256,lru_min_dirty=1.00,lru_max_dirty=2.00> > Your SERVERNAME/ALIAS - sqlhosts - /etc/services do not seem correct but I > cannot unpick them just at the moment. > > Could you tell me why do you think it's wrong? > > You said in the original post that you could start both instances together > from the command line ? How did you switch environments in that case ? Did > you use the env8.sh script? > > If i execute the script (/etc/init.d/informix start) after server boot both > instances works. > > Do you have a similar env24.sh script and what happens if you try reversing > the order of starting in the rc2.d script ?? > > If i switch boot order of instance in rc2.d script (/etc/init.d/informix) > the > first one works but the second failed due to shared memory issue. No matter > which one is first ... > > That's my problem, i don't know why both instance couldn't start in rc2.d > while manually it works ... > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a11c22eb67a498b0503cb400d

When you run /etc/init.d/infomrix start manually are yo running as root or informix or/and is there an su - informix in the script ? I run the script as root. env 8: DBDATE=DMY4/ INFORMIXDIR=/opt/IBM/informix11.7 INFORMIXSERVER=srv2_8 ONCONFIG=onconfig.srv2.8 PATH=/usr/sbin:/usr/binTZ=Europe/Paris _INIT_NET_STRATEGY=none _INIT_PREV_LEVEL=S _INIT_RUN_LEVEL=3 _INIT_RUN_NPREV=0 _INIT_UTS_ISA=sparc _INIT_UTS_MACHINE=sun4u _INIT_UTS_NODENAME=sbase8 _INIT_UTS_PLATFORM=SUNW,Sun-Fire-480R _INIT_UTS_RELEASE=5.10 _INIT_UTS_SYSNAME=SunOS _INIT_UTS_VERSION=Generic_147440-12 _INIT_ZONENAME=global env 24 : DBDATE=DMY4/ INFORMIXDIR=/opt/IBM/informix11.7 INFORMIXSERVER=srv2_24 ONCONFIG=onconfig.srv2.24 PATH=/usr/sbin:/usr/binTZ=Europe/Paris _INIT_NET_STRATEGY=none _INIT_PREV_LEVEL=S _INIT_RUN_LEVEL=3 _INIT_RUN_NPREV=0 _INIT_UTS_ISA=sparc _INIT_UTS_MACHINE=sun4u _INIT_UTS_NODENAME=sbase8 _INIT_UTS_PLATFORM=SUNW,Sun-Fire-480R _INIT_UTS_RELEASE=5.10 _INIT_UTS_SYSNAME=SunOS _INIT_UTS_VERSION=Generic_147440-12 _INIT_ZONENAME=global I don't see anything wrong ... ps: i'm a sysadmin and i never use informix's database ... I have to IPCRM the shared memory of failed instance after each reboot and execute the script again. It seems the server don't have enough RAM for both informix instances on boot... But Why it works manually after ? Thanks for your help !

Yann Sorry, reached the limits of my knowledge of Solaris and projects. I think that is where your issue lies. Anyone else able to pick up here ?? Keith On 24 September 2014 10:07, VALLETTE YANN <yannvallette@gmail.com> wrote: > When you run /etc/init.d/infomrix start manually are yo running as root or > informix or/and is there an su - informix in the script ? > > I run the script as root. > > env 8: > DBDATE=DMY4/ > INFORMIXDIR=/opt/IBM/informix11.7 > INFORMIXSERVER=srv2_8 > ONCONFIG=onconfig.srv2.8 > PATH=/usr/sbin:/usr/bin> TZ=Europe/Paris > _INIT_NET_STRATEGY=none > _INIT_PREV_LEVEL=S > _INIT_RUN_LEVEL=3 > _INIT_RUN_NPREV=0 > _INIT_UTS_ISA=sparc > _INIT_UTS_MACHINE=sun4u > _INIT_UTS_NODENAME=sbase8 > _INIT_UTS_PLATFORM=SUNW,Sun-Fire-480R > _INIT_UTS_RELEASE=5.10 > _INIT_UTS_SYSNAME=SunOS > _INIT_UTS_VERSION=Generic_147440-12 > _INIT_ZONENAME=global > > env 24 : > DBDATE=DMY4/ > INFORMIXDIR=/opt/IBM/informix11.7 > INFORMIXSERVER=srv2_24 > ONCONFIG=onconfig.srv2.24 > PATH=/usr/sbin:/usr/bin> TZ=Europe/Paris > _INIT_NET_STRATEGY=none > _INIT_PREV_LEVEL=S > _INIT_RUN_LEVEL=3 > _INIT_RUN_NPREV=0 > _INIT_UTS_ISA=sparc > _INIT_UTS_MACHINE=sun4u > _INIT_UTS_NODENAME=sbase8 > _INIT_UTS_PLATFORM=SUNW,Sun-Fire-480R > _INIT_UTS_RELEASE=5.10 > _INIT_UTS_SYSNAME=SunOS > _INIT_UTS_VERSION=Generic_147440-12 > _INIT_ZONENAME=global > > I don't see anything wrong ... > > ps: i'm a sysadmin and i never use informix's database ... > > I have to IPCRM the shared memory of failed instance after each reboot and > execute the script again. > > It seems the server don't have enough RAM for both informix instances on > boot... > But Why it works manually after ? > > Thanks for your help ! > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --90e6ba6e8fb85a2b050503ce10d9