Server Install Weirdness:

Ok, I will post this here since no at the State Of Mississippi ITS knows our ace id. Hopefully a Informix Engineer will pick up on it: Environment: zLinux Suse 11.2 on ZVM 5.4. Informix 11.7 8GB Ram allocated to zLinux , 1.5 TB Disk - 500GB DASD with 1TB Multipath to 8300 SAN. Synopsis: Install using IP address 10.242.254.100 fails. Always receives a 'shared memory not initialized for INFORMIXSERVER' randomly during install. Reproducible: YES Consistent: YES Details: Installing on zLinux Suse 11.2 using scripts that have been working for a year now, will receive a 'shared memory not initialized for INFORMIXSERVER' randomly during setup. buildmsi process and it's associated dbaccess task will hang and the sysmater, sysusers and other sys type files never complete building. Changing IP address to 10.242.254.101 or 10.242.254.99 will allow all task to complete successfully. Removing the and wiping the server at ip address 10.242.254.100 and CLONING one that has gone thru the build phase successfully (IP address 10.242.254.103) and changing the IP address to 10.242.254.100 and restarting the server results in 5 minute connection times and dbaccess returning 'shared memory not initialized for INFORMIXSERVER'. Changing IP address to 10.242.254.99 and all processes and connections works as they should. Failure is most noticeable using onspaces during attempts to setup dbspaces, new tempfs and llogdbs and plogdbs. Environment variables required for INFORMIX are there and are complete for users informix and root. Change IP address to 10.242.254.200 and server responds and acts as it should. NO OTHER SERVER or system is on the network at 10.242.254.100, all arp caches have been cleared prior to the testing. Final result: 10.242.254.100 results in repeated and consistent failure and issues of running Informix. Moving Server to different IP and the the server works, clone a working server and use IP 10.242.254.100 and it fails. Seems to be related somehow of the .100 of the IP address, as we have up and running around 30 servers that are in the 10.242.254.80 and less IP range, all built using the same scripts. Needless to say, this is a stumper. Looks like a bug inside Informix on the semaphore stuff. Ben Duncan - Business Network Solutions, Inc. 336 Elton Road Jackson MS, 39212 "Never attribute to malice, that which can be adequately explained by stupidity" - Hanlon's Razor

Hi Ben, Before say this is an Informix problem... where appear to be, because the behave is very wierd... you not mentioned nothing how is your network configuration (dns, resolve names) and the sqlhosts My first guess is you have some problem to resolve the own machine name... where sometimes informix reach the correct IP , sometimes not... (this will be a OS problem) *) how the hostnames are resolved into this Linux? (/etc/hosts.conf) *) if use DNS, it's resolve correctly your IP , hostname and reverse ip? (check nslookup, /etc/resolv.conf) *) the hostname is into /etc/hosts ? *) the sqlhosts, you use IP or the hostname? and the port? you use the /etc/services or specify an explicit port number ? *) check for FQDN over all this configuration too... If all this appear be ok... no ideas.... Regards Cesar 2013/7/10 Ben Duncan <linux4ms@aim.com> > Ok, I will post this here since no at the State Of Mississippi ITS knows > our > ace > id. Hopefully a Informix Engineer will pick up on it: > > Environment: > zLinux Suse 11.2 on ZVM 5.4. > Informix 11.7 > 8GB Ram allocated to zLinux , 1.5 TB Disk - 500GB DASD with 1TB Multipath > to 8300 SAN. > > Synopsis: Install using IP address 10.242.254.100 fails. > Always receives a 'shared memory not initialized for INFORMIXSERVER' > randomly during install. > > Reproducible: YES > Consistent: YES > > Details: > Installing on zLinux Suse 11.2 using scripts that have been working for > a year now, will receive a 'shared memory not initialized for > INFORMIXSERVER' > randomly during setup. buildmsi process and it's associated dbaccess task > will hang and the sysmater, sysusers and other sys type files never > complete > building. > > Changing IP address to 10.242.254.101 or 10.242.254.99 will allow all > task to complete successfully. Removing the and wiping the server at > ip address 10.242.254.100 and CLONING one that has gone thru the build > phase > successfully (IP address 10.242.254.103) and changing the IP address to > 10.242.254.100 and restarting the server results in 5 minute connection > times > and dbaccess returning 'shared memory not initialized for INFORMIXSERVER'. > Changing IP address to 10.242.254.99 and all processes and connections > works > as they should. Failure is most noticeable using onspaces during attempts > to > setup dbspaces, new tempfs and llogdbs and plogdbs. Environment variables > required for INFORMIX are there and are complete for users informix and > root. > > Change IP address to 10.242.254.200 and server responds and acts as it > should. > NO OTHER SERVER or system is on the network at 10.242.254.100, all arp > caches > have been cleared prior to the testing. > > Final result: > 10.242.254.100 results in repeated and consistent failure and > issues of running Informix. Moving Server to different IP and the the > server > works, clone a working server and use IP 10.242.254.100 and it fails. > Seems to be related somehow of the .100 of the IP address, as we > have up and running around 30 servers that are in the 10.242.254.80 and > less IP range, all built using the same scripts. > > Needless to say, this is a stumper. Looks like a bug inside Informix > on the semaphore stuff. > > Ben Duncan - Business Network Solutions, Inc. 336 Elton Road Jackson MS, > 39212 > "Never attribute to malice, that which can be adequately explained by > stupidity" > > - Hanlon's Razor > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --089e0149353897fc6004e12d0e98

Are you sure that there isn't already another device at the .100 address? Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Wed, Jul 10, 2013 at 2:37 PM, Ben Duncan <linux4ms@aim.com> wrote: > Ok, I will post this here since no at the State Of Mississippi ITS knows > our > ace > id. Hopefully a Informix Engineer will pick up on it: > > Environment: > zLinux Suse 11.2 on ZVM 5.4. > Informix 11.7 > 8GB Ram allocated to zLinux , 1.5 TB Disk - 500GB DASD with 1TB Multipath > to 8300 SAN. > > Synopsis: Install using IP address 10.242.254.100 fails. > Always receives a 'shared memory not initialized for INFORMIXSERVER' > randomly during install. > > Reproducible: YES > Consistent: YES > > Details: > Installing on zLinux Suse 11.2 using scripts that have been working for > a year now, will receive a 'shared memory not initialized for > INFORMIXSERVER' > randomly during setup. buildmsi process and it's associated dbaccess task > will hang and the sysmater, sysusers and other sys type files never > complete > building. > > Changing IP address to 10.242.254.101 or 10.242.254.99 will allow all > task to complete successfully. Removing the and wiping the server at > ip address 10.242.254.100 and CLONING one that has gone thru the build > phase > successfully (IP address 10.242.254.103) and changing the IP address to > 10.242.254.100 and restarting the server results in 5 minute connection > times > and dbaccess returning 'shared memory not initialized for INFORMIXSERVER'. > Changing IP address to 10.242.254.99 and all processes and connections > works > as they should. Failure is most noticeable using onspaces during attempts > to > setup dbspaces, new tempfs and llogdbs and plogdbs. Environment variables > required for INFORMIX are there and are complete for users informix and > root. > > Change IP address to 10.242.254.200 and server responds and acts as it > should. > NO OTHER SERVER or system is on the network at 10.242.254.100, all arp > caches > have been cleared prior to the testing. > > Final result: > 10.242.254.100 results in repeated and consistent failure and > issues of running Informix. Moving Server to different IP and the the > server > works, clone a working server and use IP 10.242.254.100 and it fails. > Seems to be related somehow of the .100 of the IP address, as we > have up and running around 30 servers that are in the 10.242.254.80 and > less IP range, all built using the same scripts. > > Needless to say, this is a stumper. Looks like a bug inside Informix > on the semaphore stuff. > > Ben Duncan - Business Network Solutions, Inc. 336 Elton Road Jackson MS, > 39212 > "Never attribute to malice, that which can be adequately explained by > stupidity" > > - Hanlon's Razor > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a11c3b4aeda497c04e12d7ad2

I would check to see if another server is running on IP 10.242.254.100.= In the past informix would not start if the IP:port is in use, but many customers asked for this to be relax and so it was. John F. Miller III ids-bounces@iiug.org wrote on 07/10/2013 11:37:04 AM: > From: "Ben Duncan" <linux4ms@aim.com> > To: ids@iiug.org, > Date: 07/10/2013 11:39 AM > Subject: Server Install Weirdness: [30809] > Sent by: ids-bounces@iiug.org > > Ok, I will post this here since no at the State Of Mississippi ITS kn= ows our > ace > id. Hopefully a Informix Engineer will pick up on it: > > Environment: > zLinux Suse 11.2 on ZVM 5.4. > Informix 11.7 > 8GB Ram allocated to zLinux , 1.5 TB Disk - 500GB DASD with 1TB Multi= path > to 8300 SAN. > > Synopsis: Install using IP address 10.242.254.100 fails. > Always receives a 'shared memory not initialized for INFORMIXSERVER' > randomly during install. > > Reproducible: YES > Consistent: YES > > Details: > Installing on zLinux Suse 11.2 using scripts that have been working f= or > a year now, will receive a 'shared memory not initialized for INFORMIXSERVER' > randomly during setup. buildmsi process and it's associated dbaccess = task > will hang and the sysmater, sysusers and other sys type files never complete > building. > > Changing IP address to 10.242.254.101 or 10.242.254.99 will allow all= > task to complete successfully. Removing the and wiping the server at > ip address 10.242.254.100 and CLONING one that has gone thru the buil= d phase > successfully (IP address 10.242.254.103) and changing the IP address = to > 10.242.254.100 and restarting the server results in 5 minute connecti= on times > and dbaccess returning 'shared memory not initialized for INFORMIXSERVER'. > Changing IP address to 10.242.254.99 and all processes and connection= s works > as they should. Failure is most noticeable using onspaces during atte= mpts to > setup dbspaces, new tempfs and llogdbs and plogdbs. Environment varia= bles > required for INFORMIX are there and are complete for users informix a= nd root. > > Change IP address to 10.242.254.200 and server responds and acts as > it should. > NO OTHER SERVER or system is on the network at 10.242.254.100, all ar= p caches > have been cleared prior to the testing. > > Final result: > 10.242.254.100 results in repeated and consistent failure and > issues of running Informix. Moving Server to different IP and the the= server > works, clone a working server and use IP 10.242.254.100 and it fails.= > Seems to be related somehow of the .100 of the IP address, as we > have up and running around 30 servers that are in the 10.242.254.80 a= nd > less IP range, all built using the same scripts. > > Needless to say, this is a stumper. Looks like a bug inside Informix > on the semaphore stuff. > > Ben Duncan - Business Network Solutions, Inc. 336 Elton Road JacksonM= S, 39212 > "Never attribute to malice, that which can be adequately explained by= > stupidity" > > - Hanlon's Razor > > > ***********************************************************************= ******** > Forum Note: Use "Reply" to post a response in the discussion forum.= >=

A bit of a shot in the dark - but what is your subnet mask? I'd also be interested to know if other services are functional (e.g. sshd, ping tests, etc). Also, I'm assuming the dbspace is on the SAN. What happens if you try to use local disks? It sounds more like a network configuration error that is causing the SAN connection to fail - do you have any 'chunk offline' errors in the log, or hardware related errors in /var/log/messages or dmesg? -Justin -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of John Miller iii Sent: Wednesday, July 10, 2013 12:51 PM To: ids@iiug.org Subject: Re: Server Install Weirdness: [30812] I would check to see if another server is running on IP 10.242.254.100.= In the past informix would not start if the IP:port is in use, but many customers asked for this to be relax and so it was. John F. Miller III ids-bounces@iiug.org wrote on 07/10/2013 11:37:04 AM: > From: "Ben Duncan" <linux4ms@aim.com> > To: ids@iiug.org, > Date: 07/10/2013 11:39 AM > Subject: Server Install Weirdness: [30809] > Sent by: ids-bounces@iiug.org > > Ok, I will post this here since no at the State Of Mississippi ITS kn= ows our > ace > id. Hopefully a Informix Engineer will pick up on it: > > Environment: > zLinux Suse 11.2 on ZVM 5.4. > Informix 11.7 > 8GB Ram allocated to zLinux , 1.5 TB Disk - 500GB DASD with 1TB Multi= path > to 8300 SAN. > > Synopsis: Install using IP address 10.242.254.100 fails. > Always receives a 'shared memory not initialized for INFORMIXSERVER' > randomly during install. > > Reproducible: YES > Consistent: YES > > Details: > Installing on zLinux Suse 11.2 using scripts that have been working f= or > a year now, will receive a 'shared memory not initialized for INFORMIXSERVER' > randomly during setup. buildmsi process and it's associated dbaccess = task > will hang and the sysmater, sysusers and other sys type files never complete > building. > > Changing IP address to 10.242.254.101 or 10.242.254.99 will allow all= > task to complete successfully. Removing the and wiping the server at > ip address 10.242.254.100 and CLONING one that has gone thru the buil= d phase > successfully (IP address 10.242.254.103) and changing the IP address = to > 10.242.254.100 and restarting the server results in 5 minute connecti= on times > and dbaccess returning 'shared memory not initialized for INFORMIXSERVER'. > Changing IP address to 10.242.254.99 and all processes and connection= s works > as they should. Failure is most noticeable using onspaces during atte= mpts to > setup dbspaces, new tempfs and llogdbs and plogdbs. Environment varia= bles > required for INFORMIX are there and are complete for users informix a= nd root. > > Change IP address to 10.242.254.200 and server responds and acts as > it should. > NO OTHER SERVER or system is on the network at 10.242.254.100, all ar= p caches > have been cleared prior to the testing. > > Final result: > 10.242.254.100 results in repeated and consistent failure and > issues of running Informix. Moving Server to different IP and the the= server > works, clone a working server and use IP 10.242.254.100 and it fails.= > Seems to be related somehow of the .100 of the IP address, as we > have up and running around 30 servers that are in the 10.242.254.80 a= nd > less IP range, all built using the same scripts. > > Needless to say, this is a stumper. Looks like a bug inside Informix > on the semaphore stuff. > > Ben Duncan - Business Network Solutions, Inc. 336 Elton Road JacksonM= S, 39212 > "Never attribute to malice, that which can be adequately explained by= > stupidity" > > - Hanlon's Razor > > > ***********************************************************************= ******** > Forum Note: Use "Reply" to post a response in the discussion forum.= >= ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

network issues have been absolved. Bringing DOWN the 10.242.254.100 and try to ping / traceroute etc shows no other devices, as does nmap. Services is set on hosts.conf as "host, bind" order with SERVER itself being listed in /etc/hosts file. Subnet Mask is 255.255.255.0 (slash 24) network is set as 10.242.254.0, broadcast is 10.242.254.255. DNS is set to internal 10 net server. As I said, simply changing IP to a non .100 and everything works. shared memory Failures are very randomized. makes me thing that perhaps in the shared memory key generation something is going on. On 07/10/2013 03:37 PM, Justin Killen wrote: > A bit of a shot in the dark - but what is your subnet mask? I'd also be > interested to know if other services are functional (e.g. sshd, ping tests, > etc). > > Also, I'm assuming the dbspace is on the SAN. What happens if you try to use > local disks? It sounds more like a network configuration error that is causing > the SAN connection to fail - do you have any 'chunk offline' errors in the > log, or hardware related errors in /var/log/messages or dmesg? -- Ben Duncan - Business Network Solutions, Inc. 336 Elton Road Jackson MS, 39212 "Never attribute to malice, that which can be adequately explained by stupidity" - Hanlon's Razor

So aside from IDS, other services work (ssh, http, etc) in both directions? -Justin -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Ben Duncan Sent: Wednesday, July 10, 2013 1:53 PM To: ids@iiug.org Subject: Re: Server Install Weirdness: [30814] network issues have been absolved. Bringing DOWN the 10.242.254.100 and try to ping / traceroute etc shows no other devices, as does nmap. Services is set on hosts.conf as "host, bind" order with SERVER itself being listed in /etc/hosts file. Subnet Mask is 255.255.255.0 (slash 24) network is set as 10.242.254.0, broadcast is 10.242.254.255. DNS is set to internal 10 net server. As I said, simply changing IP to a non .100 and everything works. shared memory Failures are very randomized. makes me thing that perhaps in the shared memory key generation something is going on. On 07/10/2013 03:37 PM, Justin Killen wrote: > A bit of a shot in the dark - but what is your subnet mask? I'd also be > interested to know if other services are functional (e.g. sshd, ping tests, > etc). > > Also, I'm assuming the dbspace is on the SAN. What happens if you try to use > local disks? It sounds more like a network configuration error that is causing > the SAN connection to fail - do you have any 'chunk offline' errors in the > log, or hardware related errors in /var/log/messages or dmesg? -- Ben Duncan - Business Network Solutions, Inc. 336 Elton Road Jackson MS, 39212 "Never attribute to malice, that which can be adequately explained by stupidity" - Hanlon's Razor ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Yes, we can ssh / sftp / scp without any problems. I may have network guys do a trap on packets tomorrow for weirdness (Thanks Art for that Idea). On 07/10/2013 04:02 PM, Justin Killen wrote: > So aside from IDS, other services work (ssh, http, etc) in both directions? > > -Justin -- Ben Duncan - Business Network Solutions, Inc. 336 Elton Road Jackson MS, 39212 "Never attribute to malice, that which can be adequately explained by stupidity" - Hanlon's Razor

Final result: No one and nothing is using the 10.242.254.100 address. No even an entry in arp cache on any of the routers. Change the IP back to 10.242.254.100 and BANG - start getting "shared memory errors" randomly again. Sooo... gotta be something in Informix ... On 07/10/2013 04:42 PM, Ben Duncan wrote: > Yes, we can ssh / sftp / scp without any problems. > > I may have network guys do a trap on packets tomorrow for weirdness > (Thanks Art for that Idea). > > On 07/10/2013 04:02 PM, Justin Killen wrote: >> So aside from IDS, other services work (ssh, http, etc) in both directions? >> >> -Justin > -- Ben Duncan - Business Network Solutions, Inc. 336 Elton Road Jackson MS, 39212 "Never attribute to malice, that which can be adequately explained by stupidity" - Hanlon's Razor

I'd suggest going back to Cesar's post and double checking the results from: nslookup <hostname from sqlhosts> to make sure that's the right ip. Also, when you say you're changing the IP - what exactly are you changing? (ifconfig, edit sqlhosts, etc) -Justin -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Ben Duncan Sent: Thursday, July 11, 2013 1:41 PM To: ids@iiug.org Subject: Re: Server Install Weirdness: [30827] Final result: No one and nothing is using the 10.242.254.100 address. No even an entry in arp cache on any of the routers. Change the IP back to 10.242.254.100 and BANG - start getting "shared memory errors" randomly again. Sooo... gotta be something in Informix ... On 07/10/2013 04:42 PM, Ben Duncan wrote: > Yes, we can ssh / sftp / scp without any problems. > > I may have network guys do a trap on packets tomorrow for weirdness > (Thanks Art for that Idea). > > On 07/10/2013 04:02 PM, Justin Killen wrote: >> So aside from IDS, other services work (ssh, http, etc) in both directions? >> >> -Justin > -- Ben Duncan - Business Network Solutions, Inc. 336 Elton Road Jackson MS, 39212 "Never attribute to malice, that which can be adequately explained by stupidity" - Hanlon's Razor ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Ben, If you don't have any other alternative... I suggest open a PMR or start play with "strace" command and try figure out what's happen. Regards Cesar 2013/7/11 Ben Duncan <linux4ms@aim.com> > Final result: No one and nothing is using the 10.242.254.100 address. No > even > an entry in arp cache on any of the routers. > > Change the IP back to 10.242.254.100 and BANG - start getting "shared > memory > errors" randomly again. > > Sooo... gotta be something in Informix ... > > On 07/10/2013 04:42 PM, Ben Duncan wrote: > > Yes, we can ssh / sftp / scp without any problems. > > > > I may have network guys do a trap on packets tomorrow for weirdness > > (Thanks Art for that Idea). > > > > On 07/10/2013 04:02 PM, Justin Killen wrote: > >> So aside from IDS, other services work (ssh, http, etc) in both > directions? > >> > >> -Justin > > > > -- > Ben Duncan - Business Network Solutions, Inc. 336 Elton Road Jackson MS, > 39212 > "Never attribute to malice, that which can be adequately explained by > stupidity" > > - Hanlon's Razor > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --14dae9cc929a7f1eee04e151b332

Didn't matter, since the network engineers found nothing, but to satisfy everyone's curiosity: SQLHOSTS: sup02111:~ # cat /opt/informix/etc/sqlhosts ## ## Primary Server ## pdc61103_ecf onsoctcp 10.242.254.100 7110 pdc61103_shm onipcshm 10.242.254.100 pdc61103_shm ## ## Replica Server ## rdc61103_ecf onsoctcp 10.242.254.103 7110 rdc61103_shm onipcshm 10.242.254.103 rdc61103_shm sup02111:~ # nslookup pdc61103_ecf ;; connection timed out; no servers could be reached sup02111:~ # nslookup pdc61103_shm ;; connection timed out; no servers could be reached sup02111:~ # nslookup 10.242.254.100 ;; connection timed out; no servers could be reached Sooo.. Am back to : Somethings strange with Informix Using .100 address. On 07/11/2013 04:06 PM, Justin Killen wrote: > I'd suggest going back to Cesar's post and double checking the results from: > > nslookup <hostname from sqlhosts> > SNIP -- Ben Duncan - Business Network Solutions, Inc. 336 Elton Road Jackson MS, 39212 "Never attribute to malice, that which can be adequately explained by stupidity" - Hanlon's Razor

I guess Monday I will track down thems that have the keys to submit a PMR. On 07/12/2013 09:55 AM, Cesar Inacio Martins wrote: > Ben, > > If you don't have any other alternative... I suggest open a PMR or start > play with "strace" command and try figure out what's happen. > > Regards > Cesar > -- Ben Duncan - Business Network Solutions, Inc. 336 Elton Road Jackson MS, 39212 "Never attribute to malice, that which can be adequately explained by stupidity" - Hanlon's Razor

You need to change your SQLHOSTS entry to use the IP address of the host and then restar your Informix server.