Assert Fail - onmode hangs

SunOs 5.8 IDS 9.30.UC1X6 / 9.40.UC4 We've had a problem with Assert Failures causing IDS to hang. We have opened cases with Informix and (correctly so) have been told to upgrade the 9.3 servers. We have been able to do that with some of our systems, but not yet all of them. After capturing any available onstat information, an additional problem has occurred on several occasions. When we run "onmode -ky", we never get a completion of the statement. It hangs. IDS remains running, but in a useless state. We have had to shutdown the machine to get the IDS server to start successfully again via oninit. The system reboot also hangs until we "kill -9" the oninit processes because the shutdown scripts include "onmode -ky", which isn't working. We have tried to kill the processes and then clear out shared memory, but that hasn't worked either. Has anyone ever seen this situation before? Have you been able to correct it without bouncing the machine? Is there any way to know what Informix process, shared memory, or whatever is causing it to hang? Any help would be appreciated. Thanks, Gary

Hi Gary, could you please share with us the first 20 lines of your af files? Have you seen any sort of message after you executes the onmode -ky ? I don't understand what you mean with : "We have tried to kill the processes and then clear out shared memory, but that hasn't worked either" If you kill all the oninit processes and clean up all the share memory segments the engine won't be for sure running on that box. What do you mean saying that it hasn't worked either ? Please post your af files and log files. Regards esteban

i suspectt hat you have shared memeory dumping turned on and that is why the serever appears to "hang" when it AFs What is probably happening is that activity stops while the onstat process dumps the contents of shared memory to disk. This may also explain why killing the process doesn't resolve the problem a I think its the onstat process not the oninit process that dumps the memory to disk Gary Andrus wrote: > SunOs 5.8 > IDS 9.30.UC1X6 / 9.40.UC4 > > We've had a problem with Assert Failures causing IDS to hang. We have > opened cases with Informix and (correctly so) have been told to upgrade > the 9.3 servers. We have been able to do that with some of our > systems, but not yet all of them. > > After capturing any available onstat information, an additional problem > has occurred on several occasions. When we run "onmode -ky", we never > get a completion of the statement. It hangs. IDS remains running, but > in a useless state. We have had to shutdown the machine to get the IDS > server to start successfully again via oninit. The system reboot also > hangs until we "kill -9" the oninit processes because the shutdown > scripts include "onmode -ky", which isn't working. > > We have tried to kill the processes and then clear out shared memory, > but that hasn't worked either. > > Has anyone ever seen this situation before? Have you been able to > correct it without bouncing the machine? Is there any way to know what > Informix process, shared memory, or whatever is causing it to hang? > > Any help would be appreciated. > > Thanks, > Gary

Esteban Casuscelli wrote: > Hi Gary, > could you please share with us the first 20 lines of your af files? > > Have you seen any sort of message after you executes the onmode -ky ? > > I don't understand what you mean with : "We have tried to kill the > processes and then clear out shared memory, > but that hasn't worked either" > > If you kill all the oninit processes and clean up all the share memory > segments the engine won't be for sure running on that box. > What do you mean saying that it hasn't worked either ? > > Please post your af files and log files. > > Regards > esteban Esteban, Here are the first 20 lines of the AF file: [107]% head -20 af.7b0b5bbc 15:27:41 15:27:41 Informix Dynamic Server Version 9.30.UC1X6 Software SerialNumber AAD#J292950 15:27:41 Assert Failed: Exception Caught. Type: MT_EX_OS, Context: mem 15:27:41 Who: Session(25312, informix@9d7989c7, -1, 452600008) Thread(30499, sqlexec, 18173818, 6) File: mtex.c Line: 368 15:27:41 Action: Please notify Informix Technical Support. 15:27:41 Stack for thread: 30499 sqlexec base: 0x22c12000 len: 135168 pc: 0x006b25b8 tos: 0x22c31db0 state: running vp: 6 0x006b1b24 (oninit)afhandler(0xa80000, 0xa09c9c, 0xa7fe48, 0xa0a0e4, 0x1, 0x7b0b5bbc) 0x006b1430 (oninit)affail_interface(0x22c324c8, 0x0, 0x1800a578, 0x0, 0xa0a0e4, 0x170) 0x006b5658 (oninit)mt_ex_throw_sig(0x91e0b0, 0xa07248, 0x0, 0xa0a0c4, 0xa6fce4, 0x94cce0) sidb1p:informix[/opt/prod/ids_coredumps/tlipita]-[108]% We have run kill -9 on the oninit processes and then ipcs -a followed by the necessary ipcrm commands and cleaned up all shared memory. I was still unable to successfully run an oninit command. I have since found the following message in the online log file: 16:29:58 listener-thread: err = -25572: oserr = 0: errstr = : Networkdriver cannot bind a name to the port. Is there something else that needs to be cleared up (at the system level, maybe)? Thanks, Gary

the netwrok ports are still open what is SHMDUMP set to in your online log file?? if its set to one, try setting it to 0 and see if IDS still hangs when you get an AF. This will not stop the AFs. Gary Andrus wrote: > Esteban Casuscelli wrote: > > Hi Gary, > > could you please share with us the first 20 lines of your af files? > > > > Have you seen any sort of message after you executes the onmode -ky ? > > > > I don't understand what you mean with : "We have tried to kill the > > processes and then clear out shared memory, > > but that hasn't worked either" > > > > If you kill all the oninit processes and clean up all the share memory > > segments the engine won't be for sure running on that box. > > What do you mean saying that it hasn't worked either ? > > > > Please post your af files and log files. > > > > Regards > > esteban > > Esteban, > > Here are the first 20 lines of the AF file: > > [107]% head -20 af.7b0b5bbc > 15:27:41 > 15:27:41 Informix Dynamic Server Version 9.30.UC1X6 Software Serial> Number AAD#J292950 > > 15:27:41 Assert Failed: Exception Caught. Type: MT_EX_OS, Context: mem > 15:27:41 Who: Session(25312, informix@9d7989c7, -1, 452600008) > Thread(30499, sqlexec, 18173818, 6) > File: mtex.c Line: 368 > 15:27:41 Action: Please notify Informix Technical Support. > 15:27:41 Stack for thread: 30499 sqlexec> > base: 0x22c12000 > len: 135168 > pc: 0x006b25b8 > tos: 0x22c31db0 > state: running > vp: 6 > > 0x006b1b24 (oninit)afhandler(0xa80000, 0xa09c9c, 0xa7fe48, 0xa0a0e4, > 0x1, 0x7b0b5bbc) > 0x006b1430 (oninit)affail_interface(0x22c324c8, 0x0, 0x1800a578, 0x0, > 0xa0a0e4, 0x170) > 0x006b5658 (oninit)mt_ex_throw_sig(0x91e0b0, 0xa07248, 0x0, 0xa0a0c4, > 0xa6fce4, 0x94cce0) > sidb1p:informix[/opt/prod/ids_coredumps/tlipita]-[108]% > > > We have run kill -9 on the oninit processes and then ipcs -a followed > by the necessary ipcrm commands and cleaned up all shared memory. I > was still unable to successfully run an oninit command. I have since > found the following message in the online log file: > > 16:29:58 listener-thread: err = -25572: oserr = 0: errstr = : Network> driver cannot bind a name to the port. > > Is there something else that needs to be cleared up (at the system > level, maybe)? > > Thanks, > Gary

Yes the network socket IDS was listening on. Do netstat -a and this will be in time wait. Nothing you can do but wait until the TIME_WAIT state times out and releases the port. You should never need to run kill -9 on oninit processes. What is IDS doing? We need the first 40-50 lines of the af file. The first 20 lines ends in a stack trace but we don't have all of it! 9.30 was an unstable version especially a UC1 version. Consider upgrading to at least 9.40.latest. Run onstat -g stk all and exclude the sqlexec and btcleaner threads. What else is there?