Re: Oninit from cron leaves engine in permanent fast recovery

gerry.cassidy@dsl.pipex.com wrote: > We have a requirement to automatically bounce our 9.40.FC6 instance on > AIX 5.2. > > The idea is to use cron (informix) to run a script that sets the > correct environment, etc. and then issues a 'onmode -yuk', sleeps for > 30 seconds then issues a 'oninit -v'. > > This script ... and a cut-down wee version ... runs simply and > beautifully from the informix shell. But, when cron executes the > script the instance stops cleanly but fails to restart. > > We have compared environments and have not found anything obvious that > could account for this strange behaviour. > > Have any of you experienced anything like this before? I have included > the output from the script for both shell and cron methods of > execution. I have not included any configuration information as yet. > This would appear to be a deep-rooted problem, possibly to do with > process management under AIX ... or something like that ... ?! My > AIX/UNIX knowledge doesn't stretch that far! > > Thanks in advance for any help anyone can offer. Couple of things to consider: 1. What is the current working directory when you run the command via the shell? Where does cron run things? Where does your shell run them? Does it matter? (The answer to the last is usually no, but you can design your system so that it does matter if you choose to do so - using relative pathnames instead of absolute ones.) 2. Which file descriptors are open in the shell? And which in cron? Chances are that the input in cron is coming from /dev/null; I don't think that should matter, but ... Looking at the outputs, the shell version contains these lines: Initializing DBSPACETEMP list Checking database partition index...succeeded Checking location of physical log...succeeded Initializing dataskip structure...succeeded Checking for temporary tables to drop Forking onmode_mon thread...succeeded Verbose output complete: mode = 5 Whereas the cron version stops at the first... I'm not sure what inferences to draw from that, but it certainly should give someone some ammunition. > Yours dsperately, > Mr Zap > > Script > ----------------------------------------- > #!/bin/ksh > . /opt/informixdba/Profiles/dba_profile 940 shm > /usr/informix/bin/onmode -yuk > sleep 30 > /usr/informix/bin/oninit -v > ----------------------------------------- > Output from shell > ----------------------------------------- > Checking group membership to determine server run modesucceeded > Reading configuration file > '/usr/informix/etc/onconfig.gimukldnp01_00'...succeeded > Creating /INFORMIXTMP/.infxdirs ... succeeded > Creating infos file "/usr/informix/etc/.infos.gimukldnp01_00" ... > "/usr/informix/etc/.conf.gimukldnp01_00" ... succeeded > Writing to infos file ... succeeded > Checking config parameters...succeeded > Allocating and attaching to shared memory...succeeded > Creating resident pool 43988 kbytes...succeeded > Creating buffer pool 800000 kbytes...succeeded > Creating buffer pool 8 kbytes...succeeded > Initializing rhead structure...succeeded > Initializing ASF ...succeeded > Initializing Dictionary Cache and SPL Routine Cache...succeeded > Bringing up ADM VP...succeeded > Creating VP classes...succeeded > Onlining 0 additional cpu vps...succeeded > Onlining 2 IO vps...succeeded > Initialization of Encryption...succeeded > Forking main_loop thread...succeeded > Initializing DR structures...succeeded > Forking 1 'sqlmux' listener threads...succeeded > Forking 1 'ipcshm' listener threads...succeeded > Forking 1 'soctcp' listener threads...succeeded > Forking 1 'soctcp' listener threads...succeeded > Forking 1 'soctcp' listener threads...succeeded > Forking 1 'soctcp' listener threads...succeeded > Starting tracing...succeeded > Initializing 6 flushers...succeeded > Initializing log/checkpoint information...succeeded > Opening primary chunks...succeeded > Opening mirror chunks...succeeded > Initializing dbspaces...succeeded > Validating chunks...succeeded > Initialize Async Log Flusher...succeeded > Forking btree cleaner...succeeded > Initializing DBSPACETEMP list > Checking database partition index...succeeded > Checking location of physical log...succeeded > Initializing dataskip structure...succeeded > Checking for temporary tables to drop > Forking onmode_mon thread...succeeded > Verbose output complete: mode = 5 > ----------------------------------------- > onstat - > ----------------------------------------- > IBM Informix Dynamic Server Version 9.40.FC6 -- On-Line -- Up > 00:06:42 -- 1633776 Kbytes > -----------------------------------------> Output from cron > ----------------------------------------- > Checking group membership to determine server run modesucceeded > Reading configuration file > '/usr/informix/etc/onconfig.gimukldnp01_00'...succeeded > Creating /INFORMIXTMP/.infxdirs ... succeeded > Creating infos file "/usr/informix/etc/.infos.gimukldnp01_00" ... > "/usr/informix/etc/.conf.gimukldnp01_00" ... succeeded > Writing to infos file ... succeeded > Checking config parameters...succeeded > Allocating and attaching to shared memory...succeeded > Creating resident pool 43988 kbytes...succeeded > Creating buffer pool 800000 kbytes...succeeded > Creating buffer pool 8 kbytes...succeeded > Initializing rhead structure...succeeded > Initializing ASF ...succeeded > Initializing Dictionary Cache and SPL Routine Cache...succeeded > Bringing up ADM VP...succeeded > Creating VP classes...succeeded > Onlining 0 additional cpu vps...succeeded > Onlining 2 IO vps...succeeded > Initialization of Encryption...succeeded > Forking main_loop thread...succeeded > Initializing DR structures...succeeded > Forking 1 'sqlmux' listener threads...succeeded > Forking 1 'ipcshm' listener threads...succeeded > Forking 1 'soctcp' listener threads...succeeded > Forking 1 'soctcp' listener threads...succeeded > Forking 1 'soctcp' listener threads...succeeded > Forking 1 'soctcp' listener threads...succeeded > Starting tracing...succeeded > Initializing 6 flushers...succeeded > Initializing log/checkpoint information...succeeded > Opening primary chunks...succeeded > Opening mirror chunks...succeeded > Initializing dbspaces...succeeded > Validating chunks...succeeded > Initialize Async Log Flusher...succeeded > Forking btree cleaner...succeeded > Initializing DBSPACETEMP list > ----------------------------------------- > onstat - > ----------------------------------------- > IBM Informix Dynamic Server Version 9.40.FC6 -- Fast Recovery -- Up > 00:00:06 -- 1633776 Kbytes > -----------------------------------------> > It stays in fast recovery mode so long ... we have to kill the oninit > processes! > > All of the dbspaces are valid ... including temp spaces. > -- Jonathan Leffler #include <disclaimer.h> Email: jleffler@earthlink.net, jleffler@us.ibm.com Guardian of DBD::Informix v2005.02 -- http://dbi.perl.org/

Thanks for the responses. Including a 'c' in the shutdown doesn't make any difference Dave ... sadly. I have tried flushing, forcing, advancing, clearing shm ... all to no avail. There are no relative paths referenced in the script. It's all very explicit. The script has been executed from different staring points. I don't know about open file descriptors Jonathan, but I will find out. There is no input and output is redirected to a log file in both cases. I'm not sure what to draw from the fact that it hangs on the temp space initialisation either ... but it would seem like a clue :) There is, however, nothing apparently wrong with the spaces. Thanks for the suggestions.

When it hang what does the online log say? What does onstat - say?