tbinit core dumps during recovery, urgent help needed!!!
Posted in 1995
Topics: General Discussion
We have a problem that needs urgent fixing. Informix scandinavia support didn't call back before the office closed today, so here I sit at my summer house spoiling an otherwise nice vacation evening trying to solve a rather hard problem. Fun? Not! The problem, in essence, is that we've had an Online crash, probably due to a bad index in our database. When tbinit starts it starts recovery and chews up stack space quite fast sigsegving after some seconds, leaving a nice 50 MB coredump (ulimited up the stack limit to 32 MB). The online.log leaves a message saying that a delete couldn't be done due to a bad index node or something (sorry can't reach the exact message right now). The normal way of fixing this is of course tbcheck. But, catch-22, tbcheck cannot run after tbinit has died... It *is* possible to start tbcheck *before* tbinit dies, and the problem is found, bad btree links, I think, but then it tries yet another time to make certain that Online is in Quiscent mode, which it isn't due to the since-long died tbinit (the table contains some millions of records of each some hundred bytes and six indices, so it takes a while before the index anomaly is detected, and tbinit dies in a few seconds time!). So i tried putting tbinit in a low priority, putting a busy-loop at little higher priority, and lastly put tbcheck at the highest prio, in order to starve out tbinit so it would still exist when tbcheck got to the point of the Quiscent mode check, but alas, this trial failed as well. In fact, I may have screwed up the busy loop as I used dd, /dev/zero and /dev/null, with no explicit knowledge if this would really be starving out the tbinit enough. I guess a while (1) in C would be a better bet, but I can't check right now... I now have two ideas which I don't know exactly how to try: A Remove the flag (somewhere in the reserved pages) that says that recovery is needed. Accept the incosistencies that may follow from this decisions, and run tbcheck before going online. B Somehow remove the bad index. Let tbinit recover, and create the index afterwards. C Remove the test (or somehow fake it) in tbcheck for Quiscent mode when the repairing of the bad index should start. Anyone else got other ideas? I would really appreciate them, even if they lead nowhere. I'm desparate! Our customer is quite dependent on this system as it does all the invoicing for them, and they invoice some $100K each day. I have no real belief in Informix Scandinavie, they have not acted very promptly before. Besides US is awake now, Sweden not until tomorrow.. PLEASE HELP if you can. ASAP! We must have solved this in 24hours time, otherwise a costly backup process must be started. Due to vacations and some human-factors the last consistent backup is a week old and occurred in the middle of a very delicate phase which we don't want to redo. :-( Niklas Hallqvist -- Niklas Hallqvist Phone: +46-(0)31-40 75 00 Applitron Datasystem Fax: +46-(0)31-83 39 50 Molndalsvagen 95 Email: niklas@appli.se S-412 63 GOTEBORG WWW: <A HREF="http://www.cd.chalmers.se/~nh">Here</A> Sweden