Re: help help help help help help help help help
Posted in 1994
We lost our root chunk for a moment and of course the database went down in flames. When we tried to recover we had a problem similar to yours: the DAEMON DIED message appeared and we could not convince the database that it was no longer recovering. What followed was, of course, a very long evening. We thought that the last log on the tape was bad, and was causing the recovery to fail. After several failed attempts to restore, I recovered the database from an earlier archive and rolled forward. It hung on the bad log. So we restored again, and then rolled forward logs, this time stopping before the latest tape. The message "Recovery complete" appeared in the Informix turbo.log . The tbmonitor said something like "Please wait..." but we also saw the "DAEMON DIED" message. The tbmonitor ticked away for about 45 minutes. I got tired of waiting and killed off the four orphaned tbinit -rs processes (spawn of a dead daemon perhaps???) that I saw running and then killed off the hung tbmonitor process. When I brought tbmonitor back up, I was able to bring the database into quiescent mode and then online. The turbo.log recorded some messages that looked usual for database startup: some kind of recovery and a cleanup of temp tables. A succession of tbchecks confirmed that all was right in the database, although we were a day behind. Fortunately, on that particular day for that particular database being a day behind was not a problem. I did not experiment with rolling through the bad log and killing tbinits. After 6 hours of spinning tapes, I went home. Most of this disaster occurred after Informix tech support hours. I reported the problem to Informix, and they tried to duplicate it but failed. They did not recover from tape, though. I gave up with them after they asked me to try it again here. Our platform: SCO Unix 3.2.4.1 on a i80486 Informix Online 5.01.UD2 Sigh.