Effect of System Crash on RootDbSpace
Posted in 1993
Netters: At one of our customer sites, we have set up two *identical* systems serving as database servers. One is the primary machine and the other we use in case of a failure. Due to parity error on a RAM chip the main system paniced and the crash dump analysis showed that some DMA (memory to disk) access was in progress at the time of crash. As a result of this crash our root database space was corrupted. The symptoms of the corruption were very peculiar. Online log showed no errors. On restarting the engine (tbinit), the engine came up normally. But the very first access after this point (via tbmonitor or isql) gave the error that the daemon is dead!! (Nothing in the log though !!) As the crash had occurred while the disk was being written to we theorized that the root space must have been thrashed. Of course, we quickly got the customer running on the redundant server. What we missed was that the redundant hardware was identical to the main server (same cards/config, same video, same RAM etc, same disk, partion size, DB raw device size etc.). Apparently there were two different kinds of RAM chips used to reach 40M RAM requirement, which we later found out from the manufacturer (DELL) were not compatible causing the parity error. So of course, after few hours of functioning, the second system also took the parity error hit and crashed. Again the crash analysis showed DMA actions - and again the root space was corrupted. Archive/Restore worked fine and we have removed the bad chips form both servers but are left with questions .... Question 1: We have a complete mirroring set up for our root space. Isn't mirroring suppose to protect us against exactly what happened ?? Question 2: Is it a coincidence that the same action hit the same sections of the disk so as to corrupt the disk the same way ?? Question 3: What part of the root DB will be corrupted - so that it will initialize ok but crash at first access ?? Any light thrown on the subject is welcome. Bharat ======================================================================== Informix-OnLine 5.01 SCO 3.2.4 (Odt 3.0) Informix-Isql 4.11.UC1 AST 486/66 Informix-Esql/C 5.00 Wingz 1.10 Bharat Shah - Principal Database Engineer Voice: (508)-663-7570 Teloquent Communications Corp. Fax: (508)-663-7543 4 Federal St, Billerica, Mass 01821, USA bharat@unifi.com ========================================================================