Down System Support and Data Recovery

So you are out of support and the system is down? So you have support, but it's the middle of the night? So you have down chunks? So you have a system stuck in physical recovery? So you have a system stuck in logical recovery?

Examples of Recovered Systems

Problem: Sysadmin newfs'd 360GB of disk (180 chunks) allocated to the main (750GB) dbspace.
Action: Patched the chunks back into the system and brought the dbspace back online.

Problem: SAN powered down while the engine was online, leaving 10 down chunks.
Action: Patched the chunk and dbspace structure and brought the dbspace back online. Recovery time: 15 mins. Data loss: NIL.

Problem: Catastrophic rootdbs failure on an OnLine V5 system.
Action: Extracted system information from the failed disk. Using this data, wrote custom software to extract the table structures for the failed system. Finally applied the table structures to the extracted binary data and generated ASCII unload files.

Problem: The system failed to restart after a restore.
Action: The customer was hitting bug IC56156 — IDS cannot start after restoring an ontape archive from an HDR primary server which was previously a secondary. The recommendation from IBM to upgrade was not possible as support had lapsed. Oninit® edited the rootdbs and manually reset a variety of flags that were telling the engine it used to be an RSS secondary.

Problem: Catastrophic failure when all the chunk links were removed.
Action: Stepped through the raw devices and, in conjunction with the reserved pages, recreated all the missing links.

Problem: Insufficient logical log space for rollback, leading to a locked system.
Action: Edited reserved areas using custom software and increased the logical log space to allow the transaction to roll back successfully. Recovery time: 45 mins. Data loss: NIL.

Problem: Engine crash when moving a large table. The instance crashed when the customer was trying to move a larger table to a newly created dbspace. The engine failed to start with a failure to complete logical recovery. The logical record that was failing was the creation of the new dbspace. The customer was on an old version of the engine and support had lapsed; when Oninit® were contacted they had been down for over 3 days.
Action: Manually completed the transaction and brought the dbspace online. Manually patched the table with the dbspace into the system tables. Recovery time: 45 mins. Data loss: NIL.