Re: Dumb DBA mistakes

There goes one from the backup and restore area: DBAs worked out a very elaborate backup plan for their IDS instance using ontape and nightly archiving of different levels. Level 0 on Sundays, level 1 on Mondays, Wednesdays and Fridays, level 2 on Tuesdays and Thursdays. Tapes were meticuously changed every day, right on schedule for the nightly archiving. In addition they kept all tapes for 3 weeks and after that 1 weekly tape (level 0) for 3 months and again after that 1 monthly tape (level 0) forever. All the tapes nicely and orderly stored for easy and precise access should one be needed. As the day came (after years of strictly adhering to the above scheme), they pulled out the archive tape for restore, but it didn't work. As it can be possible that something's wrong with one tape, they tried the next tape, also not restorable. Over time they found out that none of the archives on the tapes was restorable. As it turned out, all the tapes were empty. The nightly ontape commands to do the archiving were never executed. Something went wrong with the cron setup, but nobody noticed. Nobody ever checked an archive for restorability. Nobody noticed that there never was a message about an archive started / completed in the IDS message log. They had many unused tapes in their storage. In the end they had to settle on some dbexport data that was 1.5 years old but still existed to restore their system ... Regards, Martin -- Martin Fuerderer IBM Informix Development Munich, Germany Information Management Visit one of the upcoming Infobahn events in Germany to get up-to-date information on Informix products: - June 20, 2006 at the IBM Forum Muenchen - June 21, 2006 at the IBM Forum Frankfurt - June 22, 2006 at the IBM Forum Berlin For more please see: http://www.ibm.com/de/events/infobahn/

That would be the grand prize winner in my book. Because it tooks years of doing something wrong and complete arrogance not to test it. I always assume something isn't working and then try to prove that wrong. ;-) This comes back to Andy Groves "Only the Paranoid Survive." These are great mistakes. Collecting these and more mistakes and then including best practices as a counter balance would make a nice session.

bozon wrote: > That would be the grand prize winner in my book. Because it tooks years > of doing something wrong and complete arrogance not to test it. I > always assume something isn't working and then try to prove that wrong. > ;-) This comes back to Andy Groves "Only the Paranoid Survive." > > These are great mistakes. Collecting these and more mistakes and then > including best practices as a counter balance would make a nice session. > The one day that a restore was needed after a "bad batch run", and the operator accidentally did what they did every evening : ontape -s L 0 thereby kicking off a backup onto the tape they needed to restore from. Moral is to write protect your backups. A shift change looks at the tape, sees todays time and date, and assumes the backup has been done, then places it in to the backup rack. Unfortunately, the previous op had labeled and loaded the tape prior to running the backup. Moral is to write on the backup tapes AFTER running the backup.