Restarting Informix after CPIO to Other Storage
We have a DB with our Informix chunks on a SAN, and our upgrade from Informix
9.4 to Informix 10 is expected to take a long time. Any failure of the
dbimport could result in an extended outage well beyond the time available.
I'm proposing being able to roll back and restore operations simply by
stopping Informix, remounting the original LUN (and reversing the Informix
RPM), and then restarting Informix.
The plan would be to stop Informix, then mirror, and break the mirror. Hence,
the SAN would then present a copy of the storage space. The entire upgrade
would take place working on a copy of the original.
To further ensure the rollback would be a restore of the previous state, we
could make a copy of the informix directory. Therefore any and all changes
that might occur during the upgrade (to oncfg, onconfig, and so on in ~etc)
would only affect the directory that's live during the upgrade.
Informix is extremely touchy about any changes made to config files,
especially as a result of fooling around with switching data chunks. When you
restart (with "oninit"), it's not difficult to end up with informix declaring
the chunk data to be invalid.
So does anyone have a checklist of vital files that should be backed up or
watched, such that the above procedure could be performed with a reasonable
level of certainty the database would come up again? This scenario could
equally well play itself out if you mounted a copy of local storage space for
the chunks (it wouldn't have to involve a SAN).
Why not just perform the upgrade in-place? There would be no dbimport
time. You would:
1. Install IDS 10.00 in a separate directory from the 9.40 installation.
2. Copy the sqlhosts and ONCONFIG file from the 9.40 installation to the
10.00 installation
1. Make any desired changes to the ONCONFIG file to take advantage of
new features in 10.00 or
2. Change deprecated ONCONFIG parameters for new ones (ie NUMCPUVPS,
AFFNPROC, AFFSPROC, NOAGE for VPCLASS cpu,num=...)
3. Take a full archive of the server for safety and backup using 9.40
ontape or onbar as appropriate.
1. If you use onbar you will have to save the archive history as it
will be destroyed during the upgrade. See the Backup and Restore Guide
manual for details.
4. Shutdown the 9.40 instance
5. Switch INFORMIXDIR to point to the 10.00 installation (you should be
using a link for INFORMIXDIR to make this seamless - if not rename the 9.40
INFORMIXDIR and replace it with a link that you will use going forward)
6. Start up IDS 10.00 and wait for the engine to complete internal
conversions and come online.
7. Drop data distributions for all tables in all databases
1. In each database, as informix: DELETE FROM sysdistrib;
8. Recreate all data distributions per the Performance Guide
recommendations (you can use my dostats utility to make this easier and if
this is a large database use the drive_dostats script to update multiple
tables in parallel).
9. Recompile all stored procedures in all databases (except the system
databases those will have been recreated from scratch during the upgrade):
UPDATE STATISTICS FOR PROCEDURE; or let dostats do that for you as well.
10. Take another archive using the IDS 10.00 tools.
If the upgrade fails the primary recovery is:
1. Shutdown the 10.00 instance with
onmode -b 9.40
to revert any internal modifications the 10.00 software had to make.
2. Switch INFORMIXDIR to point to the 9.40 installation.
3. Start up IDS 9.40 and wait for the engine to come online.
4. Repeat steps 7-9 above
The fallback recovery is to restore the archive you took using 9.40 before
the upgrade. If you want to break the mirror before the upgrade you would
add another level of recovery between these by copying the mirrors back to
the primaries and just restarting 9.40.
Art S. Kagel
Oninit (www.oninit.com)
IIUG Board of Directors (art@iiug.org)
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Oninit, the IIUG, nor any other organization
with which I am associated either explicitly or implicitly. Neither do
those opinions reflect those of other individuals affiliated with any entity
with which I am affiliated nor those of the entities themselves.
On Mon, May 18, 2009 at 9:35 AM, CHARLES KIRBY <iiug@500pearl.com> wrote:
> We have a DB with our Informix chunks on a SAN, and our upgrade from
> Informix
> 9.4 to Informix 10 is expected to take a long time. Any failure of the
> dbimport could result in an extended outage well beyond the time available.
>
> I'm proposing being able to roll back and restore operations simply by
> stopping Informix, remounting the original LUN (and reversing the Informix
> RPM), and then restarting Informix.
>
> The plan would be to stop Informix, then mirror, and break the mirror.
> Hence,
> the SAN would then present a copy of the storage space. The entire upgrade
> would take place working on a copy of the original.
>
> To further ensure the rollback would be a restore of the previous state, we
> could make a copy of the informix directory. Therefore any and all changes
> that might occur during the upgrade (to oncfg, onconfig, and so on in ~etc)
> would only affect the directory that's live during the upgrade.
>
> Informix is extremely touchy about any changes made to config files,
> especially as a result of fooling around with switching data chunks. When
> you
> restart (with "oninit"), it's not difficult to end up with informix
> declaring
> the chunk data to be invalid.
>
> So does anyone have a checklist of vital files that should be backed up or
> watched, such that the above procedure could be performed with a reasonable
> level of certainty the database would come up again? This scenario could
> equally well play itself out if you mounted a copy of local storage space
> for
> the chunks (it wouldn't have to involve a SAN).
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--001636c923d7a11831046a323a5a
That's excellent information, thanks, Art. And that would be a very good procedure for doing such an upgrade. Unfortunately, the nature of the upgrade in our case consists of something dictated by our head office, and it's largely automated. In any case, it's important that we be able to assure a rapid return to normal operation in the event of failure. In fact, I'm sure this would be of universal interest to Informix users! What I'm proposing is to effectively isolate, and duplicate the database chunks. Likewise the informix package. So, while the informix engine is completely stopped, the plan is to copy the whole ~informix directory to a cloned partition. The same goes for the database chunks. The idea here is to be able to return to the original operation at any moment, simply by switching back to the original partitions and starting up the server, the way it looked before attempting the upgrade. I see there are directories like "CHUNK_BM", "INFO", and also "SAVE" under the ~informix/tmp directory. I wonder if there are any obscure directories or files like these that might have some input in the case where the database is restored? Perhaps the logging comes into this? Perhaps there's a change made somewhere that we're not familiar with, that could send the above plan awry? I'd like to know if there is some specific, perhaps hidden, file or directory that could be altered adversely whenever the database engine is started. Maybe one way to ensure an exact duplicate of the O.S. is to simply stop the server and remove one of the two hard drives that make up the raidset containing the operating system and Informix package. Are there any pitfalls with such a method? Imagine being able to have a clone of the Informix chunks, and a clone of the O.S.; surely (in the event of disaster) it should be possible to return to the previous database by stopping the server, and reinstalling the second half of a RAID 1 that was removed prior to the upgrade?
If the server is offline or in a checkpoint block (onmode -c block) mode,
then it is safe to just copy - or break the mirror to - the chunks and be
able to use those files to restore the engine to the point of the last
checkpoint (either the one at shutdown or the one the onmode performs)
safely. This is known as an external archive and is supported. No other
directories are needed beyond the IDS software and the chunks.
Art
Art S. Kagel
Oninit (www.oninit.com)
IIUG Board of Directors (art@iiug.org)
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Oninit, the IIUG, nor any other organization
with which I am associated either explicitly or implicitly. Neither do
those opinions reflect those of other individuals affiliated with any entity
with which I am affiliated nor those of the entities themselves.
On Wed, May 27, 2009 at 2:45 PM, CHARLES KIRBY <iiug@500pearl.com> wrote:
> That's excellent information, thanks, Art. And that would be a very good
> procedure for doing such an upgrade.
>
> Unfortunately, the nature of the upgrade in our case consists of something
> dictated by our head office, and it's largely automated. In any case, it's
> important that we be able to assure a rapid return to normal operation in
> the
> event of failure.
> In fact, I'm sure this would be of universal interest to Informix users!
>
> What I'm proposing is to effectively isolate, and duplicate the database
> chunks. Likewise the informix package.
>
> So, while the informix engine is completely stopped, the plan is to copy
> the
> whole ~informix directory to a cloned partition. The same goes for the
> database chunks. The idea here is to be able to return to the original
> operation at any moment, simply by switching back to the original
> partitions
> and starting up the server, the way it looked before attempting the
> upgrade.
>
> I see there are directories like "CHUNK_BM", "INFO", and also "SAVE" under
> the
> ~informix/tmp directory. I wonder if there are any obscure directories or
> files like these that might have some input in the case where the database
> is
> restored?
>
> Perhaps the logging comes into this? Perhaps there's a change made
> somewhere
> that we're not familiar with, that could send the above plan awry? I'd like
> to
> know if there is some specific, perhaps hidden, file or directory that
> could
> be altered adversely whenever the database engine is started.
>
> Maybe one way to ensure an exact duplicate of the O.S. is to simply stop
> the
> server and remove one of the two hard drives that make up the raidset
> containing the operating system and Informix package. Are there any
> pitfalls
> with such a method? Imagine being able to have a clone of the Informix
> chunks,
> and a clone of the O.S.; surely (in the event of disaster) it should be
> possible to return to the previous database by stopping the server, and
> reinstalling the second half of a RAID 1 that was removed prior to the
> upgrade?
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--001636c5c2f180b054046ae95366