Test Restoration failed
Posted in 2008
A user restoring an IDS 11.10 level-0 ontape backup from dev onto a test server failed because the test instance was built with one chunk fewer (datadbs2_04 missing), causing 'Cannot open chunk' and oninit shared-memory initialization failure. Advice: pre-create the missing chunk as an empty file with correct informix ownership/permissions and re-run ontape -r; also note ONDBSPACEDOWN controls whether a down dbspace aborts the server. The user ended up reinitializing the instance and restoring successfully, but then complained the restore took 15+ hours instead of 7-8; Art Kagel suspected the test box's disks were RAID5 versus RAID10 on dev. No confirmation of that cause is recorded.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Topics: Backup & Restore, Storage & Space Management, Server Administration, Logging & Checkpoints
Hello
I was trying to restore a level 0 from our dev server on to my test server
both are IDS 11.10, but the dev server had 4 data chunks and i build the test
server with only 3 chunks, rest all the chunks and their path are similar to
that of dev server. ( I made a mistake my not adding the 4th chunk and makinig
every thing identical which should have worked perfectly )
Running "ontape -r" I got the error as
oninit: Cannot open chunk '/informix/db/datadbs2_04'. errno
= 2
Physical restore failed - Cannot Open Primary Chunk '/informix/db/datadbs2_04'.
11:26:16 ipcshm buf_wait semop errno=36
And from fast recovery the database went down.tried to take it online with
"oninit" ( i did not do any onmode -ky )and go the error as
Initializing log/checkpoint information...succeeded
Initializing dbspaces...succeeded
Opening primary chunks...Bad Primary Chunk '/informix/db/datadbs2_04'.
succeeded
Opening mirror chunks...succeeded
Validating chunks...succeeded
Initialize Async Log Flusher...succeeded
Forking btree cleaner...succeeded
Initializing DBSPACETEMP list...succeeded
Checking database partition index...FAILED
oninit: Fatal error in shared memory initialization
in the path /informix/db/ is see a 0 byte file created "datadbs2_04"
I tired
onspaces -s datadbs2 -p /informix/db/datadbs2_04 -o 0 -D -ydid not worked and got shared mem not initialized.
What possibly can i do here.Or should i have to reinitialize the test instance
again?
Not sure but i guess there is some option which allows me to do such kind of
restoration.
Hi,
I think best is to create that fourth chunk (or make sure that it
has correct ownership and file access rights by now) and then
perform the restore again from scratch.
You have to restore the root dbspace and all critical dbspaces
(containing the physical log file and/or the logical log files).
Other dbspaces (containing 'only' user data) need not be
restored at time of the level 0 cold restore. They can be
restored later by doing a warm restore (that however
requires that you have backed up logical logs and can
restore them).
Saying "you have to restore the root dbspace ..." means
that all chunks that belong to root dbspace and critical
dbspaces must be restored. As you are talking about
"chunks" I assume, that this was a chunk that belonged
to the root dbspace - as such it must be restored with
the level 0 cold restore.
Regards,
Martin
--
Martin Fuerderer
IBM Informix Development Munich, Germany
Information Management
IBM Deutschland Research & Development GmbH
Chairman of the Supervisory Board: Martin Jetter
Board of Management: Erich Baier
Corporate Seat: Boeblingen, Germany
Reg.-Gericht: Amtsgericht Stuttgart, HRB 243294
ids-bounces@iiug.org wrote on 07.08.2008 13:38:02:
> Hello
>
> I was trying to restore a level 0 from our dev server on to my test
server
> both are IDS 11.10, but the dev server had 4 data chunks and i buildthe
test
> server with only 3 chunks, rest all the chunks and their path are
similar to
> that of dev server. ( I made a mistake my not adding the 4th chunk
> and makinig
> every thing identical which should have worked perfectly )
>
> Running "ontape -r" I got the error as
> oninit: Cannot open chunk '/informix/db/datadbs2_04'. errno
> = 2
> Physical restore failed - Cannot Open Primary Chunk
> '/informix/db/datadbs2_04'.
> 11:26:16 ipcshm buf_wait semop errno=36
>
> And from fast recovery the database went down.tried to take it online
with
> "oninit" ( i did not do any onmode -ky )and go the error as
>
> Initializing log/checkpoint information...succeeded
> Initializing dbspaces...succeeded
> Opening primary chunks...Bad Primary Chunk '/informix/db/datadbs2_04'.
> succeeded
> Opening mirror chunks...succeeded
> Validating chunks...succeeded
> Initialize Async Log Flusher...succeeded
> Forking btree cleaner...succeeded
> Initializing DBSPACETEMP list...succeeded
> Checking database partition index...FAILED
> oninit: Fatal error in shared memory initialization
>
> in the path /informix/db/ is see a 0 byte file created "datadbs2_04"
>
> I tired
> onspaces -s datadbs2 -p /informix/db/datadbs2_04 -o 0 -D -y> did not worked and got shared mem not initialized.
>
> What possibly can i do here.Or should i have to reinitialize the
> test instance
> again?
>
> Not sure but i guess there is some option which allows me to do such
kind of
> restoration.
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
My dev server has rootdbs,physdbs,logdbs,datadbs2_01,datadbs2_02,datadbs2_03,datadbs2_04,tempdbs_0 1 and tempdbs_02. The newly insalled test server had rootdbs,physdbs,logdbs,datadbs2_01,datadbs2_02,datadbs2_03,tempdbs_01 and tempdbs_02. so databds_04 was missing(not created)on test instance. I had used a Level 0 backup of dev instance for the cold restoration of test instance. How do i create that fourth chunk when my test instance in not coming online. Is there some thing i missing out here?? >Hi, >I think best is to create that fourth chunk (or make sure that it >has correct ownership and file access rights by now) and then >perform the restore again from scratch. >You have to restore the root dbspace and all critical dbspaces >(containing the physical log file and/or the logical log files). >Other dbspaces (containing 'only' user data) need not be >restored at time of the level 0 cold restore. They can be >restored later by doing a warm restore (that however >requires that you have backed up logical logs and can >restore them). >Saying "you have to restore the root dbspace ..." means >that all chunks that belong to root dbspace and critical >dbspaces must be restored. As you are talking about >"chunks" I assume, that this was a chunk that belonged >to the root dbspace - as such it must be restored with >the level 0 cold restore. >Regards, >Martin
Hi,
I think what martin was saying, is that you need to create the missing chunk
as an empty data file with the correct permissions and ownership, just as
you prepared the datadbs2_03 file before you ran the onspaces command.
Then re-run the ontape -r
-----Original Message-----
From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of VICKY
H
Sent: 07 August 2008 02:32 PM
To: ids@iiug.org
Subject: Re: Test Restoration failed [13014]
My dev server has
rootdbs,physdbs,logdbs,datadbs2_01,datadbs2_02,datadbs2_03,datadbs2_04,tempd
bs_01
and tempdbs_02.
The newly insalled test server had
rootdbs,physdbs,logdbs,datadbs2_01,datadbs2_02,datadbs2_03,tempdbs_01 and
tempdbs_02.
so databds_04 was missing(not created)on test instance.
I had used a Level 0 backup of dev instance for the cold restoration of test
instance.
How do i create that fourth chunk when my test instance in not coming
online.
Is there some thing i missing out here??
>Hi,
>I think best is to create that fourth chunk (or make sure that it has
>correct ownership and file access rights by now) and then perform the
>restore again from scratch.
>You have to restore the root dbspace and all critical dbspaces
>(containing the physical log file and/or the logical log files).
>Other dbspaces (containing 'only' user data) need not be restored at
>time of the level 0 cold restore. They can be restored later by doing a
>warm restore (that however requires that you have backed up logical
>logs and can restore them).
>Saying "you have to restore the root dbspace ..." means that all chunks
>that belong to root dbspace and critical dbspaces must be restored. As
>you are talking about "chunks" I assume, that this was a chunk that
>belonged to the root dbspace - as such it must be restored with the
>level 0 cold restore.
>Regards,
>Martin
****************************************************************************
***
Forum Note: Use "Reply" to post a response in the discussion forum.
Hi,
yes, Mark, that's what I was trying to say.
However, IDS can start and be on-line while a user
data dbspace is missing or corrupt. There is an
onconfig file parameter named ONDBSPACEDOWN.
IDS will continue when this is set to 0 (zero), it will
abort when this is set to 1 and it will wait (hang) when
this is set to 2.
So maybe on the system in question this is set to 1
and that's why IDS aborts?
TIA,
Martin
--
Martin Fuerderer
IBM Informix Development Munich, Germany
Information Management
IBM Deutschland Research & Development GmbH
Chairman of the Supervisory Board: Martin Jetter
Board of Management: Erich Baier
Corporate Seat: Boeblingen, Germany
Reg.-Gericht: Amtsgericht Stuttgart, HRB 243294
"Mark Tyrer" <mark.tyrer@rtt.co.za>
Sent by: ids-bounces@iiug.org
07.08.2008 14:42
Please respond to
ids@iiug.org
To
ids@iiug.org
cc
Subject
RE: Test Restoration failed [13015]
Hi,
I think what martin was saying, is that you need to create the missing
chunk
as an empty data file with the correct permissions and ownership, just as
you prepared the datadbs2_03 file before you ran the onspaces command.
Then re-run the ontape -r
-----Original Message-----
From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of
VICKY
H
Sent: 07 August 2008 02:32 PM
To: ids@iiug.org
Subject: Re: Test Restoration failed [13014]
My dev server has
rootdbs,physdbs,logdbs,datadbs2_01,datadbs2_02,datadbs2_03,datadbs2_04,tempd
bs_01
and tempdbs_02.
The newly insalled test server had
rootdbs,physdbs,logdbs,datadbs2_01,datadbs2_02,datadbs2_03,tempdbs_01 and
tempdbs_02.
so databds_04 was missing(not created)on test instance.
I had used a Level 0 backup of dev instance for the cold restoration of
test
instance.
How do i create that fourth chunk when my test instance in not coming
online.
Is there some thing i missing out here??
>Hi,
>I think best is to create that fourth chunk (or make sure that it has
>correct ownership and file access rights by now) and then perform the
>restore again from scratch.
>You have to restore the root dbspace and all critical dbspaces
>(containing the physical log file and/or the logical log files).
>Other dbspaces (containing 'only' user data) need not be restored at
>time of the level 0 cold restore. They can be restored later by doing a
>warm restore (that however requires that you have backed up logical
>logs and can restore them).
>Saying "you have to restore the root dbspace ..." means that all chunks
>that belong to root dbspace and critical dbspaces must be restored. As
>you are talking about "chunks" I assume, that this was a chunk that
>belonged to the root dbspace - as such it must be restored with the
>level 0 cold restore.
>Regards,
>Martin
****************************************************************************
***
Forum Note: Use "Reply" to post a response in the discussion forum.
*******************************************************************************
Forum Note: Use "Reply" to post a response in the discussion forum.
I have checked and
ONDBSPACEDOWN 2 # Dbspace down option: 0 = CONTINUE, 1 = ABORT, 2 = WAIT
so i dont know why am i getting the error
Opening primary chunks...Bad Primary Chunk '/informix/db/datadbs2_04'.
succeeded
Opening mirror chunks...succeeded
Validating chunks...succeeded
Initialize Async Log Flusher...succeeded
Forking btree cleaner...succeeded
Initializing DBSPACETEMP list...succeeded
Checking database partition index...FAILED
oninit: Fatal error in shared memory initialization
Thankx
vicky
>Hi,
>yes, Mark, that's what I was trying to say.
>However, IDS can start and be on-line while a user
>data dbspace is missing or corrupt. There is an
>onconfig file parameter named ONDBSPACEDOWN.
>IDS will continue when this is set to 0 (zero), it will
>abort when this is set to 1 and it will wait (hang) when
>this is set to 2.
>So maybe on the system in question this is set to 1
>and that's why IDS aborts?
>TIA,
>Martin
Finally i reinitialized the test instance and started the restore using ontape
-r.
But the restore which was suppose to finish in 7-8 hrs on this server (as
tested many times before)is not running over 15 hrs and i can see in onstat -D
that one complete chunk is yet to be restored.
onstat -D and onstat -u shows me that the restoration is in progress but dontknow the exact reason for this slow progress.
My server is solaris 10 with 4cpu and 16 Gb RAM and the backups file is 50
GB(Zipped) using backup filter.
I have restored this large backup on the dev server which is similiar to test
server many times and it completed in 7-8 hrs.
Pls let me know what should i check in order to find what is going wrong.
Should i veryfiy the backup file. Dont know if Archecker works for ontape
backup verification.
Can there be some issue with the backup restore filter used.
Pls help !
Many thanks
vicky
Bet you three chocolate dipped donuts that your 'test' system has RAID5
disks under the chunks you created while the 'dev' system is using RAID10.
Art
On Fri, Aug 8, 2008 at 4:38 AM, VICKY H <vickykh26@gmail.com> wrote:
> Finally i reinitialized the test instance and started the restore using
> ontape
> -r.>
> But the restore which was suppose to finish in 7-8 hrs on this server (as
> tested many times before)is not running over 15 hrs and i can see in onstat
> -D
> that one complete chunk is yet to be restored.
> onstat -D and onstat -u shows me that the restoration is in progress but> dont
> know the exact reason for this slow progress.
>
> My server is solaris 10 with 4cpu and 16 Gb RAM and the backups file is 50
> GB(Zipped) using backup filter.
>
> I have restored this large backup on the dev server which is similiar to
> test
> server many times and it completed in 7-8 hrs.
>
> Pls let me know what should i check in order to find what is going wrong.
> Should i veryfiy the backup file. Dont know if Archecker works for ontape
> backup verification.
> Can there be some issue with the backup restore filter used.
>
> Pls help !
>
> Many thanks
> vicky
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--
Art S. Kagel
Oninit (www.oninit.com)
IIUG Board of Directors (art@iiug.org)
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Oninit, the IIUG, nor any other organization
with which I am associated either explicitly or implicitly. Neither do those
opinions reflect those of other individuals affiliated with any entity with
which I am affiliated nor those of the entities themselves.
Art Kagel wrote: > Bet you three chocolate dipped donuts that your 'test' system has RAID5 > disks under the chunks you created while the 'dev' system is using RAID10. > Mmmm ... Dunkin Donuts .... mmmmmm.... arghlghgh -- Cheers, Obnoxio the Clown http://obotheclown.blogspot.com
One could envision an alternative career in law enforcement for Art after Oninit ... -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Obnoxio The Clown Sent: Friday, August 08, 2008 10:33 AM To: ids@iiug.org Subject: Re: Test Restoration failed [13036] Art Kagel wrote: > Bet you three chocolate dipped donuts that your 'test' system has RAID5 > disks under the chunks you created while the 'dev' system is using RAID10. > Mmmm ... Dunkin Donuts .... mmmmmm.... arghlghgh -- Cheers, Obnoxio the Clown http://obotheclown.blogspot.com ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum.
Yeah, wait until he starts whacking you with that BAARF stick! (www.baarf.com) -- Bob -------------- Original message -------------- From: "Plugge, Joe R." <JRPlugge@west.com> > One could envision an alternative career in law enforcement for Art > after Oninit ... > > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > Obnoxio The Clown > Sent: Friday, August 08, 2008 10:33 AM > To: ids@iiug.org > Subject: Re: Test Restoration failed [13036] > > Art Kagel wrote: > > Bet you three chocolate dipped donuts that your 'test' system has > RAID5 > > disks under the chunks you created while the 'dev' system is using > RAID10. > > > > Mmmm ... Dunkin Donuts .... mmmmmm.... arghlghgh > > -- > Cheers, > Obnoxio the Clown > > http://obotheclown.blogspot.com > > ************************************************************************ > ******* > Forum Note: Use "Reply" to post a response in the discussion forum. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >
Related threads
- IDS 10 table-level restore
- Informix Development Webinar December 11, 2007
- ontape -p/r with changed ROOTPATH
- Migrate from HP PA-RISC to HP ITANIUM by ontape