HELP! rootdbs corrupted!
A 4TB IDS 9.3 instance on HP-UX died after the old disk array's VDISK was deleted: during the earlier storage migration the rootdbs had never been relinked to the new array, so oninit now panics during logical recovery (assert in rslog.c, 'Dynamic Server must abort'). The poster had no level-0 archive or logical log backups. Suggestions were a cold restore (impossible without backups), restoring the old array from a system backup, repointing the rootdbs link to a valid copy and immediately taking a level 0, or engaging Oninit to salvage data from the surviving chunks. The poster noted only pre-migration rootdbs exists while chunks added later are missing; no resolution is recorded in the thread.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Hello everybody!
Please help me or my boss will kill me :(
What do we have:
HP-UX 11.11 (PA-RISC)
Informix IDS 9.3 FC6 with ~4TB database
And date of acceptance - 31/Mar/2008!!!
History:
Once customer decided to add some functionality to the system and
increase disk space. They have installed additional and more powerful
disk array and asked us to migrate the system to this array.
After successful migration everybody were happy with increased
performance and space. But the data was left as backup.
Today the nightmare became.
The old disk array became unstable and it was required to free some
space on it. When our disk array engineer asked us can he delete the
VDISK with old database from the old device we said "sure you can" and
he did it.
After that the server stopped responding. BTW online config.log does
not contain any strange messages.
When we try to oninit the serber it says (in online.log):
20:13:56 IBM Informix Dynamic Server Started.
20:14:11 Segment locked: addr=0xc00000000051e000, size=4294705152
20:14:11 Requested shared memory segment size rounded from 2285053KB
to 2285056KB
20:14:19 Segment locked: addr=0xc0000001004de000, size=2339897344
Wed Mar 26 20:14:23 2008
20:14:23 Event alarms enabled. ALARMPROG = '/apps/inst1/informix/ids.
93/etc/log_full.sh'
20:14:23 Booting Language <c> from module <>
20:14:23 Loading Module <CNULL>
20:14:23 Booting Language <builtin> from module <>
20:14:23 Loading Module <BUILTINNULL>
20:14:35 IBM Informix Dynamic Server Version 9.30.FC6X9 Software
Serial Number XXX#XXXXXXXXX
20:14:36 IBM Informix Dynamic Server Initialized -- Shared Memory
Initialized.
20:14:36 Physical Recovery Started at Page(3:144595).
20:14:38 Physical Recovery Complete: 29340 Pages Examined 29340 Pages
Restored.
20:14:38 Logical Recovery Started.
20:14:38 10 recovery worker threads will be started.
20:14:38 Assert Failed: Dynamic Server must abort
20:14:38 IBM Informix Dynamic Server Version 9.30.FC6X9
20:14:38 Who: Session(7, informix@mpmnw, 0, 335488200)
Thread(177, fast_rec, c000000113fb1088, 1)
File: rslog.c Line: 3383
20:14:38 Results: Dynamic Server must abort
20:14:38 Action: Reinitialize shared memory
20:14:38 stack trace for pid 4817 written to /apps/informix_dump/af.
499847e
20:14:38 See Also: /apps/informix_dump/af.499847e, shmem.499847e.0
20:14:49 Error writing '/apps/informix_dump/shmem.499847e.0' errno =
27
20:14:49 rslog.c, line 3383, thread 177, proc id 4817, Dynamic Server
must abort.
20:14:49 PANIC: Attempting to bring system down
After investigations it was identified that during the migration we
have made a serious mistake - the rootdbs was not relinked to the new
storage.
So currently we have 4TBs of data with old rootdbs and its mirror
(which were actual at the moment of migration).
And the question is: how to restore the rootdbs with all (or most part
of) data.
I believe it is possible. May be with manual writing lots of commands
or filling some tables in sysmaster ot something else.
Kind regards,
Pavel
Seems to me a cold restore is your only option after you relink to the
new storage. Got a level0 backups? Logical log backups?
-----Original Message-----
From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of
PAVEL LECHENKO
Sent: Wednesday, March 26, 2008 3:34 PM
To: ids@iiug.org
Subject: HELP! rootdbs corrupted! [11711]
Hello everybody!
Please help me or my boss will kill me :(
What do we have:
HP-UX 11.11 (PA-RISC)
Informix IDS 9.3 FC6 with ~4TB database
And date of acceptance - 31/Mar/2008!!!
History:
Once customer decided to add some functionality to the system and
increase disk space. They have installed additional and more powerful
disk array and asked us to migrate the system to this array.
After successful migration everybody were happy with increased
performance and space. But the data was left as backup.
Today the nightmare became.
The old disk array became unstable and it was required to free some
space on it. When our disk array engineer asked us can he delete the
VDISK with old database from the old device we said "sure you can" and
he did it.
After that the server stopped responding. BTW online config.log does
not contain any strange messages.
When we try to oninit the serber it says (in online.log):
20:13:56 IBM Informix Dynamic Server Started.
20:14:11 Segment locked: addr=0xc00000000051e000, size=4294705152
20:14:11 Requested shared memory segment size rounded from 2285053KB
to 2285056KB
20:14:19 Segment locked: addr=0xc0000001004de000, size=2339897344
Wed Mar 26 20:14:23 2008
20:14:23 Event alarms enabled. ALARMPROG = '/apps/inst1/informix/ids.
93/etc/log_full.sh'
20:14:23 Booting Language <c> from module <>
20:14:23 Loading Module <CNULL>
20:14:23 Booting Language <builtin> from module <>
20:14:23 Loading Module <BUILTINNULL>
20:14:35 IBM Informix Dynamic Server Version 9.30.FC6X9 Software
Serial Number XXX#XXXXXXXXX
20:14:36 IBM Informix Dynamic Server Initialized -- Shared Memory
Initialized.
20:14:36 Physical Recovery Started at Page(3:144595).
20:14:38 Physical Recovery Complete: 29340 Pages Examined 29340 Pages
Restored.
20:14:38 Logical Recovery Started.
20:14:38 10 recovery worker threads will be started.
20:14:38 Assert Failed: Dynamic Server must abort
20:14:38 IBM Informix Dynamic Server Version 9.30.FC6X9
20:14:38 Who: Session(7, informix@mpmnw, 0, 335488200)
Thread(177, fast_rec, c000000113fb1088, 1)
File: rslog.c Line: 3383
20:14:38 Results: Dynamic Server must abort
20:14:38 Action: Reinitialize shared memory
20:14:38 stack trace for pid 4817 written to /apps/informix_dump/af.
499847e
20:14:38 See Also: /apps/informix_dump/af.499847e, shmem.499847e.0
20:14:49 Error writing '/apps/informix_dump/shmem.499847e.0' errno =
27
20:14:49 rslog.c, line 3383, thread 177, proc id 4817, Dynamic Server
must abort.
20:14:49 PANIC: Attempting to bring system down
After investigations it was identified that during the migration we
have made a serious mistake - the rootdbs was not relinked to the new
storage.
So currently we have 4TBs of data with old rootdbs and its mirror
(which were actual at the moment of migration).
And the question is: how to restore the rootdbs with all (or most part
of) data.
I believe it is possible. May be with manual writing lots of commands
or filling some tables in sysmaster ot something else.
Kind regards,
Pavel
************************************************************************
*******
Forum Note: Use "Reply" to post a response in the discussion forum.
See you at the IIUG Informix 2008 Conference
The Power Conference for Informix Professionals
April 27 - 30, 2008 Marriott Overland Park (Kansas City), Kansas
http://www.iiug.org/conf
Registration Now Open!!
Hi Rogers, Unfortunately we do not have any backups - it was planned to the next phase. Pavel
If you don't have an Informix backup, is there a backup of your old array that you could restore? --EEM > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > PAVEL LECHENKO > Sent: Wednesday, March 26, 2008 4:08 PM > To: ids@iiug.org > Subject: Re: RE: HELP! rootdbs corrupted! [11713] > > Hi Rogers, > Unfortunately we do not have any backups - it was planned to the next > phase. > > Pavel > > > ************************************************************************ ** > ***** > Forum Note: Use "Reply" to post a response in the discussion forum. > > See you at the IIUG Informix 2008 Conference > The Power Conference for Informix Professionals > April 27 - 30, 2008 Marriott Overland Park (Kansas City), Kansas > http://www.iiug.org/conf > Registration Now Open!!
PAVEL LECHENKO wrote:
Assuming you actually copied the rootdb space to the new disk farm and
haven't made any changes to the system since, you should be able to just
point ROOTDBS link to the new copy of the rootdb space, as you should
have originally, and try to restart the instance. Unless there are
actual data tables in the rootdb space you should be OK. I would then
immediately take a level 0 archive and not miss any more archives.
Worse comes to worst, contact us a Oninit and we'll try to recover as
much of your data from the remaining chunks as possible.
Art S. Kagel
Oninit
www.oninit.com
> Hello everybody!
>
> Please help me or my boss will kill me :(
>
> What do we have:
> HP-UX 11.11 (PA-RISC)
> Informix IDS 9.3 FC6 with ~4TB database
> And date of acceptance - 31/Mar/2008!!!
>
> History:
> Once customer decided to add some functionality to the system and
> increase disk space. They have installed additional and more powerful
> disk array and asked us to migrate the system to this array.
> After successful migration everybody were happy with increased
> performance and space. But the data was left as backup.
>
> Today the nightmare became.
>
> The old disk array became unstable and it was required to free some
> space on it. When our disk array engineer asked us can he delete the
> VDISK with old database from the old device we said "sure you can" and
> he did it.
>
> After that the server stopped responding. BTW online config.log does
> not contain any strange messages.
>
> When we try to oninit the serber it says (in online.log):
> 20:13:56 IBM Informix Dynamic Server Started.
> 20:14:11 Segment locked: addr=0xc00000000051e000, size=4294705152
> 20:14:11 Requested shared memory segment size rounded from 2285053KB
> to 2285056KB
> 20:14:19 Segment locked: addr=0xc0000001004de000, size=2339897344
>
> Wed Mar 26 20:14:23 2008
>
> 20:14:23 Event alarms enabled. ALARMPROG = '/apps/inst1/informix/ids.
> 93/etc/log_full.sh'
> 20:14:23 Booting Language <c> from module <>
> 20:14:23 Loading Module <CNULL>
> 20:14:23 Booting Language <builtin> from module <>
> 20:14:23 Loading Module <BUILTINNULL>
> 20:14:35 IBM Informix Dynamic Server Version 9.30.FC6X9 Software
> Serial Number XXX#XXXXXXXXX
> 20:14:36 IBM Informix Dynamic Server Initialized -- Shared Memory
> Initialized.
>
> 20:14:36 Physical Recovery Started at Page(3:144595).
> 20:14:38 Physical Recovery Complete: 29340 Pages Examined 29340 Pages
> Restored.
>
> 20:14:38 Logical Recovery Started.
> 20:14:38 10 recovery worker threads will be started.
> 20:14:38 Assert Failed: Dynamic Server must abort
> 20:14:38 IBM Informix Dynamic Server Version 9.30.FC6X9
> 20:14:38 Who: Session(7, informix@mpmnw, 0, 335488200)
> Thread(177, fast_rec, c000000113fb1088, 1)
> File: rslog.c Line: 3383
> 20:14:38 Results: Dynamic Server must abort
> 20:14:38 Action: Reinitialize shared memory
> 20:14:38 stack trace for pid 4817 written to /apps/informix_dump/af.
> 499847e
> 20:14:38 See Also: /apps/informix_dump/af.499847e, shmem.499847e.0
> 20:14:49 Error writing '/apps/informix_dump/shmem.499847e.0' errno =
> 27
> 20:14:49 rslog.c, line 3383, thread 177, proc id 4817, Dynamic Server
> must abort.
> 20:14:49 PANIC: Attempting to bring system down
>
> After investigations it was identified that during the migration we
> have made a serious mistake - the rootdbs was not relinked to the new
> storage.
>
> So currently we have 4TBs of data with old rootdbs and its mirror
> (which were actual at the moment of migration).
>
> And the question is: how to restore the rootdbs with all (or most part
> of) data.
>
> I believe it is possible. May be with manual writing lots of commands
> or filling some tables in sysmaster ot something else.
>
> Kind regards,
>
> Pavel
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
> See you at the IIUG Informix 2008 Conference
> The Power Conference for Informix Professionals
> April 27 - 30, 2008 Marriott Overland Park (Kansas City), Kansas
> http://www.iiug.org/conf
> Registration Now Open!!
>
>
>
I would also suggest Oninit's assistance I this case. They where instrumental
in helping us with our down primary chunks issue.
Thanks,
**************************************
Ernie Knox
Sears Holding Co.
IT Database Administrator Specialist
IT Service Management, Strategy & Architecture
3333 Beverly Rd., B4-266A
Hoffman Estates, IL. 60179
Office: (847) 286-5735
Fax: (847) 645-3874
Pager: (800) 759-8352 Pin#: 7271042
Email: eknox@sears.com
" It's always a great day to watch Football ! "
**************************************
-----Original Message-----
From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Art S.
Kagel (Oninit)
Sent: Wednesday, March 26, 2008 6:28 PM
To: ids@iiug.org
Subject: Re: HELP! rootdbs corrupted! [11717]
PAVEL LECHENKO wrote:
Assuming you actually copied the rootdb space to the new disk farm and
haven't made any changes to the system since, you should be able to just
point ROOTDBS link to the new copy of the rootdb space, as you should
have originally, and try to restart the instance. Unless there are
actual data tables in the rootdb space you should be OK. I would then
immediately take a level 0 archive and not miss any more archives.
Worse comes to worst, contact us a Oninit and we'll try to recover as
much of your data from the remaining chunks as possible.
Art S. Kagel
Oninit
www.oninit.com
> Hello everybody!
>
> Please help me or my boss will kill me :(
>
> What do we have:
> HP-UX 11.11 (PA-RISC)
> Informix IDS 9.3 FC6 with ~4TB database
> And date of acceptance - 31/Mar/2008!!!
>
> History:
> Once customer decided to add some functionality to the system and
> increase disk space. They have installed additional and more powerful
> disk array and asked us to migrate the system to this array.
> After successful migration everybody were happy with increased
> performance and space. But the data was left as backup.
>
> Today the nightmare became.
>
> The old disk array became unstable and it was required to free some
> space on it. When our disk array engineer asked us can he delete the
> VDISK with old database from the old device we said "sure you can" and
> he did it.
>
> After that the server stopped responding. BTW online config.log does
> not contain any strange messages.
>
> When we try to oninit the serber it says (in online.log):
> 20:13:56 IBM Informix Dynamic Server Started.
> 20:14:11 Segment locked: addr=0xc00000000051e000, size=4294705152
> 20:14:11 Requested shared memory segment size rounded from 2285053KB
> to 2285056KB
> 20:14:19 Segment locked: addr=0xc0000001004de000, size=2339897344
>
> Wed Mar 26 20:14:23 2008
>
> 20:14:23 Event alarms enabled. ALARMPROG = '/apps/inst1/informix/ids.
> 93/etc/log_full.sh'
> 20:14:23 Booting Language <c> from module <>
> 20:14:23 Loading Module <CNULL>
> 20:14:23 Booting Language <builtin> from module <>
> 20:14:23 Loading Module <BUILTINNULL>
> 20:14:35 IBM Informix Dynamic Server Version 9.30.FC6X9 Software
> Serial Number XXX#XXXXXXXXX
> 20:14:36 IBM Informix Dynamic Server Initialized -- Shared Memory
> Initialized.
>
> 20:14:36 Physical Recovery Started at Page(3:144595).
> 20:14:38 Physical Recovery Complete: 29340 Pages Examined 29340 Pages
> Restored.
>
> 20:14:38 Logical Recovery Started.
> 20:14:38 10 recovery worker threads will be started.
> 20:14:38 Assert Failed: Dynamic Server must abort
> 20:14:38 IBM Informix Dynamic Server Version 9.30.FC6X9
> 20:14:38 Who: Session(7, informix@mpmnw, 0, 335488200)
> Thread(177, fast_rec, c000000113fb1088, 1)
> File: rslog.c Line: 3383
> 20:14:38 Results: Dynamic Server must abort
> 20:14:38 Action: Reinitialize shared memory
> 20:14:38 stack trace for pid 4817 written to /apps/informix_dump/af.
> 499847e
> 20:14:38 See Also: /apps/informix_dump/af.499847e, shmem.499847e.0
> 20:14:49 Error writing '/apps/informix_dump/shmem.499847e.0' errno =
> 27
> 20:14:49 rslog.c, line 3383, thread 177, proc id 4817, Dynamic Server
> must abort.
> 20:14:49 PANIC: Attempting to bring system down
>
> After investigations it was identified that during the migration we
> have made a serious mistake - the rootdbs was not relinked to the new
> storage.
>
> So currently we have 4TBs of data with old rootdbs and its mirror
> (which were actual at the moment of migration).
>
> And the question is: how to restore the rootdbs with all (or most part
> of) data.
>
> I believe it is possible. May be with manual writing lots of commands
> or filling some tables in sysmaster ot something else.
>
> Kind regards,
>
> Pavel
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
> See you at the IIUG Informix 2008 Conference
> The Power Conference for Informix Professionals
> April 27 - 30, 2008 Marriott Overland Park (Kansas City), Kansas
> http://www.iiug.org/conf
> Registration Now Open!!
>
>
>
*******************************************************************************
Forum Note: Use "Reply" to post a response in the discussion forum.
See you at the IIUG Informix 2008 Conference
The Power Conference for Informix Professionals
April 27 - 30, 2008 Marriott Overland Park (Kansas City), Kansas
http://www.iiug.org/conf
Registration Now Open!!
Everett, The only thing I have is an old rootdbs (before migration). After the migration some chunks were added - that's why it crashes. Pavel
Some additional info that may be helpful. Rootdbs contains only root data. Logs, temp and user data are in dedicated chunks which are up-to-date. Pavel