PANIC
A 7.31.UD1 server on Linux panics during fast recovery: logical rollforward hits "Error during recovery left index inconsistent" (rskey.c line 1374), marks two indexes unusable, fails with iserrno 105 and brings the system down. Suggestions: oncheck -cDI then drop/rebuild the indexes manually with PDQPRIORITY/PSORT settings (faster than oncheck), but since the engine never comes online, others advised opening an IBM/Informix support case so support can disable the offending indexes or skip the failing log records (risking data inconsistency); otherwise restore from backup. Also noted: SHMVIRTSIZE too small, and upgrading the fix pack cured a similar case. The original poster never reported an outcome, so no confirmed resolution is recorded.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Is there another solution for this problem
before trying to restore from
backup/archive?.
LINUX/IFX 7.31.UD1.
10:11:22 Informix Dynamic Server Initialized -- Shared Memory Initialized.
10:11:22 Physical Recovery Started.
10:11:22 Physical Recovery Complete: 64 Pages Restored.
10:11:22 Logical Recovery Started.
10:11:22 Dynamically allocated new virtual shared memory segment (size
8192KB)
10:11:22 Assert Failed: Error during recovery left index inconsistent.
10:11:22 Informix Dynamic Server Version 7.31.UD1
10:11:22 Who: Session(7, informix@bd_pfbsa00, 0, 367676640)
Thread(30, xchg_1.5, 15e800e0, 4)
File: rskey.c Line: 1374
10:11:22 Results: Index 'guardiao:"guardiao".cnfg_log' is now unusable
10:11:22 Action: Run 'oncheck -cI guardiao:"guardiao".cnfg_log'
10:11:22 Assert Failed: Error during recovery left index inconsistent.
10:11:22 Informix Dynamic Server Version 7.31.UD1
10:11:22 Who: Session(7, informix@bd_pfbsa00, 0, 367684648)
Thread(27, xchg_1.2, 15e7f1ec, 3)
File: rskey.c Line: 1374
10:11:22 Results: Index 'guardiao:"guardiao".chmd_chamada_log' is now
unusable10:11:22 Action: Run 'oncheck -cI
guardiao:"guardiao".chmd_chamada_log'
10:11:23 See Also: /tmp/af.40389fa
10:11:23 Rollforward of log record failed. iserrno = 105
10:11:23 Log Record: log = 6227, pos = 8d1088, type =
OLDRSAM:ADDITEM(28), trans = 14
10:11:23 Rollforward of log record failed. iserrno = 105
10:11:23 Log Record: log = 6227, pos = 8d1088, type =
OLDRSAM:ADDITEM(28), trans = 14
...
10:11:34 PANIC: Attempting to bring system down
Jairo
You
can run the recommended oncheck -cI (actually I'd make that -cDI) but if the
indexes are indeed corrupted do NOT let oncheck rebuild them, it's too slow.
Instead manually drop the indexes and rebuild them using the parallel sort
package (set PDQPRIORITY > 1, PSORT_NPROCS = 12 <more if you have lots of CPU
VPs configured> and PSORT_DBTEMP = a list of 3 to 6 FILESYSTEMS not dbspaces
each of which has enough freespace to hold the keys for sorting the largest
index).
Art S. Kagel
----- Original Message -----
From: Jairo Gubler <jairo.gubler@digitro.com.br>
At: 9/ 9 10:39
Is there another solution for this problem before trying to restore from
backup/archive?.
LINUX/IFX 7.31.UD1.
10:11:22 Informix Dynamic Server Initialized -- Shared Memory Initialized.
10:11:22 Physical Recovery Started.
10:11:22 Physical Recovery Complete: 64 Pages Restored.
10:11:22 Logical Recovery Started.
10:11:22 Dynamically allocated new virtual shared memory segment (size
8192KB)
10:11:22 Assert Failed: Error during recovery left index inconsistent.
10:11:22 Informix Dynamic Server Version 7.31.UD1
10:11:22 Who: Session(7, informix@bd_pfbsa00, 0, 367676640)
Thread(30, xchg_1.5, 15e800e0, 4)
File: rskey.c Line: 1374
10:11:22 Results: Index 'guardiao:"guardiao".cnfg_log' is now unusable
10:11:22 Action: Run 'oncheck -cI guardiao:"guardiao".cnfg_log'
10:11:22 Assert Failed: Error during recovery left index inconsistent.
10:11:22 Informix Dynamic Server Version 7.31.UD1
10:11:22 Who: Session(7, informix@bd_pfbsa00, 0, 367684648)
Thread(27, xchg_1.2, 15e7f1ec, 3)
File: rskey.c Line: 1374
10:11:22 Results: Index 'guardiao:"guardiao".chmd_chamada_log' is now
unusable10:11:22 Action: Run 'oncheck -cI
guardiao:"guardiao".chmd_chamada_log'
10:11:23 See Also: /tmp/af.40389fa
10:11:23 Rollforward of log record failed. iserrno = 105
10:11:23 Log Record: log = 6227, pos = 8d1088, type =
OLDRSAM:ADDITEM(28), trans = 14
10:11:23 Rollforward of log record failed. iserrno = 105
10:11:23 Log Record: log = 6227, pos = 8d1088, type =
OLDRSAM:ADDITEM(28), trans = 14
.
10:11:34 PANIC: Attempting to bring system down
Jairo
I don't think this is going to work because the
engine
went down ( scroll down in the log file that was sent
).
Jairo I would suggest you to open a case with IBM tech
support as a down system, which it is.
They have different ways to fix this issue.
HTH,
esteban.-
--- "ART KAGEL, ...." <KAGEL@bloomberg.net> escribis:
> You can run the recommended oncheck -cI (actually
> I'd make that -cDI) but if the
> indexes are indeed corrupted do NOT let oncheck
> rebuild them, it's too slow.
> Instead manually drop the indexes and rebuild them
> using the parallel sort
> package (set PDQPRIORITY > 1, PSORT_NPROCS = 12
> <more if you have lots of CPU
> VPs configured> and PSORT_DBTEMP = a list of 3 to 6
> FILESYSTEMS not dbspaces
> each of which has enough freespace to hold the keys
> for sorting the largest
> index).
>
> Art S. Kagel
>
> ----- Original Message -----
> From: Jairo Gubler <jairo.gubler@digitro.com.br>
> At: 9/ 9 10:39
>
> Is there another solution for this problem before
> trying to restore from
> backup/archive?.
> LINUX/IFX 7.31.UD1.
>
> 10:11:22 Informix Dynamic Server Initialized --
> Shared Memory Initialized.
> 10:11:22 Physical Recovery Started.
> 10:11:22 Physical Recovery Complete: 64 Pages
> Restored.
> 10:11:22 Logical Recovery Started.
> 10:11:22 Dynamically allocated new virtual shared
> memory segment (size
> 8192KB)
>
> 10:11:22 Assert Failed: Error during recovery left
> index inconsistent.
> 10:11:22 Informix Dynamic Server Version 7.31.UD1
> 10:11:22 Who: Session(7, informix@bd_pfbsa00, 0,
> 367676640)
> Thread(30, xchg_1.5, 15e800e0, 4)
> File: rskey.c Line: 1374
> 10:11:22 Results: Index
> 'guardiao:"guardiao".cnfg_log' is now unusable
> 10:11:22 Action: Run 'oncheck -cI
> guardiao:"guardiao".cnfg_log'
> 10:11:22 Assert Failed: Error during recovery left
> index inconsistent.
> 10:11:22 Informix Dynamic Server Version 7.31.UD1
> 10:11:22 Who: Session(7, informix@bd_pfbsa00, 0,
> 367684648)
> Thread(27, xchg_1.2, 15e7f1ec, 3)
> File: rskey.c Line: 1374
> 10:11:22 Results: Index
> 'guardiao:"guardiao".chmd_chamada_log' is now
> unusable10:11:22 Action: Run 'oncheck -cI
> guardiao:"guardiao".chmd_chamada_log'
> 10:11:23 See Also: /tmp/af.40389fa
> 10:11:23 Rollforward of log record failed. iserrno
> = 105
> 10:11:23 Log Record: log = 6227, pos = 8d1088, type
> =
> OLDRSAM:ADDITEM(28), trans = 14
> 10:11:23 Rollforward of log record failed. iserrno
> = 105
> 10:11:23 Log Record: log = 6227, pos = 8d1088, type
> =
> OLDRSAM:ADDITEM(28), trans = 14
>
___________________________________________________________
1GB gratis, Antivirus y Antispam
Correo Yahoo!, el mejor correo web del mundo
http://correo.yahoo.com.ar
Jairo Gubler wrote:
> Is there another solution for this problem before trying to restore from
> backup/archive?.
> LINUX/IFX 7.31.UD1.
>
> 10:11:22 Informix Dynamic Server Initialized -- Shared Memory Initialized.
> 10:11:22 Physical Recovery Started.
> 10:11:22 Physical Recovery Complete: 64 Pages Restored.
> 10:11:22 Logical Recovery Started.
> 10:11:22 Dynamically allocated new virtual shared memory segment (size
> 8192KB)
>
> 10:11:22 Assert Failed: Error during recovery left index inconsistent.
> 10:11:22 Informix Dynamic Server Version 7.31.UD1
> 10:11:22 Who: Session(7, informix@bd_pfbsa00, 0, 367676640)
> Thread(30, xchg_1.5, 15e800e0, 4)
> File: rskey.c Line: 1374
> 10:11:22 Results: Index 'guardiao:"guardiao".cnfg_log' is now unusable
> 10:11:22 Action: Run 'oncheck -cI guardiao:"guardiao".cnfg_log'
> 10:11:22 Assert Failed: Error during recovery left index inconsistent.
> 10:11:22 Informix Dynamic Server Version 7.31.UD1
> 10:11:22 Who: Session(7, informix@bd_pfbsa00, 0, 367684648)
> Thread(27, xchg_1.2, 15e7f1ec, 3)
> File: rskey.c Line: 1374
> 10:11:22 Results: Index 'guardiao:"guardiao".chmd_chamada_log' is now
> unusable10:11:22 Action: Run 'oncheck -cI
> guardiao:"guardiao".chmd_chamada_log'
> 10:11:23 See Also: /tmp/af.40389fa
> 10:11:23 Rollforward of log record failed. iserrno = 105
> 10:11:23 Log Record: log = 6227, pos = 8d1088, type =
> OLDRSAM:ADDITEM(28), trans = 14
> 10:11:23 Rollforward of log record failed. iserrno = 105
> 10:11:23 Log Record: log = 6227, pos = 8d1088, type =
> OLDRSAM:ADDITEM(28), trans = 14
> ..
> 10:11:34 PANIC: Attempting to bring system down
>
> Jairo
>
>
If you have a support contract, advanced support can bring the engine online
either disabling the indexes in question, or skipping the failing logical log
records. After that, you'll have to fix the indexes and fix the logical
inconsistencies in your data that *might* have been generated by skipping
logical log records.
If you don't have support, a restore is your only option.
--
Ciao,
Marco
______________________________________________________________________________
Marco Greco /UK /IBM Standard disclaimers apply!
Informix faq http://www.iiug.org/techinfo/faq/informix.htm
4glworks http://www.4glworks.com
Informix on Linux http://www.4glworks.com/ifmxlinux.htm
Jairo,
From the online.log it appears that IDS does not come online , so running
the suggested onchecks will not be possible.
You might want to ask Informix Support to disable the log entries for the
indexes
in question (which cause the trouble during fast recovery) before
restarting IDS.
But it is not guaranteed that this will result in success (interfering
with the
fast recovery process i always a tricky issue)
A restore might be quicker (in case you have a fairly recent backup) .
Apart from that , two points:
online.log says
> 10:11:22 Dynamically allocated new virtual shared memory segment (size
> 8192KB)
right at startup.
This indicates that you SHMVIRTSIZE parameter (ONCONFIG file) is too small.
Please set SHMVIRTSIZE to a value that is about the amount of virtual
shared memory IDS uses
during normal operation (use onstat -g seg , to monitor) .
7.31.UD1 is a rather old version. You might want to get a new fix pack.
With kind regards
Tilman Model-Bosch
--
Tilman Model-Bosch
IBM Data Management Solutions, Informix Advanced Support
c\\\\o SAP AG
TECHDEV 05
Neurrotstr.16
69190 Walldorf
forum.subscriber@iiug.org wrote on 09.09.2005 15:43:09:
> Is there another solution for this problem before trying to restore from
> backup/archive?.
> LINUX/IFX 7.31.UD1.
>
> 10:11:22 Informix Dynamic Server Initialized -- Shared Memory
Initialized.
> 10:11:22 Physical Recovery Started.
> 10:11:22 Physical Recovery Complete: 64 Pages Restored.
> 10:11:22 Logical Recovery Started.
> 10:11:22 Dynamically allocated new virtual shared memory segment (size
> 8192KB)
>
> 10:11:22 Assert Failed: Error during recovery left index inconsistent.
> 10:11:22 Informix Dynamic Server Version 7.31.UD1
> 10:11:22 Who: Session(7, informix@bd_pfbsa00, 0, 367676640)
> Thread(30, xchg_1.5, 15e800e0, 4)
> File: rskey.c Line: 1374
> 10:11:22 Results: Index 'guardiao:"guardiao".cnfg_log' is now unusable
> 10:11:22 Action: Run 'oncheck -cI guardiao:"guardiao".cnfg_log'
> 10:11:22 Assert Failed: Error during recovery left index inconsistent.
> 10:11:22 Informix Dynamic Server Version 7.31.UD1
> 10:11:22 Who: Session(7, informix@bd_pfbsa00, 0, 367684648)
> Thread(27, xchg_1.2, 15e7f1ec, 3)
> File: rskey.c Line: 1374
> 10:11:22 Results: Index 'guardiao:"guardiao".chmd_chamada_log' is now
> unusable10:11:22 Action: Run 'oncheck -cI
> guardiao:"guardiao".chmd_chamada_log'
> 10:11:23 See Also: /tmp/af.40389fa
> 10:11:23 Rollforward of log record failed. iserrno = 105
> 10:11:23 Log Record: log = 6227, pos = 8d1088, type =
> OLDRSAM:ADDITEM(28), trans = 14
> 10:11:23 Rollforward of log record failed. iserrno = 105
> 10:11:23 Log Record: log = 6227, pos = 8d1088, type =
> OLDRSAM:ADDITEM(28), trans = 14
> ..
> 10:11:34 PANIC: Attempting to bring system down
>
> Jairo
I had the same problem twice and reportedd it to
SAP/Informix.
There solution was to upgrade...from 7.31FD2 to FD6, they could not tell
what was the cause, new bugs....memory corruption....problem with the
log xxxx
-----Original Message-----
From: forum.subscriber@iiug.org [mailto:forum.subscriber@iiug.org] On
Behalf Of Tilman Mode....
Sent: Thursday, October 20, 2005 11:17 AM
To: ids@iiug.org
Subject: Re: PANIC [5870]
Jairo,
From the online.log it appears that IDS does not come online , so
running the suggested onchecks will not be possible.
You might want to ask Informix Support to disable the log entries for
the indexes in question (which cause the trouble during fast recovery)
before restarting IDS.
But it is not guaranteed that this will result in success (interfering
with the fast recovery process i always a tricky issue)
A restore might be quicker (in case you have a fairly recent backup) .
Apart from that , two points:
online.log says
> 10:11:22 Dynamically allocated new virtual shared memory segment
> (size
> 8192KB)
right at startup.
This indicates that you SHMVIRTSIZE parameter (ONCONFIG file) is too
small.
Please set SHMVIRTSIZE to a value that is about the amount of virtual
shared memory IDS uses during normal operation (use onstat -g seg , to
monitor) .
7.31.UD1 is a rather old version. You might want to get a new fix pack.
With kind regards
Tilman Model-Bosch
--
Tilman Model-Bosch
IBM Data Management Solutions, Informix Advanced Support c\\\\o SAP AG
TECHDEV 05
Neurrotstr.16
69190 Walldorf
forum.subscriber@iiug.org wrote on 09.09.2005 15:43:09:
> Is there another solution for this problem before trying to restore
> from backup/archive?.
> LINUX/IFX 7.31.UD1.
>
> 10:11:22 Informix Dynamic Server Initialized -- Shared Memory
Initialized.
> 10:11:22 Physical Recovery Started.
> 10:11:22 Physical Recovery Complete: 64 Pages Restored.
> 10:11:22 Logical Recovery Started.
> 10:11:22 Dynamically allocated new virtual shared memory segment
> (size
> 8192KB)
>
> 10:11:22 Assert Failed: Error during recovery left index
inconsistent.
> 10:11:22 Informix Dynamic Server Version 7.31.UD1
> 10:11:22 Who: Session(7, informix@bd_pfbsa00, 0, 367676640)
> Thread(30, xchg_1.5, 15e800e0, 4)
> File: rskey.c Line: 1374
> 10:11:22 Results: Index 'guardiao:"guardiao".cnfg_log' is now
unusable
> 10:11:22 Action: Run 'oncheck -cI guardiao:"guardiao".cnfg_log'
> 10:11:22 Assert Failed: Error during recovery left index
inconsistent.
> 10:11:22 Informix Dynamic Server Version 7.31.UD1
> 10:11:22 Who: Session(7, informix@bd_pfbsa00, 0, 367684648)
> Thread(27, xchg_1.2, 15e7f1ec, 3)
> File: rskey.c Line: 1374
> 10:11:22 Results: Index 'guardiao:"guardiao".chmd_chamada_log' is
now
> unusable10:11:22 Action: Run 'oncheck -cI
> guardiao:"guardiao".chmd_chamada_log'
> 10:11:23 See Also: /tmp/af.40389fa
> 10:11:23 Rollforward of log record failed. iserrno = 105
> 10:11:23 Log Record: log = 6227, pos = 8d1088, type =
> OLDRSAM:ADDITEM(28), trans = 14
> 10:11:23 Rollforward of log record failed. iserrno = 105
> 10:11:23 Log Record: log = 6227, pos = 8d1088, type =
> OLDRSAM:ADDITEM(28), trans = 14 ..
> 10:11:34 PANIC: Attempting to bring system down
>
> Jairo
Related threads
- Posting from the Informix-list
- Migrating from IDS 9.40.UC6 to 11.50.UC3
- Ip for a network session
- questions onstat -g