Re: IDS no longer online, now saying "Primary or Mirror Chunks Offline"
Posted in 2003
Topics: Storage & Space Management, Error Codes & Troubleshooting, Server Administration, Logging & Checkpoints, Cloud, Docker & Containers
I'm going to show my ignorance on this one, but how does one "back out
the logs"? I can roll them forward (although at this point in the
game, the logs are at 100%, except for one, which is at 99%), using
onmode -l, but that does not work. How do I "back out" of a log?
--Anthony
Paul Watson <paul@oninit.com> wrote in message news:<3ECDD96E.D67D9BB7@oninit.com>...
> Looks like you've run out of logs. Start in queiscent and back
> out the logs.
>
> Anthony Presley wrote:
> >
> > Group:
> >
> > I got a call from a remote client that I recently picked up, who
> > conveniently doesn't have an informix support contract, of any kind.
> >
> > Their system went down today, and the database has marked their
> > primary chunk (they have no mirror chunks), dbspace1, as Disabled (D).
> > Using dbaccess, I can get into the sysmaster and sysutils database,
> > but not the primary database (azure).
> >
> > In looking through the logs, at about 14:36 they had a reboot (most
> > likely it wasn't a planned shutdown), and the system has logged:
> >
> > 14:36:01 Event alarms enabled. ALARMPROG =
> > '/opt/informix/etc/log_full.sh'
> > 14:36:07 DR: DRAUTO is 0 (Off)
> > 14:36:07 Requested shared memory segment size rounded from 2644KB to> > 2648KB
> > 14:36:07 Informix Dynamic Server Version 7.30.UC7 Software Serial> > Number AAC#J893194
> > 14:36:07 Informix Dynamic Server Initialized -- Shared Memory> > Initialized.
> > 14:36:07 Physical Recovery Started.
> > 14:36:08 Physical Recovery Complete: 48 Pages Restored.
> > 14:36:08 Logical Recovery Started.
> > 14:36:09 Rollforward of log record failed. iserrno = 126
> > 14:36:09 Log Record: log = 2232236, pos = 9c220c, type =
> > OLDRSAM:HUPDATE(73), trans = 8
> > 14:36:09 Rollforward of log record failed. iserrno = 126
> > 14:36:09 Log Record: log = 2232236, pos = 9c17e8, type =
> > OLDRSAM:HUPDATE(73), trans = 31
> > 14:36:09 Assert Failed: Error during recovery left index> > inconsistent.
> > 14:36:09 Informix Dynamic Server Version 7.30.UC7
> > 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0)
> > Thread(29, xchg_1.4, 0, 1)
> > File: rskey.c Line: 1180
> > 14:36:09 Results: Index 'azure:"".' is now unusable
> > 14:36:09 Action: Run 'oncheck -cI azure:"".'
> > 14:36:09 See Also: /tmp/af.1d26a8
> > 14:36:09 Rollforward of log record failed. iserrno = 105
> > 14:36:09 Log Record: log = 2232236, pos = 9c1698, type =
> > OLDRSAM:ADDITEM(28), trans = 31
> > 14:36:09 Assert Failed: Error during recovery left index> > inconsistent.
> > 14:36:09 Informix Dynamic Server Version 7.30.UC7
> > 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0)
> > Thread(29, xchg_1.4, 0, 1)
> > File: rskey.c Line: 1180
> > 14:36:09 Results: Index 'azure:"".' is now unusable
> > 14:36:09 Action: Run 'oncheck -cI azure:"".'
> > 14:36:09 See Also: /tmp/af.1d26a8
> > 14:36:09 Rollforward of log record failed. iserrno = 105
> > 14:36:09 Log Record: log = 2232236, pos = 9c16cc, type =
> > OLDRSAM:ADDITEM(28), trans = 31
> > 14:36:09 Assert Failed: Error during recovery left index> > inconsistent.
> > 14:36:09 Informix Dynamic Server Version 7.30.UC7
> > 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0)
> > Thread(29, xchg_1.4, 0, 1)
> > File: rskey.c Line: 1180
> > 14:36:09 Results: Index 'azure:"".' is now unusable
> > 14:36:09 Action: Run 'oncheck -cI azure:"".'
> > 14:36:09 See Also: /tmp/af.1d26a8
> > 14:36:09 Rollforward of log record failed. iserrno = 105
> > 14:36:09 Log Record: log = 2232236, pos = 9c1700, type =
> > OLDRSAM:ADDITEM(28), trans = 31
> > 14:36:09 Assert Failed: Error during recovery left index> > inconsistent.
> > 14:36:09 Informix Dynamic Server Version 7.30.UC7
> > 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0)
> > Thread(29, xchg_1.4, 0, 1)
> > File: rskey.c Line: 1180
> > 14:36:09 Results: Index 'azure:"".' is now unusable
> > 14:36:09 Action: Run 'oncheck -cI azure:"".'
> > 14:36:09 See Also: /tmp/af.1d26a8
> > 14:36:09 Rollforward of log record failed. iserrno = 105
> > 14:36:09 Log Record: log = 2232236, pos = 9c1734, type =
> > OLDRSAM:ADDITEM(28), trans = 31
> > 14:36:09 Assert Failed: Error during recovery left index> > inconsistent.
> > 14:36:09 Informix Dynamic Server Version 7.30.UC7
> > 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0)
> > Thread(29, xchg_1.4, 0, 1)
> > File: rskey.c Line: 1180
> > 14:36:09 Results: Index 'azure:"".' is now unusable
> > 14:36:09 Action: Run 'oncheck -cI azure:"".'
> > 14:36:09 See Also: /tmp/af.1d26a8
> > 14:36:09 Rollforward of log record failed. iserrno = 105
> > 14:36:09 Log Record: log = 2232236, pos = 9c1768, type =
> > OLDRSAM:ADDITEM(28), trans = 31
> > 14:36:09 Assert Failed: Chunk 2 is being taken OFFLINE.
> > 14:36:09 Informix Dynamic Server Version 7.30.UC7
> > 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0)
> > Thread(27, xchg_1.2, 0, 1)
> > File: rsmirror.c Line: 1772
> > 14:36:09 Results: Dynamic Server will block at next checkpoint
> > 14:36:09 Action: Shutdown (onmode -k) or override (onmode -O)
> > 14:36:09 See Also: /tmp/af.1b26a7
> > 14:36:09 Rollforward of log record failed. iserrno = 101
> > 14:36:09 Log Record: log = 2232236, pos = 9c23ec, type =
> > OLDRSAM:ADDITEM(28), trans = 8
> > 14:36:09 Assert Failed: Chunk 2 is being taken OFFLINE.
> > 14:36:09 Informix Dynamic Server Version 7.30.UC7
> > 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0)
> > Thread(28, xchg_1.3, 0, 1)
> > File: rsmirror.c Line: 1772
> > 14:36:09 Results: Dynamic Server will block at next checkpoint
> > 14:36:09 Action: Shutdown (onmode -k) or override (onmode -O)
> > 14:36:09 See Also: /tmp/af.1c26a8
> > 14:36:09 Rollforward of log record failed. iserrno = 101
> > 14:36:09 Log Record: log = 2232236, pos = 9c20f0, type =
> > OLDRSAM:ADDITEM(28), trans = 31
> > 14:36:09 Rollforward of log record failed. iserrno = 101
> > 14:36:09 Log Record: log = 2232236, pos = 9c2124, type =
> > OLDRSAM:DELITEM(29), trans = 31
> > 14:36:09 Rollforward of log record failed. iserrno = 101
> > 14:36:09 Log Record: log = 2232236, pos = 9c2158, type =
> > OLDRSAM:ADDITEM(28), trans = 31
> > 14:36:09 Rollforward of log record failed. iserrno = 101
> > 14:36:09 Log Record: log = 2232236, pos = 9c218c, type =
> > OLDRSAM:DELITEM(29), trans = 31
> > 14:36:10 Logical Recovery Complete.> > 13 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks
> >
> > 14:36:10 Dataskip is now OFF for all dbspaces14:36:11 On-Line Mode
> > 14:36:11 Checkpoint blocked by down space, waiting for override or> > shutdown
> >
> > I've tried to run oncheck -cI azure:"", and I get the error "ISAM
> > error: Primary and Mirror chunks are bad"
> >
> > Running oncheck -cRe