IDS no longer online, now saying "Primary or Mirror Chunks Offline"

Group: I got a call from a remote client that I recently picked up, who conveniently doesn't have an informix support contract, of any kind. Their system went down today, and the database has marked their primary chunk (they have no mirror chunks), dbspace1, as Disabled (D). Using dbaccess, I can get into the sysmaster and sysutils database, but not the primary database (azure). In looking through the logs, at about 14:36 they had a reboot (most likely it wasn't a planned shutdown), and the system has logged: 14:36:01 Event alarms enabled. ALARMPROG = '/opt/informix/etc/log_full.sh' 14:36:07 DR: DRAUTO is 0 (Off) 14:36:07 Requested shared memory segment size rounded from 2644KB to2648KB 14:36:07 Informix Dynamic Server Version 7.30.UC7 Software SerialNumber AAC#J893194 14:36:07 Informix Dynamic Server Initialized -- Shared MemoryInitialized. 14:36:07 Physical Recovery Started. 14:36:08 Physical Recovery Complete: 48 Pages Restored. 14:36:08 Logical Recovery Started. 14:36:09 Rollforward of log record failed. iserrno = 126 14:36:09 Log Record: log = 2232236, pos = 9c220c, type = OLDRSAM:HUPDATE(73), trans = 8 14:36:09 Rollforward of log record failed. iserrno = 126 14:36:09 Log Record: log = 2232236, pos = 9c17e8, type = OLDRSAM:HUPDATE(73), trans = 31 14:36:09 Assert Failed: Error during recovery left indexinconsistent. 14:36:09 Informix Dynamic Server Version 7.30.UC7 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0) Thread(29, xchg_1.4, 0, 1) File: rskey.c Line: 1180 14:36:09 Results: Index 'azure:"".' is now unusable 14:36:09 Action: Run 'oncheck -cI azure:"".' 14:36:09 See Also: /tmp/af.1d26a8 14:36:09 Rollforward of log record failed. iserrno = 105 14:36:09 Log Record: log = 2232236, pos = 9c1698, type = OLDRSAM:ADDITEM(28), trans = 31 14:36:09 Assert Failed: Error during recovery left indexinconsistent. 14:36:09 Informix Dynamic Server Version 7.30.UC7 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0) Thread(29, xchg_1.4, 0, 1) File: rskey.c Line: 1180 14:36:09 Results: Index 'azure:"".' is now unusable 14:36:09 Action: Run 'oncheck -cI azure:"".' 14:36:09 See Also: /tmp/af.1d26a8 14:36:09 Rollforward of log record failed. iserrno = 105 14:36:09 Log Record: log = 2232236, pos = 9c16cc, type = OLDRSAM:ADDITEM(28), trans = 31 14:36:09 Assert Failed: Error during recovery left indexinconsistent. 14:36:09 Informix Dynamic Server Version 7.30.UC7 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0) Thread(29, xchg_1.4, 0, 1) File: rskey.c Line: 1180 14:36:09 Results: Index 'azure:"".' is now unusable 14:36:09 Action: Run 'oncheck -cI azure:"".' 14:36:09 See Also: /tmp/af.1d26a8 14:36:09 Rollforward of log record failed. iserrno = 105 14:36:09 Log Record: log = 2232236, pos = 9c1700, type = OLDRSAM:ADDITEM(28), trans = 31 14:36:09 Assert Failed: Error during recovery left indexinconsistent. 14:36:09 Informix Dynamic Server Version 7.30.UC7 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0) Thread(29, xchg_1.4, 0, 1) File: rskey.c Line: 1180 14:36:09 Results: Index 'azure:"".' is now unusable 14:36:09 Action: Run 'oncheck -cI azure:"".' 14:36:09 See Also: /tmp/af.1d26a8 14:36:09 Rollforward of log record failed. iserrno = 105 14:36:09 Log Record: log = 2232236, pos = 9c1734, type = OLDRSAM:ADDITEM(28), trans = 31 14:36:09 Assert Failed: Error during recovery left indexinconsistent. 14:36:09 Informix Dynamic Server Version 7.30.UC7 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0) Thread(29, xchg_1.4, 0, 1) File: rskey.c Line: 1180 14:36:09 Results: Index 'azure:"".' is now unusable 14:36:09 Action: Run 'oncheck -cI azure:"".' 14:36:09 See Also: /tmp/af.1d26a8 14:36:09 Rollforward of log record failed. iserrno = 105 14:36:09 Log Record: log = 2232236, pos = 9c1768, type = OLDRSAM:ADDITEM(28), trans = 31 14:36:09 Assert Failed: Chunk 2 is being taken OFFLINE. 14:36:09 Informix Dynamic Server Version 7.30.UC7 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0) Thread(27, xchg_1.2, 0, 1) File: rsmirror.c Line: 1772 14:36:09 Results: Dynamic Server will block at next checkpoint 14:36:09 Action: Shutdown (onmode -k) or override (onmode -O) 14:36:09 See Also: /tmp/af.1b26a7 14:36:09 Rollforward of log record failed. iserrno = 101 14:36:09 Log Record: log = 2232236, pos = 9c23ec, type = OLDRSAM:ADDITEM(28), trans = 8 14:36:09 Assert Failed: Chunk 2 is being taken OFFLINE. 14:36:09 Informix Dynamic Server Version 7.30.UC7 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0) Thread(28, xchg_1.3, 0, 1) File: rsmirror.c Line: 1772 14:36:09 Results: Dynamic Server will block at next checkpoint 14:36:09 Action: Shutdown (onmode -k) or override (onmode -O) 14:36:09 See Also: /tmp/af.1c26a8 14:36:09 Rollforward of log record failed. iserrno = 101 14:36:09 Log Record: log = 2232236, pos = 9c20f0, type = OLDRSAM:ADDITEM(28), trans = 31 14:36:09 Rollforward of log record failed. iserrno = 101 14:36:09 Log Record: log = 2232236, pos = 9c2124, type = OLDRSAM:DELITEM(29), trans = 31 14:36:09 Rollforward of log record failed. iserrno = 101 14:36:09 Log Record: log = 2232236, pos = 9c2158, type = OLDRSAM:ADDITEM(28), trans = 31 14:36:09 Rollforward of log record failed. iserrno = 101 14:36:09 Log Record: log = 2232236, pos = 9c218c, type = OLDRSAM:DELITEM(29), trans = 31 14:36:10 Logical Recovery Complete. 13 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks 14:36:10 Dataskip is now OFF for all dbspaces14:36:11 On-Line Mode 14:36:11 Checkpoint blocked by down space, waiting for override orshutdown I've tried to run oncheck -cI azure:"", and I get the error "ISAM error: Primary and Mirror chunks are bad" Running oncheck -cRe returns no errors Running oncheck -c{d,I,D} returns the same (above) ISAM error. onstat -l: Informix Dynamic Server Version 7.30.UC7 -- On-Line (CKPT REQ) -- Up 00:08:46 -- 46416 Kbytes Blocked:CKPT Physical Logging Buffer bufused bufsize numpages numwrits pages/io P-1 0 16 0 0 0.00% phybegin physize phypos phyused %used 300035 140000 16344 0 0.00 Logical Logging Buffer bufused bufsize numrecs numpages numwrits recs/pages pages/io L-1 0 16 0 0 0 0.0 0.0 Subsystem numrecs Log Space used address number flags uniqid begin size used %used 114edee4 1 U------ 2232235 1013c7 2500 2500 100.00% 114edf00 2 U---C-L 2232236 101d8b 2500 2499 99.96% 114edf1c 3 U------ 2232231 10274f 2500 2500 100.00% 114edf38 4 U------ 2232232 103113 2500

Looks like you've run out of logs. Start in queiscent and back out the logs. Anthony Presley wrote: > > Group: > > I got a call from a remote client that I recently picked up, who > conveniently doesn't have an informix support contract, of any kind. > > Their system went down today, and the database has marked their > primary chunk (they have no mirror chunks), dbspace1, as Disabled (D). > Using dbaccess, I can get into the sysmaster and sysutils database, > but not the primary database (azure). > > In looking through the logs, at about 14:36 they had a reboot (most > likely it wasn't a planned shutdown), and the system has logged: > > 14:36:01 Event alarms enabled. ALARMPROG = > '/opt/informix/etc/log_full.sh' > 14:36:07 DR: DRAUTO is 0 (Off) > 14:36:07 Requested shared memory segment size rounded from 2644KB to> 2648KB > 14:36:07 Informix Dynamic Server Version 7.30.UC7 Software Serial> Number AAC#J893194 > 14:36:07 Informix Dynamic Server Initialized -- Shared Memory> Initialized. > 14:36:07 Physical Recovery Started. > 14:36:08 Physical Recovery Complete: 48 Pages Restored. > 14:36:08 Logical Recovery Started. > 14:36:09 Rollforward of log record failed. iserrno = 126 > 14:36:09 Log Record: log = 2232236, pos = 9c220c, type = > OLDRSAM:HUPDATE(73), trans = 8 > 14:36:09 Rollforward of log record failed. iserrno = 126 > 14:36:09 Log Record: log = 2232236, pos = 9c17e8, type = > OLDRSAM:HUPDATE(73), trans = 31 > 14:36:09 Assert Failed: Error during recovery left index> inconsistent. > 14:36:09 Informix Dynamic Server Version 7.30.UC7 > 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0) > Thread(29, xchg_1.4, 0, 1) > File: rskey.c Line: 1180 > 14:36:09 Results: Index 'azure:"".' is now unusable > 14:36:09 Action: Run 'oncheck -cI azure:"".' > 14:36:09 See Also: /tmp/af.1d26a8 > 14:36:09 Rollforward of log record failed. iserrno = 105 > 14:36:09 Log Record: log = 2232236, pos = 9c1698, type = > OLDRSAM:ADDITEM(28), trans = 31 > 14:36:09 Assert Failed: Error during recovery left index> inconsistent. > 14:36:09 Informix Dynamic Server Version 7.30.UC7 > 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0) > Thread(29, xchg_1.4, 0, 1) > File: rskey.c Line: 1180 > 14:36:09 Results: Index 'azure:"".' is now unusable > 14:36:09 Action: Run 'oncheck -cI azure:"".' > 14:36:09 See Also: /tmp/af.1d26a8 > 14:36:09 Rollforward of log record failed. iserrno = 105 > 14:36:09 Log Record: log = 2232236, pos = 9c16cc, type = > OLDRSAM:ADDITEM(28), trans = 31 > 14:36:09 Assert Failed: Error during recovery left index> inconsistent. > 14:36:09 Informix Dynamic Server Version 7.30.UC7 > 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0) > Thread(29, xchg_1.4, 0, 1) > File: rskey.c Line: 1180 > 14:36:09 Results: Index 'azure:"".' is now unusable > 14:36:09 Action: Run 'oncheck -cI azure:"".' > 14:36:09 See Also: /tmp/af.1d26a8 > 14:36:09 Rollforward of log record failed. iserrno = 105 > 14:36:09 Log Record: log = 2232236, pos = 9c1700, type = > OLDRSAM:ADDITEM(28), trans = 31 > 14:36:09 Assert Failed: Error during recovery left index> inconsistent. > 14:36:09 Informix Dynamic Server Version 7.30.UC7 > 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0) > Thread(29, xchg_1.4, 0, 1) > File: rskey.c Line: 1180 > 14:36:09 Results: Index 'azure:"".' is now unusable > 14:36:09 Action: Run 'oncheck -cI azure:"".' > 14:36:09 See Also: /tmp/af.1d26a8 > 14:36:09 Rollforward of log record failed. iserrno = 105 > 14:36:09 Log Record: log = 2232236, pos = 9c1734, type = > OLDRSAM:ADDITEM(28), trans = 31 > 14:36:09 Assert Failed: Error during recovery left index> inconsistent. > 14:36:09 Informix Dynamic Server Version 7.30.UC7 > 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0) > Thread(29, xchg_1.4, 0, 1) > File: rskey.c Line: 1180 > 14:36:09 Results: Index 'azure:"".' is now unusable > 14:36:09 Action: Run 'oncheck -cI azure:"".' > 14:36:09 See Also: /tmp/af.1d26a8 > 14:36:09 Rollforward of log record failed. iserrno = 105 > 14:36:09 Log Record: log = 2232236, pos = 9c1768, type = > OLDRSAM:ADDITEM(28), trans = 31 > 14:36:09 Assert Failed: Chunk 2 is being taken OFFLINE. > 14:36:09 Informix Dynamic Server Version 7.30.UC7 > 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0) > Thread(27, xchg_1.2, 0, 1) > File: rsmirror.c Line: 1772 > 14:36:09 Results: Dynamic Server will block at next checkpoint > 14:36:09 Action: Shutdown (onmode -k) or override (onmode -O) > 14:36:09 See Also: /tmp/af.1b26a7 > 14:36:09 Rollforward of log record failed. iserrno = 101 > 14:36:09 Log Record: log = 2232236, pos = 9c23ec, type = > OLDRSAM:ADDITEM(28), trans = 8 > 14:36:09 Assert Failed: Chunk 2 is being taken OFFLINE. > 14:36:09 Informix Dynamic Server Version 7.30.UC7 > 14:36:09 Who: Session(11, root@sys3.inventconnect.com, 0, 0) > Thread(28, xchg_1.3, 0, 1) > File: rsmirror.c Line: 1772 > 14:36:09 Results: Dynamic Server will block at next checkpoint > 14:36:09 Action: Shutdown (onmode -k) or override (onmode -O) > 14:36:09 See Also: /tmp/af.1c26a8 > 14:36:09 Rollforward of log record failed. iserrno = 101 > 14:36:09 Log Record: log = 2232236, pos = 9c20f0, type = > OLDRSAM:ADDITEM(28), trans = 31 > 14:36:09 Rollforward of log record failed. iserrno = 101 > 14:36:09 Log Record: log = 2232236, pos = 9c2124, type = > OLDRSAM:DELITEM(29), trans = 31 > 14:36:09 Rollforward of log record failed. iserrno = 101 > 14:36:09 Log Record: log = 2232236, pos = 9c2158, type = > OLDRSAM:ADDITEM(28), trans = 31 > 14:36:09 Rollforward of log record failed. iserrno = 101 > 14:36:09 Log Record: log = 2232236, pos = 9c218c, type = > OLDRSAM:DELITEM(29), trans = 31 > 14:36:10 Logical Recovery Complete.> 13 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks > > 14:36:10 Dataskip is now OFF for all dbspaces14:36:11 On-Line Mode > 14:36:11 Checkpoint blocked by down space, waiting for override or> shutdown > > I've tried to run oncheck -cI azure:"", and I get the error "ISAM > error: Primary and Mirror chunks are bad" > > Running oncheck -cRe returns no errors > Running oncheck -c{d,I,D} returns the same (above) ISAM error. > > onstat -l:> > Informix Dynamic Server Version 7.30.UC7 -- On-Line (CKPT REQ) -- Up > 00:08:46 -- 46416 Kbytes > Blocked:CKPT > > Physical Logging > Buffer bufused bufsize numpages numwrits pages/io > P-1 0 16 0 0 0.00% > phybegin physize phypos phyused %used > 300035 140000 16344 0 0.00 > > Logical Logging > Buffer bufused bufsize numrecs numpages numwrits recs/pages > pages/io > L-1 0 16 0 0 0 0.0 0.0 >