backup failed

HI, Folks, IDs11.50 FC8, RHEL 6 ontape is failed, the following is the info. Any suggestions? Thanks Frank 1)command: [informix@stephanie ontape]$ ontape -s -L 0 Please mount tape 1 on /dbbackup/ontape/ontape_nsofpal and press Return to continue ... 10 percent done. Archive failed - function read archive backup failed code -1 errno 0 Program over. 2) online log: 3:17:12 Maximum server connections 65 13:17:12 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 16, Llog used 185 13:17:13 Level 0 Archive started on rootdbs, dblog1, dblog2, dberdata00, dbdata09_1, dbdata09_2, dbdata00, dbdata01, dbdata02, dbdata03, dbdata04, dbdata05, dbdata06, dbdata07, dbdata08, dbdata09, dbdata10, dbdata11, dbdw00, dbdw01, dbdw02, dbdw03, dbdw04, dbdw05, dbdw06, dbdw07, dbdw08, dbdata20, dbphylog, dbdata12, dbdata13, dbdata21, dbersbdata00, dbersbdata01, dbsbdata00 13:21:50 I/O read chunk 99, pagenum 2, pagecnt 7 --> errno = 2 13:21:50 Assert Warning: ISAM error: An error has occurred during archive back up. 13:21:50 IBM Informix Dynamic Server Version 11.50.FC8 13:21:50 Who: Session(49353, informix@stephanie, 31263, 0xc23980f0) Thread(55326, arcbackup1, c2367f30, 7) File: rsarcutl.c Line: 124 13:21:50 Action: arc_pages()/bfphysio() 13:21:50 stack trace for pid 842 written to /tmp/af.dc0640ee 13:21:52 See Also: /tmp/af.dc0640ee 13:21:52 ISAM error: An error has occurred during archive back up. 13:21:53 Archive on rootdbs, dblog1, dblog2, dberdata00, dbdata09_1, dbdata09_2, dbdata00, dbdata01, dbdata02, dbdata03, dbdata04, dbdata05, dbdata06, dbdata07, dbdata08, dbdata09, dbdata10, dbdata11, dbdw00, dbdw01, dbdw02, dbdw03, dbdw04, dbdw05, dbdw06, dbdw07, dbdw08, dbdata20, dbphylog, dbdata12, dbdata13, dbdata21, dbersbdata00, dbersbdata01, dbsbdata00 ABORTED. 13:21:53 Aborted by client. 13:22:19 Checkpoint Completed: duration was 0 seconds. 13:22:19 Tue Jun 17 - loguniq 91344, logpos 0xa46e018, timestamp: 0x3e945f51 Interval: 334495 13:22:19 Maximum server connections 65 13:22:20 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 28, Llog used 410 3) af file [informix@stephanie tmp]$ more af.dc0640ee 13:21:50 mt_aio_wait(0xc63b3608) 13:21:50 chunkio(101, 0x0xc6356c00, 0x2, 7, 0x0xc35fe9d8, 33, 1) errno 2 AIOREQ *a 00000000c70c68c0: 00000000 00000000 0a000000 00000000 ........ ........ 00000000c70c68d0: 8adc0000 00000000 00000000 00000000 ........ ........ 00000000c70c68e0: 00000000 00000000 00000000 00000000 ........ ........ 00000000c70c68f0: f8383bc6 00000000 18333bc6 00000000 .8;..... .3;..... 00000000c70c6900: 00000000 00000000 00000000 00000000 ........ ........ 00000000c70c6910: c03b0000 41494f72 65710000 00000000 .;..AIOr eq...... 00000000c70c6920: 00000000 00000000 d1070000 00000000 ........ ........ 00000000c70c6930: 00000000 00000000 04000000 01040000 ........ ........ 00000000c70c6940: 0b000000 02000000 01000000 00000000 ........ ........ 00000000c70c6950: 01000000 00000000 e0363bc6 00000000 ........ .6;..... 00000000c70c6960: 00000000 00000000 01000000 09000000 ........ ........ 00000000c70c6970: 01000000 00000000 00000000 00000000 ........ ........ 00000000c70c6980: 00000000 00000000 00000000 00000000 ........ ........ 00000000c70c6990: 00000000 00000000 ........ 13:21:50 13:21:50 IBM Informix Dynamic Server Version 11.50.FC8 Software Serial Number AAA#B000000 13:21:50 Assert Warning: ISAM error: An error has occurred during archive back up. 13:21:50 Who: Session(49353, informix@stephanie, 31263, 0xc23980f0) Thread(55326, arcbackup1, c2367f30, 7) File: rsarcutl.c Line: 124 13:21:50 Action: arc_pages()/bfphysio() 13:21:50 Raw hex dump of stack located in /tmp/af.dc0640ee.rawstk 13:21:50 Stack for thread: 55326 arcbackup1 base: 0x00000000c70b6000 len: 69632 pc: 0x00000000010c91d7 tos: 0x00000000c70c5ef0 state: running vp: 7 0x00000000010c91d7 (oninit) afstack 0x00000000010cb12c (oninit) afhandler 0x00000000010cb9a7 (oninit) afwarn_interface 0x0000000000c7d7a6 (oninit) oops_error 0x0000000000c8943a (oninit) arc_pages 0x0000000000c8959f (oninit) arc_dbspace 0x0000000000c8a890 (oninit) arc_scanner 0x000000000109a9bb (oninit) startup 13:21:52 See Also: /tmp/af.dc0640ee --------------------------------- Begin System Alarm Program Output --------------------------------- Assertion Failure Type: Warning Host Name: stephanie Database Server Name: nsofpal_tcp Time of failure: Tue Jun 17 13:21:52 UTC 2014 AF file: /tmp/af.dc0640ee Shared memory file: None System Blocking: OFF ------------------------------- End System Alarm Program Output ------------------------------- 13:21:52 sh /usr/informix/etc/evidence.sh 1 0 /tmp/af.dc0640ee 49353 0xc2367f30 55326 0xc54c53a8 1025 0 0 0 0 13:21:52 ------------------ End of assertion failure 0 ----------------- --001a113a5fd0fbac6504fc08273b

Frank This looks like a disk corruption, is the database behaving currently ?? From onstat -d do you know what is on chunk 99 ? What do onchecks against the table show ? Have you tried as second archive to see if it is a transient condition ? Make sure your latest complete archive and logs are secure and available, you may need to restore. Keith On 17 June 2014 14:31, FRANK <yunyaoqu@gmail.com> wrote: > HI, Folks, > > IDs11.50 FC8, RHEL 6 > > ontape is failed, the following is the info. > > Any suggestions? > > Thanks > Frank > > 1)command: > > [informix@stephanie ontape]$ ontape -s -L 0 > Please mount tape 1 on /dbbackup/ontape/ontape_nsofpal and press Return to > continue ... > 10 percent done. > Archive failed - function read archive backup failed code -1 errno 0 > Program over. > > 2) online log: > > 3:17:12 Maximum server connections 65 > 13:17:12 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, Plog used 16, Llog used 185 > 13:17:13 Level 0 Archive started on rootdbs, dblog1, dblog2, dberdata00, > dbdata09_1, dbdata09_2, dbdata00, dbdata01, dbdata02, dbdata03, dbdata04, > dbdata05, dbdata06, dbdata07, dbdata08, dbdata09, dbdata10, dbdata11, > dbdw00, dbdw01, dbdw02, dbdw03, dbdw04, dbdw05, dbdw06, dbdw07, dbdw08, > dbdata20, dbphylog, dbdata12, dbdata13, dbdata21, dbersbdata00, > dbersbdata01, dbsbdata00 > 13:21:50 I/O read chunk 99, pagenum 2, pagecnt 7 --> errno = 2 > 13:21:50 Assert Warning: ISAM error: An error has occurred during archive > back up. > > 13:21:50 IBM Informix Dynamic Server Version 11.50.FC8 > 13:21:50 Who: Session(49353, informix@stephanie, 31263, 0xc23980f0) > > Thread(55326, arcbackup1, c2367f30, 7) > > File: rsarcutl.c Line: 124 > 13:21:50 Action: arc_pages()/bfphysio() > 13:21:50 stack trace for pid 842 written to /tmp/af.dc0640ee > 13:21:52 See Also: /tmp/af.dc0640ee > 13:21:52 ISAM error: An error has occurred during archive back up. > > 13:21:53 Archive on rootdbs, dblog1, dblog2, dberdata00, dbdata09_1, > dbdata09_2, dbdata00, dbdata01, dbdata02, dbdata03, dbdata04, dbdata05, > dbdata06, dbdata07, dbdata08, dbdata09, dbdata10, dbdata11, dbdw00, dbdw01, > dbdw02, dbdw03, dbdw04, dbdw05, dbdw06, dbdw07, dbdw08, dbdata20, dbphylog, > dbdata12, dbdata13, dbdata21, dbersbdata00, dbersbdata01, dbsbdata00 > ABORTED. > 13:21:53 Aborted by client. > 13:22:19 Checkpoint Completed: duration was 0 seconds. > 13:22:19 Tue Jun 17 - loguniq 91344, logpos 0xa46e018, timestamp: > 0x3e945f51 Interval: 334495 > 13:22:19 Maximum server connections 65 > 13:22:20 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, Plog used 28, Llog used 410 > > 3) af file > > [informix@stephanie tmp]$ more af.dc0640ee > 13:21:50 mt_aio_wait(0xc63b3608) > 13:21:50 chunkio(101, 0x0xc6356c00, 0x2, 7, 0x0xc35fe9d8, 33, 1) errno 2 > AIOREQ *a > 00000000c70c68c0: 00000000 00000000 0a000000 00000000 ........ ........ > 00000000c70c68d0: 8adc0000 00000000 00000000 00000000 ........ ........ > 00000000c70c68e0: 00000000 00000000 00000000 00000000 ........ ........ > 00000000c70c68f0: f8383bc6 00000000 18333bc6 00000000 .8;..... .3;..... > 00000000c70c6900: 00000000 00000000 00000000 00000000 ........ ........ > 00000000c70c6910: c03b0000 41494f72 65710000 00000000 .;..AIOr eq...... > 00000000c70c6920: 00000000 00000000 d1070000 00000000 ........ ........ > 00000000c70c6930: 00000000 00000000 04000000 01040000 ........ ........ > 00000000c70c6940: 0b000000 02000000 01000000 00000000 ........ ........ > 00000000c70c6950: 01000000 00000000 e0363bc6 00000000 ........ .6;..... > 00000000c70c6960: 00000000 00000000 01000000 09000000 ........ ........ > 00000000c70c6970: 01000000 00000000 00000000 00000000 ........ ........ > 00000000c70c6980: 00000000 00000000 00000000 00000000 ........ ........ > 00000000c70c6990: 00000000 00000000 ........ > 13:21:50 > 13:21:50 IBM Informix Dynamic Server Version 11.50.FC8 Software Serial > Number AAA#B000000 > 13:21:50 Assert Warning: ISAM error: An error has occurred during archive > back up. > > 13:21:50 Who: Session(49353, informix@stephanie, 31263, 0xc23980f0) > > Thread(55326, arcbackup1, c2367f30, 7) > > File: rsarcutl.c Line: 124 > 13:21:50 Action: arc_pages()/bfphysio() > 13:21:50 Raw hex dump of stack located in /tmp/af.dc0640ee.rawstk > 13:21:50 Stack for thread: 55326 arcbackup1 > base: 0x00000000c70b6000 > len: 69632 > > pc: 0x00000000010c91d7 > tos: 0x00000000c70c5ef0 > state: running > > vp: 7 > 0x00000000010c91d7 (oninit) afstack > 0x00000000010cb12c (oninit) afhandler > 0x00000000010cb9a7 (oninit) afwarn_interface > 0x0000000000c7d7a6 (oninit) oops_error > 0x0000000000c8943a (oninit) arc_pages > 0x0000000000c8959f (oninit) arc_dbspace > 0x0000000000c8a890 (oninit) arc_scanner > 0x000000000109a9bb (oninit) startup > > 13:21:52 See Also: /tmp/af.dc0640ee > --------------------------------- > Begin System Alarm Program Output > --------------------------------- > Assertion Failure Type: Warning > Host Name: stephanie > Database Server Name: nsofpal_tcp > Time of failure: Tue Jun 17 13:21:52 UTC 2014 > AF file: /tmp/af.dc0640ee > Shared memory file: None > System Blocking: OFF > > ------------------------------- > End System Alarm Program Output > ------------------------------- > 13:21:52 sh /usr/informix/etc/evidence.sh 1 0 /tmp/af.dc0640ee 49353 > 0xc2367f30 55326 0xc54c53a8 1025 0 0 0 0 > 13:21:52 > ------------------ End of assertion failure 0 ----------------- > > --001a113a5fd0fbac6504fc08273b > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --047d7b41c3daa6203f04fc084810

Can you dd the file without any errors ? What /where is chunk 99 ? Is it readable etc ? Cheers Paul > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > FRANK > Sent: Tuesday, June 17, 2014 8:31 AM > To: ids@iiug.org > Subject: backup failed [33214] > > HI, Folks, > > IDs11.50 FC8, RHEL 6 > > ontape is failed, the following is the info. > > Any suggestions? > > Thanks > Frank > > 1)command: > > [informix@stephanie ontape]$ ontape -s -L 0 > Please mount tape 1 on /dbbackup/ontape/ontape_nsofpal and press > Return to > continue ... > 10 percent done. > Archive failed - function read archive backup failed code -1 errno 0 > Program over. > > 2) online log: > > 3:17:12 Maximum server connections 65 > 13:17:12 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, Plog used 16, Llog used 185 > 13:17:13 Level 0 Archive started on rootdbs, dblog1, dblog2, dberdata00, > dbdata09_1, dbdata09_2, dbdata00, dbdata01, dbdata02, dbdata03, > dbdata04, > dbdata05, dbdata06, dbdata07, dbdata08, dbdata09, dbdata10, dbdata11, > dbdw00, dbdw01, dbdw02, dbdw03, dbdw04, dbdw05, dbdw06, dbdw07, > dbdw08, > dbdata20, dbphylog, dbdata12, dbdata13, dbdata21, dbersbdata00, > dbersbdata01, dbsbdata00 > 13:21:50 I/O read chunk 99, pagenum 2, pagecnt 7 --> errno = 2 > 13:21:50 Assert Warning: ISAM error: An error has occurred during archive > back up. > > 13:21:50 IBM Informix Dynamic Server Version 11.50.FC8 > 13:21:50 Who: Session(49353, informix@stephanie, 31263, 0xc23980f0) > > Thread(55326, arcbackup1, c2367f30, 7) > > File: rsarcutl.c Line: 124 > 13:21:50 Action: arc_pages()/bfphysio() > 13:21:50 stack trace for pid 842 written to /tmp/af.dc0640ee > 13:21:52 See Also: /tmp/af.dc0640ee > 13:21:52 ISAM error: An error has occurred during archive back up. > > 13:21:53 Archive on rootdbs, dblog1, dblog2, dberdata00, dbdata09_1, > dbdata09_2, dbdata00, dbdata01, dbdata02, dbdata03, dbdata04, dbdata05, > dbdata06, dbdata07, dbdata08, dbdata09, dbdata10, dbdata11, dbdw00, > dbdw01, > dbdw02, dbdw03, dbdw04, dbdw05, dbdw06, dbdw07, dbdw08, dbdata20, > dbphylog, > dbdata12, dbdata13, dbdata21, dbersbdata00, dbersbdata01, dbsbdata00 > ABORTED. > 13:21:53 Aborted by client. > 13:22:19 Checkpoint Completed: duration was 0 seconds. > 13:22:19 Tue Jun 17 - loguniq 91344, logpos 0xa46e018, timestamp: > 0x3e945f51 Interval: 334495 > 13:22:19 Maximum server connections 65 > 13:22:20 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, Plog used 28, Llog used 410 > > 3) af file > > [informix@stephanie tmp]$ more af.dc0640ee > 13:21:50 mt_aio_wait(0xc63b3608) > 13:21:50 chunkio(101, 0x0xc6356c00, 0x2, 7, 0x0xc35fe9d8, 33, 1) errno 2 > AIOREQ *a > 00000000c70c68c0: 00000000 00000000 0a000000 00000000 ........ ........ > 00000000c70c68d0: 8adc0000 00000000 00000000 00000000 ........ ........ > 00000000c70c68e0: 00000000 00000000 00000000 00000000 ........ ........ > 00000000c70c68f0: f8383bc6 00000000 18333bc6 00000000 .8;..... .3;..... > 00000000c70c6900: 00000000 00000000 00000000 00000000 ........ ........ > 00000000c70c6910: c03b0000 41494f72 65710000 00000000 .;..AIOr eq...... > 00000000c70c6920: 00000000 00000000 d1070000 00000000 ........ ........ > 00000000c70c6930: 00000000 00000000 04000000 01040000 ........ ........ > 00000000c70c6940: 0b000000 02000000 01000000 00000000 ........ ........ > 00000000c70c6950: 01000000 00000000 e0363bc6 00000000 ........ .6;..... > 00000000c70c6960: 00000000 00000000 01000000 09000000 ........ ........ > 00000000c70c6970: 01000000 00000000 00000000 00000000 ........ ........ > 00000000c70c6980: 00000000 00000000 00000000 00000000 ........ ........ > 00000000c70c6990: 00000000 00000000 ........ > 13:21:50 > 13:21:50 IBM Informix Dynamic Server Version 11.50.FC8 Software Serial > Number AAA#B000000 > 13:21:50 Assert Warning: ISAM error: An error has occurred during archive > back up. > > 13:21:50 Who: Session(49353, informix@stephanie, 31263, 0xc23980f0) > > Thread(55326, arcbackup1, c2367f30, 7) > > File: rsarcutl.c Line: 124 > 13:21:50 Action: arc_pages()/bfphysio() > 13:21:50 Raw hex dump of stack located in /tmp/af.dc0640ee.rawstk > 13:21:50 Stack for thread: 55326 arcbackup1 > base: 0x00000000c70b6000 > len: 69632 > > pc: 0x00000000010c91d7 > tos: 0x00000000c70c5ef0 > state: running > > vp: 7 > 0x00000000010c91d7 (oninit) afstack > 0x00000000010cb12c (oninit) afhandler > 0x00000000010cb9a7 (oninit) afwarn_interface > 0x0000000000c7d7a6 (oninit) oops_error > 0x0000000000c8943a (oninit) arc_pages > 0x0000000000c8959f (oninit) arc_dbspace > 0x0000000000c8a890 (oninit) arc_scanner > 0x000000000109a9bb (oninit) startup > > 13:21:52 See Also: /tmp/af.dc0640ee > --------------------------------- > Begin System Alarm Program Output > --------------------------------- > Assertion Failure Type: Warning > Host Name: stephanie > Database Server Name: nsofpal_tcp > Time of failure: Tue Jun 17 13:21:52 UTC 2014 > AF file: /tmp/af.dc0640ee > Shared memory file: None > System Blocking: OFF > > ------------------------------- > End System Alarm Program Output > ------------------------------- > 13:21:52 sh /usr/informix/etc/evidence.sh 1 0 /tmp/af.dc0640ee 49353 > 0xc2367f30 55326 0xc54c53a8 1025 0 0 0 0 > 13:21:52 > ------------------ End of assertion failure 0 ----------------- > > --001a113a5fd0fbac6504fc08273b > > > ********************************************************** > ********************* > Forum Note: Use "Reply" to post a response in the discussion forum.

Ontape received a read error (errno = 2) from chunk # 99 reading the second page in the chunk during the archive. So, first thing, try a dd read of the chunk as Informix or root and see what happens. Go from there. Art Art S. Kagel, Principal Consultant ASK Database Management Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Tue, Jun 17, 2014 at 9:31 AM, FRANK <yunyaoqu@gmail.com> wrote: > HI, Folks, > > IDs11.50 FC8, RHEL 6 > > ontape is failed, the following is the info. > > Any suggestions? > > Thanks > Frank > > 1)command: > > [informix@stephanie ontape]$ ontape -s -L 0 > Please mount tape 1 on /dbbackup/ontape/ontape_nsofpal and press Return to > continue ... > 10 percent done. > Archive failed - function read archive backup failed code -1 errno 0 > Program over. > > 2) online log: > > 3:17:12 Maximum server connections 65 > 13:17:12 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, Plog used 16, Llog used 185 > 13:17:13 Level 0 Archive started on rootdbs, dblog1, dblog2, dberdata00, > dbdata09_1, dbdata09_2, dbdata00, dbdata01, dbdata02, dbdata03, dbdata04, > dbdata05, dbdata06, dbdata07, dbdata08, dbdata09, dbdata10, dbdata11, > dbdw00, dbdw01, dbdw02, dbdw03, dbdw04, dbdw05, dbdw06, dbdw07, dbdw08, > dbdata20, dbphylog, dbdata12, dbdata13, dbdata21, dbersbdata00, > dbersbdata01, dbsbdata00 > 13:21:50 I/O read chunk 99, pagenum 2, pagecnt 7 --> errno = 2 > 13:21:50 Assert Warning: ISAM error: An error has occurred during archive > back up. > > 13:21:50 IBM Informix Dynamic Server Version 11.50.FC8 > 13:21:50 Who: Session(49353, informix@stephanie, 31263, 0xc23980f0) > > Thread(55326, arcbackup1, c2367f30, 7) > > File: rsarcutl.c Line: 124 > 13:21:50 Action: arc_pages()/bfphysio() > 13:21:50 stack trace for pid 842 written to /tmp/af.dc0640ee > 13:21:52 See Also: /tmp/af.dc0640ee > 13:21:52 ISAM error: An error has occurred during archive back up. > > 13:21:53 Archive on rootdbs, dblog1, dblog2, dberdata00, dbdata09_1, > dbdata09_2, dbdata00, dbdata01, dbdata02, dbdata03, dbdata04, dbdata05, > dbdata06, dbdata07, dbdata08, dbdata09, dbdata10, dbdata11, dbdw00, dbdw01, > dbdw02, dbdw03, dbdw04, dbdw05, dbdw06, dbdw07, dbdw08, dbdata20, dbphylog, > dbdata12, dbdata13, dbdata21, dbersbdata00, dbersbdata01, dbsbdata00 > ABORTED. > 13:21:53 Aborted by client. > 13:22:19 Checkpoint Completed: duration was 0 seconds. > 13:22:19 Tue Jun 17 - loguniq 91344, logpos 0xa46e018, timestamp: > 0x3e945f51 Interval: 334495 > 13:22:19 Maximum server connections 65 > 13:22:20 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, Plog used 28, Llog used 410 > > 3) af file > > [informix@stephanie tmp]$ more af.dc0640ee > 13:21:50 mt_aio_wait(0xc63b3608) > 13:21:50 chunkio(101, 0x0xc6356c00, 0x2, 7, 0x0xc35fe9d8, 33, 1) errno 2 > AIOREQ *a > 00000000c70c68c0: 00000000 00000000 0a000000 00000000 ........ ........ > 00000000c70c68d0: 8adc0000 00000000 00000000 00000000 ........ ........ > 00000000c70c68e0: 00000000 00000000 00000000 00000000 ........ ........ > 00000000c70c68f0: f8383bc6 00000000 18333bc6 00000000 .8;..... .3;..... > 00000000c70c6900: 00000000 00000000 00000000 00000000 ........ ........ > 00000000c70c6910: c03b0000 41494f72 65710000 00000000 .;..AIOr eq...... > 00000000c70c6920: 00000000 00000000 d1070000 00000000 ........ ........ > 00000000c70c6930: 00000000 00000000 04000000 01040000 ........ ........ > 00000000c70c6940: 0b000000 02000000 01000000 00000000 ........ ........ > 00000000c70c6950: 01000000 00000000 e0363bc6 00000000 ........ .6;..... > 00000000c70c6960: 00000000 00000000 01000000 09000000 ........ ........ > 00000000c70c6970: 01000000 00000000 00000000 00000000 ........ ........ > 00000000c70c6980: 00000000 00000000 00000000 00000000 ........ ........ > 00000000c70c6990: 00000000 00000000 ........ > 13:21:50 > 13:21:50 IBM Informix Dynamic Server Version 11.50.FC8 Software Serial > Number AAA#B000000 > 13:21:50 Assert Warning: ISAM error: An error has occurred during archive > back up. > > 13:21:50 Who: Session(49353, informix@stephanie, 31263, 0xc23980f0) > > Thread(55326, arcbackup1, c2367f30, 7) > > File: rsarcutl.c Line: 124 > 13:21:50 Action: arc_pages()/bfphysio() > 13:21:50 Raw hex dump of stack located in /tmp/af.dc0640ee.rawstk > 13:21:50 Stack for thread: 55326 arcbackup1 > base: 0x00000000c70b6000 > len: 69632 > > pc: 0x00000000010c91d7 > tos: 0x00000000c70c5ef0 > state: running > > vp: 7 > 0x00000000010c91d7 (oninit) afstack > 0x00000000010cb12c (oninit) afhandler > 0x00000000010cb9a7 (oninit) afwarn_interface > 0x0000000000c7d7a6 (oninit) oops_error > 0x0000000000c8943a (oninit) arc_pages > 0x0000000000c8959f (oninit) arc_dbspace > 0x0000000000c8a890 (oninit) arc_scanner > 0x000000000109a9bb (oninit) startup > > 13:21:52 See Also: /tmp/af.dc0640ee > --------------------------------- > Begin System Alarm Program Output > --------------------------------- > Assertion Failure Type: Warning > Host Name: stephanie > Database Server Name: nsofpal_tcp > Time of failure: Tue Jun 17 13:21:52 UTC 2014 > AF file: /tmp/af.dc0640ee > Shared memory file: None > System Blocking: OFF > > ------------------------------- > End System Alarm Program Output > ------------------------------- > 13:21:52 sh /usr/informix/etc/evidence.sh 1 0 /tmp/af.dc0640ee 49353 > 0xc2367f30 55326 0xc54c53a8 1025 0 0 0 0 > 13:21:52 > ------------------ End of assertion failure 0 ----------------- > > --001a113a5fd0fbac6504fc08273b > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a11c3c44214017a04fc085c25

HI, Keith, I agree. The instance is up and online doing fine. But.... The following is the result of oncehck and onstat -d Thanks Frank Validating indexes for noaa:informix.ds_head... Index ds_head_ipk Index fragment partition dbdata05 in DBspace dbdata05 Could not check rowids and perform data<->index check ISAM error: Primary and Mirror chunks are bad Index ds_head_idx0 Index fragment partition dbdw06 in DBspace dbdw06 Could not check rowids and perform data<->index check ISAM error: Primary and Mirror chunks are bad Index w_ds_head_idx2 Index fragment partition dbdata06 in DBspace dbdata06 Could not bfget pagenum 0xec40, iserrno 0 Not owner [informix@stephanie ontape]$ onstat -d IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 22:54:56 -- 2870104 KbytesDbspaces address number flags fchunk nchunks pgsize flags owner name c20ae028 1 0x40001 1 1 2048 N B informix rootdbs c35dcb18 2 0x40001 2 3 2048 N B informix dblog1 c35dccb0 3 0x40001 3 3 2048 N B informix dblog2 c35dce48 4 0x42001 8 1 2048 N TB informix dbtemp1 c35de028 5 0x42001 9 1 2048 N TB informix dbtemp2 c35de1c0 6 0x42001 10 1 2048 N TB informix dbtemp3 c35de358 7 0x40001 11 2 2048 N B informix dberdata00 c35de4f0 8 0x48001 13 2 2048 N SB informix dbersbdata00 c35de688 9 0x48001 15 2 2048 N SB informix dbersbdata01 c35de820 10 0x40001 18 3 8192 N B informix dbdata09_1 c35de9b8 11 0x40001 72 6 2048 N B informix dbdata09_2 c35deb50 12 0x40001 19 2 2048 N B informix dbdata00 c35dece8 13 0x40001 21 3 2048 N B informix dbdata01 c35dee80 14 0x40001 23 2 2048 N B informix dbdata02 c35e0028 15 0x40001 25 2 2048 N B informix dbdata03 c35e01c0 16 0x40001 27 2 2048 N B informix dbdata04 c35e0358 17 0x40001 29 2 2048 N B informix dbdata05 c35e04f0 18 0x40001 31 4 2048 N B informix dbdata06 c35e0688 19 0x40001 33 2 2048 N B informix dbdata07 c35e0820 20 0x40001 35 3 2048 N B informix dbdata08 c35e09b8 21 0x40001 37 5 2048 N B informix dbdata09 c35e0b50 22 0x40001 39 3 2048 N B informix dbdata10 c35e0ce8 23 0x40001 41 2 2048 N B informix dbdata11 c35e0e80 24 0x40001 43 2 2048 N B informix dbdw00 c35e1028 25 0x40001 45 7 2048 N B informix dbdw01 c35e11c0 26 0x40001 47 3 2048 N B informix dbdw02 c35e1358 27 0x40001 49 2 2048 N B informix dbdw03 c35e14f0 28 0x40001 51 3 2048 N B informix dbdw04 c35e1688 29 0x40001 53 2 8192 N B informix dbdw05 c35e1820 30 0x40001 55 2 8192 N B informix dbdw06 c35e19b8 31 0x40001 57 2 8192 N B informix dbdw07 c35e1b50 32 0x40001 59 2 8192 N B informix dbdw08 c35e1ce8 33 0x40001 90 5 8192 N B informix dbdata20 c35e1e80 34 0x40001 63 1 2048 N B informix dbphylog c35e3028 35 0x40001 64 2 2048 N B informix dbdata12 c35e31c0 36 0x40001 66 6 2048 N B informix dbdata13 c35e3358 37 0x40001 93 4 8192 N B informix dbdata21 c35e34f0 38 0x48001 97 2 2048 N SB informix dbsbdata00 38 active, 2047 maximum Chunks address chunk/dbs offset size free bpages flags pathname c20ae1c0 1 1 0 600000 585195 PO-B- /usr/informix/dev-links/nsofops/rootdbs_ops c35e3688 2 2 0 5000000 199947 PO-B- /usr/informix/dev-links/nsofops/dblog1_ops c35e3878 3 3 0 5000000 199947 PO-B- /usr/informix/dev-links/nsofops/dblog2_ops c35e3a68 4 2 0 5000000 199997 PO-B- /usr/informix/dev-links/nsofops/dblog1a_ops c35e3c58 5 3 0 5000000 199997 PO-B- /usr/informix/dev-links/nsofops/dblog2a_ops c35e5028 6 2 0 5000000 199997 PO-B- /usr/informix/dev-links/nsofops/dblog1b_ops c35e5218 7 3 0 5000000 199997 PO-B- /usr/informix/dev-links/nsofops/dblog2b_ops c35e5408 8 4 0 3000000 2998596 PO-B- /usr/informix/dev-links/nsofops/dbtemp1a_ops c35e55f8 9 5 0 3000000 2998596 PO-B- /usr/informix/dev-links/nsofops/dbtemp2a_ops c35e57e8 10 6 0 3000000 2998596 PO-B- /usr/informix/dev-links/nsofops/dbtemp3a_ops c35e59d8 11 7 0 5000000 4999947 PO-B- /usr/informix/dev-links/nsofops/dberdata00_ops c35e5bc8 12 7 0 5000000 4999997 PO-B- /usr/informix/dev-links/nsofops/dberdata00a_ops c35e5db8 13 8 0 5000000 4663405 4663405 POSB- /usr/informix/dev-links/nsofops/dbersbdata00_ops Metadata 336542 27162 336542 c35e6028 14 8 0 5000000 4663451 4663451 POSB- /usr/informix/dev-links/nsofops/dbersbdata00a_ops Metadata 336546 199385 336546 c35e6218 15 9 0 5000000 4663405 4663405 POSB- /usr/informix/dev-links/nsofops/dbersbdata01_ops Metadata 336542 27162 336542 c35e6408 16 9 0 5000000 4663451 4663451 POSB- /usr/informix/dev-links/nsofops/dbersbdata01a_ops Metadata 336546 199385 336546 c35e65f8 17 21 0 5000000 4999997 PO-B- /usr/informix/dev-links/nsofops/dbdata09c_ops c35e67e8 18 10 0 1250000 1227411 PO-B- /usr/informix/dev-links/nsofops/dbdata09d_ops c35e69d8 19 12 0 5000000 2001191 PO-B- /usr/informix/dev-links/nsofops/dbdata00_ops c35e6bc8 20 12 0 5000000 4920245 PO-B- /usr/informix/dev-links/nsofops/dbdata00a_ops c35e6db8 21 13 0 5000000 1116246 PO-B- /usr/informix/dev-links/nsofops/dbdata01_ops c35e7028 22 13 0 5000000 4315797 PO-B- /usr/informix/dev-links/nsofops/dbdata01a_ops c35e7218 23 14 0 5000000 1062985 PO-B- /usr/informix/dev-links/nsofops/dbdata02_ops c35e7408 24 14 0 5000000 4279997 PO-B- /usr/informix/dev-links/nsofops/dbdata02a_ops c35e75f8 25 15 0 5000000 4154244 PO-B- /usr/informix/dev-links/nsofops/dbdata03_ops c35e77e8 26 15 0 5000000 4999997 PO-B- /usr/informix/dev-links/nsofops/dbdata03a_ops c35e79d8 27 16 0 5000000 4263203 PO-B- /usr/informix/dev-links/nsofops/dbdata04_ops c35e7bc8 28 16 0 5000000 4999997 PO-B- /usr/informix/dev-links/nsofops/dbdata04a_ops c35e7db8 29 17 0 5000000 4364412 PO-B- /usr/informix/dev-links/nsofops/dbdata05_ops c35e8028 30 17 0 5000000 4999997 PO-B- /usr/informix/dev-links/nsofops/dbdata05a_ops c35e8218 31 18 0 5000000 4982177 PO-B- /usr/informix/dev-links/nsofops/dbdata06_ops c35e8408 32 18 0 5000000 2076367 PO-B- /usr/informix/dev-links/nsofops/dbdata06a_ops c35e85f8 33 19 0 5000000 4999667 PO-B- /usr/informix/dev-links/nsofops/dbdata07_ops c35e87e8 34 19 0 5000000 4999997 PO-B- /usr/informix/dev-links/nsofops/dbdata07a_ops c35e89d8 35 20 0 5000000 762167 PO-B- /usr/informix/dev-links/nsofops/dbdata08_ops c35e8bc8 36 20 0 5000000 599509 PO-B- /usr/informix/dev-links/nsofops/dbdata08a_ops c35e8db8 37 21 0 5000000 4873082 PO-B- /usr/informix/dev-links/nsofops/dbdata09_ops c35e9028 38 21 0 5000000 4977261 PO-B- /usr/informix/dev-links/nsofops/dbdata09a_ops c35e9218 39 22 0 5000000 1258805 PO-B- /usr/informix/dev-links/nsofops/dbdata10_ops c35e9408 40 22 0 5000000 2706493 PO-B- /usr/informix/dev-links/nsofops/dbdata10a_ops c35e95f8 41 23 0 5000000 4262655 PO-B- /usr/informix/dev-links/nsofops/dbdata11_ops c35e97e8 42 23 0 5000000 4999997 PO-B- /usr/informix/dev-links/nsofops/dbdata11a_ops c35e99d8 43 24 0 5000000 1719340 PO-B- /usr/informix/dev-links/nsofops/dbdw00_ops c35e9bc8 44 24 0 5000000 1795645 PO-B- /usr/informix/dev-links/nsofops/dbdw00a_ops c35e9db8 45 25 0 5000000 513773 PO-B- /usr/informix/dev-links/nsofops/dbdw01_ops c35ea028

Chunk #99 isn't part of either of dbdata05 or dbdw06. Looks like an entire SAN array has gone south! Not RAID5 I hope! B^( Check the SAN logs for errors and hope it's a soft problem. Meanwhile, get out the last good backup and try to unload whatever tables you can for ultimate backup in case the archives or log backups are not usable. Remember "good DBAs are paranoid DBAs!" Art Art S. Kagel, Principal Consultant ASK Database Management Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Tue, Jun 17, 2014 at 9:56 AM, FRANK <yunyaoqu@gmail.com> wrote: > HI, Keith, > I agree. > The instance is up and online doing fine. But.... > > The following is the result of oncehck and onstat -d > > Thanks > Frank > > Validating indexes for noaa:informix.ds_head... > > Index ds_head_ipk > > Index fragment partition dbdata05 in DBspace dbdata05 > Could not check rowids and perform data<->index check > ISAM error: Primary and Mirror chunks are bad> > Index ds_head_idx0 > > Index fragment partition dbdw06 in DBspace dbdw06 > Could not check rowids and perform data<->index check > ISAM error: Primary and Mirror chunks are bad> > Index w_ds_head_idx2 > > Index fragment partition dbdata06 in DBspace dbdata06 > Could not bfget pagenum 0xec40, iserrno 0 > Not owner > > [informix@stephanie ontape]$ onstat -d > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 22:54:56 > -- 2870104 Kbytes> Dbspaces > address number flags fchunk nchunks pgsize flags > owner name > c20ae028 1 0x40001 1 1 2048 N B > informix rootdbs > c35dcb18 2 0x40001 2 3 2048 N B > informix dblog1 > c35dccb0 3 0x40001 3 3 2048 N B > informix dblog2 > c35dce48 4 0x42001 8 1 2048 N TB > informix dbtemp1 > c35de028 5 0x42001 9 1 2048 N TB > informix dbtemp2 > c35de1c0 6 0x42001 10 1 2048 N TB > informix dbtemp3 > c35de358 7 0x40001 11 2 2048 N B > informix dberdata00 > c35de4f0 8 0x48001 13 2 2048 N SB > informix dbersbdata00 > c35de688 9 0x48001 15 2 2048 N SB > informix dbersbdata01 > c35de820 10 0x40001 18 3 8192 N B > informix dbdata09_1 > c35de9b8 11 0x40001 72 6 2048 N B > informix dbdata09_2 > c35deb50 12 0x40001 19 2 2048 N B > informix dbdata00 > c35dece8 13 0x40001 21 3 2048 N B > informix dbdata01 > c35dee80 14 0x40001 23 2 2048 N B > informix dbdata02 > c35e0028 15 0x40001 25 2 2048 N B > informix dbdata03 > c35e01c0 16 0x40001 27 2 2048 N B > informix dbdata04 > c35e0358 17 0x40001 29 2 2048 N B > informix dbdata05 > c35e04f0 18 0x40001 31 4 2048 N B > informix dbdata06 > c35e0688 19 0x40001 33 2 2048 N B > informix dbdata07 > c35e0820 20 0x40001 35 3 2048 N B > informix dbdata08 > c35e09b8 21 0x40001 37 5 2048 N B > informix dbdata09 > c35e0b50 22 0x40001 39 3 2048 N B > informix dbdata10 > c35e0ce8 23 0x40001 41 2 2048 N B > informix dbdata11 > c35e0e80 24 0x40001 43 2 2048 N B > informix dbdw00 > c35e1028 25 0x40001 45 7 2048 N B > informix dbdw01 > c35e11c0 26 0x40001 47 3 2048 N B > informix dbdw02 > c35e1358 27 0x40001 49 2 2048 N B > informix dbdw03 > c35e14f0 28 0x40001 51 3 2048 N B > informix dbdw04 > c35e1688 29 0x40001 53 2 8192 N B > informix dbdw05 > c35e1820 30 0x40001 55 2 8192 N B > informix dbdw06 > c35e19b8 31 0x40001 57 2 8192 N B > informix dbdw07 > c35e1b50 32 0x40001 59 2 8192 N B > informix dbdw08 > c35e1ce8 33 0x40001 90 5 8192 N B > informix dbdata20 > c35e1e80 34 0x40001 63 1 2048 N B > informix dbphylog > c35e3028 35 0x40001 64 2 2048 N B > informix dbdata12 > c35e31c0 36 0x40001 66 6 2048 N B > informix dbdata13 > c35e3358 37 0x40001 93 4 8192 N B > informix dbdata21 > c35e34f0 38 0x48001 97 2 2048 N SB > informix dbsbdata00 > 38 active, 2047 maximum > Chunks > address chunk/dbs offset size free bpages > flags pathname > c20ae1c0 1 1 0 600000 585195 > PO-B- /usr/informix/dev-links/nsofops/rootdbs_ops > c35e3688 2 2 0 5000000 199947 > PO-B- /usr/informix/dev-links/nsofops/dblog1_ops > c35e3878 3 3 0 5000000 199947 > PO-B- /usr/informix/dev-links/nsofops/dblog2_ops > c35e3a68 4 2 0 5000000 199997 > PO-B- /usr/informix/dev-links/nsofops/dblog1a_ops > c35e3c58 5 3 0 5000000 199997 > PO-B- /usr/informix/dev-links/nsofops/dblog2a_ops > c35e5028 6 2 0 5000000 199997 > PO-B- /usr/informix/dev-links/nsofops/dblog1b_ops > c35e5218 7 3 0 5000000 199997 > PO-B- /usr/informix/dev-links/nsofops/dblog2b_ops > c35e5408 8 4 0 3000000 2998596 > PO-B- /usr/informix/dev-links/nsofops/dbtemp1a_ops > c35e55f8 9 5 0 3000000 2998596 > PO-B- /usr/informix/dev-links/nsofops/dbtemp2a_ops > c35e57e8 10 6 0 3000000 2998596 > PO-B- /usr/informix/dev-links/nsofops/dbtemp3a_ops > c35e59d8 11 7 0 5000000 4999947 > PO-B- /usr/informix/dev-links/nsofops/dberdata00_ops > c35e5bc8 12 7 0 5000000 4999997 > PO-B- /usr/informix/dev-links/nsofops/dberdata00a_ops > c35e5db8 13 8 0 5000000 4663405 4663405 > POSB- /usr/informix/dev-links/nsofops/dbersbdata00_ops > > Metadata 336542 27162 336542 > c35e6028 14 8 0 5000000 4663451 4663451 > POSB- /usr/informix/dev-links/nsofops/dbersbdata00a_ops > > Metadata 336546 199385 336546 > c35e6218 15 9 0 5000000 4663405 4663405 > POSB- /usr/informix/dev-links/nsofops/dbersbdata01_ops > > Metadata 336542 27162 336542 > c35e6408 16 9 0 5000000 4663451 4663451 > POSB- /usr/informix/dev-links/nsofops/dbersbdata01a_ops > > Metadata 336546 199385 336546 > c35e65f8 17 21 0 5000000 4999997 > PO-B- /usr/informix/dev-links/nsofops/dbdata09c_ops > c35e67e8 18 10 0 1250000 1227411 > PO-B- /usr/informix/dev-links/nsofops/dbdata09d_ops > c35e69d8 19 12 0 5000000 2001191 > PO-B- /usr/informix/dev-links/nsofops/dbdata00_ops > c35e6bc8 20 12 0 5000000 4920245 > PO-B- /usr/informix/dev-links/nsofops/dbdata00a_ops > c35e6db8 21 13 0 5000000 1116246 > PO-B- /usr/informix/dev-links/nsofops/dbdata01_ops > c35e7028 22 13 0 5000000 4315797 > PO-B- /usr/informix/dev-links/nsofops/dbdata01a_ops > c35e7218 23 14 0 5000000 1062985 > PO-B- /usr/informix/dev-links/nsofops/dbdata02_ops > c35e7408 24 14 0 5000000 4279997 > PO-B- /usr/informix/dev-links/nsofops/dbdata02a_ops > c35e75f8 25 15 0 5000000 4154244 > PO-B- /usr/informix/dev-links/nsofops/dbdata03_ops > c35e77e8 26 15 0 5000000 4999997 > PO-B- /usr/informix/dev-links/nsofops/dbdata03a_ops > c35e79d8 27 16 0 5000000 4263203 > PO-B- /usr/informix/dev-links/nsofops/dbdata04_ops > c35e7bc8 28 16 0 5000000 4999997 > PO-B- /usr/informix/dev-links/nsofops/dbdata04a_ops > c35e7db8 29 17 0 5000000 4364412 > PO-B- /usr/informix/dev-links/nsofops/dbdata05_ops > c35e8028 30 17 0 5000000 4999997 > PO-B- /usr/informix/dev-links/nsofops/dbdata05a_ops > c35e8218

Thank Art! Finally I could managed to get rid of (drop) the corrupted indexes from that spot and completed an new backup successfully ! Thank you all! Frank On Tue, Jun 17, 2014 at 10:05 AM, Art Kagel <art.kagel@gmail.com> wrote: > Chunk #99 isn't part of either of dbdata05 or dbdw06. Looks like an entire > SAN array has gone south! > > Not RAID5 I hope! B^( > > Check the SAN logs for errors and hope it's a soft problem. Meanwhile, get > out the last good backup and try to unload whatever tables you can for > ultimate backup in case the archives or log backups are not usable. > Remember "good DBAs are paranoid DBAs!" > > Art > > Art S. Kagel, Principal Consultant > ASK Database Management > > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and do not reflect on the IIUG, nor any other organization with which I am > associated either explicitly, implicitly, or by inference. Neither do > those opinions reflect those of other individuals affiliated with any > entity with which I am affiliated nor those of the entities themselves. > > On Tue, Jun 17, 2014 at 9:56 AM, FRANK <yunyaoqu@gmail.com> wrote: > > > HI, Keith, > > I agree. > > The instance is up and online doing fine. But.... > > > > The following is the result of oncehck and onstat -d > > > > Thanks > > Frank > > > > Validating indexes for noaa:informix.ds_head... > > > > Index ds_head_ipk > > > > Index fragment partition dbdata05 in DBspace dbdata05 > > Could not check rowids and perform data<->index check > > ISAM error: Primary and Mirror chunks are bad> > > > Index ds_head_idx0 > > > > Index fragment partition dbdw06 in DBspace dbdw06 > > Could not check rowids and perform data<->index check > > ISAM error: Primary and Mirror chunks are bad> > > > Index w_ds_head_idx2 > > > > Index fragment partition dbdata06 in DBspace dbdata06 > > Could not bfget pagenum 0xec40, iserrno 0 > > Not owner > > > > [informix@stephanie ontape]$ onstat -d > > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 22:54:56 > > -- 2870104 Kbytes> > Dbspaces > > address number flags fchunk nchunks pgsize flags > > owner name > > c20ae028 1 0x40001 1 1 2048 N B > > informix rootdbs > > c35dcb18 2 0x40001 2 3 2048 N B > > informix dblog1 > > c35dccb0 3 0x40001 3 3 2048 N B > > informix dblog2 > > c35dce48 4 0x42001 8 1 2048 N TB > > informix dbtemp1 > > c35de028 5 0x42001 9 1 2048 N TB > > informix dbtemp2 > > c35de1c0 6 0x42001 10 1 2048 N TB > > informix dbtemp3 > > c35de358 7 0x40001 11 2 2048 N B > > informix dberdata00 > > c35de4f0 8 0x48001 13 2 2048 N SB > > informix dbersbdata00 > > c35de688 9 0x48001 15 2 2048 N SB > > informix dbersbdata01 > > c35de820 10 0x40001 18 3 8192 N B > > informix dbdata09_1 > > c35de9b8 11 0x40001 72 6 2048 N B > > informix dbdata09_2 > > c35deb50 12 0x40001 19 2 2048 N B > > informix dbdata00 > > c35dece8 13 0x40001 21 3 2048 N B > > informix dbdata01 > > c35dee80 14 0x40001 23 2 2048 N B > > informix dbdata02 > > c35e0028 15 0x40001 25 2 2048 N B > > informix dbdata03 > > c35e01c0 16 0x40001 27 2 2048 N B > > informix dbdata04 > > c35e0358 17 0x40001 29 2 2048 N B > > informix dbdata05 > > c35e04f0 18 0x40001 31 4 2048 N B > > informix dbdata06 > > c35e0688 19 0x40001 33 2 2048 N B > > informix dbdata07 > > c35e0820 20 0x40001 35 3 2048 N B > > informix dbdata08 > > c35e09b8 21 0x40001 37 5 2048 N B > > informix dbdata09 > > c35e0b50 22 0x40001 39 3 2048 N B > > informix dbdata10 > > c35e0ce8 23 0x40001 41 2 2048 N B > > informix dbdata11 > > c35e0e80 24 0x40001 43 2 2048 N B > > informix dbdw00 > > c35e1028 25 0x40001 45 7 2048 N B > > informix dbdw01 > > c35e11c0 26 0x40001 47 3 2048 N B > > informix dbdw02 > > c35e1358 27 0x40001 49 2 2048 N B > > informix dbdw03 > > c35e14f0 28 0x40001 51 3 2048 N B > > informix dbdw04 > > c35e1688 29 0x40001 53 2 8192 N B > > informix dbdw05 > > c35e1820 30 0x40001 55 2 8192 N B > > informix dbdw06 > > c35e19b8 31 0x40001 57 2 8192 N B > > informix dbdw07 > > c35e1b50 32 0x40001 59 2 8192 N B > > informix dbdw08 > > c35e1ce8 33 0x40001 90 5 8192 N B > > informix dbdata20 > > c35e1e80 34 0x40001 63 1 2048 N B > > informix dbphylog > > c35e3028 35 0x40001 64 2 2048 N B > > informix dbdata12 > > c35e31c0 36 0x40001 66 6 2048 N B > > informix dbdata13 > > c35e3358 37 0x40001 93 4 8192 N B > > informix dbdata21 > > c35e34f0 38 0x48001 97 2 2048 N SB > > informix dbsbdata00 > > 38 active, 2047 maximum > > Chunks > > address chunk/dbs offset size free bpages > > flags pathname > > c20ae1c0 1 1 0 600000 585195 > > PO-B- /usr/informix/dev-links/nsofops/rootdbs_ops > > c35e3688 2 2 0 5000000 199947 > > PO-B- /usr/informix/dev-links/nsofops/dblog1_ops > > c35e3878 3 3 0 5000000 199947 > > PO-B- /usr/informix/dev-links/nsofops/dblog2_ops > > c35e3a68 4 2 0 5000000 199997 > > PO-B- /usr/informix/dev-links/nsofops/dblog1a_ops > > c35e3c58 5 3 0 5000000 199997 > > PO-B- /usr/informix/dev-links/nsofops/dblog2a_ops > > c35e5028 6 2 0 5000000 199997 > > PO-B- /usr/informix/dev-links/nsofops/dblog1b_ops > > c35e5218 7 3 0 5000000 199997 > > PO-B- /usr/informix/dev-links/nsofops/dblog2b_ops > > c35e5408 8 4 0 3000000 2998596 > > PO-B- /usr/informix/dev-links/nsofops/dbtemp1a_ops > > c35e55f8 9 5 0 3000000 2998596 > > PO-B- /usr/informix/dev-links/nsofops/dbtemp2a_ops > > c35e57e8 10 6 0 3000000 2998596 > > PO-B- /usr/informix/dev-links/nsofops/dbtemp3a_ops > > c35e59d8 11 7 0 5000000 4999947 > > PO-B- /usr/informix/dev-links/nsofops/dberdata00_ops > > c35e5bc8 12 7 0 5000000 4999997 > > PO-B- /usr/informix/dev-links/nsofops/dberdata00a_ops > > c35e5db8 13 8 0 5000000 4663405 4663405 > > POSB- /usr/informix/dev-links/nsofops/dbersbdata00_ops > > > > Metadata 336542 27162 336542 > > c35e6028 14 8 0 5000000 4663451 4663451 > > POSB- /usr/informix/dev-links/nsofops/dbersbdata00a_ops > > > > Metadata 336546 199385 336546 > > c35e6218 15 9 0 5000000 4663405 4663405 > > POSB- /usr/informix/dev-links/nsofops/dbersbdata01_ops > > > > Metadata 336542 27162 336542 > > c35e6408 16 9 0 5000000 4663451 4663451 > > POSB- /usr/informix/dev-links/nsofops/dbersbdata01a_ops > > > > Metadata 336546 199385 336546 > > c35e65f8 17 21 0 5000000 4999997 > > PO-B- /usr/informix/dev-links/nsofops/dbdata09c_ops > > c35e67e8 18 10 0 1250000 1227411 > > PO-B- /usr/informix/dev-links/nsofops/dbdata09d_ops > > c35e69d8 19 12 0 5000000 2001191 > > PO-B- /usr/informix/dev-links/nsofops/dbdata00_ops > > c35e6bc8 20 12 0 5000000 4920245 > > PO-B- /usr/informix/dev-links/nsofops/dbdata00a_ops > > c35e6db8 21 13 0 5000000 1116246 > > PO-B- /usr/informix/dev-links/nsofops/dbdata01_ops > > c35e7028 22 13 0 5000000 4315797 > > PO-B- /usr/informix/dev-links/nsofops/dbdata01a_ops > > c35e7218 23 14 0 5000000 1062985 > > PO-B- /usr/informix/dev-links/nsofops/dbdata02_ops > > c35e7408 24 14 0 5000000 4279997@

Great! So, what was the final verdict on the cause? The solution? Share! Art Art S. Kagel, Principal Consultant ASK Database Management Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Tue, Jun 17, 2014 at 4:10 PM, FRANK <yunyaoqu@gmail.com> wrote: > Thank Art! > > Finally I could managed to get rid of (drop) the corrupted indexes from > that spot and completed an new backup successfully ! > > Thank you all! > Frank > > On Tue, Jun 17, 2014 at 10:05 AM, Art Kagel <art.kagel@gmail.com> wrote: > > > Chunk #99 isn't part of either of dbdata05 or dbdw06. Looks like an > entire > > SAN array has gone south! > > > > Not RAID5 I hope! B^( > > > > Check the SAN logs for errors and hope it's a soft problem. Meanwhile, > get > > out the last good backup and try to unload whatever tables you can for > > ultimate backup in case the archives or log backups are not usable. > > Remember "good DBAs are paranoid DBAs!" > > > > Art > > > > Art S. Kagel, Principal Consultant > > ASK Database Management > > > > Blog: http://informix-myview.blogspot.com/ > > > > Disclaimer: Please keep in mind that my own opinions are my own opinions > > and do not reflect on the IIUG, nor any other organization with which I > am > > associated either explicitly, implicitly, or by inference. Neither do > > those opinions reflect those of other individuals affiliated with any > > entity with which I am affiliated nor those of the entities themselves. > > > > On Tue, Jun 17, 2014 at 9:56 AM, FRANK <yunyaoqu@gmail.com> wrote: > > > > > HI, Keith, > > > I agree. > > > The instance is up and online doing fine. But.... > > > > > > The following is the result of oncehck and onstat -d > > > > > > Thanks > > > Frank > > > > > > Validating indexes for noaa:informix.ds_head... > > > > > > Index ds_head_ipk > > > > > > Index fragment partition dbdata05 in DBspace dbdata05 > > > Could not check rowids and perform data<->index check > > > ISAM error: Primary and Mirror chunks are bad> > > > > > Index ds_head_idx0 > > > > > > Index fragment partition dbdw06 in DBspace dbdw06 > > > Could not check rowids and perform data<->index check > > > ISAM error: Primary and Mirror chunks are bad> > > > > > Index w_ds_head_idx2 > > > > > > Index fragment partition dbdata06 in DBspace dbdata06 > > > Could not bfget pagenum 0xec40, iserrno 0 > > > Not owner > > > > > > [informix@stephanie ontape]$ onstat -d > > > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 22:54:56 > > > -- 2870104 Kbytes> > > Dbspaces > > > address number flags fchunk nchunks pgsize flags > > > owner name > > > c20ae028 1 0x40001 1 1 2048 N B > > > informix rootdbs > > > c35dcb18 2 0x40001 2 3 2048 N B > > > informix dblog1 > > > c35dccb0 3 0x40001 3 3 2048 N B > > > informix dblog2 > > > c35dce48 4 0x42001 8 1 2048 N TB > > > informix dbtemp1 > > > c35de028 5 0x42001 9 1 2048 N TB > > > informix dbtemp2 > > > c35de1c0 6 0x42001 10 1 2048 N TB > > > informix dbtemp3 > > > c35de358 7 0x40001 11 2 2048 N B > > > informix dberdata00 > > > c35de4f0 8 0x48001 13 2 2048 N SB > > > informix dbersbdata00 > > > c35de688 9 0x48001 15 2 2048 N SB > > > informix dbersbdata01 > > > c35de820 10 0x40001 18 3 8192 N B > > > informix dbdata09_1 > > > c35de9b8 11 0x40001 72 6 2048 N B > > > informix dbdata09_2 > > > c35deb50 12 0x40001 19 2 2048 N B > > > informix dbdata00 > > > c35dece8 13 0x40001 21 3 2048 N B > > > informix dbdata01 > > > c35dee80 14 0x40001 23 2 2048 N B > > > informix dbdata02 > > > c35e0028 15 0x40001 25 2 2048 N B > > > informix dbdata03 > > > c35e01c0 16 0x40001 27 2 2048 N B > > > informix dbdata04 > > > c35e0358 17 0x40001 29 2 2048 N B > > > informix dbdata05 > > > c35e04f0 18 0x40001 31 4 2048 N B > > > informix dbdata06 > > > c35e0688 19 0x40001 33 2 2048 N B > > > informix dbdata07 > > > c35e0820 20 0x40001 35 3 2048 N B > > > informix dbdata08 > > > c35e09b8 21 0x40001 37 5 2048 N B > > > informix dbdata09 > > > c35e0b50 22 0x40001 39 3 2048 N B > > > informix dbdata10 > > > c35e0ce8 23 0x40001 41 2 2048 N B > > > informix dbdata11 > > > c35e0e80 24 0x40001 43 2 2048 N B > > > informix dbdw00 > > > c35e1028 25 0x40001 45 7 2048 N B > > > informix dbdw01 > > > c35e11c0 26 0x40001 47 3 2048 N B > > > informix dbdw02 > > > c35e1358 27 0x40001 49 2 2048 N B > > > informix dbdw03 > > > c35e14f0 28 0x40001 51 3 2048 N B > > > informix dbdw04 > > > c35e1688 29 0x40001 53 2 8192 N B > > > informix dbdw05 > > > c35e1820 30 0x40001 55 2 8192 N B > > > informix dbdw06 > > > c35e19b8 31 0x40001 57 2 8192 N B > > > informix dbdw07 > > > c35e1b50 32 0x40001 59 2 8192 N B > > > informix dbdw08 > > > c35e1ce8 33 0x40001 90 5 8192 N B > > > informix dbdata20 > > > c35e1e80 34 0x40001 63 1 2048 N B > > > informix dbphylog > > > c35e3028 35 0x40001 64 2 2048 N B > > > informix dbdata12 > > > c35e31c0 36 0x40001 66 6 2048 N B > > > informix dbdata13 > > > c35e3358 37 0x40001 93 4 8192 N B > > > informix dbdata21 > > > c35e34f0 38 0x48001 97 2 2048 N SB > > > informix dbsbdata00 > > > 38 active, 2047 maximum > > > Chunks > > > address chunk/dbs offset size free bpages > > > flags pathname > > > c20ae1c0 1 1 0 600000 585195 > > > PO-B- /usr/informix/dev-links/nsofops/rootdbs_ops > > > c35e3688 2 2 0 5000000 199947 > > > PO-B- /usr/informix/dev-links/nsofops/dblog1_ops > > > c35e3878 3 3 0 5000000 199947 > > > PO-B- /usr/informix/dev-links/nsofops/dblog2_ops > > > c35e3a68 4 2 0 5000000 199997 > > > PO-B- /usr/informix/dev-links/nsofops/dblog1a_ops > > > c35e3c58 5 3 0 5000000 199997 > > > PO-B- /usr/informix/dev-links/nsofops/dblog2a_ops > > > c35e5028 6 2 0 5000000 199997 > > > PO-B- /usr/informix/dev-links/nsofops/dblog1b_ops > > > c35e5218 7 3 0 5000000 199997 > > > PO-B- /usr/informix/dev-links/nsofops/dblog2b_ops > > > c35e5408 8 4 0 3000000 2998596 > > > PO-B- /usr/informix/dev-links/nsofops/dbtemp1a_ops > > > c35e55f8 9 5 0 3000000 2998596 > > > PO-B- /usr/informix/dev-links/nsofops/dbtemp2a_ops > > > c35e57e8 10 6 0 3000000 2998596 > > > PO-B- /usr/informix/dev-links/nsofops/dbtemp3a_ops > > > c35e59d8 11 7 0 5000000 4999947 > > > PO-B- /usr/informix/dev-links/nsofops/dberdata00_ops > > > c35e5bc8 12 7 0 5000000 4999997 > > > PO-B- /usr/informix/dev-links/nsofops/dberdata00a_ops > > > c35e5db8 13 8 0 5000000 4663405 4663405 > > > POSB- /usr/informix/dev-links/nsofops/dbersbdata00_ops > > > > > > Metadata 336542 27162 336542 > > > c35e6028 14 8 0 5000000 4663451 4663451 > > > POSB- /usr/informix/dev-links/nsofops/dbersbdata00a_ops > > > > > > Metadata 336546 199385 336546@@NL@