Re: help for inconsistencies index problem!!

On Sat, 10 May 2003 23:50:19 -0400, studyboy wrote: > Hi all. > > This is a urgent case.. > > > IHAC who has a problem currpted index. > > Customer's env. > > HP_UX11.0''''''IDS7.31UD4X3 > > > I want to know if this possible a bug? Possible but not any that I've heard of. > Or other possible cause for inconsistencies index problem? There are several possible causes: - Undetected disk corruption (RAID5's creaping crud problem) - System crash with an Unlogged or Buffered Log database. - Incorrect system shutdown scripts that either do not shutdown the IDS instance or do no wait for it to actually complete its shutdown before exiting. Again damage is most likely with Unlogged or Buffered Log DBs. - Similarly if a disk subsystem has to have hardware cache flushed during shutdown and this was not done or completed and the cache is volatile or the startup flush was disabled at restart. This one can even get UnBuffered Log databases. > if it is only disk error making this? Any other explain for it? > > And how to continue to diagnose it and prove it? Check logging modes of databases, check the system logs for shutdown info. Have the IDS shutdown script log its activity, make sure it is not shutting down the engine in the background. Note that onmode -ky may return before the engine is completely offline. The script should loop on something like: GO=5 while [[ $GO -eq 5 ]]; do sleep 10 onstat - >/dev/null GO=$? done To be sure the engine is offline before returning. Art S. Kagel > expect any suggestion! > > Thanks in advance! > > > > > > > > Here is a part of AF. > > > > 15:40:18 Assert Failed: Page Check Error in btsearch:bad btree node > 15:40:18 Informix Dynamic Server Version 7.31.UD4X3 15:40:18 Who: > Session(5, tellin@mscp1, 7795, -690870660) > Thread(50, sqlexec, d6cfdde0, 1) > File: rsdebug.c Line: 1013 > 15:40:18 Results: Possible inconsistencies in index> > 'scp20:"informix".syscolumns#column' > 15:40:18 stack trace for pid 7278 written to > /tellinshare/dumpdir/af.41a0f39 15:40:24 See Also: > /tellinshare/dumpdir/af.41a0f39, shmem.41a0f39.3 15:40:35 Assert> Failed: Page Check Error in btsearch:bad btree node 15:40:35 Informix > Dynamic Server Version 7.31.UD4X3 15:40:35 Who: Session(5, > tellin@mscp1, 7795, -690870660) > Thread(50, sqlexec, d6cfdde0, 1) > File: rsdebug.c Line: 1013 > 15:40:35 Results: Possible inconsistencies in index> > 'scp20:"informix".syscolumns#column' > 15:40:35 stack trace for pid 7278 written to > /tellinshare/dumpdir/af.41a0f39 15:40:41 See Also: > /tellinshare/dumpdir/af.41a0f39, shmem.41a0f39.4 15:40:50 Assert> Failed: Page Check Error in btsearch:bad btree node 15:40:50 Informix > Dynamic Server Version 7.31.UD4X3 15:40:50 Who: Session(5, > tellin@mscp1, 7795, -690870660) > Thread(50, sqlexec, d6cfdde0, 1) > File: rsdebug.c Line: 1013 > 15:40:50 Results: Possible inconsistencies in index> > 'scp20:"informix".syscolumns#column' > 15:40:50 stack trace for pid 7278 written to > /tellinshare/dumpdir/af.41a0f39 15:40:55 See Also: > /tellinshare/dumpdir/af.41a0f39, shmem.41a0f39.5 15:41:02 Assert> Failed: Page Check Error in btsearch:bad btree node 15:41:02 Informix > Dynamic Server Version 7.31.UD4X3 15:41:02 Who: Session(5, > tellin@mscp1, 7795, -690870660) > Thread(50, sqlexec, d6cfdde0, 1) > File: rsdebug.c Line: 1013 > 15:41:02 Results: Possible inconsistencies in index> > 'scp20:"informix".syscolumns#column' > 15:41:02 stack trace for pid 7278 written to > /tellinshare/dumpdir/af.41a0f39 15:41:08 See Also: > /tellinshare/dumpdir/af.41a0f39, shmem.41a0f39.6 15:41:15 > invoke_alarm(): /bin/sh -c '/opt/informix/etc/log_full.sh 5 6 "Internal> > Subsystem failure: 'Buffer'" "Thread exited with 7 buffers held. " > /tellinshare/dumpdir/af.41a0f39' > 15:41:15 invoke_alarm(): mt_exec failed, status 12, errno 0 15:41:15 > Assert Failed: Dynamic Server must abort 15:41:15 Informix Dynamic > Server Version 7.31.UD4X3 15:41:15 Who: Session(5, tellin@mscp1, 7795, > -690870660) > Thread(50, sqlexec, d6cfdde0, 1) > File: rsshmem.c Line: 1678 > 15:41:15 Results: Dynamic Server must abort 15:41:15 stack trace for> pid 7278 written to /tellinshare/dumpdir/af.41a0f39 15:41:20 See Also: > /tellinshare/dumpdir/af.41a0f39, shmem.41a0f39.7 15:41:28 rsshmem.c, > line 1678, thread 50, proc id 7278, Dynamic Server must abort. 15:41:28 > PANIC: Attempting to bring system down > > AF'''''''''''''''''''''''''''''' > > 18:24:46 bfcheck: bad page: pg_flags 4 != type 10 buffer header > c392f554: 00000000 00000000 00000000 00000001 ........ ........ > c392f564: 00000000 00000000 c36242cc c36242cc ........ .bB..bB. > c392f574: c2db1d34 c392f5b8 00000002 00000000 ...4.... ........ > c392f584: 0000000a 00000000 00000000 00010000 ........ ........ > c392f594: 00000010 02c82c4e 00500003 00000030 ......,N .P.....0 > c392f5a4: 00000000 00000000 00500549 00000001 ........ .P.I....> c392f5b4: c4951800 .... page c4951800: > 00500549 02c71a26 00000004 001807e4 .P.I...& ........ c4951810: > 00000000 00000000 00000003 00000000 ........ ........ c4951820: > 00000000 00000000 00000000 00000000 ........ ........ c4951830 * > c4951ff0: 00000000 00000000 00000000 02c71a26 ........ .......& chunk > info > d6cf4738: 00050006 00000019 000f9e0c 00094099 ........ ......@. > d6cf4748: 00050000 00400015 2f646576 2f766731 .....@.. /dev/vg1 > d6cf4758: 322f7277 6f726b64 62736c76 31000000 2/rworkd bslv1... > d6cf4768: 00000000 00000022 d6cf4218 00000001 ......." ..B..... > d6cf4778: 00000001 00000000 00000000 00000022 ........ ......." > d6cf4788: 6c6f6762 66320000 00000040 002a90a3 logbf2.. ...@.*.. > d6cf4798: 0072d9a4 00736254 00000000 0072a878 .r...sbT .....r.x > d6cf47a8: c3919c04 d6cf6018 00000001 d6cf63c0 ......`. ......c. > d6cf47b8: 00000001 00000000 00000081 c3919d94 ........ ........ > d6cf47c8: 00000003 d6d3e098 00070000 00000000 ........ ........ > d6cf47d8: 00000068 00000008 00000000 00000000 ...h.... ........ > d6cf47e8: 0000006b 00000016 00000000 00000000 ...k.... ........ > d6cf47f8: 00000000 00000000 0000082d 40000003 ........ ...-@... > d6cf4808: 00000000 00000000 00000001 00000000 ........ ........ > d6cf4818: 00000000 00000000 00000000 00000000 ........ ........ aio > req > d713f040: 00000001 00000000 00000000 00000000 ........ ........ > d713f050: 00000000 00000000 00000000 00000000 ........ ........ > d713f060: 00000000 d713f1b8 d713eec8 000001c0 ........ ........ > d713f070: 41494f72 65710000 00000000 07d10000 AIOreq.. ........ > d713f080: 00000000 00000001 00000000 00000001 ........ ........ > d713f090: 00000002 00000000 00000000 00000000 ........ ........ > d713f0a0: d713f0c0 00000000 00000001 00000000 ........ ........ > d713f0b0: 00000000 00000000 00000000 00000000 ........ ........ aio op > d713f0c0: 00000000 00000800 00000001 d71c63d0 ........ ......c. > d