Re: CRITICAL SITUATION - who has seen this Page Check Error problem

barryleb@atl.mindspring.com (Barry Leb) wrote: This is a follow up to what we know so far on this problem. I appreciate all the comments I have recieved. We have put the system back on line, but there still remain some questions. First of all we pursued this with both Informix and HP. Both groups looked into the system extensively. We do know that it is not an offset, space or linking problem. Our concerns were mainly focused on the RAID manager and the engine itself. This all started when we crashed several times with Page Check Errors. The online log told us to run oncheck -cDI's which we did and found no errors. When Informix dialed in they found the following type of situation in several places on Chunk 6: The table ps_vendor had an extent of several hundred pages. The page at 72455 had an address in the page header for chunk 6, however, page 72456 had an address for chunk 10. You can obviously see our concern and informix's. Given this, everyone was pursuing this issue as a massive corruption problem. This is now what we know. This extent happens to be the current extent for this table. We tested this by adding several hundred dummy rows to the table and watched the page header addresses change for subsequent data pages back to Chunk 6. This is where the fun starts. The Informix engineer working on this problem noticed that some things in this case resembled some existing known bugs dealing with light scans and our version of the engine. The current feeling among all those involved is that we have been chasing our tails for 2 days on a false alarm. That are real problem that is causing the engine crashes has nothing to do with the weird page header addresses. Now the question is where are these page header addresses coming from. One theory is that some how it is old information left over from a previous configuration as it was two weeks ago prior to the upgrade to 7.11. Could it be possible that chunk 10 was residing on these physical devices then. When informix looked at the bitmap pages they are showing these chunk 10 data pages as unused and test above confirmed this. It is still possible that the current engine or hardware configuration is doing this too. We don't know yet for sure and are pursuing this further. At least we're fairly certain that we don't have a massive data corruption problem at this time. Anyone else care to comment on all this? >"Jerry A. Van Zant" <jvanzant@msn.com> wrote: >>What is the size of your chunks? and their offsets? >Dbspaces >address num flags fchunk nchunks flags owner name >c50980f0 1 2 1 1 M informix rootdbs >c50c5a98 2 2001 2 1 N T informix tempdbs >c50c5b00 3 2 3 1 M informix physdbs >c50c5b68 4 1 4 2 N informix fsprod_data_01 >c50c5bd0 5 1 5 2 N informix fsprod_data_02 >c50c5c38 6 1 9 1 N informix fsprod_index_01 >c50c5ca0 7 1 10 1 N informix fsprod_index_02 >c50c5d08 8 1 8 1 N informix logdbs > 8 active, 2047 maximum >Chunks >address chk/dbs offset size free bpages flags pathname >c5098158 1 1 0 26000 25293 PO- /dev/informix/rootdbs >c5098230 1 1 0 26000 0 MO dev/informix/rootmirror_dbs >c50c5228 2 2 0 96000 95947 PO- /dev/informix/tempdbs >c50c5300 3 3 0 12000 8547 PO- /dev/informix/physdbs >c50c59c0 3 3 0 12000 0 MO-/dev/informix/mphysdbs >c50c53d8 4 4 0 750000 3 PO /dev/informix/fsprod_data_01 >c50c54b0 5 5 0 750000 4 PO-/dev/informix/fsprod_data_02 >c50c5588 6 5 0 250000 147623PO-/dev/informix/fsprod_index_01 >c50c5660 7 4 0 250000 198817 PO-/dev/informix/fsprod_index_02 >c50c5738 8 8 0 250000 1377 PO- /dev/informix/logdbs >c50c5810 9 6 250001 500000 107200PO-/dev/informix/fsprod_index_01 >c50c58e8 10 7 250000 500000 80274 PO-/dev/informix/fsprod_index_02 > 10 active, 2047 maximum >>Barry Leb <barryleb@atl.mindspring.com> wrote in article >><5njskb$fqf@camel12.mindspring.com>... >>> We are in dire straits at this time. We are running informix >>> 7.11.uc.1 on an HP 9000 h70, HP-UX 9.04. We are using the NikeArrays >>> with 6-2gig drives configured as 2 RAID 5 devices(Please don't make >>> fun, it wasn't my doing). We have split the data in two. Half the >>> tables reside on one device, with its indexes detached onto the second >>> RAID 5 device and vice versa. >>> >>> This is the problem. Over the last 2 weeks the engine has crashed 5 >>> times with Page Check Errors usually pointing to bad pages or bad >>> current nodes. Onchecks and dumps of the data pages are showing data >>> pages belonging to Chunk 10 which is dedicated to Indexes showing up >>> in Chunk 6 which is dedicated to tables only. >>> >>> Has anyone seen anything like this. We have had a production system >>> down for 2 days and neither Informix nor HP has been able to diagnose >>> this one yet. >>> >>> As a side note, we are seeing some of the same problems on a separate >>> HP 900 k400 running HPUX 10.01. Also Informix is same at 7.11.uc1. >>> However, using RAID 0 here. >>> >>> One last thing. This problem occurred within 24hrs after an upgrade to >>> 7.11 from 6.00. However, the second system has been running 7.11 for >>> over a year. >>> >>> Barry Leb >>> >>> National Linen Service >>> 1420 Peachtree Street >>> MS #230 >>> Atlanta, GA 30309 >>> e-mail: barryleb@mindspring.com >>> >>> >Barry Leb >National Linen Service >1420 Peachtree Street >MS #230 >Atlanta, GA 30309 >e-mail: barryleb@mindspring.com Barry Leb barryleb@atl.mindspring.com