Update from 10.00UC1 to 10.00UC4 poops out
Posted in 2006
I've had an IDS instance on an x86/LINUX box since 7.x something and it
has always been *ROCK* stable, upgrades to 9.21 to 9.4x to 10.00UC1
have always gone off without a hitch. Now I've got a minor upgrade that
has bit it twice, from 10.00UC1 to 10.00UC4. Reverting back to 10.00UC1
and reloading the databases and all is well. Anyone have any ideas:
Sequence
----
1. Exported the two databases with dbexport.
2. Ran a series of onchecks to see that there wasn't any problems.
oncheck -q -cr
oncheck -q cR
oncheck -q ce
oncheck -q cc miecr
oncheck -q cc cms
oncheck -q -x -ci miecr
oncheck -q -x -ci cms
oncheck -q -x -cI miecr
oncheck -q -x -cI cms
oncheck -q -cD miecr
oncheck -q -cD cms
.. these all completed without complaint.
3. Took the server offline.
4. Ran ""./ids_install" from C880JNA.tar
5. Started server via onmonitor
6. "onstat -d" shows all chunks online, queries seem fine, tested with
applications. Dropped and recreated some indexes, etc...
7. Start a test with a long running 4GL application, it runs for a very
long time (approx four hours) and then dies with -
duperr program fails with -
Program stopped at "flowcntrl.4gl", line number 232.
SQL statement error number -243.
Could not position within a table (informix.xrefr).
SYSTEM error number -105.
ISAM error: bad isam file format.
The output of "onstat -m" looks like -
09:58:41 IBM Informix Dynamic Server Version 10.00.UC4
09:58:41 Who: Session(109, intranet@KOHOCTON.morrison.iserv.net, 6814,
0x4b7a04a0)
Thread(2895, sqlexec, 4b77c06c, 1)
File: rsdebug.c Line: 1092
09:58:41 Results: Possible inconsistencies in '11121E1700
e:"11121U316071 \\uffff\\uffff".11122 $\\uffff'
09:58:41 Action: Run 'oncheck -cDI 11121E1700 e:"11121U316071 \\uffff
\\uffff".11122 $\\uffff'
09:58:41 stack trace for pid 2232 written
to /opt/informix/dump/af.f37fc10
09:58:41 See Also: /opt/informix/dump/af.f37fc10, shmem.f37fc10.0
09:58:44 Unable to create output file
'/opt/informix/dump/shmem.f37fc10.0' errno = 28
09:58:44 Page Check Error in btcurrent:bad current page
09:58:45 Assert Failed: Page Check Error in btcurrent:bad current page
09:58:45 IBM Informix Dynamic Server Version 10.00.UC4
09:58:45 Who: Session(109, intranet@KOHOCTON.morrison.iserv.net, 6814,
0x4b7a04a0)
Thread(2895, sqlexec, 4b77c06c, 1)
File: rsdebug.c Line: 1066
09:58:45 Results: Possible inconsistencies in index '11121E1700
e:"11121U316071 \\uffff\\uffff".11122 $\\uffffKey#1'
09:58:45 Action: Run 'oncheck -cI 11121E1700 e:"11121U316071 \\uffff
\\uffff".11122 $\\uffffKey#1'
09:58:45 stack trace for pid 2232 written
to /opt/informix/dump/af.f37fc10
09:58:45 See Also: /opt/informix/dump/af.f37fc10
09:58:49 Page Check Error in btcurrent:bad current page
Eh?
Dump directory unfortunately fills up (subsequently changed to somewhere
with additional disk space).
"onstat -d" now shows that some chunks are offline -
address chunk/dbs offset size free bpages flags pathname
4b689948 1 1 0 976885 856593 PO-B /opt/informix/dev/root_chunk
4b689ad0 1 1 0 976885 0 MO-B /opt/informix/dev/root_chunk_mirror
4b79bd38 2 2 1 8885090 6680557 PO-B /opt/informix/dev/datadbs_chunk
4c1ed948 2 2 1 8885090 0 MO-B /opt/informix/dev/datadbs_chunk_mirror
4c1ed018 3 3 0 8886776 ~8886776 8886776
POBB /opt/informix/dev/blobspace0
4c1edad0 3 3 0 8886776 0 MOBB /opt/informix/dev/blobspace0_mirror
4c1ed1a0 4 4 0 8886776 0 PD-B /opt/informix/dev/indexdbs
4c1edc58 4 4 0 8886776 8326382 MO-B /opt/informix/dev/indexdbs_mirror
4c1ed328 5 5 0 1000000 0 PD-B /opt/informix/dev/tmpdbs_chunk2
4c1ed4b0 6 3 0 8886776 ~8886776 8886776
POBB /opt/informix/dev/blobspace1
4c1edde0 6 3 0 8886776 0 MDBB /opt/informix/dev/blobspace1_mirror
4c1ed638 7 5 0 1000000 0 PD-B /opt/informix/dev/tmpdbs_chunk3
4c1ed7c0 8 5 0 100000 0 PD-B /opt/informix/dev/tmpdbs_chunk
8 active, 32766 maximum
Shutdown server, restored backup of /opt/informix, reinitialized first
chunk of raw device, added back chunks and dbspaces, imported databases
from the dbexport via dbimport.
Done this twice now.