IDS 7.2 - Buffer modified in inconsistent chunk
An IDS 7.2 instance panicked during logical recovery with "Assert Failed: Buffer modified in inconsistent chunk", taking chunk 2 offline and flagging possible corruption in a table; the poster asked how serious this was and what the number in the suggested 'oncheck -cDI 2097271' meant. Art Kagel replied that it's a bad chunk, either data overwritten on disk or a failing drive, that oncheck probably won't repair it, and advised replacing the drive(s) and doing a full restore from the last known good archive. The poster then found oncheck wouldn't run because shared memory wasn't initialized with the server down, and asked how to get diagnostics; no answer to that follow-up is recorded.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Hello all,
I have an issue with IDS 7.2 that would possibly require either a portion or
the entire database restore. Currently, Chunk 2 cannot be taken ONLINE with
the errors below at startup. I need to determine the characteristic of the
problem before making decision - either recreating an afected table or the
full restore. Would you please let me know how critical it is about "Buffer
modified in inconsistent chunk" and if I can invoke "oncheck -cDI 2097271" as
suggested? I do not understand what 2097271 means as it should be the database
name not the internal description of the database or whatever it is meaningful
for human operator. I will put the output of oncheck as soon as I have
understood the meaning of this onchecks to provide more information.
Thu Aug 6 10:24:34 2009
10:24:34 Event alarms enabled. ALARMPROG = '/opt/informix/log_full.sh'
10:24:39 DR: DRAUTO is 0 (Off)
10:24:40 INFORMIX-OnLine Initialized -- Shared Memory Initialized.
10:24:40 DR: Trying to connect to secondary server ...
10:26:41 DR: Cannot connect to secondary server
10:26:41 DR: Turned off on primary server
10:26:41 Physical Recovery Started.
10:26:41 Physical Recovery Complete: 0 Pages Restored.
10:26:41 Logical Recovery Started.
10:26:44 Assert Failed: Buffer modified in inconsistent chunk.
10:26:44 Who: Session(7, root@host006, 0, 169673896)
Thread(17, fast_rec, a1accd8, 1)
10:26:44 Results: Chunk 2 is being taken OFFLINE.
10:26:44 Action: Restore chunk from archive.
10:26:44 See Also: /tmp/af.11afe3, shmem.11afe3.0
10:26:46 Assert Failed: WARNING! Chunk 2 is being taken OFFLINE.
10:26:46 Who: Session(7, root@host006, 0, 169673896)
Thread(17, fast_rec, a1accd8, 1)
10:26:46 Results: OnLine will block at next checkpoint
10:26:46 Action: Shutdown (onmode -k) or override (onmode -O)
10:26:46 See Also: /tmp/af.11afe3
10:26:46 Assert Failed: Page Check Error in bfput
10:26:46 Who: Session(7, root@host006, 0, 169673896)
Thread(17, fast_rec, a1accd8, 1)
10:26:46 Results: Possible inconsistencies in 'tkms:"t2000".may21y7'
10:26:46 Action: Run 'oncheck -cDI 2097271'
10:26:46 See Also: /tmp/af.11afe3
10:26:46 rsbuff.c, line 2075, thread 17, proc id 275, OnLine must abort.
10:26:47 Process exited with return code 1: /bin/sh /bin/sh -c
/opt/informix/log_full.sh 4 5 "DBspace is off-line: 'mstk'." "Chunk 2 is being
taken OFFLINE." /tm
10:26:47 Process exited with return code 1: /bin/sh /bin/sh -c
/opt/informix/log_full.sh 4 1 "Table failure: 'tkms:"t2000".may21y7'." "Page
Check Error in bfp
10:26:47 invoke_alarm(): /bin/sh -c '/opt/informix/log_full.sh 5 6 "Internal
Subsystem failure: 'MT'" "rsbuff.c, line 2075, thread 17, proc id 275, OnLine
must abort." '
10:26:47 invoke_alarm(): mt_exec failed, status -1, errno 0
10:26:47 PANIC: Attempting to bring system down
Best regards,
You have a bad chunk. It may be a logical problem caused by someone
overwritting the data on disk or the physical drive may be going south. Try
to run the oncheck, but I don't think it will be able ot fix this one. I
would replace the drive(s) and do a full restore from the last known good
archive.
Art
Art S. Kagel
Oninit (www.oninit.com)
IIUG Board of Directors (art@iiug.org)
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Oninit, the IIUG, nor any other organization
with which I am associated either explicitly or implicitly. Neither do
those opinions reflect those of other individuals affiliated with any entity
with which I am affiliated nor those of the entities themselves.
On Mon, Aug 10, 2009 at 2:48 AM, YAS ONO <yohelpdeskjp@gmail.com> wrote:
> Hello all,
>
> I have an issue with IDS 7.2 that would possibly require either a portion
> or
> the entire database restore. Currently, Chunk 2 cannot be taken ONLINE with
> the errors below at startup. I need to determine the characteristic of the
> problem before making decision - either recreating an afected table or the
> full restore. Would you please let me know how critical it is about "Buffer
> modified in inconsistent chunk" and if I can invoke "oncheck -cDI 2097271"
> as
> suggested? I do not understand what 2097271 means as it should be the
> database
> name not the internal description of the database or whatever it is
> meaningful
> for human operator. I will put the output of oncheck as soon as I have
> understood the meaning of this onchecks to provide more information.
>
> Thu Aug 6 10:24:34 2009
>
> 10:24:34 Event alarms enabled. ALARMPROG = '/opt/informix/log_full.sh'
> 10:24:39 DR: DRAUTO is 0 (Off)
> 10:24:40 INFORMIX-OnLine Initialized -- Shared Memory Initialized.
> 10:24:40 DR: Trying to connect to secondary server ...
> 10:26:41 DR: Cannot connect to secondary server
> 10:26:41 DR: Turned off on primary server
> 10:26:41 Physical Recovery Started.
> 10:26:41 Physical Recovery Complete: 0 Pages Restored.
> 10:26:41 Logical Recovery Started.
> 10:26:44 Assert Failed: Buffer modified in inconsistent chunk.
> 10:26:44 Who: Session(7, root@host006, 0, 169673896)
>
> Thread(17, fast_rec, a1accd8, 1)
> 10:26:44 Results: Chunk 2 is being taken OFFLINE.
> 10:26:44 Action: Restore chunk from archive.
> 10:26:44 See Also: /tmp/af.11afe3, shmem.11afe3.0
> 10:26:46 Assert Failed: WARNING! Chunk 2 is being taken OFFLINE.
> 10:26:46 Who: Session(7, root@host006, 0, 169673896)
>
> Thread(17, fast_rec, a1accd8, 1)
> 10:26:46 Results: OnLine will block at next checkpoint
> 10:26:46 Action: Shutdown (onmode -k) or override (onmode -O)
> 10:26:46 See Also: /tmp/af.11afe3
> 10:26:46 Assert Failed: Page Check Error in bfput
> 10:26:46 Who: Session(7, root@host006, 0, 169673896)
>
> Thread(17, fast_rec, a1accd8, 1)
> 10:26:46 Results: Possible inconsistencies in 'tkms:"t2000".may21y7'
> 10:26:46 Action: Run 'oncheck -cDI 2097271'
> 10:26:46 See Also: /tmp/af.11afe3
> 10:26:46 rsbuff.c, line 2075, thread 17, proc id 275, OnLine must abort.
> 10:26:47 Process exited with return code 1: /bin/sh /bin/sh -c
> /opt/informix/log_full.sh 4 5 "DBspace is off-line: 'mstk'." "Chunk 2 is
> being
> taken OFFLINE." /tm
> 10:26:47 Process exited with return code 1: /bin/sh /bin/sh -c
> /opt/informix/log_full.sh 4 1 "Table failure: 'tkms:"t2000".may21y7'."
> "Page
> Check Error in bfp
> 10:26:47 invoke_alarm(): /bin/sh -c '/opt/informix/log_full.sh 5 6
> "Internal
> Subsystem failure: 'MT'" "rsbuff.c, line 2075, thread 17, proc id 275,
> OnLine
> must abort." '
> 10:26:47 invoke_alarm(): mt_exec failed, status -1, errno 0
> 10:26:47 PANIC: Attempting to bring system down
>
> Best regards,
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--001636c5b02188c16b0470cd18cd
Hi Art,
Thanks for your prompt reply. I got the following message in Japanese with
"oncheck -cDI 2097271":
"Shared memory not initialized for INFORMIXSERVER 'ifmxhost006"
Do we need to have at least some framework processes like oninit up and
running when running oncheck -cDI? The last line of IDS startup log says
"PANIC: Attempting to bring system down," indicating all IDS related processes
went down.
How could I make oncheck run at least to get some diagnostics or clue for
analyzing the issue under the circumstance?