dbspaces corrupted
User reported chunk file corruption after ontape backup aborted when disk became read-only during virtualized Informix 11.70FC4 operation. Respondents explained that write errors to read-only chunks cause Informix to mark them offline. Recovery options include: restoring from archive with logical log rollforward, running onspaces -O if mirrored, or contacting IBM Support to mark chunks online (with potential data loss).
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
We have Version 11.70FC4 sitting on a Red Hat 5.9 server that is, in turn
virtualised. All dbspaces are on cooked file systems.
We had a process that blocked the server using "onmode c block" followed by a
level 0 backup using "ontape s L 0".
At some point during the backup, the disk changed (either it ran out of space
or the volume group was flagged as read-only by the hypervisor). All dbspaces
are stored in the same place
At this point the backup aborted and immediately afterwards SOME of the chunk
files were reported as corrupt.
Can anyone tell me why the chunk files might have been corrupted?
0:06:01 Archive on rootdbs, llog, plog, prontodbs, livedbs, testdbs, cclxi,
cclxitst, ccleoy, ccleoy2 ABORTED.
00:06:01 Aborted by client.
00:06:02 Cannot unblock server blocked if it has not been blocked
by 'onmode -c block'.
00:06:25 Checkpoint Completed: duration was 0 seconds.
00:06:25 Wed Apr 1 - loguniq 23166, logpos 0x152018, timestamp: 0xddc30c92
Interval: 264134
00:06:25 Maximum server connections 217
00:06:25 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0,
Plog used 63, Llog used 31
00:11:25 Assert Warning: I/O error, Primary Chunk
'/data/dbs/pronto/cclxi/pronto_cclxi_2.dbs' -- Offline
00:11:25 IBM Informix Dynamic Server Version 11.70.FC4W1GE
00:11:25 Who: Thread(13, flush_sub(0), cfaa8870, 10)
File: rsbuff.c Line: 5246
00:11:25 Results: Chunk is now unusable
00:11:25 Action: Repair and restore from mirror or archive
00:11:25 stack trace for pid 5223 written to /pro/informix/tmp/af.3f580dd
00:11:25 Assert Warning: I/O error, Primary Chunk
'/data/dbs/pronto/cclxi/pronto_cclxi_4.dbs' -- Offline
00:11:25 IBM Informix Dynamic Server Version 11.70.FC4W1GE
00:11:25 Who: Thread(14, flush_sub(1), cfaa90b8, 9)
File: rsbuff.c Line: 5246
00:11:25 Results: Chunk is now unusable
00:11:25 Action: Repair and restore from mirror or archive
If the chunks became read-only the engine will have marked the chunk
offline if it received a write error trying to update the chunk.
Art
Art S. Kagel, President and Principal Consultant
ASK Database Management
www.askdbmgt.com
Blog: http://informix-myview.blogspot.com/
Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on the IIUG, nor any other organization with which I am
associated either explicitly, implicitly, or by inference. Neither do
those opinions reflect those of other individuals affiliated with any
entity with which I am affiliated nor those of the entities themselves.
On Wed, Apr 8, 2015 at 8:51 PM, RAY BURNS <ray.burns@velocityglobal.co.nz>
wrote:
> We have Version 11.70FC4 sitting on a Red Hat 5.9 server that is, in turn
> virtualised. All dbspaces are on cooked file systems.
>
> We had a process that blocked the server using "onmode c block" followed
> by a
> level 0 backup using "ontape s L 0".
>
> At some point during the backup, the disk changed (either it ran out of
> space
> or the volume group was flagged as read-only by the hypervisor). All
> dbspaces
> are stored in the same place
>
> At this point the backup aborted and immediately afterwards SOME of the
> chunk
> files were reported as corrupt.
>
> Can anyone tell me why the chunk files might have been corrupted?
>
> 0:06:01 Archive on rootdbs, llog, plog, prontodbs, livedbs, testdbs, cclxi,
> cclxitst, ccleoy, ccleoy2 ABORTED.
> 00:06:01 Aborted by client.
> 00:06:02 Cannot unblock server blocked if it has not been blocked
>
> by 'onmode -c block'.
> 00:06:25 Checkpoint Completed: duration was 0 seconds.
> 00:06:25 Wed Apr 1 - loguniq 23166, logpos 0x152018, timestamp: 0xddc30c92
> Interval: 264134
>
> 00:06:25 Maximum server connections 217
> 00:06:25 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked
> 0,
> Plog used 63, Llog used 31
>
> 00:11:25 Assert Warning: I/O error, Primary Chunk
> '/data/dbs/pronto/cclxi/pronto_cclxi_2.dbs' -- Offline
> 00:11:25 IBM Informix Dynamic Server Version 11.70.FC4W1GE
> 00:11:25 Who: Thread(13, flush_sub(0), cfaa8870, 10)
>
> File: rsbuff.c Line: 5246
> 00:11:25 Results: Chunk is now unusable
> 00:11:25 Action: Repair and restore from mirror or archive
> 00:11:25 stack trace for pid 5223 written to /pro/informix/tmp/af.3f580dd
> 00:11:25 Assert Warning: I/O error, Primary Chunk
> '/data/dbs/pronto/cclxi/pronto_cclxi_4.dbs' -- Offline
> 00:11:25 IBM Informix Dynamic Server Version 11.70.FC4W1GE
> 00:11:25 Who: Thread(14, flush_sub(1), cfaa90b8, 9)
>
> File: rsbuff.c Line: 5246
> 00:11:25 Results: Chunk is now unusable
> 00:11:25 Action: Repair and restore from mirror or archive
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--bcaec5171d35fbdcfe0513421f3a
Thanks so much. After fixing the read-only problem on the dbspace how does one bring the chunk back on-line?
call IBM support, they can activate the chunk again, if it is consistent. Marcus Haarmann ----- Ursprüngliche Mail ----- Von: "RAY BURNS" <ray.burns@velocityglobal.co.nz> An: ids@iiug.org Gesendet: Donnerstag, 9. April 2015 06:29:50 Betreff: Re: dbspaces corrupted [34939] Thanks so much. After fixing the read-only problem on the dbspace how does one bring the chunk back on-line? ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.
If the chunks are mirrored on the Informix side you just run "onspaces -s
-p /chunk/path -o <offset> -s <size> -O". If the chunk is not mirrored by
Informix then your only options are:
1) Restore from an archive and roll forward logical logs, or
2) Call IBM Support Down System line and ask them to mark the chunk online.
Note that option #2 MAY lose some data. Either way, if the logical log
chunks were among those that went down, you will likely lose those
transactions that were attempted while the chunks were offline - these
likely failed with an error code anyway, but FWIW.
Art
Art S. Kagel, President and Principal Consultant
ASK Database Management
www.askdbmgt.com
Blog: http://informix-myview.blogspot.com/
Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on the IIUG, nor any other organization with which I am
associated either explicitly, implicitly, or by inference. Neither do
those opinions reflect those of other individuals affiliated with any
entity with which I am affiliated nor those of the entities themselves.
On Thu, Apr 9, 2015 at 12:29 AM, RAY BURNS <ray.burns@velocityglobal.co.nz>
wrote:
> Thanks so much. After fixing the read-only problem on the dbspace how does
> one
> bring the chunk back on-line?
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--001a113ed4227c3b56051349cfa1
Out of interest, why are you blocking the server before doing an ontape
backup? This is intended to allow external storage backups to take place
safely. If you want to know the restore point, wouldn't a checkpoint suffice?
Ben.