hard disk failure
Posted in 1994
Topics: High Availability & Replication, Storage & Space Management, Server Administration, Security, Permissions & Auditing, Logging & Checkpoints, Migration, Import/Export & Data Conversion
hi all,
below is an excerpt from the errpt report which depicts only the hard
disk problems in reverse chronological order. these hits taken while
i performed my "cplv", logical volume to logical volume copy.
5DFEADCB 1229142593 U H LVDD Hardware disk block relocation achieved
5DFEADCB 1229142593 U H LVDD Hardware disk block relocation achieved
the following hits were taken during the dba's attempt at database
archival to tape:
8564E6B1 1229123493 T H hdisk7 DISK OPERATION ERROR
8564E6B1 1229123493 T H hdisk7 DISK OPERATION ERROR
5DFEADCB 1229123493 U H LVDD Hardware disk block relocation achieved
8564E6B1 1229123493 T H hdisk7 DISK OPERATION ERROR
5DFEADCB 1229123493 U H LVDD Hardware disk block relocation achieved
8564E6B1 1229123493 T H hdisk7 DISK OPERATION ERROR
the basic idea behind this migration was to get the informix
"lvdata7" logical volume off of hdisk7 where we were clearly getting
into trouble with bad blocks. unfortunately we did not act in time.
we were still in good shape with respect to the daily/weekly informix
log information although some of the errpt reports indicate that we
experienced "disk operation failures" or "bad block relocation"
errors. however, the frequency with which we were getting the disk
hits were increasing and i had no idea of the total number of "bad
block relocations" made. according to the bell atlantic ce we can
expect 21 relocations before they are exhausted.
the following is the sequence of activities i performed after the dba
made every attempt at archiving the trc database, he was successful
at archiving the iso database (trc and iso share the save LV). i was
equally successful at moving the iso database with the following
commands:
># cplv -e lvdata7t lvdata7
>cplv: All data contained on the destination logical
> volume will be destroyed.
>cplv: Do you wish to continue? y(es) n(o)?
>y
>0516-746 cplv: Destination logical volume must have
> type set to copy.
>cplv: Use chlv command with -t flag to change the type to copy.
># chlv -t copy lvdata7t
># cplv -e lvdata7t lvdata7
>cplv: All data contained on the destination logical
> volume will be destroyed.
>cplv: Do you wish to continue? y(es) n(o)?
>y
>cplv: Logical volume lvdata7 successfully copied to lvdata7t .
>#
the "chlv" was necessary to change the logical volume type to "copy"
(from "jfs") prior to moving the information.
you will also notice the good return from the "cplv":
cplv: Logical volume lvdata7 successfully copied to lvdata7t .
this is a little mind boggling. the "cplv" has succeeded where the
tape archive had failed. below there is an excerpt from the trc log
showing a number of problems with an area in the database. during
the first attempt at tape archival we see the 110 error which is
"media surface error". normally, this should not pose a problem
since the bad block relocation mechanism should come into play.
however, "bad block relocation" does not make since this is a "read"
operation and not a "write". i have to wonder what actually
happened. my indications suggest the relocation was successful but
what was relocated if the "bad" block was not successfully read?
after the 110 abend the dba tried a second time to archive the trc
database to tape and encountered a 13 error. it suggests an access
error (permission denied) but this is not reasonable considering
the fact that informix owns the logical volume.
database excerpts for trc:
12:07:20 Shutdown Mode
12:12:11 Quiescent Mode
12:20:15 Level 0 Archive Started
12:34:26 I/O error 'read': expect 16384 actual -1 addr 0x2a64 errno 110 retrys
0
12:34:32 I/O error 'read': expect 16384 actual -1 addr 0x2a64 errno 110 retrys
1
12:34:32 I/O error, Primary Chunk
'/usr/local/products/INFORMIX-4.1/dev/tempspace1' -- Offline
12:34:32 I/O read() chunk 4, pagenum 10852, pagecnt 4 --> errno = 110
12:34:33 Level 0 Archive Cancelled
12:36:00 Checkpoint Completed
12:47:17 Level 0 Archive Started
12:58:08 I/O read() chunk 4, pagenum 0, pagecnt 4 --> errno = 13
12:58:09 Level 0 Archive Cancelled
13:05:23 Level 0 Archive Started
13:16:18 I/O read() chunk 4, pagenum 0, pagecnt 4 --> errno = 13
13:16:19 Level 0 Archive Cancelled
13:35:02 Level 1 Archive Started
13:38:01 I/O read() chunk 4, pagenum 0, pagecnt 4 --> errno = 13
13:38:02 Level 1 Archive Cancelled
13:51:28 On-Line Mode
13:51:59 Shutdown Mode
13:51:59 Quiescent Mode
13:52:11 INFORMIX-OnLine Stopped
Wed Dec 29 13:53:58 1993
13:53:58 I/O read() chunk 4, pagenum 4, pagecnt 1 --> errno = 13
13:53:58 tblspace header error:
13:53:58 301a18a8: 00000000 00000000 00000000 00000000 ........ ........
13:53:58 301a18b8: 00010000 04000001 00400004 00000000 ........ .@......
13:53:58 301a18c8: 00000000 00000000 00000000 00000000 ........ ........
13:53:58 301a18d8: 00000000 00000000 00000000 00000000 ........ ........
13:53:58 301a18e8: 00000000 00000000 00000000 00000000 ........ ........
13:53:58 301a18f8: 00000000 00000000 00000000 00000000 ........ ........
13:53:58 301a1908: 00000000 00000000 00000000 00000000 ........ ........
13:53:58 301a1918: 00000000 00000000 00000000 00000000 ........ ........
13:53:58 301a1928: 00000000 00000000 00000000 00000000 ........ ........
13:53:58 301a1938: 00000000 00000000 00000000 00000000 ........ ........
13:53:58 301a1948: 00000000 00000000 00000000 00000000 ........ ........
13:53:58 301a1958: 00000000 00000000 00000000 00000000 ........ ........
13:53:58 301a1968: 00000000 00000000 00000000 00000000 ........ ........
13:53:58 301a1978: 00000000 00000000 00000000 00000000 ........ ........
13:53:58 301a1988: 00000000 00000000 00000000 00000000 ........ ........
13:53:58 301a1998: 00000000 301e0d54 053671a2 00000000 ....0..T .6q.....
13:53:58 301a19a8: 301e2c94 0.,.
13:53:58 -- Fail Consistency Check -- pthdrpage:ptalloc:bad bfget -- pid=10606
user=100 us=30001b9c
13:53:58 Cannot Open Dbspace 4
13:53:58 INFORMIX-OnLine Initialized -- Shared Memory Initialized
13:53:58 Physical Recovery Started
13:53:59 Checkpoint Completed
13:53:59 Physical Recovery Complete: 0 Pages Restored
13:53:59 Logical Recovery Complete 0 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks
13:54:00 Quiescent Mode
13:55:57 I/O read() chunk 4, pagenum 2, pagecnt 1 --> errno = 13
13:55:57 Level 0 Archive Cancelled
13:56:43 I/O read() chunk 4, pagenum 2, pagecnt 1 --> errno = 13
13:56:43 Level 0 Archive Cancelled
13:58:03 I/O read() chunk 4, pagenum 2, pagecnt 1 --> errno = 13
13:58:03 Level 0 Archive Cancelled
13:58:12 I/O read() chunk 4, pagenum 2, pagecnt 1 --> errno = 13
13:58:12 Level 0 Archive Cancelled
14:00:32 I/O read() chunk 4, pagenum 2, pagecnt 1 --> errno = 13
14:00:32 Level 0 Archive Cancelled
14:10:05 Logical Log 2272 Backed Up
14:10:15 Logical Log 2273 Backed Up
14:10:24 Logical Log 2274 Backed Up
14:10:56 Checkpoint Complete