Raw devices softlink to dbspaces were lost
After a reboot, the symlinks pointing Informix chunk paths at ~30 raw devices were lost, so onstat -d showed chunks as PD (down) and the instance was down; the poster knew the chunk/dbspace layout from oncheck -pr but not which device belonged to which chunk, and an OS (NetBackup) restore of the links would take days. Responders showed how to read each raw device's page header directly (dd/od -x on page 0, allowing for offsets and byte order) to extract the chunk number, rebuild the symlinks accordingly, and verify with oncheck -pP <chunk> 0 before restarting. It was also noted that if the root chunk was unreachable the down-chunk flags may not have been written, so the instance could come back cleanly, and that on Linux 2.6+ udev rules should be used to recreate the links persistently. No confirmation from the poster is recorded.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Hi All,
I have a DB shutted down today due to soft link of the chunks to raw devices
were lost after reboot. I can't find a copy of those which chunk is connected
to raw devices.. there are around 30+ raw devices affected. Due to this issue
"onstat -d" shows "PD-----", instead of "PO-----" status.
I know this should be an OS issue, my problem is, there's no history or logs
where i can get the correct values of the link.
Question:
Is there any way on DB perspective i can look into?
Do you have any OS backupds ... they should have the links ....
Peter Logan
Senior Database Administrator
Phone: 616/878-8309
From:
"JACK PAPA" <informix2009@gmail.com>
To:
ids@iiug.org
Date:
10/22/2009 01:14 PM
Subject:
Raw devices softlink to dbspaces were lost [17689]
Sent by:
ids-bounces@iiug.org
Hi All,
I have a DB shutted down today due to soft link of the chunks to raw
devices
were lost after reboot. I can't find a copy of those which chunk is
connected
to raw devices.. there are around 30+ raw devices affected. Due to this
issue
"onstat -d" shows "PD-----", instead of "PO-----" status.
I know this should be an OS issue, my problem is, there's no history or
logs
where i can get the correct values of the link.
Question:
Is there any way on DB perspective i can look into?
*******************************************************************************
Forum Note: Use "Reply" to post a response in the discussion forum.
You have an onstat -d output? then you have the link paths... unless I am
misunderstanding you.
MM
I thought he was needing to know what link when to what device... that
should be available from an OS backup.
Peter Logan
Senior Database Administrator
Phone: 616/878-8309
From:
"MIKE MAGIE" <jmmagie@yahoo.com>
To:
ids@iiug.org
Date:
10/22/2009 01:18 PM
Subject:
Re: Raw devices softlink to dbspaces were lost [17691]
Sent by:
ids-bounces@iiug.org
You have an onstat -d output? then you have the link paths... unless I am
misunderstanding you.
MM
*******************************************************************************
Forum Note: Use "Reply" to post a response in the discussion forum.
You can also try to run oncheck -pr - there will be a section that shows all
of your chunks and chunk paths.
I think even if he relinked the links to the devs he would still have down chunks. Once the chunks and dbspaces get flagged offline - unless mirrored - they'll stay that way until restored - in which case it would not matter where he linked to... But yeah I missed that part...
Thanks all,
Raw devices links to each chunks connected to dbspace is the problem... the
raw devices are there and also the oncheck -pr output.... which has the chunks
associated to each dbspace.
NOTE: It's not mirrored and i was thinking that if i got the link restored
from netbackup, i will just restart the db instance using onmode -ky;oninit -y.
Problem is the netbackup will take days to restore due to some process, so, im
trying to find another solution how to restore that link between raw dev and
links to dbspace.
I am not sure which version you are on, but you can do the following
and the example below show version 11.,
Assuming the device offset is 0 then you can run the following command
od -x device | head -1
0000000 0000 0000 0001 96f5 0003 1800 0130 06c0
*****
Look at the **** group of numbers, please be sure to byte swap if
you required by your processor type.
After setting all the link up and before starting the server
run oncheck -pP {chunk number} 0 | head -2 for each chunk to
make sure all is correct.
*** chunk number matches below
oncheck -pP 1 0 | head -2addr stamp chksum nslots flag type frptr frcnt next
prev
1:0 431858 96f5 3 1800 ROOTRSV 304 1728 0
0
**** chunk number matches the command above
John F. Miller III
STSM, Support Architect
miller3@us.ibm.com
503-578-5645
IBM Informix Dynamic Server (IDS)
ids-bounces@iiug.org wrote on 10/22/2009 10:36:26 AM:
> [image removed]
>
> Re: Raw devices softlink to dbspaces were lost [17698]
>
> JACK PAPA
>
> to:
>
> ids
>
> 10/22/2009 10:37 AM
>
> Sent by:
>
> ids-bounces@iiug.org
>
> Please respond to ids
>
> Thanks all,
>
> Raw devices links to each chunks connected to dbspace is the problem...
the
> raw devices are there and also the oncheck -pr output.... which has
> the chunks
> associated to each dbspace.
>
> NOTE: It's not mirrored and i was thinking that if i got the link
restored
> from netbackup, i will just restart the db instance using onmode
-ky;oninit
> -y.
>
> Problem is the netbackup will take days to restore due to some
> process, so, im
> trying to find another solution how to restore that link between raw dev
and
> links to dbspace.
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
If you are on a version that uses old school page headers the od and onchecks
will look like this:
od -x /dev/md/rdsk/d1001 | head -1 ****This is my rootchunk***
0000000 0010 0000 f0a3 8a3f 0002 1000 00d0 0724
CCCO OOOO
The CCC represents the chunk number, 001 in this case, and the O's represent
the offset into the chunk - in this case we are looking at page zero, 0.
Oncheck is easier to read:
oncheck -pP 1 0 (chunk 1 page 0)
addr
1:0
The format of that last email was whacky - the physical address of the chunk
starts at the point 0001 0000 - and like I said - this indicates Chunk number
1 - (root chunk) and page offset 0 - meaning the first 2K or 4k block in the
chunk - depending on the pagesize of your system. It is not that hard - Jon
just wanted to show you that you could use od to determine the chunk numbers
that your devices rep so you can then link them correctly - and then at the en
of the link process run the oncheck to verify that they are linked right.
On each candidate chunk use:
dd if=<actual chunk path> bs=2k count=5 | od -x|less(or some pager)
The first four bytes of each page (except the zeroth page) will contain the
chunk number in the first 12 bits (so 3 nibbles or hex characters). You can
use a calculator or bc to translate the three hex digits back to a chunk
number.
This is trivial if all of your devices have only one chunk on each and no
offsets. If you have used offsets, add a skip=<offset pages> to the above.
If more than one chunks resides on a device, you'll have to run the command
once for each chunk with the appropriate 'skip= ' setting.
If you really can't figure this out you will have to call IBM tech support
and set up a paid support call or contact us at Oninit (contact info at
www.oninit.com).
Art
Art S. Kagel
Oninit (www.oninit.com)
IIUG Board of Directors (art@iiug.org)
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Oninit, the IIUG, nor any other organization
with which I am associated either explicitly or implicitly. Neither do
those opinions reflect those of other individuals affiliated with any entity
with which I am affiliated nor those of the entities themselves.
On Thu, Oct 22, 2009 at 1:14 PM, JACK PAPA <informix2009@gmail.com> wrote:
> Hi All,
>
> I have a DB shutted down today due to soft link of the chunks to raw
> devices
> were lost after reboot. I can't find a copy of those which chunk is
> connected
> to raw devices.. there are around 30+ raw devices affected. Due to this
> issue
> "onstat -d" shows "PD-----", instead of "PO-----" status.
>
> I know this should be an OS issue, my problem is, there's no history or
> logs
> where i can get the correct values of the link.
>
> Question:
>
> Is there any way on DB perspective i can look into?
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--0015174760aaaafada04768b8d34
Not if the root chunk was inaccessible when the engine attempted to mark the chunks down. If that's the case he'll be OK except for possible lost transactions. Art Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Oct 22, 2009 at 1:24 PM, MIKE MAGIE <jmmagie@yahoo.com> wrote: > I think even if he relinked the links to the devs he would still have down > chunks. Once the chunks and dbspaces get flagged offline - unless mirrored > - > they'll stay that way until restored - in which case it would not matter > where > he linked to... > > But yeah I missed that part... > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --0015174784f0ce454d04768b93f0
True dat - he would be fine in that case...
It's been a while since I've done it, but if this is Linux 2.6 or newer, you
need to use the 'udev' process and create your own config files (under
/etc/udev/?? ) that rebuilds the links after any reboots. Either do a man on
udev or google it.
Bob
----- Original Message -----
From: "John Miller iii" <miller3@us.ibm.com>
To: ids@iiug.org
Sent: Thursday, October 22, 2009 2:21:13 PM GMT -05:00 US/Canada Eastern
Subject: Re: Raw devices softlink to dbspaces were lost [17704]
I am not sure which version you are on, but you can do the following
and the example below show version 11.,
Assuming the device offset is 0 then you can run the following command
od -x device | head -1
0000000 0000 0000 0001 96f5 0003 1800 0130 06c0
*****
Look at the **** group of numbers, please be sure to byte swap if
you required by your processor type.
After setting all the link up and before starting the server
run oncheck -pP {chunk number} 0 | head -2 for each chunk to
make sure all is correct.
*** chunk number matches below
oncheck -pP 1 0 | head -2addr stamp chksum nslots flag type frptr frcnt next
prev
1:0 431858 96f5 3 1800 ROOTRSV 304 1728 0
0
**** chunk number matches the command above
John F. Miller III
STSM, Support Architect
miller3@us.ibm.com
503-578-5645
IBM Informix Dynamic Server (IDS)
ids-bounces@iiug.org wrote on 10/22/2009 10:36:26 AM:
> [image removed]
>
> Re: Raw devices softlink to dbspaces were lost [17698]
>
> JACK PAPA
>
> to:
>
> ids
>
> 10/22/2009 10:37 AM
>
> Sent by:
>
> ids-bounces@iiug.org
>
> Please respond to ids
>
> Thanks all,
>
> Raw devices links to each chunks connected to dbspace is the problem...
the
> raw devices are there and also the oncheck -pr output.... which has
> the chunks
> associated to each dbspace.
>
> NOTE: It's not mirrored and i was thinking that if i got the link
restored
> from netbackup, i will just restart the db instance using onmode
-ky;oninit
> -y.
>
> Problem is the netbackup will take days to restore due to some
> process, so, im
> trying to find another solution how to restore that link between raw dev
and
> links to dbspace.
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
*******************************************************************************
Forum Note: Use "Reply" to post a response in the discussion forum.
Related threads
- IDS 10 table-level restore
- Informix Development Webinar December 11, 2007
- ontape -p/r with changed ROOTPATH
- Migrate from HP PA-RISC to HP ITANIUM by ontape