rollback fails Secondary HDR?

I think I have a little problem. As I am about to upgrade to a new server and my old secondary production system is giving me grief (does it know I am about to replace it?). The problem is that this is on ids 9.30.uc1. To old for support ( I think ). I used my secondary HDR server to test the myexport so I did not bog down the primary server. When I went to restore everything went perfect. Then I went to link the two servers (onmode -d .) and the HDR log recovery got into trouble. I got 40 or 50 of these errors. I googled them and found some stuff about bad disk drives and other problems but nothing related to HDR log recovery. The onstat -d does not show anything unusual like downed chunks. I did an oncheck -pe on the primary and had no errors. I did an oncheck -pe on the secondary and it had problems in the rootdbs? I don't put anything in the rootdbs and besides I just restored it clean (onbar & netbackup) so how could it have an error? The logs only show these lost log records. Dbaccess on the secondary looks ok also. I learned a little about deciphering logs at the Kansas IIUG but I am not sure I can put it to real use. (Nice to know but hard to use). I am assuming that failures like this are bad :-) and I have lost data but what I don't get is how do I find what this message means? I assume it will point me to the error. Any thoughts? I might call IBM on Monday, they have been good to me so perhaps they can tell me where to look up this error. Thanks for your help! ,,,Tim 21:30:26 Maximum server connections 0 21:30:26 Rollforward of log record failed. iserrno = 101 21:30:26 Log Record: log = 7609, pos = 0x67040, type = OLDRSAM:UNIQID(17), tran s = 257 21:30:26 Rollforward of log record failed. iserrno = 135 21:30:26 Log Record: log = 7609, pos = 0x670e8, type = OLDRSAM:ADDITEM(28), tra ns = 257 21:30:29 Rollforward of log record failed. iserrno = 135 21:30:29 Log Record: log = 7610, pos = 0xc0e4, type = OLDRSAM:ADDITEM(28), tran s = 37 21:30:29 Rollforward of log record failed. iserrno = 135 21:30:29 Log Record: log = 7610, pos = 0xc0e4, type = OLDRSAM:ADDITEM(28), tran s = 37 21:30:29 Rollforward of log record failed. iserrno = 101 21:30:29 Log Record: log = 7610, pos = 0xc150, type = OLDRSAM:HINSERT(40), tran s = 37 21:30:29 Rollforward of log record failed. iserrno = 101 21:30:29 Log Record: log = 7610, pos = 0xc150, type = OLDRSAM:HINSERT(40), tran s = 37 21:30:30 Rollforward of log record failed. iserrno = 135 21:30:30 Log Record: log = 7610, pos = 0xc11c, type = OLDRSAM:ADDITEM(28), tran s = 37 Tim Ertl 413-442-9000 x6211

I'm guessing but I think it may have to do with having dropped and recreated the tables while replication was offline. If the primary checks out OK, I would just take a clean archive and restore it to the secondary and bootstrap replication from scratch. Then start plannint to upgrade ;-) Art On Sat, Jul 19, 2008 at 10:12 PM, Tim Ertl <tim@lmrgroup.com> wrote: > I think I have a little problem. As I am about to upgrade to a new server > and my old secondary production system is giving me grief (does it know I > am > about to replace it?). The problem is that this is on ids 9.30.uc1. To old > for support ( I think ). > > I used my secondary HDR server to test the myexport so I did not bog down > the primary server. When I went to restore everything went perfect. Then I > went to link the two servers (onmode -d .) and the HDR log recovery got > into > trouble. I got 40 or 50 of these errors. I googled them and found some > stuff > about bad disk drives and other problems but nothing related to HDR log > recovery. The onstat -d does not show anything unusual like downed chunks. > I > did an oncheck -pe on the primary and had no errors. I did an oncheck -pe > on > the secondary and it had problems in the rootdbs? I don't put anything in > the rootdbs and besides I just restored it clean (onbar & netbackup) so how > could it have an error? The logs only show these lost log records. Dbaccess > on the secondary looks ok also. I learned a little about deciphering logs > at > the Kansas IIUG but I am not sure I can put it to real use. (Nice to know > but hard to use). > > I am assuming that failures like this are bad :-) and I have lost data but > what I don't get is how do I find what this message means? I assume it will > point me to the error. > > Any thoughts? > > I might call IBM on Monday, they have been good to me so perhaps they can > tell me where to look up this error. > > Thanks for your help! > > ,,,Tim > > 21:30:26 Maximum server connections 0 > > 21:30:26 Rollforward of log record failed. iserrno = 101 > > 21:30:26 Log Record: log = 7609, pos = 0x67040, type = OLDRSAM:UNIQID(17), > tran > > s = 257 > > 21:30:26 Rollforward of log record failed. iserrno = 135 > > 21:30:26 Log Record: log = 7609, pos = 0x670e8, type = OLDRSAM:ADDITEM(28), > tra > > ns = 257 > > 21:30:29 Rollforward of log record failed. iserrno = 135 > > 21:30:29 Log Record: log = 7610, pos = 0xc0e4, type = OLDRSAM:ADDITEM(28), > tran > > s = 37 > > 21:30:29 Rollforward of log record failed. iserrno = 135 > > 21:30:29 Log Record: log = 7610, pos = 0xc0e4, type = OLDRSAM:ADDITEM(28), > tran > > s = 37 > > 21:30:29 Rollforward of log record failed. iserrno = 101 > > 21:30:29 Log Record: log = 7610, pos = 0xc150, type = OLDRSAM:HINSERT(40), > tran > > s = 37 > > 21:30:29 Rollforward of log record failed. iserrno = 101 > > 21:30:29 Log Record: log = 7610, pos = 0xc150, type = OLDRSAM:HINSERT(40), > tran > > s = 37 > > 21:30:30 Rollforward of log record failed. iserrno = 135 > > 21:30:30 Log Record: log = 7610, pos = 0xc11c, type = OLDRSAM:ADDITEM(28), > tran > > s = 37 > > Tim Ertl > > 413-442-9000 x6211 > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves.

Thanks Art, I will be trying that shortly. Yesterday I did a fresh backup and restored into the secondary clean. I had removed a couple of tables the day before but that should not affect it if I took a backup afterwards. Strange that after 8 years all of a sudden I have problems with HDR. Don't you take weekends off? :-) Again Many thanks! Tim Ertl 413-442-9000 x6211 -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Art Kagel Sent: Sunday, July 20, 2008 10:11 AM To: ids@iiug.org Subject: Re: rollback fails Secondary HDR? [12802] I'm guessing but I think it may have to do with having dropped and recreated the tables while replication was offline. If the primary checks out OK, I would just take a clean archive and restore it to the secondary and bootstrap replication from scratch. Then start plannint to upgrade ;-) Art On Sat, Jul 19, 2008 at 10:12 PM, Tim Ertl <tim@lmrgroup.com> wrote: > I think I have a little problem. As I am about to upgrade to a new server > and my old secondary production system is giving me grief (does it know I > am > about to replace it?). The problem is that this is on ids 9.30.uc1. To old > for support ( I think ). > > I used my secondary HDR server to test the myexport so I did not bog down > the primary server. When I went to restore everything went perfect. Then I > went to link the two servers (onmode -d .) and the HDR log recovery got > into > trouble. I got 40 or 50 of these errors. I googled them and found some > stuff > about bad disk drives and other problems but nothing related to HDR log > recovery. The onstat -d does not show anything unusual like downed chunks. > I > did an oncheck -pe on the primary and had no errors. I did an oncheck -pe > on > the secondary and it had problems in the rootdbs? I don't put anything in > the rootdbs and besides I just restored it clean (onbar & netbackup) so how > could it have an error? The logs only show these lost log records. Dbaccess > on the secondary looks ok also. I learned a little about deciphering logs > at > the Kansas IIUG but I am not sure I can put it to real use. (Nice to know > but hard to use). > > I am assuming that failures like this are bad :-) and I have lost data but > what I don't get is how do I find what this message means? I assume it will > point me to the error. > > Any thoughts? > > I might call IBM on Monday, they have been good to me so perhaps they can > tell me where to look up this error. > > Thanks for your help! > > ,,,Tim > > 21:30:26 Maximum server connections 0 > > 21:30:26 Rollforward of log record failed. iserrno = 101 > > 21:30:26 Log Record: log = 7609, pos = 0x67040, type = OLDRSAM:UNIQID(17), > tran > > s = 257 > > 21:30:26 Rollforward of log record failed. iserrno = 135 > > 21:30:26 Log Record: log = 7609, pos = 0x670e8, type = OLDRSAM:ADDITEM(28), > tra > > ns = 257 > > 21:30:29 Rollforward of log record failed. iserrno = 135 > > 21:30:29 Log Record: log = 7610, pos = 0xc0e4, type = OLDRSAM:ADDITEM(28), > tran > > s = 37 > > 21:30:29 Rollforward of log record failed. iserrno = 135 > > 21:30:29 Log Record: log = 7610, pos = 0xc0e4, type = OLDRSAM:ADDITEM(28), > tran > > s = 37 > > 21:30:29 Rollforward of log record failed. iserrno = 101 > > 21:30:29 Log Record: log = 7610, pos = 0xc150, type = OLDRSAM:HINSERT(40), > tran > > s = 37 > > 21:30:29 Rollforward of log record failed. iserrno = 101 > > 21:30:29 Log Record: log = 7610, pos = 0xc150, type = OLDRSAM:HINSERT(40), > tran > > s = 37 > > 21:30:30 Rollforward of log record failed. iserrno = 135 > > 21:30:30 Log Record: log = 7610, pos = 0xc11c, type = OLDRSAM:ADDITEM(28), > tran > > s = 37 > > Tim Ertl > > 413-442-9000 x6211 > > > > **************************************************************************** *** > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. **************************************************************************** *** Forum Note: Use "Reply" to post a response in the discussion forum.

On call for client support this W/E and preparing for a bus trip, so... Art On Sun, Jul 20, 2008 at 11:02 AM, Tim Ertl <tim@lmrgroup.com> wrote: > Thanks Art, I will be trying that shortly. Yesterday I did a fresh backup > and restored into the secondary clean. I had removed a couple of tables the > day before but that should not affect it if I took a backup afterwards. > Strange that after 8 years all of a sudden I have problems with HDR. > > Don't you take weekends off? :-) > > Again Many thanks! > > Tim Ertl > 413-442-9000 x6211 > > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Art > Kagel > Sent: Sunday, July 20, 2008 10:11 AM > To: ids@iiug.org > Subject: Re: rollback fails Secondary HDR? [12802] > > I'm guessing but I think it may have to do with having dropped and > recreated > > the tables while replication was offline. If the primary checks out OK, I > would just take a clean archive and restore it to the secondary and > bootstrap replication from scratch. Then start plannint to upgrade ;-) > > Art > > On Sat, Jul 19, 2008 at 10:12 PM, Tim Ertl <tim@lmrgroup.com> wrote: > > > I think I have a little problem. As I am about to upgrade to a new server > > and my old secondary production system is giving me grief (does it know I > > am > > about to replace it?). The problem is that this is on ids 9.30.uc1. To > old > > > for support ( I think ). > > > > I used my secondary HDR server to test the myexport so I did not bog down > > the primary server. When I went to restore everything went perfect. Then > I > > > went to link the two servers (onmode -d .) and the HDR log recovery got > > into > > trouble. I got 40 or 50 of these errors. I googled them and found some > > stuff > > about bad disk drives and other problems but nothing related to HDR log > > recovery. The onstat -d does not show anything unusual like downed > chunks. > > > I > > did an oncheck -pe on the primary and had no errors. I did an oncheck -pe > > on > > the secondary and it had problems in the rootdbs? I don't put anything in > > the rootdbs and besides I just restored it clean (onbar & netbackup) so > how > > could it have an error? The logs only show these lost log records. > Dbaccess > > on the secondary looks ok also. I learned a little about deciphering logs > > at > > the Kansas IIUG but I am not sure I can put it to real use. (Nice to know > > but hard to use). > > > > I am assuming that failures like this are bad :-) and I have lost data > but > > > what I don't get is how do I find what this message means? I assume it > will > > point me to the error. > > > > Any thoughts? > > > > I might call IBM on Monday, they have been good to me so perhaps they can > > tell me where to look up this error. > > > > Thanks for your help! > > > > ,,,Tim > > > > 21:30:26 Maximum server connections 0 > > > > 21:30:26 Rollforward of log record failed. iserrno = 101 > > > > 21:30:26 Log Record: log = 7609, pos = 0x67040, type = > OLDRSAM:UNIQID(17), > > > tran > > > > s = 257 > > > > 21:30:26 Rollforward of log record failed. iserrno = 135 > > > > 21:30:26 Log Record: log = 7609, pos = 0x670e8, type = > OLDRSAM:ADDITEM(28), > > tra > > > > ns = 257 > > > > 21:30:29 Rollforward of log record failed. iserrno = 135 > > > > 21:30:29 Log Record: log = 7610, pos = 0xc0e4, type = > OLDRSAM:ADDITEM(28), > > > tran > > > > s = 37 > > > > 21:30:29 Rollforward of log record failed. iserrno = 135 > > > > 21:30:29 Log Record: log = 7610, pos = 0xc0e4, type = > OLDRSAM:ADDITEM(28), > > > tran > > > > s = 37 > > > > 21:30:29 Rollforward of log record failed. iserrno = 101 > > > > 21:30:29 Log Record: log = 7610, pos = 0xc150, type = > OLDRSAM:HINSERT(40), > > > tran > > > > s = 37 > > > > 21:30:29 Rollforward of log record failed. iserrno = 101 > > > > 21:30:29 Log Record: log = 7610, pos = 0xc150, type = > OLDRSAM:HINSERT(40), > > > tran > > > > s = 37 > > > > 21:30:30 Rollforward of log record failed. iserrno = 135 > > > > 21:30:30 Log Record: log = 7610, pos = 0xc11c, type = > OLDRSAM:ADDITEM(28), > > > tran > > > > s = 37 > > > > Tim Ertl > > > > 413-442-9000 x6211 > > > > > > > > > > **************************************************************************** > *** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > -- > Art S. Kagel > Oninit (www.oninit.com) > IIUG Board of Directors (art@iiug.org) > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and > > do not reflect on my employer, Oninit, the IIUG, nor any other organization > with which I am associated either explicitly or implicitly. Neither do > those > > opinions reflect those of other individuals affiliated with any entity with > which I am affiliated nor those of the entities themselves. > > > **************************************************************************** > *** > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves.

Art, I only have silver support and this is my secondary so it is not mission critical. I am running disk analyze to see if something is wrong on the disks. After a second primary backup/secondary restore and HDR start it did the same thing. I observed rollforward errors happening at the end of each logical log. Thanks for all your help! You have been great! Have a good trip. I really like the myexport despite the "my" :-). ,,,Tim Tim Ertl 413-442-9000 x6211 -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Art Kagel Sent: Sunday, July 20, 2008 12:27 PM To: ids@iiug.org Subject: Re: rollback fails Secondary HDR? [12808] On call for client support this W/E and preparing for a bus trip, so... Art On Sun, Jul 20, 2008 at 11:02 AM, Tim Ertl <tim@lmrgroup.com> wrote: > Thanks Art, I will be trying that shortly. Yesterday I did a fresh backup > and restored into the secondary clean. I had removed a couple of tables the > day before but that should not affect it if I took a backup afterwards. > Strange that after 8 years all of a sudden I have problems with HDR. > > Don't you take weekends off? :-) > > Again Many thanks! > > Tim Ertl > 413-442-9000 x6211 > > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Art > Kagel > Sent: Sunday, July 20, 2008 10:11 AM > To: ids@iiug.org > Subject: Re: rollback fails Secondary HDR? [12802] > > I'm guessing but I think it may have to do with having dropped and > recreated > > the tables while replication was offline. If the primary checks out OK, I > would just take a clean archive and restore it to the secondary and > bootstrap replication from scratch. Then start plannint to upgrade ;-) > > Art > > On Sat, Jul 19, 2008 at 10:12 PM, Tim Ertl <tim@lmrgroup.com> wrote: > > > I think I have a little problem. As I am about to upgrade to a new server > > and my old secondary production system is giving me grief (does it know I > > am > > about to replace it?). The problem is that this is on ids 9.30.uc1. To > old > > > for support ( I think ). > > > > I used my secondary HDR server to test the myexport so I did not bog down > > the primary server. When I went to restore everything went perfect. Then > I > > > went to link the two servers (onmode -d .) and the HDR log recovery got > > into > > trouble. I got 40 or 50 of these errors. I googled them and found some > > stuff > > about bad disk drives and other problems but nothing related to HDR log > > recovery. The onstat -d does not show anything unusual like downed > chunks. > > > I > > did an oncheck -pe on the primary and had no errors. I did an oncheck -pe > > on > > the secondary and it had problems in the rootdbs? I don't put anything in > > the rootdbs and besides I just restored it clean (onbar & netbackup) so > how > > could it have an error? The logs only show these lost log records. > Dbaccess > > on the secondary looks ok also. I learned a little about deciphering logs > > at > > the Kansas IIUG but I am not sure I can put it to real use. (Nice to know > > but hard to use). > > > > I am assuming that failures like this are bad :-) and I have lost data > but > > > what I don't get is how do I find what this message means? I assume it > will > > point me to the error. > > > > Any thoughts? > > > > I might call IBM on Monday, they have been good to me so perhaps they can > > tell me where to look up this error. > > > > Thanks for your help! > > > > ,,,Tim > > > > 21:30:26 Maximum server connections 0 > > > > 21:30:26 Rollforward of log record failed. iserrno = 101 > > > > 21:30:26 Log Record: log = 7609, pos = 0x67040, type = > OLDRSAM:UNIQID(17), > > > tran > > > > s = 257 > > > > 21:30:26 Rollforward of log record failed. iserrno = 135 > > > > 21:30:26 Log Record: log = 7609, pos = 0x670e8, type = > OLDRSAM:ADDITEM(28), > > tra > > > > ns = 257 > > > > 21:30:29 Rollforward of log record failed. iserrno = 135 > > > > 21:30:29 Log Record: log = 7610, pos = 0xc0e4, type = > OLDRSAM:ADDITEM(28), > > > tran > > > > s = 37 > > > > 21:30:29 Rollforward of log record failed. iserrno = 135 > > > > 21:30:29 Log Record: log = 7610, pos = 0xc0e4, type = > OLDRSAM:ADDITEM(28), > > > tran > > > > s = 37 > > > > 21:30:29 Rollforward of log record failed. iserrno = 101 > > > > 21:30:29 Log Record: log = 7610, pos = 0xc150, type = > OLDRSAM:HINSERT(40), > > > tran > > > > s = 37 > > > > 21:30:29 Rollforward of log record failed. iserrno = 101 > > > > 21:30:29 Log Record: log = 7610, pos = 0xc150, type = > OLDRSAM:HINSERT(40), > > > tran > > > > s = 37 > > > > 21:30:30 Rollforward of log record failed. iserrno = 135 > > > > 21:30:30 Log Record: log = 7610, pos = 0xc11c, type = > OLDRSAM:ADDITEM(28), > > > tran > > > > s = 37 > > > > Tim Ertl > > > > 413-442-9000 x6211 > > > > > > > > > > **************************************************************************** > *** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > -- > Art S. Kagel > Oninit (www.oninit.com) > IIUG Board of Directors (art@iiug.org) > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and > > do not reflect on my employer, Oninit, the IIUG, nor any other organization > with which I am associated either explicitly or implicitly. Neither do > those > > opinions reflect those of other individuals affiliated with any entity with > which I am affiliated nor those of the entities themselves. > > > **************************************************************************** > *** > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > **************************************************************************** *** > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. **************************************************************************** *** Forum Note: Use "Reply" to post a response in the discussion forum.