rollback fails Secondary HDR?
Tim Ertl's HDR secondary on IDS 9.30.UC1 logged dozens of 'Rollforward of log record failed' errors (iserrno 101 and 135) after restore and linking. Art Kagel guessed dropped and recreated tables while replication was offline and advised a clean archive and restore. Tim retried and the errors recurred at the end of each log; he planned a disk analysis. Unresolved.
Auto-generated by Claude from the posts below — may be imperfect; read the full thread.
I think I have a little problem. As I am about to upgrade to a new server
and my old secondary production system is giving me grief (does it know I am
about to replace it?). The problem is that this is on ids 9.30.uc1. To old
for support ( I think ).
I used my secondary HDR server to test the myexport so I did not bog down
the primary server. When I went to restore everything went perfect. Then I
went to link the two servers (onmode -d .) and the HDR log recovery got into
trouble. I got 40 or 50 of these errors. I googled them and found some stuff
about bad disk drives and other problems but nothing related to HDR log
recovery. The onstat -d does not show anything unusual like downed chunks. I
did an oncheck -pe on the primary and had no errors. I did an oncheck -pe on
the secondary and it had problems in the rootdbs? I don't put anything in
the rootdbs and besides I just restored it clean (onbar & netbackup) so how
could it have an error? The logs only show these lost log records. Dbaccess
on the secondary looks ok also. I learned a little about deciphering logs at
the Kansas IIUG but I am not sure I can put it to real use. (Nice to know
but hard to use).
I am assuming that failures like this are bad :-) and I have lost data but
what I don't get is how do I find what this message means? I assume it will
point me to the error.
Any thoughts?
I might call IBM on Monday, they have been good to me so perhaps they can
tell me where to look up this error.
Thanks for your help!
,,,Tim
21:30:26 Maximum server connections 0
21:30:26 Rollforward of log record failed. iserrno = 101
21:30:26 Log Record: log = 7609, pos = 0x67040, type = OLDRSAM:UNIQID(17),
tran
s = 257
21:30:26 Rollforward of log record failed. iserrno = 135
21:30:26 Log Record: log = 7609, pos = 0x670e8, type = OLDRSAM:ADDITEM(28),
tra
ns = 257
21:30:29 Rollforward of log record failed. iserrno = 135
21:30:29 Log Record: log = 7610, pos = 0xc0e4, type = OLDRSAM:ADDITEM(28),
tran
s = 37
21:30:29 Rollforward of log record failed. iserrno = 135
21:30:29 Log Record: log = 7610, pos = 0xc0e4, type = OLDRSAM:ADDITEM(28),
tran
s = 37
21:30:29 Rollforward of log record failed. iserrno = 101
21:30:29 Log Record: log = 7610, pos = 0xc150, type = OLDRSAM:HINSERT(40),
tran
s = 37
21:30:29 Rollforward of log record failed. iserrno = 101
21:30:29 Log Record: log = 7610, pos = 0xc150, type = OLDRSAM:HINSERT(40),
tran
s = 37
21:30:30 Rollforward of log record failed. iserrno = 135
21:30:30 Log Record: log = 7610, pos = 0xc11c, type = OLDRSAM:ADDITEM(28),
tran
s = 37
Tim Ertl
413-442-9000 x6211
I'm guessing but I think it may have to do with having dropped and recreated
the tables while replication was offline. If the primary checks out OK, I
would just take a clean archive and restore it to the secondary and
bootstrap replication from scratch. Then start plannint to upgrade ;-)
Art
On Sat, Jul 19, 2008 at 10:12 PM, Tim Ertl <tim@lmrgroup.com> wrote:
> I think I have a little problem. As I am about to upgrade to a new server
> and my old secondary production system is giving me grief (does it know I
> am
> about to replace it?). The problem is that this is on ids 9.30.uc1. To old
> for support ( I think ).
>
> I used my secondary HDR server to test the myexport so I did not bog down
> the primary server. When I went to restore everything went perfect. Then I
> went to link the two servers (onmode -d .) and the HDR log recovery got
> into
> trouble. I got 40 or 50 of these errors. I googled them and found some
> stuff
> about bad disk drives and other problems but nothing related to HDR log
> recovery. The onstat -d does not show anything unusual like downed chunks.
> I
> did an oncheck -pe on the primary and had no errors. I did an oncheck -pe
> on
> the secondary and it had problems in the rootdbs? I don't put anything in
> the rootdbs and besides I just restored it clean (onbar & netbackup) so how
> could it have an error? The logs only show these lost log records. Dbaccess
> on the secondary looks ok also. I learned a little about deciphering logs
> at
> the Kansas IIUG but I am not sure I can put it to real use. (Nice to know
> but hard to use).
>
> I am assuming that failures like this are bad :-) and I have lost data but
> what I don't get is how do I find what this message means? I assume it will
> point me to the error.
>
> Any thoughts?
>
> I might call IBM on Monday, they have been good to me so perhaps they can
> tell me where to look up this error.
>
> Thanks for your help!
>
> ,,,Tim
>
> 21:30:26 Maximum server connections 0
>
> 21:30:26 Rollforward of log record failed. iserrno = 101
>
> 21:30:26 Log Record: log = 7609, pos = 0x67040, type = OLDRSAM:UNIQID(17),
> tran
>
> s = 257
>
> 21:30:26 Rollforward of log record failed. iserrno = 135
>
> 21:30:26 Log Record: log = 7609, pos = 0x670e8, type = OLDRSAM:ADDITEM(28),
> tra
>
> ns = 257
>
> 21:30:29 Rollforward of log record failed. iserrno = 135
>
> 21:30:29 Log Record: log = 7610, pos = 0xc0e4, type = OLDRSAM:ADDITEM(28),
> tran
>
> s = 37
>
> 21:30:29 Rollforward of log record failed. iserrno = 135
>
> 21:30:29 Log Record: log = 7610, pos = 0xc0e4, type = OLDRSAM:ADDITEM(28),
> tran
>
> s = 37
>
> 21:30:29 Rollforward of log record failed. iserrno = 101
>
> 21:30:29 Log Record: log = 7610, pos = 0xc150, type = OLDRSAM:HINSERT(40),
> tran
>
> s = 37
>
> 21:30:29 Rollforward of log record failed. iserrno = 101
>
> 21:30:29 Log Record: log = 7610, pos = 0xc150, type = OLDRSAM:HINSERT(40),
> tran
>
> s = 37
>
> 21:30:30 Rollforward of log record failed. iserrno = 135
>
> 21:30:30 Log Record: log = 7610, pos = 0xc11c, type = OLDRSAM:ADDITEM(28),
> tran
>
> s = 37
>
> Tim Ertl
>
> 413-442-9000 x6211
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--
Art S. Kagel
Oninit (www.oninit.com)
IIUG Board of Directors (art@iiug.org)
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Oninit, the IIUG, nor any other organization
with which I am associated either explicitly or implicitly. Neither do those
opinions reflect those of other individuals affiliated with any entity with
which I am affiliated nor those of the entities themselves.
Thanks Art, I will be trying that shortly. Yesterday I did a fresh backup
and restored into the secondary clean. I had removed a couple of tables the
day before but that should not affect it if I took a backup afterwards.
Strange that after 8 years all of a sudden I have problems with HDR.
Don't you take weekends off? :-)
Again Many thanks!
Tim Ertl
413-442-9000 x6211
-----Original Message-----
From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Art
Kagel
Sent: Sunday, July 20, 2008 10:11 AM
To: ids@iiug.org
Subject: Re: rollback fails Secondary HDR? [12802]
I'm guessing but I think it may have to do with having dropped and recreated
the tables while replication was offline. If the primary checks out OK, I
would just take a clean archive and restore it to the secondary and
bootstrap replication from scratch. Then start plannint to upgrade ;-)
Art
On Sat, Jul 19, 2008 at 10:12 PM, Tim Ertl <tim@lmrgroup.com> wrote:
> I think I have a little problem. As I am about to upgrade to a new server
> and my old secondary production system is giving me grief (does it know I
> am
> about to replace it?). The problem is that this is on ids 9.30.uc1. To old
> for support ( I think ).
>
> I used my secondary HDR server to test the myexport so I did not bog down
> the primary server. When I went to restore everything went perfect. Then I
> went to link the two servers (onmode -d .) and the HDR log recovery got
> into
> trouble. I got 40 or 50 of these errors. I googled them and found some
> stuff
> about bad disk drives and other problems but nothing related to HDR log
> recovery. The onstat -d does not show anything unusual like downed chunks.
> I
> did an oncheck -pe on the primary and had no errors. I did an oncheck -pe
> on
> the secondary and it had problems in the rootdbs? I don't put anything in
> the rootdbs and besides I just restored it clean (onbar & netbackup) so
how
> could it have an error? The logs only show these lost log records.
Dbaccess
> on the secondary looks ok also. I learned a little about deciphering logs
> at
> the Kansas IIUG but I am not sure I can put it to real use. (Nice to know
> but hard to use).
>
> I am assuming that failures like this are bad :-) and I have lost data but
> what I don't get is how do I find what this message means? I assume it
will
> point me to the error.
>
> Any thoughts?
>
> I might call IBM on Monday, they have been good to me so perhaps they can
> tell me where to look up this error.
>
> Thanks for your help!
>
> ,,,Tim
>
> 21:30:26 Maximum server connections 0
>
> 21:30:26 Rollforward of log record failed. iserrno = 101
>
> 21:30:26 Log Record: log = 7609, pos = 0x67040, type = OLDRSAM:UNIQID(17),
> tran
>
> s = 257
>
> 21:30:26 Rollforward of log record failed. iserrno = 135
>
> 21:30:26 Log Record: log = 7609, pos = 0x670e8, type =
OLDRSAM:ADDITEM(28),
> tra
>
> ns = 257
>
> 21:30:29 Rollforward of log record failed. iserrno = 135
>
> 21:30:29 Log Record: log = 7610, pos = 0xc0e4, type = OLDRSAM:ADDITEM(28),
> tran
>
> s = 37
>
> 21:30:29 Rollforward of log record failed. iserrno = 135
>
> 21:30:29 Log Record: log = 7610, pos = 0xc0e4, type = OLDRSAM:ADDITEM(28),
> tran
>
> s = 37
>
> 21:30:29 Rollforward of log record failed. iserrno = 101
>
> 21:30:29 Log Record: log = 7610, pos = 0xc150, type = OLDRSAM:HINSERT(40),
> tran
>
> s = 37
>
> 21:30:29 Rollforward of log record failed. iserrno = 101
>
> 21:30:29 Log Record: log = 7610, pos = 0xc150, type = OLDRSAM:HINSERT(40),
> tran
>
> s = 37
>
> 21:30:30 Rollforward of log record failed. iserrno = 135
>
> 21:30:30 Log Record: log = 7610, pos = 0xc11c, type = OLDRSAM:ADDITEM(28),
> tran
>
> s = 37
>
> Tim Ertl
>
> 413-442-9000 x6211
>
>
>
>
****************************************************************************
***
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--
Art S. Kagel
Oninit (www.oninit.com)
IIUG Board of Directors (art@iiug.org)
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Oninit, the IIUG, nor any other organization
with which I am associated either explicitly or implicitly. Neither do those
opinions reflect those of other individuals affiliated with any entity with
which I am affiliated nor those of the entities themselves.
****************************************************************************
***
Forum Note: Use "Reply" to post a response in the discussion forum.
On call for client support this W/E and preparing for a bus trip, so...
Art
On Sun, Jul 20, 2008 at 11:02 AM, Tim Ertl <tim@lmrgroup.com> wrote:
> Thanks Art, I will be trying that shortly. Yesterday I did a fresh backup
> and restored into the secondary clean. I had removed a couple of tables the
> day before but that should not affect it if I took a backup afterwards.
> Strange that after 8 years all of a sudden I have problems with HDR.
>
> Don't you take weekends off? :-)
>
> Again Many thanks!
>
> Tim Ertl
> 413-442-9000 x6211
>
> -----Original Message-----
> From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Art
> Kagel
> Sent: Sunday, July 20, 2008 10:11 AM
> To: ids@iiug.org
> Subject: Re: rollback fails Secondary HDR? [12802]
>
> I'm guessing but I think it may have to do with having dropped and
> recreated
>
> the tables while replication was offline. If the primary checks out OK, I
> would just take a clean archive and restore it to the secondary and
> bootstrap replication from scratch. Then start plannint to upgrade ;-)
>
> Art
>
> On Sat, Jul 19, 2008 at 10:12 PM, Tim Ertl <tim@lmrgroup.com> wrote:
>
> > I think I have a little problem. As I am about to upgrade to a new server
> > and my old secondary production system is giving me grief (does it know I
> > am
> > about to replace it?). The problem is that this is on ids 9.30.uc1. To
> old
>
> > for support ( I think ).
> >
> > I used my secondary HDR server to test the myexport so I did not bog down
> > the primary server. When I went to restore everything went perfect. Then
> I
>
> > went to link the two servers (onmode -d .) and the HDR log recovery got
> > into
> > trouble. I got 40 or 50 of these errors. I googled them and found some
> > stuff
> > about bad disk drives and other problems but nothing related to HDR log
> > recovery. The onstat -d does not show anything unusual like downed
> chunks.
>
> > I
> > did an oncheck -pe on the primary and had no errors. I did an oncheck -pe
> > on
> > the secondary and it had problems in the rootdbs? I don't put anything in
> > the rootdbs and besides I just restored it clean (onbar & netbackup) so
> how
> > could it have an error? The logs only show these lost log records.
> Dbaccess
> > on the secondary looks ok also. I learned a little about deciphering logs
> > at
> > the Kansas IIUG but I am not sure I can put it to real use. (Nice to know
> > but hard to use).
> >
> > I am assuming that failures like this are bad :-) and I have lost data
> but
>
> > what I don't get is how do I find what this message means? I assume it
> will
> > point me to the error.
> >
> > Any thoughts?
> >
> > I might call IBM on Monday, they have been good to me so perhaps they can
> > tell me where to look up this error.
> >
> > Thanks for your help!
> >
> > ,,,Tim
> >
> > 21:30:26 Maximum server connections 0
> >
> > 21:30:26 Rollforward of log record failed. iserrno = 101
> >
> > 21:30:26 Log Record: log = 7609, pos = 0x67040, type =
> OLDRSAM:UNIQID(17),
>
> > tran
> >
> > s = 257
> >
> > 21:30:26 Rollforward of log record failed. iserrno = 135
> >
> > 21:30:26 Log Record: log = 7609, pos = 0x670e8, type =
> OLDRSAM:ADDITEM(28),
> > tra
> >
> > ns = 257
> >
> > 21:30:29 Rollforward of log record failed. iserrno = 135
> >
> > 21:30:29 Log Record: log = 7610, pos = 0xc0e4, type =
> OLDRSAM:ADDITEM(28),
>
> > tran
> >
> > s = 37
> >
> > 21:30:29 Rollforward of log record failed. iserrno = 135
> >
> > 21:30:29 Log Record: log = 7610, pos = 0xc0e4, type =
> OLDRSAM:ADDITEM(28),
>
> > tran
> >
> > s = 37
> >
> > 21:30:29 Rollforward of log record failed. iserrno = 101
> >
> > 21:30:29 Log Record: log = 7610, pos = 0xc150, type =
> OLDRSAM:HINSERT(40),
>
> > tran
> >
> > s = 37
> >
> > 21:30:29 Rollforward of log record failed. iserrno = 101
> >
> > 21:30:29 Log Record: log = 7610, pos = 0xc150, type =
> OLDRSAM:HINSERT(40),
>
> > tran
> >
> > s = 37
> >
> > 21:30:30 Rollforward of log record failed. iserrno = 135
> >
> > 21:30:30 Log Record: log = 7610, pos = 0xc11c, type =
> OLDRSAM:ADDITEM(28),
>
> > tran
> >
> > s = 37
> >
> > Tim Ertl
> >
> > 413-442-9000 x6211
> >
> >
> >
> >
>
> ****************************************************************************
> ***
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
>
> --
> Art S. Kagel
> Oninit (www.oninit.com)
> IIUG Board of Directors (art@iiug.org)
>
> Disclaimer: Please keep in mind that my own opinions are my own opinions
> and
>
> do not reflect on my employer, Oninit, the IIUG, nor any other organization
> with which I am associated either explicitly or implicitly. Neither do
> those
>
> opinions reflect those of other individuals affiliated with any entity with
> which I am affiliated nor those of the entities themselves.
>
>
> ****************************************************************************
> ***
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--
Art S. Kagel
Oninit (www.oninit.com)
IIUG Board of Directors (art@iiug.org)
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Oninit, the IIUG, nor any other organization
with which I am associated either explicitly or implicitly. Neither do those
opinions reflect those of other individuals affiliated with any entity with
which I am affiliated nor those of the entities themselves.
Art, I only have silver support and this is my secondary so it is not
mission critical. I am running disk analyze to see if something is wrong on
the disks. After a second primary backup/secondary restore and HDR start it
did the same thing. I observed rollforward errors happening at the end of
each logical log.
Thanks for all your help! You have been great! Have a good trip. I really
like the myexport despite the "my" :-).
,,,Tim
Tim Ertl
413-442-9000 x6211
-----Original Message-----
From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Art
Kagel
Sent: Sunday, July 20, 2008 12:27 PM
To: ids@iiug.org
Subject: Re: rollback fails Secondary HDR? [12808]
On call for client support this W/E and preparing for a bus trip, so...
Art
On Sun, Jul 20, 2008 at 11:02 AM, Tim Ertl <tim@lmrgroup.com> wrote:
> Thanks Art, I will be trying that shortly. Yesterday I did a fresh backup
> and restored into the secondary clean. I had removed a couple of tables
the
> day before but that should not affect it if I took a backup afterwards.
> Strange that after 8 years all of a sudden I have problems with HDR.
>
> Don't you take weekends off? :-)
>
> Again Many thanks!
>
> Tim Ertl
> 413-442-9000 x6211
>
> -----Original Message-----
> From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Art
> Kagel
> Sent: Sunday, July 20, 2008 10:11 AM
> To: ids@iiug.org
> Subject: Re: rollback fails Secondary HDR? [12802]
>
> I'm guessing but I think it may have to do with having dropped and
> recreated
>
> the tables while replication was offline. If the primary checks out OK, I
> would just take a clean archive and restore it to the secondary and
> bootstrap replication from scratch. Then start plannint to upgrade ;-)
>
> Art
>
> On Sat, Jul 19, 2008 at 10:12 PM, Tim Ertl <tim@lmrgroup.com> wrote:
>
> > I think I have a little problem. As I am about to upgrade to a new
server
> > and my old secondary production system is giving me grief (does it know
I
> > am
> > about to replace it?). The problem is that this is on ids 9.30.uc1. To
> old
>
> > for support ( I think ).
> >
> > I used my secondary HDR server to test the myexport so I did not bog
down
> > the primary server. When I went to restore everything went perfect. Then
> I
>
> > went to link the two servers (onmode -d .) and the HDR log recovery got
> > into
> > trouble. I got 40 or 50 of these errors. I googled them and found some
> > stuff
> > about bad disk drives and other problems but nothing related to HDR log
> > recovery. The onstat -d does not show anything unusual like downed
> chunks.
>
> > I
> > did an oncheck -pe on the primary and had no errors. I did an oncheck
-pe
> > on
> > the secondary and it had problems in the rootdbs? I don't put anything
in
> > the rootdbs and besides I just restored it clean (onbar & netbackup) so
> how
> > could it have an error? The logs only show these lost log records.
> Dbaccess
> > on the secondary looks ok also. I learned a little about deciphering
logs
> > at
> > the Kansas IIUG but I am not sure I can put it to real use. (Nice to
know
> > but hard to use).
> >
> > I am assuming that failures like this are bad :-) and I have lost data
> but
>
> > what I don't get is how do I find what this message means? I assume it
> will
> > point me to the error.
> >
> > Any thoughts?
> >
> > I might call IBM on Monday, they have been good to me so perhaps they
can
> > tell me where to look up this error.
> >
> > Thanks for your help!
> >
> > ,,,Tim
> >
> > 21:30:26 Maximum server connections 0
> >
> > 21:30:26 Rollforward of log record failed. iserrno = 101
> >
> > 21:30:26 Log Record: log = 7609, pos = 0x67040, type =
> OLDRSAM:UNIQID(17),
>
> > tran
> >
> > s = 257
> >
> > 21:30:26 Rollforward of log record failed. iserrno = 135
> >
> > 21:30:26 Log Record: log = 7609, pos = 0x670e8, type =
> OLDRSAM:ADDITEM(28),
> > tra
> >
> > ns = 257
> >
> > 21:30:29 Rollforward of log record failed. iserrno = 135
> >
> > 21:30:29 Log Record: log = 7610, pos = 0xc0e4, type =
> OLDRSAM:ADDITEM(28),
>
> > tran
> >
> > s = 37
> >
> > 21:30:29 Rollforward of log record failed. iserrno = 135
> >
> > 21:30:29 Log Record: log = 7610, pos = 0xc0e4, type =
> OLDRSAM:ADDITEM(28),
>
> > tran
> >
> > s = 37
> >
> > 21:30:29 Rollforward of log record failed. iserrno = 101
> >
> > 21:30:29 Log Record: log = 7610, pos = 0xc150, type =
> OLDRSAM:HINSERT(40),
>
> > tran
> >
> > s = 37
> >
> > 21:30:29 Rollforward of log record failed. iserrno = 101
> >
> > 21:30:29 Log Record: log = 7610, pos = 0xc150, type =
> OLDRSAM:HINSERT(40),
>
> > tran
> >
> > s = 37
> >
> > 21:30:30 Rollforward of log record failed. iserrno = 135
> >
> > 21:30:30 Log Record: log = 7610, pos = 0xc11c, type =
> OLDRSAM:ADDITEM(28),
>
> > tran
> >
> > s = 37
> >
> > Tim Ertl
> >
> > 413-442-9000 x6211
> >
> >
> >
> >
>
>
****************************************************************************
> ***
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
>
> --
> Art S. Kagel
> Oninit (www.oninit.com)
> IIUG Board of Directors (art@iiug.org)
>
> Disclaimer: Please keep in mind that my own opinions are my own opinions
> and
>
> do not reflect on my employer, Oninit, the IIUG, nor any other
organization
> with which I am associated either explicitly or implicitly. Neither do
> those
>
> opinions reflect those of other individuals affiliated with any entity
with
> which I am affiliated nor those of the entities themselves.
>
>
>
****************************************************************************
> ***
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
>
>
****************************************************************************
***
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--
Art S. Kagel
Oninit (www.oninit.com)
IIUG Board of Directors (art@iiug.org)
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Oninit, the IIUG, nor any other organization
with which I am associated either explicitly or implicitly. Neither do those
opinions reflect those of other individuals affiliated with any entity with
which I am affiliated nor those of the entities themselves.
****************************************************************************
***
Forum Note: Use "Reply" to post a response in the discussion forum.
Related threads
- IDS 10 table-level restore
- Informix Development Webinar December 11, 2007
- ontape -p/r with changed ROOTPATH
- Migrate from HP PA-RISC to HP ITANIUM by ontape