Secondary stuck in Fast Recovery

Ok guys, I need your help again…ugh. So I have this dev setup I’ve been working on. Sounded simple. I have two Version 11.5F9DE instances running with a primary and secondary set with HDR replication running. I wanted to try a FULL and DIFFERENTIAL backup and restore on the primary, and it has been a painful but great learning experience all week. So the backups and restore have now worked. The primary is up and running. HERE IS THE ISSUE: On the secondary, it will not work with oninit. It is stuck in Fast recovery. I messed with it all day yesterday, shutting it down, running onclean -ky when it complains about memory, and I do not know where to go from here. Any ideas? Do I have to restore the secondary if I restore the primary? I was expecting to just restore the primary and away it would go as usual once I had both instances running and HDR running. I ran onmode -kuy on the secondary From onstat - it gives shared memory not initialized…perfect. From onstat -m it gives that it went shutdown mode, quiescent mode, Server stopped…perfect So now I try and restart: oninit onstat - gives back in fast recovery!!!! arrrrrggggghhhhhh It will not come out of this state. I left it all night and tried again the same steps now.

What are your steps and commands? Let's go GreenThis email contains 100% recycled electrons. From: BENJI LONG <ruggedmouse@hotmail.com> To: ids@iiug.org Sent: Friday, September 15, 2017 9:34 AM Subject: Secondary stuck in Fast Recovery [39877] Ok guys, I need your help againugh. So I have this dev setup Ive been working on. Sounded simple. I have two Version 11.5F9DE instances running with a primary and secondary set with HDR replication running. I wanted to try a FULL and DIFFERENTIAL backup and restore on the primary, and it has been a painful but great learning experience all week. So the backups and restore have now worked. The primary is up and running. HERE IS THE ISSUE: On the secondary, it will not work with oninit. It is stuck in Fast recovery. I messed with it all day yesterday, shutting it down, running onclean -ky when it complains about memory, and I do not know where to go from here. Any ideas? Do I have to restore the secondary if I restore the primary? I was expecting to just restore the primary and away it would go as usual once I had both instances running and HDR running. I ran onmode -kuy on the secondary >From onstat - it gives shared memory not initializedperfect. >From onstat -m it gives that it went shutdown mode, quiescent mode, Server stoppedperfect So now I try and restart: oninit onstat - gives back in fast recovery!!!! arrrrrggggghhhhhh It will not come out of this state. I left it all night and tried again the same steps now. ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Hi, when a secondary is stuck in fast recovery, you did not set the secondary role by now, but the server is restored only physically. Did you execute the commands "onmode -d secondary <primary server name>" and "onmode -d primary <secondary server name>" ? The mode should then switch to secondary Marcus Haarmann Von: "BENJI LONG" <ruggedmouse@hotmail.com> An: "ids" <ids@iiug.org> Gesendet: Freitag, 15. September 2017 15:33:41 Betreff: Secondary stuck in Fast Recovery [39877] Ok guys, I need your help againugh. So I have this dev setup Ive been working on. Sounded simple. I have two Version 11.5F9DE instances running with a primary and secondary set with HDR replication running. I wanted to try a FULL and DIFFERENTIAL backup and restore on the primary, and it has been a painful but great learning experience all week. So the backups and restore have now worked. The primary is up and running. HERE IS THE ISSUE: On the secondary, it will not work with oninit. It is stuck in Fast recovery. I messed with it all day yesterday, shutting it down, running onclean -ky when it complains about memory, and I do not know where to go from here. Any ideas? Do I have to restore the secondary if I restore the primary? I was expecting to just restore the primary and away it would go as usual once I had both instances running and HDR running. I ran onmode -kuy on the secondary >From onstat - it gives shared memory not initializedperfect. >From onstat -m it gives that it went shutdown mode, quiescent mode, Server stoppedperfect So now I try and restart: oninit onstat - gives back in fast recovery!!!! arrrrrggggghhhhhh It will not come out of this state. I left it all night and tried again the same steps now. ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

I ran onmode -kuy on the secondary >From onstat - it gives shared memory not initializedperfect. >From onstat -m it gives that it went shutdown mode, quiescent mode, Server stoppedperfect So now I try and restart: oninit onstat - gives back in fast recovery!!!!

Hi Marcus, are you saying that my method will not work, and that I need to restore my secondary? Currently my primary is running in standard mode. I didn't set replication to start as I was waiting to get both instances running. I just ran onmode -d secondary newserver_hdr but it wouldn't start. log says Two servers out of sync and then it stops the instance In the message log, it now says mt_shm_remove: WARNGIN: may not have removed all/correct segments. I seem to be in a vortex. The instance won't start with oninit properly, and it quits and is giving errors that aren't really what is wrong possibly

Hi, Yes, you need to restore the seceondary to get the secondary back in sync. Regards, David. > On 15 September 2017 at 15:11 BENJI LONG <ruggedmouse@hotmail.com> wrote: > > > Hi Marcus, are you saying that my method will not work, and that I need to > restore my secondary? > > Currently my primary is running in standard mode. I didn't set replication to > start as I was waiting to get both instances running. > > I just ran onmode -d secondary newserver_hdr > > but it wouldn't start. log says Two servers out of sync and then it stops the > instance > > In the message log, it now says mt_shm_remove: WARNGIN: may not have removed > all/correct segments. > > I seem to be in a vortex. The instance won't start with oninit properly, and > it quits and is giving errors that aren't really what is wrong possibly > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >

I just want to double check that this is what has happened. I backed up and restored my primary. Does this mean that I also need to restore the secondary, even though I didn't touch it? The log on the secondary when I tried to start HDR is saying two servers out of sync. To re-establish the HDR pair, backup primary and restore to secondary. I thought that I would just restore the primary , and restart HDR and it would send its logs to the secondary and away it would go as normal.

Ok, thanks. I didn't realize that. I will try that now. Wow, cannot believe how much I have learned over the last few days, just trying to do a restore. Amazing help on this forum. appreciate it very much.

Can you provide the exact message? From the message then yes you would need to restore the secondary as well as they are no longer considered in-sync. NOTE: The backup writes to the reserved pages (Archive pages) as seen with onstat -g arc. What does onstat - onstat -g arc give for both servers? Regards, David. > On 15 September 2017 at 15:16 BENJI LONG <ruggedmouse@hotmail.com> wrote: > > > I just want to double check that this is what has happened. > > I backed up and restored my primary. Does this mean that I also need to > restore the secondary, even though I didn't touch it? > > The log on the secondary when I tried to start HDR is saying two servers out > of sync. To re-establish the HDR pair, backup primary and restore to > secondary. > > I thought that I would just restore the primary , and restart HDR and it would > send its logs to the secondary and away it would go as normal. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >

Beni: Yes, after restoring the primary instance you will need to back up the primary server and restore it to the secondary IDS instance in order to reinitiate HDR. Good luck! Best regards, Martin M. Graney Queues Enforth Development, Inc. This electronic message contains information which may be privileged, confidential, or otherwise protected from disclosure. The information contained herein is intended for the addressee or recipient only. If you are not the addressee, or not the intended recipient, any disclosure, copying, distribution, or use of the contents of this message (including any attachments) is prohibited. If you have received this electronic message in error, please notify the sender immediately and destroy the original message and all copies. (Queues Enforth Development, Inc.) -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of BENJI LONG Sent: Friday, September 15, 2017 10:17 AM To: ids@iiug.org Subject: Re: Secondary stuck in Fast Recovery [39883] I just want to double check that this is what has happened. I backed up and restored my primary. Does this mean that I also need to restore the secondary, even though I didn't touch it? The log on the secondary when I tried to start HDR is saying two servers out of sync. To re-establish the HDR pair, backup primary and restore to secondary. I thought that I would just restore the primary , and restart HDR and it would send its logs to the secondary and away it would go as normal. ************************************************************************** ***** Forum Note: Use "Reply" to post a response in the discussion forum.

I believe once you restore the primary, you'd need to redo your steps all over. However, the idea of HDR is for continuous operation, so when your primary is down (in this case you bring it down for testing, so no need to restore), your secondary will become primary -- assuming your parameter DRAUTO=2 (I think that's the right value y/n?). So when you want to reverse the status of your original primary, you may do followings:1) bring the original primary up but it'll stay as secondary, 2) bring the "new" primary down, your original primary will become primary3) bring the original secondary up, it'll stay as secondary. Good luck. Let's go GreenThis email contains 100% recycled electrons. From: BENJI LONG <ruggedmouse@hotmail.com> To: ids@iiug.org Sent: Friday, September 15, 2017 10:18 AM Subject: Re: Secondary stuck in Fast Recovery [39883] I just want to double check that this is what has happened. I backed up and restored my primary. Does this mean that I also need to restore the secondary, even though I didn't touch it? The log on the secondary when I tried to start HDR is saying two servers out of sync. To re-establish the HDR pair, backup primary and restore to secondary. I thought that I would just restore the primary , and restart HDR and it would send its logs to the secondary and away it would go as normal. ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Ok, well I think I read in the Backup and Restore manual that you can from the primary, run a backup and send it over to the secondary and restore it at the same time possibly so this is a good excuse to try that. I will read up on that to see if I have that correct or not, and if it is possible.

Thanks, I will test that next, as I'm sure at some point, this will happen, and I am fuzzy on all of that as well.

Hi, from reading other posts, it seems like I need to restore secondary so I will give that a go now.

You can't start the secondary with oninit until after you have connected to the primary with onmode -d. Here are the steps to start HDR: Backup the primary (you have done this). Restore to the secondary using ontape -p, which will leave the secondary in recovery mode. Execute onmode -d secondary primary_name on the secondary. It should appear to hang. Execute onmode -d primary secondary_name on the primary. The primary should start sending logs to the secondary. When the final log has been transmitted, the secondary will come online in read only mode. --EEM -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of BENJI LONG Sent: Friday, September 15, 2017 9:11 AM To: ids@iiug.org Subject: Re: Secondary stuck in Fast Recovery [39881] Hi Marcus, are you saying that my method will not work, and that I need to restore my secondary? Currently my primary is running in standard mode. I didn't set replication to start as I was waiting to get both instances running. I just ran onmode -d secondary newserver_hdr but it wouldn't start. log says Two servers out of sync and then it stops the instance In the message log, it now says mt_shm_remove: WARNGIN: may not have removed all/correct segments. I seem to be in a vortex. The instance won't start with oninit properly, and it quits and is giving errors that aren't really what is wrong possibly ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Thanks for the info. Putting in my notes.

Hi, for clarification, what do you mean when you say you cannot start the secondary with oninit until after you have connected to the primary with onmode -d? I thought I was supposed to use oninit to start both instances, and then start HDR on the secondary and then go to the primary and start it there? Do I have that messed up?

That is correct. After the secondary is online, then you can stop and restart it using oninit. It will reconnect and retrieve the logs on its own. It can't do that unless it already know that it's a secondary. Also, the primary will refuse to send the logs, unless it already knows that the secondary exists. --EEM -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of BENJI LONG Sent: Friday, September 15, 2017 10:10 AM To: ids@iiug.org Subject: Re: RE: Secondary stuck in Fast Recovery [39893] Hi, for clarification, what do you mean when you say you cannot start the secondary with oninit until after you have connected to the primary with onmode -d? I thought I was supposed to use oninit to start both instances, and then start HDR on the secondary and then go to the primary and start it there? Do I have that messed up? ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Ok, back up and running. Phew! That was a tough few days but a lot learned, and I appreciate all the help. Need to get one more question resolved in a new thread, and I'm good to go for this weekend's work.

You lucky bud!I'm trying to do the same thing with DB2 but can't seem to find a good forum like our IIUG here :) Let's go GreenThis email contains 100% recycled electrons. From: BENJI LONG <ruggedmouse@hotmail.com> To: ids@iiug.org Sent: Friday, September 15, 2017 10:21 AM Subject: Re: Secondary stuck in Fast Recovery [39884] Ok, thanks. I didn't realize that. I will try that now. Wow, cannot believe how much I have learned over the last few days, just trying to do a restore. Amazing help on this forum. appreciate it very much. ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Hi Kern, Give us a shout directly, I cover DB2 as well as Informix. Regards, David. > On 15 September 2017 at 16:51 Kern Doe <kern_doe@yahoo.com> wrote: > > > You lucky bud!I'm trying to do the same thing with DB2 but can't seem to find > a good forum like our IIUG here :) Let's go GreenThis email contains 100% > recycled electrons. > > From: BENJI LONG <ruggedmouse@hotmail.com> > To: ids@iiug.org > Sent: Friday, September 15, 2017 10:21 AM > Subject: Re: Secondary stuck in Fast Recovery [39884] > > Ok, thanks. I didn't realize that. I will try that now. > > Wow, cannot believe how much I have learned over the last few days, just > trying to do a restore. > > Amazing help on this forum. > > appreciate it very much. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >

David, are you serious? :) you're gonna help us? thank you :) but ... this is Informix forum, I ... don't want to bother people with DB2 questions ... do we have an effective forum for DB2? Let's go GreenThis email contains 100% recycled electrons. From: "david@smooth1.co.uk" <david@smooth1.co.uk> To: ids@iiug.org; Kern Doe <kern_doe@yahoo.com> Sent: Friday, September 15, 2017 11:59 AM Subject: Re: Secondary stuck in Fast Recovery [39897] Hi Kern, Give us a shout directly, I cover DB2 as well as Informix. Regards, David. > On 15 September 2017 at 16:51 Kern Doe <kern_doe@yahoo.com> wrote: > > > You lucky bud!I'm trying to do the same thing with DB2 but can't seem to find > a good forum like our IIUG here :) Let's go GreenThis email contains 100% > recycled electrons. > > From: BENJI LONG <ruggedmouse@hotmail.com> > To: ids@iiug.org > Sent: Friday, September 15, 2017 10:21 AM > Subject: Re: Secondary stuck in Fast Recovery [39884] > > Ok, thanks. I didn't realize that. I will try that now. > > Wow, cannot believe how much I have learned over the last few days, just > trying to do a restore. > > Amazing help on this forum. > > appreciate it very much. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >