HDR & Connection Manager
After failing over an HDR pair (3 instances, Informix 11.70.FC7), the old primary was rebooted and an init.d script retried 'oninit -v' in a loop. Two instances correctly refused to start (logging 'DR: Local and Remote server type and/or last change incompatible... Cannot start the Primary server when the paired server is of incompatible type'), but one instance eventually slipped through and came up as a second primary and registered with Connection Manager. Replies confirmed Informix is supposed to reject such a connection and suggested a timing/race issue, recommending a PMR be opened, plus pointing to APAR IP23568 regarding Connection Manager behaviour. No resolution is recorded in the thread.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Informix 11.7 FC7, CSDK 3.70 FC7
I have a HDR pair with 3 instances setup with Connection Manager. Primary/HDR
run on separate Linux boxes; as well as Connection Manager. I triggered a
failover and all 3 instances on the primary were failed over to the HDR
server, now the new-Primary. The original primary was rebooted and an init.d
process attempted to bring up the 3 instances. The init.d job attempts to
bring up the instances with 'oninit -v' in an endless loop for 5 minutes;
after which the job gives up on that particular instance, triggers an alert,
and moves on to the next. 2 out of 3 instances failed continuously to start as
expected because a new primary was recognized on the failed-over node (this is
what I want to happen for all 3 instances). The remaining instance failed a
few times, but then started up as another Primary and registered with
Connection Manager. I have ran through the process a couple of times; each
ending with 2 instances failing and the 3rd coming up (it is not always the
same instance that starts as a second primary). Is this expected behavior?
I think we can take Connection Manager out of the picture. Should Informix allow an instance to come up as primary when it's paired server was recently promoted as primary? Is this what is known as split-brain?
Yes, Jacob. If there is already one primary active node, and another instance is trying to start up as primary, Informix won't let it go. It's a failover security mechanism. Don't face it like a split-brain scenario, which might still happen if there is no network between the two nodes, ok? That case should be treated manually, generally with a "shoot in the head" approach. Regards. Alexandre Marini IBM Informix Certified Professional v10 / v11.50 / v11.70 / v12.10 IBM Information Management Informix Technical Professional IBM Infosphere DataStage Technical Professional BRIUG website administrator Informix independent consultant > To: ids@iiug.org > From: jacobshadix@gmail.com > Subject: Re: HDR & Connection Manager [33905] > Date: Fri, 3 Oct 2014 13:49:43 -0400 > > I think we can take Connection Manager out of the picture. Should Informix > allow an instance to come up as primary when it's paired server was recently > promoted as primary? Is this what is known as split-brain? > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >
in this case though, Informix eventually let it slip through and start-up as a second primary.
What version? We should be rejecting the network connection to re-establish the HDR connection if both think they are primary. Davis - any comments? From: "JACOB SHADIX" <jacobshadix@gmail.com> To: ids@iiug.org Date: 10/03/2014 01:21 PM Subject: Re: RE: HDR & Connection Manager [33909] Sent by: ids-bounces@iiug.org in this case though, Informix eventually let it slip through and start-= up as a second primary. ***********************************************************************= ******** Forum Note: Use "Reply" to post a response in the discussion forum. =
11.7 fc7 and csdk 3.70 fc7
Was there not a message about HDR being in an incompatible state and th= e network connection being rejected? From: "JACOB SHADIX" <jacobshadix@gmail.com> To: ids@iiug.org Date: 10/03/2014 02:27 PM Subject: Re: RE: HDR & Connection Manager [33912] Sent by: ids-bounces@iiug.org 11.7 fc7 and csdk 3.70 fc7 ***********************************************************************= ******** Forum Note: Use "Reply" to post a response in the discussion forum. =
It's important to try to understand why it happens. And since you apparently can reproduce it relatively easy a PMR should be opened. In a blind guess I can imagine that due to timming issues they may be starting upd at "same time" and not see each other... But again, support should look into it. Equally important, it's your statement that we should take CM out of the picture. That's wrong and you should take a look at this: http://www-01.ibm.com/support/docview.wss?uid=swg1IP23568 I seem to recall a situation (maybe it was the PMR that oiginated this APAR) where the customer explicitly wanted to transform one of the nodes into a standard server. Problem was they forgot to reconfigure the CM... now apparently it got smarter and tries to avoid this misuse. I think it also applies to your situation... In other words, worst that having two primaries is to let the clients see them both. Regards On Fri, Oct 3, 2014 at 7:16 PM, Alexandre Marini <alexandre@briug.org> wrote: > Yes, Jacob. If there is already one primary active node, and another > instance > is trying to start up as primary, Informix won't let it go. It's a failover > security mechanism. Don't face it like a split-brain scenario, which might > still happen if there is no network between the two nodes, ok? That case > should be treated manually, generally with a "shoot in the head" approach. > > Regards. > > Alexandre Marini > IBM Informix Certified Professional v10 / v11.50 / v11.70 / v12.10 > > IBM Information Management Informix Technical Professional > > IBM Infosphere DataStage Technical Professional > BRIUG website administrator > Informix independent consultant > > > To: ids@iiug.org > > From: jacobshadix@gmail.com > > Subject: Re: HDR & Connection Manager [33905] > > Date: Fri, 3 Oct 2014 13:49:43 -0400 > > > > I think we can take Connection Manager out of the picture. Should > Informix > > allow an instance to come up as primary when it's paired server was > recently > > promoted as primary? Is this what is known as split-brain? > > > > > > > > ******************************************************************************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --20cf303636db71380405048b1c74
It'll fail a good many times with a lot of these messages - 10/04/14 16:26:20 DR: Local and Remote server type and/or last change(LC) incompatible [Local type: Primary, LC: Primary mode] [Remote type: Primary, LC: ] 10/04/14 16:26:20 DR: Cannot start the Primary server when the paired server is of incompatible (Primary or Standard) type. Please convert the paired server to compatible (Secondary) type.