HDR and ER Running Together
Keith ran ER from an HDR primary and saw log messages like "ER can not send replicated transactions due to HDR hang", with ER freezing while HDR was disconnected. Madison Pruet (IBM) explained this is by design: ER won't send a transaction until it knows the HDR secondary has it, otherwise a failover could leave ER targets with transactions the HDR pair lost. ER force-flushes the HDR buffers (~1/4 sec delay); if the ACK doesn't arrive in a couple of seconds the message appears, and once the HDR ping timeout marks HDR down ER proceeds, so the maximum delay is the ping timeout. Sync vs async mode makes no difference. Art Kagel suggested using an RSS secondary for reporting instead, but Keith's ER targets are differently structured. No configuration fix was recorded — the behaviour is expected.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Lets try again, this may be easier to read !! Set up is as follows: IDS 9.4 on AIX 5.3 --> ER over LAN --> IDS 10 on Solaris 2.8 | |-> IDS 11 on Solaris 2.10 V Sync HDR over WAN | V IDS 9.4 on AIX 5.3 Note ER is Primary to Secondary (one way). With this setup I am seeing lines in the error log of the HDR/ER Primary as follows: ER can not send replicated transactions due to HDR hang ER resume transmission of ER/HDR replicated transactions Also a few days ago the HDR secondary lost contact with the primary and couldn't automatically re-establish (brief network glitch). I stopped and restarted the secondary and this HDR re-established, but while HDR was not running (and until the logs had shipped and applied up to date) ER was frozen. I am trying to understand why ER keeps stopping (albeit briefly) and will not run while HDR is not connected. My suspicion is that it is related to the fact I have HDR in Synchronous mode but would like this confirmed. Also, if this is the case, is it possible to switch between Synchronous and Asynchronous modes without restarting instances ? Many Thanks in anticipation Keith
If a replicated transaction is replicated via ER prior to ER knowing th= at it has made it to the HDR secondary, then there would be inconsistency = in the event of a failover. So we don't allow ER to replicate the transac= tion until we know that the secondary has the transaction. From: "Keith Simmons" <smiley73@gmail.com> To: ids@iiug.org, Date: 06/19/2012 02:58 AM Subject: HDR and ER Running Together [27383] Sent by: ids-bounces@iiug.org Lets try again, this may be easier to read !! Set up is as follows: IDS 9.4 on AIX 5.3 --> ER over LAN --> IDS 10 on Solaris 2.8 | |-> IDS 11 on Solaris 2.10 V Sync HDR over WAN | V IDS 9.4 on AIX 5.3 Note ER is Primary to Secondary (one way). With this setup I am seeing lines in the error log of the HDR/ER Primary as follows: ER can not send replicated transactions due to HDR hang ER resume transmission of ER/HDR replicated transactions Also a few days ago the HDR secondary lost contact with the primary and couldn't automatically re-establish (brief network glitch). I stopp= ed and restarted the secondary and this HDR re-established, but while HDR was not running (and until the logs had shipped and applied up to date) ER was frozen. I am trying to understand why ER keeps stopping (albeit briefly) and will not run while HDR is not connected. My suspicion is that it is rel= ated to the fact I have HDR in Synchronous mode but would like this confirmed. Also, if this is the case, is it possible to switch between Synchronous and Asynchronous modes without restarting instances ? Many Thanks in anticipation Keith ***********************************************************************= ******** Forum Note: Use "Reply" to post a response in the discussion forum. =
Thanks Madison, So am I correct in thinking that while HDR is stopped for any reason ER will also pause, regardless of whether HDR is in synchronous or asynchronous mode ? I'm trying to get my head around the implications of this. We use the target(s) of the ER as part of our monitoring and feed- back system and also as a read-only access for some customer facing web-pages to allow them to track orders in 'almost' real time. It would seem that if either the HDR Secondary or HDR Primary fail for any reason and require an extended portion of down-time (but not long enough to cycle the logs and require a complete re- instantiation of HDR) then we will not be able to maintain these targets in step. The alternative I think, therefore, is, during any failure likely to take more than 3-4 hours to resolve, to switch the remaining HDR server to stand- alone and then restart HDR using an archive/ restore (not ideal for a 1Tb instance on each end of a 100 mile WAN link). Please can you confirm this reasoning and conclusion. Keith On 19 June 2012 13:38, Madison Pruet <mpruet@us.ibm.com> wrote: > If a replicated transaction is replicated via ER prior to ER knowing th= > at > it has made it to the HDR secondary, then there would be inconsistency = > in > the event of a failover. So we don't allow ER to replicate the transac= > tion > until we know that the secondary has the transaction. > > From: "Keith Simmons" <smiley73@gmail.com> > To: ids@iiug.org, > Date: 06/19/2012 02:58 AM > Subject: HDR and ER Running Together [27383] > Sent by: ids-bounces@iiug.org > > Lets try again, this may be easier to read !! > > Set up is as follows: > > IDS 9.4 on AIX 5.3 --> ER over LAN --> IDS 10 on Solaris 2.8 > | |-> > IDS 11 on Solaris 2.10 > V > Sync HDR over WAN > | > V > IDS 9.4 on AIX 5.3 > > Note ER is Primary to Secondary (one way). > > With this setup I am seeing lines in the error log of the HDR/ER > Primary as follows: > ER can not send replicated transactions due to HDR hang > ER resume transmission of ER/HDR replicated transactions > Also a few days ago the HDR secondary lost contact with the primary > and couldn't automatically re-establish (brief network glitch). I stopp= > ed > and restarted the secondary and this HDR re-established, but while > HDR was not running (and until the logs had shipped and applied up > to date) ER was frozen. > I am trying to understand why ER keeps stopping (albeit briefly) and > will not run while HDR is not connected. My suspicion is that it is rel= > ated > > to the fact I have HDR in Synchronous mode but would like this > confirmed. Also, if this is the case, is it possible to switch between > Synchronous and Asynchronous modes without restarting instances ? > > Many Thanks in anticipation > > Keith > > ***********************************************************************= > ******** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > = > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >
An alternative would be to use an RSS secondary for the monitoring/feedback/reporting server. That would also allow you to upgrade the RSS secondary to an HDR secondary if one of the HDR servers goes down. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Tue, Jun 19, 2012 at 9:42 AM, Keith Simmons <smiley73@gmail.com> wrote: > Thanks Madison, > > So am I correct in thinking that while HDR is stopped for any reason > ER will also pause, regardless of whether HDR is in synchronous or > asynchronous mode ? > > I'm trying to get my head around the implications of this. > > We use the target(s) of the ER as part of our monitoring and feed- > back system and also as a read-only access for some customer > facing web-pages to allow them to track orders in 'almost' real time. > > It would seem that if either the HDR Secondary or HDR Primary fail > for any reason and require an extended portion of down-time (but > not long enough to cycle the logs and require a complete re- > instantiation of HDR) then we will not be able to maintain these > targets in step. The alternative I think, therefore, is, during any failure > likely to take more than 3-4 hours to resolve, to switch the remaining > HDR server to stand- alone and then restart HDR using an archive/ > restore (not ideal for a 1Tb instance on each end of a 100 mile WAN > link). > > Please can you confirm this reasoning and conclusion. > > Keith > > On 19 June 2012 13:38, Madison Pruet <mpruet@us.ibm.com> wrote: > > If a replicated transaction is replicated via ER prior to ER knowing th= > > at > > it has made it to the HDR secondary, then there would be inconsistency = > > in > > the event of a failover. So we don't allow ER to replicate the transac= > > tion > > until we know that the secondary has the transaction. > > > > From: "Keith Simmons" <smiley73@gmail.com> > > To: ids@iiug.org, > > Date: 06/19/2012 02:58 AM > > Subject: HDR and ER Running Together [27383] > > Sent by: ids-bounces@iiug.org > > > > Lets try again, this may be easier to read !! > > > > Set up is as follows: > > > > IDS 9.4 on AIX 5.3 --> ER over LAN --> IDS 10 on Solaris 2.8 > > | |-> > > IDS 11 on Solaris 2.10 > > V > > Sync HDR over WAN > > | > > V > > IDS 9.4 on AIX 5.3 > > > > Note ER is Primary to Secondary (one way). > > > > With this setup I am seeing lines in the error log of the HDR/ER > > Primary as follows: > > ER can not send replicated transactions due to HDR hang > > ER resume transmission of ER/HDR replicated transactions > > Also a few days ago the HDR secondary lost contact with the primary > > and couldn't automatically re-establish (brief network glitch). I stopp= > > ed > > and restarted the secondary and this HDR re-established, but while > > HDR was not running (and until the logs had shipped and applied up > > to date) ER was frozen. > > I am trying to understand why ER keeps stopping (albeit briefly) and > > will not run while HDR is not connected. My suspicion is that it is rel= > > ated > > > > to the fact I have HDR in Synchronous mode but would like this > > confirmed. Also, if this is the case, is it possible to switch between > > Synchronous and Asynchronous modes without restarting instances ? > > > > Many Thanks in anticipation > > > > Keith > > > > ***********************************************************************= > > ******** > > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > = > > > > > > > > ******************************************************************************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --e89a8f83a1291ed57904c2d39a9a
Art Thanks, but not an option, those ER targets are subsets of the tables with specific indexing to support that application and are very different from the OLTP application on the HDR pair. Also the ER targets are on Solaris with the HDR on AIX. Keith On 19 June 2012 14:53, Art Kagel <art.kagel@gmail.com> wrote: > An alternative would be to use an RSS secondary for the > monitoring/feedback/reporting server. That would also allow you to upgrade > the RSS secondary to an HDR secondary if one of the HDR servers goes down. > > Art > > Art S. Kagel > Advanced DataTools (www.advancedatatools.com) > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > other organization with which I am associated either explicitly, > implicitly, or by inference. Neither do those opinions reflect those of > other individuals affiliated with any entity with which I am affiliated nor > those of the entities themselves. > > On Tue, Jun 19, 2012 at 9:42 AM, Keith Simmons <smiley73@gmail.com> wrote: > >> Thanks Madison, >> >> So am I correct in thinking that while HDR is stopped for any reason >> ER will also pause, regardless of whether HDR is in synchronous or >> asynchronous mode ? >> >> I'm trying to get my head around the implications of this. >> >> We use the target(s) of the ER as part of our monitoring and feed- >> back system and also as a read-only access for some customer >> facing web-pages to allow them to track orders in 'almost' real time. >> >> It would seem that if either the HDR Secondary or HDR Primary fail >> for any reason and require an extended portion of down-time (but >> not long enough to cycle the logs and require a complete re- >> instantiation of HDR) then we will not be able to maintain these >> targets in step. The alternative I think, therefore, is, during any failure >> likely to take more than 3-4 hours to resolve, to switch the remaining >> HDR server to stand- alone and then restart HDR using an archive/ >> restore (not ideal for a 1Tb instance on each end of a 100 mile WAN >> link). >> >> Please can you confirm this reasoning and conclusion. >> >> Keith >> >> On 19 June 2012 13:38, Madison Pruet <mpruet@us.ibm.com> wrote: >> > If a replicated transaction is replicated via ER prior to ER knowing th= >> > at >> > it has made it to the HDR secondary, then there would be inconsistency = >> > in >> > the event of a failover. So we don't allow ER to replicate the transac= >> > tion >> > until we know that the secondary has the transaction. >> > >> > From: "Keith Simmons" <smiley73@gmail.com> >> > To: ids@iiug.org, >> > Date: 06/19/2012 02:58 AM >> > Subject: HDR and ER Running Together [27383] >> > Sent by: ids-bounces@iiug.org >> > >> > Lets try again, this may be easier to read !! >> > >> > Set up is as follows: >> > >> > IDS 9.4 on AIX 5.3 --> ER over LAN --> IDS 10 on Solaris 2.8 >> > | |-> >> > IDS 11 on Solaris 2.10 >> > V >> > Sync HDR over WAN >> > | >> > V >> > IDS 9.4 on AIX 5.3 >> > >> > Note ER is Primary to Secondary (one way). >> > >> > With this setup I am seeing lines in the error log of the HDR/ER >> > Primary as follows: >> > ER can not send replicated transactions due to HDR hang >> > ER resume transmission of ER/HDR replicated transactions >> > Also a few days ago the HDR secondary lost contact with the primary >> > and couldn't automatically re-establish (brief network glitch). I stopp= >> > ed >> > and restarted the secondary and this HDR re-established, but while >> > HDR was not running (and until the logs had shipped and applied up >> > to date) ER was frozen. >> > I am trying to understand why ER keeps stopping (albeit briefly) and >> > will not run while HDR is not connected. My suspicion is that it is rel= >> > ated >> > >> > to the fact I have HDR in Synchronous mode but would like this >> > confirmed. Also, if this is the case, is it possible to switch between >> > Synchronous and Asynchronous modes without restarting instances ? >> > >> > Many Thanks in anticipation >> > >> > Keith >> > >> > ***********************************************************************= >> > ******** >> > >> > Forum Note: Use "Reply" to post a response in the discussion forum. >> > >> > = >> > >> > >> > >> >> > ******************************************************************************* >> > Forum Note: Use "Reply" to post a response in the discussion forum. >> > >> >> >> >> > ******************************************************************************* >> Forum Note: Use "Reply" to post a response in the discussion forum. >> >> > > --e89a8f83a1291ed57904c2d39a9a > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >
Gotcha. It was a thought. Always heartening when you find someone using the right tool for the job and disheartening when it doesn't work perfectly anyway. <sigh> Madison: Your response does not indicate whether ER would wait for the secondary in the case where the secondary is running in asynchronous mode. Since the primary only waits for the secondary to ack the log buffer packets and not for the local commit, would the ER send also not wait for the secondary to commit? Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Tue, Jun 19, 2012 at 10:02 AM, Keith Simmons <smiley73@gmail.com> wrote: > Art > > Thanks, but not an option, those ER targets are subsets of the tables > with specific indexing to support that application and are very different > from the OLTP application on the HDR pair. Also the ER targets are on > Solaris with the HDR on AIX. > > Keith > > On 19 June 2012 14:53, Art Kagel <art.kagel@gmail.com> wrote: > > An alternative would be to use an RSS secondary for the > > monitoring/feedback/reporting server. That would also allow you to > upgrade > > the RSS secondary to an HDR secondary if one of the HDR servers goes > down. > > > > Art > > > > Art S. Kagel > > Advanced DataTools (www.advancedatatools.com) > > Blog: http://informix-myview.blogspot.com/ > > > > Disclaimer: Please keep in mind that my own opinions are my own opinions > > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > > other organization with which I am associated either explicitly, > > implicitly, or by inference. Neither do those opinions reflect those of > > other individuals affiliated with any entity with which I am affiliated > nor > > those of the entities themselves. > > > > On Tue, Jun 19, 2012 at 9:42 AM, Keith Simmons <smiley73@gmail.com> > wrote: > > > >> Thanks Madison, > >> > >> So am I correct in thinking that while HDR is stopped for any reason > >> ER will also pause, regardless of whether HDR is in synchronous or > >> asynchronous mode ? > >> > >> I'm trying to get my head around the implications of this. > >> > >> We use the target(s) of the ER as part of our monitoring and feed- > >> back system and also as a read-only access for some customer > >> facing web-pages to allow them to track orders in 'almost' real time. > >> > >> It would seem that if either the HDR Secondary or HDR Primary fail > >> for any reason and require an extended portion of down-time (but > >> not long enough to cycle the logs and require a complete re- > >> instantiation of HDR) then we will not be able to maintain these > >> targets in step. The alternative I think, therefore, is, during any > failure > >> likely to take more than 3-4 hours to resolve, to switch the remaining > >> HDR server to stand- alone and then restart HDR using an archive/ > >> restore (not ideal for a 1Tb instance on each end of a 100 mile WAN > >> link). > >> > >> Please can you confirm this reasoning and conclusion. > >> > >> Keith > >> > >> On 19 June 2012 13:38, Madison Pruet <mpruet@us.ibm.com> wrote: > >> > If a replicated transaction is replicated via ER prior to ER knowing > th= > >> > at > >> > it has made it to the HDR secondary, then there would be > inconsistency = > >> > in > >> > the event of a failover. So we don't allow ER to replicate the > transac= > >> > tion > >> > until we know that the secondary has the transaction. > >> > > >> > From: "Keith Simmons" <smiley73@gmail.com> > >> > To: ids@iiug.org, > >> > Date: 06/19/2012 02:58 AM > >> > Subject: HDR and ER Running Together [27383] > >> > Sent by: ids-bounces@iiug.org > >> > > >> > Lets try again, this may be easier to read !! > >> > > >> > Set up is as follows: > >> > > >> > IDS 9.4 on AIX 5.3 --> ER over LAN --> IDS 10 on Solaris 2.8 > >> > | |-> > >> > IDS 11 on Solaris 2.10 > >> > V > >> > Sync HDR over WAN > >> > | > >> > V > >> > IDS 9.4 on AIX 5.3 > >> > > >> > Note ER is Primary to Secondary (one way). > >> > > >> > With this setup I am seeing lines in the error log of the HDR/ER > >> > Primary as follows: > >> > ER can not send replicated transactions due to HDR hang > >> > ER resume transmission of ER/HDR replicated transactions > >> > Also a few days ago the HDR secondary lost contact with the primary > >> > and couldn't automatically re-establish (brief network glitch). I > stopp= > >> > ed > >> > and restarted the secondary and this HDR re-established, but while > >> > HDR was not running (and until the logs had shipped and applied up > >> > to date) ER was frozen. > >> > I am trying to understand why ER keeps stopping (albeit briefly) and > >> > will not run while HDR is not connected. My suspicion is that it is > rel= > >> > ated > >> > > >> > to the fact I have HDR in Synchronous mode but would like this > >> > confirmed. Also, if this is the case, is it possible to switch between > >> > Synchronous and Asynchronous modes without restarting instances ? > >> > > >> > Many Thanks in anticipation > >> > > >> > Keith > >> > > >> > > ***********************************************************************= > >> > ******** > >> > > >> > Forum Note: Use "Reply" to post a response in the discussion forum. > >> > > >> > = > >> > > >> > > >> > > >> > >> > > > > ******************************************************************************* > >> > Forum Note: Use "Reply" to post a response in the discussion forum. > >> > > >> > >> > >> > >> > > > > ******************************************************************************* > >> Forum Note: Use "Reply" to post a response in the discussion forum. > >> > >> > > > > --e89a8f83a1291ed57904c2d39a9a > > > > > > > > ******************************************************************************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --14dae9340a99870b2304c2d3f99c
It will wait unless HDR has been flagged as off because of connection timeouts. The issue is that when an HDR failover in an async mode occurs, there c= an be transactions committed on the old primary. Those transactions can be= come 'lost transactions' after failover. If ER were to transmit those transactions, then the ER targets would have the transaction, but the H= DR pair on which it originated would not. That's why ER has to wait until= it knows that the transaction has made it to the HDR secondary. If ER has a such a txn, it will attempt to force flush the HDR transmit= buffers. If that can not be done because of a temporary network outage= , you will get the message that you saw. From: "Keith Simmons" <smiley73@gmail.com> To: ids@iiug.org, Date: 06/19/2012 07:44 AM Subject: Re: HDR and ER Running Together [27388] Sent by: ids-bounces@iiug.org Thanks Madison, So am I correct in thinking that while HDR is stopped for any reason ER will also pause, regardless of whether HDR is in synchronous or asynchronous mode ? I'm trying to get my head around the implications of this. We use the target(s) of the ER as part of our monitoring and feed- back system and also as a read-only access for some customer facing web-pages to allow them to track orders in 'almost' real time. It would seem that if either the HDR Secondary or HDR Primary fail for any reason and require an extended portion of down-time (but not long enough to cycle the logs and require a complete re- instantiation of HDR) then we will not be able to maintain these targets in step. The alternative I think, therefore, is, during any fai= lure likely to take more than 3-4 hours to resolve, to switch the remaining HDR server to stand- alone and then restart HDR using an archive/ restore (not ideal for a 1Tb instance on each end of a 100 mile WAN link). Please can you confirm this reasoning and conclusion. Keith On 19 June 2012 13:38, Madison Pruet <mpruet@us.ibm.com> wrote: > If a replicated transaction is replicated via ER prior to ER knowing = th=3D > at > it has made it to the HDR secondary, then there would be inconsistenc= y =3D > in > the event of a failover. So we don't allow ER to replicate the transa= c=3D > tion > until we know that the secondary has the transaction. > > From: "Keith Simmons" <smiley73@gmail.com> > To: ids@iiug.org, > Date: 06/19/2012 02:58 AM > Subject: HDR and ER Running Together [27383] > Sent by: ids-bounces@iiug.org > > Lets try again, this may be easier to read !! > > Set up is as follows: > > IDS 9.4 on AIX 5.3 --> ER over LAN --> IDS 10 on Solaris 2.8 > | |-> > IDS 11 on Solaris 2.10 > V > Sync HDR over WAN > | > V > IDS 9.4 on AIX 5.3 > > Note ER is Primary to Secondary (one way). > > With this setup I am seeing lines in the error log of the HDR/ER > Primary as follows: > ER can not send replicated transactions due to HDR hang > ER resume transmission of ER/HDR replicated transactions > Also a few days ago the HDR secondary lost contact with the primary > and couldn't automatically re-establish (brief network glitch). I sto= pp=3D > ed > and restarted the secondary and this HDR re-established, but while > HDR was not running (and until the logs had shipped and applied up > to date) ER was frozen. > I am trying to understand why ER keeps stopping (albeit briefly) and > will not run while HDR is not connected. My suspicion is that it is r= el=3D > ated > > to the fact I have HDR in Synchronous mode but would like this > confirmed. Also, if this is the case, is it possible to switch betwee= n > Synchronous and Asynchronous modes without restarting instances ? > > Many Thanks in anticipation > > Keith > > *********************************************************************= **=3D > ******** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > =3D > > > ***********************************************************************= ******** > Forum Note: Use "Reply" to post a response in the discussion forum. > ***********************************************************************= ******** Forum Note: Use "Reply" to post a response in the discussion forum. =
Let's get the perspective. 1) If ER has a txn which it needs to send, but can't because the secon= dary has not receive the log buffer containing the commit, it will force the= HDR transmit buffers to be immediately transmitted. Generally this means a= bout a 1/4 sec delay in the ER transmission. 2) If ER is not able to send the HDR transmit buffer and/or the ACK of= the transmission is not received within a couple of seconds, then you get t= he message. 3) If the HDR ping timeout kicks in and we mark HDR as down, then ER doesn't wait because the HDR secondary is no longer a viable failover server. So the max delay is based on the HDR ping timeout. There is no distinction between sync and async HDR mode since ER will p= ush out the HDR buffer as soon as it is blocking on that buffer. M.P. From: "Art Kagel" <art.kagel@gmail.com> To: ids@iiug.org, Date: 06/19/2012 08:22 AM Subject: Re: HDR and ER Running Together [27392] Sent by: ids-bounces@iiug.org Gotcha. It was a thought. Always heartening when you find someone using= the right tool for the job and disheartening when it doesn't work perfe= ctly anyway. <sigh> Madison: Your response does not indicate whether ER would wait for the secondary in the case where the secondary is running in asynchronous mo= de. Since the primary only waits for the secondary to ack the log buffer packets and not for the local commit, would the ER send also not wait f= or the secondary to commit? Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinion= s and do not reflect on my employer, Advanced DataTools, the IIUG, nor an= y other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of= other individuals affiliated with any entity with which I am affiliated= nor those of the entities themselves. On Tue, Jun 19, 2012 at 10:02 AM, Keith Simmons <smiley73@gmail.com> wr= ote: > Art > > Thanks, but not an option, those ER targets are subsets of the tables= > with specific indexing to support that application and are very diffe= rent > from the OLTP application on the HDR pair. Also the ER targets are on= > Solaris with the HDR on AIX. > > Keith > > On 19 June 2012 14:53, Art Kagel <art.kagel@gmail.com> wrote: > > An alternative would be to use an RSS secondary for the > > monitoring/feedback/reporting server. That would also allow you to > upgrade > > the RSS secondary to an HDR secondary if one of the HDR servers goe= s > down. > > > > Art > > > > Art S. Kagel > > Advanced DataTools (www.advancedatatools.com) > > Blog: http://informix-myview.blogspot.com/ > > > > Disclaimer: Please keep in mind that my own opinions are my own opinions > > and do not reflect on my employer, Advanced DataTools, the IIUG, no= r any > > other organization with which I am associated either explicitly, > > implicitly, or by inference. Neither do those opinions reflect thos= e of > > other individuals affiliated with any entity with which I am affili= ated > nor > > those of the entities themselves. > > > > On Tue, Jun 19, 2012 at 9:42 AM, Keith Simmons <smiley73@gmail.com>= > wrote: > > > >> Thanks Madison, > >> > >> So am I correct in thinking that while HDR is stopped for any reas= on > >> ER will also pause, regardless of whether HDR is in synchronous or= > >> asynchronous mode ? > >> > >> I'm trying to get my head around the implications of this. > >> > >> We use the target(s) of the ER as part of our monitoring and feed-= > >> back system and also as a read-only access for some customer > >> facing web-pages to allow them to track orders in 'almost' real ti= me. > >> > >> It would seem that if either the HDR Secondary or HDR Primary fail= > >> for any reason and require an extended portion of down-time (but > >> not long enough to cycle the logs and require a complete re- > >> instantiation of HDR) then we will not be able to maintain these > >> targets in step. The alternative I think, therefore, is, during an= y > failure > >> likely to take more than 3-4 hours to resolve, to switch the remai= ning > >> HDR server to stand- alone and then restart HDR using an archive/ > >> restore (not ideal for a 1Tb instance on each end of a 100 mile WA= N > >> link). > >> > >> Please can you confirm this reasoning and conclusion. > >> > >> Keith > >> > >> On 19 June 2012 13:38, Madison Pruet <mpruet@us.ibm.com> wrote: > >> > If a replicated transaction is replicated via ER prior to ER kno= wing > th=3D > >> > at > >> > it has made it to the HDR secondary, then there would be > inconsistency =3D > >> > in > >> > the event of a failover. So we don't allow ER to replicate the > transac=3D > >> > tion > >> > until we know that the secondary has the transaction. > >> > > >> > From: "Keith Simmons" <smiley73@gmail.com> > >> > To: ids@iiug.org, > >> > Date: 06/19/2012 02:58 AM > >> > Subject: HDR and ER Running Together [27383] > >> > Sent by: ids-bounces@iiug.org > >> > > >> > Lets try again, this may be easier to read !! > >> > > >> > Set up is as follows: > >> > > >> > IDS 9.4 on AIX 5.3 --> ER over LAN --> IDS 10 on Solaris 2.8 > >> > | |-> > >> > IDS 11 on Solaris 2.10 > >> > V > >> > Sync HDR over WAN > >> > | > >> > V > >> > IDS 9.4 on AIX 5.3 > >> > > >> > Note ER is Primary to Secondary (one way). > >> > > >> > With this setup I am seeing lines in the error log of the HDR/ER= > >> > Primary as follows: > >> > ER can not send replicated transactions due to HDR hang > >> > ER resume transmission of ER/HDR replicated transactions > >> > Also a few days ago the HDR secondary lost contact with the prim= ary > >> > and couldn't automatically re-establish (brief network glitch). = I > stopp=3D > >> > ed > >> > and restarted the secondary and this HDR re-established, but whi= le > >> > HDR was not running (and until the logs had shipped and applied = up > >> > to date) ER was frozen. > >> > I am trying to understand why ER keeps stopping (albeit briefly)= and > >> > will not run while HDR is not connected. My suspicion is that it= is > rel=3D > >> > ated > >> > > >> > to the fact I have HDR in Synchronous mode but would like this > >> > confirmed. Also, if this is the case, is it possible to switch between > >> > Synchronous and Asynchronous modes without restarting instances = ? > >> > > >> > Many Thanks in anticipation > >> > > >> > Keith > >> > > >> > > *********************************************************************= **=3D > >> > ******** > >> > > >> > Forum Note: Use "Reply" to post a response in the discussion for= um. > >> > > >> > =3D > >> > > >> > > >> >@@
Madison Thanks for this explanation, it makes logical sense and ties in with what I see occasionally. However I got the following errors a couple of days ago: Primary: 13:40:25 DR: Cannot connect to secondary server 13:40:36 DR: Primary server connected 13:40:36 DR: Send error 13:40:36 ASF Echo-Thread Server: asfcode = -25580: oserr = 32: errstr = : System error occurred in network function. System error = 32. 13:40:36 DR: Failure recovery error (2) 13:40:37 DR: Turned off on primary server 13:40:37 DR: Cannot connect to secondary server 13:40:48 DR: Primary server connected 13:40:49 DR: Secondary server needs failure recovery Secondary: 13:40:35 DR: Received connection request from remote server when DR is not Off [Local type: Secondary, Current state: ?] [Remote type: Primary] 13:40:42 DR: ping timeout 13:40:42 DR: Receive error 13:40:42 ASF Echo-Thread Server: asfcode = -25582: oserr = 4: errstr = : Network connection is broken. System error = 4. 13:40:44 DR: Turned off on secondary server 13:40:47 DR: Secondary server connected 13:40:49 DR: Secondary server needs failure recovery 13:40:49 DR: Failure recovery from disk in progress ... 13:41:24 DR: Received connection request from remote server when DR is not Off [Local type: Secondary, Current state: ?] [Remote type: Primary] 13:41:36 DR: Received connection request from remote server when DR is not Off [Local type: Secondary, Current state: ?] HDR failed to reconnect for some 40 minutes during which time ER failed to send transactions. HDR only restarted when the secondary engine was restarted. Thoughts would be appreciated, but please tell me if this is better as a PMR rather than discussion here. Keith On 19 June 2012 17:10, Madison Pruet <mpruet@us.ibm.com> wrote: > Let's get the perspective. > > 1) If ER has a txn which it needs to send, but can't because the secon= > dary > has not receive the log buffer containing the commit, it will force the= > HDR > transmit buffers to be immediately transmitted. Generally this means a= > bout > a 1/4 sec delay in the ER transmission. > > 2) If ER is not able to send the HDR transmit buffer and/or the ACK of= > the > transmission is not received within a couple of seconds, then you get t= > he > message. > > 3) If the HDR ping timeout kicks in and we mark HDR as down, then ER > doesn't wait because the HDR secondary is no longer a viable failover > server. > > So the max delay is based on the HDR ping timeout. > > There is no distinction between sync and async HDR mode since ER will p= > ush > out the HDR buffer as soon as it is blocking on that buffer. > > M.P. > > From: "Art Kagel" <art.kagel@gmail.com> > To: ids@iiug.org, > Date: 06/19/2012 08:22 AM > Subject: Re: HDR and ER Running Together [27392] > Sent by: ids-bounces@iiug.org > > Gotcha. It was a thought. Always heartening when you find someone using= > > the right tool for the job and disheartening when it doesn't work perfe= > ctly > > anyway. <sigh> > > Madison: Your response does not indicate whether ER would wait for the > secondary in the case where the secondary is running in asynchronous mo= > de. > Since the primary only waits for the secondary to ack the log buffer > packets and not for the local commit, would the ER send also not wait f= > or > the secondary to commit? > > Art > > Art S. Kagel > Advanced DataTools (www.advancedatatools.com) > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinion= > s > and do not reflect on my employer, Advanced DataTools, the IIUG, nor an= > y > other organization with which I am associated either explicitly, > implicitly, or by inference. Neither do those opinions reflect those of= > > other individuals affiliated with any entity with which I am affiliated= > nor > > those of the entities themselves. > > On Tue, Jun 19, 2012 at 10:02 AM, Keith Simmons <smiley73@gmail.com> wr= > ote: > >> Art >> >> Thanks, but not an option, those ER targets are subsets of the tables= > >> with specific indexing to support that application and are very diffe= > rent > >> from the OLTP application on the HDR pair. Also the ER targets are on= > >> Solaris with the HDR on AIX. >> >> Keith >> >> On 19 June 2012 14:53, Art Kagel <art.kagel@gmail.com> wrote: >> > An alternative would be to use an RSS secondary for the >> > monitoring/feedback/reporting server. That would also allow you to >> upgrade >> > the RSS secondary to an HDR secondary if one of the HDR servers goe= > s >> down. >> > >> > Art >> > >> > Art S. Kagel >> > Advanced DataTools (www.advancedatatools.com) >> > Blog: http://informix-myview.blogspot.com/ >> > >> > Disclaimer: Please keep in mind that my own opinions are my own > opinions >> > and do not reflect on my employer, Advanced DataTools, the IIUG, no= > r > any >> > other organization with which I am associated either explicitly, >> > implicitly, or by inference. Neither do those opinions reflect thos= > e of > >> > other individuals affiliated with any entity with which I am affili= > ated > >> nor >> > those of the entities themselves. >> > >> > On Tue, Jun 19, 2012 at 9:42 AM, Keith Simmons <smiley73@gmail.com>= > >> wrote: >> > >> >> Thanks Madison, >> >> >> >> So am I correct in thinking that while HDR is stopped for any reas= > on >> >> ER will also pause, regardless of whether HDR is in synchronous or= > >> >> asynchronous mode ? >> >> >> >> I'm trying to get my head around the implications of this. >> >> >> >> We use the target(s) of the ER as part of our monitoring and feed-= > >> >> back system and also as a read-only access for some customer >> >> facing web-pages to allow them to track orders in 'almost' real ti= > me. >> >> >> >> It would seem that if either the HDR Secondary or HDR Primary fail= > >> >> for any reason and require an extended portion of down-time (but >> >> not long enough to cycle the logs and require a complete re- >> >> instantiation of HDR) then we will not be able to maintain these >> >> targets in step. The alternative I think, therefore, is, during an= > y >> failure >> >> likely to take more than 3-4 hours to resolve, to switch the remai= > ning > >> >> HDR server to stand- alone and then restart HDR using an archive/ >> >> restore (not ideal for a 1Tb instance on each end of a 100 mile WA= > N >> >> link). >> >> >> >> Please can you confirm this reasoning and conclusion. >> >> >> >> Keith >> >> >> >> On 19 June 2012 13:38, Madison Pruet <mpruet@us.ibm.com> wrote: >> >> > If a replicated transaction is replicated via ER prior to ER kno= > wing > >> th=3D >> >> > at >> >> > it has made it to the HDR secondary, then there would be >> inconsistency =3D >> >> > in >> >> > the event of a failover. So we don't allow ER to replicate the >> transac=3D >> >> > tion >> >
Original post: Madison Thanks for this explanation, it makes logical sense and ties in with what I see occasionally. However I got the following errors a couple of days ago: Primary: 13:40:25 DR: Cannot connect to secondary server 13:40:36 DR: Primary server connected 13:40:36 DR: Send error 13:40:36 ASF Echo-Thread Server: asfcode = -25580: oserr = 32: errstr = : System error occurred in network function. System error = 32. 13:40:36 DR: Failure recovery error (2) 13:40:37 DR: Turned off on primary server 13:40:37 DR: Cannot connect to secondary server 13:40:48 DR: Primary server connected 13:40:49 DR: Secondary server needs failure recovery Secondary: 13:40:35 DR: Received connection request from remote server when DR is not Off [Local type: Secondary, Current state: ?] [Remote type: Primary] 13:40:42 DR: ping timeout 13:40:42 DR: Receive error 13:40:42 ASF Echo-Thread Server: asfcode = -25582: oserr = 4: errstr = : Network connection is broken. System error = 4. 13:40:44 DR: Turned off on secondary server 13:40:47 DR: Secondary server connected 13:40:49 DR: Secondary server needs failure recovery 13:40:49 DR: Failure recovery from disk in progress ... 13:41:24 DR: Received connection request from remote server when DR is not Off [Local type: Secondary, Current state: ?] [Remote type: Primary] 13:41:36 DR: Received connection request from remote server when DR is not Off [Local type: Secondary, Current state: ?] HDR failed to reconnect for some 40 minutes during which time ER failed to send transactions. HDR only restarted when the secondary engine was restarted. Thoughts would be appreciated, but please tell me if this is better as a PMR rather than discussion here. Keith Response: Well, I'm not Madison, but what you've shown is not expected behavior. When the secondary got it's ping timeout, it should have shutdown the dr threads and then allowed the primary to reconnect. It appears something went wrong, which was preventing the secondary from handling the ping timeout properly so it then wasn't allowing the primary to reconnect. It should be a pmr, but the problem is if there wasn't data collected from when it happened, it might need to happen again to figure out what is happening (assuming that it doesn't happen at every ping timeout). Jacques Renaut IBM Informix Advanced Support APD Team