Determining cause of DR ping timeout

Good morning IDS community. It seems to happen intermittently, and there doesn't seem to be any physical / logical network switch causes, for DR ping timeout with an HDR Primary and HDR Read-Only Secondary IDS 11.50.FC6 servers. Both are Dell R610 x86-64bit systems with SuSE Linux 11.00 systems. I have a separate NIC interface and vLan for replication traffic for Logical Logs from the Primary to the Secondary systems. The only clue I get is out of the ONLINE LOG: 08/27/12 04:31:52 Logical Log 34090 Complete, timestamp: 0x2ae64284. 08/27/12 04:36:09 Logical Log 34091 Complete, timestamp: 0x2afb8012. 08/27/12 04:40:46 DR: ping timeout 08/27/12 04:40:57 DR: Received connection request from remote server when DR is not Off [Local type: Secondary, Current state: FAILED] [Remote type: Primary] What tools or techniques can you recommend to trouble-shoot the DR ping timeout problem? Thank-you for any suggestions or recommendations. Jonathan B. Smaby Pomona College -- "If you are not willing to risk the unusual, you will have to settle for the ordinary." - Jim Rohn ------------------------------------------------------------- This message has been scanned by Postini anti-virus software.

Jonathan No help I'm afraid, just to observe I'm getting this issue intermittently on IDS9.4 and am disappointed to see it on a more modern engine :-(( Are you running HDR over a LAN or WAN connection ?? Do you get any network hiccups (however brief) or maxing of any part of the link between the servers ? See my recently resurrected thread for detail needed when you raise your PMR !! Keith On 28 August 2012 17:54, Jonathan Smaby <Jonathan.Smaby@pomona.edu> wrote: > Good morning IDS community. > > It seems to happen intermittently, and there doesn't seem to be any physical / > logical network switch causes, for DR ping timeout with an HDR Primary and HDR > Read-Only Secondary IDS 11.50.FC6 servers. Both are Dell R610 x86-64bit > systems with SuSE Linux 11.00 systems. I have a separate NIC interface and > vLan for replication traffic for Logical Logs from the Primary to the > Secondary systems. The only clue I get is out of the ONLINE LOG: > > 08/27/12 04:31:52 Logical Log 34090 Complete, timestamp: 0x2ae64284. > 08/27/12 04:36:09 Logical Log 34091 Complete, timestamp: 0x2afb8012. > 08/27/12 04:40:46 DR: ping timeout > 08/27/12 04:40:57 DR: Received connection request from remote server when DR > is not Off > > [Local type: Secondary, Current state: FAILED] > > [Remote type: Primary] > > What tools or techniques can you recommend to trouble-shoot the DR ping > timeout problem? > > Thank-you for any suggestions or recommendations. > > Jonathan B. Smaby > Pomona College > -- > "If you are not willing to risk the unusual, you will have to settle for the > ordinary." - Jim Rohn > > ------------------------------------------------------------- > This message has been scanned by Postini anti-virus software. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >

Thanks Keith, My two systems are connected via LAN at 1Gbps on the same pair of stacked Cisco 3750X switches. No maxing-out of network links. The replication happens over dedicated private vLan links to avoid contention for bandwidth. I also have very small databases, most are less than 4 Gb. I've also turned off logging for Temp Tables (TEMPTAB_NOLOG 1) in ONCONFIG file. The one and only thing I can think of is I'm using 802.3ad LCAP network load-balancing and high-availability with 2 network interfaces acting as a Linux Bond interface "Bond0". The Cisco 3750X switches and the Novell SuSE Linux OS are correctly configured for 802.3ad port channel. Are there any tools IBM has to monitor HDR and DR ping? Is there verbosity settings to increase detail and information in the online log? Thanks for any advice. Jonathan B. Smaby Pomona College -- "If you are not willing to risk the unusual, you will have to settle for the ordinary." - Jim Rohn -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Keith Simmons Sent: Tuesday, August 28, 2012 10:31 AM To: ids@iiug.org Subject: Re: Determining cause of DR ping timeout [28182] Jonathan No help I'm afraid, just to observe I'm getting this issue intermittently on IDS9.4 and am disappointed to see it on a more modern engine :-(( Are you running HDR over a LAN or WAN connection ?? Do you get any network hiccups (however brief) or maxing of any part of the link between the servers ? See my recently resurrected thread for detail needed when you raise your PMR !! Keith On 28 August 2012 17:54, Jonathan Smaby <Jonathan.Smaby@pomona.edu> wrote: > Good morning IDS community. > > It seems to happen intermittently, and there doesn't seem to be any > physical / > logical network switch causes, for DR ping timeout with an HDR Primary > and HDR > Read-Only Secondary IDS 11.50.FC6 servers. Both are Dell R610 > x86-64bit systems with SuSE Linux 11.00 systems. I have a separate NIC > interface and vLan for replication traffic for Logical Logs from the > Primary to the Secondary systems. The only clue I get is out of the ONLINE LOG: > > 08/27/12 04:31:52 Logical Log 34090 Complete, timestamp: 0x2ae64284. > 08/27/12 04:36:09 Logical Log 34091 Complete, timestamp: 0x2afb8012. > 08/27/12 04:40:46 DR: ping timeout > 08/27/12 04:40:57 DR: Received connection request from remote server > when DR is not Off > > [Local type: Secondary, Current state: FAILED] > > [Remote type: Primary] > > What tools or techniques can you recommend to trouble-shoot the DR > ping timeout problem? > > Thank-you for any suggestions or recommendations. > > Jonathan B. Smaby > Pomona College > -- > "If you are not willing to risk the unusual, you will have to settle > for the ordinary." - Jim Rohn > > ------------------------------------------------------------- > This message has been scanned by Postini anti-virus software. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum. ------------------------------------------------------------- This message has been scanned by Postini anti-virus software.

Original post: Good morning IDS community. It seems to happen intermittently, and there doesn't seem to be any physical / logical network switch causes, for DR ping timeout with an HDR Primary and HDR Read-Only Secondary IDS 11.50.FC6 servers. Both are Dell R610 x86-64bit systems with SuSE Linux 11.00 systems. I have a separate NIC interface and vLan for replication traffic for Logical Logs from the Primary to the Secondary systems. The only clue I get is out of the ONLINE LOG: 08/27/12 04:31:52 Logical Log 34090 Complete, timestamp: 0x2ae64284. 08/27/12 04:36:09 Logical Log 34091 Complete, timestamp: 0x2afb8012. 08/27/12 04:40:46 DR: ping timeout 08/27/12 04:40:57 DR: Received connection request from remote server when DR is not Off [Local type: Secondary, Current state: FAILED] [Remote type: Primary] What tools or techniques can you recommend to trouble-shoot the DR ping timeout problem? Thank-you for any suggestions or recommendations. Jonathan B. Smaby Pomona College Response: In newer versions of the engine you could monitor onstat -g dri output as it has the time of the last send/receive which would at least give you an indication of some problem (as it should be sending things from the primary to the secondary nearly constantly or at least every DRINTERVAL). However, many ping time outs are caused by problems that have nothing to do with the network connection between the two servers. If there is a problem with the secondary where it can't keep up with the logs being shipped across or it gets into a hang situation, that can/will interfere with the ping mechanism the server uses and would get reported as a ping timeout, but the reason the primary can't send a ping is because the secondary server has basically stopped receiving messages, which again isn't because the network is messed up. Your best bet would be to open a case with tech support assuming you have support. Jacques Renaut IBM Informix Advanced Support APD Team

I also hate this non-descriptive errors. What Jacques wrote is very important and fits my experience.... Meanwhile what are your settings for DRTIMEOUT? What do you get on *both* online.log? Do you see the primary sessions with flag "G"? Tech support may suggest you some traces to gather more info. Regards On Aug 28, 2012 5:54 PM, "Jonathan Smaby" <Jonathan.Smaby@pomona.edu> wrote: > Good morning IDS community. > > It seems to happen intermittently, and there doesn't seem to be any > physical / > logical network switch causes, for DR ping timeout with an HDR Primary and > HDR > Read-Only Secondary IDS 11.50.FC6 servers. Both are Dell R610 x86-64bit > systems with SuSE Linux 11.00 systems. I have a separate NIC interface and > vLan for replication traffic for Logical Logs from the Primary to the > Secondary systems. The only clue I get is out of the ONLINE LOG: > > 08/27/12 04:31:52 Logical Log 34090 Complete, timestamp: 0x2ae64284. > 08/27/12 04:36:09 Logical Log 34091 Complete, timestamp: 0x2afb8012. > 08/27/12 04:40:46 DR: ping timeout > 08/27/12 04:40:57 DR: Received connection request from remote server when > DR > is not Off > > [Local type: Secondary, Current state: FAILED] > > [Remote type: Primary] > > What tools or techniques can you recommend to trouble-shoot the DR ping > timeout problem? > > Thank-you for any suggestions or recommendations. > > Jonathan B. Smaby > Pomona College > -- > "If you are not willing to risk the unusual, you will have to settle for > the > ordinary." - Jim Rohn > > ------------------------------------------------------------- > This message has been scanned by Postini anti-virus software. > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --20cf303640b3206b6c04c860f83c