RSS Failed recover

Hi I have HADR set in placed which had stabled since past 2 years. Recently there was IP change in our Prod nodes Pri and Sec. Prior changed we have tested to make sure all nodes including DR able to communicate as usual. All good to go. Unfortunately once changed, the RSS not able to replicate with PRI node. Following I shared the status : On PRIMARY IBM Informix Dynamic Server Version 12.10.FC14 -- On-Line (Prim) -- Up 2 days 09:44:30 -- 46353692 KbytesPrimary Server : ol_csmsagahdrtcp Current Log Page:14472,247189 Index page logging status: Enabled Index page logging was enabled at: 2022/03/17 12:07:23 Server ACKed Log Applied Log Supports Status (log, page) (log, page) Updates ol_csmsagapritcp 14472,247189 14472,247188 No NEAR_SYNC(HDR),Connected,On ol_csmsagarss02tcp 0,0 0,0 No ASYNC(RSS),Disconnected,Defined Although I deleted or given another instance name for rss , it still not able to replicate . On Primary log - ol_csmsagahdrtcp 15:51:43 SMX thread is exiting 15:51:45 smx creates 1 transports to server ol_csmsagarss02tcp 15:51:45 SMX thread is exiting 15:51:47 smx failure smx_connect failed rc = -1 15:51:47 rolling upgrade version mismatch ol_csmsagarss02tcp rc = -1 On RSS log - ol_csmsagarss02tcp IBM Informix Dynamic Server Version 12.10.FC14 -- Fast Recovery -- Up 2 days 01:52:19 -- 46353692 Kbytes Message Log File: /opt/IBM/Enterprise/informix/ids/12.00/log/online.log15:32:12 Maximum server connections 0 15:32:12 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 0, Llog used 0 15:32:13 Physical Restore of rootdbs, datadbs, logdbs, phydbs Completed . 15:32:13 Checkpoint Completed: duration was 0 seconds. 15:32:13 Sun Aug 18 - loguniq 14424, logpos 0x2d018, timestamp: 0x7fa8b40d Interval: 239386 15:32:13 Maximum server connections 0 15:51:55 DR: Reservation of the last logical log for log backup turned off 15:51:55 DR: new type = RSS 15:51:55 smx creates 1 transports to server ol_csmsagahdrtcp 15:51:56 SMX thread is exiting 15:51:58 Data replication type and state information reset. To start DR, usethe 'onmode -d' command and wait for the pair to be operational, before shutting down the database server 15:51:58 SMX thread is exiting 15:51:59 Defragmenter cleaner thread now running 15:51:59 Defragmenter cleaner thread cleaned:0 partitionsBoth PRI and RSS - sqlhosts #demo_on onipcshm on_hostname on_servername ol_csmsaga onipcshm pksdrsgadb01 ol_csmsaga ol_csmsagatcp onsoctcp pksdrsgadb01 service_csmsagatcp# Replication Master & DRC ol_csmsagarss02tcp onsoctcp pksdrsgadb01 sqlexec ol_csmsagapritcp onsoctcp pksprodsgadb01 sqlexec ol_csmsagahdrtcp onsoctcp pksprodsgadb02 sqlexecSeek advise to overcome the issue. Had tried restart the instance PRI and RSS onfact reboot the server but still the same. ------------------------------ ABeMie ------------------------------

The log is telling you that there is a rolling upgrade mismatch. Are you sure you have got the same versions at both instances ? It seems there has not only been an IP change but an upgrade to a newer (sub-) release.

Actually what does it mean ?. It's was not a first time I'm doing recover of RSS node server. As you can see from RSS online log, I'm doing physical restore from PRI full backup which has completed. ------------------------------ ABeMie ------------------------------

Weird, if you are doing a full restore with same release, this does not make sense to me. A rolling upgrade mismatch is the main reason for the connection to be broken according to the logfile. Did you take the backup before or after the upgrade of version ? You need to identify at which point a rolling upgrade did not finish.

Hi Marcus There is no upgrade release because both PRI and RSS including SEC are same IBM IDS version and release..It was weird since I never got such error So i do not know how to answer to your query. ------------------------------ ABeMie ------------------------------

Did you change IPs in DNS? What does `onstat -g rss` tell? What commands do you run to start the RSS? ------------------------------ Sincerely, Dennis ------------------------------

This may seem an odd question, but are you using Encryption at Rest? I have messages in the Informix logs after an HDR disconnect/reconnect that complain about a rolling upgrade version mismatch, even though both instances are at the same release level (14.10.FC6WE), however in my case, replication does reconnect okay. Have you seen this APAR? IT13994: SMX_CONNECT FAILURE DURING BOUNCE IN HDR ENVIRONMENT FOLLOWED BY A MISLEADING ERROR ABOUT ROLLING UPGRADE VERSION MISMATCH (ibm.com) It suggests unsetting or increasing INFORMIXCONTIME/INFORMIXCONRETRY. ------------------------------ Mike Walker xDB Systems, Inc www.xdbsystems.com ------------------------------

HI Mike That was not the issue because both PRI and RSS having value INFORMIXCONRETRY 1 and INFORMIXCONTIME 60 ------------------------------ ABeMie ------------------------------

Hi. The current RSS show as following status [informix@pksdrsgadb01 ~]$ onstat -g rss IBM Informix Dynamic Server Version 12.10.FC14 -- Fast Recovery -- Up 3 days 04: 57:40 -- 46353692 KbytesLocal server type: Standard Index page logging status: Enabled Index page logging was enabled at: 2022/03/17 12:07:23 Number of RSS servers: 0 2. After completed the physical restored, I issued command onmode -d RSS <primary_instance> 3. Will there be an issue if the RSS server got time different - eg. greater then PRI for 10 - 14 seconds ? ------------------------------ ABeMie ------------------------------

Is HDR already in place, and running? Run "onstat -g cluster" on your primary server so we can see what is already in place. When setting up RSS with the onmode -d RSS, isn't o l_csmsagapritcp y our PRIMARY and not ol_csmsagahdrtcp ? This message, " 15:51:55 smx creates 1 transports to server ol_csmsagahdrtcp"makes me think that you are trying to connect to an HDR instance on pksprodsgadb02 and not the PRIMARY on pksprodsgadb01. ------------------------------ Mike Walker xDB Systems, Inc www.xdbsystems.com ------------------------------

Hi Mike My initial post message already stated the output of onstat -g cluster. Yes the name "hdr" is our primary and it been at that status since last 2 years. The name "pri" is our secondary. On PRIMARY IBM Informix Dynamic Server Version 12.10.FC14 -- On-Line (Prim) -- Up 2 days 09:44:30 -- 46353692 KbytesPrimary Server : ol_csmsagahdrtcp Current Log Page:14472,247189 Index page logging status: Enabled Index page logging was enabled at: 2022/03/17 12:07:23 Server ACKed Log Applied Log Supports Status (log, page) (log, page) Updates ol_csmsagapritcp 14472,247189 14472,247188 No NEAR_SYNC(HDR),Connected,On ol_csmsagarss02tcp 0,0 0,0 No ASYNC(RSS),Disconnected,Defined ------------------------------ ABeMie ------------------------------

When you delete the rss from the primary, have you confirmed that it has gone away when you run onstat -g cluster on the primary? I would try changing the sqlhosts files to put in the IP addresses instead of the hostnames and see if that gets you any further. ------------------------------ Mike Walker xDB Systems, Inc www.xdbsystems.com ------------------------------

When you delete the rss from the primary, have you confirmed that it has gone away when you run onstat -g cluster on the primary? Yes , I can confirmed it deleted after ran the onstat -g cluster command. The hostname was referred to /etc/hosts. file. The changed of IP we just do it at /etc/hosts file. I try not to edit the setting in sqlhosts. But anyway I can try as your suggestion as well. But i still think what mismatch it been refer too. Meanwhil as work arpound since this is production, I temporary resync the logical log backup to RSS server on every 15 minute until this get resolve. ------------------------------ ABeMie ------------------------------

Hi This the latest result after i chnaged to IP Prior changed idelete the RSS and do add again with previous name On Pri 10:40:27 smx creates 1 transports to server ol_csmsagarsstcp 10:40:50 The SMX connection between high availability servers was closed because thepeer server was unresponsive for the timeout period (60 seconds times the number of retries). 10:40:52 SC_maxmsg_ping failed with rc -36 10:40:52 rolling upgrade version mismatch ol_csmsagarsstcp rc = -1ON RSS DR: new type = RSS 10:39:23 smx creates 1 transports to server ol_csmsagahdrtcp 10:40:48 The SMX connection between high availability servers was closed because thepeer server was unresponsive for the timeout period (60 seconds times the number of retries). ------------------------------ ABeMie ------------------------------

It seems you have a connectivity issue on ip layer. Can you try if the servername/port can connect using telnet? Might also be an arp issue. You can clean the arp cache and try again. Any firewall between the machines? Marcus Gesendet von Outlook für Android

Hi Able to telnet server and port vice versa. Traceroute between both server vice versa also good. Firewall service at server level was stopped while for client network, it has been set accordingly as port requested. Please share how to clean the arp cache . Thanks ------------------------------ ABeMie ------------------------------

If you are able to connect via hostname which is in sqlhosts and th same port, arp should not be an issue. Gesendet von Outlook für Android

On your RSS server, can you connect with dbaccess to the Primary server? e.g. dbaccess sysmaster@ ol_csmsagahdrtcp Check the log on the Primary to see if it shows any messages indicating that there might be a problem. ------------------------------ Mike Walker xDB Systems, Inc www.xdbsystems.com ------------------------------

Yes ... no issue to do dbaccess from RSS to Pri server ol_csmsagahdrtcp ------------------------------ ABeMie ------------------------------

Very strange, I would try to do a fresh backup and initialize the rss from backup again. Gesendet von Outlook für Android

Am also facing almost same situation ------------------------------ Hephorus Khan ------------------------------

Hephorus - can you give us some background? Did you have a change to IP addresses also? ------------------------------ Mike Walker xDB Systems, Inc www.xdbsystems.com ------------------------------

I know this is an old-ish thread, but I wanted to give an update that in my case (HDR), a simple restart of informix on the Primary server fixed the problem. For anybody interested, the full story was that a few months ago we had a DR outage following a network interruption. One of the 3 HDR instances reported the following in the log: 06:02:00 SC_maxmsg_ping failed with rc -30 06:02:00 DR: Receive error 06:02:00 dr_accept thread : asfcode = -25582: oserr = 0: errstr = : Network connection is broken. 06:02:00 DR_ERR set to -1 06:02:11 SMX thread is exiting because of network error code -25582 06:02:11 SMX Handshake Failure at line 4199 06:02:11 SMX thread is exiting because of network error code -25582 06:02:11 SMX Handshake Failure at line 4199 06:02:41 SC_maxmsg_ping failed with rc -30 06:02:41 DR: Received connection request from remote server when DR is not Off [Local type: Secondary, Current state: On] [Remote type: Primary]These messages were repeated continually. On restarting the HDR, the log showed: 06:45:32 Physical Recovery Started at Page (2:10758). 06:45:32 Physical Recovery Complete: 0 Pages Examined, 0 Pages Restored. 06:45:32 DR: Trying to connect to primary server = XXXXX 06:45:32 SMX thread is exiting because of network error code -25582 06:45:32 SMX thread is exiting because of network error code -25582 06:45:32 SMX Handshake Failure at line 4199 06:45:32 SMX Handshake Failure at line 4199 06:45:32 SC_maxmsg_ping failed with rc -21 06:45:32 rolling upgrade version mismatch XXXXX 06:45:32 SMX thread is exiting because of network error code -25582 06:45:32 SMX Handshake Failure at line 4199 06:45:32 DR: Cannot connect to primary server 06:45:32 DR: Turned off on secondary server 06:45:32 Encryption-at-rest is enabled using cipher 'aes128' 06:45:32 Dataskip is now OFF for all dbspaces 06:45:32 Restartable Restore has been ENABLED 06:45:32 Recovery Mode 06:45:33 SMX Handshake Failure at line 4188 06:45:33 SMX thread is exiting because of network error code -25582 06:45:33 SMX Handshake Failure at line 4199 06:45:33 SC_maxmsg_ping failed with rc -21 06:45:33 DR: Receive error 06:45:33 dr_accept thread : asfcode = -25582: oserr = 0: errstr = : Network connection is broken. . . 06:45:38 DR: Failure recovery from disk in progress ... 06:45:38 Logical Recovery Started. 06:45:38 10 recovery worker threads will be started. 06:45:38 Start Logical Recovery - Start Log 5541, End Log ? 06:45:38 Starting Log Position - 5541 0x73da018 06:45:50 Started processing open transactions on secondary during startupServer in fast recovery until these open transactions finish 06:45:50 SCHAPI: dbutil threads is already running. 06:45:50 SMX thread is exiting because of network error code -25582 06:45:50 SMX Handshake Failure at line 4199 06:45:50 Finished processing open transactions on secondary during startup. 06:45:50 Checkpoint Completed: duration was 0 seconds. 06:45:50 Fri Jul 19 - loguniq 5541, logpos 0x7403018, timestamp: 0x424e176c Interval: 1456286Despite these messages, HDR did reconnect. It seemed that all was fine, until recently when we started getting problems with extending a chunk on the Primary. This would fail, and the HDR would show: 08:09:15 SMX thread is exiting because of network error code -25582 08:09:15 SMX Handshake Failure at line 4199I performed a backup of the Primary and restored to the Secondary. When enabling HDR again, I would get the following messages and replication did NOT establish: 09:21:55 DR: Trying to connect to primary server = XXXXX 09:21:55 SMX thread is exiting because of network error code -25582 09:21:55 SMX Handshake Failure at line 4199 09:21:55 SMX thread is exiting because of network error code -25582 09:21:55 SMX Handshake Failure at line 4199 09:21:55 SC_maxmsg_ping failed with rc -21 09:21:55 rolling upgrade version mismatch XXXXXI also saw the rolling upgrade messages on the Primary: 09:21:23 DR: new type = primary, secondary server name = XXXXX 09:21:23 SC_maxmsg_ping failed with rc -21 09:21:23 rolling upgrade version mismatch XXXXX 09:21:55 SC_maxmsg_ping failed with rc -21 09:21:56 SC_maxmsg_ping failed with rc -21 09:21:56 rolling upgrade version mismatch XXXXXI set the Primary server to "standard" mode. Later I restarted the former Primary instance, then did a fresh backup and restored to the Secondary, followed by establishing HDR again. There were NO errors reported in the logs, and nothing about a rolling upgrade mismatch. I was then able to extend chunks with no problem. So, the solution for me was to (1) set the Primary to "standard", disabling the replication. (2) We took a production outage, and restarted Informix on the former Primary. (3) Then use a backup and restore to the Secondary and enabling HDR. I suspect that there was an issue with the SMX threads on the Primary, or the dr_ping threads. I don't remember ever running "onstat -g smx", which in retrospect would have been a good idea. ------------------------------ Mike Walker xDB Systems, Inc www.xdbsystems.com ------------------------------