Frequent message displayed in the log file of Informix Connection Manager.

Dear All, We have 2 connection managers running and following message is written to the log file of both the connection managers frequently :- 09:57:49 Arbitrator can not find any secondary nodes for failover. 09:57:49 Arbitrator will retry failover again... 10:05:20 Arbitrator can not find any secondary nodes for failover. 10:05:20 Arbitrator will retry failover again... 10:12:51 Arbitrator can not find any secondary nodes for failover. 10:12:51 Arbitrator will retry failover again... Following are the details of the systems :- 1) IDS is v.14.10.FC1AEE is running on the HDR, RSS, PRI databaseservers (i.e. 3 systems). 2) OS is RHEL v.7.5 64 bit on all the systems (i.e. HDR, RSS, PRI and the 2 Connection Managers). 3) Connection Manager's version is IBM Informix CSDK Version 4.10, IBM Informix-ESQL Version 4.10.FC4). The configuration files of the 2 connection managers are as follows :- Config file of CM 1:- NAME cm1 LOG 1 LOGFILE /usr/informix/cm.log EVENT_TIMEOUT 300 CLUSTER cluster_bela { INFORMIXSERVER krcl_ha SLA oltp_1 DBSERVERS=PRI SLA report_1 DBSERVERS=PRI \\ POLICY=WORKLOAD FOC ORDER=ENABLED \\ PRIORITY=1 } Config file of CM2 :- NAME cm2 LOG 1 LOGFILE /usr/informix/cm.log EVENT_TIMEOUT 300 CLUSTER cluster_bela { INFORMIXSERVER krcl_ha SLA oltp_1a DBSERVERS=PRI SLA report_1a DBSERVERS=PRI \\ POLICY=WORKLOAD FOC ORDER=ENABLED \\ PRIORITY=2 } I have tested by pinging from both the Connection Manager systems to the HDR, RSS, PRI database server systems individually for a long stretch of time and the network is very stable (i.e. there are no dropped packets). Even then, the error "Arbitrator can not find any secondary nodes for failover. Arbitrator will retry failover again..." is displayed in the log file of both the connection managers. There was mention in one of the technical documents that the above mentioned error could be because the "sysadmin" database could be missing/corrupted, but that is not our case also because the sysadmin database is build successfully as mentioned in the online.log file of the PRI database server. Wanted to know what could be the reason for the above mentioned messages in the log file of both the connection managers and how to rectify it. Thanks. ------------------------------ Neville Monteiro ------------------------------ #Informix

Hello. You did not wrote on the post, but I suppose krcl_ha is a Connection group on your sqlhosts right? Could you please paste it here too? That would help a lot. Thanks Best regards Alexandre Marini

Following is the sqlhosts entries in the sqlhosts file of both the connection managers :- krcl_ha group - - c=0,e=olr_rss olr_bela onsoctcp dcinfpr01.krcl.com olrbela g=krcl_ha #olr_sds onsoctcp drinfsds01.krcl.com olrsds g=krcl_ha olr_hdr onsoctcp dcinfhdr01.krcl.com olrhdr g=krcl_ha olr_rss onsoctcp drinfdb101.krcl.com olrrss g=krcl_ha #CMs for oltp oltp group - - c=1,e=oltp_1a oltp_1 onsoctcp dcjrpapp01 oltp g=oltp oltp_1a onsoctcp dcjrpapp02 oltp g=oltp #oltp_2 onsoctcp drmaoconn01 oltp g=oltp #CMs for reports report group - - c=1,e=report_1a report_1 onsoctcp dcjrpapp01 report g=report report_1a onsoctcp dcjrpapp02 report g=report #report_2 onsoctcp drmaoconn01 report g=report ------------------------------ Neville Monteiro ------------------------------

Hi Neville, Does that message happen every 451 seconds? That message occurs when the primary server shuts down or crashes, but that doesn't seem to be the case. How about enabling the DEBUG option in the Connection Manager configuration and viewing the logs? But even with detailed logs by the DEBUG option, it seems difficult to diagnose the problem myself. If you are having difficulty analyzing logs, please contact IBM. ------------------------------ SangGyu Jeong Software Engineer Infrasoft Seoul Korea, Republic of ------------------------------

Hello SangGyu Jeong, Yes, it looks like the frequency of this message is every 450-451 seconds. Now, what is it about this 450-451 seconds seconds that the message is displayed ? I will start the connection managers in DEBUG mode and check it. ------------------------------ Neville Monteiro ------------------------------

Hello, I had restarted both the connection managers and since the messages "Arbitrator can not find any secondary nodes for failover." and "Arbitrator will retry failover again..." are not displayed. Thanks a lot. ------------------------------ Neville Monteiro ------------------------------

Was it solved by just restarting the program? I'm glad that the problem is solved. ------------------------------ SangGyu Jeong Software Engineer Infrasoft Seoul Korea, Republic of ------------------------------

Yes, both the connection managers were restarted and the errors are not written to the log file of the connection manager since then. However, since we do not have a root cause for the issue, I am a little worried. The network team says that no changes were done on the network and no changes were done in the configuration file of both the connection managers when it was re-started, so it looks like the issue started on its own and was solved on its own. Thanks a lot SangGyu Jeong and Alexandre Marini. ------------------------------ Neville Monteiro ------------------------------

Have you ever verified the connection managers both were able to connect to all three db servers? You'd see this from "CM ... connected" messages in servers' message logs as well as in 'onstat -g cmsm' outputs on each of the server. ------------------------------ Andreas Legner ------------------------------

Yes, The connection Managers has successfully connected to all the 2 database servers (i.e. RSS, HDR, PRI). The onstat -g cmsm command on all the 2 database servers shows both the connection managers have connected to them. Thanks & Regards, Neville Monteiro. Sent with ProtonMail Secure Email.