RSS but not HDR?
Tom Girsch could not add a new node as an HDR secondary on 14.10.FC8X8 (SC_maxmsg_ping failed with rc -21), though joining as RSS worked. Art Kagel asked about deregistering the old secondary. Tom found the IDS ports were blocked by the firewall on the new node. Clive Eisen suggested the RSS worked because the port was open when it first connected.
Auto-generated by Claude from the posts below — may be imperfect; read the full thread.
I've got a curious one -- yes, I've opened a ticket, TS012087711 -- and I'm wondering if anyone else here has run into this. I'm adding a new node to an HDR cluster and when I try to make it a full HDR secondary, it fails to join, with the primary giving "SC_maxmsg_ping failed with rc -21"; the nodes can see each other just fine, and what's really confusing is that if I tell the new node to join as RSS rather than a full HDR secondary, it works just fine. Any thoughts? IDS 14.10.FC8X8 running on 4.18.0-425.10.1.el8_7.x86_64 #1 SMP ------------------------------ TOM GIRSCH ------------------------------
Tom: Did you deregister the original HDR secondary and put the primary into standalone mode first before attempting to replace the HDR secondary with the new one? Art ------------------------------ Art S. Kagel, President and Principal Consultant ASK Database Management Corp. www.askdbmgt.com ------------------------------
Indeed, I did. Also deleted the RSS with the same name as the would-be new secondary.
Basic order of events:
1. Shut down old secondary
2. Run onmode -d standard on primary
3. Run oninit -v -PHY on the would-be new secondary
4. Run onmode -d primary [new-secondary] on the primary
5. Run onmode -d secondary [primary] on the would-be new secondary
[I've tried flipping steps 4 and 5 but it doesn't seem to matter; same result either way.]
On the would-be new secondary, I see this:
2023-02-08 16:00:57.832 Recovery Mode
2023-02-08 16:01:53.747 DR: new type = secondary, primary server name = [primary]
16:01:53 DR: Trying to connect to primary server = [primary]
2023-02-08 16:01:54.843 smx creates 1 transports to server [primary]On the primary, I see:
2023-02-08 16:01:53.579 DR: new type = primary, secondary server name = [secondary[
16:01:53 DR: Trying to connect to secondary server = [secondary]2023-02-08 16:01:53.611 DR: Cannot connect to secondary server
2023-02-08 16:01:53.611 DR: Turned off on primary server
2023-02-08 16:01:54.825 SC_maxmsg_ping failed with rc -21
2023-02-08 16:01:55.857 DR: Cannot connect to secondary server
2023-02-08 16:01:55.857 DR: Turned off on primary server
2023-02-08 16:02:45.612 DR: Cannot connect to secondary server
2023-02-08 16:02:45.612 DR: Turned off on primary server
2023-02-08 16:03:14.630 Checkpoint Completed: duration was 0 seconds.
2023-02-08 16:03:14.630 Wed Feb 8 - loguniq 160185, logpos 0x7366018, timestamp: 0xd7d9b65a Interval: 794884
2023-02-08 16:03:14.631 Maximum server connections 81
2023-02-08 16:03:14.631 Checkpoint Statistics - Avg. Txn Block Time 0.001, # Txns blocked 0, Plog used 102, Llog used 22
2023-02-08 16:03:36.825 The SMX connection between high availability servers was closed because the
peer server was unresponsive for the timeout period (60 seconds times the
number of retries).
2023-02-08 16:03:45.620 DR: Cannot connect to secondary server
2023-02-08 16:03:45.620 DR: Turned off on primary server
------------------------------
TOM GIRSCH
------------------------------
So, shut down would-be secondary, then do this:
PRI: onmode -d standard
PRI: onmode -d add RSS [secondary]
New: oninit -v -PHY
New: onmode -d RSS [primary]
Now, on the new node, I see:
2023-02-08 16:04:31.624 Recovery Mode
2023-02-08 16:04:35.539 DR: new type = RSS
2023-02-08 16:04:35.541 DR: Trying to connect to server, [primary].
2023-02-08 16:04:36.586 smx creates 1 transports to server [primary]
2023-02-08 16:04:37.644 Peer node [primary] has version 131077
2023-02-08 16:04:37.645 RSS Server [primary] - state is now connected
2023-02-08 16:04:37.650 setting version information for [primary] 131077
2023-02-08 16:04:37.824 Logical Recovery Started.
2023-02-08 16:04:37.824 10 recovery worker threads will be started.
2023-02-08 16:04:37.851 Start Logical Recovery - Start Log 160185, End Log ?
2023-02-08 16:04:37.851 Starting Log Position - 160185 0x71de018
2023-02-08 16:04:37.988 DR: Cleared 38488 KB of logical log in 0 seconds.
2023-02-08 16:04:38.537 Defragmenter cleaner thread now running
2023-02-08 16:04:38.537 Defragmenter cleaner thread cleaned:0 partitions
2023-02-08 16:04:38.544 B-tree scanners disabled.
2023-02-08 16:04:39.250 Dynamically allocated new virtual shared memory segment (size 262144KB)
2023-02-08 16:04:39.250 Memory sizes:resident:71680 KB, virtual:1835008 KB, message:0, bufferpool:1302528, no SHMTOTAL limit
2023-02-08 16:04:39.537 DR: RSS secondary server operational
2023-02-08 16:04:41.393 Started processing open transactions on secondary during startup
Server in fast recovery until these open transactions finish
2023-02-08 16:04:41.403 Finished processing open transactions on secondary during startup.
2023-02-08 16:04:41.458 Checkpoint Completed: duration was 0 seconds.
On the PRI I see simply:
2023-02-08 16:03:49.592 DR: new type = standard
2023-02-08 16:03:53.594 RSS ids__chaos_dev__b4 added
2023-02-08 16:04:37.548 Peer node ids__chaos_dev__b4 has version 131077
2023-02-08 16:04:37.549 RSS Server ids__chaos_dev__b4 - state is now connected
2023-02-08 16:04:37.556 setting version information for ids__chaos_dev__b4 131077
2023-02-08 16:08:14.680 Checkpoint Completed: duration was 0 seconds.
2023-02-08 16:08:14.680 Wed Feb 8 - loguniq 160185, logpos 0x739a018, timestamp: 0xd7d9e720 Interval: 794885
2023-02-08 16:08:14.680 Maximum server connections 84
2023-02-08 16:08:14.680 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 113, Llog used 52
...and everything stays in sync until I shut everything down to take another crack at full HDR.
Worth noting: The PRI node is an older one that I'm trying to eliminate from the cluster. It's running 14.10.FC8X8 but it's on 3.10.0-1160.83.1.el7.x86_64 #1 SMP
I'd chalk the problem up to that OS version mismatch, but the original HDR secondary in the cluster is running exactly the same Linux version as the would-be new secondary, and it connects fine. The important difference between Existing Secondary and Would Be New Secondary is that the former is a VM running under AHV while the latter is a VM running under libvirt.
------------------------------
TOM GIRSCH
------------------------------
I finally figured out the issue here, and as is so often case, there was a certain measure of PEBKAC involved. Turns out that on the new node, I hadn't opened up the IDS ports on the firewall. Once I did that, the HDR joined as expected. However, this raises a new, puzzling question: How the _hell_ did RSS work with the port block? Because it _did_ work. Perfectly. ------------------------------ TOM GIRSCH ------------------------------
Just a guess but the port WAS open when the RSS secondary connected. Closing it on many firewalls does NOT terminate existing connections, it only stops new connections. - Clive Eisen