ER errors after 11.7 FC2 upgrade

Posted in 2011

Summary

After upgrading a test HP-UX 11.23 environment from 11.70.FC1 to FC2, Enterprise Replication would not start or stop: error -27000, "Cannot stat ATS directory", CDR DDL "Could not open table 'cdrddlpt'" (isam 29/sql -206), grouper aborts and repeated CDR internal shutdowns. Wiping syscdr, downgrading to 10.00 and rerunning the cdr conversion script didn't help. Madison Pruet asked about the conversion script and offered to look at a syscdr dbexport. The poster eventually found the cause himself: the ER server was using a shared-memory connection for the DBSERVERNAME; switching that entry to a TCP connection resolved the problem.

Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.

Topics: Installation, Setup & Upgrades

JARED HEATH — Thu, 31 Mar 2011 18:08:51 — source: IIUG Forums & Mailing Lists

we are doing testing on 11.7 and pushed our test env up to FC2. I cannot get ER to work on it at all. Leaving 11.7 FC1, replicates all working, and stopped. Convert to FC2, replicatation complains its on, and then that its off when trying to stop it. I've gone through everything I can think if, including wiping out the syscdr databases to no avail. (This has always worked in the past for me). Wit the cdr databases gone, I get a -27000 error, which won't go away with a restart of the engine or a reboot of the server.

↪ replying to JARED HEATH

JARED HEATH — Thu, 31 Mar 2011 18:10:38 — source: IIUG Forums & Mailing Lists

HP-UX 11.23 64bit OSes on both ends of the replication

↪ replying to JARED HEATH

JARED HEATH — Thu, 31 Mar 2011 22:57:19 — source: IIUG Forums & Mailing Lists

Got past the 27000 error by downgrading everything to 10.00, reloading the DBs, then going through the migration, and the concdr.sh script. It tries to come up, but now I'm getting this in the online log: 17:41:20 CDR GC: catalog recovery begin 17:41:21 Cannot stat ATS directory /usr1/isv/informix/repl 17:41:21 CDR GC: recovery of server g_hpst117ux failed (error = 78) 17:41:21 CDR GC: server recovery failed 17:41:21 CDR GC: catalog recovery failed 17:41:21 CDR queuer initialization complete 17:41:21 CDR NIF listening on asf://g_hpst117ux 17:41:21 ER checkpoint started 17:41:21 Checkpoint Completed: duration was 0 seconds. 17:41:21 Thu Mar 31 - loguniq 18211, logpos 0x146018, timestamp: 0xaae4f3c0 Interval: 13 17:41:21 Maximum server connections 2 17:41:21 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 164, Llog used 324 17:41:21 ER checkpoint logid:18211 logpos:0x146018 17:41:21 DDR Log Snooping - Snooping started in log 18211 17:41:21 CDR DDL: Could not open table 'cdrddlpt': isam=29 sql=-206 17:41:21 CDR Grouper FanOut thread is aborting. 17:41:22 DDR Log Snooping - Shutdown 17:41:22 CDR is shutting down due to internal error: Internal shutdown 17:41:22 CDR NIF Shutdown: connections all shutdown. 17:41:22 CDR The NIF sub-component has shut down. 17:41:22 CDR GC: operation gcStop failed (error 1). 17:42:09 CDR GC: catalog recovery begin 17:42:12 CDR GC: catalog recovery complete 17:42:12 CDR queuer initialization complete 17:42:12 CDR NIF listening on asf://g_hpst117ux 17:42:12 DDR Log Snooping - Snooping started in log 18211 17:42:12 CDR DDL: Could not open table 'cdrddlpt': isam=29 sql=-206 17:42:12 CDR Grouper FanOut thread is aborting. 17:42:12 DDR Log Snooping - Shutdown 17:42:12 CDR is shutting down due to internal error: Internal shutdown 17:42:12 CDR RQM: rqmSpoolHeader() failed for KEY (80/0/0x000000000000005c/0x0000000000000000) status=-1 SQLERR=0 thread 129 queue control_send 17:42:12 CDR RQM: rqmSpoolTxn() failed for KEY (80/0/0x000000000000005c/0x0000000000000000) status=-1 SQLERR=0 thread 129 queue control_send 17:42:12 CDR Queuer: rqmQueueTxn (80/0/0x5c/0x0) failed on CNTRL 17:42:12 CDR GC: operation control message construction failed (error -1). 17:42:15 CDR connection to server lost, id 82, name <g_repst117ux> Reason: ASF connect error (-27002) 17:42:15 CDR NIF Shutdown: connections all shutdown. 17:42:15 CDR The NIF sub-component has shut down. 17:42:15 CDR GC: operation gcStop failed (error 1). 17:43:19 (46) connection rejected - no calls allowed for cdraccept 17:43:37 CDR GC: catalog recovery begin 17:43:39 CDR GC: catalog recovery complete 17:43:39 CDR queuer initialization complete 17:43:39 CDR NIF listening on asf://g_hpst117ux 17:43:39 DDR Log Snooping - Snooping started in log 18211 17:43:39 CDR DDL: Could not open table 'cdrddlpt': isam=29 sql=-206 17:43:39 CDR Grouper FanOut thread is aborting. 17:43:39 CDR connection to server lost, id 82, name <g_repst117ux> Reason: ASF connect error (-27002) 17:43:39 DDR Log Snooping - Shutdown 17:43:39 CDR is shutting down due to internal error: Internal shutdown 17:43:39 CDR NIF Shutdown: connections all shutdown. 17:43:39 CDR The NIF sub-component has shut down. 17:43:39 CDR GC: operation gcStop failed (error 1). I'm seeing some things here that make me think ER in 11.70.FC2 isn't working.

↪ replying to JARED HEATH

Madison Pruet — Thu, 31 Mar 2011 23:33:28 — source: IIUG Forums & Mailing Lists

I'm confused. Are you on IDS 10 or 11.7? Why don't you send me a dbexport of your syscdr database --- assuming that the queues are currently empty... From: "JARED HEATH" <jared.heath@sbcglobal.net> To: ids@iiug.org Date: 03/31/2011 05:58 PM Subject: Re: ER errors after 11.7 FC2 upgrade [23277] Sent by: ids-bounces@iiug.org Got past the 27000 error by downgrading everything to 10.00, reloading the DBs, then going through the migration, and the concdr.sh script. It tries to come up, but now I'm getting this in the online log: 17:41:20 CDR GC: catalog recovery begin 17:41:21 Cannot stat ATS directory /usr1/isv/informix/repl 17:41:21 CDR GC: recovery of server g_hpst117ux failed (error = 78) 17:41:21 CDR GC: server recovery failed 17:41:21 CDR GC: catalog recovery failed 17:41:21 CDR queuer initialization complete 17:41:21 CDR NIF listening on asf://g_hpst117ux 17:41:21 ER checkpoint started 17:41:21 Checkpoint Completed: duration was 0 seconds. 17:41:21 Thu Mar 31 - loguniq 18211, logpos 0x146018, timestamp: 0xaae4f3c0 Interval: 13 17:41:21 Maximum server connections 2 17:41:21 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 164, Llog used 324 17:41:21 ER checkpoint logid:18211 logpos:0x146018 17:41:21 DDR Log Snooping - Snooping started in log 18211 17:41:21 CDR DDL: Could not open table 'cdrddlpt': isam=29 sql=-206 17:41:21 CDR Grouper FanOut thread is aborting. 17:41:22 DDR Log Snooping - Shutdown 17:41:22 CDR is shutting down due to internal error: Internal shutdown 17:41:22 CDR NIF Shutdown: connections all shutdown. 17:41:22 CDR The NIF sub-component has shut down. 17:41:22 CDR GC: operation gcStop failed (error 1). 17:42:09 CDR GC: catalog recovery begin 17:42:12 CDR GC: catalog recovery complete 17:42:12 CDR queuer initialization complete 17:42:12 CDR NIF listening on asf://g_hpst117ux 17:42:12 DDR Log Snooping - Snooping started in log 18211 17:42:12 CDR DDL: Could not open table 'cdrddlpt': isam=29 sql=-206 17:42:12 CDR Grouper FanOut thread is aborting. 17:42:12 DDR Log Snooping - Shutdown 17:42:12 CDR is shutting down due to internal error: Internal shutdown 17:42:12 CDR RQM: rqmSpoolHeader() failed for KEY (80/0/0x000000000000005c/0x0000000000000000) status=-1 SQLERR=0 thread 129 queue control_send 17:42:12 CDR RQM: rqmSpoolTxn() failed for KEY (80/0/0x000000000000005c/0x0000000000000000) status=-1 SQLERR=0 thread 129 queue control_send 17:42:12 CDR Queuer: rqmQueueTxn (80/0/0x5c/0x0) failed on CNTRL 17:42:12 CDR GC: operation control message construction failed (error -1). 17:42:15 CDR connection to server lost, id 82, name <g_repst117ux> Reason: ASF connect error (-27002) 17:42:15 CDR NIF Shutdown: connections all shutdown. 17:42:15 CDR The NIF sub-component has shut down. 17:42:15 CDR GC: operation gcStop failed (error 1). 17:43:19 (46) connection rejected - no calls allowed for cdraccept 17:43:37 CDR GC: catalog recovery begin 17:43:39 CDR GC: catalog recovery complete 17:43:39 CDR queuer initialization complete 17:43:39 CDR NIF listening on asf://g_hpst117ux 17:43:39 DDR Log Snooping - Snooping started in log 18211 17:43:39 CDR DDL: Could not open table 'cdrddlpt': isam=29 sql=-206 17:43:39 CDR Grouper FanOut thread is aborting. 17:43:39 CDR connection to server lost, id 82, name <g_repst117ux> Reason: ASF connect error (-27002) 17:43:39 DDR Log Snooping - Shutdown 17:43:39 CDR is shutting down due to internal error: Internal shutdown 17:43:39 CDR NIF Shutdown: connections all shutdown. 17:43:39 CDR The NIF sub-component has shut down. 17:43:39 CDR GC: operation gcStop failed (error 1). I'm seeing some things here that make me think ER in 11.70.FC2 isn't working. ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

↪ replying to Madison Pruet

JARED HEATH — Fri, 01 Apr 2011 14:45:17 — source: IIUG Forums & Mailing Lists

To answer your question...both. Our production is still 10.00.FC6 Our upgrade environment has been 11.70.FC1 for about 2 weeks. When FC2 came out, I went through the process of upgrading it. That's when the ER problems started appearing. I've downgraded/wiped and restarted back to both 10 and FC1 mulitple times and not gotten ER to work yet in FC2 at all. I'm currently working with IBM

↪ replying to JARED HEATH

Madison Pruet — Fri, 01 Apr 2011 15:10:23 — source: IIUG Forums & Mailing Lists

Did you run the cdr conversion script? From: "JARED HEATH" <jared.heath@sbcglobal.net> To: ids@iiug.org Date: 04/01/2011 09:46 AM Subject: Re: ER errors after 11.7 FC2 upgrade [23298] Sent by: ids-bounces@iiug.org To answer your question...both. Our production is still 10.00.FC6 Our upgrade environment has been 11.70.FC1 for about 2 weeks. When FC2 came out, I went through the process of upgrading it. That's when the ER problems started appearing. I've downgraded/wiped and restarted back to both 10 and FC1 mulitple times and not gotten ER to work yet in FC2 at all. I'm currently working with IBM ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

↪ replying to Madison Pruet

JARED HEATH — Fri, 01 Apr 2011 18:03:36 — source: IIUG Forums & Mailing Lists

from 10.0, yes. It didn't make a difference, still gave -27000. you can't run the script from FC1

↪ replying to JARED HEATH

JARED HEATH — Fri, 01 Apr 2011 21:38:43 — source: IIUG Forums & Mailing Lists

I've found the root of the issue. It was entirely usage of a shared memory connection vs the DBSERVERNAME value, which is something we were advised to do years ago by a consultant. Making this be a tcp connection instead made the entire issue go away.

Related threads

Error in TLR with Archecker