CDR Grouper FanOut thread is aborting
On IDS 11.10.UC2 / AIX 5.3, Enterprise Replication failed with "out of virtual shared memory" and shmat SHMSEG errors, causing "CDR grouper memory allocation failed (FanOut update link)" and the CDR Grouper FanOut thread aborting. Madison Pruet advised that a full ER rebuild isn't needed: just 'cdr stop' then 'cdr start', plus increasing the shared memory segment size. After the restart the error persisted, and the log showed the grouper's replay logical log (145822) was no longer present, so Pruet recommended running 'cdr check --repair'. The thread ends there with no confirmation of the outcome.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Folks, Build Version: 11.10.UC2 Build OS: AIX 5.3 The ER does not work properly due to a "out of virtual shared memory " problem in one site. Is there any EASY DRUG to treat "CDR Grouper FanOut thread is aborting." ? (instead of recreating whole the ER) Thanks, Frank ......................... :31:00 out of virtual shared memory 13:31:00 shmat: [EMFILE][24]: out of shared memory segments, check system SHMSEG 13:31:00 out of virtual shared memory 13:31:00 CDR grouper memory allocation failed (FanOut update link). 13:31:00 shmat: [EMFILE][24]: out of shared memory segments, check system SHMSEG 13:31:00 CDR Grouper FanOut thread is aborting. 13:31:00 out of virtual shared memory ................... --0016364d2ac12500fd0494166de1
cdr stop followed by cdr start. You don't have to recreate the whole of ER. M.P. From: "FRANK" <yunyaoqu@gmail.com> To: ids@iiug.org Date: 11/02/2010 01:53 PM Subject: CDR Grouper FanOut thread is aborting [21851] Sent by: ids-bounces@iiug.org Folks, Build Version: 11.10.UC2 Build OS: AIX 5.3 The ER does not work properly due to a "out of virtual shared memory " problem in one site. Is there any EASY DRUG to treat "CDR Grouper FanOut thread is aborting." ? (instead of recreating whole the ER) Thanks, Frank .......................... :31:00 out of virtual shared memory 13:31:00 shmat: [EMFILE][24]: out of shared memory segments, check system SHMSEG 13:31:00 out of virtual shared memory 13:31:00 CDR grouper memory allocation failed (FanOut update link). 13:31:00 shmat: [EMFILE][24]: out of shared memory segments, check system SHMSEG 13:31:00 CDR Grouper FanOut thread is aborting. 13:31:00 out of virtual shared memory .................... --0016364d2ac12500fd0494166de1 ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.
You also might want to increase the size of your shem segments. From: "FRANK" <yunyaoqu@gmail.com> To: ids@iiug.org Date: 11/02/2010 01:53 PM Subject: CDR Grouper FanOut thread is aborting [21851] Sent by: ids-bounces@iiug.org Folks, Build Version: 11.10.UC2 Build OS: AIX 5.3 The ER does not work properly due to a "out of virtual shared memory " problem in one site. Is there any EASY DRUG to treat "CDR Grouper FanOut thread is aborting." ? (instead of recreating whole the ER) Thanks, Frank .......................... :31:00 out of virtual shared memory 13:31:00 shmat: [EMFILE][24]: out of shared memory segments, check system SHMSEG 13:31:00 out of virtual shared memory 13:31:00 CDR grouper memory allocation failed (FanOut update link). 13:31:00 shmat: [EMFILE][24]: out of shared memory segments, check system SHMSEG 13:31:00 CDR Grouper FanOut thread is aborting. 13:31:00 out of virtual shared memory .................... --0016364d2ac12500fd0494166de1 ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.
Madison, We tried, but the problem is still there, see below. Thanks, Frank 09:16:10 Maximum server connections 173 09:16:10 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 1453, Llog used 8076 09:16:20 listener-thread: err = -27001: oserr = 0: errstr = : Read error occurred during connection attempt. "/dbbackup/online_log/online_ngdcprd.log" 1408 lines, 74247 characters 16:54:26 CDR NIF shutdown waiting for 0 site(s) to disconnect 16:54:26 CDR NIF Shutdown: connections all shutdown. 16:54:26 CDR The NIF sub-component has shut down. 16:54:29 CDR GC: operation gcStop failed (error 1). 16:54:43 (93) connection rejected - no calls allowed for cdraccept 16:54:43 (1132761) connection rejected - no calls allowed for cdraccept 16:55:13 (1132762) connection rejected - no calls allowed for cdraccept 16:55:13 (1132766) connection rejected - no calls allowed for cdraccept 16:55:45 CDR shutdown complete 16:55:58 (1132767) connection rejected - no calls allowed for cdraccept 16:55:58 (1132772) connection rejected - no calls allowed for cdraccept 16:56:22 Checkpoint Completed: duration was 0 seconds. 16:56:22 Tue Nov 2 - loguniq 146122, logpos 0x6c68018, timestamp: 0xf6985aff Interval: 271948 16:56:22 Maximum server connections 173 16:56:22 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 265, Llog used 79973 16:56:28 CDR GC: catalog recovery begin 16:56:32 CDR GC: catalog recovery complete 16:56:32 CDR queuer initialization complete 16:56:32 CDR Grouper: replay logical log ID (145822) is not present 16:56:32 CDR NIF listening on asf://ngdcprd_grp 16:56:32 CDR: Re-connected to server, id 1, name <osdint1_grp> 16:56:32 CDR: Re-connected to server, id 3, name <ncdcprd_grp> 16:56:32 CDR Grouper Fan Out thread is aborting. 16:56:32 CDR Grouper FanOut thread is aborting. 17:01:22 Checkpoint Completed: duration was 0 seconds. 17:01:22 Tue Nov 2 - loguniq 146122, logpos 0x7365018, timestamp: 0xf69e7fb8 Interval: 271949 On Tue, Nov 2, 2010 at 2:55 PM, Madison Pruet <mpruet@us.ibm.com> wrote: > cdr stop followed by cdr start. > > You don't have to recreate the whole of ER. > > M.P. > > From: "FRANK" <yunyaoqu@gmail.com> > > To: ids@iiug.org > > Date: 11/02/2010 01:53 PM > > Subject: CDR Grouper FanOut thread is aborting [21851] > > Sent by: ids-bounces@iiug.org > > Folks, > > Build Version: 11.10.UC2 > Build OS: AIX 5.3 > > The ER does not work properly due to a "out of virtual shared memory " > problem in one site. > > Is there any EASY DRUG to treat "CDR Grouper FanOut thread is > aborting." ? (instead of recreating whole the ER) > > Thanks, > Frank > > ........................... > :31:00 out of virtual shared memory > > 13:31:00 shmat: [EMFILE][24]: out of shared memory segments, check system > SHMSEG > > 13:31:00 out of virtual shared memory > > 13:31:00 CDR grouper memory allocation failed (FanOut update link). > 13:31:00 shmat: [EMFILE][24]: out of shared memory segments, check system > SHMSEG > > 13:31:00 CDR Grouper FanOut thread is aborting. > 13:31:00 out of virtual shared memory > ..................... > > --0016364d2ac12500fd0494166de1 > > > > ******************************************************************************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --20cf30025aa058ea790494169bcf
Madison, Looks the replay log is not there ..... 16:56:32 CDR Grouper: replay logical log ID (145822) is not present 16:56:32 CDR NIF listening on asf://ngdcprd_grp Thanks, Frank On Tue, Nov 2, 2010 at 3:05 PM, FRANK <yunyaoqu@gmail.com> wrote: > Madison, > > We tried, but the problem is still there, see below. > > Thanks, > Frank > > 09:16:10 Maximum server connections 173 > 09:16:10 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, Plog used 1453, Llog used 8076 > > 09:16:20 listener-thread: err = -27001: oserr = 0: errstr = : Read error > occurred during connection attempt. > "/dbbackup/online_log/online_ngdcprd.log" 1408 lines, 74247 characters > 16:54:26 CDR NIF shutdown waiting for 0 site(s) to disconnect > 16:54:26 CDR NIF Shutdown: connections all shutdown. > 16:54:26 CDR The NIF sub-component has shut down. > 16:54:29 CDR GC: operation gcStop failed (error 1). > 16:54:43 (93) connection rejected - no calls allowed for cdraccept > 16:54:43 (1132761) connection rejected - no calls allowed for cdraccept > 16:55:13 (1132762) connection rejected - no calls allowed for cdraccept > 16:55:13 (1132766) connection rejected - no calls allowed for cdraccept > 16:55:45 CDR shutdown complete > 16:55:58 (1132767) connection rejected - no calls allowed for cdraccept > 16:55:58 (1132772) connection rejected - no calls allowed for cdraccept > 16:56:22 Checkpoint Completed: duration was 0 seconds. > 16:56:22 Tue Nov 2 - loguniq 146122, logpos 0x6c68018, timestamp: > 0xf6985aff Interval: 271948 > > 16:56:22 Maximum server connections 173 > 16:56:22 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, Plog used 265, Llog used 79973 > > 16:56:28 CDR GC: catalog recovery begin > 16:56:32 CDR GC: catalog recovery complete > 16:56:32 CDR queuer initialization complete > 16:56:32 CDR Grouper: replay logical log ID (145822) is not present > 16:56:32 CDR NIF listening on asf://ngdcprd_grp > 16:56:32 CDR: Re-connected to server, id 1, name <osdint1_grp> > 16:56:32 CDR: Re-connected to server, id 3, name <ncdcprd_grp> > 16:56:32 CDR Grouper Fan Out thread is aborting. > 16:56:32 CDR Grouper FanOut thread is aborting. > 17:01:22 Checkpoint Completed: duration was 0 seconds. > 17:01:22 Tue Nov 2 - loguniq 146122, logpos 0x7365018, timestamp: > 0xf69e7fb8 Interval: 271949 > > On Tue, Nov 2, 2010 at 2:55 PM, Madison Pruet <mpruet@us.ibm.com> wrote: > > > cdr stop followed by cdr start. > > > > You don't have to recreate the whole of ER. > > > > M.P. > > > > From: "FRANK" <yunyaoqu@gmail.com> > > > > To: ids@iiug.org > > > > Date: 11/02/2010 01:53 PM > > > > Subject: CDR Grouper FanOut thread is aborting [21851] > > > > Sent by: ids-bounces@iiug.org > > > > Folks, > > > > Build Version: 11.10.UC2 > > Build OS: AIX 5.3 > > > > The ER does not work properly due to a "out of virtual shared memory " > > problem in one site. > > > > Is there any EASY DRUG to treat "CDR Grouper FanOut thread is > > aborting." ? (instead of recreating whole the ER) > > > > Thanks, > > Frank > > > > ........................... > > :31:00 out of virtual shared memory > > > > 13:31:00 shmat: [EMFILE][24]: out of shared memory segments, check system > > SHMSEG > > > > 13:31:00 out of virtual shared memory > > > > 13:31:00 CDR grouper memory allocation failed (FanOut update link). > > 13:31:00 shmat: [EMFILE][24]: out of shared memory segments, check system > > SHMSEG > > > > 13:31:00 CDR Grouper FanOut thread is aborting. > > 13:31:00 out of virtual shared memory > > ..................... > > > > --0016364d2ac12500fd0494166de1 > > > > > > > > > > ******************************************************************************* > > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > > > > ******************************************************************************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > --20cf30025aa058ea790494169bcf > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --00221597575a101bc0049416a91b
Well - looks like it's cdr check --repair time. From: "FRANK" <yunyaoqu@gmail.com> To: ids@iiug.org Date: 11/02/2010 02:09 PM Subject: Re: CDR Grouper FanOut thread is aborting [21855] Sent by: ids-bounces@iiug.org Madison, Looks the replay log is not there ..... 16:56:32 CDR Grouper: replay logical log ID (145822) is not present 16:56:32 CDR NIF listening on asf://ngdcprd_grp Thanks, Frank On Tue, Nov 2, 2010 at 3:05 PM, FRANK <yunyaoqu@gmail.com> wrote: > Madison, > > We tried, but the problem is still there, see below. > > Thanks, > Frank > > 09:16:10 Maximum server connections 173 > 09:16:10 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, Plog used 1453, Llog used 8076 > > 09:16:20 listener-thread: err = -27001: oserr = 0: errstr = : Read error > occurred during connection attempt. > "/dbbackup/online_log/online_ngdcprd.log" 1408 lines, 74247 characters > 16:54:26 CDR NIF shutdown waiting for 0 site(s) to disconnect > 16:54:26 CDR NIF Shutdown: connections all shutdown. > 16:54:26 CDR The NIF sub-component has shut down. > 16:54:29 CDR GC: operation gcStop failed (error 1). > 16:54:43 (93) connection rejected - no calls allowed for cdraccept > 16:54:43 (1132761) connection rejected - no calls allowed for cdraccept > 16:55:13 (1132762) connection rejected - no calls allowed for cdraccept > 16:55:13 (1132766) connection rejected - no calls allowed for cdraccept > 16:55:45 CDR shutdown complete > 16:55:58 (1132767) connection rejected - no calls allowed for cdraccept > 16:55:58 (1132772) connection rejected - no calls allowed for cdraccept > 16:56:22 Checkpoint Completed: duration was 0 seconds. > 16:56:22 Tue Nov 2 - loguniq 146122, logpos 0x6c68018, timestamp: > 0xf6985aff Interval: 271948 > > 16:56:22 Maximum server connections 173 > 16:56:22 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, Plog used 265, Llog used 79973 > > 16:56:28 CDR GC: catalog recovery begin > 16:56:32 CDR GC: catalog recovery complete > 16:56:32 CDR queuer initialization complete > 16:56:32 CDR Grouper: replay logical log ID (145822) is not present > 16:56:32 CDR NIF listening on asf://ngdcprd_grp > 16:56:32 CDR: Re-connected to server, id 1, name <osdint1_grp> > 16:56:32 CDR: Re-connected to server, id 3, name <ncdcprd_grp> > 16:56:32 CDR Grouper Fan Out thread is aborting. > 16:56:32 CDR Grouper FanOut thread is aborting. > 17:01:22 Checkpoint Completed: duration was 0 seconds. > 17:01:22 Tue Nov 2 - loguniq 146122, logpos 0x7365018, timestamp: > 0xf69e7fb8 Interval: 271949 > > On Tue, Nov 2, 2010 at 2:55 PM, Madison Pruet <mpruet@us.ibm.com> wrote: > > > cdr stop followed by cdr start. > > > > You don't have to recreate the whole of ER. > > > > M.P. > > > > From: "FRANK" <yunyaoqu@gmail.com> > > > > To: ids@iiug.org > > > > Date: 11/02/2010 01:53 PM > > > > Subject: CDR Grouper FanOut thread is aborting [21851] > > > > Sent by: ids-bounces@iiug.org > > > > Folks, > > > > Build Version: 11.10.UC2 > > Build OS: AIX 5.3 > > > > The ER does not work properly due to a "out of virtual shared memory " > > problem in one site. > > > > Is there any EASY DRUG to treat "CDR Grouper FanOut thread is > > aborting." ? (instead of recreating whole the ER) > > > > Thanks, > > Frank > > > > ........................... > > :31:00 out of virtual shared memory > > > > 13:31:00 shmat: [EMFILE][24]: out of shared memory segments, check system > > SHMSEG > > > > 13:31:00 out of virtual shared memory > > > > 13:31:00 CDR grouper memory allocation failed (FanOut update link). > > 13:31:00 shmat: [EMFILE][24]: out of shared memory segments, check system > > SHMSEG > > > > 13:31:00 CDR Grouper FanOut thread is aborting. > > 13:31:00 out of virtual shared memory > > ..................... > > > > --0016364d2ac12500fd0494166de1 > > > > > > > > > > ******************************************************************************* > > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > > > > ******************************************************************************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > --20cf30025aa058ea790494169bcf > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --00221597575a101bc0049416a91b ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.
Related threads
- System Or Internal Error InterruptedIOException
- Informix ODBC Error: Read error occured during con
- error 27001