CDR DS CDRD_xx thread is aborting

HI, All, IDS11.70 FC8 A little bit complicated system generated the massive errors ( below). System architecture : a GRID is composed of 3 instances A, B and C. A is the grid master and also a HDR primary server of a Secondary read-only server D . A and D are configured with DRAUTO 2. Operation performed: A was shutdown for hardware maintenance . D was coming up as an new primary ( see its online log below). B and C were keeping running. Problem: It seems the new primary (D) was trying to take massive CDR jobs from B and C ( true?), but it had large volume of CDR errors ( see the log) What do these errors mean ? Any type of consequences ? Thanks Frank ==================online log of the new primary) 23:00:54 Maximum server connections 21 23:00:54 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 134, Llog used 0 23:01:52 DR: Receive error 23:01:52 SMX thread is exiting 23:01:52 ASF Echo-Thread Server: asfcode = -25582: oserr = 0: errstr = : Network connection is broken. 23:01:52 DR_ERR set to -1 23:01:52 DR: Turned off on secondary server 23:01:52 Skipping failover callback. 23:01:53 Logical Recovery has reached the transaction cleanup phase. 23:01:53 Checkpoint Completed: duration was 0 seconds. 23:01:53 Thu Feb 23 - loguniq 411568, logpos 0x1a018, timestamp: 0x471a9aec Interval: 83237 23:01:53 Maximum server connections 21 23:01:53 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 5, Llog used 1 23:01:53 Logical Recovery Complete. 1726325 Committed, 590 Rolled Back, 0 Open, 0 Bad Locks 23:01:53 Logical Recovery Complete. 23:01:53 Quiescent Mode 23:01:53 Checkpoint Completed: duration was 0 seconds. 23:01:53 Thu Feb 23 - loguniq 411568, logpos 0x1c018, timestamp: 0x471a9b96 Interval: 83238 23:01:53 Maximum server connections 21 23:01:53 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 11, Llog used 2 23:01:54 B-tree scanners enabled. 23:01:54 SCHAPI: Started dbScheduler thread. 23:01:54 DR: new type = standard 23:01:54 DR: new type = primary, secondary server name = usppsdbgrid5prod 23:01:54 DR: Cannot connect to secondary server 23:01:54 DR: Turned off on primary server 23:01:54 Starting BldNotification 23:01:54 Starting BldNotification 23:01:54 Auto Registration is synced 23:01:54 SCHAPI: Started 2 dbWorker threads. 23:01:54 On-Line Mode 23:01:55 Checkpoint Completed: duration was 0 seconds. 23:01:55 Thu Feb 23 - loguniq 411568, logpos 0x38018, timestamp: 0x471a9eca Interval: 83239 23:01:55 Maximum server connections 21 23:01:55 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 27, Llog used 28 23:01:58 Defragmenter cleaner thread now running 23:01:58 Defragmenter cleaner thread cleaned:0 partitions 23:01:58 Checkpoint Completed: duration was 0 seconds. 23:01:58 Thu Feb 23 - loguniq 411568, logpos 0x3a018, timestamp: 0x471a9f3d Interval: 83240 23:01:58 Maximum server connections 21 23:01:58 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 6, Llog used 2 23:02:00 Logical Log 411568 Complete, timestamp: 0x471a9f49. 23:02:00 Logical Log 411568 - Backup Started 23:02:00 Logical Log 411568 - Backup Completed 23:02:06 (1372) connection rejected - no calls allowed for cdraccept 23:02:07 (1373) connection rejected - no calls allowed for cdraccept 23:02:36 (1374) connection rejected - no calls allowed for cdraccept 23:02:37 (1375) connection rejected - no calls allowed for cdraccept 23:02:44 CDR GC: catalog recovery begin 23:02:50 CDR GC: catalog recovery complete 23:02:50 CDR queuer initialization complete 23:02:50 CDR NIF listening on asf://group1 23:02:50 DDR Log Snooping - Snooping started in log 411568 23:02:50 CDR: Re-connected to server, id 2932, name <group3> 23:02:50 CDR: Re-connected to server, id 2934, name <group4> 23:02:50 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:50 CDR DS CDRD_4 thread is aborting. 23:02:50 CDR DS CDRD_4 thread is aborting. 23:02:51 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:51 CDR DS CDRD_1 thread is aborting. 23:02:51 CDR DS CDRD_1 thread is aborting. 23:02:51 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:51 CDR DS CDRD_6 thread is aborting. 23:02:51 CDR DS CDRD_6 thread is aborting. 23:02:51 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:51 CDR DS CDRD_5 thread is aborting. 23:02:51 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:51 CDR DS CDRD_5 thread is aborting. 23:02:51 CDR DS CDRD_9 thread is aborting. 23:02:51 CDR DS CDRD_9 thread is aborting. 23:02:51 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:51 CDR DS CDRD_6 thread is aborting. 23:02:51 CDR DS CDRD_6 thread is aborting. 23:02:51 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:51 CDR DS CDRD_5 thread is aborting. 23:02:51 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:51 CDR DS CDRD_5 thread is aborting. 23:02:51 CDR DS CDRD_9 thread is aborting. 23:02:51 CDR DS CDRD_9 thread is aborting. 23:02:52 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:52 CDR DS CDRD_10 thread is aborting. 23:02:52 CDR DS CDRD_10 thread is aborting. 23:02:52 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:52 CDR DS CDRD_3 thread is aborting. 23:02:52 CDR DS CDRD_3 thread is aborting. 23:02:52 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:52 CDR DS CDRD_15 thread is aborting. 23:02:52 CDR DS CDRD_15 thread is aborting. 23:02:52 CDR CDRDS: transaction aborted (Max attempt to apply transaction reached and failed.) with sql error 0 isam error 0. 23:02:52 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:52 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:52 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:52 CDR DS CDRD_16 thread is aborting. 23:02:52 CDR DS CDRD_16 thread is aborting. 23:02:52 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:52 CDR DS CDRD_12 thread is aborting. 23:02:52 CDR DS CDRD_12 thread is aborting. 23:02:52 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:52 CDR DS CDRD_2 thread is aborting. 23:02:52 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:52 CDR DS CDRD_7 thread is aborting. 23:02:52 CDR DS CDRD_2 thread is aborting. 23:02:52 CDR DS CDRD_7 thread is aborting. 23:02:52 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103

Your stable storage SBLOB for the receive queue was probably unlogged so when the fail-over occurred the transactions were not complete. While it might look scary, it is probably not a real issue. Items are placed in the receive queue only when the receive memory buffers become too full. This is different from the stable send queue in that when replicated transactions are spooled to disk, the replay log position is advanced. With the receive queue when an item is spooled, it is not ACKed to the sender. That means that in your case the replicated transaction still exists on the source nodes. I think that all you would need to do is to bounce ER by running 'cdr stop' followed by 'cdr start'. This should cause ER to purge anything in the disk receive queue and those transactions would be resent from the source nodes. I'm not quite sure what the edge conditions is that you ran into. Like I said, I suspect that it might be because the sblob for the receive queue is not logged, but I'd have to know much more to be certain. Madison Pruet Retired and Loving it On Friday, February 24, 2017 7:05 PM, FRANK <yunyaoqu@gmail.com> wrote: HI, All, IDS11.70 FC8 A little bit complicated system generated the massive errors ( below). System architecture : a GRID is composed of 3 instances A, B and C. A is the grid master and also a HDR primary server of a Secondary read-only server D . A and D are configured with DRAUTO 2. Operation performed: A was shutdown for hardware maintenance . D was coming up as an new primary ( see its online log below). B and C were keeping running. Problem: It seems the new primary (D) was trying to take massive CDR jobs from B and C ( true?), but it had large volume of CDR errors ( see the log) What do these errors mean ? Any type of consequences ? Thanks Frank ==================online log of the new primary) 23:00:54 Maximum server connections 21 23:00:54 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 134, Llog used 0 23:01:52 DR: Receive error 23:01:52 SMX thread is exiting 23:01:52 ASF Echo-Thread Server: asfcode = -25582: oserr = 0: errstr = : Network connection is broken. 23:01:52 DR_ERR set to -1 23:01:52 DR: Turned off on secondary server 23:01:52 Skipping failover callback. 23:01:53 Logical Recovery has reached the transaction cleanup phase. 23:01:53 Checkpoint Completed: duration was 0 seconds. 23:01:53 Thu Feb 23 - loguniq 411568, logpos 0x1a018, timestamp: 0x471a9aec Interval: 83237 23:01:53 Maximum server connections 21 23:01:53 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 5, Llog used 1 23:01:53 Logical Recovery Complete. 1726325 Committed, 590 Rolled Back, 0 Open, 0 Bad Locks 23:01:53 Logical Recovery Complete. 23:01:53 Quiescent Mode 23:01:53 Checkpoint Completed: duration was 0 seconds. 23:01:53 Thu Feb 23 - loguniq 411568, logpos 0x1c018, timestamp: 0x471a9b96 Interval: 83238 23:01:53 Maximum server connections 21 23:01:53 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 11, Llog used 2 23:01:54 B-tree scanners enabled. 23:01:54 SCHAPI: Started dbScheduler thread. 23:01:54 DR: new type = standard 23:01:54 DR: new type = primary, secondary server name = usppsdbgrid5prod 23:01:54 DR: Cannot connect to secondary server 23:01:54 DR: Turned off on primary server 23:01:54 Starting BldNotification 23:01:54 Starting BldNotification 23:01:54 Auto Registration is synced 23:01:54 SCHAPI: Started 2 dbWorker threads. 23:01:54 On-Line Mode 23:01:55 Checkpoint Completed: duration was 0 seconds. 23:01:55 Thu Feb 23 - loguniq 411568, logpos 0x38018, timestamp: 0x471a9eca Interval: 83239 23:01:55 Maximum server connections 21 23:01:55 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 27, Llog used 28 23:01:58 Defragmenter cleaner thread now running 23:01:58 Defragmenter cleaner thread cleaned:0 partitions 23:01:58 Checkpoint Completed: duration was 0 seconds. 23:01:58 Thu Feb 23 - loguniq 411568, logpos 0x3a018, timestamp: 0x471a9f3d Interval: 83240 23:01:58 Maximum server connections 21 23:01:58 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 6, Llog used 2 23:02:00 Logical Log 411568 Complete, timestamp: 0x471a9f49. 23:02:00 Logical Log 411568 - Backup Started 23:02:00 Logical Log 411568 - Backup Completed 23:02:06 (1372) connection rejected - no calls allowed for cdraccept 23:02:07 (1373) connection rejected - no calls allowed for cdraccept 23:02:36 (1374) connection rejected - no calls allowed for cdraccept 23:02:37 (1375) connection rejected - no calls allowed for cdraccept 23:02:44 CDR GC: catalog recovery begin 23:02:50 CDR GC: catalog recovery complete 23:02:50 CDR queuer initialization complete 23:02:50 CDR NIF listening on asf://group1 23:02:50 DDR Log Snooping - Snooping started in log 411568 23:02:50 CDR: Re-connected to server, id 2932, name <group3> 23:02:50 CDR: Re-connected to server, id 2934, name <group4> 23:02:50 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:50 CDR DS CDRD_4 thread is aborting. 23:02:50 CDR DS CDRD_4 thread is aborting. 23:02:51 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:51 CDR DS CDRD_1 thread is aborting. 23:02:51 CDR DS CDRD_1 thread is aborting. 23:02:51 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:51 CDR DS CDRD_6 thread is aborting. 23:02:51 CDR DS CDRD_6 thread is aborting. 23:02:51 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:51 CDR DS CDRD_5 thread is aborting. 23:02:51 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:51 CDR DS CDRD_5 thread is aborting. 23:02:51 CDR DS CDRD_9 thread is aborting. 23:02:51 CDR DS CDRD_9 thread is aborting. 23:02:51 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:51 CDR DS CDRD_6 thread is aborting. 23:02:51 CDR DS CDRD_6 thread is aborting. 23:02:51 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:51 CDR DS CDRD_5 thread is aborting. 23:02:51 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:51 CDR DS CDRD_5 thread is aborting. 23:02:51 CDR DS CDRD_9 thread is aborting. 23:02:51 CDR DS CDRD_9 thread is aborting. 23:02:52 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:52 CDR DS CDRD_10 thread is aborting. 23:02:52 CDR DS CDRD_10 thread is aborting. 23:02:52 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:52 CDR DS CDRD_3 thread is aborting. 23:02:52 CDR DS CDRD_3 thread is aborting. 23:02:52 CDR stablept_flushDescriptorsToDisk: Could not read row from table 'spttrg_receive': 103 23:02:52 CDR DS CDRD_15 thread is aborting. 23:02:52 CDR DS CDRD_15 thread is aborting. 23:02:52 CDR

Hmmm, message indicating a problem with syscdr:spttrg=5Freceive table=20 (receive queue stable progress table), '103' meaning some sort of an index = problem. -> oncheck -cDI syscdr:spttrg=5Freceive ? This table serves at telling ER apply (CDRD=5F* threads) which incoming=20 transactions already have been applied, so the same txn wouldn't be=20 attempted to apply a second time after a reconnect. Depending on ER's state at your maintenance shutdown, i.e. whether B or C = had their send queues cleared and no outstanding transactions to=20 replicate, a possible solution might be to drop and recreate this table -=20 not sure, though... Andreas From: "FRANK" <yunyaoqu@gmail.com> To: ids@iiug.org Date: 25.02.2017 02:05 Subject: CDR DS CDRD=5Fxx thread is aborting [38682] Sent by: ids-bounces@iiug.org HI, All,=20 IDS11.70 FC8=20 A little bit complicated system generated the massive errors ( below).=20 System architecture : a GRID is composed of 3 instances A, B and C. A=20 is the grid master and also a HDR primary server of a Secondary=20 read-only server D . A and D are configured with DRAUTO 2.=20 Operation performed: A was shutdown for hardware maintenance . D was=20 coming up as an new primary ( see its online log below). B and C were=20 keeping running.=20 Problem: It seems the new primary (D) was trying to take massive CDR jobs=20 from B and C ( true?), but it had large volume of CDR errors ( see the=20 log)=20 What do these errors mean ? Any type of consequences ?=20 Thanks=20 Frank=20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3Donline log of the new= primary)=20 23:00:54 Maximum server connections 21=20 23:00:54 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked = 0, Plog used 134, Llog used 0=20 23:01:52 DR: Receive error=20 23:01:52 SMX thread is exiting=20 23:01:52 ASF Echo-Thread Server: asfcode =3D -25582: oserr =3D 0: errstr = =3D :=20 Network connection is broken.=20 23:01:52 DR=5FERR set to -1=20 23:01:52 DR: Turned off on secondary server=20 23:01:52 Skipping failover callback.=20 23:01:53 Logical Recovery has reached the transaction cleanup phase.=20 23:01:53 Checkpoint Completed: duration was 0 seconds.=20 23:01:53 Thu Feb 23 - loguniq 411568, logpos 0x1a018, timestamp:=20 0x471a9aec Interval: 83237=20 23:01:53 Maximum server connections 21=20 23:01:53 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked = 0, Plog used 5, Llog used 1=20 23:01:53 Logical Recovery Complete.=20 1726325 Committed, 590 Rolled Back, 0 Open, 0 Bad Locks=20 23:01:53 Logical Recovery Complete.=20 23:01:53 Quiescent Mode=20 23:01:53 Checkpoint Completed: duration was 0 seconds.=20 23:01:53 Thu Feb 23 - loguniq 411568, logpos 0x1c018, timestamp:=20 0x471a9b96 Interval: 83238=20 23:01:53 Maximum server connections 21=20 23:01:53 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked = 0, Plog used 11, Llog used 2=20 23:01:54 B-tree scanners enabled.=20 23:01:54 SCHAPI: Started dbScheduler thread.=20 23:01:54 DR: new type =3D standard=20 23:01:54 DR: new type =3D primary, secondary server name =3D usppsdbgrid5pr= od=20 23:01:54 DR: Cannot connect to secondary server=20 23:01:54 DR: Turned off on primary server=20 23:01:54 Starting BldNotification=20 23:01:54 Starting BldNotification=20 23:01:54 Auto Registration is synced=20 23:01:54 SCHAPI: Started 2 dbWorker threads.=20 23:01:54 On-Line Mode=20 23:01:55 Checkpoint Completed: duration was 0 seconds.=20 23:01:55 Thu Feb 23 - loguniq 411568, logpos 0x38018, timestamp:=20 0x471a9eca Interval: 83239=20 23:01:55 Maximum server connections 21=20 23:01:55 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked = 0, Plog used 27, Llog used 28=20 23:01:58 Defragmenter cleaner thread now running=20 23:01:58 Defragmenter cleaner thread cleaned:0 partitions=20 23:01:58 Checkpoint Completed: duration was 0 seconds.=20 23:01:58 Thu Feb 23 - loguniq 411568, logpos 0x3a018, timestamp:=20 0x471a9f3d Interval: 83240=20 23:01:58 Maximum server connections 21=20 23:01:58 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked = 0, Plog used 6, Llog used 2=20 23:02:00 Logical Log 411568 Complete, timestamp: 0x471a9f49.=20 23:02:00 Logical Log 411568 - Backup Started=20 23:02:00 Logical Log 411568 - Backup Completed=20 23:02:06 (1372) connection rejected - no calls allowed for cdraccept=20 23:02:07 (1373) connection rejected - no calls allowed for cdraccept=20 23:02:36 (1374) connection rejected - no calls allowed for cdraccept=20 23:02:37 (1375) connection rejected - no calls allowed for cdraccept=20 23:02:44 CDR GC: catalog recovery begin=20 23:02:50 CDR GC: catalog recovery complete=20 23:02:50 CDR queuer initialization complete=20 23:02:50 CDR NIF listening on asf://group1=20 23:02:50 DDR Log Snooping - Snooping started in log 411568=20 23:02:50 CDR: Re-connected to server, id 2932, name <group3>=20 23:02:50 CDR: Re-connected to server, id 2934, name <group4>=20 23:02:50 CDR stablept=5FflushDescriptorsToDisk: Could not read row from=20 table 'spttrg=5Freceive': 103=20 23:02:50 CDR DS CDRD=5F4 thread is aborting.=20 23:02:50 CDR DS CDRD=5F4 thread is aborting.=20 23:02:51 CDR stablept=5FflushDescriptorsToDisk: Could not read row from=20 table 'spttrg=5Freceive': 103=20 23:02:51 CDR DS CDRD=5F1 thread is aborting.=20 23:02:51 CDR DS CDRD=5F1 thread is aborting.=20 23:02:51 CDR stablept=5FflushDescriptorsToDisk: Could not read row from=20 table 'spttrg=5Freceive': 103=20 23:02:51 CDR DS CDRD=5F6 thread is aborting.=20 23:02:51 CDR DS CDRD=5F6 thread is aborting.=20 23:02:51 CDR stablept=5FflushDescriptorsToDisk: Could not read row from=20 table 'spttrg=5Freceive': 103=20 23:02:51 CDR DS CDRD=5F5 thread is aborting.=20 23:02:51 CDR stablept=5FflushDescriptorsToDisk: Could not read row from=20 table 'spttrg=5Freceive': 103=20 23:02:51 CDR DS CDRD=5F5 thread is aborting.=20 23:02:51 CDR DS CDRD=5F9 thread is aborting.=20 23:02:51 CDR DS CDRD=5F9 thread is aborting.=20 23:02:51 CDR stablept=5FflushDescriptorsToDisk: Could not read row from=20 table 'spttrg=5Freceive': 103=20 23:02:51 CDR DS CDRD=5F6 thread is aborting.=20 23:02:51 CDR DS CDRD=5F6 thread is aborting.=20 23:02:51 CDR stablept=5FflushDescriptorsToDisk: Could not read row from=20 table 'spttrg=5Freceive': 103=20 23:02:51 CDR DS CDRD=5F5 thread is aborting.=20 23:02:51 CDR stablept=5FflushDescriptorsToDisk: Could not read row from=20 table 'spttrg=5Freceive': 103=20 23:02:51 CDR DS CDRD=5F5 thread is aborting.=20 23:02:51 CDR DS CDRD=5F9 thread is aborting.=20 23:02:51 CDR DS CDRD=5F9 thread is aborting.=20 23:02:52 CDR stablept=5FflushDescriptorsToDisk: Could not read row from=20 table 'spttrg=5Freceive': 103=20 23:02:52 CDR DS CDRD=5F10 thread is aborting.=20 23:02:52 CDR DS CDRD=5F10 thread is aborting.=20 23:02:52 CDR stablept=5FflushDescriptorsToDisk: Could not read row from=20 table 'spttrg=5Freceive': 103=20 23:02:52 CDR DS CDRD=5F3 thread is aborting.=20 23:02:52 CDR DS CDRD=5F3 thread is aborting.=20 23