RE: Enterprise Replication help - urgent

Thread 205 is gone (disappeared with the engine bounce) As it was heading into the night here in New Zealand, I threw a whole heap of space at the smart blob space and said "sort it in the morning". My first task when I get in to work (about an hour away) is to raise a case with IBM. This is a recent onstat -g rqm SENDQ IBM Informix Dynamic Server Version 11.50.UC5X2 -- On-Line -- Up 04:26:27 --851072 Kbytes CDR Reliable Queue Manager (RQM) Statistics: RQM Statistics for Queue (0x6df68018) trg_send Transaction Spool Name: trg_send_stxn Insert Stamp: 21156567/0 Flags: SEND_Q, SPOOLED, PROGRESS_TABLE, NEED_ACK Txns in queue: 679863 Log Events in queue: 48 Txns in memory: 50019 Txns in spool only: 629844 Txns spooled: 643554 Unspooled bytes: 9190654 Size of Data in queue: 138942204 Bytes Real memory in use: 9190654 Bytes Pending Txn Buffers: 0 Pending Txn Data: 0 Bytes Max Real memory data used: 10187870 (15728640) Bytes Max Real memory hdrs used 17016064 (15728640) Bytes Total data queued: 24095980 Bytes Total Txns queued: 679956 Total Txns spooled: 59152 Total Txns restored: 2250558 Total Txns recovered: 0 Spool Rows read: 2250558 Total Txns deleted: 93 Total Txns duplicated: 0 Total Txn Lookups: 2250568 Progress Table: Progress Table is Stable On-disk table name............: spttrg_send Flush interval (time).........: 30 Time of last flush............: 1282550241 Flush interval (serial number): 1000 Serial number of last flush...: 15 Current serial number.........: 15 Server Group Bytes Queued Acked Sent ------------------------------------------------------------------------------ 12 0xc04de 11388184 0xab/0x213b9/0x148f2d8/(nil) -0xab/0x21457/0x105d4c8/(nil) 12 0xc04b2 44144478 0xefffffff/0xefffffff/0xefffffff/0xefffffff -0xab/0x21457/0x104730c/(nil) 12 0xc04b1 112 0xab/0x213af/0x24cc344/(nil) -0xab/0x2141b/0x125d07c/(nil) 12 0xc04b0 214 0xab/0x213af/0x24cc344/(nil) -0xab/0x2141b/0x125d07c/(nil) 12 0xc03cf 100256 0xab/0x213b4/0x8a3d0/(nil) -0xab/0x21455/0x10f73d0/(nil) 12 0xc0137 0 0xab/0x207c8/0x11d01e0/(nil) -0xefffffff/0xefffffff/0xefffffff/0xefffffff 12 0xc0136 0 0xab/0xeaa0/0x22f1214/(nil) -0xefffffff/0xefffffff/0xefffffff/0xefffffff 12 0xc007b 0 0xab/0x213a8/0x20321f4/(nil) -0xefffffff/0xefffffff/0xefffffff/0xefffffff 12 0xc0040 0 0xab/0x212e3/0xc141c4/(nil) -0xefffffff/0xefffffff/0xefffffff/0xefffffff 12 0xc003f 56395008 0xab/0x213b9/0x14b6268/(nil) -0xab/0x21457/0x1059268/(nil) First Txn (0x6b2f9d18) Key: 171/132840/0x0193130c/0x00000000 Txn Stamp: 13476693/0, Reference Count: 0. Txn Flags: Spooled, Restored Txn Commit Time: (1281047843) 2010/08/06 10:37:23 Txn Size in Queue: 158 First Buf's (0x6ae1a438) Queue Flags: None First Buf's Buffer Flags: None NeedAck: Waiting for Acks from <[0004]> No open handles on txn. Last Txn (0x6bc4c418) Key: 171/136279/0x0105d4c8/0x00000000 Txn Stamp: 21156567/0, Reference Count: 0. Txn Flags: Notify Txn Commit Time: (1282551317) 2010/08/23 20:15:17 Txn Size in Queue: 90 First Buf's (0x6a6edd98) Queue Flags: Resident First Buf's Buffer Flags: TRG, Stream NeedAck: Waiting for Acks from <[0004]> No open handles on txn. Traverse handle (0x6dbe5238) for thread CDRNsT12 at txn (0x6a26aeb0) End_of_Q, Flags: None Traverse handle (0x6de02018) for thread CDRNr12 at Head_of_Q, Flags: None Traverse handle (0x6a4ca018) for thread CDRGeval0 at Head_of_Q, Flags: None Traverse handle (0x6df71018) for thread CDRGeval1 at Head_of_Q, Flags: None Traverse handle (0x668c9018) for thread CDRGeval2 at Head_of_Q, Flags: None Traverse handle (0x6de83018) for thread CDRGeval3 at Head_of_Q, Flags: None Traverse handle (0x6bfad470) for thread CDRACK_0 at Head_of_Q, Flags: None Traverse handle (0x6bd78788) for thread CDRACK_1 at Head_of_Q, Flags: None ________________________________________ From: ids-bounces@iiug.org [ids-bounces@iiug.org] On Behalf Of Madison Pruet [mpruet@us.ibm.com] Sent: Tuesday, 24 August 2010 2:41 a.m. To: ids@iiug.org Subject: Re: Enterprise Replication help - urgent [20995] On the surface this looks like you have some partially spooled transactions. First of all , what is thread 205? Can you get an onstat -g stk 205 and/or onstat -g ath | grep 205. From: "Jarrod Teale" <Jarrod.Teale@fonterra.com> To: ids@iiug.org Date: 08/23/2010 12:11 AM Subject: Enterprise Replication help - urgent [20990] Sent by: ids-bounces@iiug.org Hi, I have two machines, RHEL4, IDS 11.50.UC5X2 is a ER Leaf node, RHEL4 IDS 11.50.UC5 is the ER Enterprise node. The leaf node is in collect mode and won't transfer any data back to the ER node (we have replicates running both ways and neither are processing at the moment). I need help to stop it going into DDR Block The replication has been running fine till this morning when the onstat logs on the leaf node wrote this out 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) Since then the leaf node has been caching all the transactions, not sending them back to the Enterprise node. I have tried doing "cdr stop", "cdr disconnect server" and even a full reboot from both ends, but nothing seems to stop the leaf node queuing . The send queue from from leaf node looks like this (ugly) [sdev@sirrp71(10.71.64.130) ~]$ cdr stats rqm -T RQM Statistics for Queue number: 0 name: trg_send Transaction Spool Name: trg_send_stxn Flags: SEND_Q, SPOOLED, PROGRESS_TABLE, NEED_ACK, SENDQ_MASK, SREP_TABLE Txns in queue: 611107 Txns in memory: 51089 Txns in spool only: 560018 Txns spooled: 584408 Unspooled bytes: 6706326 Size of Data in queue: 121567408 Bytes Real memory in use: 6706326 Bytes Pending Txn Buffers: 0 Pending Txn Data: 0 Bytes Max Real memory data used: 6706326 Bytes Max Real memory hdrs used: 15897280 Bytes Total data queued: 6706326 Bytes Total Txns queued: 611107 Total Txns spooled: 0 Total Txns restored: 94961 Total Txns recovered: 0 Spool Rows read: 94962 Total Txns deleted: 0 Total Txns duplicated: 0 Total Txn Lookups: 94965 Any ideas? Jarrod Teale Team Lead - Manufacturing Execution Systems Automation