Enterprise Replication help - urgent

Hi, I have two machines, RHEL4, IDS 11.50.UC5X2 is a ER Leaf node, RHEL4 IDS 11.50.UC5 is the ER Enterprise node. The leaf node is in collect mode and won't transfer any data back to the ER node (we have replicates running both ways and neither are processing at the moment). I need help to stop it going into DDR Block The replication has been running fine till this morning when the onstat logs on the leaf node wrote this out 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) Since then the leaf node has been caching all the transactions, not sending them back to the Enterprise node. I have tried doing "cdr stop", "cdr disconnect server" and even a full reboot from both ends, but nothing seems to stop the leaf node queuing . The send queue from from leaf node looks like this (ugly) [sdev@sirrp71(10.71.64.130) ~]$ cdr stats rqm -T RQM Statistics for Queue number: 0 name: trg_send Transaction Spool Name: trg_send_stxn Flags: SEND_Q, SPOOLED, PROGRESS_TABLE, NEED_ACK, SENDQ_MASK, SREP_TABLE Txns in queue: 611107 Txns in memory: 51089 Txns in spool only: 560018 Txns spooled: 584408 Unspooled bytes: 6706326 Size of Data in queue: 121567408 Bytes Real memory in use: 6706326 Bytes Pending Txn Buffers: 0 Pending Txn Data: 0 Bytes Max Real memory data used: 6706326 Bytes Max Real memory hdrs used: 15897280 Bytes Total data queued: 6706326 Bytes Total Txns queued: 611107 Total Txns spooled: 0 Total Txns restored: 94961 Total Txns recovered: 0 Spool Rows read: 94962 Total Txns deleted: 0 Total Txns duplicated: 0 Total Txn Lookups: 94965 Any ideas? Jarrod Teale Team Lead - Manufacturing Execution Systems Automation and Process Control jarrod.teale@fonterra.com direct +64 7 850 7525 (ext 77525), mobile +64 21 968 364, fax +64 7 849 7855 Fonterra Co-operative Group Limited PO Box 459, Hamilton, 3240, Automation and Process Control, Fonterra Te Rapa, SH1, Hamilton, New Zealand ________________________________ DISCLAIMER This email contains information that is confidential and which may be legally privileged. If you have received this email in error, please notify the sender immediately and delete the email. This email is intended solely for the use of the intended recipient and you may not use or disclose this email in any way.

On the surface this looks like you have some partially spooled transactions. First of all , what is thread 205? Can you get an onstat -g stk 205 and/or onstat -g ath | grep 205. From: "Jarrod Teale" <Jarrod.Teale@fonterra.com> To: ids@iiug.org Date: 08/23/2010 12:11 AM Subject: Enterprise Replication help - urgent [20990] Sent by: ids-bounces@iiug.org Hi, I have two machines, RHEL4, IDS 11.50.UC5X2 is a ER Leaf node, RHEL4 IDS 11.50.UC5 is the ER Enterprise node. The leaf node is in collect mode and won't transfer any data back to the ER node (we have replicates running both ways and neither are processing at the moment). I need help to stop it going into DDR Block The replication has been running fine till this morning when the onstat logs on the leaf node wrote this out 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) Since then the leaf node has been caching all the transactions, not sending them back to the Enterprise node. I have tried doing "cdr stop", "cdr disconnect server" and even a full reboot from both ends, but nothing seems to stop the leaf node queuing . The send queue from from leaf node looks like this (ugly) [sdev@sirrp71(10.71.64.130) ~]$ cdr stats rqm -T RQM Statistics for Queue number: 0 name: trg_send Transaction Spool Name: trg_send_stxn Flags: SEND_Q, SPOOLED, PROGRESS_TABLE, NEED_ACK, SENDQ_MASK, SREP_TABLE Txns in queue: 611107 Txns in memory: 51089 Txns in spool only: 560018 Txns spooled: 584408 Unspooled bytes: 6706326 Size of Data in queue: 121567408 Bytes Real memory in use: 6706326 Bytes Pending Txn Buffers: 0 Pending Txn Data: 0 Bytes Max Real memory data used: 6706326 Bytes Max Real memory hdrs used: 15897280 Bytes Total data queued: 6706326 Bytes Total Txns queued: 611107 Total Txns spooled: 0 Total Txns restored: 94961 Total Txns recovered: 0 Spool Rows read: 94962 Total Txns deleted: 0 Total Txns duplicated: 0 Total Txn Lookups: 94965 Any ideas? Jarrod Teale Team Lead - Manufacturing Execution Systems Automation and Process Control jarrod.teale@fonterra.com direct +64 7 850 7525 (ext 77525), mobile +64 21 968 364, fax +64 7 849 7855 Fonterra Co-operative Group Limited PO Box 459, Hamilton, 3240, Automation and Process Control, Fonterra Te Rapa, SH1, Hamilton, New Zealand ________________________________ DISCLAIMER This email contains information that is confidential and which may be legally privileged. If you have received this email in error, please notify the sender immediately and delete the email. This email is intended solely for the use of the intended recipient and you may not use or disclose this email in any way. ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

anyone from tech support working with you? From: "Jarrod Teale" <Jarrod.Teale@fonterra.com> To: ids@iiug.org Date: 08/23/2010 12:11 AM Subject: Enterprise Replication help - urgent [20990] Sent by: ids-bounces@iiug.org Hi, I have two machines, RHEL4, IDS 11.50.UC5X2 is a ER Leaf node, RHEL4 IDS 11.50.UC5 is the ER Enterprise node. The leaf node is in collect mode and won't transfer any data back to the ER node (we have replicates running both ways and neither are processing at the moment). I need help to stop it going into DDR Block The replication has been running fine till this morning when the onstat logs on the leaf node wrote this out 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) status=-6 SQLERR=111 thread 205 queue unknown 03:04:48 CDR RQM: rqmRestoreTxn() failed for KEY (0/0/0x(nil)/0x(nil)) Since then the leaf node has been caching all the transactions, not sending them back to the Enterprise node. I have tried doing "cdr stop", "cdr disconnect server" and even a full reboot from both ends, but nothing seems to stop the leaf node queuing . The send queue from from leaf node looks like this (ugly) [sdev@sirrp71(10.71.64.130) ~]$ cdr stats rqm -T RQM Statistics for Queue number: 0 name: trg_send Transaction Spool Name: trg_send_stxn Flags: SEND_Q, SPOOLED, PROGRESS_TABLE, NEED_ACK, SENDQ_MASK, SREP_TABLE Txns in queue: 611107 Txns in memory: 51089 Txns in spool only: 560018 Txns spooled: 584408 Unspooled bytes: 6706326 Size of Data in queue: 121567408 Bytes Real memory in use: 6706326 Bytes Pending Txn Buffers: 0 Pending Txn Data: 0 Bytes Max Real memory data used: 6706326 Bytes Max Real memory hdrs used: 15897280 Bytes Total data queued: 6706326 Bytes Total Txns queued: 611107 Total Txns spooled: 0 Total Txns restored: 94961 Total Txns recovered: 0 Spool Rows read: 94962 Total Txns deleted: 0 Total Txns duplicated: 0 Total Txn Lookups: 94965 Any ideas? Jarrod Teale Team Lead - Manufacturing Execution Systems Automation and Process Control jarrod.teale@fonterra.com direct +64 7 850 7525 (ext 77525), mobile +64 21 968 364, fax +64 7 849 7855 Fonterra Co-operative Group Limited PO Box 459, Hamilton, 3240, Automation and Process Control, Fonterra Te Rapa, SH1, Hamilton, New Zealand ________________________________ DISCLAIMER This email contains information that is confidential and which may be legally privileged. If you have received this email in error, please notify the sender immediately and delete the email. This email is intended solely for the use of the intended recipient and you may not use or disclose this email in any way. ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.