onstat -g rqm ER Statistics for RQM Queues
The onstat -g rqm command displays the statistics for Reliable Queue Manager [RQM] queues.
The RQM manages the insertion and removal of items to and from the various queues. The RQM also
manages spooling of the in-memory portions of the queue to and from disk. The onstat -g rqm
command displays the contents of the queue, size of the transactions in the queue, how much
of the queue is in memory and on disk, the location of various handles to the queue, and
the contents of the various progress tables. You can choose to print information for all
queues or for just one queue by using one of the modifiers described below.
If a queue is empty, no information is printed for that queue.
The onstat -g rqm can take an optional modifier. The following table describes the values for the modifier.
| Modifier | Action |
|---|---|
| ACKQ | Prints the ack send queue |
| CNTRLQ | Prints the control send queue |
| RECVQ | Prints the receive queue |
| SBSPACES | Prints detailed statistical information about the sbspaces configured for CDR_QDATA_SBSPACE. |
| SENDQ | Prints the send queue |
| SYNCQ | Prints the sync send queue |
| FULL | Prints full information about every in-memory transaction for every queue |
| BRIEF | Prints a brief summary of the number of transactions in each of the queues and the replication servers for which the data is queued Use this modifier to quickly identify sites where a problem exists. If large amounts of data are queued for a single server, then that server is probably down or off the network. |
| VERBOSE | Prints all the buffer headers in memory |
When you specify a modifier to select a specific queue, the command prints all the statistics for that queue and information about the first and last in-memory transactions for that queue. When you select the SBSPACES modifier, the command prints information about the sbspaces being used for replication, including how full those sbspaces are.
The other modifiers of the onstat -g rqm command are used primarily as a debugging tool
The output for the SENDQ modifier contains the following sections:
- The current statistics section (Transaction spool name through Pending Txn Data): Contains information about the current contents of the queue, such as how many bytes are contained in the queue, how many transactions are in the queue, how many transactions are currently in memory, how many have been spooled to disk, how many exist only on disk, and so on. The Insert Stamp field value is used to maintain the order of the transactions within the queue. The Size of Data in queue field shows the size of the queue when combining the in-memory transactions with the spool-only transactions. The Pending Txn Buffers field contains information about transactions that are in the process of being queued into the send queue. The historical statistics section (Max Real memory data used through Total Txn Lookups): contains a summary of what has been placed in the queue in the past. The Max Real memory data used field contains the largest in memory size of the queue. The Total Txn Recovered field shows the transactions that existed only in the spool when the server was started. The Total Txns deleted field shows the number of transactions that have been removed from the queue. The Total Txns duplicated field contains the number of times attempted to queue a transaction that had already been processed. The Total Txn Lookups field is a counter of the number of times that an ER thread attempted to read a transaction.
- The Progress Table section: contains information on what is currently queued, to which server it is queued for, and what has been acknowledged from each of the participants of the replicate. The first part of the progress table section is a summary . Below the summary section is a list of the servers and group entries that contain what is currently queued for each server, what has been sent to the remote server, and what has been acknowledged from the remote server. The contents of the ACKed and Sent columns contains the key of the last transaction that was acknowledged from the remote server or sent to that server. The key is a multi-part number consisting of source_node/unique_log_id/logpos/incremental number. The transaction section contains the first and last transaction in the queue that are currently in memory. The NeedAck field shows from which server the transaction is waiting for an acknowledgment. You can use this bitmap mask with the output from the onstat -g cat command to determine the name of the server which server Enterprise Replication is waiting on for an acknowledgment.
- The Transverse handle section: contains the position within the queue that any thread is currently processing. Each thread that attempts to read a transaction from the queue, or to place a transaction into the queue must first allocate a handle. This handle is used to maintain the positioning within the queue.
onstat -g rqm sendq
CDR Reliable Queue Manager (RQM) Statistics:
RQM Statistics for Queue (0x3d914a028) trg_send
Transaction Spool Name: trg_send_stxn
Insert Stamp: 22293692
Flags: SEND_Q, SPOOLED, PROGRESS_TABLE, NEED_ACK
Txns in queue: 0
Log Events in queue: 0
Txns in memory: 0
Txns in spool only: 0
Txns spooled: 0
Unspooled bytes: 0
Size of Data in queue: 0 Bytes
Real memory in use: 0 Bytes
Pending Txn Buffers: 0
Pending Txn Data: 0 Bytes
Max Real memory data used: 4194248 (4194304) Bytes
Max Real memory hdrs used 3005856 (3081680832) Bytes
Total data queued: 14658380490 Bytes
Total Txns queued: 22293692
Total Txns spooled: 2656
Total Txns restored: 682
Total Txns recovered: 0
Spool Rows read: 682
Total Txns deleted: 22293692
Total Txns duplicated: 0
Total Txn Lookups: 66810608
Progress Table:
Progress Table is Stable
On-disk table name............: spttrg_send
Flush interval (time).........: 30
Time of last flush............: 1370363073
Flush interval (serial number): 1000
Serial number of last flush...: 109915
Current serial number.........: 109917
Server Group Bytes Queued Acked Sent
------------------------------------------------------------------------------
1 0x2003b 0 2/3668/0x898a16c/0x0 - 2/3668/0x898a16c/0x0
1 0x20019 0 2/3610/0xccf0458/0x0 - 2/3610/0xccf0458/0x0
1 0x200cd 0 2/1394/0x136cb2f0/0x0 - No Content Yet
1 0x20035 0 2/2964/0x25e32220/0x0 - No Content Yet
1 0x2008e 0 2/887/0xedd11b8/0x0 - No Content Yet
1 0x2008d 0 2/990/0x1c5683a4/0x0 - No Content Yet
1 0x20091 0 2/1384/0x1ecab428/0x0 - No Content Yet
1 0x20092 0 2/1384/0x1ec6d7b0/0x0 - No Content Yet
1 0x20011 0 2/439/0xa10f3f4/0x0 - No Content Yet
1 0x20053 0 2/3503/0x5f55638/0x0 - 2/3503/0x5f55638/0x0
1 0x20052 0 2/1929/0x137a2d8/0x0 - No Content Yet
1 0x2005b 0 2/3665/0x300071b0/0x0 - 2/3665/0x300071b0/0x0
1 0x2000c 0 2/633/0x2e8b001c/0x0 - No Content Yet
1 0x2003c 0 2/3674/0x125065c8/0x0 - 2/3674/0x125065c8/0x0
1 0x2000f 0 2/647/0x1d691318/0x0 - No Content Yet
1 0x20013 0 2/647/0x1d832434/0x0 - No Content Yet
1 0x20016 0 2/647/0x1d76d708/0x0 - No Content Yet
1 0x20055 0 2/3674/0x1365e690/0x0 - 2/3674/0x1365e690/0x0
1 0x20054 0 2/3674/0x12e0b254/0x0 - 2/3674/0x12e0b254/0x0
1 0x2006c 0 2/3674/0x12e3a31c/0x0 - 2/3674/0x12e3a31c/0x0
1 0x2000a 0 2/347/0xbb67340/0x0 - No Content Yet
1 0x20057 0 2/3674/0x14d4d7c8/0x0 - 2/3674/0x14d4d7c8/0x0
1 0x20065 0 2/3674/0x14d6137c/0x0 - 2/3674/0x14d6137c/0x0
1 0x20038 0 2/3668/0x2f7bf1b0/0x0 - 2/3668/0x2f7bf1b0/0x0
1 0x2001b 0 2/3674/0x11cb4364/0x0 - 2/3674/0x11cb4364/0x0
1 0x20001 0 2/3674/0x12f0618c/0x0 - 2/3674/0x12f0618c/0x0
Traverse handle (0x3d9788028) for thread CDRNsT1 at Head_of_Q, Flags: None
.......
Traverse handle (0x3d97c8028) for thread CDRNr1 at Head_of_Q, Flags: None
onstat -g rqm recv
CDR Reliable Queue Manager (RQM) Statistics:
RQM Statistics for Queue (0x3d919f028) trg_receive
Transaction Spool Name: trg_receive_stxn
Insert Stamp: 367458171
Communal Stamp: 0
Flags: RECV_Q, SPOOLED, PROGRESS_TABLE
Txns in queue: 6
Txns in memory: 6
Txns in spool only: 0
Txns spooled: 0
Unspooled bytes: 0
Size of Data in queue: 782281 Bytes
Real memory in use: 782281 Bytes
Pending Txn Buffers: 0
Pending Txn Data: 0 Bytes
Max Real memory data used: 5999987 (4194304) Bytes
Max Real memory hdrs used 6503584 (3081680832) Bytes
Total data queued: 23884344839 Bytes
Total Txns queued: 37092106
Total Txns spooled: 0
Total Txns restored: 0
Total Txns recovered: 0
Spool Rows read: 0
Total Txns deleted: 37092100
Total Txns duplicated: 0
Total Txn Lookups: 74184207
Progress Table:
Progress Table is Stable
On-disk table name............: spttrg_receive
Not keeping dirty list.
Server Group Bytes Queued Acked Sent
------------------------------------------------------------------------------
1 0x2001b 0 1/268769/0xe2593cc/0x0 - No Content Yet
1 0x20054 0 1/268196/0xd1bf2b8/0x0 - No Content Yet
1 0x2006c 0 1/268196/0xe02d174/0x0 - No Content Yet
1 0x20055 0 1/268196/0xdb02258/0x0 - No Content Yet
1 0x2003c 0 1/271769/0x25716258/0x0 - 1/271769/0x25716258/0x0
1 0x2000c 0 1/271759/0x36241b0/0x0 - 1/271759/0x36241b0/0x0
1 0x2003b 0 1/271761/0x2fb351b0/0x0 - 1/271761/0x2fb351b0/0x0
1 0x20019 0 1/268757/0x18987220/0x0 - No Content Yet
1 0x20090 0 1/271708/0x2d9b1318/0x0 - 1/271708/0x2d9b1318/0x1
1 0x200cd 652211 1/271769/0x2710a138/0x0 -1/271769/0x275b4140/0x2a3
1 0x20011 0 1/267739/0x186a8314/0x0 - No Content Yet
1 0x20038 0 1/268761/0x1b97b1b0/0x0 - No Content Yet
1 0x20016 0 1/271769/0x245ca0f8/0x0 -1/271769/0x245ca0f8/0x137
1 0x2008e 0 1/271761/0x73c53cc/0x0 - 1/271761/0x73c53cc/0x0
1 0x20013 0 1/271769/0x24935580/0x0 -1/271769/0x24935580/0x137
1 0x20065 0 1/268769/0xff1b37c/0x0 - No Content Yet
1 0x20057 0 1/268769/0xff147c8/0x0 - No Content Yet
1 0x2000f 0 1/271769/0x248990e0/0x0 -1/271769/0x248990e0/0x130
1 0x2000a 0 1/269782/0x275512b0/0x0 - No Content Yet
1 0x20092 0 1/271769/0x27091718/0x0 -1/271769/0x27091718/0xe8
1 0x2008d 0 1/271731/0x4239220/0x0 - 1/271731/0x4239220/0x0
1 0x200aa 0 1/267744/0x6446118/0x0 - No Content Yet
1 0x2005b 0 1/268181/0x2010e1b0/0x0 - No Content Yet
1 0x20091 0 1/271769/0x270d4068/0x0 -1/271769/0x270d4068/0x55
1 0x20001 342 1/271769/0x2710b18c/0x0 - 1/271769/0x275b518c/0x0
First Txn (0x3e43277b0) Key: 1/271769/0x2746b074/0x00000000
Txn Stamp: 367458166, Reference Count: 1.
Txn Flags: None
Txn Commit Time: (1370363010) 2013/06/04 11:23:30
Txn Size in Queue: 619337
First Buf's (0x3d9a740d0) Queue Flags: Resident
First Buf's Buffer Flags: TRG, Stream
Traverse handle (0x3dd127218) for thread CDRD_8397 at
txn (0x3e43277b0): 1/271769/0x2746b074/0x00000000 Flags: In_Transaction
Last Txn (0x3dd5b0a78) Key: 1/271769/0x275b518c/0x00000000
Txn Stamp: 367458171, Reference Count: 0.
Txn Flags: None
Txn Commit Time: (1370363011) 2013/06/04 11:23:31
Txn Size in Queue: 146
First Buf's (0x3dd5b0c90) Queue Flags: Resident
First Buf's Buffer Flags: TRG, Stream
Traverse handle (0x3dd345028) for thread CDRD_8425 at txn (0x3dd5b0a78) End_of_Q, Flags: None
Traverse handle (0x3d97c6028) for thread CDRNr1 at Head_of_Q, Flags: None
Traverse handle (0x3dc80a028) for thread CDRD_8375 at Head_of_Q, Flags: None
Traverse handle (0x3dd2a1028) for thread CDRD_8390 at Head_of_Q, Flags: None
......
Traverse handle (0x3dd127218) for thread CDRD_8397 at
txn (0x3e43277b0): 1/271769/0x2746b074/0x00000000
Flags: In_Transaction
Traverse handle (0x3dd1282c0) for thread CDRD_8397 at Head_of_Q, Flags: None
......
Traverse handle (0x3dd345028) for thread CDRD_8425 at txn (0x3dd5b0a78) End_of_Q, Flags: None
......
</div>