onstat -g rcv ER Statistics for the Receive Manager

The onstat -g rcv command displays the statistics for the receive manager.

The Statistics by Source section of the onstat -g rcv command output shows the following information for each source server. For each replicate ID:

The number of transactions that are applied from the source servers
The number of inserts, deletes, and updates within the applied transactions
The timestamp of the most recently applied transaction on the target server
The timestamp of the commit on the source server for the most recently applied transaction

onstat -g rcv full

ServerId: 1
Flags 0x0

Threads:
         Id Name         State  Handle
        285461 CDRACK_6       Idle 0x3d9d09778
        228884 CDRACK_5       Idle 0x3dcce65c8
        148457 CDRACK_4       Idle 0x3dce11128
        13832 CDRACK_3       Idle 0x3d9a02de8
        13013 CDRACK_2       Idle 0x3d9d0a208
        522 CDRACK_1       Idle 0x3d89cb750
        521 CDRACK_0       Idle 0x3d89cb610

 Receive Manager global  block 0x3d9718028
         cdrRM_inst_ct:                          1
         cdrRM_State:                    00000040
         cdrRM_numSleepers:              21
         cdrRM_DsCreated:                23
         cdrRM_MinDSThreads:             6
         cdrRM_MaxDSThreads:             24
         cdrRM_DSBlock                   0
         cdrRM_DSParallelPL              0
         cdrRM_DSFailRate                0.000675
         cdrRM_DSNumRun:                 37027814
         cdrRM_DSNumLockTimeout          121
         cdrRM_DSNumLockRB               1965
         cdrRM_DSNumDeadLocks            41474
         cdrRM_DSNumPCommits             2111170
         cdrRM_ACKwaiting                0
         cdrRM_totSleep:                 527
         cdrRM_Sleeptime:                45
         cdrRM_Workload:                 0
         cdrRM_optscale:                 4
         cdrRM_MinFloatThreads:          2
         cdrRM_MaxFloatThreads:          7
         cdrRM_AckThreadCount:           7
         cdrRM_AckWaiters:               7
         cdrRM_AckCreateStamp:           Fri May 17 20:56:37 2013
         cdrRM_DSCreateStamp:            Tue Jun  4 10:32:25 2013
         cdrRM_acksInList:               0
         cdrRM_BlobErrorBufs:            0

Receive Parallelism Statistics
Server Tot.Txn. Pending Active MaxPnd MaxAct  AvgPnd  AvgAct CommitRt
     1 37027814       313      2  14309     24  507.66   17.48   24.05

Tot Pending:313   Tot Active:2  Avg Pending:507.66  Avg Active:17.48
Commit Rate:24.05

Time Spent In RM Parallel Pipeline Levels
Lev. TimeInSec  Pcnt.
   0   1538595  99.98%
   1        90   0.01%
   2       165   0.01%

Statistics by Source

Server 1
Repl        Txn     Ins     Del     Upd Last Target Apply   Last Source Commit
131277  26828667 26760220 26730686 26764607 2013/06/04 10:34:55 2013/06/04 10:34:54
131073    85983   42989   42994       0 2013/06/04 10:34:55 2013/06/04 10:34:54
131132  3576204       0       0 3576204 2013/06/04 10:31:39 2013/06/04 10:31:38
131217  3208448 3175834 3169772 3176427 2013/06/04 10:31:56 2013/06/04 10:31:55
131087  1507674 1629310 1629111 1479734 2013/06/04 10:35:12 2013/06/04 10:35:08
131094    32052 3402854 3402854 6805708 2013/06/04 10:35:03 2013/06/04 10:35:00
131091  1677074 1666392 1666392 1666392 2013/06/04 10:30:24 2013/06/04 10:30:09
131218    42692 3175852 3175850 6351702 2013/06/04 10:31:55 2013/06/04 10:31:55
131214    58479   58377       0   58377 2013/06/04 02:17:04 2013/06/04 02:17:04
131213       32      20       0      20 2013/06/03 00:15:18 2013/06/03 00:15:17
131131     5166       0       0    5166 2013/06/04 03:06:23 2013/06/04 03:06:14
131216        2       1       0       1 2013/06/02 00:15:17 2013/06/02 00:15:17

Replicates Being Throttled
Repid      Max
           Concurrent
131213     3

TimeSeries Statistics by Source

Server 1
Repl        Txn   TSIns   TSDel   TSCmd Last Target Apply   Last Source Commit 
131084     5339    5263       0    5263 2013/06/04 00:01:02 2013/06/04 00:00:32

</div>

Receive Manager section

Field	Description
cdrRM_DSParallelPL	Shows the current level of Apply Parallelism, 0 (zero) being the highest
cdrRM_DSNumLockTimeout cdrRM_DSNumLockRB cdrRM_DSNumDeadLocks	Indicate the number of collisions between various apply threads
cdrRM_acksinList	Shows acknowledgments that are received but not yet processed

Receive Parallelism Statistics

Field	Description
Server	Source server ID
Concur	Number of transactions currently being applied in parallel
Tot.Txn.	Total number of transactions that are applied from this source server
Pending	Number of current transactions in the pending list for this source server
Active	Number of current transactions currently being applied from this source server
MaxPnd	Maximum number of transactions in the pending list queue
MaxAct	Maximum number of transactions in the active list queue
AvgPnd	Average depth of the pending list queue
AvgAct	Average depth of the active list queue
CommitRt	Commit rate of transaction from this source server, based on transactions per second

Notes

CommitRt is a good metric to monitor, a drop in rate is normally an indication of a problem

If a replicate encounters a deadlock situation or otherwise reduces the degree of parallelism by which transactions are applied, the Statistics by Source section shows the replicate and the maximum number of concurrent transactions that are possible.

Replicates that have a TimeSeries column are listed separately

A ratio of greater than 1% for cdrRM_DSNumRun:cdrRM_DSNumDeadLocks is a cause for concern