ER transactions queuing, NIF in BLOCK state

One of our servers in a 4-way update anywhere ER topology is currently not staying caught up, i.e. transactions from the other three servers are queueing but slowly getting sent and applied. The culprit seems to be the network interface. An onstat -g nif 105 shows: IBM Informix Dynamic Server Version 10.00.FC4 -- On-Line -- Up 14:03:38 -- 1214496 Kbytes NIF anchor Block: c0000000ae073318 nifGState RUN RetryTimeout 300 Detailed Site Instance Block: c0000000ae073a70 siteId 105 siteState 258 <RUN,BLOCK> siteVersion 8 siteCompress 2 siteNote 0 Send Thread 663273 <CDRNsT105> Recv Thread 663274 <CDRNr105> Connection Start (1153742441) 2006/07/24 07:00:41 Last Send (1153754528) 2006/07/24 10:22:08 Idle Timeout 300000 Flowblock Sent 0 Receive 1 NifInfo: c0000000ad273f70 Last Updated (1153742441) 2006/07/24 07:00:41 State Connected Total sent 37876 (8012974 bytes) Total recv'd 1388 (172439 bytes) Retry 0 (1 attempts) Connected 7 Protocol asf Proto block c0000000aedb67a0 assoc c0000000aefae028 state 0 signal 0 Recv Buf 0000000000000000 Recv Data 0 Send Count 0 Send Avail 0 What does the BLOCK really mean and how do I make it go away? The real kicker here is that the transactions are making their way to this server but extremely slow. BTW, this server is located at a remote site that we connect to via a VPN tunnel. I have opened a case with support but I thought that I'd ping the community to see if someone out there has a bright idea. The remote server (105) is running IDS 10.00.FC3X2, I know that we should upgrade it but currently it is what it is. Thanks, DL

DL Redden wrote: > One of our servers in a 4-way update anywhere ER topology is currently > not staying caught up, i.e. transactions from the other three servers > are queueing but slowly getting sent and applied. The culprit seems to > be the network interface. An onstat -g nif 105 shows: > > IBM Informix Dynamic Server Version 10.00.FC4 -- On-Line -- Up > 14:03:38 -- 1214496 Kbytes> > NIF anchor Block: c0000000ae073318 > nifGState RUN > RetryTimeout 300 > > Detailed Site Instance Block: c0000000ae073a70 > siteId 105 > siteState 258 <RUN,BLOCK> This means that the server has activated flow control. That occurs when the receive queue's queue mem size on the target gets full and is a request for the source to slow down a bit. This can occur because a large blob has been sent, or when the queue mem size is too small. You need to look at "onstat -g rcv full", "onstat - g rqm", "onstat -g ath", and the message log file of the server which is lagging. M.P. > siteVersion 8 > siteCompress 2 > siteNote 0 > Send Thread 663273 <CDRNsT105> > Recv Thread 663274 <CDRNr105> > Connection Start (1153742441) 2006/07/24 07:00:41 > Last Send (1153754528) 2006/07/24 10:22:08 > Idle Timeout 300000 > Flowblock Sent 0 Receive 1 > NifInfo: c0000000ad273f70 > Last Updated (1153742441) 2006/07/24 07:00:41 > State Connected > Total sent 37876 (8012974 bytes) > Total recv'd 1388 (172439 bytes) > Retry 0 (1 attempts) > Connected 7 > Protocol asf > Proto block c0000000aedb67a0 > assoc c0000000aefae028 > state 0 > signal 0 > Recv Buf 0000000000000000 > Recv Data 0 > Send Count 0 > Send Avail 0 > > > What does the BLOCK really mean and how do I make it go away? > > The real kicker here is that the transactions are making their way to > this server but extremely slow. BTW, this server is located at a remote > site that we connect to via a VPN tunnel. I have opened a case with > support but I thought that I'd ping the community to see if someone out > there has a bright idea. The remote server (105) is running IDS > 10.00.FC3X2, I know that we should upgrade it but currently it is what > it is. > > Thanks, > > DL >

I can see that there are transactions in receive queue but it does not look like they are being applied. There is nothing to speak of in any of my other queues on this server, the sendq occassionally has a transaction in it. What could be causing the received transactions to appear as though they are not being applied? onstat -g rcv full <snipped>Receive Parallelism Statistics Server Tot.Txn. Pending Active MaxPnd MaxAct AvgPnd AvgAct CommitRt 110 0 981 0 981 0 491.00 0.00 0.00 100 0 277 0 277 0 139.00 0.00 0.00 115 0 1290 0 1290 0 645.50 0.00 0.00 120 8 0 0 1 2 1.00 1.12 0.00 Tot Pending:2548 Tot Active:0 Avg Pending:529.29 Avg Active:1.12 Commit Rate:0.00 <snipped> Here are some of my CDR onconfig settings: CDR_EVALTHREADS 1,2 # evaluator threads (per-cpu-vp,additional) CDR_DSLOCKWAIT 120 # DS lockwait timeout (seconds) CDR_QUEUEMEM 4096 # Maximum amount of memory for any CDR queue (Kbytes) CDR_SUPPRESS_ATSRISWARN 2,3,4 # Which ATS or RIS messages to ignore CDR_NIFCOMPRESS 2 # Link level compression (-1 never, 0 none, 9 max) CDR_SERIAL 0,0 # Serial Column Sequence CDR_DBSPACE # dbspace for syscdr database CDR_QHDR_DBSPACE er_dbspace # CDR queue dbspace (default same as catalog) CDR_QDATA_SBSPACE er_sbspace1,er_sbspace2 # CDR queue smart blob space CDR_QDATA_SBFLAGS 0 # Log/no-log (default no log) ----- Original Message ---- From: Madison Pruet <mpruet@comcast.net> To: informix-list@iiug.org Cc: Informix List <informix-list@iiug.org> Sent: Monday, July 24, 2006 11:23:25 AM Subject: Re: ER transactions queuing, NIF in BLOCK state DL Redden wrote: > One of our servers in a 4-way update anywhere ER topology is currently > not staying caught up, i.e. transactions from the other three servers > are queueing but slowly getting sent and applied. The culprit seems to > be the network interface. An onstat -g nif 105 shows: > > IBM Informix Dynamic Server Version 10.00.FC4 -- On-Line -- Up > 14:03:38 -- 1214496 Kbytes> > NIF anchor Block: c0000000ae073318 > nifGState RUN > RetryTimeout 300 > > Detailed Site Instance Block: c0000000ae073a70 > siteId 105 > siteState 258 <RUN,BLOCK> This means that the server has activated flow control. That occurs when the receive queue's queue mem size on the target gets full and is a request for the source to slow down a bit. This can occur because a large blob has been sent, or when the queue mem size is too small. You need to look at "onstat -g rcv full", "onstat - g rqm", "onstat -g ath", and the message log file of the server which is lagging. M.P. > siteVersion 8 > siteCompress 2 > siteNote 0 > Send Thread 663273 <CDRNsT105> > Recv Thread 663274 <CDRNr105> > Connection Start (1153742441) 2006/07/24 07:00:41 > Last Send (1153754528) 2006/07/24 10:22:08 > Idle Timeout 300000 > Flowblock Sent 0 Receive 1 > NifInfo: c0000000ad273f70 > Last Updated (1153742441) 2006/07/24 07:00:41 > State Connected > Total sent 37876 (8012974 bytes) > Total recv'd 1388 (172439 bytes) > Retry 0 (1 attempts) > Connected 7 > Protocol asf > Proto block c0000000aedb67a0 > assoc c0000000aefae028 > state 0 > signal 0 > Recv Buf 0000000000000000 > Recv Data 0 > Send Count 0 > Send Avail 0 > > > What does the BLOCK really mean and how do I make it go away? > > The real kicker here is that the transactions are making their way to > this server but extremely slow. BTW, this server is located at a remote > site that we connect to via a VPN tunnel. I have opened a case with > support but I thought that I'd ping the community to see if someone out > there has a bright idea. The remote server (105) is running IDS > 10.00.FC3X2, I know that we should upgrade it but currently it is what > it is. > > Thanks, > > DL > _______________________________________________ Informix-list mailing list Informix-list@iiug.org http://www.iiug.org/mailman/listinfo/informix-list