Re: Replication Newbie Question
Posted in 2007
Topics: High Availability & Replication, Storage & Space Management, Stored Procedures & SPL, Platform-Specific Issues
I learned that one possibility for replication delay may be that I am replicating data from a blobspace, and that process was very slow. That seemed ok, except that now the send q has been growing for over 24 hours, with minimal transactions being sent to the receiving server.
Also of note, there is a repair job running for the table with the blobspace data. I understand that might clog up the Q, but there have not been any records added to the receiving table for over 24 hours. I tried to stop the repair job to see if it would clear things up and allow all the other transactions to be processed. so far, nothing is going through.
Thanks.
Here is a partial output from cdr list repl ( I have many tables replicating. they all show state = active).
REPLICATE: bcilic_psproda_103_9_company_address
STATE: Active ON:g_psproda
CONFLICT: Timestamp
FREQUENCY: immediate
QUEUE SIZE: 0
PARTICIPANT: bci_licensing:informix.company_address
OPTIONS: transaction,fullrow
REPLID: 66520 / 0x103d8
REPLMODE: PRIMARY ON:g_psproda
APPLY-AS: INFORMIX ON:g_psproda
REPLTYPE: Master
REPLICATE: bcilic_psproda_103_16_person_address
STATE: Active ON:g_psproda
CONFLICT: Timestamp
FREQUENCY: immediate
QUEUE SIZE: 15290
PARTICIPANT: bci_licensing:informix.person_address
OPTIONS: transaction,fullrow
REPLID: 66527 / 0x103df
REPLMODE: PRIMARY ON:g_psproda
APPLY-AS: INFORMIX ON:g_psproda
REPLTYPE: Master
REPLICATE: bcilic_psproda_103_4_lkup_county
STATE: Active ON:g_psproda
CONFLICT: Timestamp
FREQUENCY: immediate
QUEUE SIZE: 0
PARTICIPANT: bci_licensing:informix.lkup_county
OPTIONS: transaction,fullrow
REPLID: 66515 / 0x103d3
REPLMODE: PRIMARY ON:g_psproda
APPLY-AS: INFORMIX ON:g_psproda
REPLTYPE: Master
REPLICATE: bcilic_psproda_103_1_person
STATE: Active ON:g_psproda
CONFLICT: Timestamp
FREQUENCY: immediate
QUEUE SIZE: 339
PARTICIPANT: bci_licensing:informix.person
OPTIONS: transaction,fullrow
REPLID: 66512 / 0x103d0
REPLMODE: PRIMARY ON:g_psproda
APPLY-AS: INFORMIX ON:g_psproda
REPLTYPE: Master
This is the repair job
$ cdr list repair bciperson
RESYNCHJOB REPLICATE/REPLSET STATE
----------------------------------------------------------------------------
bciperson bcilic_psproda_103_1_person Pending Completion
SOURCE
------
bci_licensing@g_psproda:informix.person
select p_ssn , p_dob , p_last_name , p_first_name , p_middle_name , p_suffix , p_height , p_weight , p_eye_color , p_hair_color , p_drv_lic_no , p_drv_lic_state , p_home_phone , p_work_phone , p_sex , p_race , p_picture , p_web_phone , p_email_address , p_web_page , p_web_display from 'informix'.person
TARGET
-------
bci_licensing@g_psprodr:informix.person
select p_ssn , p_dob , p_last_name , p_first_name , p_middle_name , p_suffix , p_height , p_weight , p_eye_color , p_hair_color , p_drv_lic_no , p_drv_lic_state , p_home_phone , p_work_phone , p_sex , p_race , p_picture , p_web_phone , p_email_address , p_web_page , p_web_display from 'informix'.person
BLOCK SIZE: 10
TARGET ROW OPTION: Delete
PROCESSED ROWS: 0
START TIME: 2007-02-15 10:00:01
END TIME:
Laurie Gustin
IT Programmer Analyst
Department of Public Safety
lgustin@utah.gov
801-965-4410
>>> Nilesh Ozarkar <nilesho@us.ibm.com> 02/16/07 6:53 AM >>>
Check the queue stats using onstat -g rqm at source. And receive stats
using onstat -g rcv on target.
Although servers are active, check if replicates are active using 'cdr
list repl' and verify the state.
Regards,
Nilesh
"Laurie Gustin" <lgustin@utah.gov>
Sent by: informix-list-bounces@iiug.org
02/15/2007 09:46 AM
To
<informix-list@iiug.org>
cc
Subject
Replication Newbie Question
IDS10 FC5 HP-UX 11.11
I have a quick question about replication.
I have ER running between two servers and the queue on the sending server
just keeps getting larger. It appears that the 2 servers are
communicating, but how do I get the queue to write/flush so it doesnt just
keep getting bigger?
here are some outputs
$ cdr list serverSERVER ID STATE STATUS QUEUE CONNECTION CHANGED
-----------------------------------------------------------------------
g_psproda 1 Active Local 0
g_psprodr 21 Active Connected 86638846 Feb 15 08:02:36
weird but the cdr stats recv doesnt give me anything this morning
$ cdr stats recv
Receive Parallelism Statistics
Server Tot.Txn. Pending Active MaxPnd MaxAct AvgPnd AvgAct CommitRt
Receive Parallelism Statistics not available
Statistics by Source
Server Repl Txn Ins Del Upd Last Target Apply Last
Source Commit
Yes, I stopped and started CDR on the receiving server hoping to jump
start the sending of data... I know... that was dumb. Luckily, we arent
really using the psprodr server (I say it is a receiving server, even
though the replication is set up for update anywhere, all of the
updates/inserts are currently done on the psporda server)
Any hints you can give would be helpful.
Thanks.
Laurie
_______________________________________________
Informix-list mailing list
Informix-list@iiug.org
http://www.iiug.org/mailman/listinfo/informix-list
On Feb 16, 9:15 am, "Laurie Gustin" <LGUS...@utah.gov> wrote:
> I learned that one possibility for replication delay may be that I am replicating data from a blobspace, and that process was very slow. That seemed ok, except that now the send q has been growing for over 24 hours, with minimal transactions being sent to the receiving server.
>
> Also of note, there is a repair job running for the table with the blobspace data. I understand that might clog up the Q, but there have not been any records added to the receiving table for over 24 hours. I tried to stop the repair job to see if it would clear things up and allow all the other transactions to be processed. so far, nothing is going through.
>
> Thanks.
>
> Here is a partial output from cdr list repl ( I have many tables replicating. they all show state = active).
>
> REPLICATE: bcilic_psproda_103_9_company_address
> STATE: Active ON:g_psproda
> CONFLICT: Timestamp
> FREQUENCY: immediate
> QUEUE SIZE: 0
> PARTICIPANT: bci_licensing:informix.company_address
> OPTIONS: transaction,fullrow
> REPLID: 66520 / 0x103d8
> REPLMODE: PRIMARY ON:g_psproda
> APPLY-AS: INFORMIX ON:g_psproda
> REPLTYPE: Master
>
> REPLICATE: bcilic_psproda_103_16_person_address
> STATE: Active ON:g_psproda
> CONFLICT: Timestamp
> FREQUENCY: immediate
> QUEUE SIZE: 15290
> PARTICIPANT: bci_licensing:informix.person_address
> OPTIONS: transaction,fullrow
> REPLID: 66527 / 0x103df
> REPLMODE: PRIMARY ON:g_psproda
> APPLY-AS: INFORMIX ON:g_psproda
> REPLTYPE: Master
>
> REPLICATE: bcilic_psproda_103_4_lkup_county
> STATE: Active ON:g_psproda
> CONFLICT: Timestamp
> FREQUENCY: immediate
> QUEUE SIZE: 0
> PARTICIPANT: bci_licensing:informix.lkup_county
> OPTIONS: transaction,fullrow
> REPLID: 66515 / 0x103d3
> REPLMODE: PRIMARY ON:g_psproda
> APPLY-AS: INFORMIX ON:g_psproda
> REPLTYPE: Master
>
> REPLICATE: bcilic_psproda_103_1_person
> STATE: Active ON:g_psproda
> CONFLICT: Timestamp
> FREQUENCY: immediate
> QUEUE SIZE: 339
> PARTICIPANT: bci_licensing:informix.person
> OPTIONS: transaction,fullrow
> REPLID: 66512 / 0x103d0
> REPLMODE: PRIMARY ON:g_psproda
> APPLY-AS: INFORMIX ON:g_psproda
> REPLTYPE: Master
>
> This is the repair job
>
> $ cdr list repair bciperson>
> RESYNCHJOB REPLICATE/REPLSET STATE
> ----------------------------------------------------------------------------
> bciperson bcilic_psproda_103_1_person Pending Completion
>
> SOURCE
> ------
> bci_licensing@g_psproda:informix.person
> select p_ssn , p_dob , p_last_name , p_first_name , p_middle_name , p_suffix , p_height , p_weight , p_eye_color , p_hair_color , p_drv_lic_no , p_drv_lic_state , p_home_phone , p_work_phone , p_sex , p_race , p_picture , p_web_phone , p_email_address , p_web_page , p_web_display from 'informix'.person>
> TARGET
> -------
> bci_licensing@g_psprodr:informix.person
> select p_ssn , p_dob , p_last_name , p_first_name , p_middle_name , p_suffix , p_height , p_weight , p_eye_color , p_hair_color , p_drv_lic_no , p_drv_lic_state , p_home_phone , p_work_phone , p_sex , p_race , p_picture , p_web_phone , p_email_address , p_web_page , p_web_display from 'informix'.person>
> BLOCK SIZE: 10
> TARGET ROW OPTION: Delete
> PROCESSED ROWS: 0
> START TIME: 2007-02-15 10:00:01
> END TIME:
>
> Laurie Gustin
> IT Programmer Analyst
> Department of Public Safety
> lgus...@utah.gov
> 801-965-4410
>
> >>> Nilesh Ozarkar <nile...@us.ibm.com> 02/16/07 6:53 AM >>>
>
> Check the queue stats using onstat -g rqm at source. And receive stats
> using onstat -g rcv on target.
> Although servers are active, check if replicates are active using 'cdr
> list repl' and verify the state.
>
> Regards,
>
> Nilesh
>
> "Laurie Gustin" <lgus...@utah.gov>
> Sent by: informix-list-boun...@iiug.org
> 02/15/2007 09:46 AM
>
> To
> <informix-l...@iiug.org>
> cc
>
> Subject
> Replication Newbie Question
>
> IDS10 FC5 HP-UX 11.11
>
> I have a quick question about replication.
>
> I have ER running between two servers and the queue on the sending server
> just keeps getting larger. It appears that the 2 servers are
> communicating, but how do I get the queue to write/flush so it doesnt just
> keep getting bigger?
>
> here are some outputs
>
> $ cdr list server> SERVER ID STATE STATUS QUEUE CONNECTION CHANGED
> -----------------------------------------------------------------------
> g_psproda 1 Active Local 0
> g_psprodr 21 Active Connected 86638846 Feb 15 08:02:36
>
> weird but the cdr stats recv doesnt give me anything this morning
>
> $ cdr stats recv>
> Receive Parallelism Statistics
> Server Tot.Txn. Pending Active MaxPnd MaxAct AvgPnd AvgAct CommitRt
>
> Receive Parallelism Statistics not available
>
> Statistics by Source
>
> Server Repl Txn Ins Del Upd Last Target Apply Last
> Source Commit
>
> Yes, I stopped and started CDR on the receiving server hoping to jump
> start the sending of data... I know... that was dumb. Luckily, we arent
> really using the psprodr server (I say it is a receiving server, even
> though the replication is set up for update anywhere, all of the
> updates/inserts are currently done on the psporda server)
>
> Any hints you can give would be helpful.
>
> Thanks.
>
> Laurie
>
> _______________________________________________
> Informix-list mailing list
> Informix-l...@iiug.orghttp://www.iiug.org/mailman/listinfo/informix-list
1. Make sure that your sendq sbspace is not full at source server. See
for queue full messages and other error messages in server message log
file. Also check onstat -d output.
If queue is full then add space to the sendq sbspace. Otherwise ER
may not continue.
2. If queue full is not the problem then get us the following
statistics
onstat -g ath (at both source and target server)
onstat -g rcv full (at taget server)
onstat -g dss (at target server)
onstat -g rqm recvq ( at target server)
onstat -g rqm sendq ( at source server)
onstat -g nif <CDRID of target server> (at source server)
Thanks & Regards,
Nagaraju
Related threads
- Posting from the Informix-list
- Migrating from IDS 9.40.UC6 to 11.50.UC3
- Ip for a network session
- questions onstat -g