HDR in IDS 10.0xc5: slow Secondary restart

Hi, everybody, Long-awaited IDS 10.0xC5 (with a great FASTPOLL feature) is finally released... and appeared to be almost unusable for large installations, using HDR Really, really bad 'feature' was added to HDR mechanism. Every time, HDR Secondary is restarted (that is, stopped and started again), IDS completely re-writes all logical logs on Secondary. All logical logs become market with F------ flag, and all logical log pages become re-initialized with page number in a page header and zeroes inside, like: stage10@dw-1g:~$ hexdump -s 108544 -n 2048 /dev/lds1 001a800 0000 0000 0000 0000 0000 0000 0000 0000 * 001b000 Do I need to say, how slow it is? On my staging machine (older copy of production, used for QA), I have 140 GB of logical logs. The staging array (not very fast) is able to write at 50 MB/sec, but IDS (may be, because of small 1-MB block size, used for data writing with pwrite() ) only writes at 20 MB/sec. That is, it takes about two hours to simply restart Secondary... I had plan to double the total logical log volume.. With 10.0fc4, it takes me just two minutes to restart Secondary and have the HDR replication re-synchronized. From 2 minutes to two hours... Not a good upgrade path (FC4->FC5) I plan to file this problem to IBM asap š -Alexey Sonkin

This is an HDR bug in specific scenario and nothing to do with FASTPOOL. This problem will happen when secondary is bounced right after it is synchronized with primary using backup/restore. When secondary is restored from backup, it clears the log (existing behavior). If secondary is bounced before those cleared log are used, it will try to clear them again. Based on your log size, it may take long time. If logs are cycled after secondary synchronized with primary, this long cleanup problem should not happen. At the startup of secondary server will cleanup only those logs where are still marked as F (free): address number flags uniqid begin size used %used b0444b8 1 U-B---- 65 2:53 1000 5 0.50 b044500 5 U-B---- 66 2:1053 1000 1 0.10 b044548 6 U---C-L 67 2:2053 1000 3 0.30 b044590 7 F------ 0 2:3053 1000 0 0.00 <====== b0445d8 8 F------ 0 2:4053 1000 0 0.00 <====== If you upgrade to UC5 using in place upgrade and don’t initialize secondary using backup/restore you should be fine. If you have to setup the secondary from backup restore, make sure that you don’t have Free log before you bounce the secondary. Bug is identified by IDS development and fix should be available in next PID release. -Ajay Gupta

Hi, Ajay, Actually, there are two different problems with HDR in 10.0FC5: - one problem is a NETVP spinning on Secondary, when FASTPOLL=1 on Secondary; this problems reproduces always, and doesn't relate to logical log re-initialization - The other problem is a complete Logical Log re-initialization after Secondary restart I've just tested, what happens, if logical logs are completely recycled (marked as U-B----) after upgrade to FC5. Bad news: the problem still reproduces, IDS still re-initializes all the logical logs. -Alexey > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of AJAY > GUPTA > Sent: Monday, June 12, 2006 7:12 PM > To: ids@iiug.org > Subject: Re: HDR in IDS 10.0xc5: slow Secondary restart [6931] > > > This is an HDR bug in specific scenario and nothing to do with FASTPOOL. > > This problem will happen when secondary is bounced right after it is > synchronized with primary using backup/restore. When secondary is restored > from backup, it clears the log (existing behavior). If secondary is bounced > before those cleared log are used, it will try to clear them again. Based on > your log size, it may take long time. If logs are cycled after secondary > synchronized with primary, this long cleanup problem should not happen. At the > startup of secondary server will cleanup only those logs where are still > marked as F (free): > > address number flags uniqid begin size used %used > b0444b8 1 U-B---- 65 2:53 1000 5 0.50 > b044500 5 U-B---- 66 2:1053 1000 1 0.10 > b044548 6 U---C-L 67 2:2053 1000 3 0.30 > b044590 7 F------ 0 2:3053 1000 0 0.00 <====== > b0445d8 8 F------ 0 2:4053 1000 0 0.00 <====== > > If you upgrade to UC5 using in place upgrade and don't initialize secondary > using backup/restore you should be fine. If you have to setup the secondary > from backup restore, make sure that you don't have Free log before you bounce > the secondary. > > Bug is identified by IDS development and fix should be available in next PID > release. > > -Ajay Gupta > > > ************************************************************************ ****** > * > Forum Note: Use "Reply" to post a response in the discussion forum. >

Alexey, Is is all of the logical logs, or all of the logical logs created since= the last checkpoint? I'm pretty sure that the behaviour has been to zap the logical logs cre= ated after the recovery checkpoint for quite some time. M.P. = "Alexey Sonkin" = <alexeys@cidc.com = > = To Sent by: ids@iiug.org = ids-bounces@iiug. = cc org = Subj= ect RE: HDR in IDS 10.0xc5: slow = 06/13/2006 02:23 Secondary restart [6947] = PM = = = Please respond to = ids@iiug.org = = = Hi, Ajay, Actually, there are two different problems with HDR in 10.0FC5: - one problem is a NETVP spinning on Secondary, when FASTPOLL=3D1 on Secondary; this problems reproduces always, and doesn't relate to logical log re-initialization - The other problem is a complete Logical Log re-initialization after Secondary restart I've just tested, what happens, if logical logs are completely recycled (marked as U-B----) after upgrade to FC5. Bad news: the problem still reproduces, IDS still re-initializes all the logical logs. -Alexey > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of= AJAY > GUPTA > Sent: Monday, June 12, 2006 7:12 PM > To: ids@iiug.org > Subject: Re: HDR in IDS 10.0xc5: slow Secondary restart [6931] > > > This is an HDR bug in specific scenario and nothing to do with FASTPOOL. > > This problem will happen when secondary is bounced right after it is > synchronized with primary using backup/restore. When secondary is restored > from backup, it clears the log (existing behavior). If secondary is bounced > before those cleared log are used, it will try to clear them again. Based on > your log size, it may take long time. If logs are cycled after secondary > synchronized with primary, this long cleanup problem should not happen. At the > startup of secondary server will cleanup only those logs where are still > marked as F (free): > > address number flags uniqid begin size used %used > b0444b8 1 U-B---- 65 2:53 1000 5 0.50 > b044500 5 U-B---- 66 2:1053 1000 1 0.10 > b044548 6 U---C-L 67 2:2053 1000 3 0.30 > b044590 7 F------ 0 2:3053 1000 0 0.00 <=3D=3D=3D=3D=3D=3D > b0445d8 8 F------ 0 2:4053 1000 0 0.00 <=3D=3D=3D=3D=3D=3D > > If you upgrade to UC5 using in place upgrade and don't initialize secondary > using backup/restore you should be fine. If you have to setup the secondary > from backup restore, make sure that you don't have Free log before yo= u bounce > the secondary. > > Bug is identified by IDS development and fix should be available in next PID > release. > > -Ajay Gupta > > > ***********************************************************************= * ****** > * > Forum Note: Use "Reply" to post a response in the discussion forum. > ***********************************************************************= ******** Forum Note: Use "Reply" to post a response in the discussion forum. =

Madison, This is about cleaning ALL the logical logs every time Secondary is restarted (bounced). This behavior has never been in Informix before 10.0xC5 ------------------------------------------ Alexey Sonkin -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Madison Pruet Sent: Tuesday, June 13, 2006 3:43 PM To: ids@iiug.org Subject: RE: HDR in IDS 10.0xc5: slow Secondary restart [6948] Alexey, Is is all of the logical logs, or all of the logical logs created since= the last checkpoint? I'm pretty sure that the behaviour has been to zap the logical logs cre= ated after the recovery checkpoint for quite some time. M.P. = Hi, Ajay, Actually, there are two different problems with HDR in 10.0FC5: - one problem is a NETVP spinning on Secondary, when FASTPOLL=3D1 on Secondary; this problems reproduces always, and doesn't relate to logical log re-initialization - The other problem is a complete Logical Log re-initialization after Secondary restart I've just tested, what happens, if logical logs are completely recycled (marked as U-B----) after upgrade to FC5. Bad news: the problem still reproduces, IDS still re-initializes all the logical logs. -Alexey > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of= AJAY > GUPTA > Sent: Monday, June 12, 2006 7:12 PM > To: ids@iiug.org > Subject: Re: HDR in IDS 10.0xc5: slow Secondary restart [6931] > > > This is an HDR bug in specific scenario and nothing to do with FASTPOOL. > > This problem will happen when secondary is bounced right after it is > synchronized with primary using backup/restore. When secondary is restored > from backup, it clears the log (existing behavior). If secondary is bounced > before those cleared log are used, it will try to clear them again. Based on > your log size, it may take long time. If logs are cycled after secondary > synchronized with primary, this long cleanup problem should not happen. At the > startup of secondary server will cleanup only those logs where are still > marked as F (free): > > address number flags uniqid begin size used %used > b0444b8 1 U-B---- 65 2:53 1000 5 0.50 > b044500 5 U-B---- 66 2:1053 1000 1 0.10 > b044548 6 U---C-L 67 2:2053 1000 3 0.30 > b044590 7 F------ 0 2:3053 1000 0 0.00 <=3D=3D=3D=3D=3D=3D > b0445d8 8 F------ 0 2:4053 1000 0 0.00 <=3D=3D=3D=3D=3D=3D > > If you upgrade to UC5 using in place upgrade and don't initialize secondary > using backup/restore you should be fine. If you have to setup the secondary > from backup restore, make sure that you don't have Free log before yo= u bounce > the secondary. > > Bug is identified by IDS development and fix should be available in next PID > release. > > -Ajay Gupta > > > ***********************************************************************= * ****** > * > Forum Note: Use "Reply" to post a response in the discussion forum. > ***********************************************************************= ******** Forum Note: Use "Reply" to post a response in the discussion forum. = ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum.