Tuning Checkpoints

I need some help getting checkpoints durations lowered. We are running informix 9.4. I have been working with Informix Tech support and it has gotten some better but it is not as good as it should be. We have been adjusting BUFFERS, LRUS, CLEANERS, LRU_MAX_DIRTY, and LRU_MIN_DIRTY. BUFFERS = 350000 LRUS = 128 CLEANERS = 127 LRU_MAX_DIRTY = .5 LRU_MIN_DIRTY = .25 After the last changes we made, LRU writes is now lower than chunk writes for the first time. Our checkpoints have ranged up to 7 seconds and sometimes higher. Anyone have any ideas of what else we can look at? HP GL380G5 servers 32Gb Memory (2) quad core Processors The dbspaces are located on Compellent SAN fiber channel drives RAID level 10 (striped and mirrored).

ALICE BENNETT wrote: > I need some help getting checkpoints durations lowered. We are running > informix 9.4. I have been working with Informix Tech support and it has gotten > some better but it is not as good as it should be. We have been adjusting > BUFFERS, LRUS, CLEANERS, LRU_MAX_DIRTY, and LRU_MIN_DIRTY. > > BUFFERS = 350000 > LRUS = 128 > CLEANERS = 127 > LRU_MAX_DIRTY = .5 > LRU_MIN_DIRTY = .25> > After the last changes we made, LRU writes is now lower than chunk writes for > the first time. > > Our checkpoints have ranged up to 7 seconds and sometimes higher. Anyone have > any ideas of what else we can look at? > Are chunks using KAIO (ie are they RAW device)? If so you may have to tune the HPUX KAIO subsystem. Tuning KAIO on HPUX can make a huge difference in write times like checkpoint. Art S. Kagel > HP GL380G5 servers > 32Gb Memory > (2) quad core Processors > The dbspaces are located on Compellent SAN fiber channel drives RAID level 10 > (striped and mirrored). > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > >

Hi, 128 LRUs with 350000 Buffers and LRU_MAX_DIRTY 0.5 should be far enough and give you checkpoints between one and two seconds. Is your IO performance okay (KAIO an how many CPU VPs or when using normal IO how many AIO VPs)? Does your SAN storage have a write cache (battery buffered)? Maybe you may want to try lower LRU-numbers. SAP suggests LRU numbers = CPU VPs. In most cases that seems to work, too. Did you monitor your checkpoints? Run onstat - in an endless loop and as long as it shows a checkpoint get the dirty buffers (onstat -R) and the threads in critical sections (onstat -u | grep X) - maybe you don't have a problem with the dirty buffers but with sessions in critical sections which block the checkpoint. We had some problems with (degenerated) attached indices and insert statements - sometimes they blocked a checkpoint for over a minute. Regards, Andreas Kutsche > ------------------------------------------- SPAR Österreichische Warenhandels-AG Hauptzentrale A - 5015 Salzburg, Europastrasse 3 FN 34170 a Tel: +43 662 4470 24223 Mobile: +43 664 6259575 E-Mail: Andreas.KUTSCHE@spar.at Internet: http://www.spar.at Wichtiger Hinweis: Der Inhalt dieser E-Mail kann vertrauliche und rechtlich geschützte Informationen, insbesondere Betriebs- oder Geschäftsgeheimnisse, enthalten, zu deren Geheimhaltung der Empfänger verpflichtet ist. Die Informationen in dieser E-Mail sind ausschließlich für den Adressaten bestimmt. Sollten Sie die E-Mail irrtümlich erhalten haben so ersuchen wir Sie, die Nachricht von Ihrem System zu löschen und sich mit uns in Verbindung zu setzen. Über das Internet versandte E-Mails können leicht manipuliert oder unter fremdem Namen erstellt werden. Daher schließen wir die rechtliche Verbindlichkeit der in dieser Nachricht enthaltenen Informationen aus. Der Inhalt der E-Mail ist nur rechtsverbindlich, wenn er von uns schriftlich bestätigt und gezeichnet wird. Sollte trotz der von uns verwendeten Virus-Schutzprogramme durch die Zusendung von E-Mails ein Virus in Ihre Systeme gelangen, haften wir nicht für evtl. hieraus entstehende Schäden. Wir danken für Ihr Verständnis. Important notice: The contents of this e-mail may contain confidential and legally protected information that is in particular related to operational and trade secrets, which the recipient is obliged to treat as confidential. The information in this e-mail is made available exclusively for use by the addressee. In the event that the e-mail may have been sent to you in error, we would ask you to kindly delete this communication from your system and to contact us. E-mails sent via the Internet can be easily manipulated or sent out under someone else's name. We therefore do not accept legal liability for the information contained in this communication. The contents of the e-mail are only legally binding if they have been confirmed and signed by us in writing. If, in spite of our using Antivirus protection software, a virus may have penetrated your system through the sending of this e-mail, we do not accept liability for any damage that may possibly arise as a result of this. We trust that you appreciate our position. ------------------------------------------- -----Ursprüngliche Nachricht----- > Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag von > ALICE BENNETT > Gesendet: Freitag, 07. Dezember 2007 15:59 > An: ids@iiug.org > Betreff: Tuning Checkpoints [10650] > > I need some help getting checkpoints durations lowered. We are running > informix 9.4. I have been working with Informix Tech support and it has > gotten > some better but it is not as good as it should be. We have been adjusting > BUFFERS, LRUS, CLEANERS, LRU_MAX_DIRTY, and LRU_MIN_DIRTY. > > BUFFERS = 350000 > LRUS = 128 > CLEANERS = 127 > LRU_MAX_DIRTY = .5 > LRU_MIN_DIRTY = .25> > After the last changes we made, LRU writes is now lower than chunk writes > for > the first time. > > Our checkpoints have ranged up to 7 seconds and sometimes higher. Anyone > have > any ideas of what else we can look at? > > HP GL380G5 servers > 32Gb Memory > (2) quad core Processors > The dbspaces are located on Compellent SAN fiber channel drives RAID level > 10 > (striped and mirrored). > > > ************************************************************************** > ***** > Forum Note: Use "Reply" to post a response in the discussion forum.

Hi, tuning KAIO on HPUX: I didn't know that there is much to tune, except to have enough async ports. Are there interesting OS parameters? Regards Andreas Kutsche > ------------------------------------------- SPAR Österreichische Warenhandels-AG Hauptzentrale A - 5015 Salzburg, Europastrasse 3 FN 34170 a Tel: +43 662 4470 24223 Mobile: +43 664 6259575 E-Mail: Andreas.KUTSCHE@spar.at Internet: http://www.spar.at Wichtiger Hinweis: Der Inhalt dieser E-Mail kann vertrauliche und rechtlich geschützte Informationen, insbesondere Betriebs- oder Geschäftsgeheimnisse, enthalten, zu deren Geheimhaltung der Empfänger verpflichtet ist. Die Informationen in dieser E-Mail sind ausschließlich für den Adressaten bestimmt. Sollten Sie die E-Mail irrtümlich erhalten haben so ersuchen wir Sie, die Nachricht von Ihrem System zu löschen und sich mit uns in Verbindung zu setzen. Über das Internet versandte E-Mails können leicht manipuliert oder unter fremdem Namen erstellt werden. Daher schließen wir die rechtliche Verbindlichkeit der in dieser Nachricht enthaltenen Informationen aus. Der Inhalt der E-Mail ist nur rechtsverbindlich, wenn er von uns schriftlich bestätigt und gezeichnet wird. Sollte trotz der von uns verwendeten Virus-Schutzprogramme durch die Zusendung von E-Mails ein Virus in Ihre Systeme gelangen, haften wir nicht für evtl. hieraus entstehende Schäden. Wir danken für Ihr Verständnis. Important notice: The contents of this e-mail may contain confidential and legally protected information that is in particular related to operational and trade secrets, which the recipient is obliged to treat as confidential. The information in this e-mail is made available exclusively for use by the addressee. In the event that the e-mail may have been sent to you in error, we would ask you to kindly delete this communication from your system and to contact us. E-mails sent via the Internet can be easily manipulated or sent out under someone else's name. We therefore do not accept legal liability for the information contained in this communication. The contents of the e-mail are only legally binding if they have been confirmed and signed by us in writing. If, in spite of our using Antivirus protection software, a virus may have penetrated your system through the sending of this e-mail, we do not accept liability for any damage that may possibly arise as a result of this. We trust that you appreciate our position. ------------------------------------------- -----Ursprüngliche Nachricht----- > Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag von Art > S. Kagel (Oninit LLC) > Gesendet: Freitag, 07. Dezember 2007 16:47 > An: ids@iiug.org > Betreff: Re: Tuning Checkpoints [10654] > > ALICE BENNETT wrote: > > I need some help getting checkpoints durations lowered. We are running > > informix 9.4. I have been working with Informix Tech support and it has > gotten > > some better but it is not as good as it should be. We have been > adjusting > > BUFFERS, LRUS, CLEANERS, LRU_MAX_DIRTY, and LRU_MIN_DIRTY. > > > > BUFFERS = 350000 > > LRUS = 128 > > CLEANERS = 127 > > LRU_MAX_DIRTY = .5 > > LRU_MIN_DIRTY = .25> > > > After the last changes we made, LRU writes is now lower than chunk > writes > for > > the first time. > > > > Our checkpoints have ranged up to 7 seconds and sometimes higher. Anyone > have > > any ideas of what else we can look at? > > > > Are chunks using KAIO (ie are they RAW device)? If so you may have to > tune the HPUX KAIO subsystem. Tuning KAIO on HPUX can make a huge > difference in write times like checkpoint. > > Art S. Kagel > > > HP GL380G5 servers > > 32Gb Memory > > (2) quad core Processors > > The dbspaces are located on Compellent SAN fiber channel drives RAID > level > 10 > > (striped and mirrored). > > > > > > > ************************************************************************** > ***** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > ************************************************************************** > ***** > Forum Note: Use "Reply" to post a response in the discussion forum.

Alice, I would not bet on it being an Informix issue. Have you engaged HP support as well as SAN support? Are you also by any chance doing disk-to-disk copies of the SAN to your DR/Hotsite? We had Informix 9 on Solaris with EMC disks.... all was fine with checkpoints until we turned on EMC SRDF to do disk level syncs of the DB to our disaster recovery site. Our Informix checkpoints shot up to double digits (20-30+ seconds).... In our case we were running SAP on Informix, and we had a lot of Informix buffer cache, and a lot of SAP buffer cache.... as the old story goes...a 3 second or less checkpoint might be the benchmark, but really your checkpoing duration is only bad if the customer screams. Our users didn't scream about our 30+ second checkpoints because more often than not their SAP answers were in SAP cache... can you share more information about your environment in general so the experts out here have a better handle on the 'bigger picture'. Thanks, Norma Jean

Hi Andreas, You've brought up an interesting point regarding "degenerated attached indices and insert statements blocking a checkpoint for over a minute". We're currently having the same problem and the recommendation I got from IBM Informix Support was to recreate the index as detached. Right now, there's no way we'll be able to do that as our users don't approve of a downtime and we're at IDS9.40 which doesn't have the facility to do online index rebuild(drop/create). Based on your experience, is there any other way we can reduce those long checkpoints? Thanks & Regards, Edcel > To: ids@iiug.org > From: andreas.kutsche@spar.at > Subject: AW: Tuning Checkpoints [10657] > Date: Fri, 7 Dec 2007 16:41:32 -0500 > > Hi, > > 128 LRUs with 350000 Buffers and LRU_MAX_DIRTY 0.5 should be far enough and > give you checkpoints between one and two seconds. > > Is your IO performance okay (KAIO an how many CPU VPs or when using normal IO > how many AIO VPs)? Does your SAN storage have a write cache (battery > buffered)? > > Maybe you may want to try lower LRU-numbers. SAP suggests LRU numbers = CPU > VPs. In most cases that seems to work, too. > > Did you monitor your checkpoints? Run onstat - in an endless loop and as long > as it shows a checkpoint get the dirty buffers (onstat -R) and the threads in > critical sections (onstat -u | grep X) - maybe you don't have a problem with > the dirty buffers but with sessions in critical sections which block the > checkpoint. > We had some problems with (degenerated) attached indices and insert statements > - sometimes they blocked a checkpoint for over a minute. > > Regards, > Andreas Kutsche > > > > ------------------------------------------- > SPAR Österreichische Warenhandels-AG > Hauptzentrale > A - 5015 Salzburg, Europastrasse 3 > FN 34170 a > > Tel: +43 662 4470 24223 > Mobile: +43 664 6259575 > E-Mail: Andreas.KUTSCHE@spar.at > Internet: http://www.spar.at > > Wichtiger Hinweis: Der Inhalt dieser E-Mail kann vertrauliche und rechtlich > geschützte Informationen, insbesondere Betriebs- oder Geschäftsgeheimnisse, > enthalten, zu deren Geheimhaltung der Empfänger verpflichtet ist. Die > Informationen in dieser E-Mail sind ausschließlich für den Adressaten > bestimmt. Sollten Sie die E-Mail irrtümlich erhalten haben so ersuchen wir > Sie, die Nachricht von Ihrem System zu löschen und sich mit uns in Verbindung > zu setzen. > Über das Internet versandte E-Mails können leicht manipuliert oder unter > fremdem Namen erstellt werden. Daher schließen wir die rechtliche > Verbindlichkeit der in dieser Nachricht enthaltenen Informationen aus. Der > Inhalt der E-Mail ist nur rechtsverbindlich, wenn er von uns schriftlich > bestätigt und gezeichnet wird. > Sollte trotz der von uns verwendeten Virus-Schutzprogramme durch die Zusendung > von E-Mails ein Virus in Ihre Systeme gelangen, haften wir nicht für evtl. > hieraus entstehende Schäden. > Wir danken für Ihr Verständnis. > > Important notice: The contents of this e-mail may contain confidential and > legally protected information that is in particular related to operational and > trade secrets, which the recipient is obliged to treat as confidential. The > information in this e-mail is made available exclusively for use by the > addressee. In the event that the e-mail may have been sent to you in error, we > would ask you to kindly delete this communication from your system and to > contact us. > E-mails sent via the Internet can be easily manipulated or sent out under > someone else's name. We therefore do not accept legal liability for the > information contained in this communication. The contents of the e-mail are > only legally binding if they have been confirmed and signed by us in writing. > If, in spite of our using Antivirus protection software, a virus may have > penetrated your system through the sending of this e-mail, we do not accept > liability for any damage that may possibly arise as a result of this. > We trust that you appreciate our position. > > ------------------------------------------- > -----Ursprüngliche Nachricht----- > > > Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag von > > ALICE BENNETT > > Gesendet: Freitag, 07. Dezember 2007 15:59 > > An: ids@iiug.org > > Betreff: Tuning Checkpoints [10650] > > > > I need some help getting checkpoints durations lowered. We are running > > informix 9.4. I have been working with Informix Tech support and it has > > gotten > > some better but it is not as good as it should be. We have been adjusting > > BUFFERS, LRUS, CLEANERS, LRU_MAX_DIRTY, and LRU_MIN_DIRTY. > > > > BUFFERS = 350000 > > LRUS = 128 > > CLEANERS = 127 > > LRU_MAX_DIRTY = .5 > > LRU_MIN_DIRTY = .25> > > > After the last changes we made, LRU writes is now lower than chunk writes > > for > > the first time. > > > > Our checkpoints have ranged up to 7 seconds and sometimes higher. Anyone > > have > > any ideas of what else we can look at? > > > > HP GL380G5 servers > > 32Gb Memory > > (2) quad core Processors > > The dbspaces are located on Compellent SAN fiber channel drives RAID level > > 10 > > (striped and mirrored). > > > > > > ************************************************************************** > > ***** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >

Hi Edcel, you could check your btree scanner configuration - maybe you have not enough btree scanners running (or they run on some of the attached indices in a kind of an endless loop doing only compressions). Or your threshold value is too high to get some of the tables cleaned. How big is your system (CPUs / BUFFERS)? The default values from 1 to 3 scanners are very low and not suitable for a big production system. I think in our case the start of more scanners and monitoring of the process (restarting scanners when doing only compressions) did help a little. But the solution that really helped was recreating the indices (or even reorganizing the whole table, because dropping attached indices can sometimes take more or the same time than copying the whole table). Regards Andreas > ------------------------------------------- SPAR Österreichische Warenhandels-AG Hauptzentrale A - 5015 Salzburg, Europastrasse 3 FN 34170 a Tel: +43 662 4470 24223 Mobile: +43 664 6259575 E-Mail: Andreas.KUTSCHE@spar.at Internet: http://www.spar.at Wichtiger Hinweis: Der Inhalt dieser E-Mail kann vertrauliche und rechtlich geschützte Informationen, insbesondere Betriebs- oder Geschäftsgeheimnisse, enthalten, zu deren Geheimhaltung der Empfänger verpflichtet ist. Die Informationen in dieser E-Mail sind ausschließlich für den Adressaten bestimmt. Sollten Sie die E-Mail irrtümlich erhalten haben so ersuchen wir Sie, die Nachricht von Ihrem System zu löschen und sich mit uns in Verbindung zu setzen. Über das Internet versandte E-Mails können leicht manipuliert oder unter fremdem Namen erstellt werden. Daher schließen wir die rechtliche Verbindlichkeit der in dieser Nachricht enthaltenen Informationen aus. Der Inhalt der E-Mail ist nur rechtsverbindlich, wenn er von uns schriftlich bestätigt und gezeichnet wird. Sollte trotz der von uns verwendeten Virus-Schutzprogramme durch die Zusendung von E-Mails ein Virus in Ihre Systeme gelangen, haften wir nicht für evtl. hieraus entstehende Schäden. Wir danken für Ihr Verständnis. Important notice: The contents of this e-mail may contain confidential and legally protected information that is in particular related to operational and trade secrets, which the recipient is obliged to treat as confidential. The information in this e-mail is made available exclusively for use by the addressee. In the event that the e-mail may have been sent to you in error, we would ask you to kindly delete this communication from your system and to contact us. E-mails sent via the Internet can be easily manipulated or sent out under someone else's name. We therefore do not accept legal liability for the information contained in this communication. The contents of the e-mail are only legally binding if they have been confirmed and signed by us in writing. If, in spite of our using Antivirus protection software, a virus may have penetrated your system through the sending of this e-mail, we do not accept liability for any damage that may possibly arise as a result of this. We trust that you appreciate our position. ------------------------------------------- -----Ursprüngliche Nachricht----- > Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag von > Edcel Barcena > Gesendet: Montag, 10. Dezember 2007 02:45 > An: ids@iiug.org > Betreff: RE: AW: Tuning Checkpoints [10662] > > Hi Andreas, > > You've brought up an interesting point regarding "degenerated attached > indices > and insert statements blocking a checkpoint for over a minute". We're > currently having the same problem and the recommendation I got from IBM > Informix Support was to recreate the index as detached. > > Right now, there's no way we'll be able to do that as our users don't > approve > of a downtime and we're at IDS9.40 which doesn't have the facility to do > online index rebuild(drop/create). Based on your experience, is there any > other way we can reduce those long checkpoints? > > Thanks & Regards, > Edcel > > > To: ids@iiug.org > > From: andreas.kutsche@spar.at > > Subject: AW: Tuning Checkpoints [10657] > > Date: Fri, 7 Dec 2007 16:41:32 -0500 > > > > Hi, > > > > 128 LRUs with 350000 Buffers and LRU_MAX_DIRTY 0.5 should be far enough > and > > give you checkpoints between one and two seconds. > > > > Is your IO performance okay (KAIO an how many CPU VPs or when using > normal > IO > > how many AIO VPs)? Does your SAN storage have a write cache (battery > > buffered)? > > > > Maybe you may want to try lower LRU-numbers. SAP suggests LRU numbers = > CPU > > VPs. In most cases that seems to work, too. > > > > Did you monitor your checkpoints? Run onstat - in an endless loop and as > long > > as it shows a checkpoint get the dirty buffers (onstat -R) and the > threads > in > > critical sections (onstat -u | grep X) - maybe you don't have a problem > with > > the dirty buffers but with sessions in critical sections which block the > > checkpoint. > > We had some problems with (degenerated) attached indices and insert > statements > > - sometimes they blocked a checkpoint for over a minute. > > > > Regards, > > Andreas Kutsche > > > > > > > ------------------------------------------- > > SPAR Österreichische Warenhandels-AG > > Hauptzentrale > > A - 5015 Salzburg, Europastrasse 3 > > FN 34170 a > > > > Tel: +43 662 4470 24223 > > Mobile: +43 664 6259575 > > E-Mail: Andreas.KUTSCHE@spar.at > > Internet: http://www.spar.at > > > > Wichtiger Hinweis: Der Inhalt dieser E-Mail kann vertrauliche und > rechtlich > > geschützte Informationen, insbesondere Betriebs- oder > Geschäftsgeheimnisse, > > enthalten, zu deren Geheimhaltung der Empfänger verpflichtet ist. Die > > Informationen in dieser E-Mail sind ausschließlich für den Adressaten > > bestimmt. Sollten Sie die E-Mail irrtümlich erhalten haben so ersuchen > wir > > Sie, die Nachricht von Ihrem System zu löschen und sich mit uns in > Verbindung > > zu setzen. > > Über das Internet versandte E-Mails können leicht manipuliert oder unter > > fremdem Namen erstellt werden. Daher schließen wir die rechtliche > > Verbindlichkeit der in dieser Nachricht enthaltenen Informationen aus. > Der > > Inhalt der E-Mail ist nur rechtsverbindlich, wenn er von uns schriftlich > > bestätigt und gezeichnet wird. > > Sollte trotz der von uns verwendeten Virus-Schutzprogramme durch die > Zusendung > > von E-Mails ein Virus in Ihre Systeme gelangen, haften wir nicht für > evtl. > > hieraus entstehende Schäden. > > Wir danken für Ihr Verständnis. > > > > Important notice: The contents of this e-mail may contain confidential > and > > legally protected information that is in particular related to > operational > and > > trade secrets, which the recipient is obliged to treat as confidential. > The > > information in this e-mail is made available exclusively for use by the > > addressee. In the event that the e-mail may have been sent to you in > error, > we > > would ask you to kindly delete this communication from your system and > to > > contact us. > > E-mails sent vi

Over the weekend, I moved the indexes for the two tables with the most io to separate dbspaces. I also changed the check point interval to 30 seconds. I believe we are seeing some improvement but we are still seeing some slow downs and some large checkpoints. I forgot to mention in my original post that we are using HDR for replication to another server.

ALICE BENNETT wrote: > Over the weekend, I moved the indexes for the two tables with the most io to > separate dbspaces. I also changed the check point interval to 30 seconds. I > believe we are seeing some improvement but we are still seeing some slow downs > and some large checkpoints. > > I forgot to mention in my original post that we are using HDR for replication > to another server. > Hmm, I don't remember, are your logical logs each rather large? Because the log containing the new checkpoint record has to be sent to the HDR secondary and the primary will wait for the secondary to acknowledge that it has caught up to the checkpoint before it releases the checkpoint state. For its part, the secondary has to complete the roll-forward of all outstanding logical log records pushed to it up to the checkpoint record before it will acknowledge the checkpoint to the primary. There's a fair chance that this is at least part of the problem. Is the secondary slower than the primary? Is the WAN/LAN connection between them slower than disk channels? Is the secondary tuned properly? Art S. Kagel > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > >

What is the value of PHYBUFF LOGBUFF We had the same problem out LOGBUFF was 8M which caused checkpoints to go high, here is the test you can do turn off HDR for like 30 minutes and re-start it again, check how the checkpoints are when you have HDR turned off Thanks You can't build a reputation on what you are going to do." - Henry Ford (1863-1947) American industrialist, inventor -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of ALICE BENNETT Sent: Monday, December 10, 2007 11:59 AM To: ids@iiug.org Subject: Re: RE: AW: Tuning Checkpoints [10671] Over the weekend, I moved the indexes for the two tables with the most io to separate dbspaces. I also changed the check point interval to 30 seconds. I believe we are seeing some improvement but we are still seeing some slow downs and some large checkpoints. I forgot to mention in my original post that we are using HDR for replication to another server. ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum.

Hi, is there a special reason why you want to trigger a checkpoint every 30 seconds? (okay, recovery time after an instance crash will be short - but do you really want to have a lot of checkpoints/short_time_stops just to avoid a little longer downtime in case of an rare instance crash?). I would set the checkpoint interval to several minutes or even hours. Maybe you will have to adjust your PHYSFILE, too (a 2/3 full PHYSFILE will trigger a checkpoint). The work done (dirty buffers) is controlled by the LRU parameters. The longer checkpoint intervals won't give you shorter checkpoints, but they are less frequent and therefore users will suffer less. Regards, Andreas Kutsche > ------------------------------------------- SPAR Österreichische Warenhandels-AG Hauptzentrale A - 5015 Salzburg, Europastrasse 3 FN 34170 a Tel: +43 662 4470 24223 Mobile: +43 664 6259575 E-Mail: Andreas.KUTSCHE@spar.at Internet: http://www.spar.at Wichtiger Hinweis: Der Inhalt dieser E-Mail kann vertrauliche und rechtlich geschützte Informationen, insbesondere Betriebs- oder Geschäftsgeheimnisse, enthalten, zu deren Geheimhaltung der Empfänger verpflichtet ist. Die Informationen in dieser E-Mail sind ausschließlich für den Adressaten bestimmt. Sollten Sie die E-Mail irrtümlich erhalten haben so ersuchen wir Sie, die Nachricht von Ihrem System zu löschen und sich mit uns in Verbindung zu setzen. Über das Internet versandte E-Mails können leicht manipuliert oder unter fremdem Namen erstellt werden. Daher schließen wir die rechtliche Verbindlichkeit der in dieser Nachricht enthaltenen Informationen aus. Der Inhalt der E-Mail ist nur rechtsverbindlich, wenn er von uns schriftlich bestätigt und gezeichnet wird. Sollte trotz der von uns verwendeten Virus-Schutzprogramme durch die Zusendung von E-Mails ein Virus in Ihre Systeme gelangen, haften wir nicht für evtl. hieraus entstehende Schäden. Wir danken für Ihr Verständnis. Important notice: The contents of this e-mail may contain confidential and legally protected information that is in particular related to operational and trade secrets, which the recipient is obliged to treat as confidential. The information in this e-mail is made available exclusively for use by the addressee. In the event that the e-mail may have been sent to you in error, we would ask you to kindly delete this communication from your system and to contact us. E-mails sent via the Internet can be easily manipulated or sent out under someone else's name. We therefore do not accept legal liability for the information contained in this communication. The contents of the e-mail are only legally binding if they have been confirmed and signed by us in writing. If, in spite of our using Antivirus protection software, a virus may have penetrated your system through the sending of this e-mail, we do not accept liability for any damage that may possibly arise as a result of this. We trust that you appreciate our position. ------------------------------------------- -----Ursprüngliche Nachricht----- > Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag von > ALICE BENNETT > Gesendet: Montag, 10. Dezember 2007 20:59 > An: ids@iiug.org > Betreff: Re: RE: AW: Tuning Checkpoints [10671] > > Over the weekend, I moved the indexes for the two tables with the most io > to > separate dbspaces. I also changed the check point interval to 30 seconds. > I > believe we are seeing some improvement but we are still seeing some slow > downs > and some large checkpoints. > > I forgot to mention in my original post that we are using HDR for > replication > to another server. > > > ************************************************************************** > ***** > Forum Note: Use "Reply" to post a response in the discussion forum.

Hi Art, I hope your remarks about HDR and checkpoints are valid only if DRINTERVAL is set to -1 (synchronous update) ? Otherwise I can't imagine how HDR can be used for a large high performance production system? And what happens if the secondary or the network connections become unavailable for some time? Regards, Andreas Kutsche > ------------------------------------------- SPAR Österreichische Warenhandels-AG Hauptzentrale A - 5015 Salzburg, Europastrasse 3 FN 34170 a Tel: +43 662 4470 24223 Mobile: +43 664 6259575 E-Mail: Andreas.KUTSCHE@spar.at Internet: http://www.spar.at Wichtiger Hinweis: Der Inhalt dieser E-Mail kann vertrauliche und rechtlich geschützte Informationen, insbesondere Betriebs- oder Geschäftsgeheimnisse, enthalten, zu deren Geheimhaltung der Empfänger verpflichtet ist. Die Informationen in dieser E-Mail sind ausschließlich für den Adressaten bestimmt. Sollten Sie die E-Mail irrtümlich erhalten haben so ersuchen wir Sie, die Nachricht von Ihrem System zu löschen und sich mit uns in Verbindung zu setzen. Über das Internet versandte E-Mails können leicht manipuliert oder unter fremdem Namen erstellt werden. Daher schließen wir die rechtliche Verbindlichkeit der in dieser Nachricht enthaltenen Informationen aus. Der Inhalt der E-Mail ist nur rechtsverbindlich, wenn er von uns schriftlich bestätigt und gezeichnet wird. Sollte trotz der von uns verwendeten Virus-Schutzprogramme durch die Zusendung von E-Mails ein Virus in Ihre Systeme gelangen, haften wir nicht für evtl. hieraus entstehende Schäden. Wir danken für Ihr Verständnis. Important notice: The contents of this e-mail may contain confidential and legally protected information that is in particular related to operational and trade secrets, which the recipient is obliged to treat as confidential. The information in this e-mail is made available exclusively for use by the addressee. In the event that the e-mail may have been sent to you in error, we would ask you to kindly delete this communication from your system and to contact us. E-mails sent via the Internet can be easily manipulated or sent out under someone else's name. We therefore do not accept legal liability for the information contained in this communication. The contents of the e-mail are only legally binding if they have been confirmed and signed by us in writing. If, in spite of our using Antivirus protection software, a virus may have penetrated your system through the sending of this e-mail, we do not accept liability for any damage that may possibly arise as a result of this. We trust that you appreciate our position. ------------------------------------------- -----Ursprüngliche Nachricht----- > Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag von Art > S. Kagel (Oninit LLC) > Gesendet: Montag, 10. Dezember 2007 21:10 > An: ids@iiug.org > Betreff: Re: AW: Tuning Checkpoints [10672] > > ALICE BENNETT wrote: > > Over the weekend, I moved the indexes for the two tables with the most > io to > > separate dbspaces. I also changed the check point interval to 30 > seconds. I > > believe we are seeing some improvement but we are still seeing some slow > downs > > and some large checkpoints. > > > > I forgot to mention in my original post that we are using HDR for > replication > > to another server. > > > > Hmm, I don't remember, are your logical logs each rather large? Because > the log containing the new checkpoint record has to be sent to the HDR > secondary and the primary will wait for the secondary to acknowledge > that it has caught up to the checkpoint before it releases the > checkpoint state. For its part, the secondary has to complete the > roll-forward of all outstanding logical log records pushed to it up to > the checkpoint record before it will acknowledge the checkpoint to the > primary. > > There's a fair chance that this is at least part of the problem. Is the > secondary slower than the primary? Is the WAN/LAN connection between > them slower than disk channels? Is the secondary tuned properly? > > Art S. Kagel > > > > > ************************************************************************** > ***** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > ************************************************************************** > ***** > Forum Note: Use "Reply" to post a response in the discussion forum.

Hi Art, you wrote > > Are chunks using KAIO (ie are they RAW device)? If so you may have to > tune the HPUX KAIO subsystem. Tuning KAIO on HPUX can make a huge > difference in write times like checkpoint. > We are on HPUX with KAIO, too. But we did never special tuning for KAIO. Which kernel parameters should be tuned for a high performance system (SAN storage)? The systems with the most IO run on HPUX 11.23/11.31 IA64 , but suggestions for HPUX 11.11 PA-Risc would be appreciated, too. Regards Andreas Kutsche > Art S. Kagel > > > HP GL380G5 servers > > 32Gb Memory > > (2) quad core Processors > > The dbspaces are located on Compellent SAN fiber channel drives RAID > level > 10 > > (striped and mirrored). > > > > > > > ************************************************************************ ** > ***** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > ************************************************************************ ** > ***** > Forum Note: Use "Reply" to post a response in the discussion forum. ------------------------------------------- SPAR Osterreichische Warenhandels-AG Hauptzentrale A - 5015 Salzburg, Europastrasse 3 FN 34170 a Tel: +43 662 4470 24223 Mobile: +43 664 6259575 E-Mail: Andreas.KUTSCHE@spar.at Internet: http://www.spar.at Wichtiger Hinweis: Der Inhalt dieser E-Mail kann vertrauliche und rechtlich geschutzte Informationen, insbesondere Betriebs- oder Geschaftsgeheimnisse, enthalten, zu deren Geheimhaltung der Empfanger verpflichtet ist. Die Informationen in dieser E-Mail sind ausschlie?lich fur den Adressaten bestimmt. Sollten Sie die E-Mail irrtumlich erhalten haben so ersuchen wir Sie, die Nachricht von Ihrem System zu loschen und sich mit uns in Verbindung zu setzen. Uber das Internet versandte E-Mails konnen leicht manipuliert oder unter fremdem Namen erstellt werden. Daher schlie?en wir die rechtliche Verbindlichkeit der in dieser Nachricht enthaltenen Informationen aus. Der Inhalt der E-Mail ist nur rechtsverbindlich, wenn er von uns schriftlich bestatigt und gezeichnet wird. Sollte trotz der von uns verwendeten Virus-Schutzprogramme durch die Zusendung von E-Mails ein Virus in Ihre Systeme gelangen, haften wir nicht fur evtl. hieraus entstehende Schaden. Wir danken fur Ihr Verstandnis. Important notice: The contents of this e-mail may contain confidential and legally protected information that is in particular related to operational and trade secrets, which the recipient is obliged to treat as confidential. The information in this e-mail is made available exclusively for use by the addressee. In the event that the e-mail may have been sent to you in error, we would ask you to kindly delete this communication from your system and to contact us. E-mails sent via the Internet can be easily manipulated or sent out under someone else's name. We therefore do not accept legal liability for the information contained in this communication. The contents of the e-mail are only legally binding if they have been confirmed and signed by us in writing. If, in spite of our using Antivirus protection software, a virus may have penetrated your system through the sending of this e-mail, we do not accept liability for any damage that may possibly arise as a result of this. We trust that you appreciate our position. -------------------------------------------

We do not ship log files with HDR, we ship log buffers. ------------------------------------- Madison Pruet, STSM IDS Replication Architect = "Andreas.KUTSCHE@ = spar.at" = <andreas.kutsche@ = To spar.at> ids@iiug.org = Sent by: = cc ids-bounces@iiug. = org Subj= ect AW: AW: Tuning Checkpoints [106= 79] = 12/11/2007 02:40 = AM = = = Please respond to = ids@iiug.org = = = Hi Art, I hope your remarks about HDR and checkpoints are valid only if DRINTER= VAL is set to -1 (synchronous update) ? Otherwise I can't imagine how HDR can = be used for a large high performance production system? And what happens if the= secondary or the network connections become unavailable for some time? Regards, Andreas Kutsche > ------------------------------------------- SPAR =D6sterreichische Warenhandels-AG Hauptzentrale A - 5015 Salzburg, Europastrasse 3 FN 34170 a Tel: +43 662 4470 24223 Mobile: +43 664 6259575 E-Mail: Andreas.KUTSCHE@spar.at Internet: http://www.spar.at Wichtiger Hinweis: Der Inhalt dieser E-Mail kann vertrauliche und recht= lich gesch=FCtzte Informationen, insbesondere Betriebs- oder Gesch=E4ftsgehe= imnisse, enthalten, zu deren Geheimhaltung der Empf=E4nger verpflichtet ist. Die= Informationen in dieser E-Mail sind ausschlie=DFlich f=FCr den Adressat= en bestimmt. Sollten Sie die E-Mail irrt=FCmlich erhalten haben so ersuche= n wir Sie, die Nachricht von Ihrem System zu l=F6schen und sich mit uns in Verbindung zu setzen. =DCber das Internet versandte E-Mails k=F6nnen leicht manipuliert oder = unter fremdem Namen erstellt werden. Daher schlie=DFen wir die rechtliche Verbindlichkeit der in dieser Nachricht enthaltenen Informationen aus. = Der Inhalt der E-Mail ist nur rechtsverbindlich, wenn er von uns schriftlic= h best=E4tigt und gezeichnet wird. Sollte trotz der von uns verwendeten Virus-Schutzprogramme durch die Zusendung von E-Mails ein Virus in Ihre Systeme gelangen, haften wir nicht f=FCr = evtl. hieraus entstehende Sch=E4den. Wir danken f=FCr Ihr Verst=E4ndnis. Important notice: The contents of this e-mail may contain confidential = and legally protected information that is in particular related to operatio= nal and trade secrets, which the recipient is obliged to treat as confidential.= The information in this e-mail is made available exclusively for use by the= addressee. In the event that the e-mail may have been sent to you in er= ror, we would ask you to kindly delete this communication from your system and = to contact us. E-mails sent via the Internet can be easily manipulated or sent out und= er someone else's name. We therefore do not accept legal liability for the= information contained in this communication. The contents of the e-mail= are only legally binding if they have been confirmed and signed by us in writing. If, in spite of our using Antivirus protection software, a virus may ha= ve penetrated your system through the sending of this e-mail, we do not ac= cept liability for any damage that may possibly arise as a result of this. We trust that you appreciate our position. ------------------------------------------- -----Urspr=FCngliche Nachricht----- > Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag vo= n Art > S. Kagel (Oninit LLC) > Gesendet: Montag, 10. Dezember 2007 21:10 > An: ids@iiug.org > Betreff: Re: AW: Tuning Checkpoints [10672] > > ALICE BENNETT wrote: > > Over the weekend, I moved the indexes for the two tables with the m= ost > io to > > separate dbspaces. I also changed the check point interval to 30 > seconds. I > > believe we are seeing some improvement but we are still seeing some= slow > downs > > and some large checkpoints. > > > > I forgot to mention in my original post that we are using HDR for > replication > > to another server. > > > > Hmm, I don't remember, are your logical logs each rather large? Becau= se > the log containing the new checkpoint record has to be sent to the HD= R > secondary and the primary will wait for the secondary to acknowledge > that it has caught up to the checkpoint before it releases the > checkpoint state. For its part, the secondary has to complete the > roll-forward of all outstanding logical log records pushed to it up t= o > the checkpoint record before it will acknowledge the checkpoint to th= e > primary. > > There's a fair chance that this is at least part of the problem. Is t= he > secondary slower than the primary? Is the WAN/LAN connection between > them slower than disk channels? Is the secondary tuned properly? > > Art S. Kagel > > > > > ***********************************************************************= *** > ***** > > Forum Note: Use "Reply" to post a response in the discussion forum.= > > > > > > > ***********************************************************************= *** > ***** > Forum Note: Use "Reply" to post a response in the discussion forum. ***********************************************************************= ******** Forum Note: Use "Reply" to post a response in the discussion forum. =

Andreas.KUTSCHE@spar.at wrote: > Hi Art, > > I hope your remarks about HDR and checkpoints are valid only if DRINTERVAL is > set to -1 (synchronous update) ? Otherwise I can't imagine how HDR can be used > for a large high performance production system? And what happens if the > secondary or the network connections become unavailable for some time? > <Warning strong opinion zone ;-)> Yes, that would be for synchronous HDR with DRINTERVAL set to -1. Sorry, I tend to think only in terms of what I view as best practice. The possible loss of transactions that are assume to be committed by the primary server and therefore by clients software and users, I think, restricts practical HDR to synchronous only. Art S. Kagel > Regards, > Andreas Kutsche > > > <SNIP> >> Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag von Art >> S. Kagel (Oninit LLC) >> Gesendet: Montag, 10. Dezember 2007 21:10 >> An: ids@iiug.org >> Betreff: Re: AW: Tuning Checkpoints [10672] >> >> ALICE BENNETT wrote: >> >>> Over the weekend, I moved the indexes for the two tables with the most >>> >> io to >> >>> separate dbspaces. I also changed the check point interval to 30 >>> >> seconds. I >> >>> believe we are seeing some improvement but we are still seeing some slow >>> >> downs >> >>> and some large checkpoints. >>> >>> I forgot to mention in my original post that we are using HDR for >>> >> replication >> >>> to another server. >>> >>> >> Hmm, I don't remember, are your logical logs each rather large? Because >> the log containing the new checkpoint record has to be sent to the HDR >> secondary and the primary will wait for the secondary to acknowledge >> that it has caught up to the checkpoint before it releases the >> checkpoint state. For its part, the secondary has to complete the >> roll-forward of all outstanding logical log records pushed to it up to >> the checkpoint record before it will acknowledge the checkpoint to the >> primary. >> >> There's a fair chance that this is at least part of the problem. Is the >> secondary slower than the primary? Is the WAN/LAN connection between >> them slower than disk channels? Is the secondary tuned properly? >> >> Art S. Kagel >> >>> >> ************************************************************************** >> ***** >> >>> Forum Note: Use "Reply" to post a response in the discussion forum. >>> >>> >>> >> ************************************************************************** >> ***** >> Forum Note: Use "Reply" to post a response in the discussion forum. >> > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > >

Andreas.KUTSCHE@spar.at wrote: > Hi Art, > > you wrote > > >> Are chunks using KAIO (ie are they RAW device)? If so you may have to >> tune the HPUX KAIO subsystem. Tuning KAIO on HPUX can make a huge >> difference in write times like checkpoint. >> >> > > We are on HPUX with KAIO, too. But we did never special tuning for KAIO. > Which kernel parameters should be tuned for a high performance system > (SAN storage)? > > The systems with the most IO run on HPUX 11.23/11.31 IA64 , but > suggestions > for HPUX 11.11 PA-Risc would be appreciated, too. > I am far from an HPUX expert, but I'm going by postings I've seen in CDI to the effect that a poorly tuned HP KAIO subsystem is worse than not using KAIO. Search the CDI archives for postings, IB there was one earlier in the year. Art S. Kagel > Regards > Andreas Kutsche > > >> Art S. Kagel >> >> >>> HP GL380G5 servers >>> 32Gb Memory >>> (2) quad core Processors >>> The dbspaces are located on Compellent SAN fiber channel drives RAID >>> >> level >> 10 >> >>> (striped and mirrored). >>> >>> >>> >>> > ************************************************************************ > ** > >> ***** >> >>> Forum Note: Use "Reply" to post a response in the discussion forum. >>> >>>