Logical logs - push or pull?
Bill Glidden asked whether logical logs should be backed up by Informix's ALARMPROGRAM as each log fills (push) or by scheduled jobs from the CommVault Simpana storage manager (pull). The consensus was overwhelmingly for push/event-driven backup via ALARMPROGRAM (customising the supplied alarmprogram.sh), since it minimises the window in which a filled log exists only on the server and so maximises the roll-forward point after a crash. Caveats offered: ensure the storage manager target is fast (not a tape carousel) to avoid log backups falling behind, and consider backing up to local disk first and letting the storage manager move files to tape. One dissenting suggestion to delay until several logs accumulate was rebutted.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Good morning folks, We have implemented On-Bar with the CommVault Simpana storage manager (love it so far!). I'm wondering if there is any advantage either way to a) letting Informix backup the logical logs through ALARMPROGRAM (push) or b) scheduling periodic logical log backups through Simpana (pull)? I have my own opinion on this, but would like some other opinions so that I can state my case with backup :)So even if your thoughts are basic, and you feel like answering, go for it. Thanks!
I'm a HUGE fan of using the ALARMPROGRAM to archive the logs as they fill. Art Art S. Kagel, Principal Consultant Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Oct 3, 2013 at 8:01 AM, BILL GLIDDEN <bglidden1962@gmail.com> wrote: > Good morning folks, > > We have implemented On-Bar with the CommVault Simpana storage manager > (love it > so far!). I'm wondering if there is any advantage either way to a) letting > Informix backup the logical logs through ALARMPROGRAM (push) or b) > scheduling > periodic logical log backups through Simpana (pull)? > > I have my own opinion on this, but would like some other opinions so that I > can state my case with backup :)So even if your thoughts are basic, and you > feel like answering, go for it. > > Thanks! > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --089e0141a02c7afe5c04e7d523fc
PUSH. Without any doubt. Consider why you back up the logical logs....? Because the system may fail and you want to be able to restore it as close as possible to the crash time... If you let logical logs accumulate on the engine you fail on this basic purpose. This idea should also be considered as the main driver for the logical log size calculation (and I recently saw some post, blog article or discussion about having a task that will switch the logical log if it's there for more than a certain period - and this makes a lot of sense) All other arguments are fait-divers... Regards. On Thu, Oct 3, 2013 at 1:01 PM, BILL GLIDDEN <bglidden1962@gmail.com> wrote: > Good morning folks, > > We have implemented On-Bar with the CommVault Simpana storage manager > (love it > so far!). I'm wondering if there is any advantage either way to a) letting > Informix backup the logical logs through ALARMPROGRAM (push) or b) > scheduling > periodic logical log backups through Simpana (pull)? > > I have my own opinion on this, but would like some other opinions so that I > can state my case with backup :)So even if your thoughts are basic, and you > feel like answering, go for it. > > Thanks! > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --001a11c34b48fe4b6304e7d5b7b9
I think using the ALARMPROGRAM is a much better way to go if for no other reason that they get backed up as soon as they fill. If you schedule another source to grab them periodically, what happens when you get that crazy transaction that fills your logs in 2 seconds flat (yeah that is a stretch but stranger things have happened). Can you not set up a class on your storage manager strictly for the logical logs? You can go to disk and back them up to tape from there as well. Dan -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of BILL GLIDDEN Sent: Thursday, October 03, 2013 8:01 AM To: ids@iiug.org Subject: Logical logs - push or pull? [31586] Good morning folks, We have implemented On-Bar with the CommVault Simpana storage manager (love it so far!). I'm wondering if there is any advantage either way to a) letting Informix backup the logical logs through ALARMPROGRAM (push) or b) scheduling periodic logical log backups through Simpana (pull)? I have my own opinion on this, but would like some other opinions so that I can state my case with backup :)So even if your thoughts are basic, and you feel like answering, go for it. Thanks! ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.
Bill By pushing the logs you are sure they get backed up the instant they fill and so your roll-forward point is as recent as possible. If you pull the logs (depending on your interval between the pulls) you may not get a log backed up for some minutes after it fills. If you lose your machine in this period and have to restore then you will not have this log to use for roll-forward. Would not be an issue if the pull process could switch to the next log before pulling but I don't know if that is possible. Keith On 3 October 2013 13:01, BILL GLIDDEN <bglidden1962@gmail.com> wrote: > Good morning folks, > > We have implemented On-Bar with the CommVault Simpana storage manager > (love it > so far!). I'm wondering if there is any advantage either way to a) letting > Informix backup the logical logs through ALARMPROGRAM (push) or b) > scheduling > periodic logical log backups through Simpana (pull)? > > I have my own opinion on this, but would like some other opinions so that I > can state my case with backup :)So even if your thoughts are basic, and you > feel like answering, go for it. > > Thanks! > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a11332252dec53504e7d5cf0d
Thanks for the responses. Personally, I wouldn't do it any other way than to use ALARMPROGRAM and push the logs to the storage manager as they fill. But I'm the "new guy" here so it helps to have other opinions to back up my decisions. Thanks.
Hello. It is always the best practice to pull the logs, but make sure you have enough backup window in your SM/drives, so that your backups will not be hang in the line. This is great for dedicated drives, but if your SM is managing several databases/OSes backups, you might face some long periods of long backup lines. You can also modify your alarmprogram, so that Informix only starts to pull the logs after some amount of logs are not backed up. If you need help, please advice. Regards. Alexandre Marini IBM Informix Certified Professional v10 / v11.50 / v11.70 / v12.10 IBM Information Management Informix Technical Professional IBM Infosphere DataStage Technical Professional Informix Senior DBA - Orizon Brasil BRIUG website administrator Informix independent consultant > To: ids@iiug.org > From: smiley73@gmail.com > Subject: Re: Logical logs - push or pull? [31590] > Date: Thu, 3 Oct 2013 09:03:33 -0400 > > Bill > > By pushing the logs you are sure they get backed up the instant they fill > and > so your roll-forward point is as recent as possible. If you pull the logs > (depending on your interval between the pulls) you may not get a log backed > up for some minutes after it fills. If you lose your machine in this period > and > have to restore then you will not have this log to use for roll-forward. > > Would not be an issue if the pull process could switch to the next log > before > pulling but I don't know if that is possible. > > Keith > > On 3 October 2013 13:01, BILL GLIDDEN <bglidden1962@gmail.com> wrote: > > > Good morning folks, > > > > We have implemented On-Bar with the CommVault Simpana storage manager > > (love it > > so far!). I'm wondering if there is any advantage either way to a) letting > > Informix backup the logical logs through ALARMPROGRAM (push) or b) > > scheduling > > periodic logical log backups through Simpana (pull)? > > > > I have my own opinion on this, but would like some other opinions so that I > > can state my case with backup :)So even if your thoughts are basic, and you > > feel like answering, go for it. > > > > Thanks! > > > > > > > > > ******************************************************************************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > --001a11332252dec53504e7d5cf0d > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >
Put it this way. If there is a fire in your house, would you rather your alarm go off, or do you want to call someone and ask them if smoke is coming out of your house? Sent from my iPad > On Oct 3, 2013, at 7:21 AM, "BILL GLIDDEN" <bglidden1962@gmail.com> wrote: > > Thanks for the responses. Personally, I wouldn't do it any other way than to > use ALARMPROGRAM and push the logs to the storage manager as they fill. But > I'm the "new guy" here so it helps to have other opinions to back up my > decisions. Thanks. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >
The only thing to be careful of when you do PUSH log backups with the
ALARMPROGRAM using onbar is the latency in the storage manager. Make sure
that the logical logs are going to FAST storage with no latency of its own
(ie not a tape carousel). I have seen onbar log backups fall several log
files behind during peak load because of this where ontape to disk would
have kept up. However, even then it is no worse than log PULL using
scheduled onbar runs through the storage manager's scheduler.
Art
Art S. Kagel, Principal Consultant
Advanced DataTools (www.advancedatatools.com)
Blog: http://informix-myview.blogspot.com/
Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on my employer, Advanced DataTools, the IIUG, nor any
other organization with which I am associated either explicitly,
implicitly, or by inference. Neither do those opinions reflect those of
other individuals affiliated with any entity with which I am affiliated nor
those of the entities themselves.
On Thu, Oct 3, 2013 at 8:56 AM, Fernando Nunes <domusonline@gmail.com>wrote:
> PUSH. Without any doubt. Consider why you back up the logical logs....?
> Because the system may fail and you want to be able to restore it as close
> as possible to the crash time... If you let logical logs accumulate on the
> engine you fail on this basic purpose. This idea should also be considered
> as the main driver for the logical log size calculation (and I recently saw
> some post, blog article or discussion about having a task that will switch
> the logical log if it's there for more than a certain period - and this
> makes a lot of sense)
> All other arguments are fait-divers...
>
> Regards.
>
> On Thu, Oct 3, 2013 at 1:01 PM, BILL GLIDDEN <bglidden1962@gmail.com>
> wrote:
>
> > Good morning folks,
> >
> > We have implemented On-Bar with the CommVault Simpana storage manager
> > (love it
> > so far!). I'm wondering if there is any advantage either way to a)
> letting
> > Informix backup the logical logs through ALARMPROGRAM (push) or b)
> > scheduling
> > periodic logical log backups through Simpana (pull)?
> >
> > I have my own opinion on this, but would like some other opinions so
> that I
> > can state my case with backup :)So even if your thoughts are basic, and
> you
> > feel like answering, go for it.
> >
> > Thanks!
> >
> >
> >
> >
>
>
*******************************************************************************
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
>
> --
> Fernando Nunes
> Portugal
>
> http://informix-technology.blogspot.com
> My email works... but I don't check it frequently...
>
> --001a11c34b48fe4b6304e7d5b7b9
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--089e013a132a86c61004e7d6aa20
Um, Andre, what you describe would be a PUSH since the ALARMPROGRAM is pushing them out. I would disagree about delaying until there are several logs outstanding. The whole point of backing up the logs, as alluded to by others, is to make sure you can restore to the last transaction possible if you lose the entire server machine. Can't do that if several logs are only on disk in the server and not yet backed up. Art Art S. Kagel, Principal Consultant Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Oct 3, 2013 at 9:24 AM, Alexandre Marini <alexandre@briug.org>wrote: > Hello. > It is always the best practice to pull the logs, but make sure you have > enough > backup window in your SM/drives, so that your backups will not be hang in > the > line. > > This is great for dedicated drives, but if your SM is managing several > databases/OSes backups, you might face some long periods of long backup > lines. > > You can also modify your alarmprogram, so that Informix only starts to pull > the logs after some amount of logs are not backed up. If you need help, > please > advice. > > Regards. > > Alexandre Marini > IBM Informix Certified Professional v10 / v11.50 / v11.70 / v12.10 > > IBM Information Management Informix Technical Professional > > IBM Infosphere DataStage Technical Professional > Informix Senior DBA - Orizon Brasil > BRIUG website administrator > Informix independent consultant > > > To: ids@iiug.org > > From: smiley73@gmail.com > > Subject: Re: Logical logs - push or pull? [31590] > > Date: Thu, 3 Oct 2013 09:03:33 -0400 > > > > Bill > > > > By pushing the logs you are sure they get backed up the instant they fill > > and > > so your roll-forward point is as recent as possible. If you pull the logs > > (depending on your interval between the pulls) you may not get a log > backed > > up for some minutes after it fills. If you lose your machine in this > period > > and > > have to restore then you will not have this log to use for roll-forward. > > > > Would not be an issue if the pull process could switch to the next log > > before > > pulling but I don't know if that is possible. > > > > Keith > > > > On 3 October 2013 13:01, BILL GLIDDEN <bglidden1962@gmail.com> wrote: > > > > > Good morning folks, > > > > > > We have implemented On-Bar with the CommVault Simpana storage manager > > > (love it > > > so far!). I'm wondering if there is any advantage either way to a) > letting > > > Informix backup the logical logs through ALARMPROGRAM (push) or b) > > > scheduling > > > periodic logical log backups through Simpana (pull)? > > > > > > I have my own opinion on this, but would like some other opinions so > that > I > > > can state my case with backup :)So even if your thoughts are basic, and > you > > > feel like answering, go for it. > > > > > > Thanks! > > > > > > > > > > > > > > > > ******************************************************************************* > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > > > > --001a11332252dec53504e7d5cf0d > > > > > > > > ******************************************************************************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --089e012946ba5f96df04e7d6b704
Instead of push/pull, it might be better to use the term event driven o= r probe driven. A push model is event driven. And in this case we are us= ing the IDS alarm handler as the event. A pull model would use a probing technique. By that we mean some external program probes to see if thing= s need to be done and then perform the operations if they need to be done= . Event driven models tend to have lower latency. Sent from my iPad > On Oct 3, 2013, at 7:59 AM, "Alexandre Marini" <alexandre@briug.org> wrote: > > Hello. > It is always the best practice to pull the logs, but make sure you ha= ve enough > backup window in your SM/drives, so that your backups will not be han= g in the > line. > > This is great for dedicated drives, but if your SM is managing severa= l > databases/OSes backups, you might face some long periods of long back= up lines. > > You can also modify your alarmprogram, so that Informix only starts t= o pull > the logs after some amount of logs are not backed up. If you need hel= p, please > advice. > > Regards. > > Alexandre Marini > IBM Informix Certified Professional v10 / v11.50 / v11.70 / v12.10 > > IBM Information Management Informix Technical Professional > > IBM Infosphere DataStage Technical Professional > Informix Senior DBA - Orizon Brasil > BRIUG website administrator > Informix independent consultant > > > To: ids@iiug.org > > From: smiley73@gmail.com > > Subject: Re: Logical logs - push or pull? [31590] > > Date: Thu, 3 Oct 2013 09:03:33 -0400 > > > > Bill > > > > By pushing the logs you are sure they get backed up the instant the= y fill > > and > > so your roll-forward point is as recent as possible. If you pull th= e logs > > (depending on your interval between the pulls) you may not get a lo= g backed > > up for some minutes after it fills. If you lose your machine in thi= s period > > and > > have to restore then you will not have this log to use for roll-forward. > > > > Would not be an issue if the pull process could switch to the next = log > > before > > pulling but I don't know if that is possible. > > > > Keith > > > > On 3 October 2013 13:01, BILL GLIDDEN <bglidden1962@gmail.com> wrot= e: > > > > > Good morning folks, > > > > > > We have implemented On-Bar with the CommVault Simpana storage man= ager > > > (love it > > > so far!). I'm wondering if there is any advantage either way to a= ) letting > > > Informix backup the logical logs through ALARMPROGRAM (push) or b= ) > > > scheduling > > > periodic logical log backups through Simpana (pull)? > > > > > > I have my own opinion on this, but would like some other opinions= so that > I > > > can state my case with backup :)So even if your thoughts are basi= c, and > you > > > feel like answering, go for it. > > > > > > Thanks! > > > > > > > > > > > > > > > ***********************************************************************= ******** > > > Forum Note: Use "Reply" to post a response in the discussion foru= m. > > > > > > > > > > --001a11332252dec53504e7d5cf0d > > > > > > > ***********************************************************************= ******** > > Forum Note: Use "Reply" to post a response in the discussion forum.= > > > > > ***********************************************************************= ******** > Forum Note: Use "Reply" to post a response in the discussion forum.= >=
Given a free hand I would always backup via alarm program to disk and then get the storage manager to grab the files. If I have to restore I do not want to be dependent on someone having to find the correct tapes - I want them held locally. Cheers Paul Paul Watson Oninit www.oninit.com +1 913 387 7529 On Oct 3, 2013, at 8:38, "Madison Pruet" <mpruet@us.ibm.com> wrote: > Put it this way. If there is a fire in your house, would you rather your > alarm go off, or do you want to call someone and ask them if smoke is > coming out of your house? > > Sent from my iPad > >> On Oct 3, 2013, at 7:21 AM, "BILL GLIDDEN" <bglidden1962@gmail.com> > wrote: >> >> Thanks for the responses. Personally, I wouldn't do it any other way than > to >> use ALARMPROGRAM and push the logs to the storage manager as they fill. > But >> I'm the "new guy" here so it helps to have other opinions to back up my >> decisions. Thanks. > > ******************************************************************************* > >> Forum Note: Use "Reply" to post a response in the discussion forum. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum.
Hi Bill, Since a "LOG FULL" is an internal engine event (among the many other events) and that the engine executes the alarm program of your choice (the name of the alarm program should be defined in the ALARMPROGRAM onconfig parameter, I prefer to keep this solution. Plus the alarmprogram.sh shell script is provided by IBM. Use it and tailor it to your need. It is a rather complete shell that you can of course enrich and adapt to your needs. It covers all of the events that the IDS engines has. I would not use one script for triggering the log backups and another for the other events, let alone another way of controling the moment to backup the logs. The alamrprograms.sh shell is great: you can send email, trigger actions, backup logs, store the info in relation to the events in files, etc. 1. Use an alarm program in the ALARMPROGRAM parameter of the onconfig file 2. Use the alarmprogram.sh shell and enrich it or adapt it to your needs if need be. Cordialement, Regards, Khaled Bentebal Directeur Général - ConsultiX Président UGIF - User Group Informix France IIUG - Board of Directors Tél: 33 (0) 1 39 12 18 00 Fax: 33 (0) 1 39 12 18 18 Mobile: 33 (0) 6 07 78 41 97 Email: khaled.bentebal@consult-ix.fr Site Web: www.consult-ix.fr Le 03/10/13 14:01, BILL GLIDDEN a écrit : > Good morning folks, > > We have implemented On-Bar with the CommVault Simpana storage manager (love it > so far!). I'm wondering if there is any advantage either way to a) letting > Informix backup the logical logs through ALARMPROGRAM (push) or b) scheduling > periodic logical log backups through Simpana (pull)? > > I have my own opinion on this, but would like some other opinions so that I > can state my case with backup :)So even if your thoughts are basic, and you > feel like answering, go for it. > > Thanks! > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > >
I favor to PUSH too... no matter how...
Today I admin an production where we still using ontape -c , with dedicate
tape + RSS server , this way I can consider the logical logs at secondary
too...
I don't have much experience with onbar , but I plan use it some day... I
plan keep using the -c (continuos) backup with it , and try solve the tape
latency problem with the Storage Manger software, something what I know
Tivoli does (at least a Tivoli consult tell me), what is keep the last
backups at spool area (storage manager server) then some time of day they
write all at once to tape...
With some linux machines (without tape) I personalize the alarmprogram.sh
to backup the logical log , compress and copy immediately to other server
(using rsync) , this way I have them at other place at the same moment.
Always.. pushing... keep move... johnny walker style :)
2013/10/3 Art Kagel <art.kagel@gmail.com>
> The only thing to be careful of when you do PUSH log backups with the
> ALARMPROGRAM using onbar is the latency in the storage manager. Make sure
> that the logical logs are going to FAST storage with no latency of its own
> (ie not a tape carousel). I have seen onbar log backups fall several log
> files behind during peak load because of this where ontape to disk would
> have kept up. However, even then it is no worse than log PULL using
> scheduled onbar runs through the storage manager's scheduler.
>
> Art
>
> Art S. Kagel, Principal Consultant
>
> Advanced DataTools (www.advancedatatools.com)
> Blog: http://informix-myview.blogspot.com/
>
> Disclaimer: Please keep in mind that my own opinions are my own opinions
> and do not reflect on my employer, Advanced DataTools, the IIUG, nor any
> other organization with which I am associated either explicitly,
> implicitly, or by inference. Neither do those opinions reflect those of
> other individuals affiliated with any entity with which I am affiliated nor
> those of the entities themselves.
>
> On Thu, Oct 3, 2013 at 8:56 AM, Fernando Nunes <domusonline@gmail.com
> >wrote:
>
> > PUSH. Without any doubt. Consider why you back up the logical logs....?
> > Because the system may fail and you want to be able to restore it as
> close
> > as possible to the crash time... If you let logical logs accumulate on
> the
> > engine you fail on this basic purpose. This idea should also be
> considered
> > as the main driver for the logical log size calculation (and I recently
> saw
> > some post, blog article or discussion about having a task that will
> switch
> > the logical log if it's there for more than a certain period - and this
> > makes a lot of sense)
> > All other arguments are fait-divers...
> >
> > Regards.
> >
> > On Thu, Oct 3, 2013 at 1:01 PM, BILL GLIDDEN <bglidden1962@gmail.com>
> > wrote:
> >
> > > Good morning folks,
> > >
> > > We have implemented On-Bar with the CommVault Simpana storage manager
> > > (love it
> > > so far!). I'm wondering if there is any advantage either way to a)
> > letting
> > > Informix backup the logical logs through ALARMPROGRAM (push) or b)
> > > scheduling
> > > periodic logical log backups through Simpana (pull)?
> > >
> > > I have my own opinion on this, but would like some other opinions so
> > that I
> > > can state my case with backup :)So even if your thoughts are basic, and
> > you
> > > feel like answering, go for it.
> > >
> > > Thanks!
> > >
> > >
> > >
> > >
> >
> >
>
>
*******************************************************************************
> > > Forum Note: Use "Reply" to post a response in the discussion forum.
> > >
> > >
> >
> > --
> > Fernando Nunes
> > Portugal
> >
> > http://informix-technology.blogspot.com
> > My email works... but I don't check it frequently...
> >
> > --001a11c34b48fe4b6304e7d5b7b9
> >
> >
> >
> >
>
>
*******************************************************************************
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
>
> --089e013a132a86c61004e7d6aa20
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--001a1135e27e43c53304e7d8697f