database backup time versus workload
User reported IDS 11.50 backup taking 2 hours normally but 9-10 hours during heavy workload (300GB database). Respondents identified I/O bottleneck as likely cause: backup I/O combined with application workload overwhelms system. Suggested diagnostics: use onstat -p to measure I/O, check sar -d for disk contention, verify backup destination isn't sharing disk controllers/channels with busy database chunks.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Folks,
IDS11.50 FC8, RHEL5.0
Our database backup normally takes 2 hours to have level 0 backup when
the work volume is normal or not very high.
But when things get to crazy , busier work load, long checkpoint time
,...... The backup time can take up to 9-10 hours .....( database size does
not change much daily, about 300GB of ontape size ).
Any comments?
Thanks,
Frank
--001a11c2c974ed4bca04f45733c2
My guess is you are overloading the I/O. A backup does a lot of disk I/O,
but when a heavy
workload is introduced the workload creates I/O, along with the archive.
I would measure
the amount of system I/O when the backup goes normally and the amount of
system I/O when a
things go "crazy" and see what the difference is?
You can use a simple onstat -p to measure this.
John F. Miller III
STSM, Lead Architect
miller3@us.ibm.com
503-747-1366
IBM Informix Dynamic Server (IDS)
ids-bounces@iiug.org wrote on 03/11/2014 09:28:00 AM:
> From: "FRANK" <yunyaoqu@gmail.com>
> To: ids@iiug.org,
> Date: 03/11/2014 09:49 AM
> Subject: database backup time versus workload [32691]
> Sent by: ids-bounces@iiug.org
>
> Folks,
>
> IDS11.50 FC8, RHEL5.0
>
> Our database backup normally takes 2 hours to have level 0 backup when
> the work volume is normal or not very high.
>
> But when things get to crazy , busier work load, long checkpoint time
> ,...... The backup time can take up to 9-10 hours .....( database size
does
> not change much daily, about 300GB of ontape size ).
>
> Any comments?
>
> Thanks,
> Frank
>
> --001a11c2c974ed4bca04f45733c2
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
One thing to be aware of is if the archive is being written to disk is the
target on the same structures on disk as the busiest chunks of the
database? Are they sharing a controller or channel to the SAN? It may be
that there is an IO bottleneck that you can easily reconfigure.
Art
Art S. Kagel, Principal Consultant
ASK Database Management
Blog: http://informix-myview.blogspot.com/
Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on the IIUG, nor any other organization with which I am
associated either explicitly, implicitly, or by inference. Neither do
those opinions reflect those of other individuals affiliated with any
entity with which I am affiliated nor those of the entities themselves.
On Tue, Mar 11, 2014 at 6:01 PM, John Miller iii <miller3@us.ibm.com> wrote:
> My guess is you are overloading the I/O. A backup does a lot of disk I/O,
> but when a heavy
> workload is introduced the workload creates I/O, along with the archive.
> I would measure
> the amount of system I/O when the backup goes normally and the amount of
> system I/O when a
> things go "crazy" and see what the difference is?
>
> You can use a simple onstat -p to measure this.
>
> John F. Miller III
> STSM, Lead Architect
> miller3@us.ibm.com
> 503-747-1366
> IBM Informix Dynamic Server (IDS)
>
> ids-bounces@iiug.org wrote on 03/11/2014 09:28:00 AM:
>
> > From: "FRANK" <yunyaoqu@gmail.com>
> > To: ids@iiug.org,
> > Date: 03/11/2014 09:49 AM
> > Subject: database backup time versus workload [32691]
> > Sent by: ids-bounces@iiug.org
> >
> > Folks,
> >
> > IDS11.50 FC8, RHEL5.0
> >
> > Our database backup normally takes 2 hours to have level 0 backup when
> > the work volume is normal or not very high.
> >
> > But when things get to crazy , busier work load, long checkpoint time
> > ,...... The backup time can take up to 9-10 hours .....( database size
> does
> > not change much daily, about 300GB of ontape size ).
> >
> > Any comments?
> >
> > Thanks,
> > Frank
> >
> > --001a11c2c974ed4bca04f45733c2
> >
> >
> >
>
>
>
*******************************************************************************
>
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--001a11c24358c7fd4804f45be9f5
Thanks Art and John! Will monitor, check and see.. Thanks, Frank
On Tue, Mar 11, 2014 at 6:05 PM, Art Kagel <art.kagel@gmail.com> wrote:
> One thing to be aware of is if the archive is being written to disk is the
> target on the same structures on disk as the busiest chunks of the
> database? Are they sharing a controller or channel to the SAN? It may be
> that there is an IO bottleneck that you can easily reconfigure.
>
> Art
>
> Art S. Kagel, Principal Consultant
> ASK Database Management
>
> Blog: http://informix-myview.blogspot.com/
>
> Disclaimer: Please keep in mind that my own opinions are my own opinions
> and do not reflect on the IIUG, nor any other organization with which I am
> associated either explicitly, implicitly, or by inference. Neither do
> those opinions reflect those of other individuals affiliated with any
> entity with which I am affiliated nor those of the entities themselves.
>
> On Tue, Mar 11, 2014 at 6:01 PM, John Miller iii <miller3@us.ibm.com>
> wrote:
>
> > My guess is you are overloading the I/O. A backup does a lot of disk I/O,
> > but when a heavy
> > workload is introduced the workload creates I/O, along with the archive.
> > I would measure
> > the amount of system I/O when the backup goes normally and the amount of
> > system I/O when a
> > things go "crazy" and see what the difference is?
> >
> > You can use a simple onstat -p to measure this.
> >
> > John F. Miller III
> > STSM, Lead Architect
> > miller3@us.ibm.com
> > 503-747-1366
> > IBM Informix Dynamic Server (IDS)
> >
> > ids-bounces@iiug.org wrote on 03/11/2014 09:28:00 AM:
> >
> > > From: "FRANK" <yunyaoqu@gmail.com>
> > > To: ids@iiug.org,
> > > Date: 03/11/2014 09:49 AM
> > > Subject: database backup time versus workload [32691]
> > > Sent by: ids-bounces@iiug.org
> > >
> > > Folks,
> > >
> > > IDS11.50 FC8, RHEL5.0
> > >
> > > Our database backup normally takes 2 hours to have level 0 backup when
> > > the work volume is normal or not very high.
> > >
> > > But when things get to crazy , busier work load, long checkpoint time
> > > ,...... The backup time can take up to 9-10 hours .....( database size
> > does
> > > not change much daily, about 300GB of ontape size ).
> > >
> > > Any comments?
> > >
> > > Thanks,
> > > Frank
> > >
> > > --001a11c2c974ed4bca04f45733c2
> > >
> > >
> > >
> >
> >
> >
>
>
*******************************************************************************
> >
> > > Forum Note: Use "Reply" to post a response in the discussion forum.
> > >
> >
> >
> >
> >
>
>
*******************************************************************************
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
>
> --001a11c24358c7fd4804f45be9f5
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--001a11c2c9741c529304f45e2263
do you have sar running ? sar -d will quickly show you any q we use a product sarcheck that analyzes the sar files are reports bottle necks a very useful tool here is an extract below that hightlighs disk contention Average time spent waiting for I/O was 18.4 percent. I/O wait time peaked at 63.83 percent from 22:00:01 to 22:10:02. Traditional thresholds for this data show that values in excess of 7 - 15 percent are high enough to suggest an I/O bottleneck. we have backup going then
Thanks Karl! Will see. Thanks Frank On Wed, Mar 12, 2014 at 4:29 PM, KARL OLIVER <karl.oliver@maf.govt.nz>wrote: > do you have sar running ? sar -d will quickly show you any q > > we use a product sarcheck that analyzes the sar files are reports bottle > necks > a very useful tool > here is an extract below that hightlighs disk contention > > Average time spent waiting for I/O was 18.4 percent. I/O wait time > > peaked at 63.83 percent from 22:00:01 to 22:10:02. Traditional > > thresholds for this data show that values in excess of 7 - 15 percent > > are high enough to suggest an I/O bottleneck. > > we have backup going then > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a1132e232f1133a04f46fd6d8