Database backup running extremely slow
Posted in 2003
A user on IDS 7.31 found ontape level-0 backups suddenly running far slower than usual (10% done after 30 minutes). Respondents identified the known "very old page" bug (defect 101062, archive performance degradation from many arc_very_old_page() calls), typically hitting busy systems with long-unchanged data and recurring roughly weekly. Diagnosis: run onstat -g stk on the arcbackup1 thread and look for arc_very_old_page() at the top of the stack. The suggested fix was upgrading (7.31.UD5/UC7 or later, 9.21.UC6+) and setting the CCFLAGS environment variable before starting the engine, as documented in the release notes. A follow-up question about Veritas NetBackup client timeouts went unanswered.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Topics: Backup & Restore
Any
ideas why a level 0 backup using ontape on Informix 7.31 suddenly runs
extremely slow! Normally it takes our backup to run for an hour and now it
is already running for 30 minutes and it's only up to 10 percent completed.
Thanks in advance for your response.
Maria Makasakit
On
Mon, 3 Feb 2003 21:58:03 -0500 (EST), Mariamakasa.... wrote:
>Any ideas why a level 0 backup using ontape on Informix 7.31 suddenly runs
>extremely slow! Normally it takes our backup to run for an hour and now it
>is already running for 30 minutes and it's only up to 10 percent completed.
>
>Thanks in advance for your response.
>
>Maria Makasakit
>
>
>
>
All together now:
"IT'S THE "VERY OLD PAGE" BUG"!!!!
.... Unfortunately, the Product Defect Query page at IBM/Informix is currently
unavailable and I can't recall the exact causes and fix but I'm sure Art & Co.
can give you some ideas.
Sorry can't be more help
Malc
--
XS2Mail: Check your mail anywhere http://www.xs2mail.com/
It's BUG
101062.
In former versions informix had to update the timestamp of "very old pages"
(maxint behind the current timestamp) to old pages.
If you once hit this BUG, you'll hit it periodically once a week (in normal
working environment).
There is no real way out of this problem except you update to 7.31.UD5 or up/
9.21.UC6 or up, where you can set CCFLAGS to avoid this problem.
CCFLAGS (in this part) is noticed in releasenotes.
On Tue, 4 Feb 2003 04:08:56 -0500 (EST)
"Malc_p " <malc_p@btinternet.com> wrote:
> On Mon, 3 Feb 2003 21:58:03 -0500 (EST), Mariamakasa.... wrote:
> >Any ideas why a level 0 backup using ontape on Informix 7.31 suddenly runs
> >extremely slow! Normally it takes our backup to run for an hour and now it
> >is already running for 30 minutes and it's only up to 10 percent completed.
> >
> >Thanks in advance for your response.
> >
> >Maria Makasakit
> >
> >
> >
> >
>
> All together now:
>
> "IT'S THE "VERY OLD PAGE" BUG"!!!!
>
> .... Unfortunately, the Product Defect Query page at IBM/Informix is
currently unavailable and I can't recall the exact causes and fix but I'm sure
Art & Co. can give you some ideas.
>
>
> Sorry can't be more help
>
> Malc
>
> --
> XS2Mail: Check your mail anywhere http://www.xs2mail.com/
>
--
Mit freundlichen Grüßen
Gerd Kaluzinski
\\\\\\\\|//
(o o)
--------------------------------------------------ooO-(_)-Ooo---
Gerd Kaluzinski mailto:gerd.kaluzinski@bytec.de
Manager Consulting mailto:support@bytec.de
Informix Certified Senior System Engineer
http://www.bytec.de
BYTEC GmbH Telefon: 07541-585-1019
Hermann-Metzger-Str. 7 Fax : 07541-585-2019
88045 Friedrichshafen Ooo.
-------------------------------------------------.ooO----( )---
( ) (_/
\\\\_)
Maria
Do you have a relatively high throughput system but with some data that has
not changed for a very long time? Are you running UC6 or earlier? If so you
may be hitting bug 101062 - ARCHIVE PERFORMANCE DEGRADATION WHEN MANY CALLS
TO ARC_VERY_OLD_PAGE().
Rajib Sarkar posted the following checks in an earlier thread:-
Check the onstat -g stk output of the arcbackup1 thread for a period
of
time. If you consistently see the arc_very_old_page() function at
top of
the stack then you know, u've hit the bug.
The only solution is to upgrade to UC7 or above (not UD4 if you have
detached indexes !!) and set CCFLAGS environmental variable before starting
the engine.
Search the newsgroup for fuller details or contact tech support.
Keith
-> -----Original Message-----
-> From: Mariamakasa.... [mailto:Mariamakasakit@aol.com]
-> Sent: Tuesday, February 04, 2003 2:58 AM
-> To: ids@iiug.org
-> Subject: Database backup running extremely slow [220]
->
->
-> Any ideas why a level 0 backup using ontape on Informix 7.31
-> suddenly runs
-> extremely slow! Normally it takes our backup to run for an
-> hour and now it
-> is already running for 30 minutes and it's only up to 10
-> percent completed.
->
-> Thanks in advance for your response.
->
-> Maria Makasakit
->
->
->
********************************************************************************
**
This message is sent in strict confidence for the addressee only. It may
contain legally privileged information. The contents are not to be disclosed
to anyone other than the addressee. Unauthorised recipients are requested
to preserve this confidentiality and to advise the sender immediately of any
error in transmission.
This footnote also confirms that this email message has been swept for the
presence of computer viruses, however we cannot guarantee that this message
is free from such problems.
********************************************************************************
**
Is it possible that this bug could cause a backup client, in this case
Veritas Netbackup, to have a network time out.
Here is an example of an error message we are getting. We just started to
have these problems over the last 2 weeks.
10:54:17 INF - Backup by root on client psfsdb-bak using class
ps-db-offsite: timed out connecting to client.
"Simmons, Keith"
<keith.simmons@bbslimi To: ids@iiug.org
ted.co.uk> cc:
Sent by: Subject: RE: Database backup running extremely slow [230]
forum.subscriber@iiug.
org
02/04/2003 06:03 AM
Maria
Do you have a relatively high throughput system but with some data that has
not changed for a very long time? Are you running UC6 or earlier? If so you
may be hitting bug 101062 - ARCHIVE PERFORMANCE DEGRADATION WHEN MANY CALLS
TO ARC_VERY_OLD_PAGE().
Rajib Sarkar posted the following checks in an earlier thread:-
Check the onstat -g stk output of the arcbackup1 thread for a period
of
time. If you consistently see the arc_very_old_page() function at
top of
the stack then you know, u've hit the bug.
The only solution is to upgrade to UC7 or above (not UD4 if you have
detached indexes !!) and set CCFLAGS environmental variable before starting
the engine.
Search the newsgroup for fuller details or contact tech support.
Keith
-> -----Original Message-----
-> From: Mariamakasa.... [mailto:Mariamakasakit@aol.com]
-> Sent: Tuesday, February 04, 2003 2:58 AM
-> To: ids@iiug.org
-> Subject: Database backup running extremely slow [220]
->
->
-> Any ideas why a level 0 backup using ontape on Informix 7.31
-> suddenly runs
-> extremely slow! Normally it takes our backup to run for an
-> hour and now it
-> is already running for 30 minutes and it's only up to 10
-> percent completed.
->
-> Thanks in advance for your response.
->
-> Maria Makasakit
->
->
->
********************************************************************************
**
This message is sent in strict confidence for the addressee only. It may
contain legally privileged information. The contents are not to be
disclosed
to anyone other than the addressee. Unauthorised recipients are requested
to preserve this confidentiality and to advise the sender immediately of
any
error in transmission.
This footnote also confirms that this email message has been swept for the
presence of computer viruses, however we cannot guarantee that this message
is free from such problems.
********************************************************************************
**
Kipp
Sorry to contradict and correct, but syssesprof and syssessions are valid
system tables created in the sysmaster database. They are not 'real' tables
but rather read-only views into the shared memory. They can give the same
(and enhanced) information that comes from the onstat commands, but they are
accessible from sql and so the output can be easily placed into programs for
manipulation and analysis. Also known as SMI tables, a full description of
their contents can be found in the Admin Guide for the engine (try Ch 34 for
a 7.x engine).
Keith
-> -----Original Message-----
-> From: Kipp Sherry [mailto:KSHERRY@corr.state.id.us]
-> Sent: Wednesday, April 23, 2003 11:33 PM
-> To: classics@iiug.org
-> Subject: Re: Performance tunning [230]
->
->
-> Esteban,
->
-> I think what is going on is that you are reading information
-> from two =
-> different locations.
-> ONSTAT gets it's information directly from the database
-> engine. It's a =
-> gimme, comes standard.
-> SYSSESPROF and SYSSESSIONS are tables that were created in
-> the database. =
-> These were probably created during database design and are
-> probably used =
-> for some in-house tracking. Thus they are probably populated
-> by some =
-> in-house program.
->
-> Good Luck,
->
->
-> Kipp D. Sherry
-> IT Systems Analyst
-> Idaho Department of Correction
-> Administration Office / IT
-> Phone: 208-658-2089
-> Fax: 208-327-7469
->
-> >>> "ESTEBAN RIEZNIK" <erieznikr@repsolypf.com> 04/23/03 03:37PM >>>
-> Hi there. New in IIUG.
->
-> About tables syssesprof and sysptprof, i know that some
-> fields, e.g. =
-> isreads, iswrites et al. give an idea of session and table
-> I/O activity, =
-> but what DOES exactly those fields mean: read records, read
-> kbytes, read =
-> bytes, records read no matter if match a condition, or what ?
->
-> Following this topic, i've found that a query on syssesprof:
->
-> select syssesprof.sid, syssessions.username, isreads
-> fro syssesprof,syssessions
-> where syssesprof.sid =3D syssessions.sid
-> order by isreads
->
-> has little (or nothing) to do with values shown in "onstat
-> -u" (nreads and =
-> nwrites columns) Any idea why ?
->
-> Thanks in advance
-> Truly
->
->
********************************************************************************
**
This message is sent in strict confidence for the addressee only. It may
contain legally privileged information. The contents are not to be disclosed
to anyone other than the addressee. Unauthorised recipients are requested
to preserve this confidentiality and to advise the sender immediately of any
error in transmission.
This footnote also confirms that this email message has been swept for the
presence of computer viruses, however we cannot guarantee that this message
is free from such problems.
********************************************************************************
**