Update Statistics & dbspace not clearing
9.30.UC5 on Solaris 8 A C++ application has it's own idx and dbs spaces and is unable to reclaim space from deleted rows. This causes the spaces to fill over time. The programmer says that this occurs because 'update statistics' must be run periodically on the database. He is building 'update statistics' into his application so space will be reclaimed in the dbspaces. I've never seen a database behave this way. I looked at my docs and 'Informix IDS Unlocking the Mysteries Behind Update Statistics' John F. Miller III, and found no mention of 'update statistics' in this context. I also find no info about reclaiming space being a problem in any context. Has anyone seen this problem? What kinds of things should I look for in the code? Thanks, John
AFAIK, update statistics, while a very necessary thing to do, has
nothing to do with page compression.
Over time the engine will note that a page has over a threshold number of
deleted rows (I don't know what that threshold is - probably on the order of
60%). It that time it will perform a compression, and re-write all of the good
rows starting over at the start of the page, then update the freelist to
indicate a) where the next slot on this page is b) that this page has free
space. You can see the count of compressions in onstat -p output.
If the data you are deleting resides on the same subset of pages, these will
tend to go through compression quite a bit. However pages against which no
deletes are occuring may never get compressed (so you deleted 10 rows 1 year
ago and have done nothing since).
I seem to remember that since the engine always writes to the end of the page
that the space is not re-used until a compression has occurred - I could be
off on that.
The key is a) design so that you are not doing a lot of deletes. b) those
deletes that you are performing are happening to a subset of the pages - so
that they frequently become candidates for compression (an excellent reason to
fragment) c) reorg your tables occasionally to do your own 'compression'.
Space reclamation is generally not an issue unless you have a very volatile
dataset, in which case the first sign of it is wild fragmentation of extents -
which 'oncheck -pe' should report to you. When you see that, then it is time
to start planning a reorg.
You should consider isolating volatile datasets into their own dbspaces,
otherwise their extent allocation/reclamation (during a reorg) may leave all
sorts of holes at the dbspace level. Of course there are frequently more
important business considerations which can affect your fragmentation strategy.
cheers
j.
>From: "John Whitte...." <john.whittenberger@metnet.navy.mil>
>Date: Tue Aug 09 09:45:25 CDT 2005
>To: ids@iiug.org
>Subject: Update Statistics & dbspace not clearing [5586]
>9.30.UC5 on Solaris 8
>
>A C++ application has it's own idx and dbs spaces and is unable
>to reclaim space from deleted rows. This causes the spaces to
>fill over time. The programmer says that this occurs because
>'update statistics' must be run periodically on the database.
>He is building 'update statistics' into his application so space
>will be reclaimed in the dbspaces.
>
>I've never seen a database behave this way. I looked at my docs
>and 'Informix IDS Unlocking the Mysteries Behind Update
>Statistics' John F. Miller III, and found no mention of 'update
>statistics' in this context. I also find no info about reclaiming
>space being a problem in any context.
>
>Has anyone seen this problem? What kinds of things should I look
>for in the code?
>
>Thanks,
>
>John
Let me get this straight... Your programmer says that the table(s) and indexes
are growing without limit because the space from deleted rows is not reused for
new data within the tables? And he says that if he changes the app to run
update statistics periodically it will release the space? Is that the question?
Answer: No. You cannot find this documented because that's not the way that IDS
works! Within a table and its indexes all deleted row space is reused
immediately for newly inserted rows if needed with the sole exception of
blobspace blob pages (TEXT or DATA type columns stored in blobspace). The
pre-image copies of blobspace blobpages are not logged so they cannot be
released until the next ontape/onbar archive has been taken. Update statistics
has no effect on BLOB pages.
OK, physically indexes are cleaned asynchronously by the BTREE CLEANERS or
SCANNERS depending on the version. These threads compress partial nodes and
free index pages thus emptied. Space freed by key deletions that produce
partially filled index pages is not available immediately for new index pages
but are freed over time. Completely empty index nodes are freed for reuse
immediately and deleted key slots can always be reused before compression for
new individual keys that fall into such a free slot. You can tune the BTREE
scanners and can launch a scan using onmode. Update statistics has nothing to
do with it.
It is true that deleted row/key space is not released back to the server's free
extent list for reuse by other tables. But update statistics has nothing to do
with that either. To release unneeded space within a table or its indexes back
to the free extent pool you have to reorganize the table and/or rebuild the
index. Tables are reorged by one of the following methods:
1 - Unload the data, drop the table, recreate the table, reload the data,
rebuild indexes.
2 - ALTER INDEX <some index on the table> TO CLUSTER;
(if the index is already clustered uncluster it first. Requires space for
the intermediate copy of the table, locks and logical log pages unless you
alter the table to RAW first.)
3 - ALTER FRAGMENT FOR TABLE <sometable> INIT IN <dbspace or fragmentation
clause>;
(this can be done whether the table is fragmented or not and the dbspace or
fragmentation clause can be a different one than the one the table already
lives
in or the same one. Requires space for the intermediate copy of the table,
locks and logical log pages unless you alter the table to RAW first.)
Art S. Kagel
----- Original Message -----
From: John Whitte....
At: 8/ 9 12:40
9.30.UC5 on Solaris 8
A C++ application has it's own idx and dbs spaces and is unable
to reclaim space from deleted rows. This causes the spaces to
fill over time. The programmer says that this occurs because
'update statistics' must be run periodically on the database.
He is building 'update statistics' into his application so space
will be reclaimed in the dbspaces.
I've never seen a database behave this way. I looked at my docs
and 'Informix IDS Unlocking the Mysteries Behind Update
Statistics' John F. Miller III, and found no mention of 'update
statistics' in this context. I also find no info about reclaiming
space being a problem in any context.
Has anyone seen this problem? What kinds of things should I look
for in the code?
Thanks,
John
I was
under the impression that BTREE cleaners are
"started up" when an UPDATE STATISTICS was ran. Is
this absolutely incorrect?
I seem to remember a server at my old job that had a
major performance issue one day. We were able to trace
the naughty thread down to the BTREE cleaner. The
Informix ESE's that were on site gave us an
undocumented onmode command to turn it off. I'm pretty
sure that they said an UPDATE STATISTICS would start
it up. It made since at the time beacuse normally
UPDATE STATISTICS did not run on that server and wefound a programmer who had executed one.
--- "ART KAGEL, ...." <KAGEL@bloomberg.net> wrote:
>
> Let me get this straight... Your programmer says
> that the table(s) and indexes
> are growing without limit because the space from
> deleted rows is not reused for
> new data within the tables? And he says that if he
> changes the app to run
> update statistics periodically it will release the> space? Is that the question?
>
> Answer: No. You cannot find this documented because
> that's not the way that IDS
> works! Within a table and its indexes all deleted
> row space is reused
> immediately for newly inserted rows if needed with
> the sole exception of
> blobspace blob pages (TEXT or DATA type columns
> stored in blobspace). The
> pre-image copies of blobspace blobpages are not
> logged so they cannot be
> released until the next ontape/onbar archive has
> been taken. Update statistics
> has no effect on BLOB pages.
>
> OK, physically indexes are cleaned asynchronously by
> the BTREE CLEANERS or
> SCANNERS depending on the version. These threads
> compress partial nodes and
> free index pages thus emptied. Space freed by key
> deletions that produce
> partially filled index pages is not available
> immediately for new index pages
> but are freed over time. Completely empty index
> nodes are freed for reuse
> immediately and deleted key slots can always be
> reused before compression for
> new individual keys that fall into such a free slot.
> You can tune the BTREE
> scanners and can launch a scan using onmode. Update
> statistics has nothing to
> do with it.
>
> It is true that deleted row/key space is not
> released back to the server's free
> extent list for reuse by other tables. But update
> statistics has nothing to do
> with that either. To release unneeded space within
> a table or its indexes back
> to the free extent pool you have to reorganize the
> table and/or rebuild the
> index. Tables are reorged by one of the following
> methods:
>
> 1 - Unload the data, drop the table, recreate the
> table, reload the data,
> rebuild indexes.
> 2 - ALTER INDEX <some index on the table> TO
> CLUSTER;
> (if the index is already clustered uncluster it
> first. Requires space for
> the intermediate copy of the table, locks and
> logical log pages unless you
> alter the table to RAW first.)
> 3 - ALTER FRAGMENT FOR TABLE <sometable> INIT IN
> <dbspace or fragmentation
> clause>;
> (this can be done whether the table is
> fragmented or not and the dbspace or
> fragmentation clause can be a different one than the
> one the table already lives
> in or the same one. Requires space for the
> intermediate copy of the table,
> locks and logical log pages unless you alter the
> table to RAW first.)
>
> Art S. Kagel
>
> ----- Original Message -----
> From: John Whitte....
> At: 8/ 9 12:40
>
> 9.30.UC5 on Solaris 8
>
> A C++ application has it's own idx and dbs spaces
> and is unable
> to reclaim space from deleted rows. This causes the
> spaces to
> fill over time. The programmer says that this
> occurs because
> 'update statistics' must be run periodically on the
> database.
> He is building 'update statistics' into his
> application so space
> will be reclaimed in the dbspaces.
>
> I've never seen a database behave this way. I
> looked at my docs
> and 'Informix IDS Unlocking the Mysteries Behind
> Update
> Statistics' John F. Miller III, and found no mention
> of 'update
> statistics' in this context. I also find no info
> about reclaiming
> space being a problem in any context.
>
> Has anyone seen this problem? What kinds of things
> should I look
> for in the code?
>
> Thanks,
>
> John
>
>
>
>
>