Large volume updates
Posted in 1995
Topics: Performance & Tuning
Help! This question may be in the FAQ. (If there is a FAQ. It's not
at rtfm.mit.edu and it's not on comp.answers or news.answers. If it
exists, it has expired on my service provider, and I apologize
profusely!)
We are using Informix 5.04 for a large archival system. Up to
1.5 million data records are staged in an Informix table prior to
being archived by our system.[1] Everything was wonderful, until
the customer decided we needed to synchronize two separate archives.
In essence, given tables a and b, each with a field named key,
we need to partition the data (logically or physically). Best of all,
they have two types of data co-mingled in each input stream.
I'm currently trying to do this using something like:
update a set flag = 0;
update a set flag = 1 where key in
(select key from b where class = 'SPECIAL');
update a set flag = 2 where key in
(select key from b where class <> 'SPECIAL');
update b set flag = 0;
update b set flag = 1 where key in
(select key from a) and class = 'SPECIAL';
update b set flag = 2 where key in
(select key from a) and class <> 'SPECIAL';
Yes, at 1.5 million records each in a and b, this is grotesque.
I'm fried, jet lagged, exhausted, and feeling like I must be
missing something obvious. Does anyone have any suggestions on
how to solve this problem with reasonable performance? Just
the first update is taking 45 minutes.
Any help would be greatly appreciated!
-- Jonathan Crawford
Ark Technologies, Inc.
[1] For the perpetually curious, if you're wondering why we have our
own archival system, it's designed for huge data volumes. To take
one example, we have a customer who's input stream consists of
over 10 million records per day, with a ten-year retention
period and five-minute retrieval requirement. No problem.
But, it's not a "relational archive."