dbload - performance
Posted in 1995
Pholks,
I'm having a little trouble with one of my favourite utilities - dbload.
Namely it is too slow. In the past this has not been an issue - but in a
warehousing world where you need to load massive amounts of data on a daily
basis it has become an issue. In my development environment a dbload of
a simple 25MB file is taking 16-19 minutes. Granted that the engine is a
puny little thing and nothing has been optimized, but a 'LOAD FROM INSERT
INTO' takes 7-8 minutes, and a cow orker is currently working on an esql/c
program to duplicate the functionality and see if its any faster.
The problem is that I really like the ability to talk to my load utility in
a simplified language to tell it how to parse a fixed length file and have
no desire to trust and support code to pre-parse a file (for a LOAD statement)
or parse on the fly. IMHO code to support a DBLOAD statement is a lot
simpler - both to write and maintain.
Options I have considered revolve around a parallel load:
split the incoming file
run multiple copies of dbload against the same file, AND against the
multiple files I will be receiving.
Don't current versions of Online (7.0+) have a parallel load?
My question is therefore - does anyone have thoughts on optimizing dbload or
should I not expend effort in that direction and rely on a 7.0+ parallel
load? Does the parallel load rely solely on multiple processors or will it
still work with a single one?
Since I'm lazy and not very reliable when it comes to summing up a discussion,
could we keep this one public?
cheers
j.
_____________________________________________________________________________
Jack Parker - Hewlett Packard, BSMC Boise, Idaho, USA
jparker@hpbs3645.boi.hp.com
_____________________________________________________________________________
It ain't over till the digestively challenged lady sings.
_____________________________________________________________________________
Any opinions expressed herein are my own and not those of my employers.
_____________________________________________________________________________