Long transactions
Posted in 2003
mpw wanted dbaccess to commit every N rows while loading about 180000 records. rkusenet recommended Jonathan Leffler's sqlreload with its -n commit interval (default 1024). Paul Watson and Nebojsa Sevo suggested splitting the file with split and loading the pieces in separate runs. Jay Aymond suggested dropping indexes and making the table raw during the load.
Auto-generated by Claude from the posts below — may be imperfect; read the full thread.
Topics: Server Administration
If I'm using a dbaccess script to load data from a file is there a way I can
tell the load to commit the transaction after every 'x' rows processed. I'm
trying to load a file with about 180000 records. I know I can use dbload,
but with the automated tools I have dbaccess works better.
My script file is:
database db1;
load from loadfile.dat insert into table1;commit;
Thanks. M
"mpw" <em-2@cox.net> wrote
> If I'm using a dbaccess script to load data from a file is there a way I can
> tell the load to commit the transaction after every 'x' rows processed. I'm
> trying to load a file with about 180000 records. I know I can use dbload,
> but with the automated tools I have dbaccess works better.
>
> My script file is:
>
> database db1;
> load from loadfile.dat insert into table1;> commit;
use Jonathan Leffler's sqlreload.
sqlreload -d database -t table -i unload_file -n commit_after_n_rows.
The default for -n is 1024.
I find it to be the best as compared to dbaccess or dbload. Great for
automated tools.
Something like
#! /bin/ksh
split loadfile.data
for F in `ls x*`
do
dbaccess db1 <<-EOF
load from $F insert into eric; EOF
done
mpw wrote:
>
> If I'm using a dbaccess script to load data from a file is there a way I can
> tell the load to commit the transaction after every 'x' rows processed. I'm
> trying to load a file with about 180000 records. I know I can use dbload,
> but with the automated tools I have dbaccess works better.
>
> My script file is:
>
> database db1;
> load from loadfile.dat insert into table1;> commit;
>
> Thanks. M
--
Paul Watson #
Oninit Ltd # Growing old is mandatory
Tel: +44 1436 672201 # Growing up is optional
Fax: +44 1436 678693 #
Mob: +44 7818 003457 #
www.oninit.com #
You could drop the indexes, alter the table to "raw", do the load, and then
alter it back to "standard".
"mpw" <em-2@cox.net> wrote in message
news:lmXMa.564558$vU3.107079@news1.central.cox.net...
> If I'm using a dbaccess script to load data from a file is there a way I
can
> tell the load to commit the transaction after every 'x' rows processed.
I'm
> trying to load a file with about 180000 records. I know I can use dbload,
> but with the automated tools I have dbaccess works better.
>
> My script file is:
>
> database db1;
> load from loadfile.dat insert into table1;> commit;
>
> Thanks. M
>
>
On Thu, 03 Jul 2003 14:47:45 GMT, "mpw" <em-2@cox.net> wrote:
If you are running on Unix/Linux you can use split to divide your input file to
smaller pices. Here is from 'man split'
"The split command reads file and writes it in as many n-line pieces as
necessary (default 1000), onto a set of output files. The name of the first
output file is name with aa appended, and so on lexicographically. If no
output name is given, x is default."
So you can write script:
split loadfile.dat ABC
small_files=`ls ABC*`
# make SQL script file
echo " " > load.sql
for b in $small_files; do
echo "load from $b insert into table1;" >> load.sql
done
# run dbaccess
dbaccess db1 load.sql
#
rm ABC* load.sql
Nebojsa
>If I'm using a dbaccess script to load data from a file is there a way I can
>tell the load to commit the transaction after every 'x' rows processed. I'm
>trying to load a file with about 180000 records. I know I can use dbload,
>but with the automated tools I have dbaccess works better.
>
>My script file is:
>
>database db1;
>load from loadfile.dat insert into table1;>commit;
>
>Thanks. M
>
------------------------------------
Remove spam block (DELETE_) to reply