Dbexport too slow
Posted in 2016
A user on IDS 11.70.FC5 (Windows 2008 R2, NetApp storage) complained that dbexport of a 100GB database took about 4.5 hours, with inconsistent run times. Suggestions were: set FET_BUF_SIZE to cut client/server message traffic, consult the IIUG FAQ 8.35 on faster dbexport alternatives, and use John Miller's stored procedure that emulates dbexport via external tables in dirty-read mode. The user tried the external-table approach and the export dropped from 4h30 to 27 minutes. Art Kagel confirmed external tables are generally safe and fast (only caveat: very wide rows, e.g. timeseries/BLOB data) and pointed to his myexport/myimport utilities, which are dbexport/dbimport-compatible.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Topics: Installation, Setup & Upgrades, Migration, Import/Export & Data Conversion, Internationalization & Character Sets
Hi, I have a database of 100 gb which I export every day, the export put a
directory of 29gb, but the problem is that the export is too slow, nearly 4
hours for exporting unls, the command I use is dbexport database -q -ss
sometimes the export do less than 2 hours, I don't understand why but it
happens sometimes.
I have 11.70 FC5 informix installed on a Windows 2008 R2 (64 bits), and the
disks are on a NETAPP storage unit .
I want to know if is there a solution to go fast on exporting, and how to
dianose why sometimes the export is going quick.
regards
Hello,
Please refer "8.35 Is there anything faster than dbexport?" on below URL:
http://www.iiug.org/faqs/informix-faq/ifaq08b.htm.1
Hope this will help you.
Thank & Regards,
Pravin Bankar
pravinebankar@gmail.com
I have two suggestions for you.
1. As for dbexport, make sure you set FET=5FBUF=5FSIZE as it can make a huge
difference
in performance by reducing the number of network messages between client
and server.
2. This will show you how to create dbexport but use the external table
interface which is very fast.
http://www.ibmnosql.com/2012/10/how-to-create-a-stored-procedure-which-emul=
ates-a-very-fast-dbexport-in-dirty-read-mode/
John F. Miller III
STSM, Lead Architect
miller3@us.ibm.com
503-747-1366
IBM Informix Dynamic Server (IDS)
ids-bounces@iiug.org wrote on 03/07/2016 05:21:54 AM:
> From: "PRAVIN BANKAR" <pravinebankar@gmail.com>
> To: ids@iiug.org
> Date: 03/07/2016 05:22 AM
> Subject: Re: Dbexport too slow [36710]
> Sent by: ids-bounces@iiug.org
>
> Hello,
> Please refer "8.35 Is there anything faster than dbexport?" on below URL:
> http://www.iiug.org/faqs/informix-faq/ifaq08b.htm.1
>
> Hope this will help you.
>
> Thank & Regards,
> Pravin Bankar
> pravinebankar@gmail.com
>
>
>
***************************************************************************=
****
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
Or just use my myexport utility that does this for you.
Art
Art S. Kagel, President and Principal Consultant
ASK Database Management
www.askdbmgt.com
Blog: http://informix-myview.blogspot.com/
Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on the IIUG, nor any other organization with which I am
associated either explicitly, implicitly, or by inference. Neither do
those opinions reflect those of other individuals affiliated with any
entity with which I am affiliated nor those of the entities themselves.
On Mon, Mar 7, 2016 at 12:25 PM, John Miller iii <miller3@us.ibm.com> wrote:
> I have two suggestions for you.
>
> 1. As for dbexport, make sure you set FET=5FBUF=5FSIZE as it can make a
> huge
> difference
> in performance by reducing the number of network messages between client
> and server.
>
> 2. This will show you how to create dbexport but use the external table
> interface which is very fast.
>
> http://www.ibmnosql.com/2012/10/how-to-create-a-stored-procedure-which-emul=
> ates-a-very-fast-dbexport-in-dirty-read-mode/
>
> John F. Miller III
> STSM, Lead Architect
> miller3@us.ibm.com
> 503-747-1366
> IBM Informix Dynamic Server (IDS)
>
> ids-bounces@iiug.org wrote on 03/07/2016 05:21:54 AM:
>
> > From: "PRAVIN BANKAR" <pravinebankar@gmail.com>
> > To: ids@iiug.org
> > Date: 03/07/2016 05:22 AM
> > Subject: Re: Dbexport too slow [36710]
> > Sent by: ids-bounces@iiug.org
> >
> > Hello,
> > Please refer "8.35 Is there anything faster than dbexport?" on below URL:
>
> > http://www.iiug.org/faqs/informix-faq/ifaq08b.htm.1
> >
> > Hope this will help you.
> >
> > Thank & Regards,
> > Pravin Bankar
> > pravinebankar@gmail.com
> >
> >
> >
>
> ***************************************************************************=
> ****
>
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--047d7bd75f1c509114052d798eec
Hi, where to add FET=5 ....
and the second question, I have used the stored procedure it's amzing from
4h30 to 27mn , thanks a lot, but you know my colleagus are afraid from this
new features and ask me if it"s with no risk as the classic dbexport and also
if IBM support this things, if there is any problem when restoring?
thanks a lot a lot
The only think I know about is that external tables have a slightly lower
tolerance for VERY wide rows that can result from timeseries or blob data.
But you would find that out at export time anyway.
Art
Art S. Kagel, President and Principal Consultant
ASK Database Management
www.askdbmgt.com
Blog: http://informix-myview.blogspot.com/
Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on the IIUG, nor any other organization with which I am
associated either explicitly, implicitly, or by inference. Neither do
those opinions reflect those of other individuals affiliated with any
entity with which I am affiliated nor those of the entities themselves.
On Mon, Mar 14, 2016 at 4:52 PM, CHALLENGER212 ABDERRAFI <
abderrafi212@gmail.com> wrote:
> Hi, where to add FET=5 ....
>
> and the second question, I have used the stored procedure it's amzing from
> 4h30 to 27mn , thanks a lot, but you know my colleagus are afraid from this
> new features and ask me if it"s with no risk as the classic dbexport and
> also
> if IBM support this things, if there is any problem when restoring?
>
> thanks a lot a lot
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--001a113ee4fcfdffbb052e09f199
Hi, so if I correctly undesrtood, It's risky to use external, is there cases
of failure when retoring?
I'm in the processes of transforming all dbexport to this way, is it valuable?
In general it is very useful to export and import using external tables.
This method is even a bit faster than the HP Loader in most cases. The only
risk is with very wide rows. I use this method in my dbexport/dbimport
replacement utilities myexport/myimport. BTW, no need to reinvent the wheel
here. You can download the myexport package from the IIUG Repository and
get the latest utils2_ak package from my website (
www.askdbmgt.com/my-utilities I have not been uploading the latest
versions to IIUG because we are working on the new IIUG web site). Then you
will have a working replacement for dbexport and dbimport that can use
external tables, dbaccess, or the HP Loader all ready to go. Myexport and
myimport produce files that are compatible with dbexport and dbimport, so
you can export with myexport and import with dbimport and vice-versa with
one small exception. One myexport option, noted in the README files, might
require manual intervention if used, otherwise it is designed to be
compatible. Then if you want any customizations you can start with a
working model.
Art
Art S. Kagel, President and Principal Consultant
ASK Database Management
www.askdbmgt.com
Blog: http://informix-myview.blogspot.com/
Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on the IIUG, nor any other organization with which I am
associated either explicitly, implicitly, or by inference. Neither do
those opinions reflect those of other individuals affiliated with any
entity with which I am affiliated nor those of the entities themselves.
On Thu, Mar 17, 2016 at 3:57 AM, CHALLENGER212 ABDERRAFI <
abderrafi212@gmail.com> wrote:
> Hi, so if I correctly undesrtood, It's risky to use external, is there
> cases
> of failure when retoring?
>
> I'm in the processes of transforming all dbexport to this way, is it
> valuable?
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--047d7bfea0b6ce3017052e3c11d4