Slow database creation and loading

Hi everybody, On a nightly basis we update our Q/A database with a copy from production. We do this by re-creating the db and then using a combination of HPL and dbimport (for tables with a Text datatype) and this process is taking about 10 and half hours. Using the following code, that I found on these NGs, I calculated the size of the db to be 10.2GB. $INFORMIXDIR/bin/onstat -d | grep PO | grep -v bpages | sort +2 -3 -n | awk '{pri ntf("%s%s\\t%s%s\\t%s%s\\t%s%.2f \\n", " Online Size=", totsz+=$5*2, "Free=", totfre e+=$6*2, "Used=", totused+=($5-$6)*2,"Full=", totpcnt=((totused/ totsz)*100))}'>/ tmp/tmp1; grep -n Size= /tmp/tmp1>/tmp/tmp2; tail -1 /tmp/tmp2 | awk '{printf("%s %s %s %s %s\\n", $2, $3"K", $4"K", $5"K", $6"%")}'; rm /tmp/tmp1 /tmp/tmp2 This code returns the following: Online Size=23382610K Free=13177086K Used=10205524K Full=43.65% Is this 10 plus hour time the best we can do or can we speed this up somehow? If we can speed this up please guide me as I am not a DBA and we do not have a DBA on staff. We are using Informix Dynamic Server Version 7.31.UD7 on AIX on an RS/ 6000 I have been told that we have 4 GB of memory on this box. Here is what our onconfig looks like: # Root Dbspace Configuration ROOTNAME rootdbs # Root dbspace nameROOTPATH /usr/informix/dblinks/infrootlv1 # Path for device containing root dbspace ROOTOFFSET 4 # Offset of root dbspace into device (Kbytes) ROOTSIZE 2097147 # Size of root dbspace (Kbytes) # Disk Mirroring Configuration Parameters MIRROR 1 # Mirroring flag (Yes = 1, No = 0) MIRRORPATH # Path for device containing mirroredroot MIRROROFFSET 0 # Offset into mirrored device (Kbytes) # Physical Log Configuration PHYSDBS rootdbs # Location (dbspace) of physical log PHYSFILE 40000 # Physical log file size (Kbytes) # Logical Log Configuration LOGFILES 14 # Number of logical log files LOGSIZE 10000 # Logical log size (Kbytes) # Diagnostics MSGPATH /usr/informix/online.insurance # System message log file path CONSOLE /usr/informix/console.insurance # System console message path ALARMPROGRAM /usr/informix/log_full.sh # Alarm program path # System Archive Tape Device TAPEDEV /dev/null # Tape device path TAPEBLK 1024 # Tape block size (Kbytes) TAPESIZE 12000000 # Maximum amount of data to put on tape (Kbytes) # Log Archive Tape Device LTAPEDEV /dev/null # Log tape device path LTAPEBLK 1024 # Log tape block size (Kbytes) LTAPESIZE 4096000 # Max amount of data to put on log tape (Kbytes) # Optical STAGEBLOB # INFORMIX-OnLine/Optical staging area # System Configuration SERVERNUM 0 # Unique id corresponding to a OnLineinstance DBSERVERNAME insurance_online # Name of default database server DBSERVERALIASES insurance_remote # List of alternate dbservernames NETTYPE soctcp,3,600,CPU # Added for Inf. TS 2/24/97 #NETTYPE onsoctcp,3,600,NET # Changed for Inf. CSA 5/3/06 DEADLOCK_TIMEOUT 60 # Max time to wait of lock indistributed env. RESIDENT 0 # Forced residency flag (Yes = 1, No = 0) MULTIPROCESSOR 1 # 0 for single-processor, 1 for multi-processor NUMCPUVPS 3 # Number of user (cpu) vps SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vpsto one NOAGE 1 # Process aging AFF_SPROC 0 # Affinity start processor AFF_NPROCS 0 # Affinity number of processors # Shared Memory Parameters LOCKS 750000 # Maximum number of locks BUFFERS 75000 # Maximum number of shared buffers#NUMAIOVPS 1 # Number of IO vps NUMAIOVPS 36 # Number of IO vps changed CSA 05/3/06 PHYSBUFF 64 # Physical log buffer size (Kbytes) LOGBUFF 64 # Logical log buffer size (Kbytes)LOGSMAX 40 # Maximum number of logical log files CLEANERS 32 # Number of buffer cleaner processes SHMBASE 0x30000000 # Shared memory base address#SHMVIRTSIZE 131072 # initial virtual shared memory segment size SHMVIRTSIZE 524288 # initial virtual shared memorysegment size SHMADD 32768 # Size of new shared memory segments (Kbytes) SHMTOTAL 0 # Total shared memory (Kbytes). 0=>unlimited CKPTINTVL 1200 # Check point interval (in sec)# LRUS 4 # Number of LRU queues # Test config for LRUS. From NG posting seems buffers to LRUS should be a # ratio of 2560 BUFFERS to 1 LRUS. LRUS 19 # Number of LRU queues LRU_MAX_DIRTY 10 # LRU percent dirty begin cleaninglimit LRU_MIN_DIRTY 5 # LRU percent dirty end cleaning limit LTXHWM 50 # Long transaction high water markpercentage LTXEHWM 60 # Long transaction high water mark (exclusive) TXTIMEOUT 0x12c # Transaction timeout (in sec) STACKSIZE 32 # Stack size (Kbytes) # System Page Size # BUFFSIZE - OnLine no longer supports this configuration parameter. # To determine the page size used by OnLine on your platform # see the last line of output from the command, 'onstat - b'. # Recovery Variables # OFF_RECVRY_THREADS: # Number of parallel worker threads during fast recovery or an offline restore. # ON_RECVRY_THREADS: # Number of parallel worker threads during an online restore. OFF_RECVRY_THREADS 10 # Default number of offline workerthreads ON_RECVRY_THREADS 1 # Default number of online workerthreads # Data Replication Variables # DRAUTO: 0 manual, 1 retain type, 2 reverse type DRAUTO 0 # DR automatic switchover DRINTERVAL 30 # DR max time between DR buffer flushes (in sec) DRTIMEOUT 30 # DR network timeout (in sec) DRLOSTFOUND /usr/informix/etc/dr.lostfound # DR lost+found filepath # Backup/Restore variables BAR_ACT_LOG /tmp/bar_act_insurance.log BAR_MAX_BACKUP 0 BAR_RETRY 1 BAR_NB_XPORT_COUNT 10 BAR_XFER_BUF_SIZE 31 # Read Ahead Variables RA_PAGES # Number of pages to attempt to readahead RA_THRESHOLD # Number of pages left before nextgroup # DBSPACETEMP: # OnLine equivalent of DBTEMP for SE. This is the list of dbspaces # that the OnLine SQL Engine will use to cr

You are mostly running into defect/feature idsdb00100415 which is available 10.00.xC6 onwards. IDS 10 has many such performance improvements so may be time to upgrade ... Thanks and Regards, Gaurav johneevo <johneevo@gmail.c om> To Sent by: informix-list@iiug.org informix-list-bou cc nces@iiug.org Subject Slow database creation and loading 07/06/2007 18:34 Hi everybody, On a nightly basis we update our Q/A database with a copy from production. We do this by re-creating the db and then using a combination of HPL and dbimport (for tables with a Text datatype) and this process is taking about 10 and half hours. Using the following code, that I found on these NGs, I calculated the size of the db to be 10.2GB. $INFORMIXDIR/bin/onstat -d | grep PO | grep -v bpages | sort +2 -3 -n | awk '{pri ntf("%s%s\\t%s%s\\t%s%s\\t%s%.2f \\n", " Online Size=", totsz+=$5*2, "Free=", totfre e+=$6*2, "Used=", totused+=($5-$6)*2,"Full=", totpcnt=((totused/ totsz)*100))}'>/ tmp/tmp1; grep -n Size= /tmp/tmp1>/tmp/tmp2; tail -1 /tmp/tmp2 | awk '{printf("%s %s %s %s %s\\n", $2, $3"K", $4"K", $5"K", $6"%")}'; rm /tmp/tmp1 /tmp/tmp2 This code returns the following: Online Size=23382610K Free=13177086K Used=10205524K Full=43.65% Is this 10 plus hour time the best we can do or can we speed this up somehow? If we can speed this up please guide me as I am not a DBA and we do not have a DBA on staff. We are using Informix Dynamic Server Version 7.31.UD7 on AIX on an RS/ 6000 I have been told that we have 4 GB of memory on this box. Here is what our onconfig looks like: # Root Dbspace Configuration ROOTNAME rootdbs # Root dbspace nameROOTPATH /usr/informix/dblinks/infrootlv1 # Path for device containing root dbspace ROOTOFFSET 4 # Offset of root dbspace into device (Kbytes) ROOTSIZE 2097147 # Size of root dbspace (Kbytes) # Disk Mirroring Configuration Parameters MIRROR 1 # Mirroring flag (Yes = 1, No = 0) MIRRORPATH # Path for device containing mirroredroot MIRROROFFSET 0 # Offset into mirrored device (Kbytes) # Physical Log Configuration PHYSDBS rootdbs # Location (dbspace) of physical log PHYSFILE 40000 # Physical log file size (Kbytes) # Logical Log Configuration LOGFILES 14 # Number of logical log files LOGSIZE 10000 # Logical log size (Kbytes) # Diagnostics MSGPATH /usr/informix/online.insurance # System message log file path CONSOLE /usr/informix/console.insurance # System console message path ALARMPROGRAM /usr/informix/log_full.sh # Alarm program path # System Archive Tape Device TAPEDEV /dev/null # Tape device path TAPEBLK 1024 # Tape block size (Kbytes) TAPESIZE 12000000 # Maximum amount of data to put on tape (Kbytes) # Log Archive Tape Device LTAPEDEV /dev/null # Log tape device path LTAPEBLK 1024 # Log tape block size (Kbytes) LTAPESIZE 4096000 # Max amount of data to put on log tape (Kbytes) # Optical STAGEBLOB # INFORMIX-OnLine/Optical staging area # System Configuration SERVERNUM 0 # Unique id corresponding to a OnLineinstance DBSERVERNAME insurance_online # Name of default database server DBSERVERALIASES insurance_remote # List of alternate dbservernames NETTYPE soctcp,3,600,CPU # Added for Inf. TS 2/24/97 #NETTYPE onsoctcp,3,600,NET # Changed for Inf. CSA 5/3/06 DEADLOCK_TIMEOUT 60 # Max time to wait of lock indistributed env. RESIDENT 0 # Forced residency flag (Yes = 1, No = 0) MULTIPROCESSOR 1 # 0 for single-processor, 1 for multi-processor NUMCPUVPS 3 # Number of user (cpu) vps SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vpsto one NOAGE 1 # Process aging AFF_SPROC 0 # Affinity start processor AFF_NPROCS 0 # Affinity number of processors # Shared Memory Parameters LOCKS 750000 # Maximum number of locks BUFFERS 75000 # Maximum number of shared buffers#NUMAIOVPS 1 # Number of IO vps NUMAIOVPS 36 # Number of IO vps changed CSA 05/3/06 PHYSBUFF 64 # Physical log buffer size (Kbytes) LOGBUFF 64 # Logical log buffer size (Kbytes)LOGSMAX 40 # Maximum number of logical log files CLEANERS 32 # Number of buffer cleaner processes SHMBASE 0x30000000 # Shared memory base address#SHMVIRTSIZE 131072 # initial virtual shared memory segment size SHMVIRTSIZE 524288 # initial virtual shared memorysegment size SHMADD 32768 # Size of new shared memory segments (Kbytes) SHMTOTAL 0 # Total shared memory (Kbytes). 0=>unlimited CKPTINTVL 1200 # Check point interval (in sec)# LRUS 4 # Number of LRU queues # Test config for LRUS. From NG posting seems buffers to LRUS should be a # ratio of 2560 BUFFERS to 1 LRUS. LRUS 19 # Number of LRU queues LRU_MAX_DIRTY 10 # LRU percent dirty begin cleaninglimit LRU_MIN_DIRTY 5 # LRU percent dirty end cleaning limit LTXHWM 50 # Long transaction high water markpercentage LTXEHWM 60 # Long transaction high water mark (exclusive) TXTIMEOUT 0x12c # Transaction timeout (in sec) STACKSIZE 32 # Stack size (Kbytes) # System Page Size # BUFFSIZE - OnLine no longer supports this configuration parameter. # To determine the page size used by OnLine on your platform # see the last line of output from the command, 'onstat - b'. # Recovery Variables # OFF_RECVRY_THREADS: # Number of parallel worker threads during fast recovery or an offline restore. # ON_RECVRY_THREADS: # Number of parallel worker threads during an online restore. OFF_RECVRY_THREADS 10 # Default number of offline workerthreads ON_RECVRY_THREADS 1 # Default number of online workerthreads # Data Replication Variables # DRAUTO: 0 manual, 1 retain type, 2 reverse type DRAUTO 0 # DR au

On Jun 7, 10:54 am, Gaurav Saxena3 <gsaxe...@in.ibm.com> wrote: Hi Gaurav, I'm sorry I am not familiar with "idsdb00100415". Can you give me more information about what this defect/feature is or how/where to find this on IBM's site? Thanks > You are mostly running into defect/feature idsdb00100415 which is available > 10.00.xC6 onwards. IDS 10 has many such performance improvements so may be > time to upgrade ... > > Thanks and Regards, > Gaurav