RE: Periodic extremely slow ontape archives

The following Informix software defect is known to cause serious performance degradation during backups. 101062 ARCHIVE PERFORMANCE DEGRADATION WHEN MANY CALLS TO ARC_VERY_OLD_PAGE() At last year's IWUC conference, users complained about a 10-fold increase in backup time when the ARC_VERY_OLD_PAGE routine was updating timestamps during backup processing. The Informix developers informed us that this issue had been addressed in the IDS release AFTER 9.2. If you encounter significant update activity while ontape is running, you are probably being impacted by this bug and should contact Informix technical support. Rick Bernstein ALARIS Medical Systems, Inc. -----Original Message----- From: Murray Wood [mailto:murray@quanta.co.nz] Sent: Wednesday, January 05, 2000 4:33 PM To: informix-list@iiug.org Subject: RE: Periodic extremely slow ontape archives This sort of behaviour suggests that there was some resource contention. What about Tape I/O - could you measure whether there were I/O errors (recovered) as the archive was writing the tape? How busy was the tape? Did sar report any issues during this time? I suggest on SUN, increasing TAPEBLK to 1024 to get the archive moving faster. Murray Wood -----Original Message----- From: owner-informix-list@iiug.org [mailto:owner-informix-list@iiug.org]On Behalf Of Dick.Brieck@chase.com Sent: Thursday, January 06, 2000 9:58 AM To: informix-list@iiug.org Subject: Periodic extremely slow ontape archives Greetings. We have been experiencing intermittently long archives using ontape for several months. These long runs have increased in duration and frequency during this time. The box doesn't look pregnant! Most nights we finish in under 3 hours, sometimes over 21 hours. We average around 2 long archives per week. Our production batch is scheduled to run after this, so a long run adversely effects us the next day. There is no obvious correlation between system activity during the archive and it's duration. By day we do mostly OLTP and run batch at night. We schedule the archive before the nightly batch and after most daily OLTP work. Here is an interesting example of what has happened the last 2 days. Monday's archive started at 9 PM Time to complete = 21 hours 35 minutes Average processor time / 15 minute increment = 2 - 3 seconds Tuesday's archive started just 2 1/2 hrs after the long archive: Time to complete = 2 hours 23 minutes Average processor time / 15 minute increment = 23 - 24 seconds ** Note: The processor times were obtained from a unix 'ps' command taken at 15 minute intervals. CPU Utilization was < 50% both days Memory Utilization < 70% both days No apparent difference in jobs running on the two days The backup is done via an Informix Utility and uses compression on a DLT 7000 tape drive. Disk IO never spiked on Monday, but it spiked for the entire 2.5 hours on Tuesday (as it should have). It looks like we had plenty of available resources during this time. ontape was just t a a a k i ng i t ' s t i i i i m e. I'd really appreciate some help with this one as we are getting hit pretty hard and i'm stumped for now. Thanks! Here's our system specs: Hardware: Sun Enterprise 6000, 14 cpus @ 248 MHz, 7GB memory OS: Solaris 2.6 Database: IDS 7.30.UC6 Our database size is around 80 GB. of RAID 5. I really do want to change, Art. This Sun host supports 2 Informix instances. One is a magnetic cache for blobs and is archived to /dev/null. Other subsystems: Plexus Floware The Floware system allocates shared memory and has background daemons. It uses an Informix database that it manages. API calls to this system are made by client applications utilizing workflow functionality. Here's our onconfig: Informix Dynamic Server Version 7.30.UC6 -- On-Line -- Up 11 days 22:49:41 -- 1180224 Kbytes Configuration File: /usr/informix/etc/onconfig.onl #************************************************************************** # # INFORMIX SOFTWARE, INC. # # Title: onconfig for wil_online # Description: INFORMIX-OnLine Configuration Parameters # #************************************************************************** # Root Dbspace Configuration ROOTNAME rootdbs # Root dbspace nameROOTPATH /dev/vx/rdsk/rawdg/d10001 # Path for root dbspace device ROOTOFFSET 0 # Offset of root dbspace into device (Kbytes) ROOTSIZE 500000 # Size of root dbspace (Kbytes) # Disk Mirroring Configuration Parameters MIRROR 0 # Mirroring flag (Yes = 1, No = 0) MIRRORPATH # Path device containing mirrored root MIRROROFFSET 0 # Offset into mirrored device (Kbytes) # Physical Log Configuration PHYSDBS phys # Location (dbspace) of physical log PHYSFILE 30000 # Physical log file size (Kbytes) # Logical Log Configuration LOGFILES 50 # Number of logical log files LOGSIZE 10000 # Logical log size (Kbytes) # Diagnostics MSGPATH /usr/informix/onl_online.log # System message log file path CONSOLE /dev/console # System console message path ALARMPROGRAM /usr/informix/etc/no_log.sh # Alarm program path# SYSALARMPROGRAM /usr/informix/etc/evidence.sh # System Alarm program path # TBLSPACE_STATS 1 # System Archive Tape Device TAPEDEV /dev/rmt/1c # Tape device path TAPEBLK 64 # Tape block size (Kbytes) TAPESIZE 60000000 # Maximum amount of data to put on tape (Kbytes) # Log Archive Tape Device LTAPEDEV /dev/rmt/0c # Log tape device path LTAPEBLK 64 # Log tape block size (Kbytes) LTAPESIZE 35000000 # Max amount of data to put on log tape (Kbytes) # Optical STAGEBLOB # INFORMIX-OnLine/Optical staging area # System Configuration SERVERNUM 30 # Unique id corresponding to a OnLineinstance DBSERVERNAME wil_online_shm # Name of default database server DBSERVERALIASES wil_online,wil_online_2,wil_online_3 # List of alternate dbservernames NETTYPE ipcshm,3,200,CPU NETTYPE tlitcp,10,250,NET #changed 12/16/1998 tpp #NETTYPE tlitcp,8,250,NET DEADLOCK_TIMEOUT 60 # Max time to wait of lock in distributed env. RESIDENT 1 # Forced residency flag (Yes = 1, No = 0) MULTIPROCESSOR 1 # 0 for single-processor, 1 formulti-processor NUMCPUVPS 8 # Number of user (cpu) vps SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vps toone NOAGE 1 # Process aging AFF_SPROC 1 # Affinity start processor AFF_NPROCS 8 # Affinity number of processors # Shared Memory Parameters @