Re: Intermittent Performance Problem - What does oninit do?

On Mon, 08 Mar 2004 09:29:54 -0500, Joel wrote: > We have Informix V7.30 running on a SCO-Unix box. Several times during the > day, the response of the system goes in the tank. At these times, sar > indicates that there is 0% idle time. > > Our DB has around 2.5 million rows across about 30 tables. There is another > SCO-Unix box on the network with another Informix DB, and we use synonyms to > facilitate table reads across the network. > > I don't have a good tool to view the running processes, so I use ps -ef to > create an output file of the processes, wait several seconds, and then do > another ps -ef to a different output file. Using diff on the two files, > always indicates that oninit is the process getting the most time. Post your ONCONFIG file, a description of your disk farm, whether you are using raw devices or cooked files for chunks, the elapsed time since onstat -z was last run, and the following onstat output and we'll take a look: onstat -d onstat -D onstat -p onstat -P (don't remember if this one is in 7.30 or appeared in 7.31) onstat -g glo onstat -g iov onstat -g iof Art S. Kagel

Here's the info you requested Art. If you need more, please let me know. Thanks for the help. Joel onconfig file #************************************************************************** # # INFORMIX SOFTWARE, INC. # # Title: onconfig.std # Description: Informix Dynamic Server Configuration Parameters # #************************************************************************** # Root Dbspace Configuration ROOTNAME rootdbs # Root dbspace name ROOTPATH /dev/lrootdbs # Path for device containing root dbspace ROOTOFFSET 0 # Offset of root dbspace into device (Kbytes) ROOTSIZE 100000 # Size of root dbspace (Kbytes) # Disk Mirroring Configuration Parameters MIRROR 1 # Mirroring flag (Yes = 1, No = 0) MIRRORPATH # Path for device containing mirrored root MIRROROFFSET 0 # Offset into mirrored device (Kbytes) # Physical Log Configuration PHYSDBS rootdbs # Location (dbspace) of physical log PHYSFILE 30000 # Physical log file size (Kbytes) # Logical Log Configuration LOGFILES 200 # Number of logical log files LOGSIZE 2500 # Logical log size (Kbytes) # Diagnostics MSGPATH /informix/online.log # System message log file path CONSOLE /dev/console # System console message path ALARMPROGRAM /informix/etc/no_log.sh # Alarm program pathSYSALARMPROGRAM /informix/etc/evidence.sh # System Alarm program path TBLSPACE_STATS 1 # System Archive Tape Device #TAPEDEV /dev/null # Tape device path TAPEDEV /dev/rmt/0b # Tape device path TAPEBLK 128 # Tape block size (Kbytes) TAPESIZE 34500000 # Maximum amount of data to put on tape (Kbytes) # Log Archive Tape Device LTAPEDEV /dev/null # Log tape device path LTAPEBLK 16 # Log tape block size (Kbytes) LTAPESIZE 10240 # Max amount of data to put on log tape (Kbytes) # Optical STAGEBLOB # Informix Dynamic Server/Optical staging area # System Configuration SERVERNUM 2 # Unique id corresponding to a Dynamic Server instance DBSERVERNAME srswms_shm # Name of default database server DBSERVERALIASES srswms_net # List of alternate dbservernames NETTYPE ipcshm,1,200,CPU # Configure poll thread(s) for nettype NETTYPE tlitcp,1,150,NET # Configure poll thread(s) for nettype DEADLOCK_TIMEOUT 60 # Max time to wait of lock in distributed env. RESIDENT 1 # Forced residency flag (Yes = 1, No = 0) MULTIPROCESSOR 0 # 0 for single-processor, 1 for multi-processor NUMCPUVPS 1 # Number of user (cpu) vps SINGLE_CPU_VP 1 # If non-zero, limit number of cpu vps to one NOAGE 1 # Process aging AFF_SPROC 0 # Affinity start processor AFF_NPROCS 0 # Affinity number of processors # Shared Memory Parameters LOCKS 200000 # Maximum number of locks BUFFERS 30000 # Maximum number of shared buffers NUMAIOVPS 4 # Number of IO vps PHYSBUFF 32 # Physical log buffer size (Kbytes) LOGBUFF 32 # Logical log buffer size (Kbytes)LOGSMAX 200 # Maximum number of logical log files CLEANERS 8 # Number of buffer cleaner processes SHMBASE 0x82000000 # Shared memory base address SHMVIRTSIZE 64000 # initial virtual shared memory segment size SHMADD 32000 # Size of new shared memory segments (Kbytes) SHMTOTAL 0 # Total shared memory (Kbytes). 0=>unlimited CKPTINTVL 300 # Check point interval (in sec) LRUS 8 # Number of LRU queues LRU_MAX_DIRTY 60 # LRU percent dirty begin cleaning limit LRU_MIN_DIRTY 50 # LRU percent dirty end cleaning limit LTXHWM 50 # Long transaction high water mark percentage LTXEHWM 60 # Long transaction high water mark (exclusive) TXTIMEOUT 0x12c # Transaction timeout (in sec) STACKSIZE 32 # Stack size (Kbytes) # System Page Size # BUFFSIZE - Dynamic Server no longer supports this configuration parameter. # To determine the page size used by Dynamic Server on your platform # see the last line of output from the command, 'onstat -b'. # Recovery Variables # OFF_RECVRY_THREADS: # Number of parallel worker threads during fast recovery or an offline restore. # ON_RECVRY_THREADS: # Number of parallel worker threads during an online restore. OFF_RECVRY_THREADS 10 # Default number of offline worker threads ON_RECVRY_THREADS 1 # Default number of online worker threads # Data Replication Variables # DRAUTO: 0 manual, 1 retain type, 2 reverse type DRAUTO 0 # DR automatic switchover DRINTERVAL 30 # DR max time between DR buffer flushes (in sec) DRTIMEOUT 30 # DR network timeout (in sec)DRLOSTFOUND /informix/etc/dr.lostfound # DR lost+found file path # CDR Variables CDR_LOGBUFFERS 2048 # size of log reading buffer pool (Kbytes) CDR_EVALTHREADS 1,2 # evaluator threads (per-cpu-vp,additional) CDR_DSLOCKWAIT 5 # DS lockwait timeout (seconds) CDR_QUEUEMEM 4096 # Maximum amount of memory for any CDR queue (Kb ytes) # Backup/Restore variables BAR_ACT_LOG /tmp/bar_act.log BAR_MAX_BACKUP 0 BAR_RETRY 1 BAR_NB_XPORT_COUNT 10 BAR_XFER_BUF_SIZE 31 # Informix Storage Manager variables ISM_DATA_POOL ISMData # If the data pool name is changed, be sure to # update $INFORMIXDIR/bin/onbar. Change to # ism_catalog -create_bootstrap -pool <new name> ISM_LOG_POOL ISMLogs # Read Ahead Variables RA_PAGES 10 # Number of pages to attempt to read ahead RA_THRESHOLD 4 # Number of pages left before next group # DBSPACETEMP: # Dynamic Server equivalent of DBTEMP for SE. This is the list of dbspaces # that the Dynamic Server SQL Engine will use to create temp tables etc. # If specified it must be a colon separated list of dbspaces that exist # when the Dynamic Server system is brought online. If not specified, or if # all dbspaces specified are invalid, various ad hoc queries will create # temporary files in /tmp instead. DBSPACETEMP # Default temp dbspaces # DUMP*: # The following parameters control the type of diagnostics information which # is preserved when an unanticipated error condition (assertion failure) occurs # during Dynamic Server operations. # For DUMPSHMEM, DUMPGCORE and DUMPCORE 1 means Yes, 0 means No. DUMPDIR

On Fri, 12 Mar 2004 12:36:43 -0500, Joel wrote: > Here's the info you requested Art. If you need more, please let me know. > > Thanks for the help. > > Joel <SNIP> Snipped it all as it's hard to read anyway. Looks like you are suffering from buffer and/or LRU queue contention. with only 30000 buffers and close to a million read requests in the 150 minutes between stats likely it's both. I'd increase BUFFERS as much as your memory can handle. I'd want about 200000 buffers for a system that can get that busy. Increase LRUS and CLEANERS from 8 each to 128 (or about the number of concurrent users if less than 128, but avoid 32 and 64 which trigger an old unfixed bug). See what happens after that. Also, either you are using COOKED chunks or KAIO is not enabled of available on your system. Increase NUMAIOVPS from 4 to about 12. You want to see at least one of them with a value in io/wup < 1.0 in the onstat -g iov report, so play until you get that. Art S. Kagel