Re: Performance issues on HP-UX

Mike Peters said: > > We are currently suffering performance problems with our Informix > server running on HP-UX. The server appears to be paging heavily with > sar -q showing %swpocc running at or close to 100. > > The server is an HP 9000/800 with 1Gb RAM, running: > IBM Informix Dynamic Server Version 7.31.FD8 > HPUX Version B.11.11 > > At startup the online.log shows: > > 13:25:00 Segment locked: addr=0x5c5000, size=411049984 > 13:25:00 Segment locked: addr=0x18dc7000, size=115343360 > 13:25:01 The number of configured CPU poll threads exceeds > 'NUMCPUVPS'. > NETTYPE 'live_sm' poll threads started on NET VPs. > > In our onconfig we have: > > NETTYPE ipcshm,2,150,CPU > NETTYPE soctcp,1,100,NET > > NUMCPUVPS 1> >>From the error message and googling I suppose we need to set NUMCPUVPS > to 2 (it is a dual processor machine). Some sources suggest setting the > vp class value to NET on a multi-processor machine. Is that the way to > go or should we stick with CPU? > > Additionally we are seeing occasional Network recieve errors in the > online.log and clients' telnet sessions are dropping with TCP errors. > However the dropped sessions do not always correspond to the Network > recieve errors in the logs: > > 12:00:37 listener-thread: err = -25587: oserr = 0: errstr = : Network > receive failed. > > Finally we are also seeing: > > 11:56:41 shmctl: errno = 12 > > 11:56:41 Shared memory segment 0xc00000001ffaf000 could not be forced > resident. > 11:57:03 hpkaioaddseg: ASYNC_ADDSEG failed, errno = 11 Resource > temporarily unavailable > 11:57:05 shmctl: errno = 12 > > With regards the latter we have tried upping IFMX_HPKAIO_NUM_REQ to > 3000 but that does not appear to have helped. Can anyone give us any > pointers to resolving these issues? Although performance is an issue, > the most urgent thing for us at the moment is stop the dropped > sessions. Our onconfig follows: > > # Root Dbspace Configuration > > ROOTNAME rootdbs # Root dbspace name > ROOTPATH /opt/informix/chunks/c_root> # Path for device containing root > dbspace > ROOTOFFSET 0 # Offset of root dbspace into device > (Kbytes) > ROOTSIZE 204800 # Size of root dbspace (Kbytes)> > # Disk Mirroring Configuration Parameters > > MIRROR 0 # Mirroring flag (Yes = 1, No = 0) > MIRRORPATH # Path for device containing mirrored> root > MIRROROFFSET 0 # Offset into mirrored device (Kbytes)> > # Physical Log Configuration > > PHYSDBS dbs_plog # Location (dbspace) of physical log > PHYSFILE 24400 # Physical log file size (Kbytes)> > # Logical Log Configuration > > LOGFILES 50 # Number of logical log files > LOGSIZE 4000 # Logical log size (Kbytes)> > # Diagnostics > > MSGPATH /var/informix/logs/online.log # System message log file > path > CONSOLE /dev/console # System console message path > ALARMPROGRAM /opt/informix/current/etc/log_full.sh # Alarm program> path > SYSALARMPROGRAM /opt/informix/current/etc/evidence.sh > # System Alarm program path > TBLSPACE_STATS 1> > # System Archive Tape Device > > TAPEDEV /dev/rmt/1m # Tape device path > #TAPEDEV /dev/rmt/1m # Tape device path > TAPEBLK 128 # Tape block size (Kbytes) > TAPESIZE 10485760 # Maximum amount of data to put on tape > (Kbytes)> > # Log Archive Tape Device > > LTAPEDEV /dev/null # Log tape device path > LTAPEBLK 16 # Log tape block size (Kbytes) > LTAPESIZE 10240 # Max amount of data to put on log tape > (Kbytes)> > # Optical > > STAGEBLOB # Informix Dynamic Server/Optical > staging area > > # System Configuration > > SERVERNUM 3 # Unique id corresponding to a Dynamic> Server instance > DBSERVERNAME live_sm # Name of default database server > DBSERVERALIASES live_tcp # List of alternate dbservernames > NETTYPE ipcshm,2,150,CPU # Override sqlhosts nettype parameters > NETTYPE soctcp,1,100,NET # Override sqlhosts nettype parameters > > DEADLOCK_TIMEOUT 60 # Max time to wait of lock in> distributed env. > RESIDENT -1 # Forced residency flag (Yes = 1, No = > 0, -1 = Yes and merge) > > MULTIPROCESSOR 1 # 0 for single-processor, 1 for> multi-processor > NUMCPUVPS 1 # Number of user (cpu) vps > SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vps> to one > > NOAGE 1 # Process aging > AFF_SPROC 0 # Affinity start processor > AFF_NPROCS 0 # Affinity number of processors> > # Shared Memory Parameters > > LOCKS 500000 # Maximum number of locks > BUFFERS 163840 # Maximum number of shared buffers > (Pages) > NUMAIOVPS 4 # Number of IO vps > PHYSBUFF 32 # Physical log buffer size (Kbytes) > LOGBUFF 32 # Logical log buffer size (Kbytes)> LOGSMAX 50 # Maximum number of logical log files > CLEANERS 58 # Number of buffer cleaner processes > SHMBASE 0x0 # Shared memory base address > SHMVIRTSIZE 112640 # initial virtual shared memory segment> size > SHMADD 81920 # Size of new shared memory segments > (Kbytes) > SHMTOTAL 0 # Total shared memory (Kbytes). > 0=>unlimited > CKPTINTVL 150 # Check point interval (in sec) > LRUS 57 # Number of LRU queues > LRU_MAX_DIRTY 10 # LRU percent dirty begin cleaning> limit > LRU_MIN_DIRTY 5 # LRU percent dirty end cleaning limit > LTXHWM 50 # Long transaction high water mark> percentage > LTXEHWM 60 # Long transaction high water mark > (exclusive) > TXTIMEOUT 0x12c # Transaction timeout (in sec) > STACKSIZE 32 # Stack size (Kbytes)> > # System Page Size > # BUFFSIZE - Dynamic Server no longer supports this configuration > parameter. > # To determine the page size used by Dynamic Server on your > platform > # see the last line of output from the command, 'onstat -b'. > > > # Recovery Variables > # OFF_RECVRY_THREADS: > # Number of parallel worker threads during fast recovery or an offline > restore. > # ON_RECVRY_THREADS: > # Number of parallel worker threads during an online restore. > > OFF_RECVRY_THREADS 10 # Default number of offline worker> threads > ON_RECVRY_THREADS 1 # Default number of online worker> threads > > # Data Replication Var

Obnoxio The Clown wrote: > Mike Peters said: > >>We are currently suffering performance problems with our Informix >>server running on HP-UX. The server appears to be paging heavily with >>sar -q showing %swpocc running at or close to 100. >> Buy some memory then :P >>The server is an HP 9000/800 with 1Gb RAM, running: >>IBM Informix Dynamic Server Version 7.31.FD8 >>HPUX Version B.11.11 >> >>At startup the online.log shows: >> >> 13:25:00 Segment locked: addr=0x5c5000, size=411049984 >> 13:25:00 Segment locked: addr=0x18dc7000, size=115343360 >> 13:25:01 The number of configured CPU poll threads exceeds>>'NUMCPUVPS'. >> NETTYPE 'live_sm' poll threads started on NET VPs. >> >>In our onconfig we have: >> >> NETTYPE ipcshm,2,150,CPU >> NETTYPE soctcp,1,100,NET >> >> NUMCPUVPS 1>> >>>From the error message and googling I suppose we need to set NUMCPUVPS >>to 2 (it is a dual processor machine). Some sources suggest setting the >>vp class value to NET on a multi-processor machine. Is that the way to >>go or should we stick with CPU? >> Well, all it is complaining about is the (pointless) setting for 2 CPU poll threads for shared memory connection. I suggest you change this to : NETTYPE ipcshm,1,150,CPU and magically, the error message will go away :O >>Additionally we are seeing occasional Network recieve errors in the >>online.log and clients' telnet sessions are dropping with TCP errors. >>However the dropped sessions do not always correspond to the Network >>recieve errors in the logs: >> >> 12:00:37 listener-thread: err = -25587: oserr = 0: errstr = : Network>>receive failed. >> Well, if telnet AND connections to the database server are failing, you are flooding your tcp level. There are some kernel parameters you could muck around with, but all that will do is increase the outstanding queue length that can sit on the card. You could try increasing the number of poll threads for the tcp connections : NETTYPE soctcp,2,250,NET but ... I think you have a "slow card" :P >>Finally we are also seeing: >> >> 11:56:41 shmctl: errno = 12 >> >> 11:56:41 Shared memory segment 0xc00000001ffaf000 could not be forced >>resident. >> 11:57:03 hpkaioaddseg: ASYNC_ADDSEG failed, errno = 11 Resource>>temporarily unavailable >> 11:57:05 shmctl: errno = 12>> >>With regards the latter we have tried upping IFMX_HPKAIO_NUM_REQ to >>3000 but that does not appear to have helped. Well, that is basically you running out of available memory to force resident. Looks like you have allocated a load of extra virtual segments (check with onstat -g seg). You have the instance configured to start with about 550 Mb of shared memory, and then the instance is allocating some more and then ... "other things" are using the rest :P >> Can anyone give us any >>pointers to resolving these issues? Although performance is an issue, >>the most urgent thing for us at the moment is stop the dropped >>sessions. <snip> > > Upon reading your post, I must congratulate you on having grasped at the > largest number of inappropriate or irrelevant straws for a given problem > that I have ever seen! > > I must also point out that my laptop has twice as much memory as your HP > server, and that your server is rather old, to say the least. > > While you have also indicated some very useful information, it would be > useful to know how many users you have connected to this server? > What sort of connections are being dropped. Are they batch processes? > Large numbers of concurrent session connections? > Why is your read-ahead configured so high? > What kind of application is this? > Have you amended your kernel configuration in line with the release notes? > > Most importantly, have you taken this up with Tech Support? >