slow connections

IDS 10.00.FC7 on AIX 5.3 on IBM P550 server Greetings fellow IIUGgers, I'm getting sporadic slowness establishing connections OR complete connection timeouts. Not sure what could cause such behavior, the server is 95% idle. Any recommendations where to look? I can provide other info, just ask. Thanks, --IAM Following errors in online.log: 11:45:33 Checkpoint loguniq 72613, logpos 0x3310, timestamp: 0xa7a36a3c 11:45:33 Maximum server connections 281 11:47:07 listener-thread: err = -25573: oserr = 72: errstr = : Network driver cannot accept a connection on the port. System error = 72. 11:49:16 Logical Log 72612 - Backup Started 11:49:17 Logical Log 72612 - Backup Completed 11:49:36 listener-thread: err = -25573: oserr = 72: errstr = : Network driver cannot accept a connection on the port. System error = 72. 11:49:37 listener-thread: err = -25573: oserr = 72: errstr = : Network driver cannot accept a connection on the port. System error = 72. 11:50:33 Fuzzy Checkpoint Completed: duration was 0 seconds, 347 buffers not flushed. 11:50:33 Checkpoint loguniq 72613, logpos 0x4bd100, timestamp: 0xa7a38347 from /usr/include/sys/errno.h (to define oserr = 72) #define ECONNABORTED 72 /* Software caused connection abort */ from onconfig: # System Configuration SERVERNUM 1 # Unique id corresponding to a OnLine instance DBSERVERNAME biz_prod # Name of default database server DBSERVERALIASES # List of alternate dbservernames NETTYPE soctcp,4,100,NET # Configure poll thread(s) for nettype DEADLOCK_TIMEOUT 60 # Max time to wait of lock in distributed env. RESIDENT 0 # Forced residency flag (Yes = 1, No = 0) MULTIPROCESSOR 1 # 0 for single-processor, 1 for multi-processor NUMCPUVPS 3 # Number of user (cpu) vps SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vps to one NOAGE 1 # Process aging AFF_SPROC 0 # Affinity start processor AFF_NPROCS 0 # Affinity number of processors# Shared Memory Parameters LOCKS 200000 # Maximum number of locks NUMAIOVPS 2 # Number of IO vps PHYSBUFF 32 # Logical log buffer size (Kbytes) CLEANERS 12 # Number of buffer cleaner processes SHMBASE 0x700000010000000L # Shared memory base address SHMVIRTSIZE 386480 # initial virtual shared memory segment size SHMADD 32768 # Size of new shared memory segments (Kbytes) EXTSHMADD 32768 # Size of new extension shared memory segments (Kbytes) SHMTOTAL 0 # Total shared memory (Kbytes). 0=>unlimited SHMVIRT_ALLOCSEG 0 # Values between 0 and .99 are %, values > 1 are CKPTINTVL 300 # Check point interval (in sec) TXTIMEOUT 300 # Transaction timeout (in sec) STACKSIZE 64 # Stack size (Kbytes) last line of onstat -u 221 active, 384 total, 298 maximum concurrent

My idea : - OS level such as kernel parameter, error log from system , i/o busy - Network level such as network card errors , network log For IDS : - Check point duration and check point interval (system will be blocked while doing checkpoint) - try to change NETTYPE soctcp,4,100,NETto NETTYPE soctcp,1,400,NET (or 2,200) because you have only 3VPs , too much poll thread will affect VPs load - Why you use "AFF_NPROCS 0" , try "AFF_NPROCS 3" and ensure that AFF_SPROC is the correct no. Jakkrit A. ________________________________ From: ILKKA MUSTONEN <i.will.sue.you@gmail.com> To: ids@iiug.org Sent: Tuesday, December 2, 2008 1:57:46 AM Subject: slow connections [14174] IDS 10.00.FC7 on AIX 5.3 on IBM P550 server Greetings fellow IIUGgers, I'm getting sporadic slowness establishing connections OR complete connection timeouts. Not sure what could cause such behavior, the server is 95% idle. Any recommendations where to look? I can provide other info, just ask. Thanks, --IAM Following errors in online.log: 11:45:33 Checkpoint loguniq 72613, logpos 0x3310, timestamp: 0xa7a36a3c 11:45:33 Maximum server connections 281 11:47:07 listener-thread: err = -25573: oserr = 72: errstr = : Network driver cannot accept a connection on the port. System error = 72. 11:49:16 Logical Log 72612 - Backup Started 11:49:17 Logical Log 72612 - Backup Completed 11:49:36 listener-thread: err = -25573: oserr = 72: errstr = : Network driver cannot accept a connection on the port. System error = 72. 11:49:37 listener-thread: err = -25573: oserr = 72: errstr = : Network driver cannot accept a connection on the port. System error = 72. 11:50:33 Fuzzy Checkpoint Completed: duration was 0 seconds, 347 buffers not flushed. 11:50:33 Checkpoint loguniq 72613, logpos 0x4bd100, timestamp: 0xa7a38347 from /usr/include/sys/errno.h (to define oserr = 72) #define ECONNABORTED 72 /* Software caused connection abort */ from onconfig: # System Configuration SERVERNUM 1 # Unique id corresponding to a OnLine instance DBSERVERNAME biz_prod # Name of default database server DBSERVERALIASES # List of alternate dbservernames NETTYPE soctcp,4,100,NET # Configure poll thread(s) for nettype DEADLOCK_TIMEOUT 60 # Max time to wait of lock in distributed env. RESIDENT 0 # Forced residency flag (Yes = 1, No = 0) MULTIPROCESSOR 1 # 0 for single-processor, 1 for multi-processor NUMCPUVPS 3 # Number of user (cpu) vps SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vps to one NOAGE 1 # Process aging AFF_SPROC 0 # Affinity start processor AFF_NPROCS 0 # Affinity number of processors# Shared Memory Parameters LOCKS 200000 # Maximum number of locks NUMAIOVPS 2 # Number of IO vps PHYSBUFF 32 # Logical log buffer size (Kbytes) CLEANERS 12 # Number of buffer cleaner processes SHMBASE 0x700000010000000L # Shared memory base address SHMVIRTSIZE 386480 # initial virtual shared memory segment size SHMADD 32768 # Size of new shared memory segments (Kbytes) EXTSHMADD 32768 # Size of new extension shared memory segments (Kbytes) SHMTOTAL 0 # Total shared memory (Kbytes). 0=>unlimited SHMVIRT_ALLOCSEG 0 # Values between 0 and .99 are %, values > 1 are CKPTINTVL 300 # Check point interval (in sec) TXTIMEOUT 300 # Transaction timeout (in sec) STACKSIZE 64 # Stack size (Kbytes) last line of onstat -u 221 active, 384 total, 298 maximum concurrent ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

> My idea : > > - OS level such as kernel parameter, error log from system > , i/o busy > - Network level such as network card errors , network log > > For IDS : > - Check point duration and check point interval (system will be blocked while doing checkpoint) > - try to change > > NETTYPE soctcp,4,100,NET> to > > NETTYPE soctcp,1,400,NET (or 2,200) because you have only 3VPs , too much poll thread will affect VPs load > - Why you use "AFF_NPROCS 0" , try "AFF_NPROCS 3" and ensure that AFF_SPROC is the correct no. > > Jakkrit A. On OS level, which particular kernel parameter, there's quite a few of them? There is nothing in the system errlog. Disk I/O (Ultra SCSI 320) seems fine according to various tools AND other programs/applications on the server run without any hiccups. There are three informix instances on this server, only this one is having OCCASIONAL congestion. When the slowness is occurring, even locally accessing the instance with dbaccess is poor, meanwhile you can get instantaneous connect to the other two instances on the server. Can't find anything wrong with the network either. Here's some stats: ------------------------------------------------------------- ETHERNET STATISTICS (ent0) : Device Type: 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902) Elapsed Time: 14 days 22 hours 41 minutes 1 seconds Transmit Statistics: Receive Statistics: -------------------- ------------------- Packets: 199837212 Packets: 247648973 Bytes: 178970782328 Bytes: 60691832796 Interrupts: 0 Interrupts: 240159233 Transmit Errors: 0 Receive Errors: 0 Packets Dropped: 0 Packets Dropped: 0 Bad Packets: 0 Max Packets on S/W Transmit Queue: 51 S/W Transmit Queue Overflow: 0 Current S/W+H/W Transmit Queue Length: 1 Elapsed Time: 14 days 22 hours 41 minutes 0 seconds Broadcast Packets: 25403 Broadcast Packets: 3102430 Multicast Packets: 0 Multicast Packets: 27979 No Carrier Sense: 0 CRC Errors: 0 DMA Underrun: 0 DMA Overrun: 0 Lost CTS Errors: 0 Alignment Errors: 0 Max Collision Errors: 0 No Resource Errors: 0 Late Collision Errors: 0 Receive Collision Errors: 0 Deferred: 0 Packet Too Short Errors: 0 SQE Test: 0 Packet Too Long Errors: 0 Timeout Errors: 0 Packets Discarded by Adapter: 0 Single Collision Count: 0 Receiver Start Count: 0 Multiple Collision Count: 0 Current HW Transmit Queue Length: 1 General Statistics: ------------------- No mbuf Errors: 0 Adapter Reset Count: 0 Adapter Data Rate: 2000 Driver Flags: Up Broadcast Running Simplex 64BitSupport ChecksumOffload PrivateSegment LargeSend DataRateSet 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902) Specific Statistics: ------------------------------------------------------------------------ Link Status : Up Media Speed Selected: Auto negotiation Media Speed Running: 1000 Mbps Full Duplex PCI Mode: PCI-X (100-133) PCI Bus Width: 64-bit Latency Timer: 144 Cache Line Size: 128 Jumbo Frames: Disabled TCP Segmentation Offload: Enabled TCP Segmentation Offload Packets Transmitted: 20549136 TCP Segmentation Offload Packet Errors: 0 Transmit and Receive Flow Control Status: Disabled Transmit and Receive Flow Control Threshold (High): 45056 Transmit and Receive Flow Control Threshold (Low): 24576 Transmit and Receive Storage Allocation (TX/RX): 16/48 ------------------------------------------------------------- For IDS: Checkpoints (every 5 minutes) are generally 0 seconds. Rarely 1 second. So basically fractions of a second, non-issue. 11:03:26 Fuzzy Checkpoint Completed: duration was 0 seconds, 192 buffers not flushed. 11:08:26 Fuzzy Checkpoint Completed: duration was 0 seconds, 197 buffers not flushed. 11:13:26 Fuzzy Checkpoint Completed: duration was 0 seconds, 197 buffers not flushed. 11:18:26 Fuzzy Checkpoint Completed: duration was 0 seconds, 141 buffers not flushed. 11:23:26 Fuzzy Checkpoint Completed: duration was 0 seconds, 142 buffers not flushed. 11:28:26 Fuzzy Checkpoint Completed: duration was 0 seconds, 175 buffers not flushed. 11:33:26 Fuzzy Checkpoint Completed: duration was 0 seconds, 212 buffers not flushed. 11:38:26 Fuzzy Checkpoint Completed: duration was 0 seconds, 207 buffers not flushed. 11:43:26 Fuzzy Checkpoint Completed: duration was 0 seconds, 184 buffers not flushed. The NETTYPE parameter used to be NETTYPE soctcp,2,250,NET when this problem started. Since then it was changed to NETTYPE soctcp,4,100,NET but that did not make it any better. We haven't configured AFF_NPROCS/AFF_SPROC due to running 3 instances on the server. Any ideas? --IAM

Just my anecdotal analysis: I had a problem once when users called my office complaining that the database was =B3slow=B2. I could find nothing wron= g with my HPUX box, nor with the Informix engine. In fact, the system load was fairly load, the =B3slowness=B2 turned out to be massive amount of network traffic at the Core Switch from users with KaZaa, iTunes, and eDonkey file sharing traffic that was squelching bandwidth on the local network. The sad thing was, the very same users that complained about slowness were the same ones using KaZaa, iTunes, and eDonkey. I=B9m not suggesting that this is causing your specific performance issues, but it probably wouldn=B9t hurt to query your Network Admin(s) to check for unusually levels of LAN activity. **plink-plink** the sound of my 2 cents. Jonathan Smaby Pomona College --- =B3Despite the undeniable facts and indisputable evidence, it=B9s still the fault of the I.T. staff, we just haven=B9t yet figured out how.=B2 From: ILKKA MUSTONEN <i.will.sue.you@gmail.com> Reply-To: <ids@iiug.org> Date: Thu, 4 Dec 2008 12:56:31 -0500 (EST) To: <ids@iiug.org> Subject: Re: slow connections [14218] > My idea :=20 >=20 > - OS level such as kernel parameter, error log from system > , i/o busy=20 > - Network level such as network card errors , network log >=20 > For IDS :=20 > - Check point duration and check point interval (system will be blocked while doing checkpoint) > - try to change=20 >=20 > NETTYPE soctcp,4,100,NET > to=20 >=20> NETTYPE soctcp,1,400,NET (or 2,200) because you have only 3VPs , too much poll thread will affect VPs load > - Why you use "AFF_NPROCS 0" , try "AFF_NPROCS 3" and ensure that AFF_SPR= OC is the correct no.=20 >=20 > Jakkrit A.=20 On OS level, which particular kernel parameter, there's quite a few of them= ? There is nothing in the system errlog. Disk I/O (Ultra SCSI 320) seems fine according to various tools AND other programs/applications on the server run without any hiccups. There are three informix instances on this server, only this one is having OCCASIONAL congestion. When the slowness is occurring, even locally accessing the instance with dbaccess is poor, meanwhile you can get instantaneous connect to the other two instances on the server. Can't find anything wrong with the network either. Here's some stats: ------------------------------------------------------------- ETHERNET STATISTICS (ent0) : Device Type: 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902) Elapsed Time: 14 days 22 hours 41 minutes 1 seconds Transmit Statistics: Receive Statistics: -------------------- ------------------- Packets: 199837212 Packets: 247648973 Bytes: 178970782328 Bytes: 60691832796 Interrupts: 0 Interrupts: 240159233 Transmit Errors: 0 Receive Errors: 0 Packets Dropped: 0 Packets Dropped: 0 Bad Packets: 0=20 Max Packets on S/W Transmit Queue: 51 S/W Transmit Queue Overflow: 0 Current S/W+H/W Transmit Queue Length: 1 Elapsed Time: 14 days 22 hours 41 minutes 0 seconds Broadcast Packets: 25403 Broadcast Packets: 3102430 Multicast Packets: 0 Multicast Packets: 27979 No Carrier Sense: 0 CRC Errors: 0 DMA Underrun: 0 DMA Overrun: 0 Lost CTS Errors: 0 Alignment Errors: 0 Max Collision Errors: 0 No Resource Errors: 0 Late Collision Errors: 0 Receive Collision Errors: 0 Deferred: 0 Packet Too Short Errors: 0 SQE Test: 0 Packet Too Long Errors: 0 Timeout Errors: 0 Packets Discarded by Adapter: 0 Single Collision Count: 0 Receiver Start Count: 0 Multiple Collision Count: 0 Current HW Transmit Queue Length: 1 General Statistics: ------------------- No mbuf Errors: 0=20 Adapter Reset Count: 0 Adapter Data Rate: 2000 Driver Flags: Up Broadcast Running Simplex 64BitSupport ChecksumOffload PrivateSegment LargeSend DataRateSet 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902) Specific Statistics: ------------------------------------------------------------------------ Link Status : Up=20 Media Speed Selected: Auto negotiation Media Speed Running: 1000 Mbps Full Duplex PCI Mode: PCI-X (100-133) PCI Bus Width: 64-bit Latency Timer: 144=20 Cache Line Size: 128 Jumbo Frames: Disabled TCP Segmentation Offload: Enabled TCP Segmentation Offload Packets Transmitted: 20549136 TCP Segmentation Offload Packet Errors: 0 Transmit and Receive Flow Control Status: Disabled Transmit and Receive Flow Control Threshold (High): 45056 Transmit and Receive Flow Control Threshold (Low): 24576 Transmit and Receive Storage Allocation (TX/RX): 16/48 ------------------------------------------------------------- For IDS:=20 Checkpoints (every 5 minutes) are generally 0 seconds. Rarely 1 second. So basically fractions of a second, non-issue. 11:03:26 Fuzzy Checkpoint Completed: duration was 0 seconds, 192 buffers no= t flushed.=20 11:08:26 Fuzzy Checkpoint Completed: duration was 0 seconds, 197 buffers no= t flushed.=20 11:13:26 Fuzzy Checkpoint Completed: duration was 0 seconds, 197 buffers no= t flushed.=20 11:18:26 Fuzzy Checkpoint Completed: duration was 0 seconds, 141 buffers no= t flushed.=20 11:23:26 Fuzzy Checkpoint Completed: duration was 0 seconds, 142 buffers no= t flushed.=20 11:28:26 Fuzzy Checkpoint Completed: duration was 0 seconds, 175 buffers no= t flushed.=20 11:33:26 Fuzzy Checkpoint Completed: duration was 0 seconds, 212 buffers no= t flushed.=20 11:38:26 Fuzzy Checkpoint Completed: duration was 0 seconds, 207 buffers no= t flushed.=20 11:43:26 Fuzzy Checkpoint Completed: duration was 0 seconds, 184 buffers no= t flushed.=20 The NETTYPE parameter used to be NETTYPE soctcp,2,250,NET when this problem started.=20 Since then it was changed to NETTYPE soctcp,4,100,NET but that did not make it=20 any better.=20 We haven't configured AFF_NPROCS/AFF_SPROC due to running 3 instances on th= e server.=20 Any ideas?=20 --IAM=20 ***************************************************************************= * ***=20 Forum Note: Use "Reply" to post a response in the discussion forum. ------------------------------------------------------------- This message has been scanned by Postini anti-virus software. =0D