checkpoints Excessively slow

hi all we have IDS 7.31 UC4 on unix with 2GB Memory Since yesterday the system has become too slow with excessive checkpoints: 01:04:38 Checkpoint Completed: duration was 5439 seconds. 01:52:52 Checkpoint Completed: duration was 1931 seconds. 04:06:37 Checkpoint Completed: duration was 7724 seconds. there is some parametrs in onconfig : # Physical Log Configuration PHYSDBS rootdbs # Location (dbspace) of physical log PHYSFILE 50000 # Physical log file size (Kbytes) # Logical Log Configuration LOGFILES 24 # Number of logical log files LOGSIZE 20000 # Logical log size (Kbytes) LOCKS 200000 # Maximum number of locks BUFFERS 150000 # Maximum number of shared buffers NUMAIOVPS 6 # Number of IO vps PHYSBUFF 128 # Physical log buffer size (Kbytes) LOGBUFF 128 # Logical log buffer size (Kbytes)LOGSMAX 128 # Maximum number of logical log files CLEANERS 128 # Number of buffer cleaner processes SHMBASE 0x10000000 # Shared memory base address SHMVIRTSIZE 150000 SHMADD 32768 # Size of new shared memory segments (Kbytes) SHMTOTAL 1500000 # Total shared memory (Kbytes). 0=>unlimited CKPTINTVL 300 # Check point interval (in sec) LRUS 128 # Number of LRU queues LRU_MAX_DIRTY 3 # LRU percent dirty begin cleaning limit LRU_MIN_DIRTY 1 # LRU percent dirty end cleaning limit LTXHWM 50 # Long transaction high water mark percentage LTXEHWM 60 # Long transaction high water mark (exclusive) TXTIMEOUT 0x12c # Transaction timeout (in sec) STACKSIZE 128 # Stack size (Kbytes) Please help us to resolve this problem thanks,

Hi, Have you checked the devices where IDS's chunks are put ? No system error ? No message in the online.log file ? Did your application has been upgraded ? "ABDESLEM ARBAOUI" <iiug.box@laposte.net> Sent by: ids-bounces@iiug.org 02/08/2008 04:16 PM Please respond to ids@iiug.org To ids@iiug.org cc Subject checkpoints Excessively slow [11247] hi all we have IDS 7.31 UC4 on unix with 2GB Memory Since yesterday the system has become too slow with excessive checkpoints: 01:04:38 Checkpoint Completed: duration was 5439 seconds. 01:52:52 Checkpoint Completed: duration was 1931 seconds. 04:06:37 Checkpoint Completed: duration was 7724 seconds. there is some parametrs in onconfig : # Physical Log Configuration PHYSDBS rootdbs # Location (dbspace) of physical log PHYSFILE 50000 # Physical log file size (Kbytes) # Logical Log Configuration LOGFILES 24 # Number of logical log files LOGSIZE 20000 # Logical log size (Kbytes) LOCKS 200000 # Maximum number of locks BUFFERS 150000 # Maximum number of shared buffers NUMAIOVPS 6 # Number of IO vps PHYSBUFF 128 # Physical log buffer size (Kbytes) LOGBUFF 128 # Logical log buffer size (Kbytes)LOGSMAX 128 # Maximum number of logical log files CLEANERS 128 # Number of buffer cleaner processes SHMBASE 0x10000000 # Shared memory base address SHMVIRTSIZE 150000 SHMADD 32768 # Size of new shared memory segments (Kbytes) SHMTOTAL 1500000 # Total shared memory (Kbytes). 0=>unlimited CKPTINTVL 300 # Check point interval (in sec) LRUS 128 # Number of LRU queues LRU_MAX_DIRTY 3 # LRU percent dirty begin cleaning limit LRU_MIN_DIRTY 1 # LRU percent dirty end cleaning limit LTXHWM 50 # Long transaction high water mark percentage LTXEHWM 60 # Long transaction high water mark (exclusive) TXTIMEOUT 0x12c # Transaction timeout (in sec) STACKSIZE 128 # Stack size (Kbytes) Please help us to resolve this problem thanks, ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum. See you at the IIUG Informix 2008 Conference The Power Conference for Informix Professionals April 27 - 30, 2008 Marriott Overland Park (Kansas City), Kansas http://www.iiug.org/conf Registration Now Open!!

ABDESLEM ARBAOUI wrote: > hi all > > we have IDS 7.31 UC4 on unix with 2GB Memory > Platform info will normally help also: Sun, AIX, HP-PA, HP-Itanium, x86 Linux? First thing I'm going to suggest is an upgrade at least to 7.31UD8 if not going whole hog and moving to 11.10! Beyond that, I would seriously consider contracting with a well known Informix consultancy. There are several excellent firms in the US (looks like the US from you email address), the UK, and even one or two on mainland Europe among them Oninit (www.oninit.com) with which I am associated and which has a presence on both sides of the Atlantic. You need someone who can monitor your server over time. Checkpoint times that severe, especially when they are not normal, have a root cause that usually has little to do with tuning your IDS instance. Only live local or remote monitoring will route out the trouble. Art S. Kagel Oninit > Since yesterday the system has become too slow with excessive checkpoints: > > 01:04:38 Checkpoint Completed: duration was 5439 seconds. > 01:52:52 Checkpoint Completed: duration was 1931 seconds. > 04:06:37 Checkpoint Completed: duration was 7724 seconds. > > there is some parametrs in onconfig : > > # Physical Log Configuration > > PHYSDBS rootdbs # Location (dbspace) of physical log > PHYSFILE 50000 # Physical log file size (Kbytes)> > # Logical Log Configuration > > LOGFILES 24 # Number of logical log files > LOGSIZE 20000 # Logical log size (Kbytes) > > LOCKS 200000 # Maximum number of locks > BUFFERS 150000 # Maximum number of shared buffers > NUMAIOVPS 6 # Number of IO vps > PHYSBUFF 128 # Physical log buffer size (Kbytes) > LOGBUFF 128 # Logical log buffer size (Kbytes)> LOGSMAX 128 # Maximum number of logical log files > CLEANERS 128 # Number of buffer cleaner processes > SHMBASE 0x10000000 # Shared memory base address > SHMVIRTSIZE 150000 > SHMADD 32768 # Size of new shared memory segments (Kbytes) > SHMTOTAL 1500000 # Total shared memory (Kbytes). 0=>unlimited > CKPTINTVL 300 # Check point interval (in sec) > LRUS 128 # Number of LRU queues > LRU_MAX_DIRTY 3 # LRU percent dirty begin cleaning limit > LRU_MIN_DIRTY 1 # LRU percent dirty end cleaning limit > LTXHWM 50 # Long transaction high water mark percentage > LTXEHWM 60 # Long transaction high water mark (exclusive) > TXTIMEOUT 0x12c # Transaction timeout (in sec) > STACKSIZE 128 # Stack size (Kbytes)> > Please help us to resolve this problem > thanks, > ================================================================================ =========== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.oninit.com/home/disclaimer.php ================================================================================ ===========

Check with the people who have command line access to see if anyone was doing any data loads or data conversions. If those were the only incidents, and if there were embedded with numerous logical logs being filled, one of those two circumstances is likely the cause. Take care. Clifton M. Bean Informix DBA / AIX System Admin Currency Technics & Metrics Phone: (972) 812-1411 x244 -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of ABDESLEM ARBAOUI Sent: Friday, February 08, 2008 9:17 AM To: ids@iiug.org Subject: checkpoints Excessively slow [11247] hi all we have IDS 7.31 UC4 on unix with 2GB Memory Since yesterday the system has become too slow with excessive checkpoints: 01:04:38 Checkpoint Completed: duration was 5439 seconds. 01:52:52 Checkpoint Completed: duration was 1931 seconds. 04:06:37 Checkpoint Completed: duration was 7724 seconds. there is some parametrs in onconfig : # Physical Log Configuration PHYSDBS rootdbs # Location (dbspace) of physical log PHYSFILE 50000 # Physical log file size (Kbytes) # Logical Log Configuration LOGFILES 24 # Number of logical log files LOGSIZE 20000 # Logical log size (Kbytes) LOCKS 200000 # Maximum number of locks BUFFERS 150000 # Maximum number of shared buffers NUMAIOVPS 6 # Number of IO vps PHYSBUFF 128 # Physical log buffer size (Kbytes) LOGBUFF 128 # Logical log buffer size (Kbytes)LOGSMAX 128 # Maximum number of logical log files CLEANERS 128 # Number of buffer cleaner processes SHMBASE 0x10000000 # Shared memory base address SHMVIRTSIZE 150000 SHMADD 32768 # Size of new shared memory segments (Kbytes) SHMTOTAL 1500000 # Total shared memory (Kbytes). 0=>unlimited CKPTINTVL 300 # Check point interval (in sec) LRUS 128 # Number of LRU queues LRU_MAX_DIRTY 3 # LRU percent dirty begin cleaning limit LRU_MIN_DIRTY 1 # LRU percent dirty end cleaning limit LTXHWM 50 # Long transaction high water mark percentage LTXEHWM 60 # Long transaction high water mark (exclusive) TXTIMEOUT 0x12c # Transaction timeout (in sec) STACKSIZE 128 # Stack size (Kbytes) Please help us to resolve this problem thanks, **************************************************************************** *** Forum Note: Use "Reply" to post a response in the discussion forum. See you at the IIUG Informix 2008 Conference The Power Conference for Informix Professionals April 27 - 30, 2008 Marriott Overland Park (Kansas City), Kansas http://www.iiug.org/conf Registration Now Open!!

thanks for your response there is a fragment on file online.log Fri Feb 8 00:47:44 2008 00:47:44 listener-thread: err = -27001: oserr = 0: errstr = : Read error occurred during connection attempt. 01:04:38 Checkpoint Completed: duration was 5439 seconds. 01:07:00 Logical Log 216205 Complete. 01:07:03 Process exited with return code 139: /bin/sh /bin/sh -c /home/infx7/etc/log_full.sh 2 23 "Logical Log 216205 Complete." "L ogical Log 216205 Complete." 01:09:38 KAIO: out of OS resources, errno = 11, pid = 20801 01:09:38 KAIO: out of OS resources, errno = 11, pid = 20799 01:09:38 KAIO: out of OS resources, errno = 11, pid = 20802 01:09:38 KAIO: out of OS resources, errno = 11, pid = 20791

ABDESLEM ARBAOUI wrote: > thanks for your response > there is a fragment on file online.log > > Actually, if that's the only problem, you have to tune your OS Kernel Asynchronous IO to present more resources to IDS. Sounds like you are on AIX or HPUX. Don't know the details of tuning those myself, but your System Administrator should. Art S. Kagel Oninit > Fri Feb 8 00:47:44 2008 > > 00:47:44 listener-thread: err = -27001: oserr = 0: errstr = : Read error > occurred during connection attempt. > > 01:04:38 Checkpoint Completed: duration was 5439 seconds. > 01:07:00 Logical Log 216205 Complete. > 01:07:03 Process exited with return code 139: /bin/sh /bin/sh -c > /home/infx7/etc/log_full.sh 2 23 "Logical Log 216205 Complete." "L > ogical Log 216205 Complete." > 01:09:38 KAIO: out of OS resources, errno = 11, pid = 20801 > 01:09:38 KAIO: out of OS resources, errno = 11, pid = 20799 > 01:09:38 KAIO: out of OS resources, errno = 11, pid = 20802 > 01:09:38 KAIO: out of OS resources, errno = 11, pid = 20791 > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > See you at the IIUG Informix 2008 Conference > The Power Conference for Informix Professionals > April 27 - 30, 2008 Marriott Overland Park (Kansas City), Kansas > http://www.iiug.org/conf > Registration Now Open!! > > ================================================================================ =========== > Please access the attached hyperlink for an important electronic communications disclaimer: > > http://www.oninit.com/home/disclaimer.php > > ================================================================================ =========== > > ================================================================================ =========== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.oninit.com/home/disclaimer.php ================================================================================ ===========

KAIO: out of OS resources, errno = 11, pid = 20801 You have some problems with Kernel Asynchronous I/O... Check in the /usr/include/errno.h to see what is the error 11 and contact your system administrator in order to looking for error messages in the system logs. ________________________________ "ABDESLEM ARBAOUI" <iiug.box@laposte.net> Sent by: ids-bounces@iiug.org 02/08/2008 05:31 PM Please respond to ids@iiug.org To ids@iiug.org cc Subject Re: checkpoints Excessively slow [11251] thanks for your response there is a fragment on file online.log Fri Feb 8 00:47:44 2008 00:47:44 listener-thread: err = -27001: oserr = 0: errstr = : Read error occurred during connection attempt. 01:04:38 Checkpoint Completed: duration was 5439 seconds. 01:07:00 Logical Log 216205 Complete. 01:07:03 Process exited with return code 139: /bin/sh /bin/sh -c /home/infx7/etc/log_full.sh 2 23 "Logical Log 216205 Complete." "L ogical Log 216205 Complete." 01:09:38 KAIO: out of OS resources, errno = 11, pid = 20801 01:09:38 KAIO: out of OS resources, errno = 11, pid = 20799 01:09:38 KAIO: out of OS resources, errno = 11, pid = 20802 01:09:38 KAIO: out of OS resources, errno = 11, pid = 20791 ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum. See you at the IIUG Informix 2008 Conference The Power Conference for Informix Professionals April 27 - 30, 2008 Marriott Overland Park (Kansas City), Kansas http://www.iiug.org/conf Registration Now Open!!

U wrote: > 01:04:38 Checkpoint Completed: duration was 5439 seconds. > 01:07:00 Logical Log 216205 Complete. > 01:07:03 Process exited with return code 139: /bin/sh /bin/sh -c > /home/infx7/etc/log_full.sh 2 23 "Logical Log 216205 Complete." "L > ogical Log 216205 Complete." > 01:09:38 KAIO: out of OS resources, errno = 11, pid = 20801 > 01:09:38 KAIO: out of OS resources, errno = 11, pid = 20799 > 01:09:38 KAIO: out of OS resources, errno = 11, pid = 20802 > 01:09:38 KAIO: out of OS resources, errno = 11, pid = 20791 For HPUX before regening the kernal you might try putting this in the Informix .profile export IFMX_HPKAIO_NUM_REQ=2000 # or some other value You will need to restart the instance to see the result.

What OS are you on? on AIX you might try to set this environment variable IFMX_AIXKAIO_NUM_= REQ by exporting it. This may be set to the default value right now, try increase it to 4k (4096) or 8k (8192) whichever resolves your issue export IFMX_AIXKAIO_NUM_REQ=3D4096 ON HP Unix IFMX_HPKAIO_NUM_REQ An environment variable IFMX_HPKAIO_NUM_REQ is provided to specify the number of simultaneous KAIO requests that are processed by the server a= t one time. The value can be set in the range of 10 to 5000, the default being 1000. If the error Else you can tune at the OS level by tuning the kernel jp = "ABDESLEM = ARBAOUI" = <iiug.box@laposte = To .net> ids@iiug.org = Sent by: = cc ids-bounces@iiug. = org Subj= ect Re: checkpoints Excessively slow= [11251] = 02/08/2008 11:31 = AM = = = Please respond to = ids@iiug.org = = = thanks for your response there is a fragment on file online.log Fri Feb 8 00:47:44 2008 00:47:44 listener-thread: err =3D -27001: oserr =3D 0: errstr =3D : Rea= d error occurred during connection attempt. 01:04:38 Checkpoint Completed: duration was 5439 seconds. 01:07:00 Logical Log 216205 Complete. 01:07:03 Process exited with return code 139: /bin/sh /bin/sh -c /home/infx7/etc/log_full.sh 2 23 "Logical Log 216205 Complete." "L ogical Log 216205 Complete." 01:09:38 KAIO: out of OS resources, errno =3D 11, pid =3D 20801 01:09:38 KAIO: out of OS resources, errno =3D 11, pid =3D 20799 01:09:38 KAIO: out of OS resources, errno =3D 11, pid =3D 20802 01:09:38 KAIO: out of OS resources, errno =3D 11, pid =3D 20791 ***********************************************************************= ******** Forum Note: Use "Reply" to post a response in the discussion forum. See you at the IIUG Informix 2008 Conference The Power Conference for Informix Professionals April 27 - 30, 2008 Marriott Overland Park (Kansas City), Kansas http://www.iiug.org/conf Registration Now Open!! =