Poor Performanc with Multiple DB's in one Instance
A user on Solaris (Sun Fire T2000, IDS 10.00.FC5) reported poor performance and 'transaction time out' errors when several databases ran in one instance, while a single database per instance performed fine. After posting the onconfig and onstat -d/-m/-p output, Art Kagel concluded the problem was not the multiple databases but widespread misconfiguration: far too few buffers (buffer cache turning over every ~8 seconds), undersized logical logs causing the timeouts, ~20 instances each with 23 CPU VPs and 23 net poll threads on only 6 cores, NOAGE not set, overly aggressive read-ahead (RA_ parameters), ineffective AFF_ settings, and instances contending for the same disks. He recommended drastically increasing buffers, resizing logs, cutting VPs/poll threads, pinning each instance to distinct cores, or consolidating into one instance, plus an on-site consultant review. No follow-up confirming results is recorded.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Hello, We are facing some performance issues while running multiple databases in a single IDS Instance. We have just observed that when we run only one DB in one instance, the performance remains much better. Where as while running multiple databases in one instance results in very poor performance and we get issues like 'transaction time out'. I want to ask, whether there are any onconfig parameters to improve the performance while running multiple database in one IDS instance. I would also like to know about any other tips and recommendations to handle such scenarios. Thanks n Regards, Anees Ahmad
Hi,
What OS and what versoin of IDS are you using?
How large is the buffer pool (BUFFERS from the onconfig file)?
How many users are connected to each database?
The slowdown you're seeing may be due to having too few buffers to manage
the data. With a single database, the buffer pool may be large enough,
but when you need to manage the additional data from the second database,
you certainly need to configure more buffers to avoid what you're
experiencing.
However, there may be other issues as well. Please post the entire
configuration file and the output of onstat -p during a slowdown and
during acceptable performance.
Cheers,
Dick Snoke
IBM Data Management - ChannelWorks
dsnoke@us.ibm.com
(404) 487-1595
From:
"ANEES AHMAD" <aanees@i2cinc.com>
To:
ids@iiug.org
Date:
01/30/2009 06:35 AM
Subject:
Poor Performanc with Multiple DB's in one Instance [14659]
Sent by:
ids-bounces@iiug.org
Hello,
We are facing some performance issues while running multiple databases in
a
single IDS Instance.
We have just observed that when we run only one DB in one instance, the
performance remains much better. Where as while running multiple databases
in
one instance results in very poor performance and we get issues like
'transaction time out'.
I want to ask, whether there are any onconfig parameters to improve the
performance while running multiple database in one IDS instance. I would
also
like to know about any other tips and recommendations to handle such
scenarios.
Thanks n Regards,
Anees Ahmad
*******************************************************************************
Forum Note: Use "Reply" to post a response in the discussion forum.
Hello,
Please see my comments below.
What OS and what versoin of IDS are you using?
[Anees Ahmad] We are using SunOS CA-MCPDB2 5.10 Generic_125100-09 sun4v sparc
SUNW,Sun-Fire-T200 and the IDS version is 10.0. It's a 6 core machine with 4
threads per core.
How large is the buffer pool (BUFFERS from the onconfig file)?
[Anees Ahmad] This information is available in the onconfig file which is as
follows
# vi /u/informix/etc/onconfig_mcp15
#**************************************************************************
## Licensed Material - Property Of IBM
#
# "Restricted Materials of IBM"
#
# IBM Informix Dynamic Server
# (c) Copyright IBM Corporation 1996, 2005 All rights reserved.
#
# Title: onconfig.std
# Description: IBM Informix Dynamic Server Configuration Parameters
#
#**************************************************************************
# Root Dbspace Configuration
ROOTNAME rootdbs_mcp15 # Root dbspace name
ROOTPATH /u1/ids_space/rootdbs_mcp15
# Path for device containing root dbspace
ROOTOFFSET 0 # Offset of root dbspace into device (Kbytes)
ROOTSIZE 256000 # Size of root dbspace (Kbytes)
# Disk Mirroring Configuration Parameters
MIRROR 0 # Mirroring flag (Yes = 1, No = 0)
MIRRORPATH # Path for device containing mirrored root
MIRROROFFSET 0 # Offset into mirrored device (Kbytes)
# Physical Log Configuration
PHYSDBS rootdbs_mcp15 # Location (dbspace) of physical log
PHYSFILE 10000 # Physical log file size (Kbytes)
# Logical Log Configuration
LOGFILES 23 # Number of logical log files
LOGSIZE 50000 # Logical log size (Kbytes)LOG_BACKUP_MODE CONT # Logical log backup mode (MANUAL, CONT)
# Tablespace Tablespace Configuration in Root Dbspace
TBLTBLFIRST 5120 # First extent size (Kbytes) (0 = default)
TBLTBLNEXT 5120 # Next extent size (Kbytes) (0 = default)
# Security
# DBCREATE_PERMISSION:
# By default any user can create a database. Uncomment DBCREATE_PERMISSON to
# limit database creation to a specific user. Add a new DBCREATE_PERMISSION
# line for each permitted user.
#DBCREATE_PERMISSION informix
# DB_LIBRARY_PATH:
# When loading a (C or C++) shared object (for a UDR or UDT), IDS checks that
# the user-specified path starts with one of the directory prefixes listed in
# the comma-separated list of prefixes in DB_LIBRARY_PATH. The string
# "$INFORMIXDIR/extend" must be included in DB_LIBRARY_PATH in order for
# extensibility and IBM supplied blades to work correctly.
# DB_LIBRARY_PATH $INFORMIXDIR/extend
# IFX_EXTEND_ROLE:
# 0 (or off) => Disable use of EXTEND role to control who can register
# external routines.
# 1 (or on) => Enable use of EXTEND role to control who can register
# external routines. This is the default behaviour.
#
IFX_EXTEND_ROLE 1 # To control the usage of EXTEND role.
# Diagnostics
MSGPATH /u/informix/online_mcp15.log # System message log file path
CONSOLE /dev/console # System console message path
# To automatically backup logical logs, edit alarmprogram.sh and set
# BACKUPLOGS=Y
ALARMPROGRAM /u/informix/etc/alarmprogram.sh # Alarm program path
ALRM_ALL_EVENTS 1 # Triggers ALARMPROGRAM for any event occur
TBLSPACE_STATS 1 # Maintain tblspace statistics
# System Archive Tape Device
TAPEDEV /u/ids_backup/data_mcp15 # Tape device path
TAPEBLK 128 # Tape block size (Kbytes)
TAPESIZE 2048000 # Maximum amount of data to put on tape (Kbytes)
# Log Archive Tape Device
LTAPEDEV /u/ids_backup/llog_mcp15 # Log tape device path
LTAPEBLK 128 # Log tape block size (Kbytes)
LTAPESIZE 2048000 # Max amount of data to put on log tape (Kbytes)
# Optical
STAGEBLOB # Informix Dynamic Server staging area
# System Configuration
SERVERNUM 15 # Unique id corresponding to a OnLine instance
DBSERVERNAME ids_rep15 # Name of default database serverDBSERVERALIASES ids_net_rep15 # List of alternate dbservernames
NETTYPE ipcshm,23,250,CPU # Configure poll thread(s) for nettype
NETTYPE tlitcp,23,250,NET # Configure poll thread(s) for nettype
DEADLOCK_TIMEOUT 60 # Max time to wait of lock in distributed env.
RESIDENT 0 # Forced residency flag (Yes = 1, No = 0)
MULTIPROCESSOR 1 # 0 for single-processor, 1 for multi-processor
NUMCPUVPS 23 # Number of user (cpu) vps
SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vps to one
NOAGE 0 # Process aging
AFF_SPROC 0 # Affinity start processor
AFF_NPROCS 23 # Affinity number of processors
# Shared Memory Parameters
LOCKS 2000000 # Maximum number of locks
NUMAIOVPS 8 # Number of IO vps
PHYSBUFF 128 # Physical log buffer size (Kbytes)
LOGBUFF 128 # Logical log buffer size (Kbytes)
CLEANERS 16 # Number of buffer cleaner processes
SHMBASE 0x10a000000 # Shared memory base address
SHMVIRTSIZE 16384 # initial virtual shared memory segment size
SHMADD 16384 # Size of new shared memory segments (Kbytes)
EXTSHMADD 16384 # Size of new extension shared memory segments (Kbytes)
SHMTOTAL 2048000 # Total shared memory (Kbytes). 0=>unlimited
CKPTINTVL 300 # Check point interval (in sec)
TXTIMEOUT 0x12c # Transaction timeout (in sec) 300
STACKSIZE 128 # Stack size (Kbytes)
# Dynamic Logging
# DYNAMIC_LOGS:
# 2 : server automatically add a new logical log when necessary. (ON)
# 1 : notify DBA to add new logical logs when necessary. (ON)
# 0 : cannot add logical log on the fly. (OFF)
#
# When dynamic logging is on, we can have higher values for LTXHWM/LTXEHWM,
# because the server can add new logical logs during long transaction rollback.
# However, to limit the number of new logical logs being added, LTXHWM/LTXEHWM
# can be set to smaller values.
#
# If dynamic logging is off, LTXHWM/LTXEHWM need to be set to smaller values
# to avoid long transaction rollback hanging the server due to lack of logical
# log space, i.e. 50/60 or lower.
#
# In case of system configured with CDR, the difference between LTXHWM and
# LTXEHWM should be atleast 30% so that we could minimize log overrun issue.
DYNAMIC_LOGS 0
LTXHWM 50
LTXEHWM 60
# System Page Size
# BUFFSIZE - OnLine no longer supports this configuration parameter.
# To determine the page size used by OnLine on your platform
# see the last line of output from the command, 'onstat -b'.
# Recovery Variables
# OFF_RECVRY_THREADS:
# Number of parallel worker threads during fast recovery or an offline restore.
# ON_RECVRY_THREADS:
# Number of parallel worker threads during an online restore.
OFF_RECVRY_THREADS 10 # Default number of offline worker threads
ON_RECVRY_THREADS 1 # Default number of online worker threads
# Data Replication Variables
# DRAUTO: 0 manual, 1 retain type, 2 reverse type
DRAUTO 0 # DR automatic switchover
DRINTERVAL 30 # DR max time between DR buffer flushes (in sec)
DRTIMEOUT 30 # DR network timeout (in sec)
DRLOSTFOUND /u/informix/etc/dr.lostfound # DR lost+found file path
DRIDXAUTO 0 # DR automatic index repair. 0=off, 1=on
# CDR Variables
CDR_EVALTHREADS 1,2 # evaluator threads (per-cpu-vp,additional)
CDR_DSLOCKWAIT 5 # DS lockwait timeout (seconds)
CDR_QUEUEMEM
2009/1/30 ANEES AHMAD <aanees@i2cinc.com>:
> Hello,
> We are facing some performance issues while running multiple databases in a
> single IDS Instance.
> We have just observed that when we run only one DB in one instance, the
> performance remains much better. Where as while running multiple databases in
> one instance results in very poor performance and we get issues like
> 'transaction time out'.
> I want to ask, whether there are any onconfig parameters to improve the
> performance while running multiple database in one IDS instance. I would also
> like to know about any other tips and recommendations to handle such
> scenarios.
>
> Thanks n Regards,
> Anees Ahmad
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
Anees
Give us something to work with !!
Platform/
OS?
No of Disks and connection type?
IDS version?
onconfig file?
onstat -p?
onstat -m?how many is 'multiple'?
how many concurrent users?
2009/1/30 ANEES AHMAD <aanees@i2cinc.com>:
> Hello,
>
> Please see my comments below.
>
> What OS and what versoin of IDS are you using?
> [Anees Ahmad] We are using SunOS CA-MCPDB2 5.10 Generic_125100-09 sun4v sparc
> SUNW,Sun-Fire-T200 and the IDS version is 10.0. It's a 6 core machine with 4
> threads per core.
>
> How large is the buffer pool (BUFFERS from the onconfig file)?
> [Anees Ahmad] This information is available in the onconfig file which is as
> follows
>
> # vi /u/informix/etc/onconfig_mcp15
> #**************************************************************************
> #> # Licensed Material - Property Of IBM
> #
> # "Restricted Materials of IBM"
> #
> # IBM Informix Dynamic Server
> # (c) Copyright IBM Corporation 1996, 2005 All rights reserved.
> #
> # Title: onconfig.std
> # Description: IBM Informix Dynamic Server Configuration Parameters
> #
> #**************************************************************************
>
> # Root Dbspace Configuration
>
> ROOTNAME rootdbs_mcp15 # Root dbspace name
> ROOTPATH /u1/ids_space/rootdbs_mcp15>
> # Path for device containing root dbspace
> ROOTOFFSET 0 # Offset of root dbspace into device (Kbytes)
> ROOTSIZE 256000 # Size of root dbspace (Kbytes)>
> # Disk Mirroring Configuration Parameters
>
> MIRROR 0 # Mirroring flag (Yes = 1, No = 0)
> MIRRORPATH # Path for device containing mirrored root
> MIRROROFFSET 0 # Offset into mirrored device (Kbytes)>
> # Physical Log Configuration
>
> PHYSDBS rootdbs_mcp15 # Location (dbspace) of physical log
> PHYSFILE 10000 # Physical log file size (Kbytes)>
> # Logical Log Configuration
>
> LOGFILES 23 # Number of logical log files
> LOGSIZE 50000 # Logical log size (Kbytes)> LOG_BACKUP_MODE CONT # Logical log backup mode (MANUAL, CONT)
>
> # Tablespace Tablespace Configuration in Root Dbspace
>
> TBLTBLFIRST 5120 # First extent size (Kbytes) (0 = default)
> TBLTBLNEXT 5120 # Next extent size (Kbytes) (0 = default)>
> # Security
> # DBCREATE_PERMISSION:
> # By default any user can create a database. Uncomment DBCREATE_PERMISSON to
> # limit database creation to a specific user. Add a new DBCREATE_PERMISSION
> # line for each permitted user.
>
> #DBCREATE_PERMISSION informix
>
> # DB_LIBRARY_PATH:
> # When loading a (C or C++) shared object (for a UDR or UDT), IDS checks that
> # the user-specified path starts with one of the directory prefixes listed in
> # the comma-separated list of prefixes in DB_LIBRARY_PATH. The string
> # "$INFORMIXDIR/extend" must be included in DB_LIBRARY_PATH in order for
> # extensibility and IBM supplied blades to work correctly.
>
> # DB_LIBRARY_PATH $INFORMIXDIR/extend
>
> # IFX_EXTEND_ROLE:
> # 0 (or off) => Disable use of EXTEND role to control who can register
> # external routines.
> # 1 (or on) => Enable use of EXTEND role to control who can register
> # external routines. This is the default behaviour.
> #
> IFX_EXTEND_ROLE 1 # To control the usage of EXTEND role.>
> # Diagnostics
>
> MSGPATH /u/informix/online_mcp15.log # System message log file path
> CONSOLE /dev/console # System console message path>
> # To automatically backup logical logs, edit alarmprogram.sh and set
> # BACKUPLOGS=Y
> ALARMPROGRAM /u/informix/etc/alarmprogram.sh # Alarm program path
> ALRM_ALL_EVENTS 1 # Triggers ALARMPROGRAM for any event occur
> TBLSPACE_STATS 1 # Maintain tblspace statistics>
> # System Archive Tape Device
>
> TAPEDEV /u/ids_backup/data_mcp15 # Tape device path
> TAPEBLK 128 # Tape block size (Kbytes)
> TAPESIZE 2048000 # Maximum amount of data to put on tape (Kbytes)>
> # Log Archive Tape Device
>
> LTAPEDEV /u/ids_backup/llog_mcp15 # Log tape device path
> LTAPEBLK 128 # Log tape block size (Kbytes)
> LTAPESIZE 2048000 # Max amount of data to put on log tape (Kbytes)>
> # Optical
>
> STAGEBLOB # Informix Dynamic Server staging area
>
> # System Configuration
>
> SERVERNUM 15 # Unique id corresponding to a OnLine instance
> DBSERVERNAME ids_rep15 # Name of default database server> DBSERVERALIASES ids_net_rep15 # List of alternate dbservernames
> NETTYPE ipcshm,23,250,CPU # Configure poll thread(s) for nettype
> NETTYPE tlitcp,23,250,NET # Configure poll thread(s) for nettype
> DEADLOCK_TIMEOUT 60 # Max time to wait of lock in distributed env.
> RESIDENT 0 # Forced residency flag (Yes = 1, No = 0)
>
> MULTIPROCESSOR 1 # 0 for single-processor, 1 for multi-processor
> NUMCPUVPS 23 # Number of user (cpu) vps
> SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vps to one
>
> NOAGE 0 # Process aging
> AFF_SPROC 0 # Affinity start processor
> AFF_NPROCS 23 # Affinity number of processors>
> # Shared Memory Parameters
>
> LOCKS 2000000 # Maximum number of locks
> NUMAIOVPS 8 # Number of IO vps
> PHYSBUFF 128 # Physical log buffer size (Kbytes)
> LOGBUFF 128 # Logical log buffer size (Kbytes)
> CLEANERS 16 # Number of buffer cleaner processes
> SHMBASE 0x10a000000 # Shared memory base address
> SHMVIRTSIZE 16384 # initial virtual shared memory segment size
> SHMADD 16384 # Size of new shared memory segments (Kbytes)
> EXTSHMADD 16384 # Size of new extension shared memory segments (Kbytes)
> SHMTOTAL 2048000 # Total shared memory (Kbytes). 0=>unlimited
> CKPTINTVL 300 # Check point interval (in sec)
> TXTIMEOUT 0x12c # Transaction timeout (in sec) 300
> STACKSIZE 128 # Stack size (Kbytes)>
> # Dynamic Logging
> # DYNAMIC_LOGS:
> # 2 : server automatically add a new logical log when necessary. (ON)
> # 1 : notify DBA to add new logical logs when necessary. (ON)
> # 0 : cannot add logical log on the fly. (OFF)
> #
> # When dynamic logging is on, we can have higher values for LTXHWM/LTXEHWM,
> # because the server can add new logical logs during long transaction
> rollback.
> # However, to limit the number of new logical logs being added,
LTXHWM/LTXEHWM
> # can be set to smaller values.
> #
> # If dynamic logging is off, LTXHWM/LTXEHWM need to be set to smaller values
> # to avoid long transaction rollback hanging the server due to lack of
logical
> # log space, i.e. 50/60 or lower.
> #
> # In case of system configured with CDR, the difference between LTXHWM and
> # LTXEHWM should be atleast 30% so that we could minimize log overrun issue.
>
> DYNAMIC_LOGS 0
> LTXHWM 50
> LTXEHWM 60>
> # System Page Size
> # BUFFSIZE - OnLine no longer supports this configuration parameter.
> # To determine the page size used by OnLine on your platform
> # see the last line of output from the command, 'onstat -b'.
>
> # Recovery Variables
> # OFF_RECVRY_THREADS:
> # Number of parallel worker threads during fast recovery or an offline
> restore.
> # ON_RECVRY_THREADS:
> # Number of parallel worker threads during an online restore.
>
> OFF_RECVRY_THREADS 10 # Default number of offline worker threads
> ON_RECVRY_THREADS 1 # Default number of online worker threads>
> # Data Replication Variables
> # DRAUTO: 0 manual, 1 retain type, 2 reverse type
> DRAUTO 0 # DR automatic switchover> DRINTER
we are ussing SunOS CA-MCPDB2 5.10 Generic_125100-09 sun4v sparc
SUNW,Sun-Fire-T200 with IDS Version 10.0.Total concurrent users are almost 200
per DB.
We have updated the buffers to 80,000 but please see that we are using almost
20 instances in parallel. This particular instance has 5 databases running on
it. There are 50 virtual segments in this instance.
The results of onstat d , onstat m and onstat -p are as follows
====================================================
onstat -d
====================================================
IBM Informix Dynamic Server Version 10.00.FC5 -- On-Line (Prim) -- Up 4 days
05:04:17 -- 982016 Kbytes
Dbspaces
address number flags fchunk nchunks pgsize flags owner name
119babe78 1 0x40001 1 1 2048 N B informix rootdbs_mcp15
11c7e7848 2 0x40001 2 1 2048 N B informix datadbs_mcp15
11c7e79e0 3 0x40001 3 1 2048 N B informix logdbs_mcp15
11c7e7b78 4 0x42001 4 1 2048 N TB informix tempdbs_mcp15
11c7e7d10 6 0x40001 6 5 2048 N B informix datadbs_mcp17
11c7e9028 7 0x41001 7 1 2048 N B informix datadbs_mcp19
11c7e91c0 8 0x40001 8 2 2048 N B informix datadbs_mcp20
7 active, 2047 maximum
Chunks
address chunk/dbs offset size free bpages flags pathname
119bac028 1 1 0 128000 17120 PO-B /u1/ids_space/rootdbs_mcp15
119bac5f0 2 2 0 1024000 973419 PO-B /u1/ids_space/datadbs_mcp15
119bac790 3 3 0 1024000 51147 PO-B /u1/ids_space/logdbs_mcp15
119bac930 4 4 0 1024000 1023947 PO-B /u1/ids_space/tempdbs_mcp15
119bacad0 6 6 0 1024000 155 PO-B /u1/ids_space/datadbs_mcp17
119bacc70 7 7 0 1024000 462349 PO-B /u1/ids_space/datadbs_mcp19
119bace10 8 8 0 1024000 115556 PO-B /u1/ids_space/datadbs_mcp20
11c7e7028 9 6 0 512000 689 PO-B /u1/ids_space/datadbs_mcp17_1
11c7e71c8 10 6 0 512000 0 PO-B /u2/ids_space/datadbs_mcp17_2
11c7e7368 11 6 0 1024000 77565 PO-B /u3/ids_space/datadbs_mcp17_3
11c7e7508 12 8 0 512000 511997 PO-B /u1/ids_space/datadbs_mcp20_1
11c7e76a8 13 6 0 1024000 1023997 PO-B /u3/ids_space/datadbs_mcp17_4
12 active, 32766 maximum
NOTE: The values in the "size" and "free" columns for DBspace chunks are
displayed in terms of "pgsize" of the DBspace to which they belong.
Expanded chunk capacity mode: always
===================================================================
onstat -m
===================================================================
IBM Informix Dynamic Server Version 10.00.FC5 -- On-Line (Prim) -- Up 4 days
05:03:31 -- 982016 Kbytes
Message Log File: /u/informix/online_mcp15.log
06:01:24 listener-thread: err = -25582: oserr = 9: errstr = : Networkconnection is broken.
System error = 9.
06:02:34 listener-thread: err = -25582: oserr = 9: errstr = : Network
connection is broken.
System error = 9.
06:06:24 listener-thread: err = -25582: oserr = 9: errstr = : Network
connection is broken.
System error = 9.
06:06:24 Checkpoint Completed: duration was 0 seconds.
06:06:24 Checkpoint loguniq 17971, logpos 0x236a018, timestamp: 0x37cc12c0
06:06:24 Maximum server connections 501
06:07:35 listener-thread: err = -25582: oserr = 9: errstr = : Network
connection is broken.
System error = 9.
06:11:24 listener-thread: err = -25582: oserr = 9: errstr = : Network
connection is broken.
System error = 9.
06:11:25 Checkpoint Completed: duration was 0 seconds.
06:11:25 Checkpoint loguniq 17971, logpos 0x47bb018, timestamp: 0x37ccd347
06:11:25 Maximum server connections 501
06:12:35 listener-thread: err = -25582: oserr = 9: errstr = : Network
connection is broken.
============================================================
onstat -p
============================================================
Profile
dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
271298587 309917607 1355874794 79.99 945142 1401015 3611393 73.89
isamtot open start read write rewrite delete commit rollbk
1207750804 29722553 51104894 1035124601 390575 767451 160802 573155 219
gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
0 0 0 0 0 0 0
ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
0 0 0 127776.31 34386.69 1196 2398
bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
4657818 489 1228094352 0 0 167 132651 641062
ixda-RA idx-RA da-RA RA-pgsused lchwaits
29115363 67044 3597317 32611572 1262355
===================================================================
onconfig file
===================================================================
# vi /u/informix/etc/onconfig_mcp15
#**************************************************************************
## Licensed Material - Property Of IBM
#
# "Restricted Materials of IBM"
#
# IBM Informix Dynamic Server
# (c) Copyright IBM Corporation 1996, 2005 All rights reserved.
#
# Title: onconfig.std
# Description: IBM Informix Dynamic Server Configuration Parameters
#
#**************************************************************************
# Root Dbspace Configuration
ROOTNAME rootdbs_mcp15 # Root dbspace name
ROOTPATH /u1/ids_space/rootdbs_mcp15
# Path for device containing root dbspace
ROOTOFFSET 0 # Offset of root dbspace into device (Kbytes)
ROOTSIZE 256000 # Size of root dbspace (Kbytes)
# Disk Mirroring Configuration Parameters
MIRROR 0 # Mirroring flag (Yes = 1, No = 0)
MIRRORPATH # Path for device containing mirrored root
MIRROROFFSET 0 # Offset into mirrored device (Kbytes)
# Physical Log Configuration
PHYSDBS rootdbs_mcp15 # Location (dbspace) of physical log
PHYSFILE 10000 # Physical log file size (Kbytes)
# Logical Log Configuration
LOGFILES 23 # Number of logical log files
LOGSIZE 50000 # Logical log size (Kbytes)LOG_BACKUP_MODE CONT # Logical log backup mode (MANUAL, CONT)
# Tablespace Tablespace Configuration in Root Dbspace
TBLTBLFIRST 5120 # First extent size (Kbytes) (0 = default)
TBLTBLNEXT 5120 # Next extent size (Kbytes) (0 = default)
# Security
# DBCREATE_PERMISSION:
# By default any user can create a database. Uncomment DBCREATE_PERMISSON to
# limit database creation to a specific user. Add a new DBCREATE_PERMISSION
# line for each permitted user.
#DBCREATE_PERMISSION informix
# DB_LIBRARY_PATH:
# When loading a (C or C++) shared object (for a UDR or UDT), IDS checks that
# the user-specified path starts with one of the directory prefixes listed in
# the comma-separated list of prefixes in DB_LIBRARY_PATH. The string
# "$INFORMIXDIR/extend" must be included in DB_LIBRARY_PATH in order for
# extensibility and IBM supplied blades to work correctly.
# DB_LIBRARY_PATH $INFORMIXDIR/extend
# IFX_EXTEND_ROLE:
# 0 (or off) => Disable use of EXTEND role to control who can register
# external routines.
# 1 (or on) => Enable use of EXTEND role to control who can register
# external routines. This is the default behaviour.
#
IFX_EXTEND_ROLE 1 # To control the usage of EXTEND role.
# Diagnostics
MSGPATH /u/informix/online_mcp15.log # System message log file path
CONSOLE /dev/console # System console message path
# To automatically bac
Anees:
There are SO MANY problems with your setup that it would take hours to sort
it all out. None of these has ANYTHING to do with running five databases in
a single instance. I've configured and run instances with 80 databases
supporting well over 200,000 users multiplexed through ~100 connections with
no problems.
Let me tackle the biggest few:
- 6,000 buffers - Unless your entire set of five databases encompasses
fewer than about 60,000 pages on disk (about 120MB) this is WAY too small.
80,000 pages isn't much better I'm guessing.
- You have to determine the ACTUAL working set of your database, how many
rows are actively used during a typical day and on average how
many data and
index pages will have to be pulled in to satisfy those needs.
- Note that the physical organization of your database and how it is
used will affect the latter calculation. If there are frequent
deletes and
inserts then rows will be scattered around the disk and you
cannot depend on
data locality to reduce the number of pages of cache that you will need.
- Your Buffer Turnover Rate (BTR) is 418.37/hour which means that you are
replacing the entire buffer cache seven times a minute or about every 8-9
seconds! You want to see a BTR hover in the single digits. This indicates
that the cache size should be 35 and 100 times its current size. Split the
difference and the BTR suggests a cache size of about 380,000 pages!
- If you are getting Transaction timeouts then your logical logs are not
sufficient to hold all of the active transactions. You have to size the
logs based on the update activity on the server. That will be different for
each instance and certainly is likely to be greater supporting five data
bases than supporting one. Probably by a factor of five.
- Supporting 20 instances of IDS on a single machine is a bit extreme. I
have seen it done, but you must tune each instance VERY carefully and you
must tune them all in concert to reduce contention for resources.
- You do not have NOAGE set. Solaris is one of the most aggressive OSes
at reducing the run priority of long running processes and IDS's oninit
processes will hopefully be running for months at a time for you. Far
longer than anything else on the system besides OS services which are all
running as if NOAGE'd.
- You have 23 CPU VPs configured for this instance on a machine with 3
physical CPUs having 2 cores each or 6 cores. Forget that they are supposed
to run four concurrent threads. Sparc IV Niagra class threads DO NOT each
run as efficiently as a full blown Sparc core. I would recommend ignoring
them completely (indeed I am leaning towards telling clients to disable the
threading for dedicated IDS servers). Treat the machine as a 6 core box not
a 24 core box.
- If you have configured all 20 instances to each be running 23 CPU VPs
scale WAY DOWN. You are loading the machine as if it has 240 cores each
running at 500MHZ. The UltraSparc T1 processors only run at 1 or 1.2 GHZ so
you are treating the machine as if you have 120 of those T1 cores when you
only have six!
- Your RA_ parameters are too close together causing excessive read ahead
which combined with the limited cache size is probably killing your cache
hit rates.
- If all of these instances are sharing the same physical disk structures
they are all interferring with each other! That limits the IO bandwidth
available to each instance. It would have been better to configure a single
instance with all 25 databases in it. MUCH better!
- You have WAY too many NET VPS at 23. You say 200 users per DB. Is
that 200 total on this instance or 1000? Either way you do not need 23
tlitcp poll threads. Your wasting resources and if I multiply that by 20
instances - whoa!
- To support 200 TCP/IP users you can get away with a single NET VP -
though I suggest using two to improve responsiveness. (If all users are
local, using shared memory, then just use one poll thread in the tlitcp
NETTYPE.)
- If that's really 1000 users on this server, then you need between
three and five NET VPs to handle the load. Go for three or four
if activity
by the 1000 users is intermittent and five or six if the level
of activity
is consistently high.
- If all of your connections except administrative ones are TCP/IP
then you don't really need SHM poll threads running in every CPU VP (Yes I
know that's my recommendation over the years, but you may have missed that
one proviso). If many users are local, that's fine. Otherwise, reduce the
ipcshm nettype to two poll threads instead.
I would further suggest that you perform similar analysis for all 20
instances and reduce the footprint of each drastically, except in the area
of the buffer cache.
What you really need is an experienced IDS consultant to come in and review
your entire setup and make serious recommendations that you have to be
prepared to accept and act upon. If you want that kind of help, contact us
at Oninit (www.oninit.com). We can help.
Art
On Fri, Jan 30, 2009 at 9:39 AM, ANEES AHMAD <aanees@i2cinc.com> wrote:
> we are ussing SunOS CA-MCPDB2 5.10 Generic_125100-09 sun4v sparc
> SUNW,Sun-Fire-T200 with IDS Version 10.0.Total concurrent users are almost
> 200
> per DB.
>
> We have updated the buffers to 80,000 but please see that we are using
> almost
> 20 instances in parallel. This particular instance has 5 databases running
> on
> it. There are 50 virtual segments in this instance.
>
> The results of onstat d , onstat m and onstat -p are as follows
>
> ====================================================
>
> onstat -d
> ====================================================
> IBM Informix Dynamic Server Version 10.00.FC5 -- On-Line (Prim) -- Up 4> days
> 05:04:17 -- 982016 Kbytes
>
> Dbspaces
> address number flags fchunk nchunks pgsize flags owner name
> 119babe78 1 0x40001 1 1 2048 N B informix rootdbs_mcp15
> 11c7e7848 2 0x40001 2 1 2048 N B informix datadbs_mcp15
> 11c7e79e0 3 0x40001 3 1 2048 N B informix logdbs_mcp15
> 11c7e7b78 4 0x42001 4 1 2048 N TB informix tempdbs_mcp15
> 11c7e7d10 6 0x40001 6 5 2048 N B informix datadbs_mcp17
> 11c7e9028 7 0x41001 7 1 2048 N B informix datadbs_mcp19
> 11c7e91c0 8 0x40001 8 2 2048 N B informix datadbs_mcp20
> 7 active, 2047 maximum
>
> Chunks
> address chunk/dbs offset size free bpages flags pathname
> 119bac028 1 1 0 128000 17120 PO-B /u1/ids_space/rootdbs_mcp15
> 119bac5f0 2 2 0 1024000 973419 PO-B /u1/ids_space/datadbs_mcp15
> 119bac790 3 3 0 1024000 51147 PO-B /u1/ids_space/logdbs_mcp15
> 119bac930 4 4 0 1024000 1023947 PO-B /u1/ids_space/tempdbs_mcp15
> 119bacad0 6 6 0 1024000 155 PO-B /u1/ids_space/datadbs_mcp17
> 119bacc70 7 7 0 1024000 462349 PO-B /u1/ids_space/datadbs_mcp19
> 119bace10 8 8 0 1024000 115556 PO-B /u1/ids_space/datadbs_mcp20
> 11c7e7028 9 6 0 512000 689 PO-B /u1/ids_space/datadbs_mcp17_1
> 11c7e71c8 10 6 0 512000 0 PO
Forgot, one more point. You are using Affinity, but the way you are using
it is not gaining you anything.
Set up the AFF_ parameters in each instance to minimize the interference
between instances. So, limited this instance to the last two cores with:
AFF_SPROC 16 # Affinity start processor
AFF_NPROCS 8 # Affinity number of processors
Similarly tie other instances to different pairs of processors.
Art
On Fri, Jan 30, 2009 at 9:39 AM, ANEES AHMAD <aanees@i2cinc.com> wrote:
> we are ussing SunOS CA-MCPDB2 5.10 Generic_125100-09 sun4v sparc
> SUNW,Sun-Fire-T200 with IDS Version 10.0.Total concurrent users are almost
> 200
> per DB.
>
> We have updated the buffers to 80,000 but please see that we are using
> almost
> 20 instances in parallel. This particular instance has 5 databases running
> on
> it. There are 50 virtual segments in this instance.
>
> The results of onstat d , onstat m and onstat -p are as follows
> <SNIP>
--
Art S. Kagel
Oninit (www.oninit.com)
IIUG Board of Directors (art@iiug.org)
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Oninit, the IIUG, nor any other organization
with which I am associated either explicitly or implicitly. Neither do
those opinions reflect those of other individuals affiliated with any entity
with which I am affiliated nor those of the entities themselves.
--001636c5b54f1694dd0461b61233
You can reduce the number of shared memory segements by raising the
SHMVIRTSIZE parameter. Set SHMVIRTSIZE to a value approximately equal to
the sum of the sizes of the 50 segments. You should probably also raise
the SHMADD size. Both are set to 16 MB. That's very small for most
instances.
In general, 20 instances on a single server is a matter of balancing
resources. Each instance does require memory and CPU cycles. There is
no magic for avoiding that. If you only give a small amount of memory to
an instance, then it probably will run slowly. If you give too few CPU
cycles to an instance (too few CPU VPs), then it will probably run slowly.
You will have to assign appropriate resources to each instance. There is
some point at which the system just can't service all the instances fast
enough to get the performance you want. Once you pass that point, you'll
have a hard time getting all the instances to do well.
Dick Snoke
IBM Data Management - ChannelWorks
dsnoke@us.ibm.com
(404) 487-1595
From:
"ANEES AHMAD" <aanees@i2cinc.com>
To:
ids@iiug.org
Date:
01/30/2009 09:43 AM
Subject:
Re: Poor Performanc with Multiple DB's in one Inst [14664]
Sent by:
ids-bounces@iiug.org
we are ussing SunOS CA-MCPDB2 5.10 Generic_125100-09 sun4v sparc
SUNW,Sun-Fire-T200 with IDS Version 10.0.Total concurrent users are almost
200
per DB.
We have updated the buffers to 80,000 but please see that we are using
almost
20 instances in parallel. This particular instance has 5 databases running
on
it. There are 50 virtual segments in this instance.
The results of onstat âd , onstat âm and onstat -p are as follows
====================================================
onstat -d
====================================================
IBM Informix Dynamic Server Version 10.00.FC5 -- On-Line (Prim) -- Up 4days
05:04:17 -- 982016 Kbytes
Dbspaces
address number flags fchunk nchunks pgsize flags owner name
119babe78 1 0x40001 1 1 2048 N B informix rootdbs_mcp15
11c7e7848 2 0x40001 2 1 2048 N B informix datadbs_mcp15
11c7e79e0 3 0x40001 3 1 2048 N B informix logdbs_mcp15
11c7e7b78 4 0x42001 4 1 2048 N TB informix tempdbs_mcp15
11c7e7d10 6 0x40001 6 5 2048 N B informix datadbs_mcp17
11c7e9028 7 0x41001 7 1 2048 N B informix datadbs_mcp19
11c7e91c0 8 0x40001 8 2 2048 N B informix datadbs_mcp20
7 active, 2047 maximum
Chunks
address chunk/dbs offset size free bpages flags pathname
119bac028 1 1 0 128000 17120 PO-B /u1/ids_space/rootdbs_mcp15
119bac5f0 2 2 0 1024000 973419 PO-B /u1/ids_space/datadbs_mcp15
119bac790 3 3 0 1024000 51147 PO-B /u1/ids_space/logdbs_mcp15
119bac930 4 4 0 1024000 1023947 PO-B /u1/ids_space/tempdbs_mcp15
119bacad0 6 6 0 1024000 155 PO-B /u1/ids_space/datadbs_mcp17
119bacc70 7 7 0 1024000 462349 PO-B /u1/ids_space/datadbs_mcp19
119bace10 8 8 0 1024000 115556 PO-B /u1/ids_space/datadbs_mcp20
11c7e7028 9 6 0 512000 689 PO-B /u1/ids_space/datadbs_mcp17_1
11c7e71c8 10 6 0 512000 0 PO-B /u2/ids_space/datadbs_mcp17_2
11c7e7368 11 6 0 1024000 77565 PO-B /u3/ids_space/datadbs_mcp17_3
11c7e7508 12 8 0 512000 511997 PO-B /u1/ids_space/datadbs_mcp20_1
11c7e76a8 13 6 0 1024000 1023997 PO-B /u3/ids_space/datadbs_mcp17_4
12 active, 32766 maximum
NOTE: The values in the "size" and "free" columns for DBspace chunks are
displayed in terms of "pgsize" of the DBspace to which they belong.
Expanded chunk capacity mode: always
===================================================================
onstat -m
===================================================================
IBM Informix Dynamic Server Version 10.00.FC5 -- On-Line (Prim) -- Up 4days
05:03:31 -- 982016 Kbytes
Message Log File: /u/informix/online_mcp15.log
06:01:24 listener-thread: err = -25582: oserr = 9: errstr = : Network
connection is broken.
System error = 9.
06:02:34 listener-thread: err = -25582: oserr = 9: errstr = : Network
connection is broken.
System error = 9.
06:06:24 listener-thread: err = -25582: oserr = 9: errstr = : Network
connection is broken.
System error = 9.
06:06:24 Checkpoint Completed: duration was 0 seconds.
06:06:24 Checkpoint loguniq 17971, logpos 0x236a018, timestamp: 0x37cc12c0
06:06:24 Maximum server connections 501
06:07:35 listener-thread: err = -25582: oserr = 9: errstr = : Network
connection is broken.
System error = 9.
06:11:24 listener-thread: err = -25582: oserr = 9: errstr = : Network
connection is broken.
System error = 9.
06:11:25 Checkpoint Completed: duration was 0 seconds.
06:11:25 Checkpoint loguniq 17971, logpos 0x47bb018, timestamp: 0x37ccd347
06:11:25 Maximum server connections 501
06:12:35 listener-thread: err = -25582: oserr = 9: errstr = : Network
connection is broken.
============================================================
onstat -p
============================================================
Profile
dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
271298587 309917607 1355874794 79.99 945142 1401015 3611393 73.89
isamtot open start read write rewrite delete commit rollbk
1207750804 29722553 51104894 1035124601 390575 767451 160802 573155 219
gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
0 0 0 0 0 0 0
ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
0 0 0 127776.31 34386.69 1196 2398
bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
4657818 489 1228094352 0 0 167 132651 641062
ixda-RA idx-RA da-RA RA-pgsused lchwaits
29115363 67044 3597317 32611572 1262355
===================================================================
onconfig file
===================================================================
# vi /u/informix/etc/onconfig_mcp15
#**************************************************************************
#
# Licensed Material - Property Of IBM
#
# "Restricted Materials of IBM"
#
# IBM Informix Dynamic Server
# (c) Copyright IBM Corporation 1996, 2005 All rights reserved.
#
# Title: onconfig.std
# Description: IBM Informix Dynamic Server Configuration Parameters
#
#**************************************************************************
# Root Dbspace Configuration
ROOTNAME rootdbs_mcp15 # Root dbspace name
ROOTPATH /u1/ids_space/rootdbs_mcp15
# Path for device containing root dbspace
ROOTOFFSET 0 # Offset of root dbspace into device (Kbytes)
ROOTSIZE 256000 # Size of root dbspace (Kbytes)
# Disk Mirroring Configuration Parameters
MIRROR 0 # Mirroring flag (Yes = 1, No = 0)
MIRRORPATH # Path for device containing mirrored root
MIRROROFFSET 0 # Offset into mirrored device (Kbytes)
# Physical Log Configuration
PHYSDBS rootdbs_mcp15 # Location (dbspace) of physical log
PHYSFILE 10000 # Physical log file size (Kbytes)
# Logical Log Configuration
LOGFILES 23 # Number of logical log files
LOGSIZE 50000 # Logical log size (Kbytes)LOG_BACKUP_MODE CONT # Logical log backup mode (MANUAL, CONT)
# Tablespace Tablespace Configuration in Root Dbspace
TBLTBLFIRST 5120 #