Performance Problem - Mutex Waits

Hello all. I have a performance problem that I am hoping someone can assist me with please. Aix 4.33 Maintenance Level 11 4 CPU 8 GB IDS 7.31.UD4 7 Instances 150-200 Databases Per Instance All Databases have the same schema, with the same 61 tables. Buffwait Ratio is on average around 14. Buffer Turnover Ratio is on average 40-50. Anywhere from 400-600 peak users per instance. The problem occurs during peak OLTP processing activity. What I see is all update activity for the "USER" table waiting on a single mutex, for all databases in an instance. The worst I have seen is around 450 updates waiting for the same mutex, and wait times of about a minute. This is the extreme case, usually when we experience the problem wait times are around 5-10 seconds and there are 20-50 updates waiting for the same mutex. Here is the top of the output from onstat -g wmx during the problem: Informix Dynamic Server Version 7.31.UD4 -- On-Line -- Up 00:27:29 -- 433280 Kbytes Mutexes with waiters: mid addr name holder lkcnt waiter waittime 306 4008baa8 bld ref con 1360 1 1097 59 1024 59 949 59 1103 59 700 59 1121 59 1135 58 395 58 793 58 851 58 839 58 856 58 184 58 1416 58 1414 58 823 58 518 58 249 57 1081 57 922 57 900 57 898 57 1090 57 NOTE: The waiters are from several different databases all trying to update the "USER" table in each database. onstat - p output: Profile dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached 3818260 3959135 64274453 94.06 28996 31845 161127 82.00 isamtot open start read write rewrite delete commit rollbk 23507202 3131717 4975797 8632745 7909 33064 15 32490 273 gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs 0 0 0 0 0 0 0 ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes 0 0 0 1568.03 410.38 6 12 bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans 685372 313 80931581 0 0 82 131 215946 ixda-RA idx-RA da-RA RA-pgsused lchwaits 118096 588 3331895 3444806 45441 Relevant ONCONFIG Params: RESIDENT 0 # Forced residency flag (Yes = 1, No = 0) MULTIPROCESSOR 1 # 0 for single-processor, 1 formulti-processor NUMCPUVPS 4 # Number of user (cpu) vps SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vpsto one NOAGE 1 # Process aging AFF_SPROC 0 # Affinity start processor AFF_NPROCS 0 # Affinity number of processors LOCKS 600000 # Maximum number of locks BUFFERS 35000 # Maximum number of shared buffers NUMAIOVPS 1 # Number of IO vps PHYSBUFF 256 # Physical log buffer size (Kbytes) LOGBUFF 64 # Logical log buffer size (Kbytes)LOGSMAX 20 # Maximum number of logical log files CLEANERS 40 # Number of buffer cleaner processes SHMBASE 0x30000000 # Shared memory base address ( TJL added L) SHMVIRTSIZE 360000 # initial virtual shared memorysegment size SHMADD 80000 # Size of new shared memory segments (Kbytes) SHMTOTAL 0 # Total shared memory (Kbytes). 0=>unlimited CKPTINTVL 600 # Check point interval (in sec) LRUS 40 # Number of LRU queues LRU_MAX_DIRTY 2 # LRU percent dirty begin cleaninglimit LRU_MIN_DIRTY 1 # LRU percent dirty end cleaning limit LTXHWM 49 # Long transaction high water markpercentage LTXEHWM 74 # Long transaction high water mark (exclusive) TXTIMEOUT 0x12c # Transaction timeout (in sec) STACKSIZE 32 # Stack size (Kbytes) RA_PAGES # Number of pages to attempt to readahead RA_THRESHOLD # Number of pages left before nextgroup DBSPACETEMP tmpdbs # Default temp dbspaces OPTCOMPIND 2 # To hint the optimizer TBLSPACE_STATS 1 OPT_GOAL -1 DIRECTIVES 1 NETTYPE soctcp,4,150,NET # Override sqlhosts nettypeparameters I know that I need to do some serious tuning and I am going to be making several changes. Probably these to start: BUFFERS 120000 (this is a starting point, probably go higher. BR and BTR are way too high) LRUS 127 CLEANERS 127 NETTYPE soctcp,4,250,CPU (run in-line and allow moreconnections) DD_HASHSIZE 67DD_HASHMAX 200 (200 databases with the same table name) Can anyone tell me exactly what is causing the updates for different databases (all to the "USER" table, the table has the same name in each database) to all wait for a single mutex within an instance? Is there anything else that I can look at? Does anyone know exactly what a "bld ref con" mutex is? Thanks, James