malloc()/brk() failing with lots of mem. and swap space ?

Hi, We are running Sol2.4 on a Sun 670. The applic. uses informix. At one point during execution, the applic. is failing in malloc(). Although there is not sign of mem. shortage. Below are the relevant outputs of unix tools to determine mem/swap spaces etc. vmstat 1: --------- procs memory page disk faults cpu r b w swap free re mf pi po fr de sr s1 s2 s3 s4 in sy cs us sy id 0 0 0 479868 227784 0 33 1 0 0 0 0 0 0 1 13 308 273 163 8 3 89 0 0 0 407672 176224 0 10 0 0 0 0 0 0 2 20 0 143 100 73 0 2 97 0 0 0 407672 176224 0 0 0 0 0 0 0 0 0 0 0 21 107 78 0 0 100 0 0 0 407672 176224 0 8 0 0 0 0 0 0 0 0 0 147 100 73 2 0 98 0 0 0 407672 176224 0 1 0 0 0 0 0 0 0 0 0 115 114 72 5 0 95 0 0 0 407672 176224 0 0 0 0 0 0 0 0 0 0 0 28 106 83 0 2 98 0 0 0 407664 176216 0 0 0 0 0 0 0 0 0 0 0 45 137 133 0 11 89 swap -s: -------- total: 181416k bytes allocated + 142704k reserved = 324120k used, 407692k available swap -l: -------- swapfile dev swaplo blocks free /dev/dsk/c0t3d0s4 32,28 8 512712 512712 /dev/dsk/c0t2d0s1 32,17 8 410248 410248 /usr/bin/ps -elaf(only relevant processes): ------------------------------------------ F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME COMD 8 S ontrac09 2589 1 80 41 20 fcd87ce0 1627 fcbf03e6 09:39:59 ? 0:14 suserve -p 1073741835 -d 355 -v 1 8 S ontrac09 2458 1 80 41 20 fcd889a0 1619 fcbf07a6 09:22:38 ? 0:09 suserve -p 1073741832 -d 355 -v 1 8 S ontrac09 2569 1 80 41 20 fcd88340 1637 fcbf0706 09:32:40 ? 0:18 suserve -p 1073741833 -d 355 -v 1 8 S ontrac09 2248 1 80 41 20 fc851980 1785 fcbf018e 08:20:30 ? 0:24 suserve -p 1073741828 -d 355 -v 1 8 S ontrac09 2442 1 80 41 20 fcc82680 1677 fcbf031e 09:21:01 ? 0:26 suserve -p 1073741831 -d 355 -v 1 8 S ontrac09 2577 1 80 41 20 fcd87680 1617 fcbf090e 09:37:47 ? 0:11 suserve -p 1073741834 -d 355 -v 1 8 S ontrac09 2331 1 80 41 20 fccb2cc0 1733 fcbf02a6 08:37:22 ? 0:10 suserve -p 1073741830 -d 355 -v 1 8 S ontrac09 500 1 80 41 20 fcbe9338 1957 fcbf054e 07:34:13 ? 0:19 suserve -p 1073741825 -d 355 -v 1 8 S ontrac09 4884 1 80 41 20 fcd01668 741 fcdfe7fe 13:43:01 ? 0:05 suserve -p 1073741861 -d 355 -v 1 8 S ontrac09 5141 1 80 41 20 fcd02328 1617 fcbf059e 14:10:07 ? 0:05 suserve -p 1073741863 -d 355 -v 1 8 S ontrac09 562 1 80 41 20 fccb3980 737 fcbf0936 07:49:53 ? 0:04 suserve -p 1073741826 -d 355 -v 1 8 S ontrac09 5366 1 80 41 20 fcd8c320 1617 fcdfefce 14:53:24 ? 0:07 suserve -p 1073741865 -d 355 -v 1 /usr/ucb/ps -aux(only relevant processes): ------------------------------------------ USER PID %CPU %MEM SZ RSS TT S START TIME COMMAND ontrac09 3768 0.5 0.6 7028 2412 ? S 12:01:23 0:11 suserve -p 1073741 ontrac09 5906 0.3 0.1 660 456 pts/7 S 15:37:09 0:00 grep suserve ontrac09 500 0.0 0.8 7828 3636 ? S 07:34:13 0:19 suserve -p 1073741 ontrac09 5624 0.0 0.5 6468 2280 ? S 15:04:55 0:08 suserve -p 1073741 ontrac09 562 0.0 0.5 2948 2036 ? S 07:49:53 0:04 suserve -p 1073741 ontrac09 2248 0.0 0.7 7140 2872 ? S 08:20:30 0:23 suserve -p 1073741 ontrac09 2331 0.0 0.6 6932 2404 ? S 08:37:22 0:09 suserve -p 1073741 ontrac09 2442 0.0 0.7 6708 2864 ? S 09:21:01 0:25 suserve -p 1073741 ontrac09 2458 0.0 0.5 6476 2220 ? S 09:22:38 0:09 suserve -p 1073741 dmesg | grep mem: ----------------- mem = 458752K (0x1c000000) avail mem = 444170240 /etc/system: ------------ set enable_sm_wa = 1 set shmsys:shminfo_shmmax=268435456 set semsys:seminfo_semmap=64 set semsys:seminfo_semmni=4096 set semsys:seminfo_semmns=4096 set semsys:seminfo_semmnu=4096 set semsys:seminfo_semume=64 set semsys:seminfo_semmsl=100 set shmsys:shminfo_shmmin=100 set shmsys:shminfo_shmmni=100 set shmsys:shminfo_shmseg=100 set ufs_ninode=5000 set ncsize=5000 df -k ------ Filesystem kbytes used avail capacity Mounted on /dev/dsk/c0t3d0s0 51103 15027 30966 33% / /dev/dsk/c0t3d0s1 625031 430830 131701 77% /usr /proc 0 0 0 0% /proc fd 0 0 0 0% /dev/fd /dev/dsk/c0t3d0s3 143823 28044 101399 22% /var swap 405680 672 405008 0% /tmp /dev/dsk/c0t2d0s0 480815 17388 415347 4% /home /dev/dsk/c0t3d0s5 115191 63053 40628 61% /opt /dev/dsk/c0t3d0s6 77695 64203 5732 92% /usr/WI2.2 /dev/dsk/c0t2d0s4 767694 287528 403406 42% /ontrac_backup fleur:/vendor/solaris 2671847 1239208 1165459 52% /vendor/solaris mt300.cc:/bcrcc 403900 104823 258687 29% /bdfs/system/pkg/bcrcc informix configuration ---------------------- BUFFERS 40000 # Maximum number of shared buffers *man SHMBASE 0xa000000 # Shared memory base address *man SHMVIRTSIZE 50000 # initial virtual shared memory segment size * SHMADD 8192 # Size of new shared memory segments (K) *man SHMTOTAL 160000 # Total shared memory (Kbytes). 0=>unlim (Change d from 250000) **man truss -p <pid> -------------- semop(1, 0x001278A0, 1) = 0 semop(1, 0xEFFF5478, 1) = 0 semop(1, 0x001278A0, 1) = 0 brk(0x0052D4E0) = 0 brk(0x009094E0) Err#12 ENOMEM time()--(85%) = 848515876 write(5, " 1 3 : 5 1 : 1 6 R o u".., 77) = 77 time() = 848515876 write(5, " 1 3 : 5 1 : 1 6 R o u".., 82) = 82 time() = 848515876 write(5, " 1 3 : 5 1 : 1 6 R o u".., 85) = 85