CDRGfan and Database crash

Hi All, MASTER: TARGET: IDS 10.0.UC3 IDS 9.40.TC6 ELAS3 (Kernel 2.4.21-4.ELsmp) Win 2K3 DELL PE 6600, PERC4/DC We had a DB crash. Oops !!!! Logged the issue with IBM thorough our application vendor. IBM did a thorough investigation but unable to establish the cause of crash. Possible closest bug was 175936 but that also has been ruled out later on by IBM engineer. IBM engineer was happy to replicate this issue, provided we provide all steps to reproduce it. We have no clue what steps we need to provide. Unfortunately, engine is not configured for mem dump, which makes even harder for IBM to investigate.. This could be one of thing but I want to hear if some one out here has ever encountered similar situation and any work around, if possible ? As this crash was due to CDRGfan, I am not sure what info. should I post here., however, as a starting point, pl. have a look at following info. from af file. Also, pl. let me know if need more info. PART - A 1 below are the info. from af file and PART - B are from onstat -g rqm FULL when engine is online and ER is up and running. TIA. PART - A ---------------------------------------------------------------------------------------------- 11:21:29 Stack for thread: 234774 CDRGfan base: 0x7ab8a000 len: 69632 pc: 0x0873f0c7 tos: 0x7ab9a324 state: running vp: 3 0x0873f0c7 (oninit)afstack (0x7fbd23d8, 0x91c, 0x8bb3f80, 0x1, 0x1, 0x7ab9a418) 0x08741fa9 (oninit)mt_ex_throw_sig(0x1, 0x8aaf459, 0x0, 0xe, 0x0, 0x7b815e60) 0x087147a8 (oninit)afsig_handler(0xb, 0x7ab9a540, 0x7ab9a5c0, 0xb, 0x0, 0x1) 0xb75d2df0 </lib/tls/libpthread.so.0>(null)(0x7ab81018, 0x7b815e60, 0x0, 0xccc00, 0x0, 0x1) 0x08872440 (oninit)grouperFanOutThread(0x7ab81018, 0x7ab9afcc, 0x0, 0xddccbbaa, 0x7fbd23d8, 0xa) 0x08847439 (oninit)cdrTrampolineThread(0x9444a2d0, 0x7fbd23d8, 0x4, 0x0, 0x0, 0x0) 0x0871fc34 (oninit)startup (0x15, 0x7ab9b400, 0x7a956fb0, 0x400, 0xf54f27b1, 0x0) 0x00000000 (*nosymtab*)0x0 11:21:29 11:21:29 IBM Informix Dynamic Server Version 10.00.UC3 SoftwareSerial Number AAA#B000000 11:21:29 Assert Failed: No Exception Handler 11:21:29 Who: Session(158248, informix@koa.whittlesea.vic.gov.au, 0,0x79e8b418) Thread(234774, CDRGfan, 7f0fe70c, 3) File: mtex.c Line: 435 11:21:29 Results: Exception Caught. Type: MT_EX_OS, Context: mem 11:21:29 Action: Please notify IBM Informix Technical Support. 11:21:29 Stack for thread: 234774 CDRGfan base: 0x7ab8a000 len: 69632 pc: 0x0873f0c7 tos: 0x7ab99d64 state: running vp: 3 --------------------------------- Begin System Alarm Program Output --------------------------------- Assertion Failure Type: CRASH Host Name: koa.whittlesea.vic.gov.au Database Server Name: authlive Time of failure: Mon Apr 24 11:21:30 EST 2006 AF file: /tmp/af.98fe2819 Shared memory file: None System Blocking: OFF onstat -g seg: IBM Informix Dynamic Server Version 10.00.UC3 -- On-Line -- Up 13 days 06:14:25 -- 1767052 Kbytes Segment Summary: id key addr size ovhd class blkused blkfree 98306 1382041601 44000000 879407104 241668 R* 214696 3 131075 1382041602 786ab000 921600000 28776 V 102640 122360 163844 1382041603 af593000 4227072 784 M 1032 0 196613 1382041604 af99b000 4227072 784 M 1032 0 Total: - - 1809461248 - - 319400 122363 (* segment locked in memory) onstat -u ........ ........ 7f0fe70c ---P---*158248 informix - 0 0 0 0 0 onstat -g ath ....... ....... *234774 7fbd23d8 7f0fe70c 2 running 3cpuCDRGfan onstat -g mem ...... ...... CDRGfan V 7a7fc020 53248 4312 35 6 onstat -g sts ...... ...... 234774 69120 8559 12 4620 6 CDRGfan -------------------------------------------------------------------------------------- PART - B (Below stats when engine is online and ER is up and running) -------------------------------------------------------------------------------------------- IBM Informix Dynamic Server Version 10.00.UC3 -- On-Line -- Up 14 days 00:47:44 -- 1767052 Kbytes CDR Reliable Queue Manager (RQM) Statistics: RQM Statistics for Queue (0x0x7b1e3018) trg_send Transaction Spool Name: trg_send_stxn Insert Stamp: 213995/0 Flags: SEND_Q, SPOOLED, PROGRESS_TABLE, NEED_ACK Txns in queue: 0 Log Events in queue: 0 Txns in memory: 0 Txns in spool only: 0 Txns spooled: 0 Unspooled bytes: 0 Size of Data in queue: 0 Bytes Real memory in use: 0 Bytes Pending Txn Buffers: 0 Pending Txn Data: 0 Bytes Max Real memory data used: 3272154 (4194304) Bytes Max Real memory hdrs used 1283504 (4194304) Bytes Total data queued: 782908347 Bytes Total Txns queued: 213995 Total Txns spooled: 4 Total Txns restored: 0 Total Txns recovered: 0 Spool Rows read: 0 Total Txns deleted: 213995 Total Txns duplicated: 0 Total Txn Lookups: 501003 Progress Table: Progress Table is Stable On-disk table name............: spttrg_send Flush interval (time).........: 30 Time of last flush............: 1147054982 Flush interval (serial number): 1000 Serial number of last flush...: 91737 Current serial number.........: 91745 ------------------------------------------------------------------------------------