Help : Assertion Failure Type: CRASH

Hi all, We had a problem on our server. Server seems to fail on step "Checking database partition index...". Here is trace in my report file : 00:16:26 IBM Informix Dynamic Server Version 11.10.FC1 00:16:26 Who: Session(7, informix@aang, 0, 0x45216b88) Thread(54, xchg_1.4, 451da930, 1) File: rspartn.c Line: 6946 00:16:26 Results: Cannot use TBLSpace page for TBLSpace 1048854 00:16:26 Action: Run 'oncheck -pt 1048854' 00:16:26 stack trace for pid 6960 written to /opt/IBM/informix/tmp/af.41edc39 00:16:26 See Also: /opt/IBM/informix/tmp/af.41edc39 00:16:26 pthdrpage:ptalloc:bad bfget 00:16:26 Rollforward of log record failed. iserrno = 135 00:16:26 Log Record: log = 4168, pos = 0x61747e4, type = OLDRSAM:DELITEM(29), trans = 7 00:16:26 Rollforward of log record failed. iserrno = 135 00:16:26 Log Record: log = 4168, pos = 0x61747e4, type = OLDRSAM:DELITEM(29), trans = 7 00:16:26 Assert Failed: INFORMIX-OnLine Must ABORT Critical media failure. 00:16:26 IBM Informix Dynamic Server Version 11.10.FC1 00:16:26 Who: Session(7, informix@aang, 0, 0x45216e60) Thread(51, xchg_1.1, 451d90e8, 1) File: rsmirror.c Line: 1726 00:16:26 stack trace for pid 6960 written to /opt/IBM/informix/tmp/af.41bdc38 00:16:26 See Also: /opt/IBM/informix/tmp/af.41bdc38, shmem.41bdc38.0 I've checked with ipcs and there's nothing for user informix. Does anybody have an idea about how to fix this ? Thanks for your help

Have you tried running "oncheck -pt 1048854" as suggested by the asset fail? Jarrod Teale Fonterra NZ -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of JEAN-MICHEL RAMSEYER Sent: Monday, 6 April 2009 10:38 a.m. To: ids@iiug.org Subject: Help : Assertion Failure Type: CRASH [15429] Hi all, We had a problem on our server. Server seems to fail on step "Checking database partition index...". Here is trace in my report file : 00:16:26 IBM Informix Dynamic Server Version 11.10.FC1 00:16:26 Who: Session(7, informix@aang, 0, 0x45216b88) Thread(54, xchg_1.4, 451da930, 1) File: rspartn.c Line: 6946 00:16:26 Results: Cannot use TBLSpace page for TBLSpace 1048854 00:16:26 Action: Run 'oncheck -pt 1048854' 00:16:26 stack trace for pid 6960 written to /opt/IBM/informix/tmp/af.41edc39 00:16:26 See Also: /opt/IBM/informix/tmp/af.41edc39 00:16:26 pthdrpage:ptalloc:bad bfget 00:16:26 Rollforward of log record failed. iserrno = 135 00:16:26 Log Record: log = 4168, pos = 0x61747e4, type = OLDRSAM:DELITEM(29), trans = 7 00:16:26 Rollforward of log record failed. iserrno = 135 00:16:26 Log Record: log = 4168, pos = 0x61747e4, type = OLDRSAM:DELITEM(29), trans = 7 00:16:26 Assert Failed: INFORMIX-OnLine Must ABORT Critical media failure. 00:16:26 IBM Informix Dynamic Server Version 11.10.FC1 00:16:26 Who: Session(7, informix@aang, 0, 0x45216e60) Thread(51, xchg_1.1, 451d90e8, 1) File: rsmirror.c Line: 1726 00:16:26 stack trace for pid 6960 written to /opt/IBM/informix/tmp/af.41bdc38 00:16:26 See Also: /opt/IBM/informix/tmp/af.41bdc38, shmem.41bdc38.0 I've checked with ipcs and there's nothing for user informix. Does anybody have an idea about how to fix this ? Thanks for your help ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum. DISCLAIMER: This email contains confidential information and may be legally privileged. If you are not the intended recipient or have received this email in error, please notify the sender immediately and destroy this email. You may not use, disclose or copy this email or its attachments in any way. Any opinions expressed in this email are those of the author and are not necessarily those of the Fonterra Co-operative Group. http://www.fonterra.com/

Yes, I've tried this, but got error : oncheck -pt 1048854shared memory not initialized for INFORMIXSERVER 'myServerName' Thank you for helping

I've start with oninit -r and then launch oncheck but got this : ISAM error: Primary and Mirror chunks are bad

Have you looked at your assert failure files for additional info? Your original post of the online.log showed af files as follows: /opt/IBM/informix/tmp/af.41edc39 /opt/IBM/informix/tmp/af.41bdc38 There might not be any additional help in those files, but it's worth a look. Also as you stated below, it's complaining about your chunks.... You were doing a restore... is there any chance you can replace those physical disks or point to different ones and try again... hopefully it's a physical disk issue, not a media (backup tape) issue. If your media that you did the backup on is good/restorable, perhaps pointing your device/chunk links to other disk would get you past this (you did use device links in your engine setup, not actual device paths I hope)... If you have Informix support, open a ticket with them as well.... -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of JEAN-MICHEL RAMSEYER Sent: Monday, April 06, 2009 3:54 AM To: ids@iiug.org Subject: Re: RE: Help : Assertion Failure Type: CRASH [15432] I've start with oninit -r and then launch oncheck but got this : ISAM error: Primary and Mirror chunks are bad ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum.

Please run oncheck -pp 0x100001 0x116 save the output and post it. (should run while IDs is offline) Seems you have at least one bad partition page in the root chunk. If oncheck -cr returns with bad primary and mirror chunks this does not sound well. You are on IDs 11.10.FC1 - this is the first release oif IDS 11 - if possible, I'd upgrade to 11.50.FC4 or at least to the lastest available 11.10 fixpack . (check out fixcentral ) Jarrod Teale wrote: > Have you tried running "oncheck -pt 1048854" as suggested by the asset > fail? > > Jarrod Teale > Fonterra NZ > > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > JEAN-MICHEL RAMSEYER > Sent: Monday, 6 April 2009 10:38 a.m. > To: ids@iiug.org > Subject: Help : Assertion Failure Type: CRASH [15429] > > Hi all, > > We had a problem on our server. Server seems to fail on step "Checking > database partition index...". Here is trace in my report file : > > 00:16:26 IBM Informix Dynamic Server Version 11.10.FC1 > 00:16:26 Who: Session(7, informix@aang, 0, 0x45216b88) > > Thread(54, xchg_1.4, 451da930, 1) > > File: rspartn.c Line: 6946 > 00:16:26 Results: Cannot use TBLSpace page for TBLSpace 1048854 > 00:16:26 Action: Run 'oncheck -pt 1048854' > 00:16:26 stack trace for pid 6960 written to > /opt/IBM/informix/tmp/af.41edc39 > 00:16:26 See Also: /opt/IBM/informix/tmp/af.41edc39 > 00:16:26 pthdrpage:ptalloc:bad bfget > 00:16:26 Rollforward of log record failed. iserrno = 135 > 00:16:26 Log Record: log = 4168, pos = 0x61747e4, type = > OLDRSAM:DELITEM(29), trans = 7 > 00:16:26 Rollforward of log record failed. iserrno = 135 > 00:16:26 Log Record: log = 4168, pos = 0x61747e4, type = > OLDRSAM:DELITEM(29), trans = 7 > 00:16:26 Assert Failed: INFORMIX-OnLine Must ABORT > > Critical media failure. > 00:16:26 IBM Informix Dynamic Server Version 11.10.FC1 > 00:16:26 Who: Session(7, informix@aang, 0, 0x45216e60) > > Thread(51, xchg_1.1, 451d90e8, 1) > > File: rsmirror.c Line: 1726 > 00:16:26 stack trace for pid 6960 written to > /opt/IBM/informix/tmp/af.41bdc38 > 00:16:26 See Also: /opt/IBM/informix/tmp/af.41bdc38, shmem.41bdc38.0 > > I've checked with ipcs and there's nothing for user informix. Does > anybody have an idea about how to fix this ? > Thanks for your help >

Here is result for oncheck -pp 0x100001 0x116 : shared memory not initialized for INFORMIXSERVER 'myServerName' addr stamp chksum nslots flag type frptr frcnt next prev 1:11307 181156790 1d50 5 802 PARTN 324 1700 0 0 slot ptr len flg 1 24 92 0 2 116 44 0 3 160 0 0 4 160 52 0 5 212 112 0 slot 1: 0: 16 1 10 0 2 8 0 0 2c 0 0 0 0 0 1 0 ........,....... 16: d 0 0 8 45 16 2f 47 1 0 0 0 4 0 0 0 ....E./G........ 32: 4 0 0 0 34 0 0 0 34 0 0 0 0 0 0 0 ....4...4....... 48: ff ff ff ff 13 1 10 0 0 0 0 0 1 0 0 0 ................ 64: 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 ................ 80: 1 0 0 0 8 0 0 0 48 e0 13 0 ........H`...... slot 2: 0: 73 79 73 61 64 6d 69 6e 0 69 6e 66 6f 72 6d 69 sysadmin.informi 16: 78 0 69 78 5f 70 68 5f 72 75 6e 5f 30 33 0 65 x.ix_ph_run_03.e 32: 6e 5f 55 53 2e 38 31 39 0 bd ae 0 n_US.819.=...... slot 3: slot 4: 0: 19 2 0 1 2 0 c 0 0 0 0 0 13 1 10 0 ................ 16: 1 0 0 0 4 0 4 0 2 0 0 0 0 0 0 0 ................ 32: 0 0 0 0 10 0 8 0 0 0 0 0 0 0 0 0 ................ 48: 0 0 0 0 ................ slot 5: 0: 0 0 0 0 21 2f 0 0 4 0 0 0 41 48 0 0 ....!/......AH.. 16: 8 0 0 0 f1 48 0 0 c 0 0 0 7d 49 0 0 ....qH......}I.. 32: 10 0 0 0 ed 49 0 0 14 0 0 0 55 4b 0 0 ....mI......UK.. 48: 18 0 0 0 51 4c 0 0 1c 0 0 0 5d 4e 0 0 ....QL......]N.. 64: 20 0 0 0 5d 54 0 0 24 0 0 0 e3 54 0 0 ...]T..$...cT.. 80: 28 0 0 0 e5 b1 0 0 2c 0 0 0 a8 b5 0 0 (...e1..,...(5.. 96: 30 0 0 0 27 62 0 0 34 0 0 0 0 0 0 0 0...'b..4....... And here the result for oncheck -cr shared memory not initialized for INFORMIXSERVER 'myServerName' Validating IBM Informix Dynamic Server reserved pages Validating PAGE_PZERO... Validating PAGE_CONFIG... Validating PAGE_1CKPT & PAGE_2CKPT... Using check point page PAGE_2CKPT. Validating PAGE_1DBSP & PAGE_2DBSP... Using DBspace page PAGE_1DBSP. Validating PAGE_1PCHUNK & PAGE_2PCHUNK... Using primary chunk page PAGE_1PCHUNK. Validating PAGE_1ARCH & PAGE_2ARCH... Using archive page PAGE_2ARCH. And oninit -v : 18:58:07 IBM Informix Dynamic Server Started. Mon Apr 6 18:58:09 2009 18:58:09 pid 4556: scan_eh_frame(): bad CIE reference 18:58:09 Event alarms enabled. ALARMPROG = '/opt/IBM/informix/etc/alarmprogram.sh' 18:58:09 Dynamically allocated new virtual shared memory segment (size 8416KB) 18:58:09 Memory sizes:resident:13688 KB, virtual:16608 KB, no SHMTOTAL limit 18:58:09 Booting Language <c> from module <> 18:58:09 Loading Module <CNULL> 18:58:09 Booting Language <builtin> from module <> 18:58:09 Loading Module <BUILTINNULL> 18:58:14 Dynamically allocated new virtual shared memory segment (size 8192KB) 18:58:14 Memory sizes:resident:13688 KB, virtual:24800 KB, no SHMTOTAL limit 18:58:14 DR: DRAUTO is 0 (Off) 18:58:14 DR: ENCRYPT_HDR is 0 (HDR encryption Disabled) 18:58:14 Event notification facility epoll enabled. 18:58:14 IBM Informix Dynamic Server Version 11.10.FC1 Software Serial Number AAA#B000000 18:58:18 IBM Informix Dynamic Server Initialized -- Shared Memory Initialized. 18:58:18 Physical Recovery Started at Page (1:916). 18:58:18 Physical Recovery Complete: 43 Pages Examined, 43 Pages Restored. 18:58:18 Logical Recovery Started. 18:58:18 Dynamically allocated new virtual shared memory segment (size 8192KB) 18:58:18 Memory sizes:resident:13688 KB, virtual:32992 KB, no SHMTOTAL limit 18:58:18 10 recovery worker threads will be started. 18:58:18 WARNING! Physical Log size 2000 is too small. Physical Log overflows may occur during peak activity. Recommended minimum Physical Log size is 160 times maximum concurrent user threads. 18:58:18 Rollforward of log record failed. iserrno = 126 18:58:18 Log Record: log = 4168, pos = 0x6174678, type = OLDRSAM:HDELETE(41), trans = 10 18:58:18 Assert Warning: pthdrpage:ptalloc:bad bfget 18:58:18 IBM Informix Dynamic Server Version 11.10.FC1 18:58:18 Who: Session(7, <user>@<myServer>, 0, 0x45216b88) Thread(52, xchg_1.2, 451d9900, 1) File: rspartn.c Line: 6946 18:58:18 Results: Cannot use TBLSpace page for TBLSpace 1048852 18:58:18 Action: Run 'oncheck -pt 1048852' 18:58:18 stack trace for pid 4556 written to /opt/IBM/informix/tmp/af.41c34aa 18:58:18 See Also: /opt/IBM/informix/tmp/af.41c34aa 18:58:18 pthdrpage:ptalloc:bad bfget 18:58:18 Rollforward of log record failed. iserrno = 135 18:58:18 Log Record: log = 4168, pos = 0x6174770, type = OLDRSAM:DELITEM(29), trans = 7 18:58:18 Rollforward of log record failed. iserrno = 135 18:58:18 Log Record: log = 4168, pos = 0x6174770, type = OLDRSAM:DELITEM(29), trans = 7 18:58:18 Assert Warning: pthdrpage:ptalloc:bad bfget 18:58:18 IBM Informix Dynamic Server Version 11.10.FC1 18:58:18 Who: Session(7, <user>@<myServer>, 0, 0x45216b88) Thread(53, xchg_1.3, 451da118, 1) File: rspartn.c Line: 6946 18:58:18 Results: Cannot use TBLSpace page for TBLSpace 1048853 18:58:18 Action: Run 'oncheck -pt 1048853' 18:58:18 stack trace for pid 4556 written to /opt/IBM/informix/tmp/af.41d34aa 18:58:18 See Also: /opt/IBM/informix/tmp/af.41d34aa 18:58:19 pthdrpage:ptalloc:bad bfget 18:58:19 Rollforward of log record failed. iserrno = 135 18:58:19 Log Record: log = 4168, pos = 0x61747a8, type = OLDRSAM:DELITEM(29), trans = 7 18:58:19 Rollforward of log record failed. iserrno = 135 18:58:19 Log Record: log = 4168, pos = 0x61747a8, type = OLDRSAM:DELITEM(29), trans = 7 18:58:19 Assert Warning: pthdrpage:ptalloc:bad bfget 18:58:19 IBM Informix Dynamic Server Version 11.10.FC1 18:58:19 Who: Session(7, <user>@<myServer>, 0, 0x45216b88) Thread(54, xchg_1.4, 451da930, 1) File: rspartn.c Line: 6946 18:58:19 Results: Cannot use TBLSpace page for TBLSpace 1048854 18:58:19 Action: Run 'oncheck -pt 1048854' 18:58:19 stack trace for pid 4556 written to /opt/IBM/informix/tmp/af.41e34ab 18:58:19 See Also: /opt/IBM/informix/tmp/af.41e34ab 18:58:19 pthdrpage:ptalloc:bad bfget 18:58:19 Rollforward of log record failed. iserrno = 135 18:58:19 Log Record: log = 4168, pos = 0x61747e4, type = OLDRSAM:DELITEM(29), trans = 7 18:58:19 Rollforward of log record failed. iserrno = 135 18:58:19 Log Record: log = 4168, pos = 0x61747e4, type = OLDRSAM:DELITEM(29), trans = 7 18:58:19 Assert Failed: INFORMIX-OnLine Must ABORT Critical media failure. 18:58:19 IBM Informix Dynamic Server Version 11.10.FC1 18:58:19 Who: Session(7, <user>@<myServer>, 0, 0x45216e60) Thread(51, xchg_1.1, 451d90e8, 1) File: rsmirror.c Line: 1726 18:58:19 stack trace for pid 4556 written to /opt/IBM/informix/tmp/af.41b34aa 18:58:19 See Also: /opt/IBM/informix/tmp/af.41b34aa, shmem.41b34aa.0 18:58:23 rsmirror.c, line 1726, thread 51, proc id 4556, INFORMIX-OnLine Must ABORT Critical media failure.. 18:58:23 PANIC: Attempting to bring system down 18:58:23 semctl: errno = 22

I've already had a look at files : /opt/IBM/informix/tmp/af.41edc39 /opt/IBM/informix/tmp/af.41bdc38 But nothing strange to me except what I've paste before is from the second file :(