Informix Fail to Online

Hi, Our informix instance crash after server restart.refer below error message. 10:08:49 IBM Informix Dynamic Server Started. 10:08:51 Segment locked: addr=c00000000fb1a000, size=643399680 Thu Apr 16 10:08:52 2015 10:08:52 Event alarms enabled. ALARMPROG = '/usr/informix/etc/alarmprogram.sh' 10:08:52 Booting Language <c> from module <> 10:08:52 Loading Module <CNULL> 10:08:52 Booting Language <builtin> from module <> 10:08:52 Loading Module <BUILTINNULL> 10:08:57 DR: DRAUTO is 0 (Off) 10:08:57 DR: ENCRYPT_HDR is 0 (HDR encryption Disabled) 10:08:57 Dynamically allocated new message shared memory segment (size 3744KB) 10:08:57 Memory sizes:resident:628320 KB, virtual:724288 KB, no SHMTOTAL limit 10:08:57 Dynamically allocated new message shared memory segment (size 3744KB) 10:08:57 Memory sizes:resident:628320 KB, virtual:728032 KB, no SHMTOTAL limit 10:08:57 Fast poll /dev/poll enabled. 10:08:57 IBM Informix Dynamic Server Version 11.70.FC4 Software Serial Number AAA#B000000 10:08:58 (16) connection rejected - no calls allowed for sqlexec 10:08:58 (17) connection rejected - no calls allowed for sqlexec 10:08:58 (18) connection rejected - no calls allowed for sqlexec 10:08:58 (19) connection rejected - no calls allowed for sqlexec 10:08:58 (20) connection rejected - no calls allowed for sqlexec 10:08:58 (13) connection rejected - no calls allowed for sqlexec 10:08:58 (21) connection rejected - no calls allowed for sqlexec 10:08:58 (22) connection rejected - no calls allowed for sqlexec 10:08:58 (24) connection rejected - no calls allowed for sqlexec 10:08:58 (23) connection rejected - no calls allowed for sqlexec 10:08:58 (26) connection rejected - no calls allowed for sqlexec 10:08:58 (27) connection rejected - no calls allowed for sqlexec 10:08:58 (28) connection rejected - no calls allowed for sqlexec 10:08:58 (25) connection rejected - no calls allowed for sqlexec 10:08:59 (30) connection rejected - no calls allowed for sqlexec 10:08:59 (31) connection rejected - no calls allowed for sqlexec 10:08:59 (32) connection rejected - no calls allowed for sqlexec 10:08:59 (33) connection rejected - no calls allowed for sqlexec 10:08:59 (34) connection rejected - no calls allowed for sqlexec 10:09:06 (10) connection rejected - no calls allowed for sqlexec 10:09:06 (35) connection rejected - no calls allowed for sqlexec 10:09:06 (37) connection rejected - no calls allowed for sqlexec 10:09:06 (38) connection rejected - no calls allowed for sqlexec 10:09:06 (39) connection rejected - no calls allowed for sqlexec 10:09:06 (40) connection rejected - no calls allowed for sqlexec 10:09:06 (41) connection rejected - no calls allowed for sqlexec 10:09:06 (42) connection rejected - no calls allowed for sqlexec 10:09:06 (43) connection rejected - no calls allowed for sqlexec 10:09:06 (44) connection rejected - no calls allowed for sqlexec 10:09:06 (45) connection rejected - no calls allowed for sqlexec 10:09:06 (46) connection rejected - no calls allowed for sqlexec 10:09:06 (47) connection rejected - no calls allowed for sqlexec 10:09:06 (48) connection rejected - no calls allowed for sqlexec 10:09:06 (49) connection rejected - no calls allowed for sqlexec 10:09:06 (50) connection rejected - no calls allowed for sqlexec 10:09:06 (51) connection rejected - no calls allowed for sqlexec 10:09:06 (52) connection rejected - no calls allowed for sqlexec 10:09:06 (53) connection rejected - no calls allowed for sqlexec 10:09:06 (54) connection rejected - no calls allowed for sqlexec 10:09:06 (55) connection rejected - no calls allowed for sqlexec 10:09:06 (56) connection rejected - no calls allowed for sqlexec 10:09:06 (57) connection rejected - no calls allowed for sqlexec 10:09:07 (58) connection rejected - no calls allowed for sqlexec 10:09:07 (59) connection rejected - no calls allowed for sqlexec 10:09:07 (60) connection rejected - no calls allowed for sqlexec 10:09:07 (61) connection rejected - no calls allowed for sqlexec 10:09:07 (62) connection rejected - no calls allowed for sqlexec 10:09:07 (63) connection rejected - no calls allowed for sqlexec 10:09:07 (64) connection rejected - no calls allowed for sqlexec 10:09:07 (65) connection rejected - no calls allowed for sqlexec 10:09:07 (66) connection rejected - no calls allowed for sqlexec 10:09:07 (67) connection rejected - no calls allowed for sqlexec 10:09:07 (68) connection rejected - no calls allowed for sqlexec 10:09:07 IBM Informix Dynamic Server Initialized -- Shared Memory Initialized. 10:09:07 Started 1 B-tree scanners. 10:09:07 B-tree scanner threshold set at 5000. 10:09:07 B-tree scanner range scan size set to -1. 10:09:07 B-tree scanner ALICE mode set to 6. 10:09:07 B-tree scanner index compression level set to med. 10:09:07 Warning: Invalid (non-existent/blobspace/disabled) dbspace listed in DBSPACETEMP: '/mnt17/inf_tmp/tempdbs' 10:09:07 Physical Recovery Started at Page (1:1128603). 10:09:10 Physical Recovery Complete: 44637 Pages Examined, 43486 Pages Restored. 10:09:10 Logical Recovery Started. 10:09:10 10 recovery worker threads will be started. 10:09:10 Fast Recovery Switching to Log 151578 10:09:10 Fast Recovery Switching to Log 151579 10:09:11 Fast Recovery Switching to Log 151580 10:09:12 Fast Recovery Switching to Log 151581 10:09:12 Fast Recovery Switching to Log 151582 10:09:12 Fast Recovery Switching to Log 151583 10:09:13 Rollforward of log record failed. iserrno = 101 10:09:13 Log Record: log = 151583, pos = 0x37859c, type = OLDRSAM:LOGPAGE(147), trans = 786 10:09:14 Assert Warning: Chunk 2 is being taken OFFLINE. 10:09:14 IBM Informix Dynamic Server Version 11.70.FC4 10:09:14 Who: Session(74, informix@gismain, 0, c0000000670b75d0) Thread(246, xchg_1.3, c00000006707ae18, 1) File: rsmirror.c Line: 1892 10:09:14 Results: Dynamic Server will block at next checkpoint 10:09:14 Action: Shutdown (onmode -k) or override (onmode -O) 10:09:14 stack trace for pid 23196 written to /mnt17/inf_tmp/af.4de19c8 10:09:14 See Also: /mnt17/inf_tmp/af.4de19c8 10:09:15 Chunk 2 is being taken OFFLINE. 10:09:15 Rollforward of log record failed. iserrno = 101 10:09:15 Log Record: log = 151583, pos = 0x37859c, type = OLDRSAM:LOGPAGE(147), trans = 786 10:09:16 Fast Recovery Switching to Log 151584 10:09:18 Fast Recovery Switching to Log 151585 10:09:21 Fast Recovery Switching to Log 151586 10:09:24 Fast Recovery Switching to Log 151587 10:09:26 Fast Recovery Switching to Log 151588 10:09:29 Fast Recovery Switching to Log 151589 10:09:32 Fast Recovery Switching to Log 151590 10:09:36 Fast Recovery Switching to Log 151591 10:09:40 Fast Recovery Switching to Log 151592 10:09:43 Fast Recovery Switching to Log 151593 10:09:47 Fast Recovery Switching to Log 151594 10:09:50 Fast Recovery Switching to Log 151595 10:09:54 Fast Recovery Switching to Log 151596 10:09:57 Fast Recovery Switching to Log 151597 10:10:00 Fast Recovery Switching to Log 151598 10:10:04 Fast Recovery Switching to Log 151599 10:10:06 Fast Recovery Switching to Log 151600 10:10:09 Fast Recovery Switching to Log 151601 10:10:10 Fast Recovery Switching to Log 151602 10:10:12 Assert Warning: Error during recovery left index inconsistent. 10:10:12 IBM Informix Dynamic Server Version 11

Fadzil Assuming you have sipport then call IBM, at a minimum you have corrupt Logical Logs which cannot roll forward and so need to be fixed or circumvented. Now would also be a good time to locate your latest archive and logical log saved files and dust off your recovery procedures as you may well need them. Is the hardware reporting any disk issues ?? Keith On 16 April 2015 at 04:19, MOHD FADZIL JUSOH <fadzil@isianpadu.com> wrote: > Hi, > > Our informix instance crash after server restart.refer below error message. > > 10:08:49 IBM Informix Dynamic Server Started. > 10:08:51 Segment locked: addr=c00000000fb1a000, size=643399680 > > Thu Apr 16 10:08:52 2015 > > 10:08:52 Event alarms enabled. ALARMPROG = > '/usr/informix/etc/alarmprogram.sh' > 10:08:52 Booting Language <c> from module <> > 10:08:52 Loading Module <CNULL> > 10:08:52 Booting Language <builtin> from module <> > 10:08:52 Loading Module <BUILTINNULL> > 10:08:57 DR: DRAUTO is 0 (Off) > 10:08:57 DR: ENCRYPT_HDR is 0 (HDR encryption Disabled) > 10:08:57 Dynamically allocated new message shared memory segment (size > 3744KB) > 10:08:57 Memory sizes:resident:628320 KB, virtual:724288 KB, no SHMTOTAL > limit > 10:08:57 Dynamically allocated new message shared memory segment (size > 3744KB) > 10:08:57 Memory sizes:resident:628320 KB, virtual:728032 KB, no SHMTOTAL > limit > 10:08:57 Fast poll /dev/poll enabled. > 10:08:57 IBM Informix Dynamic Server Version 11.70.FC4 Software Serial > Number > AAA#B000000 > 10:08:58 (16) connection rejected - no calls allowed for sqlexec > 10:08:58 (17) connection rejected - no calls allowed for sqlexec > 10:08:58 (18) connection rejected - no calls allowed for sqlexec > 10:08:58 (19) connection rejected - no calls allowed for sqlexec > 10:08:58 (20) connection rejected - no calls allowed for sqlexec > 10:08:58 (13) connection rejected - no calls allowed for sqlexec > 10:08:58 (21) connection rejected - no calls allowed for sqlexec > 10:08:58 (22) connection rejected - no calls allowed for sqlexec > 10:08:58 (24) connection rejected - no calls allowed for sqlexec > 10:08:58 (23) connection rejected - no calls allowed for sqlexec > 10:08:58 (26) connection rejected - no calls allowed for sqlexec > 10:08:58 (27) connection rejected - no calls allowed for sqlexec > 10:08:58 (28) connection rejected - no calls allowed for sqlexec > 10:08:58 (25) connection rejected - no calls allowed for sqlexec > 10:08:59 (30) connection rejected - no calls allowed for sqlexec > 10:08:59 (31) connection rejected - no calls allowed for sqlexec > 10:08:59 (32) connection rejected - no calls allowed for sqlexec > 10:08:59 (33) connection rejected - no calls allowed for sqlexec > 10:08:59 (34) connection rejected - no calls allowed for sqlexec > 10:09:06 (10) connection rejected - no calls allowed for sqlexec > 10:09:06 (35) connection rejected - no calls allowed for sqlexec > 10:09:06 (37) connection rejected - no calls allowed for sqlexec > 10:09:06 (38) connection rejected - no calls allowed for sqlexec > 10:09:06 (39) connection rejected - no calls allowed for sqlexec > 10:09:06 (40) connection rejected - no calls allowed for sqlexec > 10:09:06 (41) connection rejected - no calls allowed for sqlexec > 10:09:06 (42) connection rejected - no calls allowed for sqlexec > 10:09:06 (43) connection rejected - no calls allowed for sqlexec > 10:09:06 (44) connection rejected - no calls allowed for sqlexec > 10:09:06 (45) connection rejected - no calls allowed for sqlexec > 10:09:06 (46) connection rejected - no calls allowed for sqlexec > 10:09:06 (47) connection rejected - no calls allowed for sqlexec > 10:09:06 (48) connection rejected - no calls allowed for sqlexec > 10:09:06 (49) connection rejected - no calls allowed for sqlexec > 10:09:06 (50) connection rejected - no calls allowed for sqlexec > 10:09:06 (51) connection rejected - no calls allowed for sqlexec > 10:09:06 (52) connection rejected - no calls allowed for sqlexec > 10:09:06 (53) connection rejected - no calls allowed for sqlexec > 10:09:06 (54) connection rejected - no calls allowed for sqlexec > 10:09:06 (55) connection rejected - no calls allowed for sqlexec > 10:09:06 (56) connection rejected - no calls allowed for sqlexec > 10:09:06 (57) connection rejected - no calls allowed for sqlexec > 10:09:07 (58) connection rejected - no calls allowed for sqlexec > 10:09:07 (59) connection rejected - no calls allowed for sqlexec > 10:09:07 (60) connection rejected - no calls allowed for sqlexec > 10:09:07 (61) connection rejected - no calls allowed for sqlexec > 10:09:07 (62) connection rejected - no calls allowed for sqlexec > 10:09:07 (63) connection rejected - no calls allowed for sqlexec > 10:09:07 (64) connection rejected - no calls allowed for sqlexec > 10:09:07 (65) connection rejected - no calls allowed for sqlexec > 10:09:07 (66) connection rejected - no calls allowed for sqlexec > 10:09:07 (67) connection rejected - no calls allowed for sqlexec > 10:09:07 (68) connection rejected - no calls allowed for sqlexec > 10:09:07 IBM Informix Dynamic Server Initialized -- Shared Memory > Initialized. > > 10:09:07 Started 1 B-tree scanners. > 10:09:07 B-tree scanner threshold set at 5000. > 10:09:07 B-tree scanner range scan size set to -1. > 10:09:07 B-tree scanner ALICE mode set to 6. > 10:09:07 B-tree scanner index compression level set to med. > 10:09:07 Warning: Invalid (non-existent/blobspace/disabled) dbspace listed > > in DBSPACETEMP: '/mnt17/inf_tmp/tempdbs' > 10:09:07 Physical Recovery Started at Page (1:1128603). > 10:09:10 Physical Recovery Complete: 44637 Pages Examined, 43486 Pages > Restored. > 10:09:10 Logical Recovery Started. > 10:09:10 10 recovery worker threads will be started. > 10:09:10 Fast Recovery Switching to Log 151578 > 10:09:10 Fast Recovery Switching to Log 151579 > 10:09:11 Fast Recovery Switching to Log 151580 > 10:09:12 Fast Recovery Switching to Log 151581 > 10:09:12 Fast Recovery Switching to Log 151582 > 10:09:12 Fast Recovery Switching to Log 151583 > 10:09:13 Rollforward of log record failed. iserrno = 101 > 10:09:13 Log Record: log = 151583, pos = 0x37859c, type = > OLDRSAM:LOGPAGE(147), trans = 786 > 10:09:14 Assert Warning: Chunk 2 is being taken OFFLINE. > 10:09:14 IBM Informix Dynamic Server Version 11.70.FC4 > 10:09:14 Who: Session(74, informix@gismain, 0, c0000000670b75d0) > > Thread(246, xchg_1.3, c00000006707ae18, 1) > > File: rsmirror.c Line: 1892 > 10:09:14 Results: Dynamic Server will block at next checkpoint > 10:09:14 Action: Shutdown (onmode -k) or override (onmode -O) > 10:09:14 stack trace for pid 23196 written to /mnt17/inf_tmp/af.4de19c8 > 10:09:14 See Also: /mnt17/inf_tmp/af.4de19c8 > 10:09:15 Chunk 2 is being taken OFFLINE. > 10:09:15 Rollforward of log record failed. iserrno = 101 > 10:09:15 Log Record: log = 151583, pos = 0x37859c, type = > OLDRSAM:LOGPAGE(147), trans = 786 > 10:09:16 Fast Recovery Switching to Log 151584 > 10:09:18 Fast Recovery Switching to Log 151585 > 10:09:21 Fast Recovery Switching to Log 151586 > 10:09:24 Fast Recovery Switching to Log 151587 > 10:09:26 Fast Recovery Switching to Log 151588 > 10:09:29 Fast Recovery Switching to Log 151589 > 10:09:32

Long before the virtual processor crash the logical log roll forward failed. Some of that seems to have been caused by chunk files not accessible. It may be that the filesystems did not mount properly after the crash or there was a problem with your SAN or other disk structures. See to that first and if you still cannot start the engine, call IBM support and link over to the Down System Support line. Art Art S. Kagel, President and Principal Consultant ASK Database Management www.askdbmgt.com Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Wed, Apr 15, 2015 at 11:19 PM, MOHD FADZIL JUSOH <fadzil@isianpadu.com> wrote: > Hi, > > Our informix instance crash after server restart.refer below error message. > > 10:08:49 IBM Informix Dynamic Server Started. > 10:08:51 Segment locked: addr=c00000000fb1a000, size=643399680 > > Thu Apr 16 10:08:52 2015 > > 10:08:52 Event alarms enabled. ALARMPROG = > '/usr/informix/etc/alarmprogram.sh' > 10:08:52 Booting Language <c> from module <> > 10:08:52 Loading Module <CNULL> > 10:08:52 Booting Language <builtin> from module <> > 10:08:52 Loading Module <BUILTINNULL> > 10:08:57 DR: DRAUTO is 0 (Off) > 10:08:57 DR: ENCRYPT_HDR is 0 (HDR encryption Disabled) > 10:08:57 Dynamically allocated new message shared memory segment (size > 3744KB) > 10:08:57 Memory sizes:resident:628320 KB, virtual:724288 KB, no SHMTOTAL > limit > 10:08:57 Dynamically allocated new message shared memory segment (size > 3744KB) > 10:08:57 Memory sizes:resident:628320 KB, virtual:728032 KB, no SHMTOTAL > limit > 10:08:57 Fast poll /dev/poll enabled. > 10:08:57 IBM Informix Dynamic Server Version 11.70.FC4 Software Serial > Number > AAA#B000000 > 10:08:58 (16) connection rejected - no calls allowed for sqlexec > 10:08:58 (17) connection rejected - no calls allowed for sqlexec > 10:08:58 (18) connection rejected - no calls allowed for sqlexec > 10:08:58 (19) connection rejected - no calls allowed for sqlexec > 10:08:58 (20) connection rejected - no calls allowed for sqlexec > 10:08:58 (13) connection rejected - no calls allowed for sqlexec > 10:08:58 (21) connection rejected - no calls allowed for sqlexec > 10:08:58 (22) connection rejected - no calls allowed for sqlexec > 10:08:58 (24) connection rejected - no calls allowed for sqlexec > 10:08:58 (23) connection rejected - no calls allowed for sqlexec > 10:08:58 (26) connection rejected - no calls allowed for sqlexec > 10:08:58 (27) connection rejected - no calls allowed for sqlexec > 10:08:58 (28) connection rejected - no calls allowed for sqlexec > 10:08:58 (25) connection rejected - no calls allowed for sqlexec > 10:08:59 (30) connection rejected - no calls allowed for sqlexec > 10:08:59 (31) connection rejected - no calls allowed for sqlexec > 10:08:59 (32) connection rejected - no calls allowed for sqlexec > 10:08:59 (33) connection rejected - no calls allowed for sqlexec > 10:08:59 (34) connection rejected - no calls allowed for sqlexec > 10:09:06 (10) connection rejected - no calls allowed for sqlexec > 10:09:06 (35) connection rejected - no calls allowed for sqlexec > 10:09:06 (37) connection rejected - no calls allowed for sqlexec > 10:09:06 (38) connection rejected - no calls allowed for sqlexec > 10:09:06 (39) connection rejected - no calls allowed for sqlexec > 10:09:06 (40) connection rejected - no calls allowed for sqlexec > 10:09:06 (41) connection rejected - no calls allowed for sqlexec > 10:09:06 (42) connection rejected - no calls allowed for sqlexec > 10:09:06 (43) connection rejected - no calls allowed for sqlexec > 10:09:06 (44) connection rejected - no calls allowed for sqlexec > 10:09:06 (45) connection rejected - no calls allowed for sqlexec > 10:09:06 (46) connection rejected - no calls allowed for sqlexec > 10:09:06 (47) connection rejected - no calls allowed for sqlexec > 10:09:06 (48) connection rejected - no calls allowed for sqlexec > 10:09:06 (49) connection rejected - no calls allowed for sqlexec > 10:09:06 (50) connection rejected - no calls allowed for sqlexec > 10:09:06 (51) connection rejected - no calls allowed for sqlexec > 10:09:06 (52) connection rejected - no calls allowed for sqlexec > 10:09:06 (53) connection rejected - no calls allowed for sqlexec > 10:09:06 (54) connection rejected - no calls allowed for sqlexec > 10:09:06 (55) connection rejected - no calls allowed for sqlexec > 10:09:06 (56) connection rejected - no calls allowed for sqlexec > 10:09:06 (57) connection rejected - no calls allowed for sqlexec > 10:09:07 (58) connection rejected - no calls allowed for sqlexec > 10:09:07 (59) connection rejected - no calls allowed for sqlexec > 10:09:07 (60) connection rejected - no calls allowed for sqlexec > 10:09:07 (61) connection rejected - no calls allowed for sqlexec > 10:09:07 (62) connection rejected - no calls allowed for sqlexec > 10:09:07 (63) connection rejected - no calls allowed for sqlexec > 10:09:07 (64) connection rejected - no calls allowed for sqlexec > 10:09:07 (65) connection rejected - no calls allowed for sqlexec > 10:09:07 (66) connection rejected - no calls allowed for sqlexec > 10:09:07 (67) connection rejected - no calls allowed for sqlexec > 10:09:07 (68) connection rejected - no calls allowed for sqlexec > 10:09:07 IBM Informix Dynamic Server Initialized -- Shared Memory > Initialized. > > 10:09:07 Started 1 B-tree scanners. > 10:09:07 B-tree scanner threshold set at 5000. > 10:09:07 B-tree scanner range scan size set to -1. > 10:09:07 B-tree scanner ALICE mode set to 6. > 10:09:07 B-tree scanner index compression level set to med. > 10:09:07 Warning: Invalid (non-existent/blobspace/disabled) dbspace listed > > in DBSPACETEMP: '/mnt17/inf_tmp/tempdbs' > 10:09:07 Physical Recovery Started at Page (1:1128603). > 10:09:10 Physical Recovery Complete: 44637 Pages Examined, 43486 Pages > Restored. > 10:09:10 Logical Recovery Started. > 10:09:10 10 recovery worker threads will be started. > 10:09:10 Fast Recovery Switching to Log 151578 > 10:09:10 Fast Recovery Switching to Log 151579 > 10:09:11 Fast Recovery Switching to Log 151580 > 10:09:12 Fast Recovery Switching to Log 151581 > 10:09:12 Fast Recovery Switching to Log 151582 > 10:09:12 Fast Recovery Switching to Log 151583 > 10:09:13 Rollforward of log record failed. iserrno = 101 > 10:09:13 Log Record: log = 151583, pos = 0x37859c, type = > OLDRSAM:LOGPAGE(147), trans = 786 > 10:09:14 Assert Warning: Chunk 2 is being taken OFFLINE. > 10:09:14 IBM Informix Dynamic Server Version 11.70.FC4 > 10:09:14 Who: Session(74, informix@gismain, 0, c0000000670b75d0) > > Thread(246, xchg_1.3, c00000006707ae18, 1) > > File: rsmirror.c Line: 1892 > 10:09:14 Results: Dynamic Server will block at next checkpoint > 10:09:14 Action: Shutdown (onmode -k) or override (onmode -O) > 10:09:14 stack trace for pid 23196 written to /mnt17/inf_tmp/af.4de19c8 >