IDS 11.50 Crashing

Hi, IDS 11.50.FC4 on Solaris 10 We migrated out test instance from IDS 11.10.FC3 to 11.50.FC4. After changing the softlinks and copying the onconfig and sqlhosts the new instance was started simply with "oninit -v". I could not see any error while the conversion process and the end lines were "Conversion Completed Successfully" "On-Line Mode" I did a level 0 backup, rebooted the instance two time and every thing was looking good untill aftersome time, I found that the instance went down. The online.log shows 12:27:35 stack trace for pid 10196 written to /opt/informix/lt3/dump/af.3f53a17 12:27:35 Assert Failed: No Exception Handler 12:27:35 IBM Informix Dynamic Server Version 11.50.FC4 12:27:35 Who: Session(1, informix@cseloukdudb05, 0, 1689fe028) Thread(13, main_loop(), 1689bb028, 1) File: mtex.c Line: 491 12:27:35 Results: Exception Caught. Type: MT_EX_OS, Context: mem 12:27:35 Action: Please notify IBM Informix Technical Support. 12:27:35 See Also: /opt/informix/lt3/dump/af.3f53a17 12:27:39 mtex.c, line 491, thread 13, proc id 10196, No Exception Handler. 12:27:39 Assert Failed: Page Check Error in rsread:bad data page 12:27:39 IBM Informix Dynamic Server Version 11.50.FC4 12:27:39 Who: Session(24, informix@, 0, 168a059e0) Thread(79, dbWorker1, 1689cec08, 3) File: rsdebug.c Line: 1080 12:27:39 Results: Possible inconsistencies in 'sysadmin:"informix".sysprocbody' 12:27:39 Action: Run 'oncheck -cD 1048770' 12:27:39 stack trace for pid 10198 written to /opt/informix/lt3/dump/af.4373a1b 12:27:39 The Master Daemon Died 12:27:39 See Also: /opt/informix/lt3/dump/af.4373a1b 12:27:39 SCHAPI: Error -397 System catalog (sysprocbody) corrupted. 12:27:39 SCHAPI: Type: TASK, Name: post_alarm_message, Location: NULL. 12:27:39 SCHAPI: Error -105 ISAM error: bad isam file format. 12:27:39 SCHAPI: Type: TASK, Name: post_alarm_message, Location: NULL. 12:27:39 SCHAPI: Error -7530 Missing parenthesis in the signature specified for user-defined routine lookup 12:27:39 SCHAPI: Type: TASK, Name: post_alarm_message, Location: NULL. 12:27:39 SCHAPI: Error -105 ISAM error: bad isam file format. 12:27:39 SCHAPI: Type: TASK, Name: post_alarm_message, Location: NULL. 12:27:39 SCHAPI: Type: TASK, Name: post_alarm_message, Error Message: Unable to find function[ procedure ph_dbs_alert ] . 12:27:39 PANIC: Attempting to bring system down 14:44:41 IBM Informix Dynamic Server Started. 14:44:41 Requested shared memory segment size rounded from 1517568KB to 1540096KB 14:44:41 Shared memory segment will use large pages with intimate shared memory (ISM) if available 14:44:42 Segment locked: addr=10a000000, size=1577058304 The effords to take instance up (oninit) failed with error: 14:57:12 IBM Informix Dynamic Server Version 11.50.FC4 Software Serial Number AAA#B000000 14:57:12 Logical Log File not found. 14:57:12 oninit: Fatal error in shared memory initialization 14:57:12 IBM Informix Dynamic Server Stopped. 14:57:12 mt_shm_remove: WARNING: may not have removed all/correct segments I checked for "ipcs |grep informix" and did "onmode -ky" 2 times but of no use. The logfile and other file permissions were intact and nothing seems to be changed externally. I restored the instance with the level 0 backup and took it online and runned UPDATE STATISTICS on every database, again every thing looked good untilltoday morning when i started another leve 0 backup which failed and instance crashed. The online.log now shows 09:56:09 Level 0 Archive started on rootdbs, physdbs, logdbs, datadbs2, datadbs8 09:56:10 Assert Warning: Archive detects that page 1:13 is corrupt. 09:56:10 IBM Informix Dynamic Server Version 11.50.FC4 09:56:10 Who: Session(31, informix@cseloukdudb05, 10585, 168a06ea8) Thread(236, arcbackup1, 1689d04f8, 3) File: rsarcbu.c Line: 3819 09:56:10 stack trace for pid 15153 written to /opt/informix/lt3/dump/af.4d4681a 09:56:10 See Also: /opt/informix/lt3/dump/af.4d4681a 09:56:11 Archive detects that page 1:13 is corrupt. 09:56:11 Assert Warning: Archive detects that page 1:14 is corrupt. 09:56:11 IBM Informix Dynamic Server Version 11.50.FC4 09:56:11 Who: Session(31, informix@cseloukdudb05, 10585, 168a06ea8) Thread(236, arcbackup1, 1689d04f8, 3) File: rsarcbu.c Line: 3819 09:56:11 stack trace for pid 15153 written to /opt/informix/lt3/dump/af.4d4681a 09:56:11 See Also: /opt/informix/lt3/dump/af.4d4681a 09:56:11 Archive detects that page 1:14 is corrupt. 09:56:11 Assert Warning: Archive detects that page 1:15 is corrupt. 09:56:11 IBM Informix Dynamic Server Version 11.50.FC4 09:56:11 Who: Session(31, informix@cseloukdudb05, 10585, 168a06ea8) Thread(236, arcbackup1, 1689d04f8, 3) File: rsarcbu.c Line: 3819 09:56:11 stack trace for pid 15153 written to /opt/informix/lt3/dump/af.4d4681a 09:56:11 See Also: /opt/informix/lt3/dump/af.4d4681a 09:56:11 Archive detects that page 1:15 is corrupt. 09:56:12 Assert Warning: Archive detects that page 1:16 is corrupt. 09:56:12 IBM Informix Dynamic Server Version 11.50.FC4 09:56:12 Who: Session(31, informix@cseloukdudb05, 10585, 168a06ea8) Thread(236, arcbackup1, 1689d04f8, 3) File: rsarcbu.c Line: 3819 . . . I checked the kernel parameters from machine notes can could not see any change. Any Help on this is appriciated Regards Vikas

The sysmaster database is not always correctly rebuild during an upgrade. You can manually rebuilt it though. Just restart the instance and run the script $INFORMIXDIR/etc/buildsmi When the script completes you can look in the log of the run: /tmp/buildsmi.out Art Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Jul 2, 2009 at 4:40 AM, VIKAS HIVARKAR <vikas.hivarkar@gmail.com>wrote: > Hi, > > IDS 11.50.FC4 on Solaris 10 > > We migrated out test instance from IDS 11.10.FC3 to 11.50.FC4. > After changing the softlinks and copying the onconfig and sqlhosts the new > instance was started simply with "oninit -v". > > I could not see any error while the conversion process and the end lines > were > "Conversion Completed Successfully" > "On-Line Mode" > > I did a level 0 backup, rebooted the instance two time and every thing was > looking good untill aftersome time, I found that the instance went down. > > The online.log shows > > 12:27:35 stack trace for pid 10196 written to > /opt/informix/lt3/dump/af.3f53a17 > 12:27:35 Assert Failed: No Exception Handler > 12:27:35 IBM Informix Dynamic Server Version 11.50.FC4 > 12:27:35 Who: Session(1, informix@cseloukdudb05, 0, 1689fe028) > > Thread(13, main_loop(), 1689bb028, 1) > > File: mtex.c Line: 491 > 12:27:35 Results: Exception Caught. Type: MT_EX_OS, Context: mem > 12:27:35 Action: Please notify IBM Informix Technical Support. > 12:27:35 See Also: /opt/informix/lt3/dump/af.3f53a17 > 12:27:39 mtex.c, line 491, thread 13, proc id 10196, No Exception Handler. > 12:27:39 Assert Failed: Page Check Error in rsread:bad data page > 12:27:39 IBM Informix Dynamic Server Version 11.50.FC4 > 12:27:39 Who: Session(24, informix@, 0, 168a059e0) > > Thread(79, dbWorker1, 1689cec08, 3) > > File: rsdebug.c Line: 1080 > 12:27:39 Results: Possible inconsistencies in > 'sysadmin:"informix".sysprocbody' > 12:27:39 Action: Run 'oncheck -cD 1048770' > 12:27:39 stack trace for pid 10198 written to > /opt/informix/lt3/dump/af.4373a1b > 12:27:39 The Master Daemon Died > 12:27:39 See Also: /opt/informix/lt3/dump/af.4373a1b > 12:27:39 SCHAPI: Error -397 System catalog (sysprocbody) corrupted. > 12:27:39 SCHAPI: Type: TASK, Name: post_alarm_message, Location: NULL. > 12:27:39 SCHAPI: Error -105 ISAM error: bad isam file format. > 12:27:39 SCHAPI: Type: TASK, Name: post_alarm_message, Location: NULL. > 12:27:39 SCHAPI: Error -7530 Missing parenthesis in the signature specified > for user-defined routine lookup > 12:27:39 SCHAPI: Type: TASK, Name: post_alarm_message, Location: NULL. > 12:27:39 SCHAPI: Error -105 ISAM error: bad isam file format. > 12:27:39 SCHAPI: Type: TASK, Name: post_alarm_message, Location: NULL. > 12:27:39 SCHAPI: Type: TASK, Name: post_alarm_message, Error Message: > Unable > to find function[ procedure ph_dbs_alert ] . > 12:27:39 PANIC: Attempting to bring system down > 14:44:41 IBM Informix Dynamic Server Started. > 14:44:41 Requested shared memory segment size rounded from 1517568KB to > 1540096KB > 14:44:41 Shared memory segment will use large pages with intimate shared > memory (ISM) if available > 14:44:42 Segment locked: addr=10a000000, size=1577058304 > > The effords to take instance up (oninit) failed with error: > > 14:57:12 IBM Informix Dynamic Server Version 11.50.FC4 Software Serial > Number > AAA#B000000 > 14:57:12 Logical Log File not found. > 14:57:12 oninit: Fatal error in shared memory initialization > > 14:57:12 IBM Informix Dynamic Server Stopped. > > 14:57:12 mt_shm_remove: WARNING: may not have removed all/correct segments > > I checked for "ipcs |grep informix" and did "onmode -ky" 2 times but of no > use. > The logfile and other file permissions were intact and nothing seems to be > changed externally. > > I restored the instance with the level 0 backup and took it online and > runned > UPDATE STATISTICS on every database, again every thing looked good untill> today morning when i started another leve 0 backup which failed and > instance > crashed. > > The online.log now shows > > 09:56:09 Level 0 Archive started on rootdbs, physdbs, logdbs, datadbs2, > datadbs8 > 09:56:10 Assert Warning: Archive detects that page 1:13 is corrupt. > 09:56:10 IBM Informix Dynamic Server Version 11.50.FC4 > 09:56:10 Who: Session(31, informix@cseloukdudb05, 10585, 168a06ea8) > > Thread(236, arcbackup1, 1689d04f8, 3) > > File: rsarcbu.c Line: 3819 > 09:56:10 stack trace for pid 15153 written to > /opt/informix/lt3/dump/af.4d4681a > 09:56:10 See Also: /opt/informix/lt3/dump/af.4d4681a > 09:56:11 Archive detects that page 1:13 is corrupt. > 09:56:11 Assert Warning: Archive detects that page 1:14 is corrupt. > 09:56:11 IBM Informix Dynamic Server Version 11.50.FC4 > 09:56:11 Who: Session(31, informix@cseloukdudb05, 10585, 168a06ea8) > > Thread(236, arcbackup1, 1689d04f8, 3) > > File: rsarcbu.c Line: 3819 > 09:56:11 stack trace for pid 15153 written to > /opt/informix/lt3/dump/af.4d4681a > 09:56:11 See Also: /opt/informix/lt3/dump/af.4d4681a > 09:56:11 Archive detects that page 1:14 is corrupt. > 09:56:11 Assert Warning: Archive detects that page 1:15 is corrupt. > 09:56:11 IBM Informix Dynamic Server Version 11.50.FC4 > 09:56:11 Who: Session(31, informix@cseloukdudb05, 10585, 168a06ea8) > > Thread(236, arcbackup1, 1689d04f8, 3) > > File: rsarcbu.c Line: 3819 > 09:56:11 stack trace for pid 15153 written to > /opt/informix/lt3/dump/af.4d4681a > 09:56:11 See Also: /opt/informix/lt3/dump/af.4d4681a > 09:56:11 Archive detects that page 1:15 is corrupt. > 09:56:12 Assert Warning: Archive detects that page 1:16 is corrupt. > 09:56:12 IBM Informix Dynamic Server Version 11.50.FC4 > 09:56:12 Who: Session(31, informix@cseloukdudb05, 10585, 168a06ea8) > > Thread(236, arcbackup1, 1689d04f8, 3) > > File: rsarcbu.c Line: 3819 > .. > .. > .. > > I checked the kernel parameters from machine notes can could not see any > change. > > Any Help on this is appriciated > > Regards > Vikas > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001636c5a4405496a6046df77362