Replication issues

I have two separate servers with two similar but separate issues. Both involve Secondary servers in a DR setup. Both are due to a power outage that caused these servers to power down without Informix being shut down. For the first server, here is the error log: 16:19:21 Event alarms enabled. ALARMPROG = '/ifmx/ifmx00175/etc/log_full.sh' 16:19:22 Booting Language <c> from module <> 16:19:22 Loading Module <CNULL> 16:19:22 Booting Language <builtin> from module <> 16:19:22 Loading Module <BUILTINNULL> 16:19:22 Could not disable priority aging: errno = 1 16:19:22 Could not disable priority aging: errno = 1 16:19:22 Could not disable priority aging: errno = 1 16:19:22 Could not disable priority aging: errno = 1 16:19:22 Could not disable priority aging: errno = 1 16:19:22 Could not disable priority aging: errno = 1 16:19:28 DR: DRAUTO is 0 (Off) 16:19:28 Requested shared memory segment size rounded from 544KB to 1024KB 16:19:29 IBM Informix Dynamic Server Version 10.00.UC6 Software Serial Number AAA#B000000 16:19:29 WARNING: Next backup of DBspace llogdbs must be level-0 backup. 16:19:29 WARNING: Next backup of DBspace plogdbs must be level-0 backup. 16:19:29 IBM Informix Dynamic Server Initialized -- Shared Memory Initialized. 16:19:29 Physical Recovery Started at Page (3:29800). 16:19:29 Physical Recovery Complete: 18 Pages Examined, 18 Pages Restored. 16:19:29 DR: Trying to connect to primary server = ifmx00174_repl 16:19:30 Dataskip is now OFF for all dbspaces 16:19:30 Restartable Restore has been ENABLED 16:19:30 Recovery Mode 16:19:33 DR: Secondary server connected 16:19:34 DR: Configuration parameter values of the paired servers do not match: Parameter: TAPESIZE Current server's value: 20480000 Paired server's value: 30720000 WARNING: The parameter values must match if ontape is used as backup/recovery tool. Please adjust this parameter value in onconfig file to correctly reflect the setting of the primary server. 16:19:34 DR: Secondary server needs failure recovery 16:19:34 DR: Failure recovery from disk in progress ... 16:19:34 Logical Recovery Started. 16:19:34 10 recovery worker threads will be started. 16:19:34 Start Logical Recovery - Start Log 54282, End Log ? 16:19:34 Starting Log Position - 54282 0x10018 16:19:34 Assert Failed: Page Check Error in page_reorg: slots overlap 16:19:34 IBM Informix Dynamic Server Version 10.00.UC6 16:19:34 Who: Session(26, informix@t5240-03-z02, 0, 3fc324e0) Thread(129, xchg_1.8, 3fc1d548, 3) File: rsdebug.c Line: 1081 16:19:34 Results: Possible inconsistencies in an index of 'campaign:"dbaparts".camptss_ses' "/ifmx/ifmx00175/online.log" [Read only] 92 lines, 5329 characters 16:19:30 Dataskip is now OFF for all dbspaces 16:19:30 Restartable Restore has been ENABLED 16:19:30 Recovery Mode 16:19:33 DR: Secondary server connected 16:19:34 DR: Configuration parameter values of the paired servers do not match: Parameter: TAPESIZE Current server's value: 20480000 Paired server's value: 30720000 WARNING: The parameter values must match if ontape is used as backup/recovery tool. Please adjust this parameter value in onconfig file to correctly reflect the setting of the primary server. 16:19:34 DR: Secondary server needs failure recovery 16:19:34 DR: Failure recovery from disk in progress ... 16:19:34 Logical Recovery Started. 16:19:34 10 recovery worker threads will be started. 16:19:34 Start Logical Recovery - Start Log 54282, End Log ? 16:19:34 Starting Log Position - 54282 0x10018 16:19:34 Assert Failed: Page Check Error in page_reorg: slots overlap 16:19:34 IBM Informix Dynamic Server Version 10.00.UC6 16:19:34 Who: Session(26, informix@t5240-03-z02, 0, 3fc324e0) Thread(129, xchg_1.8, 3fc1d548, 3) File: rsdebug.c Line: 1081 16:19:34 Results: Possible inconsistencies in an index of 'campaign:"dbaparts".camptss_ses' 16:19:34 Action: Run 'oncheck -cI campaign:"dbaparts".camptss_ses' 16:19:34 stack trace for pid 5543 written to /dbengines/informix/tmp/af.46904f6 16:19:34 See Also: /dbengines/informix/tmp/af.46904f6, shmem.46904f6.0 16:19:34 Assert Warning: Error during recovery left index inconsistent. 16:19:34 IBM Informix Dynamic Server Version 10.00.UC6 16:19:34 Who: Session(26, informix@t5240-03-z02, 0, 3fc33520) Thread(130, xchg_1.9, 3fc1da74, 5) File: rskey.c Line: 1516 16:19:34 Results: Index 'campaign:"dbaparts".camptss_ses# 120_163' is now unusable 16:19:34 Action: Run 'oncheck -cI campaign:"dbaparts".camptss_ses# 120_163' 16:19:34 stack trace for pid 5559 written to /dbengines/informix/tmp/af.46a04f6 16:19:34 See Also: /dbengines/informix/tmp/af.46a04f6 16:19:35 Rollforward of log record failed. iserrno = 126 16:19:35 Log Record: log = 54282, pos = 0x1906c, type = OLDRSAM:HINSERT(40), trans = 91 16:19:35 Error during recovery left index inconsistent. 16:19:35 Rollforward of log record failed. iserrno = 105 16:19:35 Log Record: log = 54282, pos = 0x1917c, type = OLDRSAM:ADDITEM(28), trans = 91 16:19:35 Rollforward of log record failed. iserrno = 105 16:19:35 Log Record: log = 54282, pos = 0x1917c, type = OLDRSAM:ADDITEM(28), trans = 91 16:19:44 Assert Warning: Chunk 8 is being taken OFFLINE. 16:19:44 IBM Informix Dynamic Server Version 10.00.UC6 16:19:44 Who: Session(26, informix@t5240-03-z02, 0, 3fc33520) Thread(126, xchg_1.5, 3fc1c5c4, 1) File: rsmirror.c Line: 1846 16:19:44 Results: Dynamic Server will block at next checkpoint 16:19:44 Action: Shutdown (onmode -k) or override (onmode -O) 16:19:44 stack trace for pid 5462 written to /dbengines/informix/tmp/af.46604f6 16:19:44 See Also: /dbengines/informix/tmp/af.46604f6 16:19:45 Chunk 8 is being taken OFFLINE. 16:19:45 Rollforward of log record failed. iserrno = 126 16:19:45 Log Record: log = 54282, pos = 0x1906c, type = OLDRSAM:HINSERT(40), trans = 91 16:19:45 Rollforward of log record failed. iserrno = 101 16:19:45 Log Record: log = 54282, pos = 0x1b044, type = OLDRSAM:UNIQID(17), trans = 93 16:19:45 Rollforward of log record failed. iserrno = 101 16:19:45 Log Record: log = 54282, pos = 0x1b044, type = OLDRSAM:UNIQID(17), trans = 93 16:21:38 Page Check Error in page_reorg: slots overlap 16:21:38 Rollforward of log record failed. iserrno = 105 16:21:38 Log Record: log = 54282, pos = 0x1b140, type = OLDRSAM:ADDITEM(28), trans = 93 16:21:38 Rollforward of log record failed. iserrno = 105 16:21:38 Log Record: log = 54282, pos = 0x1b140, type = OLDRSAM:ADDITEM(28), trans = 93 16:21:38 Checkpoint blocked by down space, waiting for override or shutdown I tried to run 'oncheck -cI campaign:"dbaparts".camptss_ses,' but it tells me that I can't run it because I don't own that table. The chunk that is down includes all of the indexes for this instance. Is it possible to drop the chunk, recreate the chunk, and then recreate the indexes, or will we need to restore the data from a backup? 17:13:34 Event alarms enabled. ALARMPROG = '/ifmx/ifmx00804/etc/event.pl' 17:13:34 Booting Language <c> from module <> 17:13:34 Loading Module <CNULL> 17:13:34 Booting Language <builtin>

Keith Don't know if any else has responded direct, or if you have resolved this now ? I would suggest in both cases (after you have checked the physical disks are OK) is to switch both Primaries to Standard Mode then re-initialise HDR by taking archives (make sure you resolve the tape size issue0 and restoring to the secondaries. You very limited about what actions can be undertaken on Secondaries and I would look very carefully at your physical disk before doing anything as it looks suspiciously like you have some bad areas (head crash/corruption from the power issue). Keith (Simmons) On 26 January 2013 01:30, Schleicher, Keith <Keith.Schleicher@searshc.com>wrote: > I have two separate servers with two similar but separate issues. Both > involve > Secondary servers in a DR setup. Both are due to a power outage that caused > these servers to power down without Informix being shut down. > > For the first server, here is the error log: > > 16:19:21 Event alarms enabled. ALARMPROG = > '/ifmx/ifmx00175/etc/log_full.sh' > 16:19:22 Booting Language <c> from module <> > 16:19:22 Loading Module <CNULL> > 16:19:22 Booting Language <builtin> from module <> > 16:19:22 Loading Module <BUILTINNULL> > 16:19:22 Could not disable priority aging: errno = 1 > 16:19:22 Could not disable priority aging: errno = 1 > 16:19:22 Could not disable priority aging: errno = 1 > 16:19:22 Could not disable priority aging: errno = 1 > 16:19:22 Could not disable priority aging: errno = 1 > 16:19:22 Could not disable priority aging: errno = 1 > 16:19:28 DR: DRAUTO is 0 (Off) > 16:19:28 Requested shared memory segment size rounded from 544KB to 1024KB > 16:19:29 IBM Informix Dynamic Server Version 10.00.UC6 Software Serial > Number > AAA#B000000 > 16:19:29 WARNING: Next backup of DBspace llogdbs must be level-0 backup. > 16:19:29 WARNING: Next backup of DBspace plogdbs must be level-0 backup. > 16:19:29 IBM Informix Dynamic Server Initialized -- Shared Memory > Initialized. > > 16:19:29 Physical Recovery Started at Page (3:29800). > 16:19:29 Physical Recovery Complete: 18 Pages Examined, 18 Pages Restored. > 16:19:29 DR: Trying to connect to primary server = ifmx00174_repl > 16:19:30 Dataskip is now OFF for all dbspaces > 16:19:30 Restartable Restore has been ENABLED > 16:19:30 Recovery Mode > 16:19:33 DR: Secondary server connected > 16:19:34 DR: Configuration parameter values of the paired servers do not > match: > > Parameter: TAPESIZE > > Current server's value: 20480000 > > Paired server's value: 30720000 > > WARNING: The parameter values must match if ontape is used as > > backup/recovery tool. > > Please adjust this parameter value in onconfig file to correctly > > reflect the setting of the primary server. > 16:19:34 DR: Secondary server needs failure recovery > > 16:19:34 DR: Failure recovery from disk in progress ... > 16:19:34 Logical Recovery Started. > 16:19:34 10 recovery worker threads will be started. > 16:19:34 Start Logical Recovery - Start Log 54282, End Log ? > 16:19:34 Starting Log Position - 54282 0x10018 > 16:19:34 Assert Failed: Page Check Error in page_reorg: slots overlap > 16:19:34 IBM Informix Dynamic Server Version 10.00.UC6 > 16:19:34 Who: Session(26, informix@t5240-03-z02, 0, 3fc324e0) > > Thread(129, xchg_1.8, 3fc1d548, 3) > > File: rsdebug.c Line: 1081 > 16:19:34 Results: Possible inconsistencies in an index of > 'campaign:"dbaparts".camptss_ses' > "/ifmx/ifmx00175/online.log" [Read only] 92 lines, 5329 characters > 16:19:30 Dataskip is now OFF for all dbspaces > 16:19:30 Restartable Restore has been ENABLED > 16:19:30 Recovery Mode > 16:19:33 DR: Secondary server connected > 16:19:34 DR: Configuration parameter values of the paired servers do not > match: > > Parameter: TAPESIZE > > Current server's value: 20480000 > > Paired server's value: 30720000 > > WARNING: The parameter values must match if ontape is used as > > backup/recovery tool. > > Please adjust this parameter value in onconfig file to correctly > > reflect the setting of the primary server. > 16:19:34 DR: Secondary server needs failure recovery > > 16:19:34 DR: Failure recovery from disk in progress ... > 16:19:34 Logical Recovery Started. > 16:19:34 10 recovery worker threads will be started. > 16:19:34 Start Logical Recovery - Start Log 54282, End Log ? > 16:19:34 Starting Log Position - 54282 0x10018 > 16:19:34 Assert Failed: Page Check Error in page_reorg: slots overlap > 16:19:34 IBM Informix Dynamic Server Version 10.00.UC6 > 16:19:34 Who: Session(26, informix@t5240-03-z02, 0, 3fc324e0) > > Thread(129, xchg_1.8, 3fc1d548, 3) > > File: rsdebug.c Line: 1081 > 16:19:34 Results: Possible inconsistencies in an index of > 'campaign:"dbaparts".camptss_ses' > 16:19:34 Action: Run 'oncheck -cI campaign:"dbaparts".camptss_ses' > 16:19:34 stack trace for pid 5543 written to > /dbengines/informix/tmp/af.46904f6 > 16:19:34 See Also: /dbengines/informix/tmp/af.46904f6, shmem.46904f6.0 > 16:19:34 Assert Warning: Error during recovery left index inconsistent. > 16:19:34 IBM Informix Dynamic Server Version 10.00.UC6 > 16:19:34 Who: Session(26, informix@t5240-03-z02, 0, 3fc33520) > > Thread(130, xchg_1.9, 3fc1da74, 5) > > File: rskey.c Line: 1516 > 16:19:34 Results: Index 'campaign:"dbaparts".camptss_ses# 120_163' is now > unusable > 16:19:34 Action: Run 'oncheck -cI campaign:"dbaparts".camptss_ses# 120_163' > 16:19:34 stack trace for pid 5559 written to > /dbengines/informix/tmp/af.46a04f6 > 16:19:34 See Also: /dbengines/informix/tmp/af.46a04f6 > 16:19:35 Rollforward of log record failed. iserrno = 126 > 16:19:35 Log Record: log = 54282, pos = 0x1906c, type = > OLDRSAM:HINSERT(40), > trans = 91 > 16:19:35 Error during recovery left index inconsistent. > 16:19:35 Rollforward of log record failed. iserrno = 105 > 16:19:35 Log Record: log = 54282, pos = 0x1917c, type = > OLDRSAM:ADDITEM(28), > trans = 91 > 16:19:35 Rollforward of log record failed. iserrno = 105 > 16:19:35 Log Record: log = 54282, pos = 0x1917c, type = > OLDRSAM:ADDITEM(28), > trans = 91 > 16:19:44 Assert Warning: Chunk 8 is being taken OFFLINE. > 16:19:44 IBM Informix Dynamic Server Version 10.00.UC6 > 16:19:44 Who: Session(26, informix@t5240-03-z02, 0, 3fc33520) > > Thread(126, xchg_1.5, 3fc1c5c4, 1) > > File: rsmirror.c Line: 1846 > 16:19:44 Results: Dynamic Server will block at next checkpoint > 16:19:44 Action: Shutdown (onmode -k) or override (onmode -O) > 16:19:44 stack trace for pid 5462 written to > /dbengines/informix/tmp/af.46604f6 > 16:19:44 See Also: /dbengines/informix/tmp/af.46604f6 > 16:19:45 Chunk 8 is being taken OFFLINE. > 16:19:45 Rollforward of log record failed. iserrno = 126 > 16:19:45 Log Record: log = 54282, pos = 0x1906c, type = > OLDRSAM:HINSERT(40), > trans = 91 > 16:19:45 Rollforward of log record failed. iserrno = 101 > 16:19:45 Log Record: log = 54282, pos = 0x1b044, type = OLDRSAM:UNIQID(17), > trans = 93 > 16:19:45 Rollforward of log record failed. iserrno = 101 > 16:19:45 Log

I would take a level 0 archive of the primaries and reestablish replication from scratch. It might be a good idea to run onchecks on the primaries to make sure that the inconsistencies did not originate there. The nochecks that are complaining about ownership should be run by user informix. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Jan 25, 2013 at 8:30 PM, Schleicher, Keith < Keith.Schleicher@searshc.com> wrote: > I have two separate servers with two similar but separate issues. Both > involve > Secondary servers in a DR setup. Both are due to a power outage that caused > these servers to power down without Informix being shut down. > > For the first server, here is the error log: > > 16:19:21 Event alarms enabled. ALARMPROG = > '/ifmx/ifmx00175/etc/log_full.sh' > 16:19:22 Booting Language <c> from module <> > 16:19:22 Loading Module <CNULL> > 16:19:22 Booting Language <builtin> from module <> > 16:19:22 Loading Module <BUILTINNULL> > 16:19:22 Could not disable priority aging: errno = 1 > 16:19:22 Could not disable priority aging: errno = 1 > 16:19:22 Could not disable priority aging: errno = 1 > 16:19:22 Could not disable priority aging: errno = 1 > 16:19:22 Could not disable priority aging: errno = 1 > 16:19:22 Could not disable priority aging: errno = 1 > 16:19:28 DR: DRAUTO is 0 (Off) > 16:19:28 Requested shared memory segment size rounded from 544KB to 1024KB > 16:19:29 IBM Informix Dynamic Server Version 10.00.UC6 Software Serial > Number > AAA#B000000 > 16:19:29 WARNING: Next backup of DBspace llogdbs must be level-0 backup. > 16:19:29 WARNING: Next backup of DBspace plogdbs must be level-0 backup. > 16:19:29 IBM Informix Dynamic Server Initialized -- Shared Memory > Initialized. > > 16:19:29 Physical Recovery Started at Page (3:29800). > 16:19:29 Physical Recovery Complete: 18 Pages Examined, 18 Pages Restored. > 16:19:29 DR: Trying to connect to primary server = ifmx00174_repl > 16:19:30 Dataskip is now OFF for all dbspaces > 16:19:30 Restartable Restore has been ENABLED > 16:19:30 Recovery Mode > 16:19:33 DR: Secondary server connected > 16:19:34 DR: Configuration parameter values of the paired servers do not > match: > > Parameter: TAPESIZE > > Current server's value: 20480000 > > Paired server's value: 30720000 > > WARNING: The parameter values must match if ontape is used as > > backup/recovery tool. > > Please adjust this parameter value in onconfig file to correctly > > reflect the setting of the primary server. > 16:19:34 DR: Secondary server needs failure recovery > > 16:19:34 DR: Failure recovery from disk in progress ... > 16:19:34 Logical Recovery Started. > 16:19:34 10 recovery worker threads will be started. > 16:19:34 Start Logical Recovery - Start Log 54282, End Log ? > 16:19:34 Starting Log Position - 54282 0x10018 > 16:19:34 Assert Failed: Page Check Error in page_reorg: slots overlap > 16:19:34 IBM Informix Dynamic Server Version 10.00.UC6 > 16:19:34 Who: Session(26, informix@t5240-03-z02, 0, 3fc324e0) > > Thread(129, xchg_1.8, 3fc1d548, 3) > > File: rsdebug.c Line: 1081 > 16:19:34 Results: Possible inconsistencies in an index of > 'campaign:"dbaparts".camptss_ses' > "/ifmx/ifmx00175/online.log" [Read only] 92 lines, 5329 characters > 16:19:30 Dataskip is now OFF for all dbspaces > 16:19:30 Restartable Restore has been ENABLED > 16:19:30 Recovery Mode > 16:19:33 DR: Secondary server connected > 16:19:34 DR: Configuration parameter values of the paired servers do not > match: > > Parameter: TAPESIZE > > Current server's value: 20480000 > > Paired server's value: 30720000 > > WARNING: The parameter values must match if ontape is used as > > backup/recovery tool. > > Please adjust this parameter value in onconfig file to correctly > > reflect the setting of the primary server. > 16:19:34 DR: Secondary server needs failure recovery > > 16:19:34 DR: Failure recovery from disk in progress ... > 16:19:34 Logical Recovery Started. > 16:19:34 10 recovery worker threads will be started. > 16:19:34 Start Logical Recovery - Start Log 54282, End Log ? > 16:19:34 Starting Log Position - 54282 0x10018 > 16:19:34 Assert Failed: Page Check Error in page_reorg: slots overlap > 16:19:34 IBM Informix Dynamic Server Version 10.00.UC6 > 16:19:34 Who: Session(26, informix@t5240-03-z02, 0, 3fc324e0) > > Thread(129, xchg_1.8, 3fc1d548, 3) > > File: rsdebug.c Line: 1081 > 16:19:34 Results: Possible inconsistencies in an index of > 'campaign:"dbaparts".camptss_ses' > 16:19:34 Action: Run 'oncheck -cI campaign:"dbaparts".camptss_ses' > 16:19:34 stack trace for pid 5543 written to > /dbengines/informix/tmp/af.46904f6 > 16:19:34 See Also: /dbengines/informix/tmp/af.46904f6, shmem.46904f6.0 > 16:19:34 Assert Warning: Error during recovery left index inconsistent. > 16:19:34 IBM Informix Dynamic Server Version 10.00.UC6 > 16:19:34 Who: Session(26, informix@t5240-03-z02, 0, 3fc33520) > > Thread(130, xchg_1.9, 3fc1da74, 5) > > File: rskey.c Line: 1516 > 16:19:34 Results: Index 'campaign:"dbaparts".camptss_ses# 120_163' is now > unusable > 16:19:34 Action: Run 'oncheck -cI campaign:"dbaparts".camptss_ses# 120_163' > 16:19:34 stack trace for pid 5559 written to > /dbengines/informix/tmp/af.46a04f6 > 16:19:34 See Also: /dbengines/informix/tmp/af.46a04f6 > 16:19:35 Rollforward of log record failed. iserrno = 126 > 16:19:35 Log Record: log = 54282, pos = 0x1906c, type = > OLDRSAM:HINSERT(40), > trans = 91 > 16:19:35 Error during recovery left index inconsistent. > 16:19:35 Rollforward of log record failed. iserrno = 105 > 16:19:35 Log Record: log = 54282, pos = 0x1917c, type = > OLDRSAM:ADDITEM(28), > trans = 91 > 16:19:35 Rollforward of log record failed. iserrno = 105 > 16:19:35 Log Record: log = 54282, pos = 0x1917c, type = > OLDRSAM:ADDITEM(28), > trans = 91 > 16:19:44 Assert Warning: Chunk 8 is being taken OFFLINE. > 16:19:44 IBM Informix Dynamic Server Version 10.00.UC6 > 16:19:44 Who: Session(26, informix@t5240-03-z02, 0, 3fc33520) > > Thread(126, xchg_1.5, 3fc1c5c4, 1) > > File: rsmirror.c Line: 1846 > 16:19:44 Results: Dynamic Server will block at next checkpoint > 16:19:44 Action: Shutdown (onmode -k) or override (onmode -O) > 16:19:44 stack trace for pid 5462 written to > /dbengines/informix/tmp/af.46604f6 > 16:19:44 See Also: /dbengines/informix/tmp/af.46604f6 > 16:19:45 Chunk 8 is being taken OFFLINE. > 16:19:45 Rollforward of log record failed. iserrno = 126 > 16:19:45 Log Record: log = 54282, pos = 0x1906c, type = > OLDRSAM:HINSERT(40), > trans = 91 > 16:19:45 Rollforw