HDR Issue / ER Differences
A hardware hang on the HDR secondary (IDS 9.40.FC6, Solaris 9) stalled the primary with a 650-second checkpoint, then 'Rollback error 126' asserts, two corrupted chunks and a server panic; IBM support restored the chunks and suspected a bug. The poster asked whether ER would insulate the primary better. Replies explained ER replicates per-transaction with fully independent servers (better isolation, but costlier/trickier setup, no auto-replication of new objects or triggers/SPs), while HDR is simpler one-way failover; one contributor noted the hang stemmed from the secondary hanging rather than crashing and suggested lowering DRTIMEOUT, saying the rollback error was a separate bug unrelated to HDR. No definitive root-cause fix is recorded; the poster planned to trial ER.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Hi All,
We are running HDR between two informix 9.4FC6 instances on solaris
9....we have just experienced a chronic hardware failure on our
secondary (Sun still trying to find out why - no syslog errors or
diagnostics yet) which caused a long checkpoint followed by a rollback
error (see below) which managed to corrupt two data chunks on our
primary server. IBM tech support dialed in and managed to bring the
chunks online which enabled us to recover the data affected. Tech
Support believe this to be a bug but are still looking into the
diagnostic information.
The real reason for bringing this to the group's attention is to try to
understand if we had been running ER rather than HDR would there be less
chance of suffering a failure on our primary server ? Are there
architectural differences between the two replication methods which
would make the primary server less reliant on whats happening on the
secondary server ?
Thanks in advance
Phil Torkington
Primary Online Log.... (can't get to the secondary one yet ;-) )
09:27:15 Checkpoint Completed: duration was 4 seconds.
09:27:15 Checkpoint loguniq 36786, logpos 0xec558c, timestamp:
0xc18b696e
09:27:15 Maximum server connections 921
09:43:05 DR: ping timeout
09:43:05 DR: Receive error
09:43:05 ASF Echo-Thread Server: asfcode = -25582: oserr = 0: errstr =
: Network connection is broken.
09:43:05 Checkpoint Completed: duration was 650 seconds.
09:43:05 Checkpoint loguniq 36786, logpos 0x1303018, timestamp:
0xc18f9e3d
09:43:05 Maximum server connections 921
09:43:06 Assert Failed: Rollback error 126
09:43:06 IBM Informix Dynamic Server Version 9.40.FC6
09:43:06 Who: Session(721514, ebask13@prdpctb1701.lastminute.com, -1,
1a0e76178)
Thread(766809, sqlexec, 1a1b11388, 4)
File: rstrans.c Line: 2427
09:43:06 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x1309570 was not rolled back
09:43:06 Action: Use 'onlog' to view the transaction and repair
manually.
09:43:06 stack trace for pid 4211 written to
/temp/informix_dump/af.b7413aa9
09:43:06 See Also: /temp/informix_dump/af.b7413aa9, shmem.b7413aa9.0
09:43:07 DR: Turned off on primary server
09:43:07 DR: Cannot connect to secondary server
09:43:07 Assert Failed: Rollback error 126
09:43:07 IBM Informix Dynamic Server Version 9.40.FC6
09:43:07 Who: Session(721935, ebask13@prdpctb1711.lastminute.com, -1,
19fdac1a8)
Thread(767232, sqlexec, 199ec30c0, 3)
File: rstrans.c Line: 2427
09:43:07 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x130a030 was not rolled back
09:43:07 Action: Use 'onlog' to view the transaction and repair
manually.
09:43:07 stack trace for pid 4210 written to
/temp/informix_dump/af.b8e83aab
09:43:07 See Also: /temp/informix_dump/af.b8e83aab, shmem.b8e83aab.0
09:43:09 Assert Failed: Rollback error 126
09:43:09 IBM Informix Dynamic Server Version 9.40.FC6
09:43:09 Who: Session(720848, ebask13@prdpctb1801.lastminute.com, -1,
19d749aa8)
Thread(766143, sqlexec, 199ed2570, 1)
File: rstrans.c Line: 2427
09:43:09 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x13093d0 was not rolled back
09:43:09 Action: Use 'onlog' to view the transaction and repair
manually.
09:43:09 stack trace for pid 4208 written to
/temp/informix_dump/af.b4a73aac
09:43:09 See Also: /temp/informix_dump/af.b4a73aac, shmem.b4a73aac.0
09:47:09 Rollback error 126
09:47:09 Rollback error 126
09:47:09 Rollback error 126
09:47:10 Assert Failed: Rollback error 126
09:47:10 IBM Informix Dynamic Server Version 9.40.FC6
09:47:10 Assert Failed: Rollback error 126
09:47:10 Who: Session(720848, ebask13@prdpctb1801.lastminute.com, -1,
19d749aa8)
Thread(766143, sqlexec, 199ed2570, 1)
File: rsextlog.c Line: 1643
09:47:10 IBM Informix Dynamic Server Version 9.40.FC6
09:47:10 Who: Session(721935, ebask13@prdpctb1711.lastminute.com, -1,
19fdac1a8)
Thread(767232, sqlexec, 199ec30c0, 3)
File: rsextlog.c Line: 1643
09:47:10 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x130a030 was not rolled back
09:47:10 Action: Use 'onlog' to view the transaction and repair
manually.
09:47:10 stack trace for pid 4210 written to
/temp/informix_dump/af.b8e83aab
09:47:10 See Also: /temp/informix_dump/af.b8e83aab
09:47:10 Assert Failed: Rollback error 126
09:47:10 IBM Informix Dynamic Server Version 9.40.FC6
09:47:10 Who: Session(721514, ebask13@prdpctb1701.lastminute.com, -1,
1a0e76178)
Thread(766809, sqlexec, 1a1b11388, 4)
File: rsextlog.c Line: 1643
09:47:10 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x1309570 was not rolled back
09:47:10 Action: Use 'onlog' to view the transaction and repair
manually.
09:47:10 stack trace for pid 4211 written to
/temp/informix_dump/af.b7413aa9
09:47:10 See Also: /temp/informix_dump/af.b7413aa9
09:47:10 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x13093d0 was not rolled back
09:47:10 Action: Use 'onlog' to view the transaction and repair
manually.
09:47:10 stack trace for pid 4208 written to
/temp/informix_dump/af.b4a73aac
09:47:10 See Also: /temp/informix_dump/af.b4a73aac
09:47:30 Rollback error 126
09:47:30 Rollback error 126
09:47:30 Rollback error 126
09:47:31 Assert Failed: Dynamic Server must abort
09:47:31 IBM Informix Dynamic Server Version 9.40.FC6
09:47:31 Who: Session(721514, ebask13@prdpctb1701.lastminute.com, -1,
1a0e76178)
Thread(766809, sqlexec, 1a1b11388, 4)
File: rslog.c Line: 3511
09:47:31 Results: Dynamic Server must abort
09:47:31 Action: Reinitialize shared memory
09:47:31 stack trace for pid 4211 written to
/temp/informix_dump/af.b7413aa9
09:47:31 See Also: /temp/informix_dump/af.b7413aa9
09:47:32 Assert Failed: Dynamic Server must abort
09:47:32 IBM Informix Dynamic Server Version 9.40.FC6
09:47:32 Who: Session(720848, ebask13@prdpctb1801.lastminute.com, -1,
19d749aa8)
Thread(766143, sqlexec, 199ed2570, 1)
File: rslog.c Line: 3511
09:47:32 Results: Dynamic Server must abort
09:47:32 Action: Reinitialize shared memory
09:47:32 stack trace for pid 4208 written to
/temp/informix_dump/af.b4a73aac
09:47:32 See Also: /temp/informix_dump/af.b4a73aac
09:47:32 Assert Failed: Dynamic Server must abort
09:47:32 IBM Informix Dynamic Server Version 9.40.FC6
09:47:32 Who: Session(721935, ebask13@prdpctb1711.lastminute.com, -1,
19fdac1a8)
Thread(767232, sqlexec, 199ec30c0, 3)
File: rslog.c Line: 3511
09:47:32 Results: Dynamic Server must abort
09:47:32 Action: Reinitialize shared memory
09:47:32 stack trace for pid 4210 written to
/temp/informix_dump/af.b8e83aab
09:47:32 See Also: /temp/informix_dump/af.b8e83aab
09:47:51 rslog.c, line 3511, thread 766809, proc id 4211, Dynamic
Server must abort.
09:47:51 rslog.c, line 3511, thread 767232, proc id 4210, Dynamic
Server must abort.
09:47:51 rslog.c, line 3511, thread 766143, proc id 4208, Dynamic
Server must abort.
09:47:52 Fatal error in ADM VP at mt.c:12358
09:47:52 Unexpected virtual processor termination, pid = 4210, exit =
0x100
09:47:53 PANIC: Attempting to bring system down
**********************************************************************
This email and any files transmitted with i
HDR v/s
ER. Interesting.
It depends on your configuration. In the case of ER, the transactions
get piled up in the primary server logs and when the secondary comes up,
it will try to apply the transaction. Is your secondary fast enough to
apply the transactions. How long can you tolerate the down time in your
secondary. How big is your transactional log in the primary. If the
transaction that needs to be applied to the secondary is present in the
next available log, then you either need to allocate more
logs/Blobspaces, otherwise it will go for a DDR lock ( I think it is
called as DDR lock, I might be wrong).
In my own experience with ER, I had issues when the server
configurations aren't the same. Please take a look at the Informix
Administrator and ER manuals.
Thank you,
Kannan Thirugnanam
-----Original Message-----
From: forum.subscriber@iiug.org [mailto:forum.subscriber@iiug.org] On
Behalf Of Phil Torkington
Sent: Thursday, December 15, 2005 12:23 PM
To: ids@iiug.org
Subject: HDR Issue / ER Differences [6113]
Hi All,
We are running HDR between two informix 9.4FC6 instances on solaris
9....we have just experienced a chronic hardware failure on our
secondary (Sun still trying to find out why - no syslog errors or
diagnostics yet) which caused a long checkpoint followed by a rollback
error (see below) which managed to corrupt two data chunks on our
primary server. IBM tech support dialed in and managed to bring the
chunks online which enabled us to recover the data affected. Tech
Support believe this to be a bug but are still looking into the
diagnostic information.
The real reason for bringing this to the group's attention is to try to
understand if we had been running ER rather than HDR would there be less
chance of suffering a failure on our primary server ? Are there
architectural differences between the two replication methods which
would make the primary server less reliant on whats happening on the
secondary server ?
Thanks in advance
Phil Torkington
Primary Online Log.... (can't get to the secondary one yet ;-) )
09:27:15 Checkpoint Completed: duration was 4 seconds.
09:27:15 Checkpoint loguniq 36786, logpos 0xec558c, timestamp:
0xc18b696e
09:27:15 Maximum server connections 921
09:43:05 DR: ping timeout
09:43:05 DR: Receive error
09:43:05 ASF Echo-Thread Server: asfcode = -25582: oserr = 0: errstr =
: Network connection is broken.
09:43:05 Checkpoint Completed: duration was 650 seconds.
09:43:05 Checkpoint loguniq 36786, logpos 0x1303018, timestamp:
0xc18f9e3d
09:43:05 Maximum server connections 921
09:43:06 Assert Failed: Rollback error 126
09:43:06 IBM Informix Dynamic Server Version 9.40.FC6
09:43:06 Who: Session(721514, ebask13@prdpctb1701.lastminute.com, -1,
1a0e76178)
Thread(766809, sqlexec, 1a1b11388, 4)
File: rstrans.c Line: 2427
09:43:06 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x1309570 was not rolled back
09:43:06 Action: Use 'onlog' to view the transaction and repair
manually.
09:43:06 stack trace for pid 4211 written to
/temp/informix_dump/af.b7413aa9
09:43:06 See Also: /temp/informix_dump/af.b7413aa9, shmem.b7413aa9.0
09:43:07 DR: Turned off on primary server
09:43:07 DR: Cannot connect to secondary server
09:43:07 Assert Failed: Rollback error 126
09:43:07 IBM Informix Dynamic Server Version 9.40.FC6
09:43:07 Who: Session(721935, ebask13@prdpctb1711.lastminute.com, -1,
19fdac1a8)
Thread(767232, sqlexec, 199ec30c0, 3)
File: rstrans.c Line: 2427
09:43:07 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x130a030 was not rolled back
09:43:07 Action: Use 'onlog' to view the transaction and repair
manually.
09:43:07 stack trace for pid 4210 written to
/temp/informix_dump/af.b8e83aab
09:43:07 See Also: /temp/informix_dump/af.b8e83aab, shmem.b8e83aab.0
09:43:09 Assert Failed: Rollback error 126
09:43:09 IBM Informix Dynamic Server Version 9.40.FC6
09:43:09 Who: Session(720848, ebask13@prdpctb1801.lastminute.com, -1,
19d749aa8)
Thread(766143, sqlexec, 199ed2570, 1)
File: rstrans.c Line: 2427
09:43:09 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x13093d0 was not rolled back
09:43:09 Action: Use 'onlog' to view the transaction and repair
manually.
09:43:09 stack trace for pid 4208 written to
/temp/informix_dump/af.b4a73aac
09:43:09 See Also: /temp/informix_dump/af.b4a73aac, shmem.b4a73aac.0
09:47:09 Rollback error 126
09:47:09 Rollback error 126
09:47:09 Rollback error 126
09:47:10 Assert Failed: Rollback error 126
09:47:10 IBM Informix Dynamic Server Version 9.40.FC6
09:47:10 Assert Failed: Rollback error 126
09:47:10 Who: Session(720848, ebask13@prdpctb1801.lastminute.com, -1,
19d749aa8)
Thread(766143, sqlexec, 199ed2570, 1)
File: rsextlog.c Line: 1643
09:47:10 IBM Informix Dynamic Server Version 9.40.FC6
09:47:10 Who: Session(721935, ebask13@prdpctb1711.lastminute.com, -1,
19fdac1a8)
Thread(767232, sqlexec, 199ec30c0, 3)
File: rsextlog.c Line: 1643
09:47:10 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x130a030 was not rolled back
09:47:10 Action: Use 'onlog' to view the transaction and repair
manually.
09:47:10 stack trace for pid 4210 written to
/temp/informix_dump/af.b8e83aab
09:47:10 See Also: /temp/informix_dump/af.b8e83aab
09:47:10 Assert Failed: Rollback error 126
09:47:10 IBM Informix Dynamic Server Version 9.40.FC6
09:47:10 Who: Session(721514, ebask13@prdpctb1701.lastminute.com, -1,
1a0e76178)
Thread(766809, sqlexec, 1a1b11388, 4)
File: rsextlog.c Line: 1643
09:47:10 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x1309570 was not rolled back
09:47:10 Action: Use 'onlog' to view the transaction and repair
manually.
09:47:10 stack trace for pid 4211 written to
/temp/informix_dump/af.b7413aa9
09:47:10 See Also: /temp/informix_dump/af.b7413aa9
09:47:10 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x13093d0 was not rolled back
09:47:10 Action: Use 'onlog' to view the transaction and repair
manually.
09:47:10 stack trace for pid 4208 written to
/temp/informix_dump/af.b4a73aac
09:47:10 See Also: /temp/informix_dump/af.b4a73aac
09:47:30 Rollback error 126
09:47:30 Rollback error 126
09:47:30 Rollback error 126
09:47:31 Assert Failed: Dynamic Server must abort
09:47:31 IBM Informix Dynamic Server Version 9.40.FC6
09:47:31 Who: Session(721514, ebask13@prdpctb1701.lastminute.com, -1,
1a0e76178)
Thread(766809, sqlexec, 1a1b11388, 4)
File: rslog.c Line: 3511
09:47:31 Results: Dynamic Server must abort
09:47:31 Action: Reinitialize shared memory
09:47:31 stack trace for pid 4211 written to
/temp/informix_dump/af.b7413aa9
09:47:31 See Also: /temp/informix_dump/af.b7413aa9
09:47:32 Assert Failed: Dynamic Server must abort
09:47:32 IBM Informix Dynamic Server Version 9.40.FC6
09:47:32 Who: Session(720848, ebask13@prdpctb1801.lastminute.com, -1,
19d749aa8)
Thread(766143, sqlexec, 199ed2570, 1)
File: rslog.c Line: 3511
09:47:32 Results: Dynamic Server must abort
09:47:32 Action: Reinitialize shared memory
09:47:32 stack trace for pid 4208 written to
/temp/informix_dump/af.b4a73aac
09:47:32 Se
Hi Phil,
In my experience ER works at the transaction level whereas HDR appears
more to apply changes at the chunk level (I am unsure quite how this
works as both use the logical log entries).
Regards,
Nick
-----Original Message-----
From: forum.subscriber@iiug.org [mailto:forum.subscriber@iiug.org] On
Behalf Of Phil Torkington
Sent: Thursday, December 15, 2005 9:23 AM
To: ids@iiug.org
Subject: HDR Issue / ER Differences [6113]
Hi All,
We are running HDR between two informix 9.4FC6 instances on solaris
9....we have just experienced a chronic hardware failure on our
secondary (Sun still trying to find out why - no syslog errors or
diagnostics yet) which caused a long checkpoint followed by a rollback
error (see below) which managed to corrupt two data chunks on our
primary server. IBM tech support dialed in and managed to bring the
chunks online which enabled us to recover the data affected. Tech
Support believe this to be a bug but are still looking into the
diagnostic information.
The real reason for bringing this to the group's attention is to try to
understand if we had been running ER rather than HDR would there be less
chance of suffering a failure on our primary server ? Are there
architectural differences between the two replication methods which
would make the primary server less reliant on whats happening on the
secondary server ?
Thanks in advance
Phil Torkington
Primary Online Log.... (can't get to the secondary one yet ;-) )
09:27:15 Checkpoint Completed: duration was 4 seconds.
09:27:15 Checkpoint loguniq 36786, logpos 0xec558c, timestamp:
0xc18b696e
09:27:15 Maximum server connections 921
09:43:05 DR: ping timeout
09:43:05 DR: Receive error
09:43:05 ASF Echo-Thread Server: asfcode = -25582: oserr = 0: errstr =
: Network connection is broken.
09:43:05 Checkpoint Completed: duration was 650 seconds.
09:43:05 Checkpoint loguniq 36786, logpos 0x1303018, timestamp:
0xc18f9e3d
09:43:05 Maximum server connections 921
09:43:06 Assert Failed: Rollback error 126
09:43:06 IBM Informix Dynamic Server Version 9.40.FC6
09:43:06 Who: Session(721514, ebask13@prdpctb1701.lastminute.com, -1,
1a0e76178)
Thread(766809, sqlexec, 1a1b11388, 4)
File: rstrans.c Line: 2427
09:43:06 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x1309570 was not rolled back
09:43:06 Action: Use 'onlog' to view the transaction and repair
manually.
09:43:06 stack trace for pid 4211 written to
/temp/informix_dump/af.b7413aa9
09:43:06 See Also: /temp/informix_dump/af.b7413aa9, shmem.b7413aa9.0
09:43:07 DR: Turned off on primary server
09:43:07 DR: Cannot connect to secondary server
09:43:07 Assert Failed: Rollback error 126
09:43:07 IBM Informix Dynamic Server Version 9.40.FC6
09:43:07 Who: Session(721935, ebask13@prdpctb1711.lastminute.com, -1,
19fdac1a8)
Thread(767232, sqlexec, 199ec30c0, 3)
File: rstrans.c Line: 2427
09:43:07 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x130a030 was not rolled back
09:43:07 Action: Use 'onlog' to view the transaction and repair
manually.
09:43:07 stack trace for pid 4210 written to
/temp/informix_dump/af.b8e83aab
09:43:07 See Also: /temp/informix_dump/af.b8e83aab, shmem.b8e83aab.0
09:43:09 Assert Failed: Rollback error 126
09:43:09 IBM Informix Dynamic Server Version 9.40.FC6
09:43:09 Who: Session(720848, ebask13@prdpctb1801.lastminute.com, -1,
19d749aa8)
Thread(766143, sqlexec, 199ed2570, 1)
File: rstrans.c Line: 2427
09:43:09 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x13093d0 was not rolled back
09:43:09 Action: Use 'onlog' to view the transaction and repair
manually.
09:43:09 stack trace for pid 4208 written to
/temp/informix_dump/af.b4a73aac
09:43:09 See Also: /temp/informix_dump/af.b4a73aac, shmem.b4a73aac.0
09:47:09 Rollback error 126
09:47:09 Rollback error 126
09:47:09 Rollback error 126
09:47:10 Assert Failed: Rollback error 126 09:47:10 IBM Informix
Dynamic Server Version 9.40.FC6 09:47:10 Assert Failed: Rollback error
126
09:47:10 Who: Session(720848, ebask13@prdpctb1801.lastminute.com, -1,
19d749aa8)
Thread(766143, sqlexec, 199ed2570, 1)
File: rsextlog.c Line: 1643 09:47:10 IBM Informix
Dynamic Server Version 9.40.FC6
09:47:10 Who: Session(721935, ebask13@prdpctb1711.lastminute.com, -1,
19fdac1a8)
Thread(767232, sqlexec, 199ec30c0, 3)
File: rsextlog.c Line: 1643
09:47:10 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x130a030 was not rolled back
09:47:10 Action: Use 'onlog' to view the transaction and repair
manually.
09:47:10 stack trace for pid 4210 written to
/temp/informix_dump/af.b8e83aab
09:47:10 See Also: /temp/informix_dump/af.b8e83aab
09:47:10 Assert Failed: Rollback error 126 09:47:10 IBM Informix
Dynamic Server Version 9.40.FC6
09:47:10 Who: Session(721514, ebask13@prdpctb1701.lastminute.com, -1,
1a0e76178)
Thread(766809, sqlexec, 1a1b11388, 4)
File: rsextlog.c Line: 1643
09:47:10 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x1309570 was not rolled back
09:47:10 Action: Use 'onlog' to view the transaction and repair
manually.
09:47:10 stack trace for pid 4211 written to
/temp/informix_dump/af.b7413aa9
09:47:10 See Also: /temp/informix_dump/af.b7413aa9
09:47:10 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x13093d0 was not rolled back
09:47:10 Action: Use 'onlog' to view the transaction and repair
manually.
09:47:10 stack trace for pid 4208 written to
/temp/informix_dump/af.b4a73aac
09:47:10 See Also: /temp/informix_dump/af.b4a73aac
09:47:30 Rollback error 126
09:47:30 Rollback error 126
09:47:30 Rollback error 126
09:47:31 Assert Failed: Dynamic Server must abort
09:47:31 IBM Informix Dynamic Server Version 9.40.FC6
09:47:31 Who: Session(721514, ebask13@prdpctb1701.lastminute.com, -1,
1a0e76178)
Thread(766809, sqlexec, 1a1b11388, 4)
File: rslog.c Line: 3511
09:47:31 Results: Dynamic Server must abort
09:47:31 Action: Reinitialize shared memory
09:47:31 stack trace for pid 4211 written to
/temp/informix_dump/af.b7413aa9
09:47:31 See Also: /temp/informix_dump/af.b7413aa9
09:47:32 Assert Failed: Dynamic Server must abort
09:47:32 IBM Informix Dynamic Server Version 9.40.FC6
09:47:32 Who: Session(720848, ebask13@prdpctb1801.lastminute.com, -1,
19d749aa8)
Thread(766143, sqlexec, 199ed2570, 1)
File: rslog.c Line: 3511
09:47:32 Results: Dynamic Server must abort
09:47:32 Action: Reinitialize shared memory
09:47:32 stack trace for pid 4208 written to
/temp/informix_dump/af.b4a73aac
09:47:32 See Also: /temp/informix_dump/af.b4a73aac
09:47:32 Assert Failed: Dynamic Server must abort
09:47:32 IBM Informix Dynamic Server Version 9.40.FC6
09:47:32 Who: Session(721935, ebask13@prdpctb1711.lastminute.com, -1,
19fdac1a8)
Thread(767232, sqlexec, 199ec30c0, 3)
File: rslog.c Line: 3511
09:47:32 Results: Dynamic Server must abort
09:47:32 Action: Reinitialize shared memory
09:47:32 stack trace for pid 4210 written to
/temp/informix_dump/af.b8e83aab
09:47:32 See Also: /temp/informix_dump/af.b8e83aab
09:47:51 rslog.c, line 3511, thread 766809, proc id 4211, Dynamic
Server must abort.
Yes, there are big differences between HDR and ER.
HDR is usually implemented as a failover methodology for disaster recovery. Or
it is widely used if you would want to use the secondary to handle all the
reporting work thus minimizing the load on primary server, and isolating the
primary for direct updates, deletes, inserts and such heavy OLTP related DML
queries which are business critical. HDR is the best replication method if you
do not need update anywhere mechanism which revolves around conflict
resolutions methodology for your business.
ER is implemented when a business needs update anywhere mechanism. (like
Oracle Streams but better because ER is easier to manage). And, ER allows you
to add more instances to the topology as the business grows.
To answer your question in a disaster recovery point of view, ER is better
because you would usually have multiple copies of the same database in several
locations. However, ER implementation is very costly but the advantages are
tremendous. So, ER is sort of like having primary servers everywhere.
HDR on the other hand is one way stream (A>B not B>A) but relatively much
easier to manage as opposed to ER and its very inexpensive.
To answer your question about "reliance", all replication methodologies have
an internal locking mechanism (some are more complex) to assure that the
transactions have been applied to the other counterparts regardless of the
method being used. So, had you used ER, still the internal locking is the same
but you would have had more avenues for uptime since you would have had at
least more than one secondary box.
Now, Informix new versions allow both HDR and ER in the same topology.
My experienced has been that regardless of which methodology being used, if
you run a huge transaction in a replicated environment 9 out of 10 it will
lock up the systems. For instance, you do not want to create an index on a
table consists of 400 million rows that is part of a replicated server without
turning off replication first. Also, trying to alter a table with 200 million
rows could end up the servers in the same spot if its not done after turning
off replication.
Hope these helps.
Ravi.
"Lello, Nick" <Nick.Lello@NielsenEDI.com> wrote: Hi Phil,
In my experience ER works at the transaction level whereas HDR appears
more to apply changes at the chunk level (I am unsure quite how this
works as both use the logical log entries).
Regards,
Nick
-----Original Message-----
From: forum.subscriber@iiug.org [mailto:forum.subscriber@iiug.org] On
Behalf Of Phil Torkington
Sent: Thursday, December 15, 2005 9:23 AM
To: ids@iiug.org
Subject: HDR Issue / ER Differences [6113]
Hi All,
We are running HDR between two informix 9.4FC6 instances on solaris
9....we have just experienced a chronic hardware failure on our
secondary (Sun still trying to find out why - no syslog errors or
diagnostics yet) which caused a long checkpoint followed by a rollback
error (see below) which managed to corrupt two data chunks on our
primary server. IBM tech support dialed in and managed to bring the
chunks online which enabled us to recover the data affected. Tech
Support believe this to be a bug but are still looking into the
diagnostic information.
The real reason for bringing this to the group's attention is to try to
understand if we had been running ER rather than HDR would there be less
chance of suffering a failure on our primary server ? Are there
architectural differences between the two replication methods which
would make the primary server less reliant on whats happening on the
secondary server ?
Thanks in advance
Phil Torkington
Primary Online Log.... (can't get to the secondary one yet ;-) )
09:27:15 Checkpoint Completed: duration was 4 seconds.
09:27:15 Checkpoint loguniq 36786, logpos 0xec558c, timestamp:
0xc18b696e
09:27:15 Maximum server connections 921
09:43:05 DR: ping timeout
09:43:05 DR: Receive error
09:43:05 ASF Echo-Thread Server: asfcode = -25582: oserr = 0: errstr =
: Network connection is broken.
09:43:05 Checkpoint Completed: duration was 650 seconds.
09:43:05 Checkpoint loguniq 36786, logpos 0x1303018, timestamp:
0xc18f9e3d
09:43:05 Maximum server connections 921
09:43:06 Assert Failed: Rollback error 126
09:43:06 IBM Informix Dynamic Server Version 9.40.FC6
09:43:06 Who: Session(721514, ebask13@prdpctb1701.lastminute.com, -1,
1a0e76178)
Thread(766809, sqlexec, 1a1b11388, 4)
File: rstrans.c Line: 2427
09:43:06 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x1309570 was not rolled back
09:43:06 Action: Use 'onlog' to view the transaction and repair
manually.
09:43:06 stack trace for pid 4211 written to
/temp/informix_dump/af.b7413aa9
09:43:06 See Also: /temp/informix_dump/af.b7413aa9, shmem.b7413aa9.0
09:43:07 DR: Turned off on primary server
09:43:07 DR: Cannot connect to secondary server
09:43:07 Assert Failed: Rollback error 126
09:43:07 IBM Informix Dynamic Server Version 9.40.FC6
09:43:07 Who: Session(721935, ebask13@prdpctb1711.lastminute.com, -1,
19fdac1a8)
Thread(767232, sqlexec, 199ec30c0, 3)
File: rstrans.c Line: 2427
09:43:07 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x130a030 was not rolled back
09:43:07 Action: Use 'onlog' to view the transaction and repair
manually.
09:43:07 stack trace for pid 4210 written to
/temp/informix_dump/af.b8e83aab
09:43:07 See Also: /temp/informix_dump/af.b8e83aab, shmem.b8e83aab.0
09:43:09 Assert Failed: Rollback error 126
09:43:09 IBM Informix Dynamic Server Version 9.40.FC6
09:43:09 Who: Session(720848, ebask13@prdpctb1801.lastminute.com, -1,
19d749aa8)
Thread(766143, sqlexec, 199ed2570, 1)
File: rstrans.c Line: 2427
09:43:09 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x13093d0 was not rolled back
09:43:09 Action: Use 'onlog' to view the transaction and repair
manually.
09:43:09 stack trace for pid 4208 written to
/temp/informix_dump/af.b4a73aac
09:43:09 See Also: /temp/informix_dump/af.b4a73aac, shmem.b4a73aac.0
09:47:09 Rollback error 126
09:47:09 Rollback error 126
09:47:09 Rollback error 126
09:47:10 Assert Failed: Rollback error 126 09:47:10 IBM Informix
Dynamic Server Version 9.40.FC6 09:47:10 Assert Failed: Rollback error
126
09:47:10 Who: Session(720848, ebask13@prdpctb1801.lastminute.com, -1,
19d749aa8)
Thread(766143, sqlexec, 199ed2570, 1)
File: rsextlog.c Line: 1643 09:47:10 IBM Informix
Dynamic Server Version 9.40.FC6
09:47:10 Who: Session(721935, ebask13@prdpctb1711.lastminute.com, -1,
19fdac1a8)
Thread(767232, sqlexec, 199ec30c0, 3)
File: rsextlog.c Line: 1643
09:47:10 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x130a030 was not rolled back
09:47:10 Action: Use 'onlog' to view the transaction and repair
manually.
09:47:10 stack trace for pid 4210 written to
/temp/informix_dump/af.b8e83aab
09:47:10 See Also: /temp/informix_dump/af.b8e83aab
09:47:10 Assert Failed: Rollback error 126 09:47:10 IBM Informix
Dynamic Server Version 9.40.FC6
09:47:10 Who: Session(721514, ebask13@prdpctb1701.lastminute.com, -1,
1a0e76178)
Thread(766809,
Hang on primary is caused because secondary system/machine was hanging. If machine or IDS were crashed on secondary system, you should not have seen the hang on primary. To avoid such hang on primary, it is better to set the DRTIMEOUT value to a smaller number. This will allow primary to quickly timeout if primary does not get response from secondary in the desired time. Other problem that you are seeing is some bug in rollback code. It has nothing to do with HDR.
Thanks
for all the advice
The fact that the two servers are totally independent is very appealing
but also take note of the extra set-up / maintenance over HDR.
We will test ER in the new year and get a feel for how much more setting
up/maintenance is needed.
________________________________
From: Madison Pruet [mailto:mpruet@us.ibm.com]
Sent: 15 December 2005 20:24
To: Phil Torkington
Cc: forum.subscriber@iiug.org; ids@iiug.org
Subject: Re: HDR Issue / ER Differences [6113]
With ER the servers are totally independent. However ER is a bit more
tricky to set up.
I think that we've improved the setup issues significantly with v10,
especially with the additional functionality which was activated in
v10UC4. This would allow you, for instance, to set up replication via
templates with basically only two commands.
However some things to think about. ER does not currently automatically
create replication on objects which do not yet exist. Also, ER does not
currently support the automatic creation of secondary objects such as
triggers or stored procedures.
M.P.
"Phil Torkington" <PTorkington@lastminute.com>
"Phil Torkington" <PTorkington@lastminute.com>
Sent by: forum.subscriber@iiug.org
12/15/2005 11:23 AM
To
ids@iiug.org
cc
Subject
HDR Issue / ER Differences [6113]
Hi All,
We are running HDR between two informix 9.4FC6 instances on solaris
9....we have just experienced a chronic hardware failure on our
secondary (Sun still trying to find out why - no syslog errors or
diagnostics yet) which caused a long checkpoint followed by a rollback
error (see below) which managed to corrupt two data chunks on our
primary server. IBM tech support dialed in and managed to bring the
chunks online which enabled us to recover the data affected. Tech
Support believe this to be a bug but are still looking into the
diagnostic information.
The real reason for bringing this to the group's attention is to try to
understand if we had been running ER rather than HDR would there be less
chance of suffering a failure on our primary server ? Are there
architectural differences between the two replication methods which
would make the primary server less reliant on whats happening on the
secondary server ?
Thanks in advance
Phil Torkington
Primary Online Log.... (can't get to the secondary one yet ;-) )
09:27:15 Checkpoint Completed: duration was 4 seconds.
09:27:15 Checkpoint loguniq 36786, logpos 0xec558c, timestamp:
0xc18b696e
09:27:15 Maximum server connections 921
09:43:05 DR: ping timeout
09:43:05 DR: Receive error
09:43:05 ASF Echo-Thread Server: asfcode = -25582: oserr = 0: errstr =
: Network connection is broken.
09:43:05 Checkpoint Completed: duration was 650 seconds.
09:43:05 Checkpoint loguniq 36786, logpos 0x1303018, timestamp:
0xc18f9e3d
09:43:05 Maximum server connections 921
09:43:06 Assert Failed: Rollback error 126
09:43:06 IBM Informix Dynamic Server Version 9.40.FC6
09:43:06 Who: Session(721514, ebask13@prdpctb1701.lastminute.com, -1,
1a0e76178)
Thread(766809, sqlexec, 1a1b11388, 4)
File: rstrans.c Line: 2427
09:43:06 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x1309570 was not rolled back
09:43:06 Action: Use 'onlog' to view the transaction and repair
manually.
09:43:06 stack trace for pid 4211 written to
/temp/informix_dump/af.b7413aa9
09:43:06 See Also: /temp/informix_dump/af.b7413aa9, shmem.b7413aa9.0
09:43:07 DR: Turned off on primary server
09:43:07 DR: Cannot connect to secondary server
09:43:07 Assert Failed: Rollback error 126
09:43:07 IBM Informix Dynamic Server Version 9.40.FC6
09:43:07 Who: Session(721935, ebask13@prdpctb1711.lastminute.com, -1,
19fdac1a8)
Thread(767232, sqlexec, 199ec30c0, 3)
File: rstrans.c Line: 2427
09:43:07 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x130a030 was not rolled back
09:43:07 Action: Use 'onlog' to view the transaction and repair
manually.
09:43:07 stack trace for pid 4210 written to
/temp/informix_dump/af.b8e83aab
09:43:07 See Also: /temp/informix_dump/af.b8e83aab, shmem.b8e83aab.0
09:43:09 Assert Failed: Rollback error 126
09:43:09 IBM Informix Dynamic Server Version 9.40.FC6
09:43:09 Who: Session(720848, ebask13@prdpctb1801.lastminute.com, -1,
19d749aa8)
Thread(766143, sqlexec, 199ed2570, 1)
File: rstrans.c Line: 2427
09:43:09 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x13093d0 was not rolled back
09:43:09 Action: Use 'onlog' to view the transaction and repair
manually.
09:43:09 stack trace for pid 4208 written to
/temp/informix_dump/af.b4a73aac
09:43:09 See Also: /temp/informix_dump/af.b4a73aac, shmem.b4a73aac.0
09:47:09 Rollback error 126
09:47:09 Rollback error 126
09:47:09 Rollback error 126
09:47:10 Assert Failed: Rollback error 126
09:47:10 IBM Informix Dynamic Server Version 9.40.FC6
09:47:10 Assert Failed: Rollback error 126
09:47:10 Who: Session(720848, ebask13@prdpctb1801.lastminute.com, -1,
19d749aa8)
Thread(766143, sqlexec, 199ed2570, 1)
File: rsextlog.c Line: 1643
09:47:10 IBM Informix Dynamic Server Version 9.40.FC6
09:47:10 Who: Session(721935, ebask13@prdpctb1711.lastminute.com, -1,
19fdac1a8)
Thread(767232, sqlexec, 199ec30c0, 3)
File: rsextlog.c Line: 1643
09:47:10 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x130a030 was not rolled back
09:47:10 Action: Use 'onlog' to view the transaction and repair
manually.
09:47:10 stack trace for pid 4210 written to
/temp/informix_dump/af.b8e83aab
09:47:10 See Also: /temp/informix_dump/af.b8e83aab
09:47:10 Assert Failed: Rollback error 126
09:47:10 IBM Informix Dynamic Server Version 9.40.FC6
09:47:10 Who: Session(721514, ebask13@prdpctb1701.lastminute.com, -1,
1a0e76178)
Thread(766809, sqlexec, 1a1b11388, 4)
File: rsextlog.c Line: 1643
09:47:10 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x1309570 was not rolled back
09:47:10 Action: Use 'onlog' to view the transaction and repair
manually.
09:47:10 stack trace for pid 4211 written to
/temp/informix_dump/af.b7413aa9
09:47:10 See Also: /temp/informix_dump/af.b7413aa9
09:47:10 Results: Log record (OLDRSAM:HUPDATE) in log 36786, offset
0x13093d0 was not rolled back
09:47:10 Action: Use 'onlog' to view the transaction and repair
manually.
09:47:10 stack trace for pid 4208 written to
/temp/informix_dump/af.b4a73aac
09:47:10 See Also: /temp/informix_dump/af.b4a73aac
09:47:30 Rollback error 126
09:47:30 Rollback error 126
09:47:30 Rollback error 126
09:47:31 Assert Failed: Dynamic Server must abort
09:47:31 IBM Informix Dynamic Server Version 9.40.FC6
09:47:31 Who: Session(721514, ebask13@prdpctb1701.lastminute.com, -1,
1a0e76178)
Thread(766809, sqlexec, 1a1b11388, 4)
File: rslog.c Line: 3511
09:47:31 Results: Dynamic Server must abort
09:47:31 Action: Reinitialize shared memory
09:47:31 stack trace for pid 4211 written to
/temp/informix_dump/af.b7413aa9
09:47:31 See Also: /temp/informix_dump/af.b7413aa9
09:47:32 Assert Failed: Dynamic Server must abort
09:47:32 IBM Informix Dynamic Server Ve
Related threads
- System Or Internal Error InterruptedIOException
- 25582 error on high volume of short live trans
- ASF Echo-Thread Server: asfcode = -25582 oserr = 4