HDR crashed after upgradation

hi all i was using IDS 11.5FC4, i have two SDS and one HDR instance.after upgrading all the databases to 11.5FC5 i started the primary database and then i start the application wothout starting the SDS and HDR databases.After some time primary database blocked and stated showing the following status -------------online-Prim (CKPT REQ)---------------- Blocked CKPT Could not shutdown the primary database so thats the reason i killed the processes and shutdown the database and then start the primary database. after that i try to start the HDR database but it crashed and shows following error in the message log. 14:35:40 DR_ERR set to -3 14:35:40 Assert Failed: DR: Log Record Apply Thread Exited Abnormally. Internal Error. A restart of the database server shall be required to correct this problem. 14:35:40 IBM Informix Dynamic Server Version 11.50.FC4 14:35:40 Who: Session(151, informix@, 0, c0000002e3693ae8) Thread(294, dr_secapply, c0000002e364fb00, 3) File: hdr/hdrApply.c Line: 511 14:35:40 Results: Dynamic Server must abort 14:35:40 Action: Reinitialize shared memory 14:35:40 stack trace for pid 24130 written to /work/tmp/af.50e86e4 14:35:40 See Also: /work/tmp/af.50e86e4, shmem.50e86e4.0 14:36:44 hdr/hdrApply.c, line 511, thread 294, proc id 24130, DR: Log Record Apply Thread Exited Abnormally. Internal Error. A restart of the database server shall be required to correct this problem. . 14:36:45 Fatal error in ADM VP at mt.c:13836 14:36:45 Unexpected virtual processor termination, pid = 24130, exit = 0x100 14:36:45 PANIC: Attempting to bring system down what could be the reason of this crash? Was there anything wrong with the way databases upgraded? Poonam

Open a case with IBM. They will be able to fix this. It sounds like the engine did not shutdown properly before you restarted with the new version. Art Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Sep 11, 2009 at 3:10 AM, POONAM TANDEL <poonam_tandel28@yahoo.co.in>wrote: > hi all > > i was using IDS 11.5FC4, i have two SDS and one HDR instance.after > upgrading > all the databases to 11.5FC5 i started the primary database and then i > start > the application wothout starting the SDS and HDR databases.After some time > primary database blocked and stated showing the following status > > -------------online-Prim (CKPT REQ)---------------- > Blocked CKPT > > Could not shutdown the primary database so thats the reason i killed the > processes and shutdown the database and then start the primary database. > after that i try to start the HDR database but it crashed and shows > following > error in the message log. > > 14:35:40 DR_ERR set to -3 > 14:35:40 Assert Failed: DR: Log Record Apply Thread Exited Abnormally. > Internal Error. > > A restart of the database server shall be required to correct > > this problem. > > 14:35:40 IBM Informix Dynamic Server Version 11.50.FC4 > 14:35:40 Who: Session(151, informix@, 0, c0000002e3693ae8) > > Thread(294, dr_secapply, c0000002e364fb00, 3) > > File: hdr/hdrApply.c Line: 511 > 14:35:40 Results: Dynamic Server must abort > 14:35:40 Action: Reinitialize shared memory > 14:35:40 stack trace for pid 24130 written to /work/tmp/af.50e86e4 > 14:35:40 See Also: /work/tmp/af.50e86e4, shmem.50e86e4.0 > 14:36:44 hdr/hdrApply.c, line 511, thread 294, proc id 24130, DR: Log > Record > Apply Thread Exited Abnormally. Internal Error. > > A restart of the database server shall be required to correct > > this problem. > .. > 14:36:45 Fatal error in ADM VP at mt.c:13836 > 14:36:45 Unexpected virtual processor termination, pid = 24130, exit = > 0x100 > > 14:36:45 PANIC: Attempting to bring system down > > what could be the reason of this crash? > Was there anything wrong with the way databases upgraded? > > Poonam > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --0023545bda2866551204734c0e4e

HI Art did u mean that that crash happend becz i didnt shutdown the primary database properly. I thought that the crash happend becz of the way i upgraded the databases. To upgrade the database i have performed the following steps. 1) Shutdown the HDR and SDS instances 2) Shutdown the Primary database 3)Installed the binaries on all the databases 4) Started the primary database 5) Then started the SDS and HDR Is this a right way to upgrade the database in MACH11 cluster?

Your procedure is correct. I would only add that I would have done only one thing different. I would install the new version BEFORE shutting down the existing instances but install into a different directory. I always use a soft link for INFORMIXDIR and point that to the actual install directory. So, I would install the new release in a new directory - say /opt/informix/ifmx.11.50.FC5 - then: - Shutdown all instances. - Link the value I use for INFORMIXDIR away from the prrevious install directory - say /opt/informix/infmx.11.50.FC4 - to now point to the new install directory. - Restart all instances. This method has several advantages: 1. Reduces downtime to seconds instead of from 10 minutes to an hour if you have to wait for the installs to complete. 2. You still don't have to change any scripts or have users reset their environment 3. You can revert to the previous version in the same several seconds by simply: 1. For major upgrades run the onmode reversion to the previous release. Not needed for minor release upgrades. 2. Shutdown 3. Relink INFORMIXDIR to the old release 4. Restart the servers using the older software 5. Optionally, drop and recreate the support databases: sysmaster, sysadmin, sysusers - in case these were updated by the new release. But that has nothing to do with your problem, I know. Yes, IB that the shutdown did not complete correctly and left some partial logical log record. That inconsistency is what is confusing the engine at startup. IBM can connect to your system and truncate the last logical log to a point where the engine will be able to complete fast recovery successfully. You might lose a transaction or two, but since you were performing a controlled shutdown that is unlikely. Art Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Sep 11, 2009 at 9:43 AM, POONAM TANDEL <poonam_tandel28@yahoo.co.in>wrote: > HI Art > > did u mean that that crash happend becz i didnt shutdown the primary > database > properly. > I thought that the crash happend becz of the way i upgraded the databases. > To upgrade the database i have performed the following steps. > 1) Shutdown the HDR and SDS instances > 2) Shutdown the Primary database > 3)Installed the binaries on all the databases > 4) Started the primary database > 5) Then started the SDS and HDR > > Is this a right way to upgrade the database in MACH11 cluster? > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001517448ad24911ee04734dabb9

If I read your post right, you said you upgraded all your servers to 11.50.FC5, but the portion of your post from the MSGPATH file on the secondary server that had the dr_secapply thread abort shows the version of the engine you were using was 11.50.FC4. That could be the reason the server was having problems. It is never a good idea to mix product versions between HDR primary/secondary pairs. On your secondary, could you have accidentally brought it up with the incorrect version? Jacques Renaut IBM Informix Advanced Support APD Team Your post: hi all i was using IDS 11.5FC4, i have two SDS and one HDR instance.after upgrading all the databases to 11.5FC5 i started the primary database and then i start the application wothout starting the SDS and HDR databases.After some time primary database blocked and stated showing the following status -------------online-Prim (CKPT REQ)---------------- Blocked CKPT Could not shutdown the primary database so thats the reason i killed the processes and shutdown the database and then start the primary database. after that i try to start the HDR database but it crashed and shows following error in the message log. 14:35:40 DR_ERR set to -3 14:35:40 Assert Failed: DR: Log Record Apply Thread Exited Abnormally. Internal Error. A restart of the database server shall be required to correct this problem. 14:35:40 IBM Informix Dynamic Server Version 11.50.FC4 14:35:40 Who: Session(151, informix@, 0, c0000002e3693ae8) Thread(294, dr_secapply, c0000002e364fb00, 3) File: hdr/hdrApply.c Line: 511 14:35:40 Results: Dynamic Server must abort 14:35:40 Action: Reinitialize shared memory 14:35:40 stack trace for pid 24130 written to /work/tmp/af.50e86e4 14:35:40 See Also: /work/tmp/af.50e86e4, shmem.50e86e4.0 14:36:44 hdr/hdrApply.c, line 511, thread 294, proc id 24130, DR: Log Record Apply Thread Exited Abnormally. Internal Error. A restart of the database server shall be required to correct this problem. . 14:36:45 Fatal error in ADM VP at mt.c:13836 14:36:45 Unexpected virtual processor termination, pid = 24130, exit = 0x100 14:36:45 PANIC: Attempting to bring system down what could be the reason of this crash? Was there anything wrong with the way databases upgraded?