Informix 11.5 and Server 2012R2 – Database stuck in Quiescent Mode

I am experiencing an issue with an archive Informix database that is stuck in quiescent mode. When trying to start the service in windows services I get the following error: "The informix IDS – service on the Local Computer started and stopped. Some services stop automatically if they are not in use by other services or programs". When I run the 'onstat –' command in the Informix terminal I get the following 'IBM Informix Dynamic Server Version 11.50.FC9 – Fast Recovery <CKPT REQ> -- UP Blocked CKPT' . It seems that the DB may have been interrupted during a checkpoint and is preventing itself from starting. I was wondering if the community would happen to know how to approach this issue as I am not very knowledgeable in Informix or what steps need to be done to correct and put the database into standard mode. I appreciate any help. Thank you ------------------------------ Eric Lovato ------------------------------ #Informix

This could be a number of things, some of them not good. But check to make sure that your logical logs are all backed up, as that can cause the engine to block, although I remember this having a different message. Run "onstat -l" (that's an "L") and make sure that the flags show U-B and not U-- Also check the Informix log for more information. You should be able to see the end of this with onstat -m ------------------------------ Mike Walker xDB Systems, Inc www.xdbsystems.com ------------------------------

Good Afternoon Mike, I was able to run the command on the terminal but i am noticing U___C_L flag listed on the 13th partition and the rest of the partitions being listed as U-- .Thank you for your quick reply ------------------------------ Eric Lovato ------------------------------

Okay, this is good. You need your logical logs to be backed up. The logical logs are reused but informix cannot use the next logical log because it has not been archived, and if it were overwritten you may not be able to recover if you had to do a restore...so Informix will hang until the log has been backed up. I don't know how you normally back up your logs, so you will need to find out why they are not being actively backed up. Check the informix log for errors. If you use ontape, check LTAPEDEV in your informix configuration file - perhaps you are writing to a filesystem that has filled, or you are writing to a tape drive that has no tape in it. Maybe the ontape command needs to be run with -a option, but can only guess at this point. ------------------------------ Mike Walker xDB Systems, Inc www.xdbsystems.com ------------------------------

That is normal. The "L" indicates the logical log that contains the last checkpoint entry and the "C" indicates the logical log that is the one currently being used. The "C"urrent log is normally not backed up. If you run ontape -a without the -d it would prompt you asking whether you want to backup the "C"urrent log. With the -d "N"o is assumed and ontape will not prompt. Art ------------------------------ Art S. Kagel, President and Principal Consultant ASK Database Management Corp. www.askdbmgt.com ------------------------------

There is a script for backing up the database as follows that uses the ontape -s -L 0 command : REM change for your install path set INFORMIXDIR=D:\\Progra~1\\InformixREM change if your sever instance name is different that the default set INFORMIXSERVER=worx_server set ONCONFIG=ONCONFIG.worx_server set PATH=D:\\Progra~1\\Informix\\bin;%PATH%set CLASSPATH=%INFORMIXDIR%\\extend\\krakatoa\\krakatoa.jar;%INFORMIXDIR%\\extend\\krakatoa\\jdbc.jar;%CLASSPATH% set DBTEMP=d:\\PROGRA~1\\informix\\infxtmp set CLIENT_LOCALE=EN_US.CP1252 set DB_LOCALE=EN_US.8859-1mode con codepage select=1252 REM set the path for the logfile set LOGFILE=d:\\PROGRA~1\\informix\\level0_backup.log cls date /T >>%LOGFILE% echo "starting level 0 backup" >>%LOGFILE% ontape -s -L 0 >>%LOGFILE%echo "Completed level 0 backup" >>%LOGFILE% date /T >>%LOGFILE% I took a look at the level0_backup.log file and the last good backup was done on October 16 however the folder that would contain this backup E:\\backups has been cleared presumably by ontape running and not completing. LTAPEDEV variable is set to E:\\backups and should be the correct path on a separate windows drive. When I try to run the ontape -s -L 0 command outside of my script I get 'Server is in incompatible state or user authentication failed'. Should i try to execute the ontape command with -a option? ------------------------------ Eric Lovato ------------------------------

Eric: The ontape -a option only backs up the logical logs. You are saying that your full level 0 archives seem to have been failing. To double check that, first ask the Informix engine: onstat -g arcHere's output from my server: $ onstat -g arc IBM Informix Dynamic Server Version 14.10.FC9DE -- On-Line -- Up 2 days 04:40:47 -- 2339972 Kbytes 2022-11-30 16:57:10num DBSpace Q Size Q Len Buffer partnum size scanner Dbspaces - Archive Status name number level date log log-position rootdbs 1 0 11/07/2022.18:12:56 3243 0x2d018 llogspace 2 0 11/07/2022.18:12:56 3243 0x2d018 plogspace 3 0 11/07/2022.18:12:56 3243 0x2d018 datadbs_1 4 0 11/07/2022.18:12:56 3243 0x2d018 indexdbs_1 5 0 11/07/2022.18:12:56 3243 0x2d018 sbspace 7 0 11/07/2022.18:12:56 3243 0x2d018 cdrspace 8 0 11/07/2022.18:12:56 3243 0x2d018 indexdbs 9 0 11/07/2022.18:12:56 3243 0x2d018 datadbs_2 10 0 11/07/2022.18:12:56 3243 0x2d018 So that is saying that the server, and all of its dbspaces. were backed up last on November 7. I don't automatically back up this server as it is used only to test software and Informix. It's only backed up manually about once a month or so. You can see where the backup file(s) are from the last two archives here: $ egrep '^TAPEDEV' $INFORMIXDIR/etc/$ONCONFIG TAPEDEV /home/art/GoogleDrive/Informix/Archives $ dir /home/art/GoogleDrive/Informix/Archivestotal 6401072 drwxrwxrwx 2 art art 4096 Nov 7 18:13 ./ drwxrwxr-x 4 informix informix 4096 Sep 28 16:27 ../ -rwxrwxrwx 1 art art 3277094912 Oct 12 18:32 Elbereth_1_20221012_183241_L0.FC8* -rwxrwxrwx 1 art art 3277586432 Nov 7 18:13 Elbereth_1_L0* Also you can look in the message log (parameter MSG) for the last time an archive started and whether it completed successfully or failed and why: 18:12:56 Level 0 Archive started on rootdbs, llogspace, plogspace, datadbs_1, indexdbs_1, cdrspace, indexdbs, datadbs_2, sbspace ... 18:13:15 Archive on rootdbs, llogspace, plogspace, datadbs_1, indexdbs_1, cdrspace, indexdbs, datadbs_2, sbspace Completed. ------------------------------Art S. Kagel, President and Principal Consultant ASK Database Management Corp. www.askdbmgt.com ------------------------------

Yes, try running ontape -a Hopefully this will just back up the logical logs and all will be well. However, because the engine is not fully online, I am not sure that it will work. But try it anyway... ------------------------------ Mike Walker xDB Systems, Inc www.xdbsystems.com ------------------------------

Art: I ran the onstat -g arc command and received the following: I took a look to verfiy the ONCONFIG file and confirmed that the TAPEDEV variable was set to e:\\backups I then went to verify that there we no files in the backup directory. I took a look at the MSG parameter and reviewed the log file but did not see any information on the backup. Then I remembered we are redirecting the output to a different %LOGFILE% in the script above and have attached the following output ------------------------------ Eric Lovato ------------------------------

Eric: And there are no files in E:\\backups? That is strange. The last recorded backup according to onstat -g arc was on the 14th but your log files shows completed archives on the 15th and 16th (though the one on the 16th is strange because there are not percent done messages and that's unusual. At this point I would open a PMR with IBM and let them sort it out. Art ------------------------------ Art S. Kagel, President and Principal Consultant ASK Database Management Corp. www.askdbmgt.com ------------------------------

Check your informix log to see if it has a message for late on 10/16 or early on 10/17 about a failed or aborted backup. Maybe it will tell you more information. I wonder if you have another script that removes the old backup, or otherwise the failed backup on 10/17 zapped it. Maybe there is a script that copies the backup somewhere else for safe keeping? The bad thing is that it looks like you have no archive available to restore from should you need to...unless it gets copied somewhere else. ------------------------------ Mike Walker xDB Systems, Inc www.xdbsystems.com ------------------------------

Thank you, Art and Mike ... I am reaching out to IBM to better understand the PMR process and will be meeting with them today (via webex). I appreciate your time in helping me troubleshoot and I am hoping that it can be resolved ------------------------------ Eric Lovato ------------------------------