Logical logs backups acting weird after upgrade from 12.10.FC13 to 14.10.FC10

IDS 12.10.FC13 to 14.10.FC10 NAME="SLES" VERSION="15-SP5" Our ERP finally released IDS14 for us to upgrade to. I ran the upgrade on our test/dev vm and the upgrade went well. However the logical log backups are just acting weird. We use the alarmprogram (BACKUP_CMD="ontape -a -d") to "/backups" directory. Sometimes it seems to do the backup and other times it seems to just be waiting. Granted not a lot of activity is happened currently on the test/dev so don't expect lots of logs. I'm getting emails from IDS with subject of Logs at 81.5% full or whatever the % is and body of Hello, The logical logs on the onconf.cars OnLine Engine are at 81.5% full. Please take appropriate steps. Thank You I'm assuming IDS is generating these as not sure what else would. We have a mixture of buffered and unbuffered databases. adamski cars: echo "select name[1,20], is_logging, is_buff_log from sysdatabases" | dbaccess sysmaster Database selected. name is_logging is_buff_log sysmaster 1 0 sysutils 1 0 sysuser 1 0 sysadmin 1 0 uni93 1 0 cars_audit 1 0 cars 1 0 uni97 1 1 train 1 1 uni103 1 1 10 row(s) retrieved. Snip-it from onconf ALARMPROGRAM $INFORMIXDIR/etc/alarmprogram.sh ALRM_ALL_EVENTS 0 STORAGE_FULL_ALARM 600,3SYSALARMPROGRAM $INFORMIXDIR/etc/evidence.sh LTAPEDEV /backups LTAPEBLK 512 LTAPESIZE 0 Snip-it of alarmprogram.sh -- make sure top line is #!/bin/kshBACKUPLOGS=Y ALARMADMIN=0 ALARMPAGER=0 ADMINEMAIL= PAGEREMAIL= BACKUP_CMD="ontape -a -d" RM="rm -f" ONSTATCMD="onstat" adamski cars: ll onconf.cars -rw-r--r-- 1 informix informix 90619 Feb 7 12:37 onconf.cars adamski cars: ll alarmprogram.sh -rwxr-xr-x 1 informix informix 20022 Feb 7 13:01 alarmprogram.sh I have made a full instance backup, bounced IDS and even rebooted server and still sometimes I get the logs to backup other times just the emails saying logs getting full. Anyone have a suggested starting point to troubleshoot as I been through all my notes from prior upgrades and nothing I did in the past to get logs backing up have seemed to help this time. John David Adamski Sr. Sysadmin/DBA Graceland University 1 University Place, Lamoni, IA 50140 adamski@graceland.edu 641-784-5267

When you upgrade, if you are using the ALARMPROGRAM to backup logs, you have to remember to either copy the old alarmprogram script or modify the newly installed one since log backups are disabled in the delivered script by default! ------------------------------ Art S. Kagel, President and Principal Consultant ASK Database Management Corp. www.askdbmgt.com ------------------------------

Oops, never mind. I see the changes you made to alarmprogram.sh. Red herring. Art Art S. Kagel, President and Principal Consultant ASK Database Management www.askdbmgt.com Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves.

The BACKUP_CMD you're citing doesn't look to be the default one, so which alarmprogram.sh is this in - v12's or v14's? In other words: is this really the command being executed? Do you see log backups getting created in /backups? And regarding your logical logs, when getting those emails: does "onstat -l" show multiple logs carry U (used) but no B (backed up) flag? I.e. are logs not getting backed up when they should. The ALARMPROGRAM should be called with each logical log filling or, more precisely, with each switch to a new logical log. I'd start by tracing this script, e.g. by putting some "echo $(date) 'did this and that, on line x' >> /tmp/alarmprog.trace". HTH, Andreas ------------------------------ Andreas Legner ------------------------------

Sorry my post formatting was a bit off, since last Outlook update been having problems with what I type and what is sent being totally different format wise. The current alarmprogram.sh I have in place is the v12 version, and was working prior to the upgrade with no problems. So no new stuff from v14. Still not working. I originally tried the v14 one with the BACKUPLOGS and BACLUP_CMD changed to what I need. When it does work, I see a corresponding log in /backups and onstat -l shows the B to that log. When it not working I see one or more logs with U but no B, as expected. Even stranger I am not getting two emails each 15 odd minutes with the % full message. Had to deal with a "the sky is falling" situation and didn't have any time today to look at. Will add debug line in script to see if that points to something. Hopefully I will have a quieter Friday to do some debugging. Thanks for the suggestions. john

Hi, First make sure the alarmprogram is executable by informix and really gets called. For debugging the alarm events, put a line somewhere near the beginning like: echo "`date +%Y%m%d%H%M%S` $1,$2,$3,$4,$5" >> /tmp/alarm_${INFORMIXSERVER}.out You should see each event triggered in the log (informix user should be able to write to the file ...) After the case "$EVENT_SEVERITY" block, put another echo ${BACKUPLOGS} ${LTAPEDEV} ${BACKUP_CMD} >> /tmp/alarm_${INFORMIXSERVER}.out You should get a 23 event (LOGICAL LOG FILE COMPLETED) for each log filling up. Add more output there for investigation, echo the full command call to the alarm_${INFORMIXSERVER}.out file. Redirect the error output of the BACKUP_CMD call to the same output to see what really happens. Are you using onbar or ontape ? This event can be manually enforced in a testing environment with onmode -l (force next log). We have been using almost the same script for this purpose since version 11.x I think. So any script from 12.x should at least do something with the common events (which did not change). Hope this helps.

Hi John. Firstly, you should never use alarmprogram.sh from a previous version. Always reapply your differences to the new version to ensure it's compatible and to have bug fixes. You should make a copy of the standard version, make your changes, then keep a copy of that in case it's overwritten in an upgrade: cd $INFORMIXDIR/etc cp -p alarmprogram.sh alarmprogram.sh.std vi alarmprogram.sh cp -p alarmprogram.sh alarmprogram.sh.new These are my recommended differences: $ diff alarmprogram.sh.std alarmprogram.sh28c28 < BACKUPLOGS=N --- > BACKUPLOGS=Y 30c30 < ALARMADMIN=0 --- > ALARMADMIN=3 32c32 < ADMINEMAIL= --- > ADMINEMAIL=informix@localhost # redirect via ~/.forward 35c35,39 < MAILUTILITY=/usr/bin/mail --- > MAILUTILITY="sendmail -t" # message headers in content > > exec 2>> $INFORMIXDIR/tmp/alarmprogram.$INFORMIXSERVER.log # record any errors > > cd # in case IDS started in non-writable directory which breaks this 56c60 < BACKUP_CMD="onbar -b -l" --- > BACKUP_CMD="ontape -a -d" The "cd" might fix your problem. If not, recording errors should reveal it. You should also always have email alerts configured which are frequently essential. The one you mentioned was definitely not from IDS. ------------------------------ Doug Lawry Oninit Consulting ------------------------------

Also, you can check whether IDS was started in a non-writable directory on Linux with lsof -a -d cwd -p PID where PID is the process ID of the main "oninit" process with PPID 1. ------------------------------ Doug Lawry Oninit Consulting ------------------------------

Thanks for your suggestions. I had problems when we went from 11.x to 12.x and not sure if we are using the 11.x or the 12.x alarmprogram. When I do a onmode -l the logs do get backed up. When I do some sql the logs don't seem to get backup automatically. Hopefully I will have time to later today to debug, this morning not looking promising. :-( John

I am confused as to why it works sometimes. I would start by just putting a statement at the start of alarmprogram to echo something to a file in /tmp to see if it is even being called. Then dump out the environment, as I have seen problems before when the environment was not set properly when informix was started. Are you starting informix differently now, e.g. with a service? ------------------------------ Mike Walker xDB Systems, Inc www.xdbsystems.com ------------------------------

I was able to do some debugging and testing. I ran a sql to update a large number of records to make sure I would get logs to roll to next. And the old log got backed up and the alarmprogram show correct information from the debug. Not sure what make the alarmprogram to act more like it should, as all I did was add a few debug lines, saved it and then ran the sql and worked as should. Undoing the debug lines seem to not effect it and still seems to be working correctly. It looks like what happening now, is if the current active log is over some a percentage of full I start getting emails from database that the log is getting full. I don't exactly know what producing the email as I don't remember setting up any monitoring or warning. So now a dungeon search to figure out what is producing the emails. As the warning is nice but getting one every 15 minutes on a quiet system annoying. Especially if for last few hours been at 80.7%. Thanks again for suggestions on the alarmprogram. john

Hi, cd does not fix all issues. If the the current working directory (cwd) for the server no longer exists (e.g. when started under automation) then the server will not be able to fork a process to run the alarmprogram! I cannot remember if something appears in the online.log for this condition. Regards, David. ------------------------------ David Williams ------------------------------