log_full.sh getting Aborted by client.

While I was away our developers decided to do some cleanup of old records in our audit database. As I understand this there where deleting millions of old records when a lock halted their sql . once they cleared the other person who had the lock the logs haven't automatically rolled using the log_full.sh script. They been keeping the system running while I was gone by doing a periodic ontape -a Since this happened two days ago a full backup has happened twice (ontape -s -L 0) and the server was rebooted. I'm not sure how to find the cause or fix this. Any ideas of what wrong and how to fix it????? John David Adamski, Sr. Network Specialist Graceland University, 1 Univeristy Place, Lamoni, IA 50140 adamski@graceland.edu Not sure if these logs might help bar_dbug.log 2017-03-18 01:37:59 27887 27875 dsm_obj.c:658 dsmMakeObjectCursorList: No more records from object table. 2017-03-18 01:38:00 27887 27875 dsm_obj.c:658 dsmMakeObjectCursorList: No more records from object table. 2017-03-18 01:38:01 27887 27875 dsm_obj.c:658 dsmMakeObjectCursorList: No more records from object table. 2017-03-18 01:38:01 27887 27875 dsm_obj.c:658 dsmMakeObjectCursorList: No more records from object table. 2017-03-18 01:38:02 27887 27875 dsm_obj.c:658 dsmMakeObjectCursorList: No more records from object table. 2017-03-18 01:38:02 27887 27875 dsm_obj.c:658 dsmMakeObjectCursorList: No more records from object table. 2017-03-18 01:38:03 27887 27875 dsm_obj.c:658 dsmMakeObjectCursorList: No more records from object table. 2017-03-18 04:10:58 2248 2245 smdbopen.c:117 nsmOpenDatabase: Count: opened = 8, missing = 0. 2017-03-18 04:10:58 2248 2245 smobject.c:1580 Error -88001: C-ISAM function 'dsmFetchDetail' retu rned error (17). 2017-03-18 04:10:58 2248 2245 smtranid.c:176 Error -88001: C-ISAM function 'isread' returned erro r (110). 2017-03-18 04:10:58 2248 2245 dsm_obj.c:658 dsmMakeObjectCursorList: No more records from object t able. 2017-03-18 04:10:58 2248 2245 xbsa_so.c:1406 psmQueryObject: nsmOpenObjectCursor() returned 17 2017-03-18 04:10:58 2248 2245 smtranid.c:176 Error -88001: C-ISAM function 'isread' returned erro r (110). 2017-03-18 04:12:10 3141 3139 smdbopen.c:117 nsmOpenDatabase: Count: opened = 8, missing = 0. 2017-03-18 04:12:11 3141 3139 smobject.c:1580 Error -88001: C-ISAM function 'dsmFetchDetail' retu rned error (17). 2017-03-18 04:12:11 3141 3139 smtranid.c:176 Error -88001: C-ISAM function 'isread' returned erro r (110). 2017-03-18 04:12:11 3141 3139 dsm_obj.c:658 dsmMakeObjectCursorList: No more records from object t able. informix cars: cat cars.log Fri Mar 17 23:57:00 CDT 2017 Sat Mar 18 00:10:01 2017 00:10:01 Logical Log 402881 Complete, timestamp: 0x105918c7. 00:10:02 Logical Log 402861 - Backup Started 00:10:08 Logical Log 402861 - Backup Aborted Aborted by client. 00:10:14 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402881 Complete, timestamp: 0x105918c7." "Logical Log 402881 Complete, timestamp: 0x105918c7." "" 23001 00:10:30 Logical Log 402882 Complete, timestamp: 0x105a30b1. 00:10:31 Logical Log 402861 - Backup Started 00:10:34 Logical Log 402861 - Backup Aborted Aborted by client. 00:10:40 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402882 Complete, timestamp: 0x105a30b1." "Logical Log 402882 Complete, timestamp: 0x105a30b1." "" 23001 00:11:02 Logical Log 402883 Complete, timestamp: 0x105bb8af. 00:11:02 Logical Log 402861 - Backup Started 00:11:07 Logical Log 402861 - Backup Aborted Aborted by client. 00:11:13 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402883 Complete, timestamp: 0x105bb8af." "Logical Log 402883 Complete, timestamp: 0x105bb8af." "" 23001 00:20:45 Logical Log 402884 Complete, timestamp: 0x10648e20. 00:20:46 Logical Log 402861 - Backup Started 00:20:50 Logical Log 402861 - Backup Aborted Aborted by client. 00:20:56 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402884 Complete, timestamp: 0x10648e20." "Logical Log 402884 Complete, timestamp: 0x10648e20." "" 23001 00:34:21 Logical Log 402885 Complete, timestamp: 0x1069922a. 00:34:22 Logical Log 402861 - Backup Started 00:34:26 Logical Log 402861 - Backup Aborted Aborted by client. 00:34:32 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402885 Complete, timestamp: 0x1069922a." "Logical Log 402885 Complete, timestamp: 0x1069922a." "" 23001 00:36:33 Logical Log 402886 Complete, timestamp: 0x106bca74. 00:36:33 Logical Log 402887 Complete, timestamp: 0x106bca74. 00:36:34 Logical Log 402861 - Backup Started 00:36:34 Logical Log 402888 Complete, timestamp: 0x106c6983. 00:36:37 Logical Log 402861 - Backup Aborted Aborted by client. 00:36:41 Process exited with return code 152: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402887 Complete, timestamp: 0x106bca74." "Logical Log 402887 Complete, timestamp: 0x106bca74." "" 23001 00:36:42 Process exited with return code 152: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402888 Complete, timestamp: 0x106c6983." "Logical Log 402888 Complete, timestamp: 0x106c6983." "" 23001 00:36:43 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402886 Complete, timestamp: 0x106bca74." "Logical Log 402886 Complete, timestamp: 0x106bca74." "" 23001 00:49:13 Logical Log 402889 Complete, timestamp: 0x1071d576. 00:49:14 Logical Log 402861 - Backup Started 00:49:16 Logical Log 402861 - Backup Aborted Aborted by client. 00:49:22 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402889 Complete, timestamp: 0x1071d576." "Logical Log 402889 Complete, timestamp: 0x1071d576." "" 23001 01:00:04 Logical Log 402890 Complete, timestamp: 0x107754cc. 01:00:06 Logical Log 402861 - Backup Started 01:00:11 Logical Log 402861 - Backup Aborted Aborted by client. 01:00:15 Logical Log 402891 Complete, timestamp: 0x107869db. 01:00:15 Logical Log 402861 - Backup Started 01:00:16 Logical Log 402861 - Backup Aborted Aborted by client. 01:00:17 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402890 Complete, timestamp: 0x107754cc." "Logical Log 402890 Complete, timestamp: 0x107754cc." "" 23001 01:00:20 Logical Log 402892 Complete, timestamp: 0x107939d5. 01:00:20 Logical Log 402861 - Backup Started 01:00:22 Process exited with return code 19: /bin/sh /bin/sh -c /opt/informix/etc/log_full.sh 2 23 "Logical Log 402891 Complete, timestamp: 0x107869db." "Logical Log 402891 Complete, timestamp: 0x107869db." @@DQ@

Hi Adam, so you're using PSM, right? So onbar for log backups? How would this fit = the two ontape full backups? Can we have Informix version and platform info, please? And the bar=5Fact.log ... Andreas

We are on an HPUX 11.31 server running Informix 12.10.FC6 - sorry for forgetting that just got back from 19 hour car rid from funeral. Brain wasn't wake. We use the log_full script (onbar?) for logs and use tape to back up the database (ontape -s -L 0). I set this up 10+ years ago when I was 50% DBA 50% systems admin, now I'm only 10% dba and 90% server room manager. So rusty on my Informix. I now just make sure the backups work and bailout the developers when they do something the database doesn't like. Informix just run like the energizer bunny so my job priorities where changed. The only onbar log I see is the bar_dbug.log that I took a snippet from in my original email. I haven't talked with the developers in more detail since the panic call Thursday, will later today when they come in. maybe some new information might come of this. John -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Andreas Legner Sent: Monday, March 20, 2017 3:13 AM To: ids@iiug.org Subject: Re: log_full.sh getting Aborted by client. [38784] Hi Adam, so you're using PSM, right? So onbar for log backups? How would this fit = the two ontape full backups? Can we have Informix version and platform info, please? And the bar=5Fact.log ... Andreas ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Also to clarify, we have had two database backups (ontape -s -L 0 ) and we also rebooted the server to see if that would get things working again. None of these seem to have changed the fact the onbar process is aborting. John -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of John Adamski (Work Account) Sent: Monday, March 20, 2017 7:46 AM To: ids@iiug.org Subject: RE: log_full.sh getting Aborted by client. [38785] [This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing] We are on an HPUX 11.31 server running Informix 12.10.FC6 - sorry for forgetting that just got back from 19 hour car rid from funeral. Brain wasn't wake. We use the log_full script (onbar?) for logs and use tape to back up the database (ontape -s -L 0). I set this up 10+ years ago when I was 50% DBA 50% systems admin, now I'm only 10% dba and 90% server room manager. So rusty on my Informix. I now just make sure the backups work and bailout the developers when they do something the database doesn't like. Informix just run like the energizer bunny so my job priorities where changed. The only onbar log I see is the bar_dbug.log that I took a snippet from in my original email. I haven't talked with the developers in more detail since the panic call Thursday, will later today when they come in. maybe some new information might come of this. John -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Andreas Legner Sent: Monday, March 20, 2017 3:13 AM To: ids@iiug.org Subject: Re: log_full.sh getting Aborted by client. [38784] Hi Adam, so you're using PSM, right? So onbar for log backups? How would this fit = the two ontape full backups? Can we have Informix version and platform info, please? And the bar=5Fact.log ... Andreas ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum. ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Looks like the location got turn to read-only - have no idea how or why this happened. informix cars: onpsm -D list Informix Primary Storage Manager Device List Type Prio Block/Size (MB) Pool Name Device Name FILE HIGHEST --/-- DBSPOOL /opt/informix/backups/infx/spaces FILE READ-ONLY --/-- LOGPOOL /opt/informix/tlog/hold informix cars: onpsm -D update /opt/informix/tlog/hold -g LOGPOOL -p HIGHEST informix cars: onpsm -D list Informix Primary Storage Manager Device List Type Prio Block/Size (MB) Pool Name Device Name FILE HIGHEST --/-- DBSPOOL /opt/informix/backups/infx/spaces FILE HIGHEST --/-- LOGPOOL /opt/informix/tlog/hold Log seem to be backing up now. John -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of John Adamski (Work Account) Sent: Monday, March 20, 2017 7:46 AM To: ids@iiug.org Subject: RE: log_full.sh getting Aborted by client. [38785] [This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing] We are on an HPUX 11.31 server running Informix 12.10.FC6 - sorry for forgetting that just got back from 19 hour car rid from funeral. Brain wasn't wake. We use the log_full script (onbar?) for logs and use tape to back up the database (ontape -s -L 0). I set this up 10+ years ago when I was 50% DBA 50% systems admin, now I'm only 10% dba and 90% server room manager. So rusty on my Informix. I now just make sure the backups work and bailout the developers when they do something the database doesn't like. Informix just run like the energizer bunny so my job priorities where changed. The only onbar log I see is the bar_dbug.log that I took a snippet from in my original email. I haven't talked with the developers in more detail since the panic call Thursday, will later today when they come in. maybe some new information might come of this. John -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Andreas Legner Sent: Monday, March 20, 2017 3:13 AM To: ids@iiug.org Subject: Re: log_full.sh getting Aborted by client. [38784] Hi Adam, so you're using PSM, right? So onbar for log backups? How would this fit = the two ontape full backups? Can we have Informix version and platform info, please? And the bar=5Fact.log ... Andreas ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum. ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Hi John, it's not really a recommended / supported combination, using ontape for=20 dbspace backups and onbar for log backups - it might work during a=20 restore if you split the restore into a physical and a logical restore,=20 but no guarantees. Now that you're using onbar for the logs, and Informix PSM obviously as=20 your storage manager (which might be an unconscious decision as PSM just=20 work out of the box, not asking for configuration, if you haven't=20 configured BAR=5FBSALIB=5FPATH), where are these backups going to, who's=20 taking care of disk space management? -> check $INFORMIXDIR/backup directory if you haven't explicitly=20 configured differently. Try 'onbar -m' to learn about bar=5Fact.log file (BAR=5FACT=5FLOG onconfi= g=20 param) and see last 20 lines. -> might want to post a few more than these 20 lines. Just seeing you've already resolved the problem, still sending this out,=20 mainly to alert you to my first point above. Cheers, Andreas From: "John Adamski (Work Account)" <adamski@graceland.edu> To: ids@iiug.org Date: 20.03.2017 14:33 Subject: RE: log=5Ffull.sh getting Aborted by client. [38786] Sent by: ids-bounces@iiug.org Also to clarify, we have had two database backups (ontape -s -L 0 ) and we = also rebooted the server to see if that would get things working again.=20 None=20 of these seem to have changed the fact the onbar process is aborting.=20 John=20 -----Original Message-----=20 From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of John = Adamski (Work Account)=20 Sent: Monday, March 20, 2017 7:46 AM=20 To: ids@iiug.org=20 Subject: RE: log=5Ffull.sh getting Aborted by client. [38785]=20 [This sender failed our fraud detection checks and may not be who they=20 appear=20 to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]=20 We are on an HPUX 11.31 server running Informix 12.10.FC6 - sorry for=20 forgetting that just got back from 19 hour car rid from funeral. Brain=20 wasn't=20 wake.=20 We use the log=5Ffull script (onbar?) for logs and use tape to back up the = database (ontape -s -L 0).=20 I set this up 10+ years ago when I was 50% DBA 50% systems admin, now I'm=20 only=20 10% dba and 90% server room manager. So rusty on my Informix. I now just=20 make=20 sure the backups work and bailout the developers when they do something=20 the=20 database doesn't like. Informix just run like the energizer bunny so my=20 job=20 priorities where changed.=20 The only onbar log I see is the bar=5Fdbug.log that I took a snippet from i= n=20 my=20 original email.=20 I haven't talked with the developers in more detail since the panic call=20 Thursday, will later today when they come in. maybe some new information=20 might=20 come of this.=20 John=20 -----Original Message-----=20 From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of=20 Andreas=20 Legner=20 Sent: Monday, March 20, 2017 3:13 AM=20 To: ids@iiug.org=20 Subject: Re: log=5Ffull.sh getting Aborted by client. [38784]=20 Hi Adam,=20 so you're using PSM, right? So onbar for log backups? How would this fit = =3D=20 the two ontape full backups?=20 Can we have Informix version and platform info, please?=20 And the bar=3D5Fact.log ...=20 Andreas=20 ***************************************************************************= ****=20 Forum Note: Use "Reply" to post a response in the discussion forum.=20 ***************************************************************************= ****=20 Forum Note: Use "Reply" to post a response in the discussion forum.=20 ***************************************************************************= ****=20 Forum Note: Use "Reply" to post a response in the discussion forum.=20