Problem with 9.3 Onbar backup
Posted in 2006
Mark Cory reported ON-Bar level-0 backups of a large smart-blob space (sb_a20) failing under Legato Networker on IDS 9.3, with "ASSERT: file bar_xbsa.ec barBeginTxn() line 912", exit code 159, a 2 GB core file, and the engine log saying the backup was "aborted by client". Martin Fuerderer suggested it looked like a memory bug likely fixed in later versions, and advised a debugger stack trace, higher BAR_DEBUG and checking the XBSA side. Raising BAR_DEBUG to 7 yielded nothing new, the backup vendor only confirmed status 159, and with 9.3 out of support no fix was found; Mark fell back to using ontape to disk.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Topics: Backup & Restore, Storage & Space Management, Server Administration
Onbar and legato networker backups have recently been failing with the following messages in the bar_act log (pids and timestamps removed): Begin level 0 backup sb_a20. ASSERT: file bar_xbsa.ecbarBeginTxn() line 912 - contact product support See also: /bml/logs/informix/core.23797 Process completed. The ON-Bar process 23797 exited with a problem (exit code 159 (0x9f), signal 0). The engine log simply states that the backup was "aborted by client." Of course, the core file is binary and I have no idea how to try to decipher the 2 GB. file. Blobspace sb_a20 consists of eight chunks of two GB each, and is one of multiple fragments of a round-robin partitioned table used for a binary file storage repository. I'm in the process of following-up with the group supporting the legato networker backup server to investigate the status of the xbsa and am still awaiting their feedback. To make this truly interesting to resolve, management chose to discontinue maintenance and support of Informix a year or two ago, so I have no product support available. Other than making management admit their mistake and getting support again from IBM, do any of you have any suggestions? Thank you for your attention, consideration, and enlightened feedback! Mark
Hi,
9.3 is quite old now and a lot of problems have been fixed
in newer version. Management decision to discontinue the
support most probably happened because 9.3 is out of regular
support. And to still get support for such a version is quite
difficult and expensive ...
If I understand you correctly, you are saying that the core file from
onbar is 2 GB? This is very large for an onbar process and I would
suspect some memory problem that surely has been fixed in newer
versions. You can use a debugger to retrieve a function call stack
from the core file. Though I assume that this will not give you a hint
for resolving the problem without upgrading. :-(
Regards,
Martin
--
Martin Fuerderer
IBM Informix Development Munich, Germany
Information Management
Informix URLs list: http://home.arcor.de/mfu1/informix/urls.html
Informix in the news:
http://www.crn.com/sections/software/software.jhtml?articleId=193004835
IBM Information On Demand Global Conference
October 15-20, 2006, Anaheim, California
see http://www.ibm.com/events/informationondemand
ids-bounces@iiug.org wrote on 29.09.2006 18:18:46:
>
> Onbar and legato networker backups have recently been failing with the
> following messages in the bar_act log (pids and timestamps removed):
>
> Begin level 0 backup sb_a20.
> ASSERT: file bar_xbsa.ecbarBeginTxn() line 912 - contact product support
> See also: /bml/logs/informix/core.23797
> Process completed.
> The ON-Bar process 23797 exited with a problem (exit code 159 (0x9f),
signal
> 0).
>
> The engine log simply states that the backup was "aborted by client." Of
> course, the core file is binary and I have no idea how to try to
decipher the
> 2 GB. file. Blobspace sb_a20 consists of eight chunks of two GB each,
and is
> one of multiple fragments of a round-robin partitioned table used for a
binary
> file storage repository. I'm in the process of following-up with the
group
> supporting the legato networker backup server to investigate the status
of the
> xbsa and am still awaiting their feedback.
>
> To make this truly interesting to resolve, management chose to
discontinue
> maintenance and support of Informix a year or two ago, so I have no
product
> support available. Other than making management admit their mistake and
> getting support again from IBM, do any of you have any suggestions?
>
> Thank you for your attention, consideration, and enlightened feedback!
>
> Mark
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
Thank you for your feedback, Martin! I managed to get a low level debug (BAR_DEBUG=3) executed this weekend, but it also ended in failure. The only significant entries I could find in the bar_act log were: 2006-09-30 20:12:38 bar_debug: enter 2006-09-30 20:12:38 bar_debug: Called from file bar_xbsa.ecbarBeginTxn() line 912 2006-09-30 20:13:04 bar_debug: return 2006-09-30 20:13:05 bar_upd_bootfile: input: bar_objdesc obj_id 26 obj_name 'sb_a20' obj_type 'ND' act_id 85826 act_type 1 act_status 159 act_start '2006-09-30 18:48:25' act_end '2006-09-30 20:13:04' ins_time 0 rsam_time -2147483648 level 0 copyid hi:lo 1159667306:1159667307 req_act_id 0 logstream 0 est_pages hi:lo 0:3813279 first_log 0 chpt_log 0 last_log 0 partial_flag 0 do_query 0 ins_sm_id 0 ins_sm_name '' ins_verify 0 ins_verify_date '' restore order 0:0 objInfo '' retry 0 in_catalog 1 in_bootfile 0 child_pid 0 child_state 2 bkup_host '' I don't know if it would be of any benefit to increase BAR_DEBUG to seven and try again. I wasn't able to find much help from looking around on the web over the weekend, other than something that referred to the original exit code (159 / 0x9f) as an unspecified bug. I'm still persuing this as a possible issue with the xbsa software, awaiting feedback from the outsource vendor that supports the network backup server. As before, any and all feedback is greatly appreciated! Thank you all for your attention and insight! Mark
Hi, this really does not look significant to the error/problem. Increasing the debug level will give more detailed info on function calls and their parameters. So it may be possible to get "closer" to the failure. Though I can't guarantee. Also looking on the XBSA side definitely is a good idea. Regards, Martin -- Martin Fuerderer IBM Informix Development Munich, Germany Information Management Informix URLs list: http://home.arcor.de/mfu1/informix/urls.html Informix in the news: http://www.crn.com/sections/software/software.jhtml?articleId=193004835 IBM Information On Demand Global Conference October 15-20, 2006, Anaheim, California see http://www.ibm.com/events/informationondemand ids-bounces@iiug.org wrote on 02.10.2006 17:55:40: > > Thank you for your feedback, Martin! I managed to get a low level debug > (BAR_DEBUG=3) executed this weekend, but it also ended in failure. The only > significant entries I could find in the bar_act log were: > > 2006-09-30 20:12:38 bar_debug: enter > 2006-09-30 20:12:38 bar_debug: Called from file bar_xbsa.ecbarBeginTxn() line > 912 > 2006-09-30 20:13:04 bar_debug: return > 2006-09-30 20:13:05 bar_upd_bootfile: input: bar_objdesc > obj_id 26 obj_name 'sb_a20' obj_type 'ND' act_id 85826 act_type 1 act_status > 159 > act_start '2006-09-30 18:48:25' act_end '2006-09-30 20:13:04' > ins_time 0 rsam_time -2147483648 level 0 copyid hi:lo 1159667306:1159667307 > req_act_id 0 > logstream 0 est_pages hi:lo 0:3813279 first_log 0 chpt_log 0 last_log 0 > partial_flag 0 do_query 0 ins_sm_id 0 ins_sm_name '' > ins_verify 0 ins_verify_date '' restore order 0:0 > objInfo '' > retry 0 in_catalog 1 in_bootfile 0 child_pid 0 child_state 2 > bkup_host '' > > I don't know if it would be of any benefit to increase BAR_DEBUG to seven and > try again. I wasn't able to find much help from looking around on the web over > the weekend, other than something that referred to the original exit code (159 > / 0x9f) as an unspecified bug. I'm still persuing this as a possible issue > with the xbsa software, awaiting feedback from the outsource vendor that > supports the network backup server. > > As before, any and all feedback is greatly appreciated! Thank you all for your > attention and insight! > > Mark > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >
Mark, Martin,
can you recap the problem for me...
i have quite a bit of experience debugging IDS onbar with Symantec (Veritas)
Netbackup.
your bar-debug info looked interesting, what's in the logs for your backup
utility? (i.e. sometimes IDS doesn't tell me as much as Netbackup does.
frequently it takes debugging both IDS and netbackup to understand the
problem/solution).
thanks,
Norma Jean
________________________________
From: ids-bounces@iiug.org on behalf of Martin Fuerderer
Sent: Wed 10/4/2006 8:58 AM
To: ids@iiug.org
Subject: Re: Problem with 9.3 Onbar backup [7575]
Hi,
this really does not look significant to the error/problem.
Increasing the debug level will give more detailed info
on function calls and their parameters. So it may be
possible to get "closer" to the failure. Though I can't
guarantee.
Also looking on the XBSA side definitely is a good idea.
Regards,
Martin
--
Martin Fuerderer
IBM Informix Development Munich, Germany
Information Management
Informix URLs list: http://home.arcor.de/mfu1/informix/urls.html
Informix in the news:
http://www.crn.com/sections/software/software.jhtml?articleId=193004835
IBM Information On Demand Global Conference
October 15-20, 2006, Anaheim, California
see http://www.ibm.com/events/informationondemand
ids-bounces@iiug.org wrote on 02.10.2006 17:55:40:
>
> Thank you for your feedback, Martin! I managed to get a low level debug
> (BAR_DEBUG=3) executed this weekend, but it also ended in failure. The
only
> significant entries I could find in the bar_act log were:
>
> 2006-09-30 20:12:38 bar_debug: enter
> 2006-09-30 20:12:38 bar_debug: Called from file bar_xbsa.ecbarBeginTxn()
line
> 912
> 2006-09-30 20:13:04 bar_debug: return
> 2006-09-30 20:13:05 bar_upd_bootfile: input: bar_objdesc
> obj_id 26 obj_name 'sb_a20' obj_type 'ND' act_id 85826 act_type 1
act_status
> 159
> act_start '2006-09-30 18:48:25' act_end '2006-09-30 20:13:04'
> ins_time 0 rsam_time -2147483648 level 0 copyid hi:lo
1159667306:1159667307
> req_act_id 0
> logstream 0 est_pages hi:lo 0:3813279 first_log 0 chpt_log 0 last_log 0
> partial_flag 0 do_query 0 ins_sm_id 0 ins_sm_name ''
> ins_verify 0 ins_verify_date '' restore order 0:0
> objInfo ''
> retry 0 in_catalog 1 in_bootfile 0 child_pid 0 child_state 2
> bkup_host ''
>
> I don't know if it would be of any benefit to increase BAR_DEBUG to
seven and
> try again. I wasn't able to find much help from looking around on the
web over
> the weekend, other than something that referred to the original exit
code (159
> / 0x9f) as an unspecified bug. I'm still persuing this as a possible
issue
> with the xbsa software, awaiting feedback from the outsource vendor that
> supports the network backup server.
>
> As before, any and all feedback is greatly appreciated! Thank you all
for your
> attention and insight!
>
> Mark
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
*******************************************************************************
Forum Note: Use "Reply" to post a response in the discussion forum.
============================================================
The information contained in this message may be privileged
and confidential and protected from disclosure. If the reader
of this message is not the intended recipient, or an employee
or agent responsible for delivering this message to the
intended recipient, you are hereby notified that any reproduction,
dissemination or distribution of this communication is strictly
prohibited. If you have received this communication in error,
please notify us immediately by replying to the message and
deleting it from your computer. Thank you. Tellabs
============================================================
Sorry for the delayed reply; I kept getting told that I wasn't logged in and
not allowed to post a response. I attempted another backup with BAR_DEBUG set
to seven, but the output was no different than it was when BAR_DEBUG was set
to three (aside from time stamps, naturally). The only thing I can get out of
the outsource vendor that runs the backup server is a screenshot of a window
titled "Group Detail" that shows two core dumps and "onbar returned status of
159", which we already knew from the onbar log. Considering that 9.3 hasn't
been supported by IBM for over a year, management does not expect to get any
affordable support options. (Of course, that's their own fault for cancelling
maintenance nearly two years ago.) I'm currently working on a backup solution
using ontape directly to disk.