Logical log backup SLOW to NetBackup
We are in the process of changing our enterprise backup solution from Tivoli Storage manager (TSM) to Symantec NetBackup (NBU). Our Informix logical logs average about 32mb in size and are able to backup to TSM in 5 seconds or less. The Informix databases that have been converted to NBU take MUCH longer to backup logical logs (1 3 minutes/logical log) and cant keep up. I would prefer not to change the size or number of the logical logs to work around this problem, but it might be inevitable with NBU. The actual read/write of a logical log backup to NBU only takes seconds. It appears that most of the time to backup a logical log in NBU occurs during the phase in which the backup process requests a resource and then is granted a resource for the backup of each logical log (see example below). Does anyone have any experience with making a persistent connection for Informix logical logs to NBU so that each logical log backup does NOT need to establish a new connection to NBU each time for a backup? Also, does anyone know how Informix spawns processes for logical log backups? Specifically, as transactions progress through various logical logs, does Informix spawn separate backup processes for each logical log or is only one logical log backed up at a time? 05/02/2011 05:43:13 - requesting resource NBU_DISK_Archive_SUG 05/02/2011 05:43:13 - requesting resource nbu_master.acmeco.com.NBU_CLIENT.MAXJOBS.node.acmeco.com 05/02/2011 05:43:13 - requesting resource nbu_master.acmeco.com.NBU_POLICY.MAXJOBS.Informix_Logs_RDC_Prod 05/02/2011 05:46:47 - granted resource nbu_master.acmeco.com.NBU_CLIENT.MAXJOBS.node.acmeco.com 05/02/2011 05:46:47 - granted resource nbu_master.acmeco.com.NBU_POLICY.MAXJOBS.Informix_Logs_RDC_Prod 05/02/2011 05:46:47 - granted resource MediaID=@aaaaB;DiskVolume=/nbu_archive1;DiskPool=node_AD_Archive;Path=/nbu_archi ve1;StorageServer=node.acmeco.com;MediaServer=node.acmeco.com 05/02/2011 05:46:47 - granted resource node-Disk-Archive 05/02/2011 05:46:48 - estimated 0 kbytes needed 05/02/2011 05:46:52 - started process bpbrm (pid=5953) 05/02/2011 05:46:56 - connecting 05/02/2011 05:47:07 - connected; connect time: 0:00:00 05/02/2011 05:47:12 - begin writing 05/02/2011 05:47:24 - end writing; write time: 0:00:12 the requested operation was successfully completed (0) Thanks, Scott
Looks like someone need to look at TSM) system you are using as it is
obviously the problem.
Anyway
Logical log back can be started by having onconfig option
ALARMPROGRAM /opt/informix/etc/alarmprogram.sh
and in /opt/informix/etc/alarmprogram.sh
have
BACKUPLOGS=Y
or you can have continuous backup which would mean tape is open all the time
probably not a good idea for TSM) though.
I think onbar can have more that one backuop of log going. Look at your onbar
log.
TSM calls onbar I believe.
I use ontape for backing up our logical logs and database. Using the contiuous
option for backing up logs works very well.
What is the opinion of the SM supplier? Informix uses a public interface (XBSA) and I would expect it to call the same API functions independently of the SM. Note that the XBSA implementation is done inside the XBSA library which is specific for every SM. Why does the "resource" take so long to be obtained? Unfortunately I've seen this with the same SM, I raised the issue inside the customer team but things haven't improved. It takes much longer to establish the connection than to transfer the data... for me, personally, this is a bit hard to accept. And please remember the effect of this if you need to restore with logs... It will greatly increase the recovery time. I would love to see some answers for this issue.... Regards On Tue, May 3, 2011 at 2:20 PM, SCOTT YOHONN <syohonn@gmail.com> wrote: > We are in the process of changing our enterprise backup solution from > Tivoli > Storage manager (TSM) to Symantec NetBackup (NBU). Our Informix logical > logs > average about 32mb in size and are able to backup to TSM in 5 seconds or > less. > The Informix databases that have been converted to NBU take MUCH longer to > backup logical logs (1 3 minutes/logical log) and cant keep up. I would > prefer not to change the size or number of the logical logs to work around > this problem, but it might be inevitable with NBU. > The actual read/write of a logical log backup to NBU only takes seconds. It > appears that most of the time to backup a logical log in NBU occurs during > the > phase in which the backup process requests a resource and then is granted > a > resource for the backup of each logical log (see example below). Does > anyone > have any experience with making a persistent connection for Informix > logical > logs to NBU so that each logical log backup does NOT need to establish a > new > connection to NBU each time for a backup? Also, does anyone know how > Informix > spawns processes for logical log backups? Specifically, as transactions > progress through various logical logs, does Informix spawn separate backup > processes for each logical log or is only one logical log backed up at a > time? > > 05/02/2011 05:43:13 - requesting resource NBU_DISK_Archive_SUG > 05/02/2011 05:43:13 - requesting resource > nbu_master.acmeco.com.NBU_CLIENT.MAXJOBS.node.acmeco.com > 05/02/2011 05:43:13 - requesting resource > nbu_master.acmeco.com.NBU_POLICY.MAXJOBS.Informix_Logs_RDC_Prod > 05/02/2011 05:46:47 - granted resource > nbu_master.acmeco.com.NBU_CLIENT.MAXJOBS.node.acmeco.com > 05/02/2011 05:46:47 - granted resource > nbu_master.acmeco.com.NBU_POLICY.MAXJOBS.Informix_Logs_RDC_Prod > 05/02/2011 05:46:47 - granted resource > > MediaID=@aaaaB;DiskVolume=/nbu_archive1;DiskPool=node_AD_Archive;Path=/nbu_archi ve1;StorageServer= > node.acmeco.com;MediaServer=node.acmeco.com > 05/02/2011 05:46:47 - granted resource node-Disk-Archive > 05/02/2011 05:46:48 - estimated 0 kbytes needed > 05/02/2011 05:46:52 - started process bpbrm (pid=5953) > 05/02/2011 05:46:56 - connecting > 05/02/2011 05:47:07 - connected; connect time: 0:00:00 > 05/02/2011 05:47:12 - begin writing > 05/02/2011 05:47:24 - end writing; write time: 0:00:12 > the requested operation was successfully completed (0) > > Thanks, > Scott > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --0016e68e8026b812f404a265e9af
All, Just to provide an update on this issue... I have been working with Symantec (NetBackup) support and they have provided us some configuration settings to apply our our NetBackup Master and Media servers. Our Storage and UNIX teams will handle this and they only one want apply one change at a time due to the centralized nature our NBU and we have no test NBU environment. We have not gotten the maintenance windows to do this yet (It won't be until the end of May), but here is was Symantec is suggesting so far: 1. Disable the tcp_fusion on Solaris 10 servers that act as Master or Media NBU servers. Here is a URL on how to do this http://www.symantec.com/docs/TECH62004 2. Change parameters within certain files that reside in the usr/openv/var/global directory A. emm.conf file on Master and Media NBU servers to have these values NUM_DB_BROWSE_CONNECTIONS=10 NUM_DB_CONNECTIONS=11 NUM_ORB_THREADS=21 B.server.conf on Master and Media NBU servers to have these values -c 400M -cl 400M -ch 750M C.nbrb.conf file on Master and Media NBU servers to have these values SECONDS_FOR_EVAL_LOOP_RELEASE = 180 RESPECT_REQUEST_PRIORITY = 0 DO_INTERMITTENT_UNLOADS = 1 I will provide an update on how this effects NBU being able to more quicky find resources for the backup of logical logs. This will take time due to the nature of how we can get maintenance windows.