Re: Very slow onbar restore

Thanks Bruce. I note the comment about the old version. Most of your comments seem to relate to backup speeds: in this case the backup speeds are fine and the problem is with *restore* times. regards Neil "Brice Avila" <briceavila@hotmail.com> wrote in message news:c46e3ec5.0405270550.41fc6238@posting.google.com... > Neil, > > NetBackup 3.4_1 is a OLD, non-supported version of NetBackup, and > there have been some performance improvements in newer vesions. The > latest version is 5.0 MP1. That said, try the following: > > 1. Make sure that BAR_DEBUG is not set in the onconfig file. If it is, > remove it and restart the backup. BAR_DEBUG is used for debugging and > causes performance slowdowns. > > 2. Set BAR_MAX_BACKUP to zero or a high number in the onconfig file. > The BAR_MAX_BACKUP onconfig parameter specifies the maximum number of > parallel child processes that will be forked for each dbspace backup, > one per dbspace, depending on the amount of shared memory that one has > available. Generally more is better. Setting BAR_MAX_BACKUP to zero > means an unlimited number of processes can be forked. > > 3. Make sure that the backup is run in parallel by not using the "-w" > flag. But also make sure that you have a corresponding set of storage > units in your class as BAR_MAX_BACKUP. Informix backups by default run > in parallel mode. Using the "-w" flag will cause a backup to run in > serial mode, which is safer but slower. > > Another way backups can be run in serial mode is if BAR_MAX_BACKUP is > set to 1. Make sure that BAR_MAX_BACKUP is set to 0, or a high number. > > 4. If you have a bptm log > ("/usr/openv/netbackup/logs/bptm/log.<date>") on your media server and > have VERBOSE=5 in your /usr/openv/netbackup/bp.conf file set you'll > see messages similar to the following: > > "write_data: waited for full buffer 2134 times, delayed 9846 times" > "read_data: waited for empty buffer 100 times, delayed 5456 times" > > "waiting for full buffer" is NetBackup waiting for information from > onbar. "waiting for empty buffer" is the tape library taking > information from NetBackup. Each 10,000 delays is 5 minutes. This will > help you focus on which side (or both) you need to concentrate your > efforts on. > > 5. Make sure that Informix and Veritas have adequate kernel resources. > Since both Informix and Veritas user memory, CPU and I/O resources one > must ensure that both products have adequate kernel resources to run > both. Informix has kernel recommendations for it's product in the > release notes at "$INFORMIXDIR/release/en_us/0333/IDS_7.3". Veritas > has kernel recommendations at "http://support.veritas.com/docs/264967"; > > 6. Increase the BAR_NB_XPORT_COUNT and BAR_XFER_BUF_SIZE onconfig > parameters until an "onstat -g -stq" shows diminishing returns. > > The BAR_NB_XPORT_COUNT Informix parameter specifies the number of data > buffers that each onbar_d process can use to exchange data with the > database server. The BAR_XFER_BUFSIZE Informix parameter specifies the > size of each transfer buffer. In general, more of each means faster > transfer of data, and these parameters have a direct effect on the > backup and restore performance. The default values of both are: > > BAR_NB_XPORT_COUNT: 10 > BAR_XFER_BUF_SIZE: 31 for 2K page ports, 15 for 4K page ports (see > your release notes for the page size of your instance) > > Please note that if the BAR_XFER_BUFSIZE is changed, all previous > Informix backups cannot be restored. > > Hope this information helps. > > Brice Avila > Minneapolis, Minnesota > > "Neil Truby" <neil.truby@ardenta.com> wrote in message news:<2hjtk3Fdofe8U1@uni-berlin.de>... > > IDS 7.31 on HP-UX 11.11 > > > > The customer did a whole back-up onbar -b -w -L 0 on Saturday. Took about > > six hours. Netbackup 3.4.1 is used to backup to a tape device attached to > > another server on the network. Now they are doing a restore > > (onbar -r -p -w). Although it's running OK, it is so sllllooooowwwwww. For > > instance, if I zero out the stats, I see from onstat -D that the writes > > clock up at perhaps a few thousand a minute: about one-tenth of the rate I > > would expect. > > > > Using sar shows fast (sub 4ms) response times on the target disks. > > onstat -p also shows periods of inactivity, followed by a mass of writes,> > followed by more inactivity. onstat -g ses shows the ontape thread > > constantly waiting on condition. there is nothing useful in the onbar, > > Informix or system syslogs. I've added another cpu vp (it's using kaio), to > > no avail. > > > > All the above points towards a systemic problem with the tape stacker. But > > it's not a network one: the only network card configured on the target > > server shows instant response times to a ping to the server to which the > > stacker is connected. The customer has previously tested restores on > > identical hardware/OS/Informix/ Netbackup and they have flown through at the > > expected rates. > > > > The customer has also executed a file system restore using the same stacker > > (not the samne device obviously) with the expected fast restore times. The > > customer also reckons that she started the restore yesterday and it went at > > normal speeds until it failed with a tape positioning error. > > > > Any ideas? Suggestions gratefully received. > > > > IBM support call 403279 refers. > > > > thanks > > Neil

Neil, All of the suggestions also relate to restore performance tuning except #4: 1. "waiting for full buffer" is NetBackup waiting for data from the tape library 2. "waiting for empty buffer" is NetBackup waiting for onbar to take data from it Hope this helps. Brice "Neil Truby" <neil.truby@ardenta.com> wrote in message news:<2hmhuqFetch6U1@uni-berlin.de>... > Thanks Bruce. > > I note the comment about the old version. > Most of your comments seem to relate to backup speeds: in this case the > backup speeds are fine and the problem is with *restore* times. > > regards > Neil > > "Brice Avila" <briceavila@hotmail.com> wrote in message > news:c46e3ec5.0405270550.41fc6238@posting.google.com... > > Neil, > > > > NetBackup 3.4_1 is a OLD, non-supported version of NetBackup, and > > there have been some performance improvements in newer vesions. The > > latest version is 5.0 MP1. That said, try the following: > > > > 1. Make sure that BAR_DEBUG is not set in the onconfig file. If it is, > > remove it and restart the backup. BAR_DEBUG is used for debugging and > > causes performance slowdowns. > > > > 2. Set BAR_MAX_BACKUP to zero or a high number in the onconfig file. > > The BAR_MAX_BACKUP onconfig parameter specifies the maximum number of > > parallel child processes that will be forked for each dbspace backup, > > one per dbspace, depending on the amount of shared memory that one has > > available. Generally more is better. Setting BAR_MAX_BACKUP to zero > > means an unlimited number of processes can be forked. > > > > 3. Make sure that the backup is run in parallel by not using the "-w" > > flag. But also make sure that you have a corresponding set of storage > > units in your class as BAR_MAX_BACKUP. Informix backups by default run > > in parallel mode. Using the "-w" flag will cause a backup to run in > > serial mode, which is safer but slower. > > > > Another way backups can be run in serial mode is if BAR_MAX_BACKUP is > > set to 1. Make sure that BAR_MAX_BACKUP is set to 0, or a high number. > > > > 4. If you have a bptm log > > ("/usr/openv/netbackup/logs/bptm/log.<date>") on your media server and > > have VERBOSE=5 in your /usr/openv/netbackup/bp.conf file set you'll > > see messages similar to the following: > > > > "write_data: waited for full buffer 2134 times, delayed 9846 times" > > "read_data: waited for empty buffer 100 times, delayed 5456 times" > > > > "waiting for full buffer" is NetBackup waiting for information from > > onbar. "waiting for empty buffer" is the tape library taking > > information from NetBackup. Each 10,000 delays is 5 minutes. This will > > help you focus on which side (or both) you need to concentrate your > > efforts on. > > > > 5. Make sure that Informix and Veritas have adequate kernel resources. > > Since both Informix and Veritas user memory, CPU and I/O resources one > > must ensure that both products have adequate kernel resources to run > > both. Informix has kernel recommendations for it's product in the > > release notes at "$INFORMIXDIR/release/en_us/0333/IDS_7.3". Veritas > > has kernel recommendations at "http://support.veritas.com/docs/264967"; > > > > 6. Increase the BAR_NB_XPORT_COUNT and BAR_XFER_BUF_SIZE onconfig > > parameters until an "onstat -g -stq" shows diminishing returns. > > > > The BAR_NB_XPORT_COUNT Informix parameter specifies the number of data > > buffers that each onbar_d process can use to exchange data with the > > database server. The BAR_XFER_BUFSIZE Informix parameter specifies the > > size of each transfer buffer. In general, more of each means faster > > transfer of data, and these parameters have a direct effect on the > > backup and restore performance. The default values of both are: > > > > BAR_NB_XPORT_COUNT: 10 > > BAR_XFER_BUF_SIZE: 31 for 2K page ports, 15 for 4K page ports (see > > your release notes for the page size of your instance) > > > > Please note that if the BAR_XFER_BUFSIZE is changed, all previous > > Informix backups cannot be restored. > > > > Hope this information helps. > > > > Brice Avila > > Minneapolis, Minnesota > > > > "Neil Truby" <neil.truby@ardenta.com> wrote in message > news:<2hjtk3Fdofe8U1@uni-berlin.de>... > > > IDS 7.31 on HP-UX 11.11 > > > > > > The customer did a whole back-up onbar -b -w -L 0 on Saturday. Took > about > > > six hours. Netbackup 3.4.1 is used to backup to a tape device attached > to > > > another server on the network. Now they are doing a restore > > > (onbar -r -p -w). Although it's running OK, it is so sllllooooowwwwww. > For > > > instance, if I zero out the stats, I see from onstat -D that the writes > > > clock up at perhaps a few thousand a minute: about one-tenth of the rate > I > > > would expect. > > > > > > Using sar shows fast (sub 4ms) response times on the target disks. > > > onstat -p also shows periods of inactivity, followed by a mass of > writes,> > > followed by more inactivity. onstat -g ses shows the ontape thread > > > constantly waiting on condition. there is nothing useful in the onbar, > > > Informix or system syslogs. I've added another cpu vp (it's using > kaio), to > > > no avail. > > > > > > All the above points towards a systemic problem with the tape stacker. > But > > > it's not a network one: the only network card configured on the target > > > server shows instant response times to a ping to the server to which the > > > stacker is connected. The customer has previously tested restores on > > > identical hardware/OS/Informix/ Netbackup and they have flown through at > the > > > expected rates. > > > > > > The customer has also executed a file system restore using the same > stacker > > > (not the samne device obviously) with the expected fast restore times. > The > > > customer also reckons that she started the restore yesterday and it went > at > > > normal speeds until it failed with a tape positioning error. > > > > > > Any ideas? Suggestions gratefully received. > > > > > > IBM support call 403279 refers. > > > > > > thanks > > > Neil