RE: ONBAR-VERITAS DB restore performance issue... [293]

This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C3FFCC.A5B6FCB0 Content-Type: text/plain; charset="iso-8859-1" Hi, I would like to post some updates on this performance issue topic to get some more suggestions. From the bptm logs. it shows that the delay is on the client side. But, we are yet to isolate the issue btw the DB server, Solaris 9 and the storage array Sun storedge A5200 on this host client. We put all the latest patches for Solaris 9, Volume manager 3.5 mp2, Veritas Netbackup 4.5 MP5. In fact, we actually applied all the firmware revision for the storedge array. We defined our DB volumes layout on RAID-5 with 8 columns and 32 k stripe size. I see the I/O is utilized to 35 % only at the max. We did perform the alternate client restore with RAID 5 h/w on the Sun storedge T3 Array. But this one is with the VERITAS volume manager 3.5 MP2 - RAID5 software level. Iam not sure whether that is causing the performance issue. From the Informix side, there is nothing much to tune for ONBAR except the BAR_NB_XPORT_COUNT since iam doing the serial whole system restore. I even bumped up that parameter to 10000 from 100. I found that there is a KAIO bug fixed in Solaris 9 but thats for Oracle. So, we even tried disabling the KAIO and ran the restore with AIO threads and still didnt improve the performance at all. We still didnt get the 10000 KB/SEC and we get only 500 KB/SEC. Please post your comments/suggestions on this topic. I would like to clarify one more thing here. My question: During the whole system restore, since i had the performance issues, Every time i troubleshoot and fix something, i got to re-start the restore using "onbar -RESTART" since i enabled the Restartable restore in the Onconfig file. Since the troubleshooting has gone for few days and there are times before i start the restartable restore, i changed the ixbar file to include the latest incremental backups to restore. Now today, it just completed the whole system restore and for some reasons, it didnt restore the incremental backup at all. Hence I'm performing the log restores for all logs since the last level-0 restore that is just completed. In the past, i changed the ixbar files during the restartable restore to include the latest logs from PRD. That used to restore upto the last logs that i just included in the ixbar file. Iam not sure why it didnt restore the level-1 the same way in this instance. 1) Can't the ixbar file be changed during the restartable restore to purposely include the latest level-1's ? Thanks RAJ -----Original Message----- From: Andrew J. Dibbins [mailto:andy_d@rapier.demon.co.uk] Sent: Friday, February 27, 2004 4:08 PM To: Rajasekaran, Rajesh Subject: RE: ONBAR-VERITAS DB restore performance issue... [293] Hi Rajesh, Just to confirm: SIZE_DATA_BUFFERS = 16 NUMBER_DATA_BUFFERS = 1048576 Surely, you mean the other way round SIZE_DATA_BUFFERS = 1048576 NUMBER = 16 Anyway, the size figure is way too high, it should be 65536 for dlt 7k, and should never be bigger. I don't think its a fix for the problems your experiencing, but it will surely improve overal backup performance. Andy -----Original Message----- From: Rajasekaran, Rajesh [mailto:RRajasekaran@forestpharm.com] Sent: Friday, February 27, 2004 1:40 PM To: 'Andrew J. Dibbins'; Rajasekaran, Rajesh Subject: RE: ONBAR-VERITAS DB restore performance issue... [293] Andy, We have the SUN L40 DLT 7000 tape drives. There are 2 such drives in the robotic library. Our SIZE_DATA_BUFFERS,NUMBER_DATA_BUFFERS are set to 16,1048576 respectively. Thanks RAJ -----Original Message----- From: Andrew J. Dibbins [ mailto:andy_d@rapier.demon.co.uk <mailto:andy_d@rapier.demon.co.uk> ] Sent: Thursday, February 26, 2004 3:12 PM To: Rajasekaran.... Subject: RE: ONBAR-VERITAS DB restore performance issue... [293] Hi Rajash, This probably won't matter, because of your bptm logging, but check /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS and NUMBER_DATA_BUFFERS Your don't mention the tape drives in use, but consider SIZE_DATA_BUFFERS set to: 64k for dlt 256k for lto's and stk9940B and our best results for the number of buffers is around 16, but although I've never used this there is a extra parameter RESTORE_DATA_BUFFERS (check manual). Also, make sure shared memory paramters are in /etc/system on the NBU master, as its this that gets the used when the above gets allocated. Clearly the raid 5 isn't clever, you could consider changing the strip size to 64K, some benefit, perhaps. Keep us posted. Andy Home: andrew.dibbins@rapier.demon.co.uk Work: andrew.dibbins@sun.com -----Original Message----- From: forum.subscriber@iiug.org [ mailto:forum.subscriber@iiug.org <mailto:forum.subscriber@iiug.org> ]On Behalf Of Rajasekaran.... Sent: Thursday, February 26, 2004 2:00 PM To: sapmix@iiug.org Subject: ONBAR-VERITAS DB restore performance issue... [293] Hi, We have a performance issue with ONBAR restore for our IDS 7.31UD2XG. We use the Veritas Netbackup 4.5GA under Solaris 8. The backup (whole system serial) that we are trying to restore is the whole system restore. The restore throughput used to be 10 mb/sec and now its around 3 mb/sec. The restore is going over the network. The resore client has 100 MBPS network and the master server is 1000 Mbps/sec (gigabit network). I dont see much usage on the CPU,memory on both the box. Our network ppl say that the load between the two host is normal. We somehow identified that the slow down could be very specific to the restore client and it may be due to disk writes. But, we also have the view that the storage array is only 25% used during the restore. The storage array is Sun storedge A5200 array on Sun Fire V880 host. The DB volume layouts are RAID5 with 8 cols and 32k stripe size. I know RAID5 is going to slow down writes due to parity writing but i dont think it should degrade to the extent of more than 75%. Our VERITAS bptm logs from the master server shows that the delay is after it sends the data to the client. For an object, 6.5 hrs out of 10 hrs was the delay for one object. This used to complete in 4.5 hrs. But on this instance, the disk is not heavily loaded and hence more data can be pumped up. Considering the storage array is not utilized much, Should the BAR_NB_XPORT_COUNT and the BAR_NB_XFER_SIZE be increased ? I just left the values to 100 & 31 respectively. I am assuming that increasing this parameter will only have effect on parallel backups/restores. Also, BAR_NB_XFER_SIZE is set to 31 by default on 2k pagesize system. But it says xbsa has 64kb limit. At the same time, it can have values upto 1 million.Will it help if i increase beyond 32 k ? Will tuning both these parameters for a serial whole system restore help ? Please pass on your suggestions/comments. Regards Rajesh Rajas

Rajasekaran, Rajesh wrote: > This message is in MIME format. Don't post MIME. Or HTML - it's incredibly horrible. > We defined our DB volumes layout on RAID-5 Careful - you'll set Art off. Try searching for 'NO RAID 5' written by Art Kagel in groups.google.com - read and learn. -- Jonathan Leffler #include <disclaimer.h> Email: jleffler@earthlink.net, jleffler@us.ibm.com Guardian of DBD::Informix v2003.04 -- http://dbi.perl.org/