Logical Log Sizing

Hey Folks, IDS 11.70.FC8XC4 AIX 7.2 The short of it is that I am looking for some logical log sizing suggestions. Let me explain what I am facing I have 644 logical logs sized at 100Mg each. On most days I will use approx. 200 of these. On peak days, I will use 600 thus the total number of logs that I can go a day w/o backing up. I also just swapped from TSM to NBU storage managers. During peak times, I will go through 4 logs a minute and the storage manager (NBU) cannot keep up with backing them up. On Sunday, I had > 300 that needed to be backed up. Backups of the logs were happening however a lot slower than I was using them. They were backing up at about 25 - 30 seconds per log. Coincidentally, TSM never experienced this lag. Working with the storage manager folks here, they tell me that this is the best they can do and that I need to use larger logs because this size is not efficient. It takes longer for the storage manager overhead than it does to actually back up the log. Finally, I have always been apprehensive about very large logical logs because if I lose that disk, I lose more data. So, is anyone using very large logical logs? If so, what size are you using? Also, for any NBU users out there. Are there any magic bullets you could throw my way or any "suggested" sizes I should be thinking about? Sorry for the long winded issue, Dan

Dan: One suggestion: I have found that when using Netbackup with Informix on a high transaction rate system, you need to set up a separate Netbackup server just for the Informix engines. It will do a better job of keeping up if the server is not working on a hundred filesystems in addition to the Informix logical logs. Art Art S. Kagel, President and Principal Consultant ASK Database Management www.askdbmgt.com Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Tue, Aug 15, 2017 at 4:50 PM, DAN MUELLER <ddmueller@west.com> wrote: > Hey Folks, > > IDS 11.70.FC8XC4 > AIX 7.2 > > The short of it is that I am looking for some logical log sizing > suggestions. > Let me explain what I am facing > > I have 644 logical logs sized at 100Mg each. On most days I will use > approx. > 200 of these. On peak days, I will use 600 thus the total number of logs > that > I can go a day w/o backing up. > > I also just swapped from TSM to NBU storage managers. During peak times, I > will go through 4 logs a minute and the storage manager (NBU) cannot keep > up > with backing them up. On Sunday, I had > 300 that needed to be backed up. > Backups of the logs were happening however a lot slower than I was using > them. > They were backing up at about 25 - 30 seconds per log. Coincidentally, TSM > never experienced this lag. > > Working with the storage manager folks here, they tell me that this is the > best they can do and that I need to use larger logs because this size is > not > efficient. It takes longer for the storage manager overhead than it does to > actually back up the log. > > Finally, I have always been apprehensive about very large logical logs > because > if I lose that disk, I lose more data. > > So, is anyone using very large logical logs? If so, what size are you > using? > Also, for any NBU users out there. Are there any magic bullets you could > throw > my way or any "suggested" sizes I should be thinking about? > > Sorry for the long winded issue, > Dan > > > ************************************************************ > ******************* > Forum Note: Use "Reply" to post a response in the discussion forum. > >

Do you use on-line compression of logical logs ? BACKUP_FILTER on $ONCONFIG, this will reduce size of LOGS on disk 10 x .. On 15.08.2017 23:50, DAN MUELLER wrote: > Hey Folks, > > IDS 11.70.FC8XC4 > AIX 7.2 > > The short of it is that I am looking for some logical log sizing suggestions. > Let me explain what I am facing > > I have 644 logical logs sized at 100Mg each. On most days I will use approx. > 200 of these. On peak days, I will use 600 thus the total number of logs that > I can go a day w/o backing up. > > I also just swapped from TSM to NBU storage managers. During peak times, I > will go through 4 logs a minute and the storage manager (NBU) cannot keep up > with backing them up. On Sunday, I had > 300 that needed to be backed up. > Backups of the logs were happening however a lot slower than I was using them. > They were backing up at about 25 - 30 seconds per log. Coincidentally, TSM > never experienced this lag. > > Working with the storage manager folks here, they tell me that this is the > best they can do and that I need to use larger logs because this size is not > efficient. It takes longer for the storage manager overhead than it does to > actually back up the log. > > Finally, I have always been apprehensive about very large logical logs because > if I lose that disk, I lose more data. > > So, is anyone using very large logical logs? If so, what size are you using? > Also, for any NBU users out there. Are there any magic bullets you could throw > my way or any "suggested" sizes I should be thinking about? > > Sorry for the long winded issue, > Dan > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >

Where does NBU spend it's time? Dealing with connection establishment or with the transfer of data? I've seen places where it takes a long time to "setup" the backup (20-30s) and then it takes a couple of seconds to transfer the data... If the issue is in the setup, than it's something they need to check/improve (I've seen places where it doesn't happen). If the time is spent in the "setup" then you need to understand where exactly... in the simplest cases it may be just delays due to DNS (reverse lookups). Eventually you'd need to send the logs to a pool of disks instead of directly to tape (which will take longer to mount obviously). Another option would be to send the logs to disk and then use NBU to back them up (basically what they probably do with Oracle). But this will be tricky to setup and specially to restore (when you need "fast and simple") Regards. On Tue, Aug 15, 2017 at 10:50 PM, DAN MUELLER <ddmueller@west.com> wrote: > Hey Folks, > > IDS 11.70.FC8XC4 > AIX 7.2 > > The short of it is that I am looking for some logical log sizing > suggestions. > Let me explain what I am facing > > I have 644 logical logs sized at 100Mg each. On most days I will use > approx. > 200 of these. On peak days, I will use 600 thus the total number of logs > that > I can go a day w/o backing up. > > I also just swapped from TSM to NBU storage managers. During peak times, I > will go through 4 logs a minute and the storage manager (NBU) cannot keep > up > with backing them up. On Sunday, I had > 300 that needed to be backed up. > Backups of the logs were happening however a lot slower than I was using > them. > They were backing up at about 25 - 30 seconds per log. Coincidentally, TSM > never experienced this lag. > > Working with the storage manager folks here, they tell me that this is the > best they can do and that I need to use larger logs because this size is > not > efficient. It takes longer for the storage manager overhead than it does to > actually back up the log. > > Finally, I have always been apprehensive about very large logical logs > because > if I lose that disk, I lose more data. > > So, is anyone using very large logical logs? If so, what size are you > using? > Also, for any NBU users out there. Are there any magic bullets you could > throw > my way or any "suggested" sizes I should be thinking about? > > Sorry for the long winded issue, > Dan > > > ************************************************************ > ******************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently...

Hi, I'd think you can make the logs bigger. So big that you have "only" about 100 logical logs (providing the same total log space you have now). That way the percentage calculations (e.g. for the high watermarks) still work reasonably accurate. If you worry about loosing logical log records due to a disk problem, you may consider mirroring the dbspace(s) that contain the logical logs (using plain old Informix mirroring or some other method e.g. on a file system level). Regards, Martin -- Martin Fuerderer Informix Development Germany HCL Technologies Germany GmbH Frankfurter Ring 17 80807 Munich, Germany e-mail: martin.fuerderer1@de.ibm.com [1]http://www.hcltech.com/de --DISCLAIMER-- ---------------------------------------------------------------------- ---------------------------------------------------------------------- -------- The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects. ---------------------------------------------------------------------- ---------------------------------------------------------------------- -------- From: "DAN MUELLER" <ddmueller@west.com> To: ids@iiug.org Date: 08/15/2017 11:51 PM Subject: Logical Log Sizing [39714] Sent by: ids-bounces@iiug.org _________________________________________________________________ Hey Folks, IDS 11.70.FC8XC4 AIX 7.2 The short of it is that I am looking for some logical log sizing suggestions. Let me explain what I am facing I have 644 logical logs sized at 100Mg each. On most days I will use approx. 200 of these. On peak days, I will use 600 thus the total number of logs that I can go a day w/o backing up. I also just swapped from TSM to NBU storage managers. During peak times, I will go through 4 logs a minute and the storage manager (NBU) cannot keep up with backing them up. On Sunday, I had > 300 that needed to be backed up. Backups of the logs were happening however a lot slower than I was using them. They were backing up at about 25 - 30 seconds per log. Coincidentally, TSM never experienced this lag. Working with the storage manager folks here, they tell me that this is the best they can do and that I need to use larger logs because this size is not efficient. It takes longer for the storage manager overhead than it does to actually back up the log. Finally, I have always been apprehensive about very large logical logs because if I lose that disk, I lose more data. So, is anyone using very large logical logs? If so, what size are you using? Also, for any NBU users out there. Are there any magic bullets you could throw my way or any "suggested" sizes I should be thinking about? Sorry for the long winded issue, Dan ********************************************************************** ********* Forum Note: Use "Reply" to post a response in the discussion forum. References 1. http://www.hcltech.com/de

Dan, not much choice here. I would size the logs larger to such an extent that NBU stays current so in the event the disk fails you recover up to current - 1 log. Just keep your high water marks in mind. Our largest customer uses 256 mg logs and nothing adverse has ever come of it. You probably will have to move fast on this since if you have to restore all 600+ logs it will take around 6 hours or so, not fun. Mark

Hi Dan, I don't think your problem is inherent with Netbackup. Just looking at one of our system which uses 200 Mb logs and can fill one of these in well under a minute before calling onbar to send it to Netbackup taking around 12 s to do so. 20:01:40 Logical Log 733193 Complete, timestamp: 0xee4da9cd. 20:01:40 Logical Log 733193 - Backup Started 20:01:52 Logical Log 733193 - Backup Completed In general it is true that you might need larger logs because there is a "backup set up time" incurred once regardless of the logs you intend to send. However, I would suggest your NBU system may benefit from tuning especially if network latency could be an issue. Informix side, I think you can only tune: BAR_NB_XPORT_COUNT 100 BAR_XFER_BUF_SIZE 31 I have shown the settings we use. We don't have a NBU server dedicated to Informix. Like with Informix, Netbackup performance can be heavily dependent on the storage used for backups. Finally, on your comment of log size versus data loss. I doubt any business will state acceptable data loss in terms of the maximum size of an Informix logical log. If you want zero or close to zero data loss in the event of the complete loss of a server the best way to do this is to have a HDR or RSS server running at another site. Then to recover logical logs you roll forward with NBU at least as far as a log that is current on your HDR/RSS and do the last logs by pairing up with the HDR/RSS. This process is in the IBM docs somewhere. Ben.

Hi Dan, NBU has built in slowness. With one customer I saw on average it takes 9 seconds to backup a 20MB logical log, with a fibre connection and a Veritas server that is almost dedicated to INfomrix (does so small backups once a day. All of the time was in building the XBSA connection and starting the XBSA session (4 Seconds) and then closing the XBSA Connection (3 Second). Onbar also has a built in delay of 1 second per backup, so as you can see the actual time to backup was 1 second, but it took 9 seconds. The backup is quite fast, so the larger the logical log the faster effective backup. For instance a 60 MB logical log, using NBU, only took 11 seconds in the same environment. With the advent of significantly more stable drives, a larger sized logical log is not the danger it once was. -Mark