Logical log size
Larry asked whether enlarging logical logs on IDS 11.50.FC7 (Solaris, with HDR) has downsides, after getting "log size may be too small" warnings during batch jobs that fill a log every 20-30 seconds and run 5-6 hours. Replies gave sizing rules of thumb: size a single log by how much data you can afford to lose, keep enough total log space for ~4-5 days (e.g. a long weekend of failed log backups), use dynamic log addition, and keep off-machine copies. On HDR, Marcus and Art noted replication ships log buffers as transactions complete, so log size won't change the volume sent; Art suggested checking the pipe bandwidth versus log fill rate, switching to an RSS secondary, or raising recovery threads on the secondary. No confirmed outcome is recorded.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
What is the downside to increasing the logical log size on IDS 11.50.FC7 running on Solaris? I have been receiving occasional messages telling me that my log size may be too small. This occurs during large batch processing operations. We are also running HDR. Are there any other implications to increasing the size of the logical logs that I need to look at? Thanks you. Larry
Hi, the size of a single logical log file should be set in a way to not lose much data when it comes to the most severe condition that you would have to restore a database system which is down and inaccessible (so you cannot get any more the active log using salvage logs). We have set these up in a way that under normal circumstances, a logical log gets full each 5-10 minutes, on high load databases also each 2 minutes. The number of logical log files limits the maximum size of a big transaction. Also, we take into consideration that a system might be running but is not able to backup the logs for some reason. In that situation, a system should survive without blocking (because logfiles are full) for at least 12 hours. Also, make use of the option to dynamically add logfiles to prevent either long transactions or blocking because of backup needed. There are no "good defaults" for each situation, every server has its own load to cope with. This could be 20 logfiles with 10 mb each, or 200 logfiles with 100mb each. Marcus Haarmann ----- Ursprüngliche Mail ----- Von: "LARRY SORENSEN" <lsorensen25@msn.com> An: ids@iiug.org Gesendet: Montag, 28. Juli 2014 17:34:06 Betreff: Logical log size [33444] What is the downside to increasing the logical log size on IDS 11.50.FC7 running on Solaris? I have been receiving occasional messages telling me that my log size may be too small. This occurs during large batch processing operations. We are also running HDR. Are there any other implications to increasing the size of the logical logs that I need to look at? Thanks you. Larry ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.
Marcus's review is a good one. I would just add that the total size and number of logs should be sufficient that if the log backups start to fail over a holiday weekend, everything can keep running without you having to interrupt your days off to go in and force a log backup or fix the problem. So, you want to have at least 4 days of logs online. That's the rule of thumb I use for total log space. For single log sizing, it is, as Marcus pointed out, how much data can you afford to lose if the hardware catastrophically fails just before the current log fills, backs up, and is copied to a safe location. Art Art S. Kagel, Principal Consultant ASK Database Management Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Mon, Jul 28, 2014 at 11:34 AM, LARRY SORENSEN <lsorensen25@msn.com> wrote: > What is the downside to increasing the logical log size on IDS 11.50.FC7 > running on Solaris? I have been receiving occasional messages telling me > that > my log size may be too small. This occurs during large batch processing > operations. We are also running HDR. > > Are there any other implications to increasing the size of the logical logs > that I need to look at? > > Thanks you. > > Larry > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a11347f468e4a7204ff43279d
And if you are backing them up to disk then make sure you get an 'off machine' copy Cheers Paul > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Art > Kagel > Sent: Monday, July 28, 2014 11:10 AM > To: ids@iiug.org > Subject: Re: Logical log size [33446] > > Marcus's review is a good one. I would just add that the total size and > number of logs should be sufficient that if the log backups start to fail > over a holiday weekend, everything can keep running without you having to > interrupt your days off to go in and force a log backup or fix the problem. > So, you want to have at least 4 days of logs online. That's the rule of > thumb I use for total log space. > > For single log sizing, it is, as Marcus pointed out, how much data can you > afford to lose if the hardware catastrophically fails just before the > current log fills, backs up, and is copied to a safe location. > > Art > > Art S. Kagel, Principal Consultant > ASK Database Management > > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and do not reflect on the IIUG, nor any other organization with which I am > associated either explicitly, implicitly, or by inference. Neither do > those opinions reflect those of other individuals affiliated with any > entity with which I am affiliated nor those of the entities themselves. > > On Mon, Jul 28, 2014 at 11:34 AM, LARRY SORENSEN > <lsorensen25@msn.com> > wrote: > > > What is the downside to increasing the logical log size on IDS 11.50.FC7 > > running on Solaris? I have been receiving occasional messages telling me > > that > > my log size may be too small. This occurs during large batch processing > > operations. We are also running HDR. > > > > Are there any other implications to increasing the size of the logical logs > > that I need to look at? > > > > Thanks you. > > > > Larry > > > > > > > > > ********************************************************** > ********************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > --001a11347f468e4a7204ff43279d > > > ********************************************************** > ********************* > Forum Note: Use "Reply" to post a response in the discussion forum.
My rule of thumb is total log space is 5 times the average log space used in a day, that way in the event of a four day weekend (say your company calls off on Christmas Eve as well as Christmas Day, and Christmas is on a Friday). That way you can have all of your transactions for the four day weekend and some padding for emergencies. --EEM > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > Art Kagel > Sent: Monday, July 28, 2014 11:10 AM > To: ids@iiug.org > Subject: Re: Logical log size [33446] > > Marcus's review is a good one. I would just add that the total size and > number of logs should be sufficient that if the log backups start to > fail over a holiday weekend, everything can keep running without you > having to interrupt your days off to go in and force a log backup or > fix the problem. > So, you want to have at least 4 days of logs online. That's the rule of > thumb I use for total log space. > > For single log sizing, it is, as Marcus pointed out, how much data can > you afford to lose if the hardware catastrophically fails just before > the current log fills, backs up, and is copied to a safe location. > > Art > > Art S. Kagel, Principal Consultant > ASK Database Management > > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own > opinions and do not reflect on the IIUG, nor any other organization > with which I am associated either explicitly, implicitly, or by > inference. Neither do those opinions reflect those of other individuals > affiliated with any entity with which I am affiliated nor those of the > entities themselves. > > On Mon, Jul 28, 2014 at 11:34 AM, LARRY SORENSEN <lsorensen25@msn.com> > wrote: > > > What is the downside to increasing the logical log size on IDS > > 11.50.FC7 running on Solaris? I have been receiving occasional > > messages telling me that my log size may be too small. This occurs > > during large batch processing operations. We are also running HDR. > > > > Are there any other implications to increasing the size of the > logical > > logs that I need to look at? > > > > Thanks you. > > > > Larry > > > > > > > > > *********************************************************************** > ******** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > --001a11347f468e4a7204ff43279d > > > *********************************************************************** > ******** > Forum Note: Use "Reply" to post a response in the discussion forum.
Thank you all for your advice. Another part of the question is: Currently, we have plenty of logical logs to last several days. My question is, during large batching, we fill up a logical log every 20-30 seconds or so. The batches seem to be taking longer than they should. 1 - I was wondering how much of a factor that running a remote HDR secondary setup is. 2- Is it very possible that having larger logical logs might speed up the batch processing because logs aren't filling up so quickly and flooding the pipe to the secondary? The connection pipe between servers is ok for daily operations; however, it is not ideal. > To: ids@iiug.org > From: Everett.Mills@nationalbeef.com > Subject: RE: Logical log size [33448] > Date: Mon, 28 Jul 2014 12:19:22 -0400 > > My rule of thumb is total log space is 5 times the average log space used in a > day, that way in the event of a four day weekend (say your company calls off > on Christmas Eve as well as Christmas Day, and Christmas is on a Friday). That > way you can have all of your transactions for the four day weekend and some > padding for emergencies. > > --EEM > > > -----Original Message----- > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > > Art Kagel > > Sent: Monday, July 28, 2014 11:10 AM > > To: ids@iiug.org > > Subject: Re: Logical log size [33446] > > > > Marcus's review is a good one. I would just add that the total size and > > number of logs should be sufficient that if the log backups start to > > fail over a holiday weekend, everything can keep running without you > > having to interrupt your days off to go in and force a log backup or > > fix the problem. > > So, you want to have at least 4 days of logs online. That's the rule of > > thumb I use for total log space. > > > > For single log sizing, it is, as Marcus pointed out, how much data can > > you afford to lose if the hardware catastrophically fails just before > > the current log fills, backs up, and is copied to a safe location. > > > > Art > > > > Art S. Kagel, Principal Consultant > > ASK Database Management > > > > Blog: http://informix-myview.blogspot.com/ > > > > Disclaimer: Please keep in mind that my own opinions are my own > > opinions and do not reflect on the IIUG, nor any other organization > > with which I am associated either explicitly, implicitly, or by > > inference. Neither do those opinions reflect those of other individuals > > affiliated with any entity with which I am affiliated nor those of the > > entities themselves. > > > > On Mon, Jul 28, 2014 at 11:34 AM, LARRY SORENSEN <lsorensen25@msn.com> > > wrote: > > > > > What is the downside to increasing the logical log size on IDS > > > 11.50.FC7 running on Solaris? I have been receiving occasional > > > messages telling me that my log size may be too small. This occurs > > > during large batch processing operations. We are also running HDR. > > > > > > Are there any other implications to increasing the size of the > > logical > > > logs that I need to look at? > > > > > > Thanks you. > > > > > > Larry > > > > > > > > > > > > > > *********************************************************************** > > ******** > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > > > > --001a11347f468e4a7204ff43279d > > > > > > *********************************************************************** > > ******** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >
Another piece of information is, at it's busiest during batch processing, the server is only at about 40%-50% utilized. I am just looking for something to cut the processing from 5-6 hours down to 2-3 hours. I don't have any control over the application, so I am trying to do what I can on the database/OS side. (IDS 11.50.FC7GE) > To: ids@iiug.org > From: Everett.Mills@nationalbeef.com > Subject: RE: Logical log size [33448] > Date: Mon, 28 Jul 2014 12:19:22 -0400 > > My rule of thumb is total log space is 5 times the average log space used in a > day, that way in the event of a four day weekend (say your company calls off > on Christmas Eve as well as Christmas Day, and Christmas is on a Friday). That > way you can have all of your transactions for the four day weekend and some > padding for emergencies. > > --EEM > > > -----Original Message----- > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > > Art Kagel > > Sent: Monday, July 28, 2014 11:10 AM > > To: ids@iiug.org > > Subject: Re: Logical log size [33446] > > > > Marcus's review is a good one. I would just add that the total size and > > number of logs should be sufficient that if the log backups start to > > fail over a holiday weekend, everything can keep running without you > > having to interrupt your days off to go in and force a log backup or > > fix the problem. > > So, you want to have at least 4 days of logs online. That's the rule of > > thumb I use for total log space. > > > > For single log sizing, it is, as Marcus pointed out, how much data can > > you afford to lose if the hardware catastrophically fails just before > > the current log fills, backs up, and is copied to a safe location. > > > > Art > > > > Art S. Kagel, Principal Consultant > > ASK Database Management > > > > Blog: http://informix-myview.blogspot.com/ > > > > Disclaimer: Please keep in mind that my own opinions are my own > > opinions and do not reflect on the IIUG, nor any other organization > > with which I am associated either explicitly, implicitly, or by > > inference. Neither do those opinions reflect those of other individuals > > affiliated with any entity with which I am affiliated nor those of the > > entities themselves. > > > > On Mon, Jul 28, 2014 at 11:34 AM, LARRY SORENSEN <lsorensen25@msn.com> > > wrote: > > > > > What is the downside to increasing the logical log size on IDS > > > 11.50.FC7 running on Solaris? I have been receiving occasional > > > messages telling me that my log size may be too small. This occurs > > > during large batch processing operations. We are also running HDR. > > > > > > Are there any other implications to increasing the size of the > > logical > > > logs that I need to look at? > > > > > > Thanks you. > > > > > > Larry > > > > > > > > > > > > > > *********************************************************************** > > ******** > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > > > > --001a11347f468e4a7204ff43279d > > > > > > *********************************************************************** > > ******** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >
Hi, HDR is not transferring logfiles in normal mode, only for recovery (when the servers are out of sync). When you have an active/synchronized secondary, transfer occurs for each finished transaction, not depending on the log size. I don't thing the log size will modify that behaviour in a way that you can transfer more/less data. Marcus Haarmann ----- Ursprüngliche Mail ----- Von: "LARRY SORENSEN" <lsorensen25@msn.com> An: ids@iiug.org Gesendet: Montag, 28. Juli 2014 18:28:06 Betreff: RE: Logical log size [33449] Thank you all for your advice. Another part of the question is: Currently, we have plenty of logical logs to last several days. My question is, during large batching, we fill up a logical log every 20-30 seconds or so. The batches seem to be taking longer than they should. 1 - I was wondering how much of a factor that running a remote HDR secondary setup is. 2- Is it very possible that having larger logical logs might speed up the batch processing because logs aren't filling up so quickly and flooding the pipe to the secondary? The connection pipe between servers is ok for daily operations; however, it is not ideal. > To: ids@iiug.org > From: Everett.Mills@nationalbeef.com > Subject: RE: Logical log size [33448] > Date: Mon, 28 Jul 2014 12:19:22 -0400 > > My rule of thumb is total log space is 5 times the average log space used in a > day, that way in the event of a four day weekend (say your company calls off > on Christmas Eve as well as Christmas Day, and Christmas is on a Friday). That > way you can have all of your transactions for the four day weekend and some > padding for emergencies. > > --EEM > > > -----Original Message----- > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > > Art Kagel > > Sent: Monday, July 28, 2014 11:10 AM > > To: ids@iiug.org > > Subject: Re: Logical log size [33446] > > > > Marcus's review is a good one. I would just add that the total size and > > number of logs should be sufficient that if the log backups start to > > fail over a holiday weekend, everything can keep running without you > > having to interrupt your days off to go in and force a log backup or > > fix the problem. > > So, you want to have at least 4 days of logs online. That's the rule of > > thumb I use for total log space. > > > > For single log sizing, it is, as Marcus pointed out, how much data can > > you afford to lose if the hardware catastrophically fails just before > > the current log fills, backs up, and is copied to a safe location. > > > > Art > > > > Art S. Kagel, Principal Consultant > > ASK Database Management > > > > Blog: http://informix-myview.blogspot.com/ > > > > Disclaimer: Please keep in mind that my own opinions are my own > > opinions and do not reflect on the IIUG, nor any other organization > > with which I am associated either explicitly, implicitly, or by > > inference. Neither do those opinions reflect those of other individuals > > affiliated with any entity with which I am affiliated nor those of the > > entities themselves. > > > > On Mon, Jul 28, 2014 at 11:34 AM, LARRY SORENSEN <lsorensen25@msn.com> > > wrote: > > > > > What is the downside to increasing the logical log size on IDS > > > 11.50.FC7 running on Solaris? I have been receiving occasional > > > messages telling me that my log size may be too small. This occurs > > > during large batch processing operations. We are also running HDR. > > > > > > Are there any other implications to increasing the size of the > > logical > > > logs that I need to look at? > > > > > > Thanks you. > > > > > > Larry > > > > > > > > > > > > > > *********************************************************************** > > ******** > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > > > > --001a11347f468e4a7204ff43279d > > > > > > *********************************************************************** > > ******** > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.
It's possible. Is the secondary set up in synchronous mode or asynch? What's the rating on the pipe versus the rate at which the logs are filling during batch processing? So 2-3 logs a minute at what size? Use that to calculate the amount of data you are trying to push down the pipe per second and see if the pipe can handle that much. If not, get a wider pipe or switch to RSS secondary which is asynch and full duplex and can handle being behind better. Art Art S. Kagel, Principal Consultant ASK Database Management Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Mon, Jul 28, 2014 at 12:28 PM, LARRY SORENSEN <lsorensen25@msn.com> wrote: > Thank you all for your advice. > > Another part of the question is: > > Currently, we have plenty of logical logs to last several days. My question > is, during large batching, we fill up a logical log every 20-30 seconds or > so. > The batches seem to be taking longer than they should. > > 1 - I was wondering how much of a factor that running a remote HDR > secondary > setup is. > > 2- Is it very possible that having larger logical logs might speed up the > batch processing because logs aren't filling up so quickly and flooding the > pipe to the secondary? The connection pipe between servers is ok for daily > operations; however, it is not ideal. > > > To: ids@iiug.org > > From: Everett.Mills@nationalbeef.com > > Subject: RE: Logical log size [33448] > > Date: Mon, 28 Jul 2014 12:19:22 -0400 > > > > My rule of thumb is total log space is 5 times the average log space > used in > a > > day, that way in the event of a four day weekend (say your company calls > off > > on Christmas Eve as well as Christmas Day, and Christmas is on a Friday). > That > > way you can have all of your transactions for the four day weekend and > some > > padding for emergencies. > > > > --EEM > > > > > -----Original Message----- > > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > > > Art Kagel > > > Sent: Monday, July 28, 2014 11:10 AM > > > To: ids@iiug.org > > > Subject: Re: Logical log size [33446] > > > > > > Marcus's review is a good one. I would just add that the total size and > > > number of logs should be sufficient that if the log backups start to > > > fail over a holiday weekend, everything can keep running without you > > > having to interrupt your days off to go in and force a log backup or > > > fix the problem. > > > So, you want to have at least 4 days of logs online. That's the rule of > > > thumb I use for total log space. > > > > > > For single log sizing, it is, as Marcus pointed out, how much data can > > > you afford to lose if the hardware catastrophically fails just before > > > the current log fills, backs up, and is copied to a safe location. > > > > > > Art > > > > > > Art S. Kagel, Principal Consultant > > > ASK Database Management > > > > > > Blog: http://informix-myview.blogspot.com/ > > > > > > Disclaimer: Please keep in mind that my own opinions are my own > > > opinions and do not reflect on the IIUG, nor any other organization > > > with which I am associated either explicitly, implicitly, or by > > > inference. Neither do those opinions reflect those of other individuals > > > affiliated with any entity with which I am affiliated nor those of the > > > entities themselves. > > > > > > On Mon, Jul 28, 2014 at 11:34 AM, LARRY SORENSEN <lsorensen25@msn.com> > > > wrote: > > > > > > > What is the downside to increasing the logical log size on IDS > > > > 11.50.FC7 running on Solaris? I have been receiving occasional > > > > messages telling me that my log size may be too small. This occurs > > > > during large batch processing operations. We are also running HDR. > > > > > > > > Are there any other implications to increasing the size of the > > > logical > > > > logs that I need to look at? > > > > > > > > Thanks you. > > > > > > > > Larry > > > > > > > > > > > > > > > > > > > *********************************************************************** > > > ******** > > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > > > > > > > > --001a11347f468e4a7204ff43279d > > > > > > > > > *********************************************************************** > > > ******** > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > > ******************************************************************************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a11c33d7013e88104ff4416d5
You can also increase the number of offline recovery threads on the secondary so that it can process the logs faster (assuming the pipe is wide enough to get the data over there). Art Art S. Kagel, Principal Consultant ASK Database Management Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Mon, Jul 28, 2014 at 1:03 PM, LARRY SORENSEN <lsorensen25@msn.com> wrote: > Another piece of information is, at it's busiest during batch processing, > the > server is only at about 40%-50% utilized. > > I am just looking for something to cut the processing from 5-6 hours down > to > 2-3 hours. I don't have any control over the application, so I am trying > to do > what I can on the database/OS side. (IDS 11.50.FC7GE) > > > To: ids@iiug.org > > From: Everett.Mills@nationalbeef.com > > Subject: RE: Logical log size [33448] > > Date: Mon, 28 Jul 2014 12:19:22 -0400 > > > > My rule of thumb is total log space is 5 times the average log space > used in > a > > day, that way in the event of a four day weekend (say your company calls > off > > on Christmas Eve as well as Christmas Day, and Christmas is on a Friday). > That > > way you can have all of your transactions for the four day weekend and > some > > padding for emergencies. > > > > --EEM > > > > > -----Original Message----- > > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > > > Art Kagel > > > Sent: Monday, July 28, 2014 11:10 AM > > > To: ids@iiug.org > > > Subject: Re: Logical log size [33446] > > > > > > Marcus's review is a good one. I would just add that the total size and > > > number of logs should be sufficient that if the log backups start to > > > fail over a holiday weekend, everything can keep running without you > > > having to interrupt your days off to go in and force a log backup or > > > fix the problem. > > > So, you want to have at least 4 days of logs online. That's the rule of > > > thumb I use for total log space. > > > > > > For single log sizing, it is, as Marcus pointed out, how much data can > > > you afford to lose if the hardware catastrophically fails just before > > > the current log fills, backs up, and is copied to a safe location. > > > > > > Art > > > > > > Art S. Kagel, Principal Consultant > > > ASK Database Management > > > > > > Blog: http://informix-myview.blogspot.com/ > > > > > > Disclaimer: Please keep in mind that my own opinions are my own > > > opinions and do not reflect on the IIUG, nor any other organization > > > with which I am associated either explicitly, implicitly, or by > > > inference. Neither do those opinions reflect those of other individuals > > > affiliated with any entity with which I am affiliated nor those of the > > > entities themselves. > > > > > > On Mon, Jul 28, 2014 at 11:34 AM, LARRY SORENSEN <lsorensen25@msn.com> > > > wrote: > > > > > > > What is the downside to increasing the logical log size on IDS > > > > 11.50.FC7 running on Solaris? I have been receiving occasional > > > > messages telling me that my log size may be too small. This occurs > > > > during large batch processing operations. We are also running HDR. > > > > > > > > Are there any other implications to increasing the size of the > > > logical > > > > logs that I need to look at? > > > > > > > > Thanks you. > > > > > > > > Larry > > > > > > > > > > > > > > > > > > > *********************************************************************** > > > ******** > > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > > > > > > > > --001a11347f468e4a7204ff43279d > > > > > > > > > *********************************************************************** > > > ******** > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > > ******************************************************************************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a1133642056537804ff441e00
Not transaction by transaction exactly, but log buffer by log buffer is transferred. If all of your databases are unbuffered logging that amounts to the same thing, almost, since a buffer can contain data from multiple transactions. Art Art S. Kagel, Principal Consultant ASK Database Management Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Mon, Jul 28, 2014 at 1:04 PM, Marcus Haarmann <marcus.haarmann@midoco.de> wrote: > Hi, > > HDR is not transferring logfiles in normal mode, only for recovery (when > the > servers are out of sync). > When you have an active/synchronized secondary, transfer occurs for each > finished transaction, > not depending on the log size. I don't thing the log size will modify that > behaviour in a way that > you can transfer more/less data. > > Marcus Haarmann > > ----- Ursprüngliche Mail ----- > > Von: "LARRY SORENSEN" <lsorensen25@msn.com> > An: ids@iiug.org > Gesendet: Montag, 28. Juli 2014 18:28:06 > Betreff: RE: Logical log size [33449] > > Thank you all for your advice. > > Another part of the question is: > > Currently, we have plenty of logical logs to last several days. My question > is, during large batching, we fill up a logical log every 20-30 seconds or > so. > The batches seem to be taking longer than they should. > > 1 - I was wondering how much of a factor that running a remote HDR > secondary > setup is. > > 2- Is it very possible that having larger logical logs might speed up the > batch processing because logs aren't filling up so quickly and flooding the > pipe to the secondary? The connection pipe between servers is ok for daily > operations; however, it is not ideal. > > > To: ids@iiug.org > > From: Everett.Mills@nationalbeef.com > > Subject: RE: Logical log size [33448] > > Date: Mon, 28 Jul 2014 12:19:22 -0400 > > > > My rule of thumb is total log space is 5 times the average log space > used in > a > > day, that way in the event of a four day weekend (say your company calls > off > > on Christmas Eve as well as Christmas Day, and Christmas is on a Friday). > That > > way you can have all of your transactions for the four day weekend and > some > > padding for emergencies. > > > > --EEM > > > > > -----Original Message----- > > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > > > Art Kagel > > > Sent: Monday, July 28, 2014 11:10 AM > > > To: ids@iiug.org > > > Subject: Re: Logical log size [33446] > > > > > > Marcus's review is a good one. I would just add that the total size and > > > number of logs should be sufficient that if the log backups start to > > > fail over a holiday weekend, everything can keep running without you > > > having to interrupt your days off to go in and force a log backup or > > > fix the problem. > > > So, you want to have at least 4 days of logs online. That's the rule of > > > thumb I use for total log space. > > > > > > For single log sizing, it is, as Marcus pointed out, how much data can > > > you afford to lose if the hardware catastrophically fails just before > > > the current log fills, backs up, and is copied to a safe location. > > > > > > Art > > > > > > Art S. Kagel, Principal Consultant > > > ASK Database Management > > > > > > Blog: http://informix-myview.blogspot.com/ > > > > > > Disclaimer: Please keep in mind that my own opinions are my own > > > opinions and do not reflect on the IIUG, nor any other organization > > > with which I am associated either explicitly, implicitly, or by > > > inference. Neither do those opinions reflect those of other individuals > > > affiliated with any entity with which I am affiliated nor those of the > > > entities themselves. > > > > > > On Mon, Jul 28, 2014 at 11:34 AM, LARRY SORENSEN <lsorensen25@msn.com> > > > wrote: > > > > > > > What is the downside to increasing the logical log size on IDS > > > > 11.50.FC7 running on Solaris? I have been receiving occasional > > > > messages telling me that my log size may be too small. This occurs > > > > during large batch processing operations. We are also running HDR. > > > > > > > > Are there any other implications to increasing the size of the > > > logical > > > > logs that I need to look at? > > > > > > > > Thanks you. > > > > > > > > Larry > > > > > > > > > > > > > > > > > > > *********************************************************************** > > > ******** > > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > > > > > > > > --001a11347f468e4a7204ff43279d > > > > > > > > > *********************************************************************** > > > ******** > > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > > > ******************************************************************************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --047d7b87461abaeb7704ff44222c
Larry,
You mentioned remote HDR, are you running sync or a-sync mode?
If your primary shows Logical Log buffer waits "G" in onstat -u then you
may want to increase the OFF_RECVRY_THREADS on the secondary to 3 or 4
times the number of CPU vp on the primary.
Your will also want to consider increasing the size of LOGBUFF on both
systems ( which will increase the size of the HDR buffers reducing the
waits )
Other parameters in that arena are: LOG_STAGING_DIR and DELAY_APPLY
The size of the Logical Logs will not have any significant impact on the
throughput during peak times ( within reason ) size them by the advise
already given.
The message about Logical logs too small appears to be triggered by the
duration a log is opened and will trigger the message the first time a
duration hits below 30 seconds. We see that happen when an archive
completes just after a log fills. The log change associated with the
archive complete will trigger this message in the online log if the log was
opened less then 30 seconds before the switch caused by the archive finish.
As for the 40-50% utilized during peak load - check to see if you have more
CPU then CPU vp ( might be the DB can not run the machine at 100% because
you are limited too far on the num of CPU vp, check for ). Also look into
the possibility that the SQL that is running during peak processing is not
efficient enough to max out the machine ( too many sequential scans that
could be using an index to speed performance) .
It is a little hard to attempt to diagnose possible issues with as little
information as you can nicely craft in an E-mail, the idea above should
give you items to possible check, but don't be surprised if none of them
are all that helpful.
George.
From: "LARRY SORENSEN" <lsorensen25@msn.com>
To: ids@iiug.org,
Date: 07/28/2014 12:03 PM
Subject: RE: Logical log size [33451]
Sent by: ids-bounces@iiug.org
Another piece of information is, at it's busiest during batch processing,
the
server is only at about 40%-50% utilized.
I am just looking for something to cut the processing from 5-6 hours down
to
2-3 hours. I don't have any control over the application, so I am trying to
do
what I can on the database/OS side. (IDS 11.50.FC7GE)
> To: ids@iiug.org
> From: Everett.Mills@nationalbeef.com
> Subject: RE: Logical log size [33448]
> Date: Mon, 28 Jul 2014 12:19:22 -0400
>
> My rule of thumb is total log space is 5 times the average log space used
in
a
> day, that way in the event of a four day weekend (say your company calls
off
> on Christmas Eve as well as Christmas Day, and Christmas is on a Friday).
That
> way you can have all of your transactions for the four day weekend and
some
> padding for emergencies.
>
> --EEM
>
> > -----Original Message-----
> > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of
> > Art Kagel
> > Sent: Monday, July 28, 2014 11:10 AM
> > To: ids@iiug.org
> > Subject: Re: Logical log size [33446]
> >
> > Marcus's review is a good one. I would just add that the total size and
> > number of logs should be sufficient that if the log backups start to
> > fail over a holiday weekend, everything can keep running without you
> > having to interrupt your days off to go in and force a log backup or
> > fix the problem.
> > So, you want to have at least 4 days of logs online. That's the rule of
> > thumb I use for total log space.
> >
> > For single log sizing, it is, as Marcus pointed out, how much data can
> > you afford to lose if the hardware catastrophically fails just before
> > the current log fills, backs up, and is copied to a safe location.
> >
> > Art
> >
> > Art S. Kagel, Principal Consultant
> > ASK Database Management
> >
> > Blog: http://informix-myview.blogspot.com/
> >
> > Disclaimer: Please keep in mind that my own opinions are my own
> > opinions and do not reflect on the IIUG, nor any other organization
> > with which I am associated either explicitly, implicitly, or by
> > inference. Neither do those opinions reflect those of other individuals
> > affiliated with any entity with which I am affiliated nor those of the
> > entities themselves.
> >
> > On Mon, Jul 28, 2014 at 11:34 AM, LARRY SORENSEN <lsorensen25@msn.com>
> > wrote:
> >
> > > What is the downside to increasing the logical log size on IDS
> > > 11.50.FC7 running on Solaris? I have been receiving occasional
> > > messages telling me that my log size may be too small. This occurs
> > > during large batch processing operations. We are also running HDR.
> > >
> > > Are there any other implications to increasing the size of the
> > logical
> > > logs that I need to look at?
> > >
> > > Thanks you.
> > >
> > > Larry
> > >
> > >
> > >
> > >
> > ***********************************************************************
> > ********
> > > Forum Note: Use "Reply" to post a response in the discussion forum.
> > >
> > >
> >
> > --001a11347f468e4a7204ff43279d
> >
> >
> > ***********************************************************************
> > ********
> > Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
*******************************************************************************
Forum Note: Use "Reply" to post a response in the discussion forum.
Thank you all for the helpful information. I have some extra things to
consider as well now.
> To: ids@iiug.org
> From: George_Palmer@aotx.uscourts.gov
> Subject: RE: Logical log size [33456]
> Date: Mon, 28 Jul 2014 13:34:56 -0400
>
> Larry,
>
> You mentioned remote HDR, are you running sync or a-sync mode?
> If your primary shows Logical Log buffer waits "G" in onstat -u then you
> may want to increase the OFF_RECVRY_THREADS on the secondary to 3 or 4
> times the number of CPU vp on the primary.
> Your will also want to consider increasing the size of LOGBUFF on both
> systems ( which will increase the size of the HDR buffers reducing the
> waits )
> Other parameters in that arena are: LOG_STAGING_DIR and DELAY_APPLY
> The size of the Logical Logs will not have any significant impact on the
> throughput during peak times ( within reason ) size them by the advise
> already given.
> The message about Logical logs too small appears to be triggered by the
> duration a log is opened and will trigger the message the first time a
> duration hits below 30 seconds. We see that happen when an archive
> completes just after a log fills. The log change associated with the
> archive complete will trigger this message in the online log if the log was
> opened less then 30 seconds before the switch caused by the archive finish.
>
> As for the 40-50% utilized during peak load - check to see if you have more
> CPU then CPU vp ( might be the DB can not run the machine at 100% because
> you are limited too far on the num of CPU vp, check for ). Also look into
> the possibility that the SQL that is running during peak processing is not
> efficient enough to max out the machine ( too many sequential scans that
> could be using an index to speed performance) .
>
> It is a little hard to attempt to diagnose possible issues with as little
> information as you can nicely craft in an E-mail, the idea above should
> give you items to possible check, but don't be surprised if none of them
> are all that helpful.
>
> George.
>
> From: "LARRY SORENSEN" <lsorensen25@msn.com>
> To: ids@iiug.org,
> Date: 07/28/2014 12:03 PM
> Subject: RE: Logical log size [33451]
> Sent by: ids-bounces@iiug.org
>
> Another piece of information is, at it's busiest during batch processing,
> the
> server is only at about 40%-50% utilized.
>
> I am just looking for something to cut the processing from 5-6 hours down
> to
> 2-3 hours. I don't have any control over the application, so I am trying to
> do
> what I can on the database/OS side. (IDS 11.50.FC7GE)
>
> > To: ids@iiug.org
> > From: Everett.Mills@nationalbeef.com
> > Subject: RE: Logical log size [33448]
> > Date: Mon, 28 Jul 2014 12:19:22 -0400
> >
> > My rule of thumb is total log space is 5 times the average log space used
> in
> a
> > day, that way in the event of a four day weekend (say your company calls
> off
> > on Christmas Eve as well as Christmas Day, and Christmas is on a Friday).
>
> That
> > way you can have all of your transactions for the four day weekend and
> some
> > padding for emergencies.
> >
> > --EEM
> >
> > > -----Original Message-----
> > > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of
> > > Art Kagel
> > > Sent: Monday, July 28, 2014 11:10 AM
> > > To: ids@iiug.org
> > > Subject: Re: Logical log size [33446]
> > >
> > > Marcus's review is a good one. I would just add that the total size and
>
> > > number of logs should be sufficient that if the log backups start to
> > > fail over a holiday weekend, everything can keep running without you
> > > having to interrupt your days off to go in and force a log backup or
> > > fix the problem.
> > > So, you want to have at least 4 days of logs online. That's the rule of
>
> > > thumb I use for total log space.
> > >
> > > For single log sizing, it is, as Marcus pointed out, how much data can
> > > you afford to lose if the hardware catastrophically fails just before
> > > the current log fills, backs up, and is copied to a safe location.
> > >
> > > Art
> > >
> > > Art S. Kagel, Principal Consultant
> > > ASK Database Management
> > >
> > > Blog: http://informix-myview.blogspot.com/
> > >
> > > Disclaimer: Please keep in mind that my own opinions are my own
> > > opinions and do not reflect on the IIUG, nor any other organization
> > > with which I am associated either explicitly, implicitly, or by
> > > inference. Neither do those opinions reflect those of other individuals
>
> > > affiliated with any entity with which I am affiliated nor those of the
> > > entities themselves.
> > >
> > > On Mon, Jul 28, 2014 at 11:34 AM, LARRY SORENSEN <lsorensen25@msn.com>
> > > wrote:
> > >
> > > > What is the downside to increasing the logical log size on IDS
> > > > 11.50.FC7 running on Solaris? I have been receiving occasional
> > > > messages telling me that my log size may be too small. This occurs
> > > > during large batch processing operations. We are also running HDR.
> > > >
> > > > Are there any other implications to increasing the size of the
> > > logical
> > > > logs that I need to look at?
> > > >
> > > > Thanks you.
> > > >
> > > > Larry
> > > >
> > > >
> > > >
> > > >
> > > ***********************************************************************
>
> > > ********
> > > > Forum Note: Use "Reply" to post a response in the discussion forum.
> > > >
> > > >
> > >
> > > --001a11347f468e4a7204ff43279d
> > >
> > >
> > > ***********************************************************************
>
> > > ********
> > > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
> >
>
>
*******************************************************************************
>
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
>
>
>
*******************************************************************************
>
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
Larry,
regarding the log log duration question, I have seen a cool way to garantee
that a logical log will not remain too long (you have to define what "too
long" means for you) without being backed up.
Sincerly, this is a a smart way of doing it.
Check here:
http://planetids.com/content/ensuring-logical-logs-do-not-get-old
For your batch period consuming logs very fast(which is a problem
performancewise), why not trying the following (has to be scripted obviously).
This works if your batches are exectuted out of OLTP/Office hours, else forget.
Synoptic:
at 'opening of batch window'
1) create big logical logs ( up to 2 gb ), and remember their log #
those logs will have to be big and numerous enough to stand all the batch
period
2) skip logs to position the current log to the first of those big logs
by running successively onmode -l
At 'close of batch window'
1) skip to one "small" logical log, ensure no big log is needed anymore.
2) drop the big logical logs ( you remembered them in your creation scripts
I would test those scripts before :-)
The thing is: is there any unknown limitation of the number of times you can
create and delete new logical logs, so that you would received an error
message because you cannot create anymore logs, years after initiating this
process for instance.
I don't think there is such a limitation. Anyone?
Eric