Fw: Optimizer
On AIX 6.1 with IDS 11.70.FC3, a 10-table MicroStrategy report query slowed from ~1 minute to ~4 minutes after the optimizer changed its table join order; adding an ORDERED directive restored the fast plan, but the reporting tool can't embed directives. Fernando Nunes suggested checking AUTO_STAT_MODE/stale distributions, noted the optimizer must guess selectivity for the unknown result of the weekid() function subquery (behaviour changed in recent fixpacks, worth trying 11.70.FC4), and suggested external directives or expression tricks to avoid an index. Art Kagel suggested SET OPTIMIZATION LOW, since 10 tables means millions of candidate plans. The poster couldn't get external directives to take effect; no resolution is recorded.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Aix 6.1 IDS 11.70.fc3 The below query was executing ok, and then recently it started taking a lot longer... In looking at the query plans, the order of the tables changed a few weeks ago... Amount of data and stats in the tables haven't changed dramatically ... The question I have, is ... Is the optimizer smart enough to execute the (select weekid(today, -1) from systables where tabid = 1 ... I can duplicate if I take that select out and hard code the fiscal week id .. runs about a minute .. then runs about 4 minutes. The query plan changes .. If I do run the query with the directive ... then I can consistently get it to run around a minute .. The problem is that this is a Micro Strategies report, so that optimizer directive can't be included .... Any help would be appreciated ... Peter Logan Senior Database Administrator Phone: 616/878-8309 ----- Forwarded by Peter Logan/Corporate/Spartan on 03/28/2012 03:53 PM ----- From: plisad0 (spartan02) <plisad0@spartanstores.com> To: Peter_Logan@spartanstores.com Date: 03/28/2012 03:52 PM Subject: Optimizer -- select --+ORDERED select a18.mdse_grp_key mdse_grp_key, a12.fiscal_week_id fiscal_week_id, a13.catgy_manager_key catgy_manager_key, (sum(a11.total_sales_amt) * 0.001) WJXBFS1, sum(a11.ext_profit_amt) WJXBFS2, sum(a11.total_sales_amt) WJXBFS3, sum(a11.ext_profit_amt) WJXBFS4 from week_to_year_lkp a12, channel a16, chain a15, line a14, efin_wk_line_cls a11, mdse_class_manager a13, mdse_class a17, mdse_category a18, mdse_group a19, department a110, business_dept a111 where a11.fiscal_week_id = a12.fiscal_wty_id and a11.mdse_class_key = a13.mdse_class_key and a11.sales_line_id = a14.sales_line_id and a14.sales_chain_id = a15.sales_chain_id and a15.sales_channel_id = a16.sales_channel_id and a11.mdse_class_key = a17.mdse_class_key and a17.mdse_catgy_key = a18.mdse_catgy_key and a18.mdse_grp_key = a19.mdse_grp_key and a19.dept_key = a110.dept_key and a110.bus_dept_key = a111.bus_dept_key and (a16.enterprise_id in (18) and a13.catgy_manager_key in (70) and a110.dept_grp_key not in (525, 9) and a14.format_type_id in ('SUPERMKT ') and a12.fiscal_week_id in (201253) and a111.dept_grp_type_key in (1)) group by a18.mdse_grp_key, a12.fiscal_week_id, a13.catgy_manager_key into temp ZZMD03 with no log
I'm afraid I didn't understand your point correctly. You mention the select from systables but I didn't understand how it's relevant for the problem. Are you saying that query presented works fine but if you use the select from systables instead of: and a12.fiscal_week_id in (201253) It will break the query plan? In any case a few warnings and maybe some useful information: - 11.7 alreay includes the AUTO_STAT_MODE parameter. This activates or deactivates the optimization that says to the engine that "if the table doesn't change too much don't bother to update it's statistics". I had a recent situation where the default threshold was causing some stalled statistics and it made a difference on the query plans. And the probability that this happens with queries involving time/dates is higher than with other types of filters (because these fields many times tend to be populated with constantly increasing values, so statistically something that makes the filter "column > today - some_time" look good will in time lead to bad choices) - You can activate optimizer directives for any query outside the applications by using the External Directives feature. I will not go into details, because it could take a while, but if you're not aware of this and need further information, please tell. In any case I'm against the principle of using directives to solve query plans problems... after all we have an optimizer... we're not Oracle :) So, my advise: - Verify that the distributions are really up to date (disable AUTO_STAT_MODE) and run them (it's dynamic and I believe there's a way to control it inside the session (either with SET ENVIRONMENT or with a environment variable) - If you never used the external optimizer directives, please investigate them. Sometimes you can get away with them to quickly solve a performance problem. But my advise is to always investigate this kind of problems and find a definitive solution - You could of course re-post your problem in a manner I can understand, or wait until someone smarter to answer you :) Regards. On Wed, Mar 28, 2012 at 9:00 PM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > Aix 6.1 > IDS 11.70.fc3 > > The below query was executing ok, and then recently it started taking a > lot longer... In looking at the query plans, the order of the tables > changed a few weeks ago... Amount of data and stats in the tables haven't > changed dramatically ... > > The question I have, is ... Is the optimizer smart enough to execute the > (select weekid(today, -1) from systables where tabid = 1 ... I can > duplicate if I take that select out and hard code the fiscal week id .. > runs about a minute .. then runs about 4 minutes. The query plan changes > ... If I do run the query with the directive ... then I can consistently > get it to run around a minute .. The problem is that this is a Micro > Strategies report, so that optimizer directive can't be included .... > Any help would be appreciated ... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > ----- Forwarded by Peter Logan/Corporate/Spartan on 03/28/2012 03:53 PM > ----- > > From: plisad0 (spartan02) <plisad0@spartanstores.com> > To: Peter_Logan@spartanstores.com > Date: 03/28/2012 03:52 PM > Subject: Optimizer > > -- select --+ORDERED > select > a18.mdse_grp_key mdse_grp_key, > > a12.fiscal_week_id fiscal_week_id, > > a13.catgy_manager_key catgy_manager_key, > > (sum(a11.total_sales_amt) * 0.001) WJXBFS1, > > sum(a11.ext_profit_amt) WJXBFS2, > > sum(a11.total_sales_amt) WJXBFS3, > > sum(a11.ext_profit_amt) WJXBFS4 > from > > week_to_year_lkp a12, > > channel a16, > > chain a15, > > line a14, > efin_wk_line_cls a11, > > mdse_class_manager a13, > > mdse_class a17, > > mdse_category a18, > > mdse_group a19, > > department a110, > > business_dept a111 > where a11.fiscal_week_id = a12.fiscal_wty_id and > > a11.mdse_class_key = a13.mdse_class_key and > > a11.sales_line_id = a14.sales_line_id and > > a14.sales_chain_id = a15.sales_chain_id and > > a15.sales_channel_id = a16.sales_channel_id and > > a11.mdse_class_key = a17.mdse_class_key and > > a17.mdse_catgy_key = a18.mdse_catgy_key and > > a18.mdse_grp_key = a19.mdse_grp_key and > > a19.dept_key = a110.dept_key and > > a110.bus_dept_key = a111.bus_dept_key > > and (a16.enterprise_id in (18) > > and a13.catgy_manager_key in (70) > > and a110.dept_grp_key not in (525, 9) > > and a14.format_type_id in ('SUPERMKT ') > > and a12.fiscal_week_id in (201253) > > and a111.dept_grp_type_key in (1)) > group by a18.mdse_grp_key, > > a12.fiscal_week_id, > > a13.catgy_manager_key > into temp ZZMD03 with no log > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --0023544710d08e6fef04bc540bf4
Thanks for the reply ... Yes, the auto stats updating has been disabled .. and I am aware of the external optimizer directives ... Let me try to sum this up a little differently ... when the date was say 9/15/2011 .... one query plan was followed ... the one that works ... now that the date is 3/28/2012 ... the optimizer is choosing a different path .. so, what it seems to me is that the optimizer must be evaluating that part of the query .. and then figuring out the query plan from there ... I could understand that if this was a constant, but this isn't ... the select weekid(today, -1) from systables where tabid = 1 is a procedure that brings back the fiscal week id .. ie 201252 ... does that make a little ore sense ... I'm trying to figure out if the optimizer is making that evaluation .. then coming up with the plan.... and if so, how to keep it from doing that .... Peter Logan Senior Database Administrator Phone: 616/878-8309 From: "Fernando Nunes" <domusonline@gmail.com> To: ids@iiug.org Date: 03/28/2012 05:11 PM Subject: Re: Fw: Optimizer [26592] Sent by: ids-bounces@iiug.org I'm afraid I didn't understand your point correctly. You mention the select from systables but I didn't understand how it's relevant for the problem. Are you saying that query presented works fine but if you use the select from systables instead of: and a12.fiscal_week_id in (201253) It will break the query plan? In any case a few warnings and maybe some useful information: - 11.7 alreay includes the AUTO_STAT_MODE parameter. This activates or deactivates the optimization that says to the engine that "if the table doesn't change too much don't bother to update it's statistics". I had a recent situation where the default threshold was causing some stalled statistics and it made a difference on the query plans. And the probability that this happens with queries involving time/dates is higher than with other types of filters (because these fields many times tend to be populated with constantly increasing values, so statistically something that makes the filter "column > today - some_time" look good will in time lead to bad choices) - You can activate optimizer directives for any query outside the applications by using the External Directives feature. I will not go into details, because it could take a while, but if you're not aware of this and need further information, please tell. In any case I'm against the principle of using directives to solve query plans problems... after all we have an optimizer... we're not Oracle :) So, my advise: - Verify that the distributions are really up to date (disable AUTO_STAT_MODE) and run them (it's dynamic and I believe there's a way to control it inside the session (either with SET ENVIRONMENT or with a environment variable) - If you never used the external optimizer directives, please investigate them. Sometimes you can get away with them to quickly solve a performance problem. But my advise is to always investigate this kind of problems and find a definitive solution - You could of course re-post your problem in a manner I can understand, or wait until someone smarter to answer you :) Regards. On Wed, Mar 28, 2012 at 9:00 PM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > Aix 6.1 > IDS 11.70.fc3 > > The below query was executing ok, and then recently it started taking a > lot longer... In looking at the query plans, the order of the tables > changed a few weeks ago... Amount of data and stats in the tables haven't > changed dramatically ... > > The question I have, is ... Is the optimizer smart enough to execute the > (select weekid(today, -1) from systables where tabid = 1 ... I can > duplicate if I take that select out and hard code the fiscal week id .. > runs about a minute .. then runs about 4 minutes. The query plan changes > ... If I do run the query with the directive ... then I can consistently > get it to run around a minute .. The problem is that this is a Micro > Strategies report, so that optimizer directive can't be included .... > Any help would be appreciated ... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > ----- Forwarded by Peter Logan/Corporate/Spartan on 03/28/2012 03:53 PM > ----- > > From: plisad0 (spartan02) <plisad0@spartanstores.com> > To: Peter_Logan@spartanstores.com > Date: 03/28/2012 03:52 PM > Subject: Optimizer > > -- select --+ORDERED > select > a18.mdse_grp_key mdse_grp_key, > > a12.fiscal_week_id fiscal_week_id, > > a13.catgy_manager_key catgy_manager_key, > > (sum(a11.total_sales_amt) * 0.001) WJXBFS1, > > sum(a11.ext_profit_amt) WJXBFS2, > > sum(a11.total_sales_amt) WJXBFS3, > > sum(a11.ext_profit_amt) WJXBFS4 > from > > week_to_year_lkp a12, > > channel a16, > > chain a15, > > line a14, > efin_wk_line_cls a11, > > mdse_class_manager a13, > > mdse_class a17, > > mdse_category a18, > > mdse_group a19, > > department a110, > > business_dept a111 > where a11.fiscal_week_id = a12.fiscal_wty_id and > > a11.mdse_class_key = a13.mdse_class_key and > > a11.sales_line_id = a14.sales_line_id and > > a14.sales_chain_id = a15.sales_chain_id and > > a15.sales_channel_id = a16.sales_channel_id and > > a11.mdse_class_key = a17.mdse_class_key and > > a17.mdse_catgy_key = a18.mdse_catgy_key and > > a18.mdse_grp_key = a19.mdse_grp_key and > > a19.dept_key = a110.dept_key and > > a110.bus_dept_key = a111.bus_dept_key > > and (a16.enterprise_id in (18) > > and a13.catgy_manager_key in (70) > > and a110.dept_grp_key not in (525, 9) > > and a14.format_type_id in ('SUPERMKT ') > > and a12.fiscal_week_id in (201253) > > and a111.dept_grp_type_key in (1)) > group by a18.mdse_grp_key, > > a12.fiscal_week_id, > > a13.catgy_manager_key > into temp ZZMD03 with no log > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --0023544710d08e6fef04bc540bf4 ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.
Hmmm.... Yes, I think I can understand the point, although tables schema would help. I believe the problem is a well known issue (notice I use "issue" instead of "bug"). And recent versions have been seeing changes in that area. Would I be correct to assume the bad query plan is using a date/datetime index? This could be one source of the problem, but in your case it's probably another issue. You're asking the optimizer to calculate a query plan based on the unknown result of a user function. As such the optimizer must consider/estimate a selectivity. Too low will make that index non attractive, and too high makes it too attractive. And the dramatic issue is that it has no way to make a good estimate. I believe that recent versions (11.50.FC9 and maybe 11.70.FC2/3) may have change the way this selectivity is calculated. This solved many customer's problems and probably caused problems to many others... I'm not in developing but I believe 11.70.FC4 may have changed this. And eventually some improvements may appear in the future. Would it be too hard for you to try that query on 11.70.FC4? Of course, if you had the chance to avoid the use of the function and give it the value the problem should not happen. If you're sure that the query should NEVER use that index (on the column you're comparing with the function result), you can use some sort of operation on it. For example: my_column + 0 UNITS DAY = ( SELECT WEEKID(...) FROM systables WHERE tabid = 0) This will avoid the use of any index on that column and will not change the result set... hmmmm... But again, if that's a query generated from a tool this may not be feasible. P.S.: Above I avoided using the term "bug" because this is a nasty situation... We must consider some value for selectivity, but we have no idea about the possible or probable result of the function. So basically we're blind... I can't imagine a good solution (that will fit all the scenarios) for this. Regards. On Thu, Mar 29, 2012 at 1:27 PM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > Thanks for the reply ... Yes, the auto stats updating has been disabled > ... and I am aware of the external optimizer directives ... > > Let me try to sum this up a little differently ... when the date was say > 9/15/2011 .... one query plan was followed ... the one that works ... now > that the date is 3/28/2012 ... the optimizer is choosing a different path > ... so, what it seems to me is that the optimizer must be evaluating that > part of the query .. and then figuring out the query plan from there ... I > could understand that if this was a constant, but this isn't ... the > select weekid(today, -1) from systables where tabid = 1 is a procedure > that brings back the fiscal week id .. ie 201252 ... does that make a > little ore sense ... I'm trying to figure out if the optimizer is making > that evaluation .. then coming up with the plan.... and if so, how to keep > it from doing that .... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > From: "Fernando Nunes" <domusonline@gmail.com> > To: ids@iiug.org > Date: 03/28/2012 05:11 PM > Subject: Re: Fw: Optimizer [26592] > Sent by: ids-bounces@iiug.org > > I'm afraid I didn't understand your point correctly. You mention the > select > from systables but I didn't understand how it's relevant for the problem. > Are you saying that query presented works fine but if you use the select > from systables instead of: > > and a12.fiscal_week_id in (201253) > > It will break the query plan? > > In any case a few warnings and maybe some useful information: > > - 11.7 alreay includes the AUTO_STAT_MODE parameter. This activates or > deactivates the optimization that says to the engine that "if the table > doesn't change too much don't bother to update it's statistics". I had a > recent situation where the default threshold was causing some stalled > statistics and it made a difference on the query plans. And the > probability > that this happens with queries involving time/dates is higher than with > other types of filters (because these fields many times tend to be > populated with constantly increasing values, so statistically something > that makes the filter "column > today - some_time" look good will in time > lead to bad choices) > > - You can activate optimizer directives for any query outside the > applications by using the External Directives feature. I will not go into > details, because it could take a while, but if you're not aware of this > and > need further information, please tell. In any case I'm against the > principle of using directives to solve query plans problems... after all > we > have an optimizer... we're not Oracle :) > > So, my advise: > - Verify that the distributions are really up to date (disable > AUTO_STAT_MODE) and run them (it's dynamic and I believe there's a way to > control it inside the session (either with SET ENVIRONMENT or with a > environment variable) > - If you never used the external optimizer directives, please investigate > them. Sometimes you can get away with them to quickly solve a performance > problem. But my advise is to always investigate this kind of problems and > find a definitive solution > - You could of course re-post your problem in a manner I can understand, > or > wait until someone smarter to answer you :) > > Regards. > > On Wed, Mar 28, 2012 at 9:00 PM, Peter_Logan@spartanstores.com < > Peter_Logan@spartanstores.com> wrote: > > > Aix 6.1 > > IDS 11.70.fc3 > > > > The below query was executing ok, and then recently it started taking a > > lot longer... In looking at the query plans, the order of the tables > > changed a few weeks ago... Amount of data and stats in the tables > haven't > > changed dramatically ... > > > > The question I have, is ... Is the optimizer smart enough to execute the > > > (select weekid(today, -1) from systables where tabid = 1 ... I can > > duplicate if I take that select out and hard code the fiscal week id .. > > runs about a minute .. then runs about 4 minutes. The query plan changes > > > ... If I do run the query with the directive ... then I can consistently > > > get it to run around a minute .. The problem is that this is a Micro > > Strategies report, so that optimizer directive can't be included .... > > Any help would be appreciated ... > > > > Peter Logan > > Senior Database Administrator > > Phone: 616/878-8309 > > ----- Forwarded by Peter Logan/Corporate/Spartan on 03/28/2012 03:53 PM > > ----- > > > > From: plisad0 (spartan02) <plisad0@spartanstores.com> > > To: Peter_Logan@spartanstores.com > > Date: 03/28/2012 03:52 PM > > Subject: Optimizer > > > > -- select --+ORDERED > > select > > a18.mdse_grp_key mdse_grp_key, > > > > a12.fiscal_week_id fiscal_week_id, > > > > a13.catgy_manager_key catgy_manager_key, > > > > (sum(a11.total_sales_amt) * 0.001) WJXBFS1, > > > > sum(a11.ext_profit_amt) WJXBFS2, > > > > sum(a11.total_sales_amt) WJXBFS3, > > > > sum(a11.e
Ok .. here is the really strange thing ... If I do away with the function ... and put in place the fiscal week id ... then up until week 36 one query plan is used ... and then from week 37 on, a different query plan is used ... The first plan runs in a minute...the second several minutes ... when I put the directive on the query and force it to use the order of the tables that was chosen by the optimizer it always runs in a minute ... I'm trying to see if an external directive will work for this one .. just to give them that as a choice .. but I can't seem to get that working ... I can save the external directive, but when I execute the sql, the explain doesn't show the directive being used ... and I'm using the exact sql to create the directive as well as execute the sql ... wondering if the function in the sql is messing that up ... Peter Logan Senior Database Administrator Phone: 616/878-8309 From: "Fernando Nunes" <domusonline@gmail.com> To: ids@iiug.org Date: 03/29/2012 09:52 AM Subject: Re: Fw: Optimizer [26597] Sent by: ids-bounces@iiug.org Hmmm.... Yes, I think I can understand the point, although tables schema would help. I believe the problem is a well known issue (notice I use "issue" instead of "bug"). And recent versions have been seeing changes in that area. Would I be correct to assume the bad query plan is using a date/datetime index? This could be one source of the problem, but in your case it's probably another issue. You're asking the optimizer to calculate a query plan based on the unknown result of a user function. As such the optimizer must consider/estimate a selectivity. Too low will make that index non attractive, and too high makes it too attractive. And the dramatic issue is that it has no way to make a good estimate. I believe that recent versions (11.50.FC9 and maybe 11.70.FC2/3) may have change the way this selectivity is calculated. This solved many customer's problems and probably caused problems to many others... I'm not in developing but I believe 11.70.FC4 may have changed this. And eventually some improvements may appear in the future. Would it be too hard for you to try that query on 11.70.FC4? Of course, if you had the chance to avoid the use of the function and give it the value the problem should not happen. If you're sure that the query should NEVER use that index (on the column you're comparing with the function result), you can use some sort of operation on it. For example: my_column + 0 UNITS DAY = ( SELECT WEEKID(...) FROM systables WHERE tabid = 0) This will avoid the use of any index on that column and will not change the result set... hmmmm... But again, if that's a query generated from a tool this may not be feasible. P.S.: Above I avoided using the term "bug" because this is a nasty situation... We must consider some value for selectivity, but we have no idea about the possible or probable result of the function. So basically we're blind... I can't imagine a good solution (that will fit all the scenarios) for this. Regards. On Thu, Mar 29, 2012 at 1:27 PM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > Thanks for the reply ... Yes, the auto stats updating has been disabled > ... and I am aware of the external optimizer directives ... > > Let me try to sum this up a little differently ... when the date was say > 9/15/2011 .... one query plan was followed ... the one that works ... now > that the date is 3/28/2012 ... the optimizer is choosing a different path > ... so, what it seems to me is that the optimizer must be evaluating that > part of the query .. and then figuring out the query plan from there ... I > could understand that if this was a constant, but this isn't ... the > select weekid(today, -1) from systables where tabid = 1 is a procedure > that brings back the fiscal week id .. ie 201252 ... does that make a > little ore sense ... I'm trying to figure out if the optimizer is making > that evaluation .. then coming up with the plan.... and if so, how to keep > it from doing that .... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > From: "Fernando Nunes" <domusonline@gmail.com> > To: ids@iiug.org > Date: 03/28/2012 05:11 PM > Subject: Re: Fw: Optimizer [26592] > Sent by: ids-bounces@iiug.org > > I'm afraid I didn't understand your point correctly. You mention the > select > from systables but I didn't understand how it's relevant for the problem. > Are you saying that query presented works fine but if you use the select > from systables instead of: > > and a12.fiscal_week_id in (201253) > > It will break the query plan? > > In any case a few warnings and maybe some useful information: > > - 11.7 alreay includes the AUTO_STAT_MODE parameter. This activates or > deactivates the optimization that says to the engine that "if the table > doesn't change too much don't bother to update it's statistics". I had a > recent situation where the default threshold was causing some stalled > statistics and it made a difference on the query plans. And the > probability > that this happens with queries involving time/dates is higher than with > other types of filters (because these fields many times tend to be > populated with constantly increasing values, so statistically something > that makes the filter "column > today - some_time" look good will in time > lead to bad choices) > > - You can activate optimizer directives for any query outside the > applications by using the External Directives feature. I will not go into > details, because it could take a while, but if you're not aware of this > and > need further information, please tell. In any case I'm against the > principle of using directives to solve query plans problems... after all > we > have an optimizer... we're not Oracle :) > > So, my advise: > - Verify that the distributions are really up to date (disable > AUTO_STAT_MODE) and run them (it's dynamic and I believe there's a way to > control it inside the session (either with SET ENVIRONMENT or with a > environment variable) > - If you never used the external optimizer directives, please investigate > them. Sometimes you can get away with them to quickly solve a performance > problem. But my advise is to always investigate this kind of problems and > find a definitive solution > - You could of course re-post your problem in a manner I can understand, > or > wait until someone smarter to answer you :) > > Regards. > > On Wed, Mar 28, 2012 at 9:00 PM, Peter_Logan@spartanstores.com < > Peter_Logan@spartanstores.com> wrote: > > > Aix 6.1 > > IDS 11.70.fc3 > > > > The below query was executing ok, and then recently it started taking a > > lot longer... In looking at the query plans, the order of the tables > > changed a few weeks ago... Amount of data and stats in the tables > haven't > > changed dramatically ... > > > > The question I have, is ... Is the optimize
One thing Peter, have you tried running this query under SET OPTIMIZATION LOW? You have 10 tables in there, I'm willing to bet that a significant portion of the runtime is optimizing the query since there are 3,628,800 query plans that have to be examined and have cost calculated under the default HIGH optimization method versus only 54 query plans under LOW optimization. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Mar 29, 2012 at 10:06 AM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > Ok .. here is the really strange thing ... > > If I do away with the function ... and put in place the fiscal week id ... > then up until week 36 one query plan is used ... and then from week 37 on, > a different query plan is used ... The first plan runs in a minute...the > second several minutes ... when I put the directive on the query and > force it to use the order of the tables that was chosen by the optimizer > it always runs in a minute ... > > I'm trying to see if an external directive will work for this one .. just > to give them that as a choice .. but I can't seem to get that working ... > I can save the external directive, but when I execute the sql, the > explain doesn't show the directive being used ... and I'm using the exact > sql to create the directive as well as execute the sql ... wondering if > the function in the sql is messing that up ... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > From: "Fernando Nunes" <domusonline@gmail.com> > To: ids@iiug.org > Date: 03/29/2012 09:52 AM > Subject: Re: Fw: Optimizer [26597] > Sent by: ids-bounces@iiug.org > > Hmmm.... Yes, I think I can understand the point, although tables schema > would help. > I believe the problem is a well known issue (notice I use "issue" instead > of "bug"). And recent versions have been seeing changes in that area. > > Would I be correct to assume the bad query plan is using a date/datetime > index? This could be one source of the problem, but in your case it's > probably another issue. > You're asking the optimizer to calculate a query plan based on the unknown > > result of a user function. As such the optimizer must consider/estimate a > selectivity. Too low will make that index non attractive, and too high > makes it too attractive. And the dramatic issue is that it has no way to > make a good estimate. > > I believe that recent versions (11.50.FC9 and maybe 11.70.FC2/3) may have > change the way this selectivity is calculated. This solved many customer's > > problems and probably caused problems to many others... > > I'm not in developing but I believe 11.70.FC4 may have changed this. And > eventually some improvements may appear in the future. Would it be too > hard > for you to try that query on 11.70.FC4? > > Of course, if you had the chance to avoid the use of the function and give > > it the value the problem should not happen. > If you're sure that the query should NEVER use that index (on the column > you're comparing with the function result), you can use some sort of > operation on it. For example: > > my_column + 0 UNITS DAY = ( SELECT WEEKID(...) FROM systables WHERE tabid > = > 0) > > This will avoid the use of any index on that column and will not change > the > result set... hmmmm... But again, if that's a query generated from a tool > this may not be feasible. > > P.S.: Above I avoided using the term "bug" because this is a nasty > situation... We must consider some value for selectivity, but we have no > idea about the possible or probable result of the function. So basically > we're blind... I can't imagine a good solution (that will fit all the > scenarios) for this. > > Regards. > > On Thu, Mar 29, 2012 at 1:27 PM, Peter_Logan@spartanstores.com < > Peter_Logan@spartanstores.com> wrote: > > > Thanks for the reply ... Yes, the auto stats updating has been disabled > > ... and I am aware of the external optimizer directives ... > > > > Let me try to sum this up a little differently ... when the date was say > > > 9/15/2011 .... one query plan was followed ... the one that works ... > now > > that the date is 3/28/2012 ... the optimizer is choosing a different > path > > ... so, what it seems to me is that the optimizer must be evaluating > that > > part of the query .. and then figuring out the query plan from there ... > I > > could understand that if this was a constant, but this isn't ... the > > select weekid(today, -1) from systables where tabid = 1 is a procedure > > that brings back the fiscal week id .. ie 201252 ... does that make a > > little ore sense ... I'm trying to figure out if the optimizer is making > > > that evaluation .. then coming up with the plan.... and if so, how to > keep > > it from doing that .... > > > > Peter Logan > > Senior Database Administrator > > Phone: 616/878-8309 > > > > From: "Fernando Nunes" <domusonline@gmail.com> > > To: ids@iiug.org > > Date: 03/28/2012 05:11 PM > > Subject: Re: Fw: Optimizer [26592] > > Sent by: ids-bounces@iiug.org > > > > I'm afraid I didn't understand your point correctly. You mention the > > select > > from systables but I didn't understand how it's relevant for the > problem. > > Are you saying that query presented works fine but if you use the select > > > from systables instead of: > > > > and a12.fiscal_week_id in (201253) > > > > It will break the query plan? > > > > In any case a few warnings and maybe some useful information: > > > > - 11.7 alreay includes the AUTO_STAT_MODE parameter. This activates or > > deactivates the optimization that says to the engine that "if the table > > doesn't change too much don't bother to update it's statistics". I had a > > > recent situation where the default threshold was causing some stalled > > statistics and it made a difference on the query plans. And the > > probability > > that this happens with queries involving time/dates is higher than with > > other types of filters (because these fields many times tend to be > > populated with constantly increasing values, so statistically something > > that makes the filter "column > today - some_time" look good will in > time > > lead to bad choices) > > > > - You can activate optimizer directives for any query outside the > > applications by using the External Directives feature. I will not go > into > > details, because it could take a while, but if you're not aware of this > > and > > need further information, please tell. In any case I'm against the > > principle of using directives to solve query plans p
Art, yes .. and that does help this particular query ... the issue though is that this is only one part of a larger set of queries...and the optimization being set low has a negative impact on the entire stream ... and they can't turn that off and back on .... can only set it for the session ... Peter Logan Senior Database Administrator Phone: 616/878-8309 From: "Art Kagel" <art.kagel@gmail.com> To: ids@iiug.org Date: 03/29/2012 10:24 AM Subject: Re: Fw: Optimizer [26600] Sent by: ids-bounces@iiug.org One thing Peter, have you tried running this query under SET OPTIMIZATION LOW? You have 10 tables in there, I'm willing to bet that a significant portion of the runtime is optimizing the query since there are 3,628,800 query plans that have to be examined and have cost calculated under the default HIGH optimization method versus only 54 query plans under LOW optimization. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Mar 29, 2012 at 10:06 AM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > Ok .. here is the really strange thing ... > > If I do away with the function ... and put in place the fiscal week id ... > then up until week 36 one query plan is used ... and then from week 37 on, > a different query plan is used ... The first plan runs in a minute...the > second several minutes ... when I put the directive on the query and > force it to use the order of the tables that was chosen by the optimizer > it always runs in a minute ... > > I'm trying to see if an external directive will work for this one .. just > to give them that as a choice .. but I can't seem to get that working ... > I can save the external directive, but when I execute the sql, the > explain doesn't show the directive being used ... and I'm using the exact > sql to create the directive as well as execute the sql ... wondering if > the function in the sql is messing that up ... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > From: "Fernando Nunes" <domusonline@gmail.com> > To: ids@iiug.org > Date: 03/29/2012 09:52 AM > Subject: Re: Fw: Optimizer [26597] > Sent by: ids-bounces@iiug.org > > Hmmm.... Yes, I think I can understand the point, although tables schema > would help. > I believe the problem is a well known issue (notice I use "issue" instead > of "bug"). And recent versions have been seeing changes in that area. > > Would I be correct to assume the bad query plan is using a date/datetime > index? This could be one source of the problem, but in your case it's > probably another issue. > You're asking the optimizer to calculate a query plan based on the unknown > > result of a user function. As such the optimizer must consider/estimate a > selectivity. Too low will make that index non attractive, and too high > makes it too attractive. And the dramatic issue is that it has no way to > make a good estimate. > > I believe that recent versions (11.50.FC9 and maybe 11.70.FC2/3) may have > change the way this selectivity is calculated. This solved many customer's > > problems and probably caused problems to many others... > > I'm not in developing but I believe 11.70.FC4 may have changed this. And > eventually some improvements may appear in the future. Would it be too > hard > for you to try that query on 11.70.FC4? > > Of course, if you had the chance to avoid the use of the function and give > > it the value the problem should not happen. > If you're sure that the query should NEVER use that index (on the column > you're comparing with the function result), you can use some sort of > operation on it. For example: > > my_column + 0 UNITS DAY = ( SELECT WEEKID(...) FROM systables WHERE tabid > = > 0) > > This will avoid the use of any index on that column and will not change > the > result set... hmmmm... But again, if that's a query generated from a tool > this may not be feasible. > > P.S.: Above I avoided using the term "bug" because this is a nasty > situation... We must consider some value for selectivity, but we have no > idea about the possible or probable result of the function. So basically > we're blind... I can't imagine a good solution (that will fit all the > scenarios) for this. > > Regards. > > On Thu, Mar 29, 2012 at 1:27 PM, Peter_Logan@spartanstores.com < > Peter_Logan@spartanstores.com> wrote: > > > Thanks for the reply ... Yes, the auto stats updating has been disabled > > ... and I am aware of the external optimizer directives ... > > > > Let me try to sum this up a little differently ... when the date was say > > > 9/15/2011 .... one query plan was followed ... the one that works ... > now > > that the date is 3/28/2012 ... the optimizer is choosing a different > path > > ... so, what it seems to me is that the optimizer must be evaluating > that > > part of the query .. and then figuring out the query plan from there ... > I > > could understand that if this was a constant, but this isn't ... the > > select weekid(today, -1) from systables where tabid = 1 is a procedure > > that brings back the fiscal week id .. ie 201252 ... does that make a > > little ore sense ... I'm trying to figure out if the optimizer is making > > > that evaluation .. then coming up with the plan.... and if so, how to > keep > > it from doing that .... > > > > Peter Logan > > Senior Database Administrator > > Phone: 616/878-8309 > > > > From: "Fernando Nunes" <domusonline@gmail.com> > > To: ids@iiug.org > > Date: 03/28/2012 05:11 PM > > Subject: Re: Fw: Optimizer [26592] > > Sent by: ids-bounces@iiug.org > > > > I'm afraid I didn't understand your point correctly. You mention the > > select > > from systables but I didn't understand how it's relevant for the > problem. > > Are you saying that query presented works fine but if you use the select > > > from systables instead of: > > > > and a12.fiscal_week_id in (201253) > > > > It will break the query plan? > > > > In any case a few warnings and maybe some useful information: > > > > - 11.7 alreay includes the AUTO_STAT_MODE parameter. This activates or > > deactivates the optimization that says to the engine that "if the table > > doesn't change too much don't bother to update it's statistics". I had a > > > recent situation where the default threshold was causing some stalled > > statistics and it made a difference on the query plans. And the > > pro
No, you can switch between OPTIMIZATION LOW; and OPTIMIZATION HIGH; between queries. It would be nice if they would just add these are optimizer directives though, like ALL_ROWS and FIRST_ROWS optimizer goals. That's a hint Fernando! Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Mar 29, 2012 at 10:43 AM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > Art, > > yes .. and that does help this particular query ... the issue though is > that this is only one part of a larger set of queries...and the > optimization being set low has a negative impact on the entire stream ... > and they can't turn that off and back on .... can only set it for the > session ... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > From: "Art Kagel" <art.kagel@gmail.com> > To: ids@iiug.org > Date: 03/29/2012 10:24 AM > Subject: Re: Fw: Optimizer [26600] > Sent by: ids-bounces@iiug.org > > One thing Peter, have you tried running this query under SET OPTIMIZATION > LOW? You have 10 tables in there, I'm willing to bet that a significant > portion of the runtime is optimizing the query since there are 3,628,800 > query plans that have to be examined and have cost calculated under the > default HIGH optimization method versus only 54 query plans under LOW > optimization. > > Art > > Art S. Kagel > Advanced DataTools (www.advancedatatools.com) > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > other organization with which I am associated either explicitly, > implicitly, or by inference. Neither do those opinions reflect those of > other individuals affiliated with any entity with which I am affiliated > nor > those of the entities themselves. > > On Thu, Mar 29, 2012 at 10:06 AM, Peter_Logan@spartanstores.com < > Peter_Logan@spartanstores.com> wrote: > > > Ok .. here is the really strange thing ... > > > > If I do away with the function ... and put in place the fiscal week id > .... > > then up until week 36 one query plan is used ... and then from week 37 > on, > > a different query plan is used ... The first plan runs in a minute...the > > > second several minutes ... when I put the directive on the query and > > force it to use the order of the tables that was chosen by the optimizer > > > it always runs in a minute ... > > > > I'm trying to see if an external directive will work for this one .. > just > > to give them that as a choice .. but I can't seem to get that working > .... > > I can save the external directive, but when I execute the sql, the > > explain doesn't show the directive being used ... and I'm using the > exact > > sql to create the directive as well as execute the sql ... wondering if > > the function in the sql is messing that up ... > > > > Peter Logan > > Senior Database Administrator > > Phone: 616/878-8309 > > > > From: "Fernando Nunes" <domusonline@gmail.com> > > To: ids@iiug.org > > Date: 03/29/2012 09:52 AM > > Subject: Re: Fw: Optimizer [26597] > > Sent by: ids-bounces@iiug.org > > > > Hmmm.... Yes, I think I can understand the point, although tables schema > > > would help. > > I believe the problem is a well known issue (notice I use "issue" > instead > > of "bug"). And recent versions have been seeing changes in that area. > > > > Would I be correct to assume the bad query plan is using a date/datetime > > > index? This could be one source of the problem, but in your case it's > > probably another issue. > > You're asking the optimizer to calculate a query plan based on the > unknown > > > > result of a user function. As such the optimizer must consider/estimate > a > > selectivity. Too low will make that index non attractive, and too high > > makes it too attractive. And the dramatic issue is that it has no way to > > > make a good estimate. > > > > I believe that recent versions (11.50.FC9 and maybe 11.70.FC2/3) may > have > > change the way this selectivity is calculated. This solved many > customer's > > > > problems and probably caused problems to many others... > > > > I'm not in developing but I believe 11.70.FC4 may have changed this. And > > > eventually some improvements may appear in the future. Would it be too > > hard > > for you to try that query on 11.70.FC4? > > > > Of course, if you had the chance to avoid the use of the function and > give > > > > it the value the problem should not happen. > > If you're sure that the query should NEVER use that index (on the column > > > you're comparing with the function result), you can use some sort of > > operation on it. For example: > > > > my_column + 0 UNITS DAY = ( SELECT WEEKID(...) FROM systables WHERE > tabid > > = > > 0) > > > > This will avoid the use of any index on that column and will not change > > the > > result set... hmmmm... But again, if that's a query generated from a > tool > > this may not be feasible. > > > > P.S.: Above I avoided using the term "bug" because this is a nasty > > situation... We must consider some value for selectivity, but we have no > > > idea about the possible or probable result of the function. So basically > > > we're blind... I can't imagine a good solution (that will fit all the > > scenarios) for this. > > > > Regards. > > > > On Thu, Mar 29, 2012 at 1:27 PM, Peter_Logan@spartanstores.com < > > Peter_Logan@spartanstores.com> wrote: > > > > > Thanks for the reply ... Yes, the auto stats updating has been > disabled > > > ... and I am aware of the external optimizer directives ... > > > > > > Let me try to sum this up a little differently ... when the date was > say > > > > > 9/15/2011 .... one query plan was followed ... the one that works ... > > now > > > that the date is 3/28/2012 ... the optimizer is choosing a different > > path > > > ... so, what it seems to me is that the optimizer must be evaluating > > that > > > part of the query .. and then figuring out the query plan from there > .... > > I > > > could understand that if this was a constant, but this isn't ... the > > > select weekid(today, -1) from systables where tabid = 1 is a procedure > > > > that brings back the fiscal week id .. ie 201252 ... does that make a > > > little ore sense ... I'm trying to figure out if the optimizer is > making > > > > > that evaluation .. then coming up with the plan.... and if so, how to > > keep > > > it from doing that ..
I understand ... but from what I understand the client tool can only set this at the beginning .... Peter Logan Senior Database Administrator Phone: 616/878-8309 From: "Art Kagel" <art.kagel@gmail.com> To: ids@iiug.org Date: 03/29/2012 10:54 AM Subject: Re: Fw: Optimizer [26602] Sent by: ids-bounces@iiug.org No, you can switch between OPTIMIZATION LOW; and OPTIMIZATION HIGH; between queries. It would be nice if they would just add these are optimizer directives though, like ALL_ROWS and FIRST_ROWS optimizer goals. That's a hint Fernando! Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Mar 29, 2012 at 10:43 AM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > Art, > > yes .. and that does help this particular query ... the issue though is > that this is only one part of a larger set of queries...and the > optimization being set low has a negative impact on the entire stream ... > and they can't turn that off and back on .... can only set it for the > session ... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > From: "Art Kagel" <art.kagel@gmail.com> > To: ids@iiug.org > Date: 03/29/2012 10:24 AM > Subject: Re: Fw: Optimizer [26600] > Sent by: ids-bounces@iiug.org > > One thing Peter, have you tried running this query under SET OPTIMIZATION > LOW? You have 10 tables in there, I'm willing to bet that a significant > portion of the runtime is optimizing the query since there are 3,628,800 > query plans that have to be examined and have cost calculated under the > default HIGH optimization method versus only 54 query plans under LOW > optimization. > > Art > > Art S. Kagel > Advanced DataTools (www.advancedatatools.com) > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > other organization with which I am associated either explicitly, > implicitly, or by inference. Neither do those opinions reflect those of > other individuals affiliated with any entity with which I am affiliated > nor > those of the entities themselves. > > On Thu, Mar 29, 2012 at 10:06 AM, Peter_Logan@spartanstores.com < > Peter_Logan@spartanstores.com> wrote: > > > Ok .. here is the really strange thing ... > > > > If I do away with the function ... and put in place the fiscal week id > .... > > then up until week 36 one query plan is used ... and then from week 37 > on, > > a different query plan is used ... The first plan runs in a minute...the > > > second several minutes ... when I put the directive on the query and > > force it to use the order of the tables that was chosen by the optimizer > > > it always runs in a minute ... > > > > I'm trying to see if an external directive will work for this one .. > just > > to give them that as a choice .. but I can't seem to get that working > .... > > I can save the external directive, but when I execute the sql, the > > explain doesn't show the directive being used ... and I'm using the > exact > > sql to create the directive as well as execute the sql ... wondering if > > the function in the sql is messing that up ... > > > > Peter Logan > > Senior Database Administrator > > Phone: 616/878-8309 > > > > From: "Fernando Nunes" <domusonline@gmail.com> > > To: ids@iiug.org > > Date: 03/29/2012 09:52 AM > > Subject: Re: Fw: Optimizer [26597] > > Sent by: ids-bounces@iiug.org > > > > Hmmm.... Yes, I think I can understand the point, although tables schema > > > would help. > > I believe the problem is a well known issue (notice I use "issue" > instead > > of "bug"). And recent versions have been seeing changes in that area. > > > > Would I be correct to assume the bad query plan is using a date/datetime > > > index? This could be one source of the problem, but in your case it's > > probably another issue. > > You're asking the optimizer to calculate a query plan based on the > unknown > > > > result of a user function. As such the optimizer must consider/estimate > a > > selectivity. Too low will make that index non attractive, and too high > > makes it too attractive. And the dramatic issue is that it has no way to > > > make a good estimate. > > > > I believe that recent versions (11.50.FC9 and maybe 11.70.FC2/3) may > have > > change the way this selectivity is calculated. This solved many > customer's > > > > problems and probably caused problems to many others... > > > > I'm not in developing but I believe 11.70.FC4 may have changed this. And > > > eventually some improvements may appear in the future. Would it be too > > hard > > for you to try that query on 11.70.FC4? > > > > Of course, if you had the chance to avoid the use of the function and > give > > > > it the value the problem should not happen. > > If you're sure that the query should NEVER use that index (on the column > > > you're comparing with the function result), you can use some sort of > > operation on it. For example: > > > > my_column + 0 UNITS DAY = ( SELECT WEEKID(...) FROM systables WHERE > tabid > > = > > 0) > > > > This will avoid the use of any index on that column and will not change > > the > > result set... hmmmm... But again, if that's a query generated from a > tool > > this may not be feasible. > > > > P.S.: Above I avoided using the term "bug" because this is a nasty > > situation... We must consider some value for selectivity, but we have no > > > idea about the possible or probable result of the function. So basically > > > we're blind... I can't imagine a good solution (that will fit all the > > scenarios) for this. > > > > Regards. > > > > On Thu, Mar 29, 2012 at 1:27 PM, Peter_Logan@spartanstores.com < > > Peter_Logan@spartanstores.com> wrote: > > > > > Thanks for the reply ... Yes, the auto stats updating has been > disabled > > > ... and I am aware of the external optimizer directives ... > > > > > > Let me try to sum this up a little differently ... when the date was > say > > > > > 9/15/2011 .... one query plan was followed ... the one that works ... > > now > > > that the date is 3/28/2012 ... the optimizer is choosing a different > > path > > > ... so, what it seems to me is that the optimizer must be evaluating > > that > > > part of the query .. and then figuri
Don't know the reporting tool you are using, but couldn't you just include the statement in the stream of SQL statements? LIke: "set optimization low; <big query>; set optimization high;" Also, are you completely certain that the data distributions on the two tables that are being directly affected by the week id (so week_to_year_lkp and efin_wk_line_cls) are current and sufficiently detailed? Have you tried increasing the resolution for the HIGH on the week_to_year_lkp.fiscal_week_id, efin_wk_line_cls.fiscal_week_id, and week_to_year_lkp.fiscal_wty_id columns beyond the default 2.5%? Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Mar 29, 2012 at 10:57 AM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > I understand ... but from what I understand the client tool can only set > this at the beginning .... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > From: "Art Kagel" <art.kagel@gmail.com> > To: ids@iiug.org > Date: 03/29/2012 10:54 AM > Subject: Re: Fw: Optimizer [26602] > Sent by: ids-bounces@iiug.org > > No, you can switch between OPTIMIZATION LOW; and OPTIMIZATION HIGH; > between > queries. It would be nice if they would just add these are optimizer > directives though, like ALL_ROWS and FIRST_ROWS optimizer goals. > > That's a hint Fernando! > > Art > > Art S. Kagel > Advanced DataTools (www.advancedatatools.com) > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > other organization with which I am associated either explicitly, > implicitly, or by inference. Neither do those opinions reflect those of > other individuals affiliated with any entity with which I am affiliated > nor > those of the entities themselves. > > On Thu, Mar 29, 2012 at 10:43 AM, Peter_Logan@spartanstores.com < > Peter_Logan@spartanstores.com> wrote: > > > Art, > > > > yes .. and that does help this particular query ... the issue though is > > that this is only one part of a larger set of queries...and the > > optimization being set low has a negative impact on the entire stream > .... > > and they can't turn that off and back on .... can only set it for the > > session ... > > > > Peter Logan > > Senior Database Administrator > > Phone: 616/878-8309 > > > > From: "Art Kagel" <art.kagel@gmail.com> > > To: ids@iiug.org > > Date: 03/29/2012 10:24 AM > > Subject: Re: Fw: Optimizer [26600] > > Sent by: ids-bounces@iiug.org > > > > One thing Peter, have you tried running this query under SET > OPTIMIZATION > > LOW? You have 10 tables in there, I'm willing to bet that a significant > > portion of the runtime is optimizing the query since there are 3,628,800 > > > query plans that have to be examined and have cost calculated under the > > default HIGH optimization method versus only 54 query plans under LOW > > optimization. > > > > Art > > > > Art S. Kagel > > Advanced DataTools (www.advancedatatools.com) > > Blog: http://informix-myview.blogspot.com/ > > > > Disclaimer: Please keep in mind that my own opinions are my own opinions > > > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > > > other organization with which I am associated either explicitly, > > implicitly, or by inference. Neither do those opinions reflect those of > > other individuals affiliated with any entity with which I am affiliated > > nor > > those of the entities themselves. > > > > On Thu, Mar 29, 2012 at 10:06 AM, Peter_Logan@spartanstores.com < > > Peter_Logan@spartanstores.com> wrote: > > > > > Ok .. here is the really strange thing ... > > > > > > If I do away with the function ... and put in place the fiscal week id > > > .... > > > then up until week 36 one query plan is used ... and then from week 37 > > > on, > > > a different query plan is used ... The first plan runs in a > minute...the > > > > > second several minutes ... when I put the directive on the query and > > > force it to use the order of the tables that was chosen by the > optimizer > > > > > it always runs in a minute ... > > > > > > I'm trying to see if an external directive will work for this one .. > > just > > > to give them that as a choice .. but I can't seem to get that working > > .... > > > I can save the external directive, but when I execute the sql, the > > > explain doesn't show the directive being used ... and I'm using the > > exact > > > sql to create the directive as well as execute the sql ... wondering > if > > > the function in the sql is messing that up ... > > > > > > Peter Logan > > > Senior Database Administrator > > > Phone: 616/878-8309 > > > > > > From: "Fernando Nunes" <domusonline@gmail.com> > > > To: ids@iiug.org > > > Date: 03/29/2012 09:52 AM > > > Subject: Re: Fw: Optimizer [26597] > > > Sent by: ids-bounces@iiug.org > > > > > > Hmmm.... Yes, I think I can understand the point, although tables > schema > > > > > would help. > > > I believe the problem is a well known issue (notice I use "issue" > > instead > > > of "bug"). And recent versions have been seeing changes in that area. > > > > > > Would I be correct to assume the bad query plan is using a > date/datetime > > > > > index? This could be one source of the problem, but in your case it's > > > probably another issue. > > > You're asking the optimizer to calculate a query plan based on the > > unknown > > > > > > result of a user function. As such the optimizer must > consider/estimate > > a > > > selectivity. Too low will make that index non attractive, and too high > > > > makes it too attractive. And the dramatic issue is that it has no way > to > > > > > make a good estimate. > > > > > > I believe that recent versions (11.50.FC9 and maybe 11.70.FC2/3) may > > have > > > change the way this selectivity is calculated. This solved many > > customer's > > > > > > problems and probably caused problems to many others... > > > > > > I'm not in developing but I believe 11.70.FC4 may have changed this. > And > > > > > eventually some improvements may appear in the future. Would it be too > > > > hard > > > for you to try that query on 11.70.FC4? > > > > > > Of course, if you had the chance to avoid the use of the function and > > give > > > > > > it the value the problem should not happen. > > > If you're sure that the query should NEVER use that index (on the@
Haven't tried that yet ... But what I don't understand .. is why the optimizer would pick a different path ... the only difference is the day the query is run .....which obviously will change the returned value from the weekid function ... is it somehow executing that part of the sql to determine the query plan .... Peter Logan Senior Database Administrator Phone: 616/878-8309 From: "Art Kagel" <art.kagel@gmail.com> To: ids@iiug.org Date: 03/29/2012 11:44 AM Subject: Re: Fw: Optimizer [26604] Sent by: ids-bounces@iiug.org Don't know the reporting tool you are using, but couldn't you just include the statement in the stream of SQL statements? LIke: "set optimization low; <big query>; set optimization high;" Also, are you completely certain that the data distributions on the two tables that are being directly affected by the week id (so week_to_year_lkp and efin_wk_line_cls) are current and sufficiently detailed? Have you tried increasing the resolution for the HIGH on the week_to_year_lkp.fiscal_week_id, efin_wk_line_cls.fiscal_week_id, and week_to_year_lkp.fiscal_wty_id columns beyond the default 2.5%? Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Mar 29, 2012 at 10:57 AM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > I understand ... but from what I understand the client tool can only set > this at the beginning .... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > From: "Art Kagel" <art.kagel@gmail.com> > To: ids@iiug.org > Date: 03/29/2012 10:54 AM > Subject: Re: Fw: Optimizer [26602] > Sent by: ids-bounces@iiug.org > > No, you can switch between OPTIMIZATION LOW; and OPTIMIZATION HIGH; > between > queries. It would be nice if they would just add these are optimizer > directives though, like ALL_ROWS and FIRST_ROWS optimizer goals. > > That's a hint Fernando! > > Art > > Art S. Kagel > Advanced DataTools (www.advancedatatools.com) > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > other organization with which I am associated either explicitly, > implicitly, or by inference. Neither do those opinions reflect those of > other individuals affiliated with any entity with which I am affiliated > nor > those of the entities themselves. > > On Thu, Mar 29, 2012 at 10:43 AM, Peter_Logan@spartanstores.com < > Peter_Logan@spartanstores.com> wrote: > > > Art, > > > > yes .. and that does help this particular query ... the issue though is > > that this is only one part of a larger set of queries...and the > > optimization being set low has a negative impact on the entire stream > .... > > and they can't turn that off and back on .... can only set it for the > > session ... > > > > Peter Logan > > Senior Database Administrator > > Phone: 616/878-8309 > > > > From: "Art Kagel" <art.kagel@gmail.com> > > To: ids@iiug.org > > Date: 03/29/2012 10:24 AM > > Subject: Re: Fw: Optimizer [26600] > > Sent by: ids-bounces@iiug.org > > > > One thing Peter, have you tried running this query under SET > OPTIMIZATION > > LOW? You have 10 tables in there, I'm willing to bet that a significant > > portion of the runtime is optimizing the query since there are 3,628,800 > > > query plans that have to be examined and have cost calculated under the > > default HIGH optimization method versus only 54 query plans under LOW > > optimization. > > > > Art > > > > Art S. Kagel > > Advanced DataTools (www.advancedatatools.com) > > Blog: http://informix-myview.blogspot.com/ > > > > Disclaimer: Please keep in mind that my own opinions are my own opinions > > > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > > > other organization with which I am associated either explicitly, > > implicitly, or by inference. Neither do those opinions reflect those of > > other individuals affiliated with any entity with which I am affiliated > > nor > > those of the entities themselves. > > > > On Thu, Mar 29, 2012 at 10:06 AM, Peter_Logan@spartanstores.com < > > Peter_Logan@spartanstores.com> wrote: > > > > > Ok .. here is the really strange thing ... > > > > > > If I do away with the function ... and put in place the fiscal week id > > > .... > > > then up until week 36 one query plan is used ... and then from week 37 > > > on, > > > a different query plan is used ... The first plan runs in a > minute...the > > > > > second several minutes ... when I put the directive on the query and > > > force it to use the order of the tables that was chosen by the > optimizer > > > > > it always runs in a minute ... > > > > > > I'm trying to see if an external directive will work for this one .. > > just > > > to give them that as a choice .. but I can't seem to get that working > > .... > > > I can save the external directive, but when I execute the sql, the > > > explain doesn't show the directive being used ... and I'm using the > > exact > > > sql to create the directive as well as execute the sql ... wondering > if > > > the function in the sql is messing that up ... > > > > > > Peter Logan > > > Senior Database Administrator > > > Phone: 616/878-8309 > > > > > > From: "Fernando Nunes" <domusonline@gmail.com> > > > To: ids@iiug.org > > > Date: 03/29/2012 09:52 AM > > > Subject: Re: Fw: Optimizer [26597] > > > Sent by: ids-bounces@iiug.org > > > > > > Hmmm.... Yes, I think I can understand the point, although tables > schema > > > > > would help. > > > I believe the problem is a well known issue (notice I use "issue" > > instead > > > of "bug"). And recent versions have been seeing changes in that area. > > > > > > Would I be correct to assume the bad query plan is using a > date/datetime > > > > > index? This could be one source of the problem, but in your case it's > > > probably another issue. > > > You're asking the optimizer to calculate a query plan based on the > > unknown > > > > > > result of a user function. As such the optimizer must > consider/estimate > > a > > > selectivity. Too low will make that index non attractive, and too high > > > > makes it too attractive. And the dramatic issue is that it has no way > to > > > > > make a good estimate. > > > > > > I believe that rec
Yeah, I don't think so (on the optimizer executing the function). However, you note that the query plan changes even when you hardwire different values for the week id that would be the output of the function, so I'm still thinking it has something to do with the quality of the data distributions. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Mar 29, 2012 at 11:50 AM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > Haven't tried that yet ... > > But what I don't understand .. is why the optimizer would pick a different > path ... the only difference is the day the query is run .....which > obviously will change the returned value from the weekid function ... is > it somehow executing that part of the sql to determine the query plan .... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > From: "Art Kagel" <art.kagel@gmail.com> > To: ids@iiug.org > Date: 03/29/2012 11:44 AM > Subject: Re: Fw: Optimizer [26604] > Sent by: ids-bounces@iiug.org > > Don't know the reporting tool you are using, but couldn't you just include > > the statement in the stream of SQL statements? LIke: > > "set optimization low; <big query>; set optimization high;" > > Also, are you completely certain that the data distributions on the two > tables that are being directly affected by the week id (so > week_to_year_lkp > and efin_wk_line_cls) are current and sufficiently detailed? Have you > tried increasing the resolution for the HIGH on the > week_to_year_lkp.fiscal_week_id, efin_wk_line_cls.fiscal_week_id, and > week_to_year_lkp.fiscal_wty_id columns beyond the default 2.5%? > > Art > > Art S. Kagel > Advanced DataTools (www.advancedatatools.com) > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > other organization with which I am associated either explicitly, > implicitly, or by inference. Neither do those opinions reflect those of > other individuals affiliated with any entity with which I am affiliated > nor > those of the entities themselves. > > On Thu, Mar 29, 2012 at 10:57 AM, Peter_Logan@spartanstores.com < > Peter_Logan@spartanstores.com> wrote: > > > I understand ... but from what I understand the client tool can only set > > > this at the beginning .... > > > > Peter Logan > > Senior Database Administrator > > Phone: 616/878-8309 > > > > From: "Art Kagel" <art.kagel@gmail.com> > > To: ids@iiug.org > > Date: 03/29/2012 10:54 AM > > Subject: Re: Fw: Optimizer [26602] > > Sent by: ids-bounces@iiug.org > > > > No, you can switch between OPTIMIZATION LOW; and OPTIMIZATION HIGH; > > between > > queries. It would be nice if they would just add these are optimizer > > directives though, like ALL_ROWS and FIRST_ROWS optimizer goals. > > > > That's a hint Fernando! > > > > Art > > > > Art S. Kagel > > Advanced DataTools (www.advancedatatools.com) > > Blog: http://informix-myview.blogspot.com/ > > > > Disclaimer: Please keep in mind that my own opinions are my own opinions > > > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > > > other organization with which I am associated either explicitly, > > implicitly, or by inference. Neither do those opinions reflect those of > > other individuals affiliated with any entity with which I am affiliated > > nor > > those of the entities themselves. > > > > On Thu, Mar 29, 2012 at 10:43 AM, Peter_Logan@spartanstores.com < > > Peter_Logan@spartanstores.com> wrote: > > > > > Art, > > > > > > yes .. and that does help this particular query ... the issue though > is > > > that this is only one part of a larger set of queries...and the > > > optimization being set low has a negative impact on the entire stream > > .... > > > and they can't turn that off and back on .... can only set it for the > > > session ... > > > > > > Peter Logan > > > Senior Database Administrator > > > Phone: 616/878-8309 > > > > > > From: "Art Kagel" <art.kagel@gmail.com> > > > To: ids@iiug.org > > > Date: 03/29/2012 10:24 AM > > > Subject: Re: Fw: Optimizer [26600] > > > Sent by: ids-bounces@iiug.org > > > > > > One thing Peter, have you tried running this query under SET > > OPTIMIZATION > > > LOW? You have 10 tables in there, I'm willing to bet that a > significant > > > portion of the runtime is optimizing the query since there are > 3,628,800 > > > > > query plans that have to be examined and have cost calculated under > the > > > default HIGH optimization method versus only 54 query plans under LOW > > > optimization. > > > > > > Art > > > > > > Art S. Kagel > > > Advanced DataTools (www.advancedatatools.com) > > > Blog: http://informix-myview.blogspot.com/ > > > > > > Disclaimer: Please keep in mind that my own opinions are my own > opinions > > > > > and do not reflect on my employer, Advanced DataTools, the IIUG, nor > any > > > > > other organization with which I am associated either explicitly, > > > implicitly, or by inference. Neither do those opinions reflect those > of > > > other individuals affiliated with any entity with which I am > affiliated > > > nor > > > those of the entities themselves. > > > > > > On Thu, Mar 29, 2012 at 10:06 AM, Peter_Logan@spartanstores.com < > > > Peter_Logan@spartanstores.com> wrote: > > > > > > > Ok .. here is the really strange thing ... > > > > > > > > If I do away with the function ... and put in place the fiscal week > id > > > > > .... > > > > then up until week 36 one query plan is used ... and then from week > 37 > > > > > on, > > > > a different query plan is used ... The first plan runs in a > > minute...the > > > > > > > second several minutes ... when I put the directive on the query and > > > > > force it to use the order of the tables that was chosen by the > > optimizer > > > > > > > it always runs in a minute ... > > > > > > > > I'm trying to see if an external directive will work for this one .. > > > > just > > > > to give them that as a choice .. but I can't seem to get that > working > > > .... > > > > I can save the external directive, but when I execute the sql, the > > > > explain doesn't show the directive being used ... and I'm using the > > > exact > > > > sql to create the directive as well as execute the sql ... wondering > > > if > > > > the function in the sql is messing that up ... > > > > > > > > Peter
You may be right ... I just updated the stats on all the tables and now I get the bad plan no matter what ... Peter Logan Senior Database Administrator Phone: 616/878-8309 From: "Art Kagel" <art.kagel@gmail.com> To: ids@iiug.org Date: 03/29/2012 12:43 PM Subject: Re: Fw: Optimizer [26606] Sent by: ids-bounces@iiug.org Yeah, I don't think so (on the optimizer executing the function). However, you note that the query plan changes even when you hardwire different values for the week id that would be the output of the function, so I'm still thinking it has something to do with the quality of the data distributions. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Mar 29, 2012 at 11:50 AM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > Haven't tried that yet ... > > But what I don't understand .. is why the optimizer would pick a different > path ... the only difference is the day the query is run .....which > obviously will change the returned value from the weekid function ... is > it somehow executing that part of the sql to determine the query plan .... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > From: "Art Kagel" <art.kagel@gmail.com> > To: ids@iiug.org > Date: 03/29/2012 11:44 AM > Subject: Re: Fw: Optimizer [26604] > Sent by: ids-bounces@iiug.org > > Don't know the reporting tool you are using, but couldn't you just include > > the statement in the stream of SQL statements? LIke: > > "set optimization low; <big query>; set optimization high;" > > Also, are you completely certain that the data distributions on the two > tables that are being directly affected by the week id (so > week_to_year_lkp > and efin_wk_line_cls) are current and sufficiently detailed? Have you > tried increasing the resolution for the HIGH on the > week_to_year_lkp.fiscal_week_id, efin_wk_line_cls.fiscal_week_id, and > week_to_year_lkp.fiscal_wty_id columns beyond the default 2.5%? > > Art > > Art S. Kagel > Advanced DataTools (www.advancedatatools.com) > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > other organization with which I am associated either explicitly, > implicitly, or by inference. Neither do those opinions reflect those of > other individuals affiliated with any entity with which I am affiliated > nor > those of the entities themselves. > > On Thu, Mar 29, 2012 at 10:57 AM, Peter_Logan@spartanstores.com < > Peter_Logan@spartanstores.com> wrote: > > > I understand ... but from what I understand the client tool can only set > > > this at the beginning .... > > > > Peter Logan > > Senior Database Administrator > > Phone: 616/878-8309 > > > > From: "Art Kagel" <art.kagel@gmail.com> > > To: ids@iiug.org > > Date: 03/29/2012 10:54 AM > > Subject: Re: Fw: Optimizer [26602] > > Sent by: ids-bounces@iiug.org > > > > No, you can switch between OPTIMIZATION LOW; and OPTIMIZATION HIGH; > > between > > queries. It would be nice if they would just add these are optimizer > > directives though, like ALL_ROWS and FIRST_ROWS optimizer goals. > > > > That's a hint Fernando! > > > > Art > > > > Art S. Kagel > > Advanced DataTools (www.advancedatatools.com) > > Blog: http://informix-myview.blogspot.com/ > > > > Disclaimer: Please keep in mind that my own opinions are my own opinions > > > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > > > other organization with which I am associated either explicitly, > > implicitly, or by inference. Neither do those opinions reflect those of > > other individuals affiliated with any entity with which I am affiliated > > nor > > those of the entities themselves. > > > > On Thu, Mar 29, 2012 at 10:43 AM, Peter_Logan@spartanstores.com < > > Peter_Logan@spartanstores.com> wrote: > > > > > Art, > > > > > > yes .. and that does help this particular query ... the issue though > is > > > that this is only one part of a larger set of queries...and the > > > optimization being set low has a negative impact on the entire stream > > .... > > > and they can't turn that off and back on .... can only set it for the > > > session ... > > > > > > Peter Logan > > > Senior Database Administrator > > > Phone: 616/878-8309 > > > > > > From: "Art Kagel" <art.kagel@gmail.com> > > > To: ids@iiug.org > > > Date: 03/29/2012 10:24 AM > > > Subject: Re: Fw: Optimizer [26600] > > > Sent by: ids-bounces@iiug.org > > > > > > One thing Peter, have you tried running this query under SET > > OPTIMIZATION > > > LOW? You have 10 tables in there, I'm willing to bet that a > significant > > > portion of the runtime is optimizing the query since there are > 3,628,800 > > > > > query plans that have to be examined and have cost calculated under > the > > > default HIGH optimization method versus only 54 query plans under LOW > > > optimization. > > > > > > Art > > > > > > Art S. Kagel > > > Advanced DataTools (www.advancedatatools.com) > > > Blog: http://informix-myview.blogspot.com/ > > > > > > Disclaimer: Please keep in mind that my own opinions are my own > opinions > > > > > and do not reflect on my employer, Advanced DataTools, the IIUG, nor > any > > > > > other organization with which I am associated either explicitly, > > > implicitly, or by inference. Neither do those opinions reflect those > of > > > other individuals affiliated with any entity with which I am > affiliated > > > nor > > > those of the entities themselves. > > > > > > On Thu, Mar 29, 2012 at 10:06 AM, Peter_Logan@spartanstores.com < > > > Peter_Logan@spartanstores.com> wrote: > > > > > > > Ok .. here is the really strange thing ... > > > > > > > > If I do away with the function ... and put in place the fiscal week > id > > > > > .... > > > > then up until week 36 one query plan is used ... and then from week > 37 > > > > > on, > > > > a different query plan is used ... The first plan runs in a > > minute...the > > > > > > > second several minutes ... when I put the directive on the query and > > > > > force it to use the order of the tables that was chosen by the > > optimizer > > > > > > > it always runs in a minute ... > > > > > > > > I'm trying to see if an external directive will work
Hmmmm... How do you have the EXT_DIRECTIVES parameter in onconfig? Regards. On Thu, Mar 29, 2012 at 3:06 PM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > Ok .. here is the really strange thing ... > > If I do away with the function ... and put in place the fiscal week id ... > then up until week 36 one query plan is used ... and then from week 37 on, > a different query plan is used ... The first plan runs in a minute...the > second several minutes ... when I put the directive on the query and > force it to use the order of the tables that was chosen by the optimizer > it always runs in a minute ... > > I'm trying to see if an external directive will work for this one .. just > to give them that as a choice .. but I can't seem to get that working ... > I can save the external directive, but when I execute the sql, the > explain doesn't show the directive being used ... and I'm using the exact > sql to create the directive as well as execute the sql ... wondering if > the function in the sql is messing that up ... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > From: "Fernando Nunes" <domusonline@gmail.com> > To: ids@iiug.org > Date: 03/29/2012 09:52 AM > Subject: Re: Fw: Optimizer [26597] > Sent by: ids-bounces@iiug.org > > Hmmm.... Yes, I think I can understand the point, although tables schema > would help. > I believe the problem is a well known issue (notice I use "issue" instead > of "bug"). And recent versions have been seeing changes in that area. > > Would I be correct to assume the bad query plan is using a date/datetime > index? This could be one source of the problem, but in your case it's > probably another issue. > You're asking the optimizer to calculate a query plan based on the unknown > > result of a user function. As such the optimizer must consider/estimate a > selectivity. Too low will make that index non attractive, and too high > makes it too attractive. And the dramatic issue is that it has no way to > make a good estimate. > > I believe that recent versions (11.50.FC9 and maybe 11.70.FC2/3) may have > change the way this selectivity is calculated. This solved many customer's > > problems and probably caused problems to many others... > > I'm not in developing but I believe 11.70.FC4 may have changed this. And > eventually some improvements may appear in the future. Would it be too > hard > for you to try that query on 11.70.FC4? > > Of course, if you had the chance to avoid the use of the function and give > > it the value the problem should not happen. > If you're sure that the query should NEVER use that index (on the column > you're comparing with the function result), you can use some sort of > operation on it. For example: > > my_column + 0 UNITS DAY = ( SELECT WEEKID(...) FROM systables WHERE tabid > = > 0) > > This will avoid the use of any index on that column and will not change > the > result set... hmmmm... But again, if that's a query generated from a tool > this may not be feasible. > > P.S.: Above I avoided using the term "bug" because this is a nasty > situation... We must consider some value for selectivity, but we have no > idea about the possible or probable result of the function. So basically > we're blind... I can't imagine a good solution (that will fit all the > scenarios) for this. > > Regards. > > On Thu, Mar 29, 2012 at 1:27 PM, Peter_Logan@spartanstores.com < > Peter_Logan@spartanstores.com> wrote: > > > Thanks for the reply ... Yes, the auto stats updating has been disabled > > ... and I am aware of the external optimizer directives ... > > > > Let me try to sum this up a little differently ... when the date was say > > > 9/15/2011 .... one query plan was followed ... the one that works ... > now > > that the date is 3/28/2012 ... the optimizer is choosing a different > path > > ... so, what it seems to me is that the optimizer must be evaluating > that > > part of the query .. and then figuring out the query plan from there ... > I > > could understand that if this was a constant, but this isn't ... the > > select weekid(today, -1) from systables where tabid = 1 is a procedure > > that brings back the fiscal week id .. ie 201252 ... does that make a > > little ore sense ... I'm trying to figure out if the optimizer is making > > > that evaluation .. then coming up with the plan.... and if so, how to > keep > > it from doing that .... > > > > Peter Logan > > Senior Database Administrator > > Phone: 616/878-8309 > > > > From: "Fernando Nunes" <domusonline@gmail.com> > > To: ids@iiug.org > > Date: 03/28/2012 05:11 PM > > Subject: Re: Fw: Optimizer [26592] > > Sent by: ids-bounces@iiug.org > > > > I'm afraid I didn't understand your point correctly. You mention the > > select > > from systables but I didn't understand how it's relevant for the > problem. > > Are you saying that query presented works fine but if you use the select > > > from systables instead of: > > > > and a12.fiscal_week_id in (201253) > > > > It will break the query plan? > > > > In any case a few warnings and maybe some useful information: > > > > - 11.7 alreay includes the AUTO_STAT_MODE parameter. This activates or > > deactivates the optimization that says to the engine that "if the table > > doesn't change too much don't bother to update it's statistics". I had a > > > recent situation where the default threshold was causing some stalled > > statistics and it made a difference on the query plans. And the > > probability > > that this happens with queries involving time/dates is higher than with > > other types of filters (because these fields many times tend to be > > populated with constantly increasing values, so statistically something > > that makes the filter "column > today - some_time" look good will in > time > > lead to bad choices) > > > > - You can activate optimizer directives for any query outside the > > applications by using the External Directives feature. I will not go > into > > details, because it could take a while, but if you're not aware of this > > and > > need further information, please tell. In any case I'm against the > > principle of using directives to solve query plans problems... after all > > > we > > have an optimizer... we're not Oracle :) > > > > So, my advise: > > - Verify that the distributions are really up to date (disable > > AUTO_STAT_MODE) and run them (it's dynamic and I believe there's a way > to > > control it inside the session (either with SET ENVIRONMENT or with a > > environment variable) > > - If you never used the external optimizer directives, please > investigate > > them. Sometimes you can get away with them to quickly solve a > performance > > problem. But my advise is to always investigate this kind of problems > and > > find a definitive solution > > - You could of course re-post your problem in a manner I can understand, > > > or > > wait until someone smarter to answer you :) > > > > Regards. >
After a point in time, statistically there are no records that match your query. This will not happen if you run the query after updatting statistics. Have you? On Thu, Mar 29, 2012 at 4:50 PM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > Haven't tried that yet ... > > But what I don't understand .. is why the optimizer would pick a different > path ... the only difference is the day the query is run .....which > obviously will change the returned value from the weekid function ... is > it somehow executing that part of the sql to determine the query plan .... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > From: "Art Kagel" <art.kagel@gmail.com> > To: ids@iiug.org > Date: 03/29/2012 11:44 AM > Subject: Re: Fw: Optimizer [26604] > Sent by: ids-bounces@iiug.org > > Don't know the reporting tool you are using, but couldn't you just include > > the statement in the stream of SQL statements? LIke: > > "set optimization low; <big query>; set optimization high;" > > Also, are you completely certain that the data distributions on the two > tables that are being directly affected by the week id (so > week_to_year_lkp > and efin_wk_line_cls) are current and sufficiently detailed? Have you > tried increasing the resolution for the HIGH on the > week_to_year_lkp.fiscal_week_id, efin_wk_line_cls.fiscal_week_id, and > week_to_year_lkp.fiscal_wty_id columns beyond the default 2.5%? > > Art > > Art S. Kagel > Advanced DataTools (www.advancedatatools.com) > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > other organization with which I am associated either explicitly, > implicitly, or by inference. Neither do those opinions reflect those of > other individuals affiliated with any entity with which I am affiliated > nor > those of the entities themselves. > > On Thu, Mar 29, 2012 at 10:57 AM, Peter_Logan@spartanstores.com < > Peter_Logan@spartanstores.com> wrote: > > > I understand ... but from what I understand the client tool can only set > > > this at the beginning .... > > > > Peter Logan > > Senior Database Administrator > > Phone: 616/878-8309 > > > > From: "Art Kagel" <art.kagel@gmail.com> > > To: ids@iiug.org > > Date: 03/29/2012 10:54 AM > > Subject: Re: Fw: Optimizer [26602] > > Sent by: ids-bounces@iiug.org > > > > No, you can switch between OPTIMIZATION LOW; and OPTIMIZATION HIGH; > > between > > queries. It would be nice if they would just add these are optimizer > > directives though, like ALL_ROWS and FIRST_ROWS optimizer goals. > > > > That's a hint Fernando! > > > > Art > > > > Art S. Kagel > > Advanced DataTools (www.advancedatatools.com) > > Blog: http://informix-myview.blogspot.com/ > > > > Disclaimer: Please keep in mind that my own opinions are my own opinions > > > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > > > other organization with which I am associated either explicitly, > > implicitly, or by inference. Neither do those opinions reflect those of > > other individuals affiliated with any entity with which I am affiliated > > nor > > those of the entities themselves. > > > > On Thu, Mar 29, 2012 at 10:43 AM, Peter_Logan@spartanstores.com < > > Peter_Logan@spartanstores.com> wrote: > > > > > Art, > > > > > > yes .. and that does help this particular query ... the issue though > is > > > that this is only one part of a larger set of queries...and the > > > optimization being set low has a negative impact on the entire stream > > .... > > > and they can't turn that off and back on .... can only set it for the > > > session ... > > > > > > Peter Logan > > > Senior Database Administrator > > > Phone: 616/878-8309 > > > > > > From: "Art Kagel" <art.kagel@gmail.com> > > > To: ids@iiug.org > > > Date: 03/29/2012 10:24 AM > > > Subject: Re: Fw: Optimizer [26600] > > > Sent by: ids-bounces@iiug.org > > > > > > One thing Peter, have you tried running this query under SET > > OPTIMIZATION > > > LOW? You have 10 tables in there, I'm willing to bet that a > significant > > > portion of the runtime is optimizing the query since there are > 3,628,800 > > > > > query plans that have to be examined and have cost calculated under > the > > > default HIGH optimization method versus only 54 query plans under LOW > > > optimization. > > > > > > Art > > > > > > Art S. Kagel > > > Advanced DataTools (www.advancedatatools.com) > > > Blog: http://informix-myview.blogspot.com/ > > > > > > Disclaimer: Please keep in mind that my own opinions are my own > opinions > > > > > and do not reflect on my employer, Advanced DataTools, the IIUG, nor > any > > > > > other organization with which I am associated either explicitly, > > > implicitly, or by inference. Neither do those opinions reflect those > of > > > other individuals affiliated with any entity with which I am > affiliated > > > nor > > > those of the entities themselves. > > > > > > On Thu, Mar 29, 2012 at 10:06 AM, Peter_Logan@spartanstores.com < > > > Peter_Logan@spartanstores.com> wrote: > > > > > > > Ok .. here is the really strange thing ... > > > > > > > > If I do away with the function ... and put in place the fiscal week > id > > > > > .... > > > > then up until week 36 one query plan is used ... and then from week > 37 > > > > > on, > > > > a different query plan is used ... The first plan runs in a > > minute...the > > > > > > > second several minutes ... when I put the directive on the query and > > > > > force it to use the order of the tables that was chosen by the > > optimizer > > > > > > > it always runs in a minute ... > > > > > > > > I'm trying to see if an external directive will work for this one .. > > > > just > > > > to give them that as a choice .. but I can't seem to get that > working > > > .... > > > > I can save the external directive, but when I execute the sql, the > > > > explain doesn't show the directive being used ... and I'm using the > > > exact > > > > sql to create the directive as well as execute the sql ... wondering > > > if > > > > the function in the sql is messing that up ... > > > > > > > > Peter Logan > > > > Senior Database Administrator > > > > Phone: 616/878-8309 > > > > > > > > From: "Fernando Nunes" <domusonline@gmail.com> > > > > To: ids@iiug.org > > > > Date: 03/29/2012 09:52 AM > > > > Subject: Re: Fw: Optimizer [26597] > > > > Sent by: ids-bounces@iiug.org > > > > > > > > Hmmm.... Yes, I think I can understand the point, although tables > > schema > > > > > > > would help. > > > > I believe the problem is a well known issue (notice I use "issue" > > > instead > > > > of "bug"). And recent versions have been seeing changes in that > area. > > > > > > > > Would I
What level of stats? Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Mar 29, 2012 at 1:24 PM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > You may be right ... I just updated the stats on all the tables and now I > get the bad plan no matter what ... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > From: "Art Kagel" <art.kagel@gmail.com> > To: ids@iiug.org > Date: 03/29/2012 12:43 PM > Subject: Re: Fw: Optimizer [26606] > Sent by: ids-bounces@iiug.org > > Yeah, I don't think so (on the optimizer executing the function). However, > > you note that the query plan changes even when you hardwire different > values for the week id that would be the output of the function, so I'm > still thinking it has something to do with the quality of the data > distributions. > > Art > > Art S. Kagel > Advanced DataTools (www.advancedatatools.com) > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > other organization with which I am associated either explicitly, > implicitly, or by inference. Neither do those opinions reflect those of > other individuals affiliated with any entity with which I am affiliated > nor > those of the entities themselves. > > On Thu, Mar 29, 2012 at 11:50 AM, Peter_Logan@spartanstores.com < > Peter_Logan@spartanstores.com> wrote: > > > Haven't tried that yet ... > > > > But what I don't understand .. is why the optimizer would pick a > different > > path ... the only difference is the day the query is run .....which > > obviously will change the returned value from the weekid function ... is > > > it somehow executing that part of the sql to determine the query plan > ..... > > > > Peter Logan > > Senior Database Administrator > > Phone: 616/878-8309 > > > > From: "Art Kagel" <art.kagel@gmail.com> > > To: ids@iiug.org > > Date: 03/29/2012 11:44 AM > > Subject: Re: Fw: Optimizer [26604] > > Sent by: ids-bounces@iiug.org > > > > Don't know the reporting tool you are using, but couldn't you just > include > > > > the statement in the stream of SQL statements? LIke: > > > > "set optimization low; <big query>; set optimization high;" > > > > Also, are you completely certain that the data distributions on the two > > tables that are being directly affected by the week id (so > > week_to_year_lkp > > and efin_wk_line_cls) are current and sufficiently detailed? Have you > > tried increasing the resolution for the HIGH on the > > week_to_year_lkp.fiscal_week_id, efin_wk_line_cls.fiscal_week_id, and > > week_to_year_lkp.fiscal_wty_id columns beyond the default 2.5%? > > > > Art > > > > Art S. Kagel > > Advanced DataTools (www.advancedatatools.com) > > Blog: http://informix-myview.blogspot.com/ > > > > Disclaimer: Please keep in mind that my own opinions are my own opinions > > > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > > > other organization with which I am associated either explicitly, > > implicitly, or by inference. Neither do those opinions reflect those of > > other individuals affiliated with any entity with which I am affiliated > > nor > > those of the entities themselves. > > > > On Thu, Mar 29, 2012 at 10:57 AM, Peter_Logan@spartanstores.com < > > Peter_Logan@spartanstores.com> wrote: > > > > > I understand ... but from what I understand the client tool can only > set > > > > > this at the beginning .... > > > > > > Peter Logan > > > Senior Database Administrator > > > Phone: 616/878-8309 > > > > > > From: "Art Kagel" <art.kagel@gmail.com> > > > To: ids@iiug.org > > > Date: 03/29/2012 10:54 AM > > > Subject: Re: Fw: Optimizer [26602] > > > Sent by: ids-bounces@iiug.org > > > > > > No, you can switch between OPTIMIZATION LOW; and OPTIMIZATION HIGH; > > > between > > > queries. It would be nice if they would just add these are optimizer > > > directives though, like ALL_ROWS and FIRST_ROWS optimizer goals. > > > > > > That's a hint Fernando! > > > > > > Art > > > > > > Art S. Kagel > > > Advanced DataTools (www.advancedatatools.com) > > > Blog: http://informix-myview.blogspot.com/ > > > > > > Disclaimer: Please keep in mind that my own opinions are my own > opinions > > > > > and do not reflect on my employer, Advanced DataTools, the IIUG, nor > any > > > > > other organization with which I am associated either explicitly, > > > implicitly, or by inference. Neither do those opinions reflect those > of > > > other individuals affiliated with any entity with which I am > affiliated > > > nor > > > those of the entities themselves. > > > > > > On Thu, Mar 29, 2012 at 10:43 AM, Peter_Logan@spartanstores.com < > > > Peter_Logan@spartanstores.com> wrote: > > > > > > > Art, > > > > > > > > yes .. and that does help this particular query ... the issue though > > > is > > > > that this is only one part of a larger set of queries...and the > > > > optimization being set low has a negative impact on the entire > stream > > > .... > > > > and they can't turn that off and back on .... can only set it for > the > > > > session ... > > > > > > > > Peter Logan > > > > Senior Database Administrator > > > > Phone: 616/878-8309 > > > > > > > > From: "Art Kagel" <art.kagel@gmail.com> > > > > To: ids@iiug.org > > > > Date: 03/29/2012 10:24 AM > > > > Subject: Re: Fw: Optimizer [26600] > > > > Sent by: ids-bounces@iiug.org > > > > > > > > One thing Peter, have you tried running this query under SET > > > OPTIMIZATION > > > > LOW? You have 10 tables in there, I'm willing to bet that a > > significant > > > > portion of the runtime is optimizing the query since there are > > 3,628,800 > > > > > > > query plans that have to be examined and have cost calculated under > > the > > > > default HIGH optimization method versus only 54 query plans under > LOW > > > > optimization. > > > > > > > > Art > > > > > > > > Art S. Kagel > > > > Advanced DataTools (www.advancedatatools.com) > > > > Blog: http://informix-myview.blogspot.com/ > > > > > > > > Disclaimer: Please keep in mind that my own opinions are my own > > opinions > > > > > > > and do not reflect on my employer, Advanced DataTools, the IIUG, nor > > > any > > > > > > > other organization with which I am associated either explicitly, > > > > implicitly, or by inference. Neither do tho
set to 1 .. tried a simple sql and it worked fine ... Peter Logan Senior Database Administrator Phone: 616/878-8309 From: "Fernando Nunes" <domusonline@gmail.com> To: ids@iiug.org Date: 03/29/2012 01:50 PM Subject: Re: Fw: Optimizer [26608] Sent by: ids-bounces@iiug.org Hmmmm... How do you have the EXT_DIRECTIVES parameter in onconfig? Regards. On Thu, Mar 29, 2012 at 3:06 PM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > Ok .. here is the really strange thing ... > > If I do away with the function ... and put in place the fiscal week id ... > then up until week 36 one query plan is used ... and then from week 37 on, > a different query plan is used ... The first plan runs in a minute...the > second several minutes ... when I put the directive on the query and > force it to use the order of the tables that was chosen by the optimizer > it always runs in a minute ... > > I'm trying to see if an external directive will work for this one .. just > to give them that as a choice .. but I can't seem to get that working ... > I can save the external directive, but when I execute the sql, the > explain doesn't show the directive being used ... and I'm using the exact > sql to create the directive as well as execute the sql ... wondering if > the function in the sql is messing that up ... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > From: "Fernando Nunes" <domusonline@gmail.com> > To: ids@iiug.org > Date: 03/29/2012 09:52 AM > Subject: Re: Fw: Optimizer [26597] > Sent by: ids-bounces@iiug.org > > Hmmm.... Yes, I think I can understand the point, although tables schema > would help. > I believe the problem is a well known issue (notice I use "issue" instead > of "bug"). And recent versions have been seeing changes in that area. > > Would I be correct to assume the bad query plan is using a date/datetime > index? This could be one source of the problem, but in your case it's > probably another issue. > You're asking the optimizer to calculate a query plan based on the unknown > > result of a user function. As such the optimizer must consider/estimate a > selectivity. Too low will make that index non attractive, and too high > makes it too attractive. And the dramatic issue is that it has no way to > make a good estimate. > > I believe that recent versions (11.50.FC9 and maybe 11.70.FC2/3) may have > change the way this selectivity is calculated. This solved many customer's > > problems and probably caused problems to many others... > > I'm not in developing but I believe 11.70.FC4 may have changed this. And > eventually some improvements may appear in the future. Would it be too > hard > for you to try that query on 11.70.FC4? > > Of course, if you had the chance to avoid the use of the function and give > > it the value the problem should not happen. > If you're sure that the query should NEVER use that index (on the column > you're comparing with the function result), you can use some sort of > operation on it. For example: > > my_column + 0 UNITS DAY = ( SELECT WEEKID(...) FROM systables WHERE tabid > = > 0) > > This will avoid the use of any index on that column and will not change > the > result set... hmmmm... But again, if that's a query generated from a tool > this may not be feasible. > > P.S.: Above I avoided using the term "bug" because this is a nasty > situation... We must consider some value for selectivity, but we have no > idea about the possible or probable result of the function. So basically > we're blind... I can't imagine a good solution (that will fit all the > scenarios) for this. > > Regards. > > On Thu, Mar 29, 2012 at 1:27 PM, Peter_Logan@spartanstores.com < > Peter_Logan@spartanstores.com> wrote: > > > Thanks for the reply ... Yes, the auto stats updating has been disabled > > ... and I am aware of the external optimizer directives ... > > > > Let me try to sum this up a little differently ... when the date was say > > > 9/15/2011 .... one query plan was followed ... the one that works ... > now > > that the date is 3/28/2012 ... the optimizer is choosing a different > path > > ... so, what it seems to me is that the optimizer must be evaluating > that > > part of the query .. and then figuring out the query plan from there ... > I > > could understand that if this was a constant, but this isn't ... the > > select weekid(today, -1) from systables where tabid = 1 is a procedure > > that brings back the fiscal week id .. ie 201252 ... does that make a > > little ore sense ... I'm trying to figure out if the optimizer is making > > > that evaluation .. then coming up with the plan.... and if so, how to > keep > > it from doing that .... > > > > Peter Logan > > Senior Database Administrator > > Phone: 616/878-8309 > > > > From: "Fernando Nunes" <domusonline@gmail.com> > > To: ids@iiug.org > > Date: 03/28/2012 05:11 PM > > Subject: Re: Fw: Optimizer [26592] > > Sent by: ids-bounces@iiug.org > > > > I'm afraid I didn't understand your point correctly. You mention the > > select > > from systables but I didn't understand how it's relevant for the > problem. > > Are you saying that query presented works fine but if you use the select > > > from systables instead of: > > > > and a12.fiscal_week_id in (201253) > > > > It will break the query plan? > > > > In any case a few warnings and maybe some useful information: > > > > - 11.7 alreay includes the AUTO_STAT_MODE parameter. This activates or > > deactivates the optimization that says to the engine that "if the table > > doesn't change too much don't bother to update it's statistics". I had a > > > recent situation where the default threshold was causing some stalled > > statistics and it made a difference on the query plans. And the > > probability > > that this happens with queries involving time/dates is higher than with > > other types of filters (because these fields many times tend to be > > populated with constantly increasing values, so statistically something > > that makes the filter "column > today - some_time" look good will in > time > > lead to bad choices) > > > > - You can activate optimizer directives for any query outside the > > applications by using the External Directives feature. I will not go > into > > details, because it could take a while, but if you're not aware of this > > and > > need further information, please tell. In any case I'm against the > > principle of using directives to solve query plans problems... after all > > > we > > have an optimizer... we're not Oracle :) > > > > So, my advise: > > - Verify that the distributions are really up to date (disable > > AUTO_STAT_MODE) and run them (it's dynamic and I believe there's a way > to > > control it inside the session (either with SET
medium ... distributions only ... high fiscal_week_id (lead index) low (index columns) Peter Logan Senior Database Administrator Phone: 616/878-8309 From: "Art Kagel" <art.kagel@gmail.com> To: ids@iiug.org Date: 03/29/2012 02:00 PM Subject: Re: Fw: Optimizer [26610] Sent by: ids-bounces@iiug.org What level of stats? Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Mar 29, 2012 at 1:24 PM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > You may be right ... I just updated the stats on all the tables and now I > get the bad plan no matter what ... > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > From: "Art Kagel" <art.kagel@gmail.com> > To: ids@iiug.org > Date: 03/29/2012 12:43 PM > Subject: Re: Fw: Optimizer [26606] > Sent by: ids-bounces@iiug.org > > Yeah, I don't think so (on the optimizer executing the function). However, > > you note that the query plan changes even when you hardwire different > values for the week id that would be the output of the function, so I'm > still thinking it has something to do with the quality of the data > distributions. > > Art > > Art S. Kagel > Advanced DataTools (www.advancedatatools.com) > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > other organization with which I am associated either explicitly, > implicitly, or by inference. Neither do those opinions reflect those of > other individuals affiliated with any entity with which I am affiliated > nor > those of the entities themselves. > > On Thu, Mar 29, 2012 at 11:50 AM, Peter_Logan@spartanstores.com < > Peter_Logan@spartanstores.com> wrote: > > > Haven't tried that yet ... > > > > But what I don't understand .. is why the optimizer would pick a > different > > path ... the only difference is the day the query is run .....which > > obviously will change the returned value from the weekid function ... is > > > it somehow executing that part of the sql to determine the query plan > ..... > > > > Peter Logan > > Senior Database Administrator > > Phone: 616/878-8309 > > > > From: "Art Kagel" <art.kagel@gmail.com> > > To: ids@iiug.org > > Date: 03/29/2012 11:44 AM > > Subject: Re: Fw: Optimizer [26604] > > Sent by: ids-bounces@iiug.org > > > > Don't know the reporting tool you are using, but couldn't you just > include > > > > the statement in the stream of SQL statements? LIke: > > > > "set optimization low; <big query>; set optimization high;" > > > > Also, are you completely certain that the data distributions on the two > > tables that are being directly affected by the week id (so > > week_to_year_lkp > > and efin_wk_line_cls) are current and sufficiently detailed? Have you > > tried increasing the resolution for the HIGH on the > > week_to_year_lkp.fiscal_week_id, efin_wk_line_cls.fiscal_week_id, and > > week_to_year_lkp.fiscal_wty_id columns beyond the default 2.5%? > > > > Art > > > > Art S. Kagel > > Advanced DataTools (www.advancedatatools.com) > > Blog: http://informix-myview.blogspot.com/ > > > > Disclaimer: Please keep in mind that my own opinions are my own opinions > > > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > > > other organization with which I am associated either explicitly, > > implicitly, or by inference. Neither do those opinions reflect those of > > other individuals affiliated with any entity with which I am affiliated > > nor > > those of the entities themselves. > > > > On Thu, Mar 29, 2012 at 10:57 AM, Peter_Logan@spartanstores.com < > > Peter_Logan@spartanstores.com> wrote: > > > > > I understand ... but from what I understand the client tool can only > set > > > > > this at the beginning .... > > > > > > Peter Logan > > > Senior Database Administrator > > > Phone: 616/878-8309 > > > > > > From: "Art Kagel" <art.kagel@gmail.com> > > > To: ids@iiug.org > > > Date: 03/29/2012 10:54 AM > > > Subject: Re: Fw: Optimizer [26602] > > > Sent by: ids-bounces@iiug.org > > > > > > No, you can switch between OPTIMIZATION LOW; and OPTIMIZATION HIGH; > > > between > > > queries. It would be nice if they would just add these are optimizer > > > directives though, like ALL_ROWS and FIRST_ROWS optimizer goals. > > > > > > That's a hint Fernando! > > > > > > Art > > > > > > Art S. Kagel > > > Advanced DataTools (www.advancedatatools.com) > > > Blog: http://informix-myview.blogspot.com/ > > > > > > Disclaimer: Please keep in mind that my own opinions are my own > opinions > > > > > and do not reflect on my employer, Advanced DataTools, the IIUG, nor > any > > > > > other organization with which I am associated either explicitly, > > > implicitly, or by inference. Neither do those opinions reflect those > of > > > other individuals affiliated with any entity with which I am > affiliated > > > nor > > > those of the entities themselves. > > > > > > On Thu, Mar 29, 2012 at 10:43 AM, Peter_Logan@spartanstores.com < > > > Peter_Logan@spartanstores.com> wrote: > > > > > > > Art, > > > > > > > > yes .. and that does help this particular query ... the issue though > > > is > > > > that this is only one part of a larger set of queries...and the > > > > optimization being set low has a negative impact on the entire > stream > > > .... > > > > and they can't turn that off and back on .... can only set it for > the > > > > session ... > > > > > > > > Peter Logan > > > > Senior Database Administrator > > > > Phone: 616/878-8309 > > > > > > > > From: "Art Kagel" <art.kagel@gmail.com> > > > > To: ids@iiug.org > > > > Date: 03/29/2012 10:24 AM > > > > Subject: Re: Fw: Optimizer [26600] > > > > Sent by: ids-bounces@iiug.org > > > > > > > > One thing Peter, have you tried running this query under SET > > > OPTIMIZATION > > > > LOW? You have 10 tables in there, I'm willing to bet that a > > significant > > > > portion of the runtime is optimizing the query since there are > > 3,628,800 > > > > > > > query plans that have to be examined and have cost calculated under > > the > > > > default HIGH optimization method versus only 54 query plans under > LOW > > > > optimization. > > > > > > > > Art > > > > > > > > Art S. Kagel@@NL@
Stats on fiscal_week_id is HIGH in BOTH tables? What about fiscal_wty_id? That should be HIGH as well since it is an important filter column for the join. Also try increasing the resolution from 2.5% to 1% to go from 40 bins to 100 bins for those three columns. I was going to suggest that you drop distributions altogether for those two tables so that the older v5 style optimizer has to kick in (it often makes better decisions in strange situations like this one), but I'm afraid what the affect on other queries will be if you do that. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Mar 29, 2012 at 2:04 PM, Peter_Logan@spartanstores.com < Peter_Logan@spartanstores.com> wrote: > medium ... distributions only ... > > high fiscal_week_id (lead index) > low (index columns) > > Peter Logan > Senior Database Administrator > Phone: 616/878-8309 > > From: "Art Kagel" <art.kagel@gmail.com> > To: ids@iiug.org > Date: 03/29/2012 02:00 PM > Subject: Re: Fw: Optimizer [26610] > Sent by: ids-bounces@iiug.org > > What level of stats? > > Art > > Art S. Kagel > Advanced DataTools (www.advancedatatools.com) > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > other organization with which I am associated either explicitly, > implicitly, or by inference. Neither do those opinions reflect those of > other individuals affiliated with any entity with which I am affiliated > nor > those of the entities themselves. > > On Thu, Mar 29, 2012 at 1:24 PM, Peter_Logan@spartanstores.com < > Peter_Logan@spartanstores.com> wrote: > > > You may be right ... I just updated the stats on all the tables and now > I > > get the bad plan no matter what ... > > > > Peter Logan > > Senior Database Administrator > > Phone: 616/878-8309 > > > > From: "Art Kagel" <art.kagel@gmail.com> > > To: ids@iiug.org > > Date: 03/29/2012 12:43 PM > > Subject: Re: Fw: Optimizer [26606] > > Sent by: ids-bounces@iiug.org > > > > Yeah, I don't think so (on the optimizer executing the function). > However, > > > > you note that the query plan changes even when you hardwire different > > values for the week id that would be the output of the function, so I'm > > still thinking it has something to do with the quality of the data > > distributions. > > > > Art > > > > Art S. Kagel > > Advanced DataTools (www.advancedatatools.com) > > Blog: http://informix-myview.blogspot.com/ > > > > Disclaimer: Please keep in mind that my own opinions are my own opinions > > > and do not reflect on my employer, Advanced DataTools, the IIUG, nor any > > > other organization with which I am associated either explicitly, > > implicitly, or by inference. Neither do those opinions reflect those of > > other individuals affiliated with any entity with which I am affiliated > > nor > > those of the entities themselves. > > > > On Thu, Mar 29, 2012 at 11:50 AM, Peter_Logan@spartanstores.com < > > Peter_Logan@spartanstores.com> wrote: > > > > > Haven't tried that yet ... > > > > > > But what I don't understand .. is why the optimizer would pick a > > different > > > path ... the only difference is the day the query is run .....which > > > obviously will change the returned value from the weekid function ... > is > > > > > it somehow executing that part of the sql to determine the query plan > > ..... > > > > > > Peter Logan > > > Senior Database Administrator > > > Phone: 616/878-8309 > > > > > > From: "Art Kagel" <art.kagel@gmail.com> > > > To: ids@iiug.org > > > Date: 03/29/2012 11:44 AM > > > Subject: Re: Fw: Optimizer [26604] > > > Sent by: ids-bounces@iiug.org > > > > > > Don't know the reporting tool you are using, but couldn't you just > > include > > > > > > the statement in the stream of SQL statements? LIke: > > > > > > "set optimization low; <big query>; set optimization high;" > > > > > > Also, are you completely certain that the data distributions on the > two > > > tables that are being directly affected by the week id (so > > > week_to_year_lkp > > > and efin_wk_line_cls) are current and sufficiently detailed? Have you > > > tried increasing the resolution for the HIGH on the > > > week_to_year_lkp.fiscal_week_id, efin_wk_line_cls.fiscal_week_id, and > > > week_to_year_lkp.fiscal_wty_id columns beyond the default 2.5%? > > > > > > Art > > > > > > Art S. Kagel > > > Advanced DataTools (www.advancedatatools.com) > > > Blog: http://informix-myview.blogspot.com/ > > > > > > Disclaimer: Please keep in mind that my own opinions are my own > opinions > > > > > and do not reflect on my employer, Advanced DataTools, the IIUG, nor > any > > > > > other organization with which I am associated either explicitly, > > > implicitly, or by inference. Neither do those opinions reflect those > of > > > other individuals affiliated with any entity with which I am > affiliated > > > nor > > > those of the entities themselves. > > > > > > On Thu, Mar 29, 2012 at 10:57 AM, Peter_Logan@spartanstores.com < > > > Peter_Logan@spartanstores.com> wrote: > > > > > > > I understand ... but from what I understand the client tool can only > > > set > > > > > > > this at the beginning .... > > > > > > > > Peter Logan > > > > Senior Database Administrator > > > > Phone: 616/878-8309 > > > > > > > > From: "Art Kagel" <art.kagel@gmail.com> > > > > To: ids@iiug.org > > > > Date: 03/29/2012 10:54 AM > > > > Subject: Re: Fw: Optimizer [26602] > > > > Sent by: ids-bounces@iiug.org > > > > > > > > No, you can switch between OPTIMIZATION LOW; and OPTIMIZATION HIGH; > > > > between > > > > queries. It would be nice if they would just add these are optimizer > > > > > directives though, like ALL_ROWS and FIRST_ROWS optimizer goals. > > > > > > > > That's a hint Fernando! > > > > > > > > Art > > > > > > > > Art S. Kagel > > > > Advanced DataTools (www.advancedatatools.com) > > > > Blog: http://informix-myview.blogspot.com/ > > > > > > > > Disclaimer: Please keep in mind that my own opinions are my own > > opinions > > > > > > > and do not reflect on my employer, Advanced DataTools, the IIUG, nor > > > any > > > > > > > other organization with which I am associated either explicitly, > > > > implicitly, or by inference. Neither do those opinions reflect those > > > of > > > > other individuals affiliated with any entity with which