Too much IO Wait time

Hi list, First sorry about my english, I have migrate my informix server from 10.0 to 11.5FC5 and also migrate hardware. I move to a new 2xQuad Xeon+4x150 Gb Disk 10000RPM+8Gb of Ram. In tests it seems to be a much faster, but when i go to my production enviroment I see some Queries taking too much to be resolved. My plataform is Linux. Syntoms when I execute theses queries: 1) Like 35% of CPU in Waiting (I/O) 2) Status of session changing between "IO WAIT" and "yield bufwait-" Example of query: Explain out: QUERY: (OPTIMIZATION TIMESTAMP: 09-14-2009 15:27:32) ------ SELECT posicion.fechaposicion as fechaposicion , posicion.holderid as^M holderid , posicion.eventoid as eventoid , holder.dominio as dominio ^M FROM posicion INNER JOIN holder ON (holder.holderid=posicion.holderid) ^M WHERE ( posicion.holderid in^M (14273,14440,14448,14274,14442,15171,14275,14271,14443,14253,13045,14278,14292,1 4279,14250,13044,13011,13002,14306,14305,14291) ) AND (^M posicion.eventoid in (7, 6, 4) ) AND ( posicion.fechaposicion >=^M '2009-09-08 04:00:00' ) AND ( posicion.fechaposicion <= '2009-09-14 03:59:00' ) ORDER BY fechaposicion Estimated Cost: 87861 Estimated # of Rows Returned: 9117 Temporary Files Required For: Order By 1) informix.holder: INDEX PATH (1) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = 14273 (2) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = 14440 (3) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = 14448 (4) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = 14274 (5) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = 14442 (6) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = 15171 (7) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = 14275 (8) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) (9) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = 14443 (10) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = 14253 (11) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = 13045 (12) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = 14278 (13) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = 14292 (14) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = 14279 (15) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = 14250 (16) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = 13044 (17) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = 13011 (18) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = 13002 (19) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = 14306 (20) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = 14305 (21) Index Name: root. 216_289 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = 14291 2) informix.posicion: INDEX PATH Filters: ((informix.posicion.fechaposicion >= datetime(2009-09-08 04:00:00) year to second AND informix.posicion.eventoid IN (7 , 6 , 4 )) AND informix.posicion.fechaposicion <= datetime(2009-09-14 03:59:00) year to second ) (1) Index Name: root. 100_909 Index Keys: holderid (Serial, fragments: ALL) Lower Index Filter: informix.holder.holderid = informix.posicion.holderid NESTED LOOP JOIN Query statistics: ----------------- Table map : ---------------------------- Internal name Table name ---------------------------- t1 holder t2 posicion type table rows_prod est_rows rows_scan time est_cost ------------------------------------------------------------------- scan t1 21 21 21 00:00.00 5 type table rows_prod est_rows rows_scan time est_cost ------------------------------------------------------------------- scan t2 49511 9117 844729 06:31.26 4006 type rows_prod est_rows time est_cost ------------------------------------------------- nljoin 49511 9118 06:31.27 87861 type rows_sort est_rows rows_cons time ------------------------------------------------- sort 49511 9118 49511 06:32.55 # onstat -g ses 5382 IBM Informix Dynamic Server Version 11.50.FC5 -- On-Line -- Up 01:39:11 -- 6651320 Kbytes session effective #RSAM total used dynamic id user user tty pid hostname threads memory memory explain 5382 informix - reflex 8 Emiliano 1 253952 225008 off tid name rstcb flags curstk status 5532 sqlexec 19ee4ebd8 ---PR-- 13488 *IO Wait-* Memory pools count 3 name class addr totalsize freesize #allocfrag #freefrag 5382 V 1a027b040 167936 17168 240 16 5382*O0 V 1a1131040 4096 808 1 1 5382_SORT_0 V 1a23e8040 81920 10968 19 3 name free used name free used overhead 0 9864 scb 0 144 opentable 0 8096 filetable 0 2240 ru 0 600 log 0 16536 temprec 0 2320 keys 0 216 ralloc 0 74160 gentcb 0 1584 ostcb 0 2920 sort 0 67488 sqscb 0 21736 sql 0 72 srtmembuf 0 200 rdahead 0 1120 hashfiletab 0 552 osenv 0 2072 sqtcb 0 8528 fragman 0 3640 shmblklist 0 776 sapi 0 64 sqscb info scb sqscb optofc pdqpriority sqlstats optcompind directives 1a15dd028 19f728028 0 0 0 2 1 Sess SQL Current Iso Lock SQL ISAM F.E. Id Stmt type Database Lvl Mode ERR ERR Vers Explain 5382 SELECT db3g CR Not Wait 0 0 9.29 On Current statement name : _ixceb #onstat -g ses 5382 IBM Informix Dynamic Server Version 11.50.FC5 -- On-Line -- Up 01:39:20 -- 6651320 Kbytes session effective #RSAM total used dynamic id user user tty pid hostname threa

Mr. Romero, 1) It is always recommended doing a full update statistics after a version migration. Did you do it? 2) If you new machine has 8 cores (2 quad cores), you could optimize your cpu´s vps 3) are you using KAIO modes to improve the IO transfers ??? 4) After all above, check on manuals how to optimize the b-tree index scanning (alice mode is great!!!!) Regards. Alexandre Marini Tecnologia da Informação - DBA SEFAZ-MS / SGI-UIMP / Sistemas IBM-Informix IIUG Member <http://www.iiug.org>; Emiliano Romero escreveu: > Hi list, > > First sorry about my english, > > I have migrate my informix server from 10.0 to 11.5FC5 and also migrate > hardware. I move to a new 2xQuad Xeon+4x150 Gb Disk 10000RPM+8Gb of Ram. > In tests it seems to be a much faster, but when i go to my production > enviroment I see some Queries taking too much to be resolved. > > My plataform is Linux. > > Syntoms when I execute theses queries: > 1) Like 35% of CPU in Waiting (I/O) > 2) Status of session changing between "IO WAIT" and "yield bufwait-" > > Example of query: > Explain out: > QUERY: (OPTIMIZATION TIMESTAMP: 09-14-2009 15:27:32) > ------ > SELECT posicion.fechaposicion as fechaposicion , posicion.holderid as^M> > holderid , posicion.eventoid as eventoid , holder.dominio as > dominio ^M > > FROM posicion INNER JOIN holder ON > (holder.holderid=posicion.holderid) ^M > > WHERE ( posicion.holderid in^M > > > (14273,14440,14448,14274,14442,15171,14275,14271,14443,14253,13045,14278,14292,1 4279,14250,13044,13011,13002,14306,14305,14291) > ) AND (^M > > posicion.eventoid in (7, 6, 4) ) AND ( posicion.fechaposicion >=^M > > '2009-09-08 04:00:00' ) AND ( posicion.fechaposicion <= '2009-09-14 > 03:59:00' ) ORDER BY fechaposicion > > Estimated Cost: 87861 > Estimated # of Rows Returned: 9117 > Temporary Files Required For: Order By > > 1) informix.holder: INDEX PATH > > (1) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14273 > > (2) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14440 > > (3) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14448 > > (4) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14274 > > (5) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14442 > > (6) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 15171 > > (7) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14275 > > (8) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > (9) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14443 > > (10) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14253 > > (11) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 13045 > > (12) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14278 > > (13) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14292 > > (14) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14279 > > (15) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14250 > > (16) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 13044 > > (17) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 13011 > > (18) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 13002 > > (19) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14306 > > (20) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14305 > > (21) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14291 > > 2) informix.posicion: INDEX PATH > > Filters: ((informix.posicion.fechaposicion >= > datetime(2009-09-08 04:00:00) year to second AND > informix.posicion.eventoid IN (7 , 6 , 4 )) AND > informix.posicion.fechaposicion <= datetime(2009-09-14 03:59:00) year to > second ) > > (1) Index Name: root. 100_909 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = > informix.posicion.holderid > NESTED LOOP JOIN > > Query statistics: > ----------------- > > Table map : > ---------------------------- > Internal name Table name > ---------------------------- > t1 holder > t2 posicion > > type table rows_prod est_rows rows_scan time est_cost > ------------------------------------------------------------------- > scan t1 21 21 21 00:00.00 5 > > type table rows_prod est_rows rows_scan time est_cost > ------------------------------------------------------------------- > scan t2 49511 9117 844729 06:31.26 4006 > > type rows_prod est_rows time est_cost > ------------------------------------------------- > nljoin 49511 9118 06:31.27 87861 > > type rows_sort est_rows rows_cons time > ------------------------------------------------- > sort 49511 9118 49511 06:32.55 > > # onstat -g ses 5382> > IBM Informix Dynamic Server Version 11.50.FC5 -- On-Line -- Up > 01:39:11 -- 6651320 Kbytes> > session effective #RSAM > total used dynamic > id user user tty pid hostname threads > memory memory explain > 5382 informix - reflex 8 Emiliano 1 > 253952 225008 off > > tid name rstcb flags curstk status > 5532 sqlexec 19ee4ebd8 ---PR-- 13488 *IO Wait-* > > Memory pools count 3 > name class addr totalsize freesize #allocfrag > #freefrag > 5382 V 1a027b040 167936 17168 240 > 16 > 5382*O0 V 1a1131040 4096 808 1 > 1 > 5382_SORT_0 V 1a23e8040 81920 10968 19 > 3@

Alexandre, thanks for your reply. I have do a full update statistics after migration, dbimport do that for me. And I do an update statistics this morning. I add more cpu's vps. Now I have 7. I have KAIO support. I also add a BTSCANNER and change alice mode to 11. After restart of informix now i have 80% of cpu in Waiting. As I read in Informix information center, with KAIO, my cpu vps are used for IO. Maybe that's why I have 80% of my cpu in waiting state. I open a ticket in informix support, so I keep you updated with what they say. Thanks again and Regards Alexandre Marini escribió: > Mr. Romero, > 1) It is always recommended doing a full update statistics after a > version migration. Did you do it? > 2) If you new machine has 8 cores (2 quad cores), you could optimize > your cpu´s vps > 3) are you using KAIO modes to improve the IO transfers ??? > 4) After all above, check on manuals how to optimize the b-tree index > scanning (alice mode is great!!!!) > > Regards. > > Alexandre Marini > > Tecnologia da Informação - DBA > > SEFAZ-MS / SGI-UIMP / Sistemas IBM-Informix > > IIUG Member > > <http://www.iiug.org>; > > Emiliano Romero escreveu: > >> Hi list, >> >> First sorry about my english, >> >> I have migrate my informix server from 10.0 to 11.5FC5 and also migrate >> hardware. I move to a new 2xQuad Xeon+4x150 Gb Disk 10000RPM+8Gb of Ram. >> In tests it seems to be a much faster, but when i go to my production >> enviroment I see some Queries taking too much to be resolved. >> >> My plataform is Linux. >> >> Syntoms when I execute theses queries: >> 1) Like 35% of CPU in Waiting (I/O) >> 2) Status of session changing between "IO WAIT" and "yield bufwait-" >> >> Example of query: >> Explain out: >> QUERY: (OPTIMIZATION TIMESTAMP: 09-14-2009 15:27:32) >> ------ >> SELECT posicion.fechaposicion as fechaposicion , posicion.holderid as^M>> >> holderid , posicion.eventoid as eventoid , holder.dominio as >> dominio ^M >> >> FROM posicion INNER JOIN holder ON >> (holder.holderid=posicion.holderid) ^M >> >> WHERE ( posicion.holderid in^M >> >> >> >> > (14273,14440,14448,14274,14442,15171,14275,14271,14443,14253,13045,14278,14292,1 4279,14250,13044,13011,13002,14306,14305,14291) > >> ) AND (^M >> >> posicion.eventoid in (7, 6, 4) ) AND ( posicion.fechaposicion >=^M >> >> '2009-09-08 04:00:00' ) AND ( posicion.fechaposicion <= '2009-09-14 >> 03:59:00' ) ORDER BY fechaposicion >> >> Estimated Cost: 87861 >> Estimated # of Rows Returned: 9117 >> Temporary Files Required For: Order By >> >> 1) informix.holder: INDEX PATH >> >> (1) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 14273 >> >> (2) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 14440 >> >> (3) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 14448 >> >> (4) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 14274 >> >> (5) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 14442 >> >> (6) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 15171 >> >> (7) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 14275 >> >> (8) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> (9) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 14443 >> >> (10) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 14253 >> >> (11) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 13045 >> >> (12) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 14278 >> >> (13) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 14292 >> >> (14) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 14279 >> >> (15) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 14250 >> >> (16) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 13044 >> >> (17) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 13011 >> >> (18) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 13002 >> >> (19) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 14306 >> >> (20) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 14305 >> >> (21) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 14291 >> >> 2) informix.posicion: INDEX PATH >> >> Filters: ((informix.posicion.fechaposicion >= >> datetime(2009-09-08 04:00:00) year to second AND >> informix.posicion.eventoid IN (7 , 6 , 4 )) AND >> informix.posicion.fechaposicion <= datetime(2009-09-14 03:59:00) year to >> second ) >> >> (1) Index Name: root. 100_909 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = >> informix.posicion.holderid >> NESTED LOOP JOIN >> >> Query statistics: >> ----------------- >> >> Table map : >> ---------------------------- >> Internal name Table name >> ---------------------------- >> t1 holder >> t2 posicion >> >> type table rows_prod est_rows rows_scan time est_cost >> ------------------------------------------------------------------- >> scan t1 21 21 21 00:00.00 5 >> >> type table rows_prod est_rows rows_scan time est_cost >> ------------------------------------------------------------------- >> scan t2 49511 9117 844729 06:31.26 4006 >> >> type rows_prod est_rows time est_cost >>

Ok any additional tip could be great. Regards. Alexandre Marini Tecnologia da Informação - DBA SEFAZ-MS / SGI-UIMP / Sistemas IBM-Informix IIUG Member <http://www.iiug.org>; Emiliano Romero escreveu: > Alexandre, thanks for your reply. > I have do a full update statistics after migration, dbimport do that for > me. And I do an update statistics this morning. > > I add more cpu's vps. Now I have 7. I have KAIO support. I also add a > BTSCANNER and change alice mode to 11. After restart of informix now i > have 80% of cpu in Waiting. > > As I read in Informix information center, with KAIO, my cpu vps are used > for IO. Maybe that's why I have 80% of my cpu in waiting state. > > I open a ticket in informix support, so I keep you updated with what > they say. > > Thanks again and Regards > > Alexandre Marini escribió: > >> Mr. Romero, >> 1) It is always recommended doing a full update statistics after a >> version migration. Did you do it? >> 2) If you new machine has 8 cores (2 quad cores), you could optimize >> your cpu´s vps >> 3) are you using KAIO modes to improve the IO transfers ??? >> 4) After all above, check on manuals how to optimize the b-tree index >> scanning (alice mode is great!!!!) >> >> Regards. >> >> Alexandre Marini >> >> Tecnologia da Informação - DBA >> >> SEFAZ-MS / SGI-UIMP / Sistemas IBM-Informix >> >> IIUG Member >> >> <http://www.iiug.org>; >> >> Emiliano Romero escreveu: >> >> >>> Hi list, >>> >>> First sorry about my english, >>> >>> I have migrate my informix server from 10.0 to 11.5FC5 and also migrate >>> hardware. I move to a new 2xQuad Xeon+4x150 Gb Disk 10000RPM+8Gb of Ram. >>> In tests it seems to be a much faster, but when i go to my production >>> enviroment I see some Queries taking too much to be resolved. >>> >>> My plataform is Linux. >>> >>> Syntoms when I execute theses queries: >>> 1) Like 35% of CPU in Waiting (I/O) >>> 2) Status of session changing between "IO WAIT" and "yield bufwait-" >>> >>> Example of query: >>> Explain out: >>> QUERY: (OPTIMIZATION TIMESTAMP: 09-14-2009 15:27:32) >>> ------ >>> SELECT posicion.fechaposicion as fechaposicion , posicion.holderid as^M>>> >>> holderid , posicion.eventoid as eventoid , holder.dominio as >>> dominio ^M >>> >>> FROM posicion INNER JOIN holder ON >>> (holder.holderid=posicion.holderid) ^M >>> >>> WHERE ( posicion.holderid in^M >>> >>> >>> >>> >>> > (14273,14440,14448,14274,14442,15171,14275,14271,14443,14253,13045,14278,14292,1 4279,14250,13044,13011,13002,14306,14305,14291) > >>> ) AND (^M >>> >>> posicion.eventoid in (7, 6, 4) ) AND ( posicion.fechaposicion >=^M >>> >>> '2009-09-08 04:00:00' ) AND ( posicion.fechaposicion <= '2009-09-14 >>> 03:59:00' ) ORDER BY fechaposicion >>> >>> Estimated Cost: 87861 >>> Estimated # of Rows Returned: 9117 >>> Temporary Files Required For: Order By >>> >>> 1) informix.holder: INDEX PATH >>> >>> (1) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = 14273 >>> >>> (2) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = 14440 >>> >>> (3) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = 14448 >>> >>> (4) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = 14274 >>> >>> (5) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = 14442 >>> >>> (6) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = 15171 >>> >>> (7) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = 14275 >>> >>> (8) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> (9) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = 14443 >>> >>> (10) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = 14253 >>> >>> (11) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = 13045 >>> >>> (12) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = 14278 >>> >>> (13) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = 14292 >>> >>> (14) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = 14279 >>> >>> (15) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = 14250 >>> >>> (16) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = 13044 >>> >>> (17) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = 13011 >>> >>> (18) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = 13002 >>> >>> (19) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = 14306 >>> >>> (20) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = 14305 >>> >>> (21) Index Name: root. 216_289 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = 14291 >>> >>> 2) informix.posicion: INDEX PATH >>> >>> Filters: ((informix.posicion.fechaposicion >= >>> datetime(2009-09-08 04:00:00) year to second AND >>> informix.posicion.eventoid IN (7 , 6 , 4 )) AND >>> informix.posicion.fechaposicion <= datetime(2009-09-14 03:59:00) year to >>> second ) >>> >>> (1) Index Name: root. 100_909 >>> >>> Index Keys: holderid (Serial, fragments: ALL) >>> >>> Lower Index Filter: informix.holder.holderid = >>> informix.posicion.holderid >>> NESTED LOOP JOIN >>> >>> Query statistics: >>> ----------------- >>> >>> Table map : >>> ----------------------------

There are several problems with this query and your configuration. The ONCONFIG first. There are a couple of parameters that I would configure differently, however, the only one that is affecting this query severely is: DS_NONPDQ_QUERY_MEM 128 This controls how much memory is available for sorting in memory when PDQPRIORITY is zero or unset. In earlier engines (before 11.xx) the environment variable DBUPSPACE controlled this and the default was 15MB. In the onconfig.std the default setting for this new parameter, which overrides DBUPSPACE BTW, is 128KB and that is the setting which you have retained. That means that you have about 1/120th the space in memory for sorting data for ORDER BY and other requirements. This query required between 700KB and 1MB of memory to sort the results. That means that this query must do its sorting to the tempdbspaces and disk is MUCH slower than memory. That may be the main cause of the IO waits. Increase DS_NONPDQ_QUERY_MEM to 15360 (15MB) to get back to the default for the IDS version you came from or, better, increase it to 51240 (50MB) to allow even larger sorts to happen in memory. Next, the table posicion had to be index scanned to determine which rows satisfied the secondary filter criteria on the eventoid and fechaposicion columns. You would do better with an index on those two columns along with the join column holderid in this order: (holderid, eventoid, fechaposicion). Without this index to filter, the engine had to scan over 800,000 rows in order to return just 50,000 rows. Another major source or excess IO processing. If that index exists already and wasn't used, that may be due to the next problem I see: Looking at the SET EXPLAIN output, I notice in the statistics section that while the engine had to scan almost 50000 rows it estimated that it would only have to scan just over 9000 rows. This indicates that the data distributions for the index key and filter columns are not sufficient to give the optimizer the data it needs to formulate a good query plan. If you relied on the IDS 11.50's ability to build minimal data distributions on key columns during the index build process to gather the statistics, you should know that only the first column of each index key is used to build those statistical distributions which is not sufficient for more complex queries like this one. Look in the Performance Guide manual for details of the recommended suite of commands that you need to run to gather a useful level of statistics with minimal effort and time. You can also use my dostats utility which implements those recommendations automatically for you. Dostats is free and is contained in the package utils2_ak which you can download from either the Oninit web site (www.oninit.com/utils) or the IIUG Software Repository (www.iiug.org/software) Art Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Tue, Sep 15, 2009 at 10:56 AM, Emiliano Romero <eromero@sitrack.com>wrote: > Hi list, > > First sorry about my english, > > I have migrate my informix server from 10.0 to 11.5FC5 and also migrate > hardware. I move to a new 2xQuad Xeon+4x150 Gb Disk 10000RPM+8Gb of Ram. > In tests it seems to be a much faster, but when i go to my production > enviroment I see some Queries taking too much to be resolved. > > My plataform is Linux. > > Syntoms when I execute theses queries: > 1) Like 35% of CPU in Waiting (I/O) > 2) Status of session changing between "IO WAIT" and "yield bufwait-" > > Example of query: > Explain out: > QUERY: (OPTIMIZATION TIMESTAMP: 09-14-2009 15:27:32) > ------ > SELECT posicion.fechaposicion as fechaposicion , posicion.holderid as^M> > holderid , posicion.eventoid as eventoid , holder.dominio as > dominio ^M > > FROM posicion INNER JOIN holder ON > (holder.holderid=posicion.holderid) ^M > > WHERE ( posicion.holderid in^M > > > > (14273,14440,14448,14274,14442,15171,14275,14271,14443,14253,13045,14278,14292,1 4279,14250,13044,13011,13002,14306,14305,14291) > ) AND (^M > > posicion.eventoid in (7, 6, 4) ) AND ( posicion.fechaposicion >=^M > > '2009-09-08 04:00:00' ) AND ( posicion.fechaposicion <= '2009-09-14 > 03:59:00' ) ORDER BY fechaposicion > > Estimated Cost: 87861 > Estimated # of Rows Returned: 9117 > Temporary Files Required For: Order By > > 1) informix.holder: INDEX PATH > > (1) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14273 > > (2) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14440 > > (3) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14448 > > (4) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14274 > > (5) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14442 > > (6) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 15171 > > (7) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14275 > > (8) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > (9) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14443 > > (10) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14253 > > (11) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 13045 > > (12) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14278 > > (13) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14292 > > (14) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14279 > > (15) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter: informix.holder.holderid = 14250 > > (16) Index Name: root. 216_289 > > Index Keys: holderid (Serial, fragments: ALL) > > Lower Index Filter:

Art, thanks for your reply. I Change DS_NONPDQ_QUERY_MEM too 51240, also run dostats. And there is no change, all our queries that have to access a lot of rows takes too much time. Now I'm going to install Informix 11.5FC5 in my old server, and check if there's a problem with my OS or hardware. Thanks again for your help, and any other ideas are welcome :) # onstat -g mgmMemory Grant Manager (MGM) -------------------------- MAX_PDQPRIORITY: 100 DS_MAX_QUERIES: 400 DS_MAX_SCANS: 1048576 DS_NONPDQ_QUERY_MEM: 51240 KB DS_TOTAL_MEMORY: 204960 KB Queries: Active Ready Maximum 0 0 400 Memory: Total Free Quantum (KB) 204960 204960 512 Scans: Total Free Quantum 1048576 1048576 1 Load Control: (Memory) (Scans) (Priority) (Max Queries) (Reinit) Gate 1 Gate 2 Gate 3 Gate 4 Gate 5 (Queue Length) 0 0 0 0 0 Active Queries: None Ready Queries: None Free Resource Average # Minimum # -------------- --------------- --------- Memory 0.0 +- 0.0 25620 Scans 0.0 +- 0.0 1048576 Queries Average # Maximum # Total # -------------- --------------- --------- ------- Active 0.0 +- 0.0 0 0 Ready 0.0 +- 0.0 0 0 Resource/Lock Cycle Prevention count: 0 Emiliano Romero Art Kagel escribió: > There are several problems with this query and your configuration. The > ONCONFIG first. There are a couple of parameters that I would configure > differently, however, the only one that is affecting this query severely is: > > DS_NONPDQ_QUERY_MEM 128> > This controls how much memory is available for sorting in memory when > PDQPRIORITY is zero or unset. In earlier engines (before 11.xx) the > environment variable DBUPSPACE controlled this and the default was 15MB. In > the onconfig.std the default setting for this new parameter, which overrides > DBUPSPACE BTW, is 128KB and that is the setting which you have retained. > That means that you have about 1/120th the space in memory for sorting data > for ORDER BY and other requirements. This query required between 700KB and > 1MB of memory to sort the results. That means that this query must do its > sorting to the tempdbspaces and disk is MUCH slower than memory. That may > be the main cause of the IO waits. Increase DS_NONPDQ_QUERY_MEM to 15360 > (15MB) to get back to the default for the IDS version you came from or, > better, increase it to 51240 (50MB) to allow even larger sorts to happen in > memory. > > Next, the table posicion had to be index scanned to determine which rows > satisfied the secondary filter criteria on the eventoid and fechaposicion > columns. You would do better with an index on those two columns along with > the join column holderid in this order: (holderid, eventoid, > fechaposicion). Without this index to filter, the engine had to scan over > 800,000 rows in order to return just 50,000 rows. Another major source or > excess IO processing. If that index exists already and wasn't used, that > may be due to the next problem I see: > > Looking at the SET EXPLAIN output, I notice in the statistics section that > while the engine had to scan almost 50000 rows it estimated that it would > only have to scan just over 9000 rows. This indicates that the data > distributions for the index key and filter columns are not sufficient to > give the optimizer the data it needs to formulate a good query plan. > > If you relied on the IDS 11.50's ability to build minimal data distributions > on key columns during the index build process to gather the statistics, you > should know that only the first column of each index key is used to build > those statistical distributions which is not sufficient for more complex > queries like this one. Look in the Performance Guide manual for details of > the recommended suite of commands that you need to run to gather a useful > level of statistics with minimal effort and time. You can also use my > dostats utility which implements those recommendations automatically for > you. Dostats is free and is contained in the package utils2_ak which you > can download from either the Oninit web site (www.oninit.com/utils) or the > IIUG Software Repository (www.iiug.org/software) > > Art > > Art S. Kagel > Oninit (www.oninit.com) > IIUG Board of Directors (art@iiug.org) > > Disclaimer: Please keep in mind that my own opinions are my own opinions and > do not reflect on my employer, Oninit, the IIUG, nor any other organization > with which I am associated either explicitly or implicitly. Neither do > those opinions reflect those of other individuals affiliated with any entity > with which I am affiliated nor those of the entities themselves. > > On Tue, Sep 15, 2009 at 10:56 AM, Emiliano Romero <eromero@sitrack.com>wrote: > > >> Hi list, >> >> First sorry about my english, >> >> I have migrate my informix server from 10.0 to 11.5FC5 and also migrate >> hardware. I move to a new 2xQuad Xeon+4x150 Gb Disk 10000RPM+8Gb of Ram. >> In tests it seems to be a much faster, but when i go to my production >> enviroment I see some Queries taking too much to be resolved. >> >> My plataform is Linux. >> >> Syntoms when I execute theses queries: >> 1) Like 35% of CPU in Waiting (I/O) >> 2) Status of session changing between "IO WAIT" and "yield bufwait-" >> >> Example of query: >> Explain out: >> QUERY: (OPTIMIZATION TIMESTAMP: 09-14-2009 15:27:32) >> ------ >> SELECT posicion.fechaposicion as fechaposicion , posicion.holderid as^M>> >> holderid , posicion.eventoid as eventoid , holder.dominio as >> dominio ^M >> >> FROM posicion INNER JOIN holder ON >> (holder.holderid=posicion.holderid) ^M >> >> WHERE ( posicion.holderid in^M >> >> >> >> >> > (14273,14440,14448,14274,14442,15171,14275,14271,14443,14253,13045,14278,14292,1 4279,14250,13044,13011,13002,14306,14305,14291) > >> ) AND (^M >> >> posicion.eventoid in (7, 6, 4) ) AND ( posicion.fechaposicion >=^M >> >> '2009-09-08 04:00:00' ) AND ( posicion.fechaposicion <= '2009-09-14 >> 03:59:00' ) ORDER BY fechaposicion >> >> Estimated Cost: 87861 >> Estimated # of Rows Returned: 9117 >> Temporary Files Required For: Order By >> >> 1) informix.holder: INDEX PATH >> >> (1) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 14273 >> >> (2) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 14440 >> >> (3) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 14448 >> >> (4) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 14274 >> >> (5) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragments: ALL) >> >> Lower Index Filter: informix.holder.holderid = 14442 >> >> (6) Index Name: root. 216_289 >> >> Index Keys: holderid (Serial, fragmen