Long lasting dostats

Hi all Informix 12.10.FC4W1XB, Solaris 10 I'm running dostats on a very big table. It seems it won't come to an end. Runs now since Monday, the 5th of December. Table myschema: CREATE TABLE "informix".xxxxxxxxxxxxx ( pe_id SERIAL8(15712838227) NOT NULL, pe_re INTEGER, pe_re_lfd SMALLINT, pe_vo SMALLINT, pe_pos SMALLINT, pe_ebene CHAR(2), pe_pruefung SMALLINT, pe_pruefstart DATETIME YEAR TO SECOND, pe_pruefende DATETIME YEAR TO SECOND, pe_pruefart CHAR(1), pe_bearbeiter CHAR(8), pe_wfstatusvor DECIMAL(8,4), pe_wfstatusnach DECIMAL(8,4), pe_ergx SMALLINT, pe_ergxart CHAR(1), pe_berinr INTEGER, pe_textnr INTEGER, pe_folgeberinr INTEGER, pe_jobid INTEGER, pe_jobart CHAR(1), pe_pruefzeit DECIMAL(12,2), pe_pruefzeit_ges DECIMAL(12,2), pe_wfnr SMALLINT, pe_nachbeding SMALLINT, pe_folge_ga INTEGER, pe_lfdnr_verf INTEGER ) FRAGMENT BY EXPRESSION PARTITION xxxxxxxxx23 ((pe_id >= 2200000000 ) AND (pe_id < 2400000000 ) ) IN xxxxxxxxx23, PARTITION xxxxxxxxx24 ((pe_id >= 2400000000 ) AND (pe_id < 2600000000 ) ) IN xxxxxxxxx24, PARTITION xxxxxxxxx25 ((pe_id >= 2600000000 ) AND (pe_id < 2800000000 ) ) IN xxxxxxxxx25, PARTITION xxxxxxxxx26 ((pe_id >= 2800000000 ) AND (pe_id < 3000000000 ) ) IN xxxxxxxxx26, PARTITION xxxxxxxxx27 ((pe_id >= 3000000000 ) AND (pe_id < 3200000000 ) ) IN xxxxxxxxx27, PARTITION xxxxxxxxx28 ((pe_id >= 3200000000 ) AND (pe_id < 3400000000 ) ) IN xxxxxxxxx28, PARTITION xxxxxxxxx29 ((pe_id >= 3400000000 ) AND (pe_id < 3600000000 ) ) IN xxxxxxxxx29, PARTITION xxxxxxxxx30 ((pe_id >= 3600000000 ) AND (pe_id < 3800000000 ) ) IN xxxxxxxxx30, PARTITION xxxxxxxxx31 ((pe_id >= 3800000000 ) AND (pe_id < 4000000000 ) ) IN xxxxxxxxx31, PARTITION xxxxxxxxx14 ((pe_id >= 4000000000 ) AND (pe_id < 4200000000 ) ) IN xxxxxxxxx14, PARTITION xxxxxxxxx5 ((pe_id >= 4200000000 ) AND (pe_id < 4400000000 ) ) IN xxxxxxxxx5, PARTITION xxxxxxxxx13 ((pe_id >= 4400000000 ) AND (pe_id < 4600000000 ) ) IN xxxxxxxxx13, PARTITION xxxxxxxxx11 ((pe_id >= 4800000000 ) AND (pe_id < 5000000000 ) ) IN xxxxxxxxx11, PARTITION xxxxxxxxx22 ((pe_id >= 5000000000 ) AND (pe_id < 5200000000 ) ) IN xxxxxxxxx22, PARTITION xxxxxxxxx3 ((pe_id >= 5200000000 ) AND (pe_id < 5400000000 ) ) IN xxxxxxxxx3, PARTITION xxxxxxxxx21 ((pe_id >= 5400000000 ) AND (pe_id < 5600000000 ) ) IN xxxxxxxxx21, PARTITION xxxxxxxxx2 ((pe_id >= 5600000000 ) AND (pe_id < 5800000000 ) ) IN xxxxxxxxx2, PARTITION xxxxxxxxx1 ((pe_id >= 5800000000 ) AND (pe_id < 6000000000 ) ) IN xxxxxxxxx1, PARTITION xxxxxxxxx10 ((pe_id >= 6000000000 ) AND (pe_id < 6200000000 ) ) IN xxxxxxxxx10, PARTITION xxxxxxxxx15 ((pe_id >= 6200000000 ) AND (pe_id < 6400000000 ) ) IN xxxxxxxxx15, PARTITION xxxxxxxxx9 ((pe_id >= 6400000000 ) AND (pe_id < 6600000000 ) ) IN xxxxxxxxx9, PARTITION xxxxxxxxx8 ((pe_id >= 6600000000 ) AND (pe_id < 6800000000 ) ) IN xxxxxxxxx8, PARTITION xxxxxxxxx7 ((pe_id >= 6800000000 ) AND (pe_id < 7000000000 ) ) IN xxxxxxxxx7, PARTITION xxxxxxxxx6 ((pe_id >= 7000000000 ) AND (pe_id < 7200000000 ) ) IN xxxxxxxxx6, PARTITION xxxxxxxxx20 ((pe_id >= 7200000000 ) AND (pe_id < 7400000000 ) ) IN xxxxxxxxx20, PARTITION xxxxxxxxx19 ((pe_id >= 7400000000 ) AND (pe_id < 7600000000 ) ) IN xxxxxxxxx19, PARTITION xxxxxxxxx18 ((pe_id >= 7600000000 ) AND (pe_id < 7800000000 ) ) IN xxxxxxxxx18, PARTITION xxxxxxxxx17 ((pe_id >= 7800000000 ) AND (pe_id < 8000000000 ) ) IN xxxxxxxxx17, PARTITION xxxxxxxxx16 ((pe_id >= 8000000000 ) AND (pe_id < 8200000000 ) ) IN xxxxxxxxx16, PARTITION xxxxxxxxx4 ((pe_id >= 8200000000 ) AND (pe_id < 8400000000 ) ) IN xxxxxxxxx4, PARTITION xxxxxxxxx32 ((pe_id >= 8400000000 ) AND (pe_id < 8600000000 ) ) IN xxxxxxxxx32, PARTITION xxxxxxxxx33 ((pe_id >= 8600000000 ) AND (pe_id < 8800000000 ) ) IN xxxxxxxxx33, PARTITION xxxxxxxxx34 ((pe_id >= 8800000000 ) AND (pe_id < 9000000000 ) ) IN xxxxxxxxx34, PARTITION xxxxxxxxx35 ((pe_id >= 9000000000 ) AND (pe_id < 9200000000 ) ) IN xxxxxxxxx35, PARTITION xxxxxxxxx36 ((pe_id >= 9200000000 ) AND (pe_id < 9400000000 ) ) IN xxxxxxxxx36, PARTITION xxxxxxxxx37 ((pe_id >= 9400000000 ) AND (pe_id < 9600000000 ) ) IN xxxxxxxxx37, PARTITION xxxxxxxxx38 ((pe_id >= 9600000000 ) AND (pe_id < 9800000000 ) ) IN xxxxxxxxx38, PARTITION xxxxxxxxx39 ((pe_id >= 9800000000 ) AND (pe_id < 10000000000 ) ) IN xxxxxxxxx39, PARTITION xxxxxxxxx40 ((pe_id >= 10000000000 ) AND (pe_id < 10200000000 ) ) IN xxxxxxxxx40, PARTITION xxxxxxxxx41 ((pe_id >= 10200000000 ) AND (pe_id < 10400000000 ) ) IN xxxxxxxxx41, PARTITION xxxxxxxxx42 ((pe_id >= 10400000000 ) AND (pe_id < 10600000000 ) ) IN xxxxxxxxx42, PARTITION xxxxxxxxx43 ((pe_id >= 10600000000 ) AND (pe_id < 10800000000 ) ) IN xxxxxxxxx43, PARTITION xxxxxxxxx44 ((pe_id >= 10800000000 ) AND (pe_id < 11000000000 ) ) IN xxxxxxxxx44, PARTITION xxxxxxxxx45 ((pe_id >= 11000000000 ) AND (pe_id < 11200000000 ) ) IN xxxxxxxxx45, PARTITION xxxxxxxxx46 ((pe_id >= 11200000000 ) AND (pe_id < 11400000000 ) ) IN xxxxxxxxx46, PARTITION xxxxxxxxx47 ((pe_id >= 11400000000 ) AND (pe_id < 11600000000 ) ) IN xxxxxxxxx47, PARTITION xxxxxxxxx48 ((pe_id >= 11600000000 ) AND (pe_id < 11800000000 ) ) IN xxxxxxxxx48, PARTITION xxxxxxxxx49 ((pe_id >= 11800000000 ) AND (pe_id < 12000000000 ) ) IN xxxxxxxxx49, PARTITION xxxxxxxxx50 ((pe_id >= 12000000000 ) AND (pe_id < 12200000000 ) ) IN xxxxxxxxx50, PARTITION xxxxxxxxx51 ((pe_id >= 12200000000 ) AND (pe_id < 12400000000 ) ) IN xxxxxxxxx51, PARTITION xxxxxxxxx52 ((pe_id >= 12400000000 ) AND (pe_id < 12600000000 ) ) IN xxxxxxxxx52, PARTITION xxxxxxxxx53 ((pe_id >= 12600000000 ) AND (pe_id < 12800000000 ) ) IN xxxxxxxxx53, PARTITION xxxxxxxxx54 ((pe_id >= 12800000000 ) AND (pe_id < 13000000000 ) ) IN xxxxxxxxx54, PARTITION xxxxxxxxx55 ((pe_id >= 13000000000 ) AND (pe_id < 13200000000 ) ) IN xxxxxxxxx55, PARTITION xxxxxxxxx12 ((pe_id >= 4600000000 ) AND (pe_id < 4800000000 ) ) IN xxxxxxxxx12, PARTITION xxxxxxxxx56 ((pe_id >= 13200000000 ) AND (pe_id < 13400000000 ) ) IN xxxxxxxxx56, PARTITION xxxxxxxxx57 ((pe_id >= 13400000000 ) AND (pe_id < 13600000000 ) ) IN xxxxxxxxx57, PARTITION xxxxxxxxx58 ((pe_id >= 13600000000 ) AND (pe_id < 13800000000 ) ) IN xxxxxxxxx58, PARTITION xxxxxxxxx59 ((pe_id >= 13800000000 ) AND (pe_id < 14000000000 ) ) IN xxxxxxxxx59, PARTITION xxxxxxxxx60 ((pe_id >= 14000000000 ) AND (pe_id < 14200000000 ) ) IN xxxxxxxxx60, PARTITION xxxxxxxxx61 ((pe_id >= 14200000000 ) AND (pe_id < 14400000000 ) ) IN xxxxxxxxx61, PARTITION xxxxxxxxx62 ((pe_id >= 14400000000 ) AND (pe_id < 14600000000 ) ) IN xxxxxxxxx62, PARTITION xxxxxxxxx63 ((pe_id >= 14600000000 ) AND (pe_id < 14800000000 ) ) IN xxxxxxxxx63

Reinhard: Welllllll, you've got a table that, at least on the surface, seems to have on the order of 10^10 rows and you don't have PDQPRIORITY or PSORT_NPROCS set so this is a single threaded operation on a table that has well over 30 partitions (I keep losing count). It's just going to take time. You have -X set... I'd like to see the sqexplain file lines from this run (you can send these to me directly so as not to clog the list. Art Art S. Kagel, President and Principal Consultant ASK Database Management www.askdbmgt.com Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Dec 9, 2016 at 7:41 AM, Habichtsberg, Reinhard < RHabichtsberg@arz-emmendingen.de> wrote: > Hi all > > Informix 12.10.FC4W1XB, Solaris 10 > > I'm running dostats on a very big table. It seems it won't come to an end. > Runs now since Monday, the 5th of December. > > Table myschema: > > CREATE TABLE "informix".xxxxxxxxxxxxx ( > > pe_id SERIAL8(15712838227) NOT NULL, > > pe_re INTEGER, > > pe_re_lfd SMALLINT, > > pe_vo SMALLINT, > > pe_pos SMALLINT, > > pe_ebene CHAR(2), > > pe_pruefung SMALLINT, > > pe_pruefstart DATETIME YEAR TO SECOND, > > pe_pruefende DATETIME YEAR TO SECOND, > > pe_pruefart CHAR(1), > > pe_bearbeiter CHAR(8), > > pe_wfstatusvor DECIMAL(8,4), > > pe_wfstatusnach DECIMAL(8,4), > > pe_ergx SMALLINT, > > pe_ergxart CHAR(1), > > pe_berinr INTEGER, > > pe_textnr INTEGER, > > pe_folgeberinr INTEGER, > > pe_jobid INTEGER, > > pe_jobart CHAR(1), > > pe_pruefzeit DECIMAL(12,2), > > pe_pruefzeit_ges DECIMAL(12,2), > > pe_wfnr SMALLINT, > > pe_nachbeding SMALLINT, > > pe_folge_ga INTEGER, > > pe_lfdnr_verf INTEGER > ) FRAGMENT BY EXPRESSION > > PARTITION xxxxxxxxx23 ((pe_id >= 2200000000 ) AND (pe_id < 2400000000 ) ) > IN > xxxxxxxxx23, > > PARTITION xxxxxxxxx24 ((pe_id >= 2400000000 ) AND (pe_id < 2600000000 ) ) > IN > xxxxxxxxx24, > > PARTITION xxxxxxxxx25 ((pe_id >= 2600000000 ) AND (pe_id < 2800000000 ) ) > IN > xxxxxxxxx25, > > PARTITION xxxxxxxxx26 ((pe_id >= 2800000000 ) AND (pe_id < 3000000000 ) ) > IN > xxxxxxxxx26, > > PARTITION xxxxxxxxx27 ((pe_id >= 3000000000 ) AND (pe_id < 3200000000 ) ) > IN > xxxxxxxxx27, > > PARTITION xxxxxxxxx28 ((pe_id >= 3200000000 ) AND (pe_id < 3400000000 ) ) > IN > xxxxxxxxx28, > > PARTITION xxxxxxxxx29 ((pe_id >= 3400000000 ) AND (pe_id < 3600000000 ) ) > IN > xxxxxxxxx29, > PARTITION xxxxxxxxx30 ((pe_id >= 3600000000 ) AND (pe_id < 3800000000 ) ) > IN > xxxxxxxxx30, > > PARTITION xxxxxxxxx31 ((pe_id >= 3800000000 ) AND (pe_id < 4000000000 ) ) > IN > xxxxxxxxx31, > > PARTITION xxxxxxxxx14 ((pe_id >= 4000000000 ) AND (pe_id < 4200000000 ) ) > IN > xxxxxxxxx14, > > PARTITION xxxxxxxxx5 ((pe_id >= 4200000000 ) AND (pe_id < 4400000000 ) ) IN > xxxxxxxxx5, > > PARTITION xxxxxxxxx13 ((pe_id >= 4400000000 ) AND (pe_id < 4600000000 ) ) > IN > xxxxxxxxx13, > > PARTITION xxxxxxxxx11 ((pe_id >= 4800000000 ) AND (pe_id < 5000000000 ) ) > IN > xxxxxxxxx11, > > PARTITION xxxxxxxxx22 ((pe_id >= 5000000000 ) AND (pe_id < 5200000000 ) ) > IN > xxxxxxxxx22, > > PARTITION xxxxxxxxx3 ((pe_id >= 5200000000 ) AND (pe_id < 5400000000 ) ) IN > xxxxxxxxx3, > > PARTITION xxxxxxxxx21 ((pe_id >= 5400000000 ) AND (pe_id < 5600000000 ) ) > IN > xxxxxxxxx21, > > PARTITION xxxxxxxxx2 ((pe_id >= 5600000000 ) AND (pe_id < 5800000000 ) ) IN > xxxxxxxxx2, > > PARTITION xxxxxxxxx1 ((pe_id >= 5800000000 ) AND (pe_id < 6000000000 ) ) IN > xxxxxxxxx1, > > PARTITION xxxxxxxxx10 ((pe_id >= 6000000000 ) AND (pe_id < 6200000000 ) ) > IN > xxxxxxxxx10, > > PARTITION xxxxxxxxx15 ((pe_id >= 6200000000 ) AND (pe_id < 6400000000 ) ) > IN > xxxxxxxxx15, > > PARTITION xxxxxxxxx9 ((pe_id >= 6400000000 ) AND (pe_id < 6600000000 ) ) IN > xxxxxxxxx9, > > PARTITION xxxxxxxxx8 ((pe_id >= 6600000000 ) AND (pe_id < 6800000000 ) ) IN > xxxxxxxxx8, > > PARTITION xxxxxxxxx7 ((pe_id >= 6800000000 ) AND (pe_id < 7000000000 ) ) IN > xxxxxxxxx7, > > PARTITION xxxxxxxxx6 ((pe_id >= 7000000000 ) AND (pe_id < 7200000000 ) ) IN > xxxxxxxxx6, > > PARTITION xxxxxxxxx20 ((pe_id >= 7200000000 ) AND (pe_id < 7400000000 ) ) > IN > xxxxxxxxx20, > > PARTITION xxxxxxxxx19 ((pe_id >= 7400000000 ) AND (pe_id < 7600000000 ) ) > IN > xxxxxxxxx19, > > PARTITION xxxxxxxxx18 ((pe_id >= 7600000000 ) AND (pe_id < 7800000000 ) ) > IN > xxxxxxxxx18, > > PARTITION xxxxxxxxx17 ((pe_id >= 7800000000 ) AND (pe_id < 8000000000 ) ) > IN > xxxxxxxxx17, > > PARTITION xxxxxxxxx16 ((pe_id >= 8000000000 ) AND (pe_id < 8200000000 ) ) > IN > xxxxxxxxx16, > > PARTITION xxxxxxxxx4 ((pe_id >= 8200000000 ) AND (pe_id < 8400000000 ) ) IN > xxxxxxxxx4, > > PARTITION xxxxxxxxx32 ((pe_id >= 8400000000 ) AND (pe_id < 8600000000 ) ) > IN > xxxxxxxxx32, > > PARTITION xxxxxxxxx33 ((pe_id >= 8600000000 ) AND (pe_id < 8800000000 ) ) > IN > xxxxxxxxx33, > > PARTITION xxxxxxxxx34 ((pe_id >= 8800000000 ) AND (pe_id < 9000000000 ) ) > IN > xxxxxxxxx34, > > PARTITION xxxxxxxxx35 ((pe_id >= 9000000000 ) AND (pe_id < 9200000000 ) ) > IN > xxxxxxxxx35, > > PARTITION xxxxxxxxx36 ((pe_id >= 9200000000 ) AND (pe_id < 9400000000 ) ) > IN > xxxxxxxxx36, > > PARTITION xxxxxxxxx37 ((pe_id >= 9400000000 ) AND (pe_id < 9600000000 ) ) > IN > xxxxxxxxx37, > > PARTITION xxxxxxxxx38 ((pe_id >= 9600000000 ) AND (pe_id < 9800000000 ) ) > IN > xxxxxxxxx38, > > PARTITION xxxxxxxxx39 ((pe_id >= 9800000000 ) AND (pe_id < 10000000000 ) ) > IN > xxxxxxxxx39, > > PARTITION xxxxxxxxx40 ((pe_id >= 10000000000 ) AND (pe_id < 10200000000 ) > ) IN > xxxxxxxxx40, > > PARTITION xxxxxxxxx41 ((pe_id >= 10200000000 ) AND (pe_id < 10400000000 ) > ) IN > xxxxxxxxx41, > > PARTITION xxxxxxxxx42 ((pe_id >= 10400000000 ) AND (pe_id < 10600000000 ) > ) IN > xxxxxxxxx42, > > PARTITION xxxxxxxxx43 ((pe_id >= 10600000000 ) AND (pe_id < 10800000000 ) > ) IN > xxxxxxxxx43, > > PARTITION xxxxxxxxx44 ((pe_id >= 10800000000 ) AND (pe_id < 11000000000 ) > ) IN > xxxxxxxxx44, > > PARTITION xxxxxxxxx45 ((pe_id >= 11000000000 ) AND (pe_id < 11200000000 ) > ) IN > xxxxxxxxx45, > > PARTITION xxxxxxxxx46 ((pe_id >= 11200000000 ) AND (pe_id < 11400000000 ) > ) IN > xxxxxxxxx46, > > PARTITION xxxxxxxxx47 ((pe_id >= 11400000000 ) AND (pe_id < 11600000000 ) > ) IN > xxxxxxxxx47, > > PARTITION xxxxxxxxx48 ((pe_id >= 11600000000 ) AND (pe_id <

Art: There are only a few lines in sqexplain.out: UPDATE STATISTICS:================== Table: informix.xxxxxxxxx Mode: HIGH Number of Bins: 267 Bin size 67362549 Sort data 964908.6 MB Sort memory granted 15.0 MB Estimated number of table scans 5 PASS #1 pe_id PASS #2 pe_bearbeiter PASS #3 pe_pruefstart PASS #4 pe_wfstatusvor PASS #5 pe_re Scan 0 Sort 0 Build 34591 Insert 0 Close 0 Total 34591 Completed pass 1 in 576 minutes 31 seconds Scan 0 Sort 0 Build 141308 Insert 0 Close 0 Total 141308 Completed pass 2 in 2355 minutes 8 seconds Scan 0 Sort 0 Build 36106 Insert 0 Close 0 Total 36106 Completed pass 3 in 601 minutes 46 seconds Btw: The table has 71 partitions. The table is always heavily used and has about seven million of write ops a day. Furthermore the table is part of many queries. Thank you, Reinhard. -----Ursprüngliche Nachricht----- Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag von Art Kagel Gesendet: Freitag, 9. Dezember 2016 14:34 An: ids@iiug.org Betreff: Re: Long lasting dostats [38274] Reinhard: Welllllll, you've got a table that, at least on the surface, seems to have on the order of 10^10 rows and you don't have PDQPRIORITY or PSORT_NPROCS set so this is a single threaded operation on a table that has well over 30 partitions (I keep losing count). It's just going to take time. You have -X set... I'd like to see the sqexplain file lines from this run (you can send these to me directly so as not to clog the list. Art Art S. Kagel, President and Principal Consultant ASK Database Management www.askdbmgt.com Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Dec 9, 2016 at 7:41 AM, Habichtsberg, Reinhard < RHabichtsberg@arz-emmendingen.de> wrote: > Hi all > > Informix 12.10.FC4W1XB, Solaris 10 > > I'm running dostats on a very big table. It seems it won't come to an end. > Runs now since Monday, the 5th of December. > > Table myschema: > > CREATE TABLE "informix".xxxxxxxxxxxxx ( > > pe_id SERIAL8(15712838227) NOT NULL, > > pe_re INTEGER, > > pe_re_lfd SMALLINT, > > pe_vo SMALLINT, > > pe_pos SMALLINT, > > pe_ebene CHAR(2), > > pe_pruefung SMALLINT, > > pe_pruefstart DATETIME YEAR TO SECOND, > > pe_pruefende DATETIME YEAR TO SECOND, > > pe_pruefart CHAR(1), > > pe_bearbeiter CHAR(8), > > pe_wfstatusvor DECIMAL(8,4), > > pe_wfstatusnach DECIMAL(8,4), > > pe_ergx SMALLINT, > > pe_ergxart CHAR(1), > > pe_berinr INTEGER, > > pe_textnr INTEGER, > > pe_folgeberinr INTEGER, > > pe_jobid INTEGER, > > pe_jobart CHAR(1), > > pe_pruefzeit DECIMAL(12,2), > > pe_pruefzeit_ges DECIMAL(12,2), > > pe_wfnr SMALLINT, > > pe_nachbeding SMALLINT, > > pe_folge_ga INTEGER, > > pe_lfdnr_verf INTEGER > ) FRAGMENT BY EXPRESSION > > PARTITION xxxxxxxxx23 ((pe_id >= 2200000000 ) AND (pe_id < 2400000000 > ) ) IN xxxxxxxxx23, > > PARTITION xxxxxxxxx24 ((pe_id >= 2400000000 ) AND (pe_id < 2600000000 > ) ) IN xxxxxxxxx24, > > PARTITION xxxxxxxxx25 ((pe_id >= 2600000000 ) AND (pe_id < 2800000000 > ) ) IN xxxxxxxxx25, > > PARTITION xxxxxxxxx26 ((pe_id >= 2800000000 ) AND (pe_id < 3000000000 > ) ) IN xxxxxxxxx26, > > PARTITION xxxxxxxxx27 ((pe_id >= 3000000000 ) AND (pe_id < 3200000000 > ) ) IN xxxxxxxxx27, > > PARTITION xxxxxxxxx28 ((pe_id >= 3200000000 ) AND (pe_id < 3400000000 > ) ) IN xxxxxxxxx28, > > PARTITION xxxxxxxxx29 ((pe_id >= 3400000000 ) AND (pe_id < 3600000000 > ) ) IN xxxxxxxxx29, PARTITION xxxxxxxxx30 ((pe_id >= 3600000000 ) AND > (pe_id < 3800000000 ) ) IN xxxxxxxxx30, > > PARTITION xxxxxxxxx31 ((pe_id >= 3800000000 ) AND (pe_id < 4000000000 > ) ) IN xxxxxxxxx31, > > PARTITION xxxxxxxxx14 ((pe_id >= 4000000000 ) AND (pe_id < 4200000000 > ) ) IN xxxxxxxxx14, > > PARTITION xxxxxxxxx5 ((pe_id >= 4200000000 ) AND (pe_id < 4400000000 ) > ) IN xxxxxxxxx5, > > PARTITION xxxxxxxxx13 ((pe_id >= 4400000000 ) AND (pe_id < 4600000000 > ) ) IN xxxxxxxxx13, > > PARTITION xxxxxxxxx11 ((pe_id >= 4800000000 ) AND (pe_id < 5000000000 > ) ) IN xxxxxxxxx11, > > PARTITION xxxxxxxxx22 ((pe_id >= 5000000000 ) AND (pe_id < 5200000000 > ) ) IN xxxxxxxxx22, > > PARTITION xxxxxxxxx3 ((pe_id >= 5200000000 ) AND (pe_id < 5400000000 ) > ) IN xxxxxxxxx3, > > PARTITION xxxxxxxxx21 ((pe_id >= 5400000000 ) AND (pe_id < 5600000000 > ) ) IN xxxxxxxxx21, > > PARTITION xxxxxxxxx2 ((pe_id >= 5600000000 ) AND (pe_id < 5800000000 ) > ) IN xxxxxxxxx2, > > PARTITION xxxxxxxxx1 ((pe_id >= 5800000000 ) AND (pe_id < 6000000000 ) > ) IN xxxxxxxxx1, > > PARTITION xxxxxxxxx10 ((pe_id >= 6000000000 ) AND (pe_id < 6200000000 > ) ) IN xxxxxxxxx10, > > PARTITION xxxxxxxxx15 ((pe_id >= 6200000000 ) AND (pe_id < 6400000000 > ) ) IN xxxxxxxxx15, > > PARTITION xxxxxxxxx9 ((pe_id >= 6400000000 ) AND (pe_id < 6600000000 ) > ) IN xxxxxxxxx9, > > PARTITION xxxxxxxxx8 ((pe_id >= 6600000000 ) AND (pe_id < 6800000000 ) > ) IN xxxxxxxxx8, > > PARTITION xxxxxxxxx7 ((pe_id >= 6800000000 ) AND (pe_id < 7000000000 ) > ) IN xxxxxxxxx7, > > PARTITION xxxxxxxxx6 ((pe_id >= 7000000000 ) AND (pe_id < 7200000000 ) > ) IN xxxxxxxxx6, > > PARTITION xxxxxxxxx20 ((pe_id >= 7200000000 ) AND (pe_id < 7400000000 > ) ) IN xxxxxxxxx20, > > PARTITION xxxxxxxxx19 ((pe_id >= 7400000000 ) AND (pe_id < 7600000000 > ) ) IN xxxxxxxxx19, > > PARTITION xxxxxxxxx18 ((pe_id >= 7600000000 ) AND (pe_id < 7800000000 > ) ) IN xxxxxxxxx18, > > PARTITION xxxxxxxxx17 ((pe_id >= 7800000000 ) AND (pe_id < 8000000000 > ) ) IN xxxxxxxxx17, > > PARTITION xxxxxxxxx16 ((pe_id >= 8000000000 ) AND (pe_id < 8200000000 > ) ) IN xxxxxxxxx16, > > PARTITION xxxxxxxxx4 ((pe_id >= 8200000000 ) AND (pe_id < 8400000000 ) > ) IN xxxxxxxxx4, > > PARTITION xxxxxxxxx32 ((pe_id >= 8400000000 ) AND (pe_id < 8600000000 > ) ) IN xxxxxxxxx32, > > PARTITION xxxxxxxxx33 ((pe_id >= 8600000000 ) AND (pe_id < 8800000000 > ) ) IN xxxxxxxxx33, > > PARTITION xxxxxxxxx34 ((pe_id >= 8800000000 ) AND (pe_id < 9000000000 > ) ) IN xxxxxxxxx34, > > PARTITION xxxxxxxxx35 ((pe_id >= 9000000000 ) AND (pe_id < 9200000000 > ) ) IN xxxxxxxxx35, > > PARTITION xxxxxxxxx36 ((pe_id >= 9200000000 ) AND (pe_id < 9400000000 > ) ) IN xxxxxxxxx36, > > PARTITION xxxxxxxxx37 ((pe_id >= 9400000000 ) AND (pe_id < 9600000000 > ) ) IN xxxxxxxxx37, > > PARTITION xxxxxxxxx38 ((pe_id >= 9600000000 ) AND (pe_id < 9800000000 > ) ) IN xxxxxxxxx38, > > PARTITION xxxxxxxxx39 ((pe_id >= 9800000000 ) AND (pe_id < 10000000000 > ) ) IN xxxxxxxxx39, > > P

OK, here's what's going on: - You didn't turn on PDQPRIORITY so the update statistics high command has to scan all 71 partitions sequentially. - You didn't increase the amount of memory allocated for sorting so it only has 15MB available to sort 5 columns of keys totaling 960MB five different ways (once ordering by each key) - DS_NONPDQ_QUERY_MEM without PDQ or DS_TOTAL_MEMORY and DS_MAX_QUERIES with PDQPRIORITY > 0 - You didn't enable PSORT_NPROCS so the non-PDQ sorting is also single threaded. At the time you extracted the sqexplain output the update statistics command was beginning to process the 4th sort of 5, so your between 60% & 80% done give or take a few percentages if the different keys are of significantly different sizes. Art Art S. Kagel, President and Principal Consultant ASK Database Management www.askdbmgt.com Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Dec 9, 2016 at 10:22 AM, Habichtsberg, Reinhard < RHabichtsberg@arz-emmendingen.de> wrote: > Art: > > There are only a few lines in sqexplain.out: > > UPDATE STATISTICS:> ================== > > Table: informix.xxxxxxxxx > Mode: HIGH > Number of Bins: 267 Bin size 67362549 > Sort data 964908.6 MB Sort memory granted 15.0 MB > Estimated number of table scans 5 > PASS #1 pe_id > PASS #2 pe_bearbeiter > PASS #3 pe_pruefstart > PASS #4 pe_wfstatusvor > PASS #5 pe_re > Scan 0 Sort 0 Build 34591 Insert 0 Close 0 Total 34591 > Completed pass 1 in 576 minutes 31 seconds > Scan 0 Sort 0 Build 141308 Insert 0 Close 0 Total 141308 > Completed pass 2 in 2355 minutes 8 seconds > Scan 0 Sort 0 Build 36106 Insert 0 Close 0 Total 36106 > Completed pass 3 in 601 minutes 46 seconds > > Btw: The table has 71 partitions. The table is always heavily used and has > about seven million of write ops a day. Furthermore the table is part of > many > queries. > > Thank you, > Reinhard. > > -----Ursprüngliche Nachricht----- > Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag von Art > Kagel > Gesendet: Freitag, 9. Dezember 2016 14:34 > An: ids@iiug.org > Betreff: Re: Long lasting dostats [38274] > > Reinhard: > > Welllllll, you've got a table that, at least on the surface, seems to have > on > the order of 10^10 rows and you don't have PDQPRIORITY or PSORT_NPROCS set > so > this is a single threaded operation on a table that has well over 30 > partitions (I keep losing count). It's just going to take time. > > You have -X set... I'd like to see the sqexplain file lines from this run > (you > can send these to me directly so as not to clog the list. > > Art > > Art S. Kagel, President and Principal Consultant ASK Database Management > www.askdbmgt.com > > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and > do not reflect on the IIUG, nor any other organization with which I am > associated either explicitly, implicitly, or by inference. Neither do those > opinions reflect those of other individuals affiliated with any entity with > which I am affiliated nor those of the entities themselves. > > On Fri, Dec 9, 2016 at 7:41 AM, Habichtsberg, Reinhard < > RHabichtsberg@arz-emmendingen.de> wrote: > > > Hi all > > > > Informix 12.10.FC4W1XB, Solaris 10 > > > > I'm running dostats on a very big table. It seems it won't come to an > end. > > Runs now since Monday, the 5th of December. > > > > Table myschema: > > > > CREATE TABLE "informix".xxxxxxxxxxxxx ( > > > > pe_id SERIAL8(15712838227) NOT NULL, > > > > pe_re INTEGER, > > > > pe_re_lfd SMALLINT, > > > > pe_vo SMALLINT, > > > > pe_pos SMALLINT, > > > > pe_ebene CHAR(2), > > > > pe_pruefung SMALLINT, > > > > pe_pruefstart DATETIME YEAR TO SECOND, > > > > pe_pruefende DATETIME YEAR TO SECOND, > > > > pe_pruefart CHAR(1), > > > > pe_bearbeiter CHAR(8), > > > > pe_wfstatusvor DECIMAL(8,4), > > > > pe_wfstatusnach DECIMAL(8,4), > > > > pe_ergx SMALLINT, > > > > pe_ergxart CHAR(1), > > > > pe_berinr INTEGER, > > > > pe_textnr INTEGER, > > > > pe_folgeberinr INTEGER, > > > > pe_jobid INTEGER, > > > > pe_jobart CHAR(1), > > > > pe_pruefzeit DECIMAL(12,2), > > > > pe_pruefzeit_ges DECIMAL(12,2), > > > > pe_wfnr SMALLINT, > > > > pe_nachbeding SMALLINT, > > > > pe_folge_ga INTEGER, > > > > pe_lfdnr_verf INTEGER > > ) FRAGMENT BY EXPRESSION > > > > PARTITION xxxxxxxxx23 ((pe_id >= 2200000000 ) AND (pe_id < 2400000000 > > ) ) IN xxxxxxxxx23, > > > > PARTITION xxxxxxxxx24 ((pe_id >= 2400000000 ) AND (pe_id < 2600000000 > > ) ) IN xxxxxxxxx24, > > > > PARTITION xxxxxxxxx25 ((pe_id >= 2600000000 ) AND (pe_id < 2800000000 > > ) ) IN xxxxxxxxx25, > > > > PARTITION xxxxxxxxx26 ((pe_id >= 2800000000 ) AND (pe_id < 3000000000 > > ) ) IN xxxxxxxxx26, > > > > PARTITION xxxxxxxxx27 ((pe_id >= 3000000000 ) AND (pe_id < 3200000000 > > ) ) IN xxxxxxxxx27, > > > > PARTITION xxxxxxxxx28 ((pe_id >= 3200000000 ) AND (pe_id < 3400000000 > > ) ) IN xxxxxxxxx28, > > > > PARTITION xxxxxxxxx29 ((pe_id >= 3400000000 ) AND (pe_id < 3600000000 > > ) ) IN xxxxxxxxx29, PARTITION xxxxxxxxx30 ((pe_id >= 3600000000 ) AND > > (pe_id < 3800000000 ) ) IN xxxxxxxxx30, > > > > PARTITION xxxxxxxxx31 ((pe_id >= 3800000000 ) AND (pe_id < 4000000000 > > ) ) IN xxxxxxxxx31, > > > > PARTITION xxxxxxxxx14 ((pe_id >= 4000000000 ) AND (pe_id < 4200000000 > > ) ) IN xxxxxxxxx14, > > > > PARTITION xxxxxxxxx5 ((pe_id >= 4200000000 ) AND (pe_id < 4400000000 ) > > ) IN xxxxxxxxx5, > > > > PARTITION xxxxxxxxx13 ((pe_id >= 4400000000 ) AND (pe_id < 4600000000 > > ) ) IN xxxxxxxxx13, > > > > PARTITION xxxxxxxxx11 ((pe_id >= 4800000000 ) AND (pe_id < 5000000000 > > ) ) IN xxxxxxxxx11, > > > > PARTITION xxxxxxxxx22 ((pe_id >= 5000000000 ) AND (pe_id < 5200000000 > > ) ) IN xxxxxxxxx22, > > > > PARTITION xxxxxxxxx3 ((pe_id >= 5200000000 ) AND (pe_id < 5400000000 ) > > ) IN xxxxxxxxx3, > > > > PARTITION xxxxxxxxx21 ((pe_id >= 5400000000 ) AND (pe_id < 5600000000 > > ) ) IN xxxxxxxxx21, > > > > PARTITION xxxxxxxxx2 ((pe_id >= 5600000000 ) AND (pe_id < 5800000000 ) > > ) IN xxxxxxxxx2, > > > > PARTITION xxxxxxxxx1 ((pe_id >= 5800000000 ) AND (pe_id < 6000000000 ) > > ) IN xxxxxxxxx1, > > > > PARTITION xxxxxxxxx10 ((pe_id >= 6000000000 ) AND (pe_id < 6200000000 > > ) ) IN xxxxxxxxx10, > > > > PARTITION xxxxxxxxx15 ((pe_id >= 6200000000 ) AND (pe_id < 6400000000 > > ) ) IN xxxxxxxxx15, > > > > PARTITION xxxxxxxxx9 ((pe_id >= 64

Oops that got away from me. Need to add: This is why dostats prints out those warnings at the beginning if you don't enable parallelism and enough memory. Art Art S. Kagel, President and Principal Consultant ASK Database Management www.askdbmgt.com Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Dec 9, 2016 at 10:59 AM, Art Kagel <art.kagel@gmail.com> wrote: > OK, here's what's going on: > > - You didn't turn on PDQPRIORITY so the update statistics high command > has to scan all 71 partitions sequentially. > - You didn't increase the amount of memory allocated for sorting so it > only has 15MB available to sort 5 columns of keys totaling 960MB five > different ways (once ordering by each key) - DS_NONPDQ_QUERY_MEM without > PDQ or DS_TOTAL_MEMORY and DS_MAX_QUERIES with PDQPRIORITY > 0 > - You didn't enable PSORT_NPROCS so the non-PDQ sorting is also single > threaded. > > At the time you extracted the sqexplain output the update statistics > command was beginning to process the 4th sort of 5, so your between 60% & > 80% done give or take a few percentages if the different keys are of > significantly different sizes. > > Art > > > Art S. Kagel, President and Principal Consultant > ASK Database Management > www.askdbmgt.com > > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and do not reflect on the IIUG, nor any other organization with which I am > associated either explicitly, implicitly, or by inference. Neither do > those opinions reflect those of other individuals affiliated with any > entity with which I am affiliated nor those of the entities themselves. > > > On Fri, Dec 9, 2016 at 10:22 AM, Habichtsberg, Reinhard < > RHabichtsberg@arz-emmendingen.de> wrote: > >> Art: >> >> There are only a few lines in sqexplain.out: >> >> UPDATE STATISTICS:>> ================== >> >> Table: informix.xxxxxxxxx >> Mode: HIGH >> Number of Bins: 267 Bin size 67362549 >> Sort data 964908.6 MB Sort memory granted 15.0 MB >> Estimated number of table scans 5 >> PASS #1 pe_id >> PASS #2 pe_bearbeiter >> PASS #3 pe_pruefstart >> PASS #4 pe_wfstatusvor >> PASS #5 pe_re >> Scan 0 Sort 0 Build 34591 Insert 0 Close 0 Total 34591 >> Completed pass 1 in 576 minutes 31 seconds >> Scan 0 Sort 0 Build 141308 Insert 0 Close 0 Total 141308 >> Completed pass 2 in 2355 minutes 8 seconds >> Scan 0 Sort 0 Build 36106 Insert 0 Close 0 Total 36106 >> Completed pass 3 in 601 minutes 46 seconds >> >> Btw: The table has 71 partitions. The table is always heavily used and has >> about seven million of write ops a day. Furthermore the table is part of >> many >> queries. >> >> Thank you, >> Reinhard. >> >> -----Ursprüngliche Nachricht----- >> Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag von >> Art >> Kagel >> Gesendet: Freitag, 9. Dezember 2016 14:34 >> An: ids@iiug.org >> Betreff: Re: Long lasting dostats [38274] >> >> Reinhard: >> >> Welllllll, you've got a table that, at least on the surface, seems to >> have on >> the order of 10^10 rows and you don't have PDQPRIORITY or PSORT_NPROCS >> set so >> this is a single threaded operation on a table that has well over 30 >> partitions (I keep losing count). It's just going to take time. >> >> You have -X set... I'd like to see the sqexplain file lines from this run >> (you >> can send these to me directly so as not to clog the list. >> >> Art >> >> Art S. Kagel, President and Principal Consultant ASK Database Management >> www.askdbmgt.com >> >> Blog: http://informix-myview.blogspot.com/ >> >> Disclaimer: Please keep in mind that my own opinions are my own opinions >> and >> do not reflect on the IIUG, nor any other organization with which I am >> associated either explicitly, implicitly, or by inference. Neither do >> those >> opinions reflect those of other individuals affiliated with any entity >> with >> which I am affiliated nor those of the entities themselves. >> >> On Fri, Dec 9, 2016 at 7:41 AM, Habichtsberg, Reinhard < >> RHabichtsberg@arz-emmendingen.de> wrote: >> >> > Hi all >> > >> > Informix 12.10.FC4W1XB, Solaris 10 >> > >> > I'm running dostats on a very big table. It seems it won't come to an >> end. >> > Runs now since Monday, the 5th of December. >> > >> > Table myschema: >> > >> > CREATE TABLE "informix".xxxxxxxxxxxxx ( >> > >> > pe_id SERIAL8(15712838227) NOT NULL, >> > >> > pe_re INTEGER, >> > >> > pe_re_lfd SMALLINT, >> > >> > pe_vo SMALLINT, >> > >> > pe_pos SMALLINT, >> > >> > pe_ebene CHAR(2), >> > >> > pe_pruefung SMALLINT, >> > >> > pe_pruefstart DATETIME YEAR TO SECOND, >> > >> > pe_pruefende DATETIME YEAR TO SECOND, >> > >> > pe_pruefart CHAR(1), >> > >> > pe_bearbeiter CHAR(8), >> > >> > pe_wfstatusvor DECIMAL(8,4), >> > >> > pe_wfstatusnach DECIMAL(8,4), >> > >> > pe_ergx SMALLINT, >> > >> > pe_ergxart CHAR(1), >> > >> > pe_berinr INTEGER, >> > >> > pe_textnr INTEGER, >> > >> > pe_folgeberinr INTEGER, >> > >> > pe_jobid INTEGER, >> > >> > pe_jobart CHAR(1), >> > >> > pe_pruefzeit DECIMAL(12,2), >> > >> > pe_pruefzeit_ges DECIMAL(12,2), >> > >> > pe_wfnr SMALLINT, >> > >> > pe_nachbeding SMALLINT, >> > >> > pe_folge_ga INTEGER, >> > >> > pe_lfdnr_verf INTEGER >> > ) FRAGMENT BY EXPRESSION >> > >> > PARTITION xxxxxxxxx23 ((pe_id >= 2200000000 ) AND (pe_id < 2400000000 >> > ) ) IN xxxxxxxxx23, >> > >> > PARTITION xxxxxxxxx24 ((pe_id >= 2400000000 ) AND (pe_id < 2600000000 >> > ) ) IN xxxxxxxxx24, >> > >> > PARTITION xxxxxxxxx25 ((pe_id >= 2600000000 ) AND (pe_id < 2800000000 >> > ) ) IN xxxxxxxxx25, >> > >> > PARTITION xxxxxxxxx26 ((pe_id >= 2800000000 ) AND (pe_id < 3000000000 >> > ) ) IN xxxxxxxxx26, >> > >> > PARTITION xxxxxxxxx27 ((pe_id >= 3000000000 ) AND (pe_id < 3200000000 >> > ) ) IN xxxxxxxxx27, >> > >> > PARTITION xxxxxxxxx28 ((pe_id >= 3200000000 ) AND (pe_id < 3400000000 >> > ) ) IN xxxxxxxxx28, >> > >> > PARTITION xxxxxxxxx29 ((pe_id >= 3400000000 ) AND (pe_id < 3600000000 >> > ) ) IN xxxxxxxxx29, PARTITION xxxxxxxxx30 ((pe_id >= 3600000000 ) AND >> > (pe_id < 3800000000 ) ) IN xxxxxxxxx30, >> > >> > PARTITION xxxxxxxxx31 ((pe_id >= 3800000000 ) AND (pe_id < 4000000000 >> > ) ) IN xxxxxxxxx31, >> > >> > PARTITION xxxxxxxxx14 ((pe_id >= 4000000000 ) AND (pe_id < 4200000000 >> > ) ) IN xxxxxxxxx14, >> > >> > PARTITION xxxxxxxxx5 ((pe_id >= 4200000000 ) AND (pe_id < 4400000000 ) >> > ) IN xxxxxxxxx5, >> > >> > PARTITION xxxxxxxxx13 ((p

Ok, next time I will give a try to: PSORT_NPROCS=50; export PSORT_NPROCS PDQPRIORITY=80; export PDQPRIORITY dostats -d sl2000 -t slda_ergebnis -G -P 0 -f dostats_slda_ergebnis -X -v 2 -E --isolation d with onconfig-params: MAX_PDQPRIORITY 100 DS_MAX_QUERIES 10000 DS_TOTAL_MEMORY 10000000 DS_MAX_SCANS 100 DS_NONPDQ_QUERY_MEM 250000 The currently running update statistics high command lastet about 60 hrs the last time the sqexplain.out where updated. So I'm hopeful we get ready 40 hours later, that is Sunday in the early morning. Thank you and have a good weekend, Reinhard. -----Ursprüngliche Nachricht----- Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag von Art Kagel Gesendet: Freitag, 9. Dezember 2016 17:02 An: ids@iiug.org Betreff: Re: Long lasting dostats [38278] Oops that got away from me. Need to add: This is why dostats prints out those warnings at the beginning if you don't enable parallelism and enough memory. Art Art S. Kagel, President and Principal Consultant ASK Database Management www.askdbmgt.com Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Dec 9, 2016 at 10:59 AM, Art Kagel <art.kagel@gmail.com> wrote: > OK, here's what's going on: > > - You didn't turn on PDQPRIORITY so the update statistics high command > has to scan all 71 partitions sequentially. > - You didn't increase the amount of memory allocated for sorting so it > only has 15MB available to sort 5 columns of keys totaling 960MB five > different ways (once ordering by each key) - DS_NONPDQ_QUERY_MEM > without PDQ or DS_TOTAL_MEMORY and DS_MAX_QUERIES with PDQPRIORITY > 0 > - You didn't enable PSORT_NPROCS so the non-PDQ sorting is also single > threaded. > > At the time you extracted the sqexplain output the update statistics > command was beginning to process the 4th sort of 5, so your between > 60% & 80% done give or take a few percentages if the different keys > are of significantly different sizes. > > Art > > > Art S. Kagel, President and Principal Consultant ASK Database > Management www.askdbmgt.com > > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own > opinions and do not reflect on the IIUG, nor any other organization > with which I am associated either explicitly, implicitly, or by > inference. Neither do those opinions reflect those of other > individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. > > > On Fri, Dec 9, 2016 at 10:22 AM, Habichtsberg, Reinhard < > RHabichtsberg@arz-emmendingen.de> wrote: > >> Art: >> >> There are only a few lines in sqexplain.out: >> >> UPDATE STATISTICS:>> ================== >> >> Table: informix.xxxxxxxxx >> Mode: HIGH >> Number of Bins: 267 Bin size 67362549 Sort data 964908.6 MB Sort >> memory granted 15.0 MB Estimated number of table scans 5 PASS #1 >> pe_id PASS #2 pe_bearbeiter PASS #3 pe_pruefstart PASS #4 >> pe_wfstatusvor PASS #5 pe_re Scan 0 Sort 0 Build 34591 Insert 0 Close >> 0 Total 34591 Completed pass 1 in 576 minutes 31 seconds Scan 0 Sort >> 0 Build 141308 Insert 0 Close 0 Total 141308 Completed pass 2 in 2355 >> minutes 8 seconds Scan 0 Sort 0 Build 36106 Insert 0 Close 0 Total >> 36106 Completed pass 3 in 601 minutes 46 seconds >> >> Btw: The table has 71 partitions. The table is always heavily used >> and has about seven million of write ops a day. Furthermore the table >> is part of many queries. >> >> Thank you, >> Reinhard. >> >> -----Ursprüngliche Nachricht----- >> Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag >> von Art Kagel >> Gesendet: Freitag, 9. Dezember 2016 14:34 >> An: ids@iiug.org >> Betreff: Re: Long lasting dostats [38274] >> >> Reinhard: >> >> Welllllll, you've got a table that, at least on the surface, seems to >> have on the order of 10^10 rows and you don't have PDQPRIORITY or >> PSORT_NPROCS set so this is a single threaded operation on a table >> that has well over 30 partitions (I keep losing count). It's just >> going to take time. >> >> You have -X set... I'd like to see the sqexplain file lines from this >> run (you can send these to me directly so as not to clog the list. >> >> Art >> >> Art S. Kagel, President and Principal Consultant ASK Database >> Management www.askdbmgt.com >> >> Blog: http://informix-myview.blogspot.com/ >> >> Disclaimer: Please keep in mind that my own opinions are my own >> opinions and do not reflect on the IIUG, nor any other organization >> with which I am associated either explicitly, implicitly, or by >> inference. Neither do those opinions reflect those of other >> individuals affiliated with any entity with which I am affiliated nor >> those of the entities themselves. >> >> On Fri, Dec 9, 2016 at 7:41 AM, Habichtsberg, Reinhard < >> RHabichtsberg@arz-emmendingen.de> wrote: >> >> > Hi all >> > >> > Informix 12.10.FC4W1XB, Solaris 10 >> > >> > I'm running dostats on a very big table. It seems it won't come to >> > an >> end. >> > Runs now since Monday, the 5th of December. >> > >> > Table myschema: >> > >> > CREATE TABLE "informix".xxxxxxxxxxxxx ( >> > >> > pe_id SERIAL8(15712838227) NOT NULL, >> > >> > pe_re INTEGER, >> > >> > pe_re_lfd SMALLINT, >> > >> > pe_vo SMALLINT, >> > >> > pe_pos SMALLINT, >> > >> > pe_ebene CHAR(2), >> > >> > pe_pruefung SMALLINT, >> > >> > pe_pruefstart DATETIME YEAR TO SECOND, >> > >> > pe_pruefende DATETIME YEAR TO SECOND, >> > >> > pe_pruefart CHAR(1), >> > >> > pe_bearbeiter CHAR(8), >> > >> > pe_wfstatusvor DECIMAL(8,4), >> > >> > pe_wfstatusnach DECIMAL(8,4), >> > >> > pe_ergx SMALLINT, >> > >> > pe_ergxart CHAR(1), >> > >> > pe_berinr INTEGER, >> > >> > pe_textnr INTEGER, >> > >> > pe_folgeberinr INTEGER, >> > >> > pe_jobid INTEGER, >> > >> > pe_jobart CHAR(1), >> > >> > pe_pruefzeit DECIMAL(12,2), >> > >> > pe_pruefzeit_ges DECIMAL(12,2), >> > >> > pe_wfnr SMALLINT, >> > >> > pe_nachbeding SMALLINT, >> > >> > pe_folge_ga INTEGER, >> > >> > pe_lfdnr_verf INTEGER >> > ) FRAGMENT BY EXPRESSION >> > >> > PARTITION xxxxxxxxx23 ((pe_id >= 2200000000 ) AND (pe_id < >> > 2400000000 >> > ) ) IN xxxxxxxxx23, >> > >> > PARTITION xxxxxxxxx24 ((pe_id >= 2400000000 ) AND (pe_id < >> > 2600000000 >> > ) ) IN xxxxxxxxx24, >> > >> > PARTITION xxxxxxxxx25 ((pe_id >= 2600000000 ) AND (pe_id < >> > 2800000000 >> > ) ) IN xxxxxxxxx25, >> > >> > PARTITION xxxxxxxxx26 ((pe_id >= 2800000000 ) AND (pe_id < >> > 3000000000

Despite my hope there was no significant progress over the weekend. Not even pass #4 has finished. So I try to kill the run. What about the impact I have to face in a high production system if I start again with the adjustments below? TIA Reinhard. -----Ursprüngliche Nachricht----- Von: Habichtsberg, Reinhard Gesendet: Freitag, 9. Dezember 2016 18:28 An: 'ids@iiug.org' Cc: Habichtsberg, Reinhard Betreff: AW: Long lasting dostats [38278] Ok, next time I will give a try to: PSORT_NPROCS=50; export PSORT_NPROCS PDQPRIORITY=80; export PDQPRIORITY dostats -d sl2000 -t slda_ergebnis -G -P 0 -f dostats_slda_ergebnis -X -v 2 -E --isolation d with onconfig-params: MAX_PDQPRIORITY 100 DS_MAX_QUERIES 10000 DS_TOTAL_MEMORY 10000000 DS_MAX_SCANS 100 DS_NONPDQ_QUERY_MEM 250000 The currently running update statistics high command lastet about 60 hrs the last time the sqexplain.out where updated. So I'm hopeful we get ready 40 hours later, that is Sunday in the early morning. Thank you and have a good weekend, Reinhard. -----Ursprüngliche Nachricht----- Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag von Art Kagel Gesendet: Freitag, 9. Dezember 2016 17:02 An: ids@iiug.org Betreff: Re: Long lasting dostats [38278] Oops that got away from me. Need to add: This is why dostats prints out those warnings at the beginning if you don't enable parallelism and enough memory. Art Art S. Kagel, President and Principal Consultant ASK Database Management www.askdbmgt.com Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, Dec 9, 2016 at 10:59 AM, Art Kagel <art.kagel@gmail.com> wrote: > OK, here's what's going on: > > - You didn't turn on PDQPRIORITY so the update statistics high command > has to scan all 71 partitions sequentially. > - You didn't increase the amount of memory allocated for sorting so it > only has 15MB available to sort 5 columns of keys totaling 960MB five > different ways (once ordering by each key) - DS_NONPDQ_QUERY_MEM > without PDQ or DS_TOTAL_MEMORY and DS_MAX_QUERIES with PDQPRIORITY > 0 > - You didn't enable PSORT_NPROCS so the non-PDQ sorting is also single > threaded. > > At the time you extracted the sqexplain output the update statistics > command was beginning to process the 4th sort of 5, so your between > 60% & 80% done give or take a few percentages if the different keys > are of significantly different sizes. > > Art > > > Art S. Kagel, President and Principal Consultant ASK Database > Management www.askdbmgt.com > > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own > opinions and do not reflect on the IIUG, nor any other organization > with which I am associated either explicitly, implicitly, or by > inference. Neither do those opinions reflect those of other > individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. > > > On Fri, Dec 9, 2016 at 10:22 AM, Habichtsberg, Reinhard < > RHabichtsberg@arz-emmendingen.de> wrote: > >> Art: >> >> There are only a few lines in sqexplain.out: >> >> UPDATE STATISTICS:>> ================== >> >> Table: informix.xxxxxxxxx >> Mode: HIGH >> Number of Bins: 267 Bin size 67362549 Sort data 964908.6 MB Sort >> memory granted 15.0 MB Estimated number of table scans 5 PASS #1 >> pe_id PASS #2 pe_bearbeiter PASS #3 pe_pruefstart PASS #4 >> pe_wfstatusvor PASS #5 pe_re Scan 0 Sort 0 Build 34591 Insert 0 Close >> 0 Total 34591 Completed pass 1 in 576 minutes 31 seconds Scan 0 Sort >> 0 Build 141308 Insert 0 Close 0 Total 141308 Completed pass 2 in 2355 >> minutes 8 seconds Scan 0 Sort 0 Build 36106 Insert 0 Close 0 Total >> 36106 Completed pass 3 in 601 minutes 46 seconds >> >> Btw: The table has 71 partitions. The table is always heavily used >> and has about seven million of write ops a day. Furthermore the table >> is part of many queries. >> >> Thank you, >> Reinhard. >> >> -----Ursprüngliche Nachricht----- >> Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag >> von Art Kagel >> Gesendet: Freitag, 9. Dezember 2016 14:34 >> An: ids@iiug.org >> Betreff: Re: Long lasting dostats [38274] >> >> Reinhard: >> >> Welllllll, you've got a table that, at least on the surface, seems to >> have on the order of 10^10 rows and you don't have PDQPRIORITY or >> PSORT_NPROCS set so this is a single threaded operation on a table >> that has well over 30 partitions (I keep losing count). It's just >> going to take time. >> >> You have -X set... I'd like to see the sqexplain file lines from this >> run (you can send these to me directly so as not to clog the list. >> >> Art >> >> Art S. Kagel, President and Principal Consultant ASK Database >> Management www.askdbmgt.com >> >> Blog: http://informix-myview.blogspot.com/ >> >> Disclaimer: Please keep in mind that my own opinions are my own >> opinions and do not reflect on the IIUG, nor any other organization >> with which I am associated either explicitly, implicitly, or by >> inference. Neither do those opinions reflect those of other >> individuals affiliated with any entity with which I am affiliated nor >> those of the entities themselves. >> >> On Fri, Dec 9, 2016 at 7:41 AM, Habichtsberg, Reinhard < >> RHabichtsberg@arz-emmendingen.de> wrote: >> >> > Hi all >> > >> > Informix 12.10.FC4W1XB, Solaris 10 >> > >> > I'm running dostats on a very big table. It seems it won't come to >> > an >> end. >> > Runs now since Monday, the 5th of December. >> > >> > Table myschema: >> > >> > CREATE TABLE "informix".xxxxxxxxxxxxx ( >> > >> > pe_id SERIAL8(15712838227) NOT NULL, >> > >> > pe_re INTEGER, >> > >> > pe_re_lfd SMALLINT, >> > >> > pe_vo SMALLINT, >> > >> > pe_pos SMALLINT, >> > >> > pe_ebene CHAR(2), >> > >> > pe_pruefung SMALLINT, >> > >> > pe_pruefstart DATETIME YEAR TO SECOND, >> > >> > pe_pruefende DATETIME YEAR TO SECOND, >> > >> > pe_pruefart CHAR(1), >> > >> > pe_bearbeiter CHAR(8), >> > >> > pe_wfstatusvor DECIMAL(8,4), >> > >> > pe_wfstatusnach DECIMAL(8,4), >> > >> > pe_ergx SMALLINT, >> > >> > pe_ergxart CHAR(1), >> > >> > pe_berinr INTEGER, >> > >> > pe_textnr INTEGER, >> > >> > pe_folgeberinr INTEGER, >> > >> > pe_jobid INTEGER, >> > >> > pe_jobart CHAR(1), >> > >> > pe_pruefzeit DECIMAL(12,2), >> > >> > pe_pruefzeit_ges DECIMAL(12,2), >> > >> > pe_wfnr SMALLINT, >> > >> > pe_nachbeding SMALLINT, >> > >> > pe_folge_ga INTEGER, >> > >> > pe_lfdnr_verf INTEGER

Normally I would just recommend doing this at a quiet time, but in your case this table is so big that it's going to overrun any quiet period. It's always a trade off. Do you go for PDQ 80 or 100 to get this done as quickly as possible to limit impact on production or go with PDQ 10 or 20 to limit impact on production by limiting resources and let it run longer. Touch call. No easy answer. If you have a QA system you can test this on to get a handle on time both ways do that. I'd recommend setting USTLOW_SAMPLE but that only helps with the LOW calculations and it seems to be the HIGH run that's taking the most time. Art Art S. Kagel, President and Principal Consultant ASK Database Management www.askdbmgt.com Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Mon, Dec 12, 2016 at 3:46 AM, Habichtsberg, Reinhard < RHabichtsberg@arz-emmendingen.de> wrote: > Despite my hope there was no significant progress over the weekend. Not > even > pass #4 has finished. So I try to kill the run. > > What about the impact I have to face in a high production system if I start > again with the adjustments below? > > TIA > Reinhard. > > -----Ursprüngliche Nachricht----- > Von: Habichtsberg, Reinhard > Gesendet: Freitag, 9. Dezember 2016 18:28 > An: 'ids@iiug.org' > Cc: Habichtsberg, Reinhard > Betreff: AW: Long lasting dostats [38278] > > Ok, next time I will give a try to: > > PSORT_NPROCS=50; export PSORT_NPROCS > PDQPRIORITY=80; export PDQPRIORITY > dostats -d sl2000 -t slda_ergebnis -G -P 0 -f dostats_slda_ergebnis -X -v > 2 -E > --isolation d > > with onconfig-params: > > MAX_PDQPRIORITY 100 > DS_MAX_QUERIES 10000 > DS_TOTAL_MEMORY 10000000 > DS_MAX_SCANS 100 > DS_NONPDQ_QUERY_MEM 250000> > The currently running update statistics high command lastet about 60 hrs > the > last time the sqexplain.out where updated. So I'm hopeful we get ready 40 > hours later, that is Sunday in the early morning. > > Thank you and have a good weekend, > Reinhard. > > -----Ursprüngliche Nachricht----- > Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag von Art > Kagel > Gesendet: Freitag, 9. Dezember 2016 17:02 > An: ids@iiug.org > Betreff: Re: Long lasting dostats [38278] > > Oops that got away from me. Need to add: > > This is why dostats prints out those warnings at the beginning if you don't > enable parallelism and enough memory. > > Art > > Art S. Kagel, President and Principal Consultant ASK Database Management > www.askdbmgt.com > > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and > do not reflect on the IIUG, nor any other organization with which I am > associated either explicitly, implicitly, or by inference. Neither do those > opinions reflect those of other individuals affiliated with any entity with > which I am affiliated nor those of the entities themselves. > > On Fri, Dec 9, 2016 at 10:59 AM, Art Kagel <art.kagel@gmail.com> wrote: > > > OK, here's what's going on: > > > > - You didn't turn on PDQPRIORITY so the update statistics high command > > has to scan all 71 partitions sequentially. > > - You didn't increase the amount of memory allocated for sorting so it > > only has 15MB available to sort 5 columns of keys totaling 960MB five > > different ways (once ordering by each key) - DS_NONPDQ_QUERY_MEM > > without PDQ or DS_TOTAL_MEMORY and DS_MAX_QUERIES with PDQPRIORITY > 0 > > - You didn't enable PSORT_NPROCS so the non-PDQ sorting is also single > > threaded. > > > > At the time you extracted the sqexplain output the update statistics > > command was beginning to process the 4th sort of 5, so your between > > 60% & 80% done give or take a few percentages if the different keys > > are of significantly different sizes. > > > > Art > > > > > > Art S. Kagel, President and Principal Consultant ASK Database > > Management www.askdbmgt.com > > > > Blog: http://informix-myview.blogspot.com/ > > > > Disclaimer: Please keep in mind that my own opinions are my own > > opinions and do not reflect on the IIUG, nor any other organization > > with which I am associated either explicitly, implicitly, or by > > inference. Neither do those opinions reflect those of other > > individuals affiliated with any entity with which I am affiliated nor > those > of the entities themselves. > > > > > > On Fri, Dec 9, 2016 at 10:22 AM, Habichtsberg, Reinhard < > > RHabichtsberg@arz-emmendingen.de> wrote: > > > >> Art: > >> > >> There are only a few lines in sqexplain.out: > >> > >> UPDATE STATISTICS:> >> ================== > >> > >> Table: informix.xxxxxxxxx > >> Mode: HIGH > >> Number of Bins: 267 Bin size 67362549 Sort data 964908.6 MB Sort > >> memory granted 15.0 MB Estimated number of table scans 5 PASS #1 > >> pe_id PASS #2 pe_bearbeiter PASS #3 pe_pruefstart PASS #4 > >> pe_wfstatusvor PASS #5 pe_re Scan 0 Sort 0 Build 34591 Insert 0 Close > >> 0 Total 34591 Completed pass 1 in 576 minutes 31 seconds Scan 0 Sort > >> 0 Build 141308 Insert 0 Close 0 Total 141308 Completed pass 2 in 2355 > >> minutes 8 seconds Scan 0 Sort 0 Build 36106 Insert 0 Close 0 Total > >> 36106 Completed pass 3 in 601 minutes 46 seconds > >> > >> Btw: The table has 71 partitions. The table is always heavily used > >> and has about seven million of write ops a day. Furthermore the table > >> is part of many queries. > >> > >> Thank you, > >> Reinhard. > >> > >> -----Ursprüngliche Nachricht----- > >> Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag > >> von Art Kagel > >> Gesendet: Freitag, 9. Dezember 2016 14:34 > >> An: ids@iiug.org > >> Betreff: Re: Long lasting dostats [38274] > >> > >> Reinhard: > >> > >> Welllllll, you've got a table that, at least on the surface, seems to > >> have on the order of 10^10 rows and you don't have PDQPRIORITY or > >> PSORT_NPROCS set so this is a single threaded operation on a table > >> that has well over 30 partitions (I keep losing count). It's just > >> going to take time. > >> > >> You have -X set... I'd like to see the sqexplain file lines from this > >> run (you can send these to me directly so as not to clog the list. > >> > >> Art > >> > >> Art S. Kagel, President and Principal Consultant ASK Database > >> Management www.askdbmgt.com > >> > >> Blog: http://informix-myview.blogspot.com/ > >> > >> Disclaimer: Please keep in mind that my own opinions are my own > >> opinions and do not reflect on the IIUG, nor any other organization > >> with which I am associated either explicitly, implicitly, or by > >> inference. Neither do those opinions reflect those of other > >> individuals affilia

Hi Richard, Interesting to see that turning on all the PDQ bells and whistles isn't much faster in terms of time elapsed, if at all. My tests on server grade hardware show setting PDQPRIORITY and PSORT_NPROCS only reduces the time by 10-15% at the expense of a lot more I/O. This is a complicated subject and I did a lot of testing for IIUG presentations in 2015 and 2016 on this subject :-) Using "statlevel fragment" might help in your situation if any of the partitions do not get updated. There is a requirement for an sbspace to use this feature (see docs). To speed up the LOW mode part you can switch on USTLOWTS_SAMPLE which is really the only tuneable parameter. There are no reasons not to use this I know if. I think that without fragmented indices you cannot benefit from parallelism in LOW mode. Your explain plan is only for the HIGH or MEDIUM mode work. For MEDIUM and HIGH mode I usually recommend not using PDQ and updating stats on multiple tables in parallel to maximise throughput because it is difficult to improve the run times on individual tables (see above). If you have one huge table, much bigger than all the rest, this strategy will not work. I do set DBUPSPACE='0:50:0' to increase sort memory from 15 Mb to 50 Mb, but this parameter does not make a lot of difference as long as you don't disable using indices for sorting. If you do set PDQPRIORITY you will enable light scans which avoids churning your buffer cache: this is one possible advantage of PDQ. Ask yourself if it's a problem that UPDATE STATISTICS takes a long time. If you can just leave it running, leaving PDQPRIORITY off will minimise the number of threads and the impact on the rest of your system. Yes there is an issue that a running UPDATE STATISTICS process can cause waits on data dictionary mutexes while it's running but this depends on your application and schema (specifically use of foreign keys). Finally changing your distributions to MEDIUM mode from HIGH mode will speed things up and probably won't affect query plans unless you use LIKE queries on chars/varchars. Ben.

Hi Art and Ben, thank you for your answers.. I now tried to only update statistics medium. PSORT_NPROCS=50; export PSORT_NPROCS dbaccess - dostats_xxxxxxxxxx.sql dostats_xxxxxxxxxxx.sql: DATABASE yyyyy; SET ISOLATION DIRTY READ; SET EXPLAIN ON; SET PDQPRIORITY 80; UPDATE STATISTICS MEDIUM FOR TABLE xxxxxxxxxx (pe_re_lfd, pe_vo, pe_pos, pe_ebene, pe_pruefung, pe_pruefende, pe_pruefart, pe_wfstatusnach, pe_ergebnis, pe_ergebnisart, pe_berinr, pe_textnr, pe_folgeber inr, pe_jobid, pe_jobart, pe_pruefzeit, pe_pruefzeit_ges, pe_wfnr, pe_nachbeding, pe_folge_ga, pe_lfdnr_verf); SET EXPLAIN OFF; This run finished after a minute or so with error -647 Error evaluating math library function function-name. The data type of the argument is correct, but the value passed to the function is not appropriate for the math function. Check that you are passing a legitimate value to the function. Hmm, have we reached any limits? TIA Reinhard. -----Ursprüngliche Nachricht----- Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag von BENJAMIN THOMPSON Gesendet: Montag, 12. Dezember 2016 16:54 An: ids@iiug.org Betreff: Re: Long lasting dostats [38296] Hi Richard, Interesting to see that turning on all the PDQ bells and whistles isn't much faster in terms of time elapsed, if at all. My tests on server grade hardware show setting PDQPRIORITY and PSORT_NPROCS only reduces the time by 10-15% at the expense of a lot more I/O. This is a complicated subject and I did a lot of testing for IIUG presentations in 2015 and 2016 on this subject :-) Using "statlevel fragment" might help in your situation if any of the partitions do not get updated. There is a requirement for an sbspace to use this feature (see docs). To speed up the LOW mode part you can switch on USTLOWTS_SAMPLE which is really the only tuneable parameter. There are no reasons not to use this I know if. I think that without fragmented indices you cannot benefit from parallelism in LOW mode. Your explain plan is only for the HIGH or MEDIUM mode work. For MEDIUM and HIGH mode I usually recommend not using PDQ and updating stats on multiple tables in parallel to maximise throughput because it is difficult to improve the run times on individual tables (see above). If you have one huge table, much bigger than all the rest, this strategy will not work. I do set DBUPSPACE='0:50:0' to increase sort memory from 15 Mb to 50 Mb, but this parameter does not make a lot of difference as long as you don't disable using indices for sorting. If you do set PDQPRIORITY you will enable light scans which avoids churning your buffer cache: this is one possible advantage of PDQ. Ask yourself if it's a problem that UPDATE STATISTICS takes a long time. If you can just leave it running, leaving PDQPRIORITY off will minimise the number of threads and the impact on the rest of your system. Yes there is an issue that a running UPDATE STATISTICS process can cause waits on data dictionary mutexes while it's running but this depends on your application and schema (specifically use of foreign keys). Finally changing your distributions to MEDIUM mode from HIGH mode will speed things up and probably won't affect query plans unless you use LIKE queries on chars/varchars. Ben. ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

I don't think so. This is an internal error inside the engine. I'd open a PMR about it. Art Art S. Kagel, President and Principal Consultant ASK Database Management www.askdbmgt.com Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Tue, Dec 13, 2016 at 12:10 PM, Habichtsberg, Reinhard < RHabichtsberg@arz-emmendingen.de> wrote: > Hi Art and Ben, > > thank you for your answers.. > > I now tried to only update statistics medium. > > PSORT_NPROCS=50; export PSORT_NPROCS > dbaccess - dostats_xxxxxxxxxx.sql > > dostats_xxxxxxxxxxx.sql: > DATABASE yyyyy; > SET ISOLATION DIRTY READ; > SET EXPLAIN ON; > SET PDQPRIORITY 80; > UPDATE STATISTICS MEDIUM FOR TABLE xxxxxxxxxx (pe_re_lfd, pe_vo, pe_pos, > pe_ebene, pe_pruefung, pe_pruefende, pe_pruefart, pe_wfstatusnach, > pe_ergebnis, pe_ergebnisart, pe_berinr, pe_textnr, pe_folgeber > inr, pe_jobid, pe_jobart, pe_pruefzeit, pe_pruefzeit_ges, pe_wfnr, > pe_nachbeding, pe_folge_ga, pe_lfdnr_verf); > SET EXPLAIN OFF;> > This run finished after a minute or so with error > -647 Error evaluating math library function function-name. > > The data type of the argument is correct, but the value passed to the > function is not appropriate for the math function. Check that you are > passing a legitimate value to the function. > > Hmm, have we reached any limits? > > TIA > Reinhard. > > -----Ursprüngliche Nachricht----- > Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag von > BENJAMIN THOMPSON > Gesendet: Montag, 12. Dezember 2016 16:54 > An: ids@iiug.org > Betreff: Re: Long lasting dostats [38296] > > Hi Richard, > > Interesting to see that turning on all the PDQ bells and whistles isn't > much > faster in terms of time elapsed, if at all. My tests on server grade > hardware > show setting PDQPRIORITY and PSORT_NPROCS only reduces the time by 10-15% > at > the expense of a lot more I/O. > > This is a complicated subject and I did a lot of testing for IIUG > presentations in 2015 and 2016 on this subject :-) > > Using "statlevel fragment" might help in your situation if any of the > partitions do not get updated. There is a requirement for an sbspace to use > this feature (see docs). > > To speed up the LOW mode part you can switch on USTLOWTS_SAMPLE which is > really the only tuneable parameter. There are no reasons not to use this I > know if. I think that without fragmented indices you cannot benefit from > parallelism in LOW mode. > > Your explain plan is only for the HIGH or MEDIUM mode work. For MEDIUM and > HIGH mode I usually recommend not using PDQ and updating stats on multiple > tables in parallel to maximise throughput because it is difficult to > improve > the run times on individual tables (see above). If you have one huge table, > much bigger than all the rest, this strategy will not work. I do set > DBUPSPACE='0:50:0' to increase sort memory from 15 Mb to 50 Mb, but this > parameter does not make a lot of difference as long as you don't disable > using > indices for sorting. > > If you do set PDQPRIORITY you will enable light scans which avoids churning > your buffer cache: this is one possible advantage of PDQ. > > Ask yourself if it's a problem that UPDATE STATISTICS takes a long time. If > you can just leave it running, leaving PDQPRIORITY off will minimise the > number of threads and the impact on the rest of your system. Yes there is > an > issue that a running UPDATE STATISTICS process can cause waits on data > dictionary mutexes while it's running but this depends on your application > and > schema (specifically use of foreign keys). > > Finally changing your distributions to MEDIUM mode from HIGH mode will > speed > things up and probably won't affect query plans unless you use LIKE > queries on > chars/varchars. > > Ben. > > > ************************************************************ > ******************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > > ************************************************************ > ******************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --94eb2c12cad0a8fe5c05438e8e1e

Hi Richard, I'd say this is a bug. But also that it's Groundhog Day: having googled it, you asked a very similar question to the group about "update statistics medium" on a similar large table in May 2009 and Art then recommended raising a PMR. Did you get an answer from IBM back then? :-) Having read the 2009 post you also hit this issue in 2006. One thing you could try to do is isolate or work around the problem by doing column individually instead of passing them as one long list. You may also get a different result if you don't set PDQPRIORITY for high/medium stats, as I suggested in my last post. Ben.

It sounds like: IC80550 UPDATE STATISTICS MAY RAISE ERROR 647 ON SOLARIS But if I read correctly you're on 12.10.xC4W1... which should have this fixed... So it would be either a regression or a similar thing with a slightly different context. So, yes... A PMR would be in order... shouldn't be that complex Regards. On Tue, Dec 13, 2016 at 10:13 PM, BENJAMIN THOMPSON < benjamin.thompson@skybettingandgaming.com> wrote: > Hi Richard, > > I'd say this is a bug. > > But also that it's Groundhog Day: having googled it, you asked a very > similar > question to the group about "update statistics medium" on a similar large > table in May 2009 and Art then recommended raising a PMR. Did you get an > answer from IBM back then? :-) > > Having read the 2009 post you also hit this issue in 2006. > > One thing you could try to do is isolate or work around the problem by > doing > column individually instead of passing them as one long list. You may also > get > a different result if you don't set PDQPRIORITY for high/medium stats, as I > suggested in my last post. > > Ben. > > > ************************************************************ > ******************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --94eb2c030928eaed3405439469e8

Oh, that's a bit embarrassing. The error seems to obsess me. I didn't remember and it seems I didn't open a PMR then. Ok, I did it now. I'm curious what they say. Thanks for your help. Reinhard. -----Ursprüngliche Nachricht----- Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag von BENJAMIN THOMPSON Gesendet: Dienstag, 13. Dezember 2016 23:14 An: ids@iiug.org Betreff: Re: Long lasting dostats [38313] Hi Richard, I'd say this is a bug. But also that it's Groundhog Day: having googled it, you asked a very similar question to the group about "update statistics medium" on a similar large table in May 2009 and Art then recommended raising a PMR. Did you get an answer from IBM back then? :-) Having read the 2009 post you also hit this issue in 2006. One thing you could try to do is isolate or work around the problem by doing column individually instead of passing them as one long list. You may also get a different result if you don't set PDQPRIORITY for high/medium stats, as I suggested in my last post. Ben. ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

It's misterious but a second and third run of update statistics medium finished without error. Thank you Reinhard -----Ursprüngliche Nachricht----- Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag von BENJAMIN THOMPSON Gesendet: Dienstag, 13. Dezember 2016 23:14 An: ids@iiug.org Betreff: Re: Long lasting dostats [38313] Hi Richard, I'd say this is a bug. But also that it's Groundhog Day: having googled it, you asked a very similar question to the group about "update statistics medium" on a similar large table in May 2009 and Art then recommended raising a PMR. Did you get an answer from IBM back then? :-) Having read the 2009 post you also hit this issue in 2006. One thing you could try to do is isolate or work around the problem by doing column individually instead of passing them as one long list. You may also get a different result if you don't set PDQPRIORITY for high/medium stats, as I suggested in my last post. Ben. ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.