long checkpoint
SangGyu Jeong's customer on 11.70.FC9 saw checkpoints grow from 1-2 seconds to about 30 after moving from AIX E850 (smt4) to E950 (smt8) with more CPU VPs and cleaners. Art Kagel and Eric Vercelletto advised returning to smt4; Paul Watson said smt interacts badly with Informix. Vladimir Kolobrodov suspected storage. SangGyu was checking storage and IT27510. Unresolved.
Auto-generated by Claude from the posts below — may be imperfect; read the full thread.
My customer changed the server a few days ago and modified the Informix onconfig parameters vpclass and cleaners.
Perhaps because of that, the checkpoint duration has increased to around 30 seconds. Before, it usually took 1 or 2 seconds.
Is the parameter change related to the checkpoint duration?
-CPU core information and parameter values of old server
AIX 7.1(E850), 4 core, smt4
CLEANERS 12
VPCLASS cpu,num=10,noage-CPU core information and parameter values of new server
AIX 7.1(E950), 4 core, smt8
CLEANERS 20
VPCLASS cpu,num=20,noageBelow is the result of onstat -g ckp on the current server.
IBM Informix Dynamic Server Version 11.70.FC9 -- On-Line -- Up 4 days 05:46:44 -- 46601584 Kbytes
AUTO_CKPTS=On RTO_SERVER_RESTART=OffCritical Sections Physical Log Logical Log
Clock Total Flush Block # Ckpt Wait Long # Dirty Dskflu Total Avg Total Avg
Interval Time Trigger LSN Time Time Time Waits Time Time Time Buffers /Sec Pages /Sec Pages /Sec
1065136 10:48:40 HA 175881:0x488d878 1.5 1.5 0.0 3 0.0 0.0 0.0 18366 12564 50016 806 24469 394
1065137 10:49:42 HA 175882:0x5cae040 3.8 3.5 0.0 7 0.2 0.2 0.3 23897 6876 46506 762 30315 496
1065138 10:51:21 HA 175883:0x17c95a8 34.9 34.9 0.0 1 0.0 0.0 0.0 16901 484 85227 1272 13462 200
1065139 10:52:26 HA 175883:0x354c018 34.9 34.8 0.0 1 0.0 0.0 0.0 17433 500 86972 1338 8530 131
1065140 10:57:01 CKPTINTVL 175884:0x120018 8.6 8.5 0.0 1 0.0 0.0 0.0 16721 1965 3485 11 11921 39
1065141 11:02:37 CKPTINTVL 175884:0x54f4018 35.0 35.0 0.0 0 0.0 0.0 0.0 17636 503 4084 13 22435 72
1065142 11:08:51 CKPTINTVL 175885:0x24c4018 104.3 104.1 0.0 14 0.0 0.0 0.0 19202 184 4966 16 17400 57
1065143 11:10:01 HA 175885:0x374e018 69.9 69.8 0.0 5 0.0 0.0 0.0 21778 312 3414 32 7329 70
1065144 11:11:11 HA 175885:0x4185018 69.8 69.7 0.0 14 0.0 0.1 0.1 20655 296 2217 31 4522 64
1065145 11:11:47 HA 175885:0x4933018 35.8 35.6 0.0 65 0.1 0.1 0.2 16512 463 2261 32 2549 36
1065146 11:17:13 CKPTINTVL 175886:0x1e49058 59.3 36.2 0.0 313 23.0 13.0 23.0 22340 617 6231 19 14668 44
1065147 11:22:53 CKPTINTVL 175886:0x4cc1018 70.3 70.2 0.0 1 0.0 0.0 0.0 21282 303 5921 19 14930 48
1065148 11:23:29 HA 175886:0x58d7018 34.9 34.9 0.0 0 0.0 0.0 0.0 19559 560 2349 33 5360 75
1065149 11:28:31 CKPTINTVL 175887:0x2b1d018 34.8 34.7 0.0 0 0.0 0.0 0.0 18766 540 5537 18 14646 48
1065150 11:33:37 CKPTINTVL 175887:0x517e018 35.9 35.9 0.0 2 0.0 0.0 0.0 22631 630 4347 14 12552 41
1065151 11:38:43 CKPTINTVL 175888:0x206c018 36.3 36.2 0.0 1 0.0 0.0 0.0 17839 492 4411 14 13636 44
1065152 11:43:49 CKPTINTVL 175888:0x3e34018 36.7 36.6 0.0 2 0.0 0.0 0.0 23057 630 3415 11 8323 27
1065153 11:48:31 CKPTINTVL 175888:0x60cc018 12.6 12.5 0.0 1 0.0 0.0 0.0 20291 1618 3290 10 9044 29
1065154 11:54:32 CKPTINTVL 175889:0x12a3018 61.6 61.6 0.0 0 0.0 0.0 0.0 13622 221 2107 6 7364 23
1065155 11:54:34 HA 175889:0x1bef810 1.0 1.0 0.0 1 0.0 0.0 0.0 8194 8194 734 11 2386 38
Max Plog Max Llog Max Dskflush Avg Dskflush Avg Dirty Blocked
pages/sec pages/sec Time pages/sec pages/sec Time
8692 4715 391 1195 0 0
------------------------------
SangGyu Jeong
Software Engineer
Infrasoft
Seoul Korea, Republic of
------------------------------
#Informix
SangGyu: I am assuming that there were Power8 processors in the E850 and Power9 processors in the E950 now. It looks like they had half of the SMT threads disabled in the E850 running them at SMT4 rather than the default of SMT8. This is the recommended configuration for running Informix (or any of the major RDBMS systems) on Power8 or Power9 processors. Note that they are now running the E950 in SMT8 mode. I think that that is the problem! Those last 4 SMT threads run at only about 40% of the main core and running sessions on them slow down sessions running on the first four SMT threads. The primary thread can run at up to 100% of the rated speed of the core the second thread at about 85% and the next two threads at about 70-75%. The remaining threads, as I noted above, are much slower. I think that is the main problem. Art Art S. Kagel, President and Principal Consultant ASK Database Management www.askdbmgt.com Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves.
Hi SangGyu, I would try setting smt back to 4. I have observed that increasing smt to more than 4 rarely provides better performance, and often decreased performance. I was going to explain that phenomenon, but just reading Art's email that explains it perfectly: in short, raising smt gives you less power Playing with smt with high values is not good for Informix at least with current versions of AIX BR Eric
I am not an OS expert, but what i have read on hyper threading, i'll have to agree with Art & Eric. Don't assume that more the SMT threads, better the performance. Actually your load will determine what is best for your environment. So you need to tune your system. I would say try reverting your settings one by one and see if that makes and difference. If yes then its all good else a support case with ifxcollect might help you. ------------------------------ Gaurav Kumar ------------------------------
Art, Eric: There is a disk storage issue related to the longer checkpoint time than before. So the customer is checking the storage status. If there are performance issues after the storage issue has been resolved, we will also advise customers to adjust from smt8 to smt4. Gaurav: To explain the background of this problem, I opened a case because the problem of the connection manager not being reconnected is related to a long checkpoint time. As a result of opening the case, there is a possibility that IT27510 is a related defect. If the checkpoint time recovers as before due to storage problems, it is not necessary to upgrade. ------------------------------ SangGyu Jeong Software Engineer Infrasoft Seoul Korea, Republic of ------------------------------
AIX and Informix, on dedicated servers, runs faster with SMT off, or at
least that is my experience.
I was once told by a IBM'er that would know what they talking about was to
just max out the cleaners and leave them alone.
Cheers
Paul
> My customer changed the server a few days ago and modified the Informix
> parameters vpclass and cleaners.
> Perhaps because of that, the checkpoint duration has increased to around
> 30 seconds. Before, it usually took 1 or 2 seconds.
> Does parameter adjustment have anything to do with checkpoint time?
>
> -CPU core information and parameter values of existing equipment
> AIX 7.1(E850), 4 core, smt4
> CLEANERS 12
> VPCLASS cpu,num=20,noage>
> -CPU core information and parameter values of new equipment
> AIX 7.1(E950), 4 core, smt8
> CLEANERS 20
> VPCLASS cpu,num=20,noage>
>
> Below is the result of onstat -g ckp on the current server.
> IBM Informix Dynamic Server Version 11.70.FC9 -- On-Line -- Up 4 days
> 05:46:44 -- 46601584 Kbytes
>
> AUTO_CKPTS=On RTO_SERVER_RESTART=Off
>
> Critical> Sections
>
>
>
>
> Physical
> Log
> Logical
> Log
> Clock Total Flush Block #
> Ckpt Wait Long # Dirty Dskflu Total Avg Total
> Avg
> Interval Time Trigger LSN Time Time Time Waits
> Time Time Time Buffers /Sec Pages /Sec Pages /Sec
> 1065136 10:48:40 HA 175881:0x488d878 1.5 1.5 0.0 3
> 0.0 0.0 0.0 18366 12564 50016 806 24469 394
> 1065137 10:49:42 HA 175882:0x5cae040 3.8 3.5 0.0 7
> 0.2 0.2 0.3 23897 6876 46506 762 30315 496
> 1065138 10:51:21 HA 175883:0x17c95a8 34.9 34.9 0.0 1
> 0.0 0.0 0.0 16901 484 85227 1272 13462 200
> 1065139 10:52:26 HA 175883:0x354c018 34.9 34.8 0.0 1
> 0.0 0.0 0.0 17433 500 86972 1338 8530 131
> 1065140 10:57:01 CKPTINTVL 175884:0x120018 8.6 8.5 0.0 1
> 0.0 0.0 0.0 16721 1965 3485 11 11921 39
> 1065141 11:02:37 CKPTINTVL 175884:0x54f4018 35.0 35.0 0.0 0
> 0.0 0.0 0.0 17636 503 4084 13 22435 72
> 1065142 11:08:51 CKPTINTVL 175885:0x24c4018 104.3 104.1 0.0 14
> 0.0 0.0 0.0 19202 184 4966 16 17400 57
> 1065143 11:10:01 HA 175885:0x374e018 69.9 69.8 0.0 5
> 0.0 0.0 0.0 21778 312 3414 32 7329 70
> 1065144 11:11:11 HA 175885:0x4185018 69.8 69.7 0.0 14
> 0.0 0.1 0.1 20655 296 2217 31 4522 64
> 1065145 11:11:47 HA 175885:0x4933018 35.8 35.6 0.0 65
> 0.1 0.1 0.2 16512 463 2261 32 2549 36
> 1065146 11:17:13 CKPTINTVL 175886:0x1e49058 59.3 36.2 0.0 313
> 23.0 13.0 23.0 22340 617 6231 19 14668 44
> 1065147 11:22:53 CKPTINTVL 175886:0x4cc1018 70.3 70.2 0.0 1
> 0.0 0.0 0.0 21282 303 5921 19 14930 48
> 1065148 11:23:29 HA 175886:0x58d7018 34.9 34.9 0.0 0
> 0.0 0.0 0.0 19559 560 2349 33 5360 75
> 1065149 11:28:31 CKPTINTVL 175887:0x2b1d018 34.8 34.7 0.0 0
> 0.0 0.0 0.0 18766 540 5537 18 14646 48
> 1065150 11:33:37 CKPTINTVL 175887:0x517e018 35.9 35.9 0.0 2
> 0.0 0.0 0.0 22631 630 4347 14 12552 41
> 1065151 11:38:43 CKPTINTVL 175888:0x206c018 36.3 36.2 0.0 1
> 0.0 0.0 0.0 17839 492 4411 14 13636 44
> 1065152 11:43:49 CKPTINTVL 175888:0x3e34018 36.7 36.6 0.0 2
> 0.0 0.0 0.0 23057 630 3415 11 8323 27
> 1065153 11:48:31 CKPTINTVL 175888:0x60cc018 12.6 12.5 0.0 1
> 0.0 0.0 0.0 20291 1618 3290 10 9044 29
> 1065154 11:54:32 CKPTINTVL 175889:0x12a3018 61.6 61.6 0.0 0
> 0.0 0.0 0.0 13622 221 2107 6 7364 23
> 1065155 11:54:34 HA 175889:0x1bef810 1.0 1.0 0.0 1
> 0.0 0.0 0.0 8194 8194 734 11 2386 38
>
> Max Plog Max Llog Max Dskflush Avg Dskflush Avg Dirty
> Blocked
> pages/sec pages/sec Time pages/sec pages/sec
> Time
> 8692 4715 391 1195 0
> 0
>
> ???
>
> ------------------------------
> SangGyu Jeong
> Software Engineer
> Infrasoft
> Seoul Korea, Republic of
> ------------------------------
>
>
> Reply to Sender :
>
https://community.ibm.com/eGroups/PostReply/?GroupId=4147&MID=198967&SenderKey=321aa177-684f-4844-a5ad-34ecda64d7a9&MDATE=7575465468&UserKey=29a6c229-a98c-46da-9248-4df042a4a263&sKey=KeyRemoved
>
> Reply to Discussion :
>
https://community.ibm.com/eGroups/PostReply/?GroupId=4147&MID=198967&MDATE=7575465468&UserKey=29a6c229-a98c-46da-9248-4df042a4a263&sKey=KeyRemoved
>
>
>
> You are subscribed to "Informix" as
paul@oninit.com
. To change your
> subscriptions, go to
>
http://community.ibm.com/community/user/preferences?section=Subscriptions&MDATE=7575465468&UserKey=29a6c229-a98c-46da-9248-4df042a4a263&sKey=KeyRemoved.
> To unsubscribe from this community discussion, go to
>
http://community.ibm.com/HigherLogic/eGroups/Unsubscribe.aspx?UserKey=29a6c229-a98c-46da-9248-4df042a4a263&sKey=KeyRemoved&GroupKey=60eb97a2-57c0-4130-9b6d-57174f97d5a8.
On the record the IBM kernel architects state smt8 is not an issue with
anything, off the record they agree that smtX and the Informix internal
thread scheduling don't play nicely together
Cheers
Paul
> Hi SangGyu,
>
>
> I would try setting smt back to 4. I have observed that increasing smt to
> more than 4 rarely provides better performance, and often decreased
> performance.
> I was going to explain that phenomenon, but just reading Art's email that
> explains it perfectly: in short, raising smt gives you less power Playing
> with smt with high values is not good for Informix at least with current
> versions of AIX??????
>
> BR
> Eric
>
>
>
> -------------------------------------------
> Original Message:
> Sent: 10/13/2020 11:08:00 PM
> From: SangGyu Jeong
> Subject: long checkpoint
>
> My customer changed the server a few days ago and modified the Informix
> onconfig parameters vpclass and cleaners.
> Perhaps because of that, the checkpoint duration has increased to around
> 30 seconds. Before, it usually took 1 or 2 seconds.
> Is the parameter change related to the checkpoint duration?
>
> -CPU core information and parameter values of old server
> AIX 7.1(E850), 4 core, smt4
> CLEANERS 12
> VPCLASS cpu,num=10,noage>
> -CPU core information and parameter values of new server
> AIX 7.1(E950), 4 core, smt8
> CLEANERS 20
> VPCLASS cpu,num=20,noage>
>
> Below is the result of onstat -g ckp on the current server.
>
> IBM Informix Dynamic Server Version 11.70.FC9 -- On-Line -- Up 4 days
> 05:46:44 -- 46601584 Kbytes
>
> AUTO_CKPTS=On RTO_SERVER_RESTART=Off
>
> Critical> Sections
>
>
>
>
> Physical
> Log
> Logical
> Log
> Clock Total Flush Block #
> Ckpt Wait Long # Dirty Dskflu Total Avg Total
> Avg
> Interval Time Trigger LSN Time Time Time Waits
> Time Time Time Buffers /Sec Pages /Sec Pages /Sec
> 1065136 10:48:40 HA 175881:0x488d878 1.5 1.5 0.0 3
> 0.0 0.0 0.0 18366 12564 50016 806 24469 394
> 1065137 10:49:42 HA 175882:0x5cae040 3.8 3.5 0.0 7
> 0.2 0.2 0.3 23897 6876 46506 762 30315 496
> 1065138 10:51:21 HA 175883:0x17c95a8 34.9 34.9 0.0 1
> 0.0 0.0 0.0 16901 484 85227 1272 13462 200
> 1065139 10:52:26 HA 175883:0x354c018 34.9 34.8 0.0 1
> 0.0 0.0 0.0 17433 500 86972 1338 8530 131
> 1065140 10:57:01 CKPTINTVL 175884:0x120018 8.6 8.5 0.0 1
> 0.0 0.0 0.0 16721 1965 3485 11 11921 39
> 1065141 11:02:37 CKPTINTVL 175884:0x54f4018 35.0 35.0 0.0 0
> 0.0 0.0 0.0 17636 503 4084 13 22435 72
> 1065142 11:08:51 CKPTINTVL 175885:0x24c4018 104.3 104.1 0.0 14
> 0.0 0.0 0.0 19202 184 4966 16 17400 57
> 1065143 11:10:01 HA 175885:0x374e018 69.9 69.8 0.0 5
> 0.0 0.0 0.0 21778 312 3414 32 7329 70
> 1065144 11:11:11 HA 175885:0x4185018 69.8 69.7 0.0 14
> 0.0 0.1 0.1 20655 296 2217 31 4522 64
> 1065145 11:11:47 HA 175885:0x4933018 35.8 35.6 0.0 65
> 0.1 0.1 0.2 16512 463 2261 32 2549 36
> 1065146 11:17:13 CKPTINTVL 175886:0x1e49058 59.3 36.2 0.0 313
> 23.0 13.0 23.0 22340 617 6231 19 14668 44
> 1065147 11:22:53 CKPTINTVL 175886:0x4cc1018 70.3 70.2 0.0 1
> 0.0 0.0 0.0 21282 303 5921 19 14930 48
> 1065148 11:23:29 HA 175886:0x58d7018 34.9 34.9 0.0 0
> 0.0 0.0 0.0 19559 560 2349 33 5360 75
> 1065149 11:28:31 CKPTINTVL 175887:0x2b1d018 34.8 34.7 0.0 0
> 0.0 0.0 0.0 18766 540 5537 18 14646 48
> 1065150 11:33:37 CKPTINTVL 175887:0x517e018 35.9 35.9 0.0 2
> 0.0 0.0 0.0 22631 630 4347 14 12552 41
> 1065151 11:38:43 CKPTINTVL 175888:0x206c018 36.3 36.2 0.0 1
> 0.0 0.0 0.0 17839 492 4411 14 13636 44
> 1065152 11:43:49 CKPTINTVL 175888:0x3e34018 36.7 36.6 0.0 2
> 0.0 0.0 0.0 23057 630 3415 11 8323 27
> 1065153 11:48:31 CKPTINTVL 175888:0x60cc018 12.6 12.5 0.0 1
> 0.0 0.0 0.0 20291 1618 3290 10 9044 29
> 1065154 11:54:32 CKPTINTVL 175889:0x12a3018 61.6 61.6 0.0 0
> 0.0 0.0 0.0 13622 221 2107 6 7364 23
> 1065155 11:54:34 HA 175889:0x1bef810 1.0 1.0 0.0 1
> 0.0 0.0 0.0 8194 8194 734 11 2386 38
>
> Max Plog Max Llog Max Dskflush Avg Dskflush Avg Dirty
> Blocked
> pages/sec pages/sec Time pages/sec pages/sec
> Time
> 8692 4715 391 1195 0
> 0???
>
>
>
>
> ------------------------------
> SangGyu Jeong
> Software Engineer
> Infrasoft
> Seoul Korea, Republic of
> ------------------------------
>
> Reply to Sender :
>
https://community.ibm.com/eGroups/PostReply/?GroupId=4147&MID=198979&SenderKey=3f135a6d-8b93-447d-a75b-f8d9649841d6&MDATE=7575465469&UserKey=29a6c229-a98c-46da-9248-4df042a4a263&sKey=KeyRemoved
>
> Reply to Discussion :
>
https://community.ibm.com/eGroups/PostReply/?GroupId=4147&MID=198979&MDATE=7575465469&UserKey=29a6c229-a98c-46da-9248-4df042a4a263&sKey=KeyRemoved
>
>
>
> You are subscribed to "Informix" as
paul@oninit.com
. To change your
> subscriptions, go to
>
http://community.ibm.com/community/user/preferences?section=Subscriptions&MDATE=7575465469&UserKey=29a6c229-a98c-46da-9248-4df042a4a263&sKey=KeyRemoved.
> To unsubscribe from this community discussion, go to
>
http://community.ibm.com/HigherLogic/eGroups/Unsubscribe.aspx?UserKey=29a6c229-a98c-46da-9248-4df042a4a263&sKey=KeyRemoved&GroupKey=60eb97a2-57c0-4130-9b6d-57174f97d5a8.
Yes, the issue most likely is related to storage. Quick check would be to look at the value for the RAS_PLOG_SPEED in "onconfig", if instance had been setup from scratch the number reflects (relative) I/O performance of the device used for the physical log. If, on POWER9, it's much lower compared to what it's on POWER8 - that would be an indicator that storage does not perform as well.
Better way would be to monitor onstat -g iof during checkpoint and/or , if Informix version supports it, to look at onstat -g his after.
One other thing to keep in mind is the size of the buffer pool and value for the LRU max, which on busy system would usually represent number of modified buffer pool pages needed to be flushed during checkpoint. However sometimes, with slow storage, number of modified buffer pages may go over LRU max, monitoring onstat -R will tell for sure.
You can also check / compare "Avg Dskflush pages/sec" number between P8 and P9 - if you have it.
If storage is not the issue - than you need to look for the buffer(s) where Informix need to wait before they can be flushed.
------------------------------
Vladimir Kolobrodov
------------------------------
Correction - I meant onstat -g ioh , not onstat -g his, of course.
------------------------------
Vladimir Kolobrodov
------------------------------
Paul et al: I got my understanding of PowerX and Informix from a whitepapre IBM put out. Don't know if it is still available on the IBM sites, though I do have a copy of the file. The title was: IBM Informix on Power7, Best Practices, a Technical White Paper Anyway, it recommends running the Power7 processor cores which only supported up to 4 SMT threads, in SMT0 or SMT2 mode depending on the workload. In later discussions with the IBM techs, they said to extend that recommendation to the Power8 and later processors by disabling at least half of the available SMT threads. We are better off, it seems, relying on Informix's thread model and multiple CPU VPs on the bare Power cores than using the SMT threads. Art Art S. Kagel, President and Principal Consultant ASK Database Management www.askdbmgt.com Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves.
Thank you for your reply. Paul. Even after the storage issue is resolved, if it is necessary to improve performance, we plan to adjust parameters such as cleaner. ------------------------------ SangGyu Jeong Software Engineer Infrasoft Seoul Korea, Republic of ------------------------------
Vladimir: Thanks for the good information like the RAS_PLOG_SPEED value. The Informix version is 11.70, so the -ioh option cannot be used. ------------------------------ SangGyu Jeong Software Engineer Infrasoft Seoul Korea, Republic of ------------------------------
Art: I read the document too. However, based on that document, it was difficult to know which setup would be better in POWER9 and smt8 environments. We plan to gradually adjust VPCLASS, which can be adjusted online. ------------------------------ SangGyu Jeong Software Engineer Infrasoft Seoul Korea, Republic of ------------------------------
You can generate your own onstat -g ioh - we have been doing it for years - you don't need the sysadmin task at all
Cheers
Paul
Paul Watson
Oninit LLC
+1-913-387-7529
www.oninit.com
Oninit® is a registered trademark of Oninit LLC
Related threads
- Posting from the Informix-list
- Migrating from IDS 9.40.UC6 to 11.50.UC3
- Ip for a network session
- questions onstat -g