wait for mutex
Frank saw many user threads in onstat -u flagged 'S' (waiting on a mutex) on IDS 11.50.FC8 under RHEL 6, alongside slow Enterprise Replication with data queuing. Fernando Nunes and Paul Watson suggested onstat -g wmx and -g lmx to identify the mutex; the output showed dozens of threads queued behind ER mutexes (CDRDSn0022, rqmut_trg_send). Madison Pruet explained this is ER queue lock contention, noting the ER queue locking was rewritten in 12.10.xC4 to greatly reduce it. No fix for the 11.50 system itself is recorded.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Folks,
Normally I do not see S---- in output of onstat -u, but now I see.
Any comments?
Thanks
Frank
RHEL 6
[informix@walter ngdc_er_20140820]$ onstat -u
IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:38:50
-- 11847728 KbytesUserthreads
address flags sessid user tty wait tout
locks nreads nwrites
1d81b3028 ---P--D 1 informix - 0 0
0 398 1015
1d81b3880 ---P--F 0 informix - 0 0
0 0 12778
1d81b40d8 ---P--F 0 informix - 0 0
0 0 18875
1d81b4930 ---P--F 0 informix - 0 0
0 0 11092
1d81b5188 ---P--F 0 informix - 0 0
0 0 18161
1d81b59e0 ---P--F 0 informix - 0 0
0 0 26058
1d81b6238 ---P--F 0 informix - 0 0
0 0 35724
1d81b6a90 ---P--F 0 informix - 0 0
0 0 12501
1d81b72e8 ---P--F 0 informix - 0 0
0 0 36521
1d81b7b40 ---P--F 0 informix - 0 0
0 0 15588
1d81b8398 ---P--F 0 informix - 0 0
0 0 46021
1d81b8bf0 ---P--F 0 informix - 0 0
0 0 2964
1d81b9448 ---P--F 0 informix - 0 0
0 0 183203
1d81b9ca0 ---P--F 0 informix - 0 0
0 0 47100
1d81ba4f8 ---P--F 0 informix - 0 0
0 0 18092
1d81bad50 ---P--F 0 informix - 0 0
0 0 47221
1d81bb5a8 ---P--F 0 informix - 0 0
0 0 13200
1d81bbe00 ---P--- 18 informix - 0 0
0 0 63
1d81bc658 ---P--B 19 informix - 0 0
0 515817 4096
1d81bceb0 Y--P--B 20 informix - 1d8cf0e50 0
0 459361 3977
1d81bdf60 ---P--D 24 informix - 0 0
0 4 0
1d81be7b8 ---P--D 23 informix - 0 0
0 0 0
1d81bf010 ---P--- 43 informix - 0 0
1 240 21860
1d81bf868 Y--P--- 143 informix - 1db2aadc8 0
1 8 22528
1d81c00c0 ---P--- 33 informix - 0 0
0 0 0
1d81c0918 Y--P--- 141 informix - 1db2ccb78 0
1 9 3
1d81c1170 ---P--- 35 informix - 0 0
1 0 0
1d81c19c8 Y--P--- 36 informix - 1d9910448 0
3 0 0
1d81c2220 Y--P--D 41 informix - 48bdf970 0
0 0 0
1d81c2a78 ---P--- 44 informix - 0 0
1 666 32901
1d81c32d0 ---P--- 45 informix - 0 0
1 711 32575
1d81c3b28 ---P--- 46 informix - 0 0
0 3 0
1d81c4380 ---P--- 47 informix - 0 0
1 240 442
1d81c4bd8 ---P--- 48 informix - 0 0
1 237 317
1d81c5430 ---P--- 49 informix - 0 0
1 197 329
1d81c5c88 ---P--- 50 informix - 0 0
1 165 552
1d81c64e0 ---P--- 51 informix - 0 0
2 202 252
1d81c6d38 ---P--- 52 informix - 0 0
1 304 76
1d81c7590 ---P--- 53 informix - 0 0
1 211 859
1d81c7de8 ---P--- 54 informix - 0 0
2 223 298
1d81c8640 ---P--- 55 informix - 0 0
1 204 414
1d81c8e98 ---P--- 56 informix - 0 0
1 263 329
1d81c96f0 ---P--- 57 informix - 0 0
2 237 364
1d81c9f48 ---P--- 58 informix - 0 0
2 302 174
1d81ca7a0 ---P--- 136 informix - 0 0
1 4 0
1d81caff8 S--P--- 137 informix - 1d9fbc8f0 0
1 186322 0
1d81cb850 ---P--- 138 informix - 0 0
0 272298 0
1d81cc0a8 ---P--- 139 informix - 0 0
1 44 0
1d81cc900 S--P--- 142 informix - 1dbb72ee0 0
0 0 211
1d81cd158 Y--P--- 144 informix - 1db2aadc8 0
1 9 23680
1d81cd9b0 S--P--- 4692 informix - 1dbb72ee0 0
2 3 4
1d81ce208 S--P--- 4531 informix - 1dbb72ee0 0
2 0 47
1d81cea60 Y--P--- 235 saapmgr - 1def80778 0
1 125 0
1d81cf2b8 S--P--- 4693 informix - 1dbb72ee0 0
2 3 1
1d81d0bc0 S--P--- 151 informix - 1dbb72ee0 0
2 13166 33392
1d81d1418 S--P--- 152 informix - 1dbb72ee0 0
2 14148 32374
1d81d1c70 S--P--- 153 informix - 1dbb72ee0 0
2 13705 31162
1d81d2d20 ---P--- 155 informix - 0 0
1 8890 0
1d81d3dd0 Y--P--- 157 informix - 1db2aaca8 0
2 13781 32324
1d81d4628 Y--P--- 4827 informix - 1db2aaca8 0
2 4 0
1d81d4e80 S--P--- 159 informix - 1dbb72ee0 0
2 13480 32593
1d81d56d8 S--P--- 160 informix - 1dbb72ee0 0
2 14200 32926
1d81d5f30 Y--P--- 2693 saapmgr - 1e5a69148 0
1 0 46
1d81d6788 Y--P--- 173 informix - 1db2aadc8 0
1 2 17280
1d81d6fe0 Y--P--- 238 saapmgr - 1e029cca8 0
1 0 0
1d81d7838 Y--P--- 239 saapmgr - 1e3846028 0
1 2 0
1d81d8090 Y--P--- 237 saapmgr - 1dc95d4c8 0
1 1 0
1d81d88e8 Y--P--- 260 saapmgr - 1dcaccd78 0
2 757 3042
1d81d9140 Y--P--- 244 saapmgr - 1dfffeb78 0
1 40 0
1d81d9998 Y--P--- 258 saapmgr - 1e3176be0 0
1 0 0
1d81da1f0 Y--P--- 236 saapmgr - 1de8514c8 0
1 1 0
1d81daa48 Y--P--- 241 saapmgr - 1e0d85c28 0
1 2 0
1d81db2a0 Y--P--- 262 saapmgr - 1dcb5d538 0
2 454 2717
1d81dbaf8 Y--P--- 242 saapmgr - 1dcadf148 0
1 0 0
1d81dc350 Y--P--- 240 saapmgr - 1de777308 0
1 1 0
1d81dcba8 Y--P--- 254 saapmgr - 1dcb2f788 0
1 3 0
1d81dd400 Y--P--- 243 saapmgr - 1dd7a1df8 0
1 1 0
1d81ddc58 Y--P--- 259 saapmgr - 1dcb5d1d8 0
2 820 2800
1d81de4b0 Y--P--- 256 saapmgr - 1ddc9ac68 0
1 9 0
1d81ded08 Y--P--- 2647 wwwmgr - 1dd5c9708 0
1 26 76
1d81df560 Y--P--- 261 saapmgr - 1dd7ec2f8 0
2 773 3510
1d81dfdb8 Y--P--- 263 saapmgr - 1e12b86c0 0
2 743 3245
1d81e0610 Y--P--- 246 saapmgr - 1dd7c64a8 0
1 0 0
1d81e0e68 Y--P--- 264 saapmgr - 1e12b8ca8 0
2 440 3917
1d81e16c0 Y--P--- 265 saapmgr - 1dea7b3d0 0
2 727 3044
1d81e1f18 Y--P--- 266 saapmgr - 1dea7ba48 0
1 460 3202
1d81e2770 Y--P--- 267 saapmgr - 1e12b8f78 0
2 651 3378
1d81e2fc8 Y--P--- 268 saapmgr - 1e04de580 0
1 891 3256
1d81e3820 Y--P--- 269 saapmgr - 1e04debf8 0
1 856 1164
1d81e4078 Y--P--- 270 saapmgr - 1de6233d0 0
1 2122 1273
1d81e48d0 Y--P--- 271 saapmgr - 1e00966e8 0
1 69 1045
1d81e5128 Y--P--- 272 saapmgr - 1de623df0 0
1 127 1460
1d81e5980 Y--P--- 273 saapmgr - 1df259318 0
1 63 1225
1d81e61d8 Y--P--- 274 saapmgr - 1df259900 0
1 65 1009
1d81e6a30 Y--P--- 275 saapmgr - 1e028fa88 0
1 86 1122
1d81e7288 Y--P--- 276 saapmgr - 1e0324740 0
1 73 1041
1d81e7ae0 Y--P--- 277 saapmgr - 1e0324d28 0
1 117 1273
1d81e8338 Y--P--- 278 saapmgr - 1dcb43958 0
1 69 1117
1d81e8b90 Y--P--- 279 saapmgr - 1e317a630 0
1 128223 247740
1d81e93e8 Y--P--- 2737 saapmgr - 1dbb7f268 0
1 0 26
1d81e9c40 Y--P--- 4588 saapmgr - 1e182d428 0
1 1 0
1d81ea498 Y--P--- 282 saapmgr - 1dd7f96e8 0
1 294 6235
1d81eacf0 Y--P--- 283 saapmgr - 1dd7f9808 0
1 186523 286151
1d81eb548 Y--P--- 284 saapmgr - 1dd7b6788 0
1 2224 337
1d81ebda0 Y--P--- 285 saapmgr - 1dcb9bda8 0
1 0 0
1d81ec5f8 S--P--- 4473 informix - 1dbb72ee0 0
6 4 140
1d81ece50 S--P--- 4500 informix - 1dbb72ee0 0
2 7 16
1d81ed6a8 Y--P--- 288 saapmgr - 1e1638e38 0
1 0 0
1d81edf00 Y--P--- 289 saapmgr - 1de7e7028 0
1 0 0
1d81ee758 Y--P--- 290 saapmgr - 1e0cf49b8 0
1 0 0
1d81eefb0 Y--P--- 291 saapmgr - 1de713db0 0
1 0 0
1d81ef808 Y--P--- 292 saapmgr - 1df78b308 0
1 0 23
1d81f0060 Y--P--- 293 saapmgr - 1de9f9ef8 0
1 0 51
1d81f08b8 Y--P--- 297 saapmgr - 1e0886418 0
1 5 0
1d81f1110 Y--P--- 2738 saapmgr - 1dd875ce0 0
1 0 0
1d81f1968 Y--P--- 318 saapmgr - 1dfeff290 0
1 94881 281121
1d81f21c0 S--P--- 4393 informix - 1dbb72ee0 0
2 15 438
1d81f3270 Y--P--- 433 saapmgr - 1df83c278 0
1 5 0
1d81f3ac8 Y--P--- 1078 saapmgr - 1de046f18 0
1 14 0
1d81f4320 S--P--- 4828 informix - 1dbb72ee0 0
2 0 5
1d81f4b78 Y--P--- 4501 informix - 1db2aaca8 0
2 7 18
1e5116880 ---PR-- 5020 saapmgr - 0 0
1 8901 0
1e5117930 Y--P--- 2495 wwwmgr - 1dff80c28 0
1 0 0
1e5118188 L--P--- 4548 informix - 49a59840 120
2 5 13@@
onstat -g wmx and onstat -g lmx to see what is the mutex...
It can be hard to "see"...Regards
On Wed, Aug 20, 2014 at 3:34 PM, FRANK <yunyaoqu@gmail.com> wrote:
> Folks,
>
> Normally I do not see S---- in output of onstat -u, but now I see.
>
> Any comments?
>
> Thanks
> Frank
>
> RHEL 6
>
> [informix@walter ngdc_er_20140820]$ onstat -u
> IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:38:50
> -- 11847728 Kbytes> Userthreads
> address flags sessid user tty wait tout
> locks nreads nwrites
> 1d81b3028 ---P--D 1 informix - 0 0
> 0 398 1015
> 1d81b3880 ---P--F 0 informix - 0 0
> 0 0 12778
> 1d81b40d8 ---P--F 0 informix - 0 0
> 0 0 18875
> 1d81b4930 ---P--F 0 informix - 0 0
> 0 0 11092
> 1d81b5188 ---P--F 0 informix - 0 0
> 0 0 18161
> 1d81b59e0 ---P--F 0 informix - 0 0
> 0 0 26058
> 1d81b6238 ---P--F 0 informix - 0 0
> 0 0 35724
> 1d81b6a90 ---P--F 0 informix - 0 0
> 0 0 12501
> 1d81b72e8 ---P--F 0 informix - 0 0
> 0 0 36521
> 1d81b7b40 ---P--F 0 informix - 0 0
> 0 0 15588
> 1d81b8398 ---P--F 0 informix - 0 0
> 0 0 46021
> 1d81b8bf0 ---P--F 0 informix - 0 0
> 0 0 2964
> 1d81b9448 ---P--F 0 informix - 0 0
> 0 0 183203
> 1d81b9ca0 ---P--F 0 informix - 0 0
> 0 0 47100
> 1d81ba4f8 ---P--F 0 informix - 0 0
> 0 0 18092
> 1d81bad50 ---P--F 0 informix - 0 0
> 0 0 47221
> 1d81bb5a8 ---P--F 0 informix - 0 0
> 0 0 13200
> 1d81bbe00 ---P--- 18 informix - 0 0
> 0 0 63
> 1d81bc658 ---P--B 19 informix - 0 0
> 0 515817 4096
> 1d81bceb0 Y--P--B 20 informix - 1d8cf0e50 0
> 0 459361 3977
> 1d81bdf60 ---P--D 24 informix - 0 0
> 0 4 0
> 1d81be7b8 ---P--D 23 informix - 0 0
> 0 0 0
> 1d81bf010 ---P--- 43 informix - 0 0
> 1 240 21860
> 1d81bf868 Y--P--- 143 informix - 1db2aadc8 0
> 1 8 22528
> 1d81c00c0 ---P--- 33 informix - 0 0
> 0 0 0
> 1d81c0918 Y--P--- 141 informix - 1db2ccb78 0
> 1 9 3
> 1d81c1170 ---P--- 35 informix - 0 0
> 1 0 0
> 1d81c19c8 Y--P--- 36 informix - 1d9910448 0
> 3 0 0
> 1d81c2220 Y--P--D 41 informix - 48bdf970 0
> 0 0 0
> 1d81c2a78 ---P--- 44 informix - 0 0
> 1 666 32901
> 1d81c32d0 ---P--- 45 informix - 0 0
> 1 711 32575
> 1d81c3b28 ---P--- 46 informix - 0 0
> 0 3 0
> 1d81c4380 ---P--- 47 informix - 0 0
> 1 240 442
> 1d81c4bd8 ---P--- 48 informix - 0 0
> 1 237 317
> 1d81c5430 ---P--- 49 informix - 0 0
> 1 197 329
> 1d81c5c88 ---P--- 50 informix - 0 0
> 1 165 552
> 1d81c64e0 ---P--- 51 informix - 0 0
> 2 202 252
> 1d81c6d38 ---P--- 52 informix - 0 0
> 1 304 76
> 1d81c7590 ---P--- 53 informix - 0 0
> 1 211 859
> 1d81c7de8 ---P--- 54 informix - 0 0
> 2 223 298
> 1d81c8640 ---P--- 55 informix - 0 0
> 1 204 414
> 1d81c8e98 ---P--- 56 informix - 0 0
> 1 263 329
> 1d81c96f0 ---P--- 57 informix - 0 0
> 2 237 364
> 1d81c9f48 ---P--- 58 informix - 0 0
> 2 302 174
> 1d81ca7a0 ---P--- 136 informix - 0 0
> 1 4 0
> 1d81caff8 S--P--- 137 informix - 1d9fbc8f0 0
> 1 186322 0
> 1d81cb850 ---P--- 138 informix - 0 0
> 0 272298 0
> 1d81cc0a8 ---P--- 139 informix - 0 0
> 1 44 0
> 1d81cc900 S--P--- 142 informix - 1dbb72ee0 0
> 0 0 211
> 1d81cd158 Y--P--- 144 informix - 1db2aadc8 0
> 1 9 23680
> 1d81cd9b0 S--P--- 4692 informix - 1dbb72ee0 0
> 2 3 4
> 1d81ce208 S--P--- 4531 informix - 1dbb72ee0 0
> 2 0 47
> 1d81cea60 Y--P--- 235 saapmgr - 1def80778 0
> 1 125 0
> 1d81cf2b8 S--P--- 4693 informix - 1dbb72ee0 0
> 2 3 1
> 1d81d0bc0 S--P--- 151 informix - 1dbb72ee0 0
> 2 13166 33392
> 1d81d1418 S--P--- 152 informix - 1dbb72ee0 0
> 2 14148 32374
> 1d81d1c70 S--P--- 153 informix - 1dbb72ee0 0
> 2 13705 31162
> 1d81d2d20 ---P--- 155 informix - 0 0
> 1 8890 0
> 1d81d3dd0 Y--P--- 157 informix - 1db2aaca8 0
> 2 13781 32324
> 1d81d4628 Y--P--- 4827 informix - 1db2aaca8 0
> 2 4 0
> 1d81d4e80 S--P--- 159 informix - 1dbb72ee0 0
> 2 13480 32593
> 1d81d56d8 S--P--- 160 informix - 1dbb72ee0 0
> 2 14200 32926
> 1d81d5f30 Y--P--- 2693 saapmgr - 1e5a69148 0
> 1 0 46
> 1d81d6788 Y--P--- 173 informix - 1db2aadc8 0
> 1 2 17280
> 1d81d6fe0 Y--P--- 238 saapmgr - 1e029cca8 0
> 1 0 0
> 1d81d7838 Y--P--- 239 saapmgr - 1e3846028 0
> 1 2 0
> 1d81d8090 Y--P--- 237 saapmgr - 1dc95d4c8 0
> 1 1 0
> 1d81d88e8 Y--P--- 260 saapmgr - 1dcaccd78 0
> 2 757 3042
> 1d81d9140 Y--P--- 244 saapmgr - 1dfffeb78 0
> 1 40 0
> 1d81d9998 Y--P--- 258 saapmgr - 1e3176be0 0
> 1 0 0
> 1d81da1f0 Y--P--- 236 saapmgr - 1de8514c8 0
> 1 1 0
> 1d81daa48 Y--P--- 241 saapmgr - 1e0d85c28 0
> 1 2 0
> 1d81db2a0 Y--P--- 262 saapmgr - 1dcb5d538 0
> 2 454 2717
> 1d81dbaf8 Y--P--- 242 saapmgr - 1dcadf148 0
> 1 0 0
> 1d81dc350 Y--P--- 240 saapmgr - 1de777308 0
> 1 1 0
> 1d81dcba8 Y--P--- 254 saapmgr - 1dcb2f788 0
> 1 3 0
> 1d81dd400 Y--P--- 243 saapmgr - 1dd7a1df8 0
> 1 1 0
> 1d81ddc58 Y--P--- 259 saapmgr - 1dcb5d1d8 0
> 2 820 2800
> 1d81de4b0 Y--P--- 256 saapmgr - 1ddc9ac68 0
> 1 9 0
> 1d81ded08 Y--P--- 2647 wwwmgr - 1dd5c9708 0
> 1 26 76
> 1d81df560 Y--P--- 261 saapmgr - 1dd7ec2f8 0
> 2 773 3510
> 1d81dfdb8 Y--P--- 263 saapmgr - 1e12b86c0 0
> 2 743 3245
> 1d81e0610 Y--P--- 246 saapmgr - 1dd7c64a8 0
> 1 0 0
> 1d81e0e68 Y--P--- 264 saapmgr - 1e12b8ca8 0
> 2 440 3917
> 1d81e16c0 Y--P--- 265 saapmgr - 1dea7b3d0 0
> 2 727 3044
> 1d81e1f18 Y--P--- 266 saapmgr - 1dea7ba48 0
> 1 460 3202
> 1d81e2770 Y--P--- 267 saapmgr - 1e12b8f78 0
> 2 651 3378
> 1d81e2fc8 Y--P--- 268 saapmgr - 1e04de580 0
> 1 891 3256
> 1d81e3820 Y--P--- 269 saapmgr - 1e04debf8 0
> 1 856 1164
> 1d81e4078 Y--P--- 270 saapmgr - 1de6233d0 0
> 1 2122 1273
> 1d81e48d0 Y--P--- 271 saapmgr - 1e00966e8 0
> 1 69 1045
> 1d81e5128 Y--P--- 272 saapmgr - 1de623df0 0
> 1 127 1460
> 1d81e5980 Y--P--- 273 saapmgr - 1df259318 0
> 1 63 1225
> 1d81e61d8 Y--P--- 274 saapmgr - 1df259900 0
> 1 65 1009
> 1d81e6a30 Y--P--- 275 saapmgr - 1e028fa88 0
> 1 86 1122
> 1d81e7288 Y--P--- 276 saapmgr - 1e0324740 0
> 1 73 1041
> 1d81e7ae0 Y--P--- 277 saapmgr - 1e0324d28 0
> 1 117 1273
> 1d81e8338 Y--P--- 278 saapmgr - 1dcb43958 0
> 1 69 1117
> 1d81e8b90 Y--P--- 279 saapmgr - 1e317a630 0
> 1 128223 247740
> 1d81e93e8 Y--P--- 2737 saapmgr - 1dbb7f268 0
> 1 0 26
> 1d81e9c40 Y--P--- 4588 saapmgr - 1e182d428 0
> 1 1 0
> 1d81ea498 Y--P--- 282 saapmgr - 1dd7f96e8 0
> 1 294 6235
> 1d81eacf0 Y--P--- 283 saapmgr - 1dd7f9808 0
> 1 186523 286151
> 1d81eb548 Y--P--- 284 saapmgr - 1dd7b6788 0
> 1 2224 337
> 1d81ebda0 Y--P--- 285 saapmgr - 1dcb9bda8 0
> 1 0 0
> 1d81ec5f8 S--P--- 4473 informix - 1dbb72ee0 0
> 6 4 140
> 1d81ece50 S--P--- 4500 informix - 1dbb72ee0 0
> 2 7 16
> 1d81ed6a8 Y--P--- 288 saapmgr - 1e1638e38 0
> 1 0 0
> 1d81edf00 Y--P--- 289 saapmgr - 1de7e7028 0
> 1 0 0
> 1d81ee758 Y--P--- 290 saapmgr - 1e0cf49b8 0
> 1 0 0
> 1d81eefb0 Y--P--- 291 saapmgr - 1de713db0 0
> 1 0 0
> 1d81ef808 Y--P--- 292 saapmgr - 1df78b308 0
> 1 0 23
> 1d81f0060 Y--P--- 293 saapmgr - 1de9f9ef8 0
> 1 0 5
Are they fleeting ? Run onstat -g wmx|lmx but you hopefully won't catch
them
Cheers
Paul
> -----Original Message-----
> From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of
> FRANK
> Sent: Wednesday, August 20, 2014 9:34 AM
> To: ids@iiug.org
> Subject: wait for mutex [33534]
>
> Folks,
>
> Normally I do not see S---- in output of onstat -u, but now I see.
>
> Any comments?
>
> Thanks
> Frank
>
> RHEL 6
>
> [informix@walter ngdc_er_20140820]$ onstat -u
> IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:38:50
> -- 11847728 Kbytes> Userthreads
> address flags sessid user tty wait tout
> locks nreads nwrites
> 1d81b3028 ---P--D 1 informix - 0 0
> 0 398 1015
> 1d81b3880 ---P--F 0 informix - 0 0
> 0 0 12778
> 1d81b40d8 ---P--F 0 informix - 0 0
> 0 0 18875
> 1d81b4930 ---P--F 0 informix - 0 0
> 0 0 11092
> 1d81b5188 ---P--F 0 informix - 0 0
> 0 0 18161
> 1d81b59e0 ---P--F 0 informix - 0 0
> 0 0 26058
> 1d81b6238 ---P--F 0 informix - 0 0
> 0 0 35724
> 1d81b6a90 ---P--F 0 informix - 0 0
> 0 0 12501
> 1d81b72e8 ---P--F 0 informix - 0 0
> 0 0 36521
> 1d81b7b40 ---P--F 0 informix - 0 0
> 0 0 15588
> 1d81b8398 ---P--F 0 informix - 0 0
> 0 0 46021
> 1d81b8bf0 ---P--F 0 informix - 0 0
> 0 0 2964
> 1d81b9448 ---P--F 0 informix - 0 0
> 0 0 183203
> 1d81b9ca0 ---P--F 0 informix - 0 0
> 0 0 47100
> 1d81ba4f8 ---P--F 0 informix - 0 0
> 0 0 18092
> 1d81bad50 ---P--F 0 informix - 0 0
> 0 0 47221
> 1d81bb5a8 ---P--F 0 informix - 0 0
> 0 0 13200
> 1d81bbe00 ---P--- 18 informix - 0 0
> 0 0 63
> 1d81bc658 ---P--B 19 informix - 0 0
> 0 515817 4096
> 1d81bceb0 Y--P--B 20 informix - 1d8cf0e50 0
> 0 459361 3977
> 1d81bdf60 ---P--D 24 informix - 0 0
> 0 4 0
> 1d81be7b8 ---P--D 23 informix - 0 0
> 0 0 0
> 1d81bf010 ---P--- 43 informix - 0 0
> 1 240 21860
> 1d81bf868 Y--P--- 143 informix - 1db2aadc8 0
> 1 8 22528
> 1d81c00c0 ---P--- 33 informix - 0 0
> 0 0 0
> 1d81c0918 Y--P--- 141 informix - 1db2ccb78 0
> 1 9 3
> 1d81c1170 ---P--- 35 informix - 0 0
> 1 0 0
> 1d81c19c8 Y--P--- 36 informix - 1d9910448 0
> 3 0 0
> 1d81c2220 Y--P--D 41 informix - 48bdf970 0
> 0 0 0
> 1d81c2a78 ---P--- 44 informix - 0 0
> 1 666 32901
> 1d81c32d0 ---P--- 45 informix - 0 0
> 1 711 32575
> 1d81c3b28 ---P--- 46 informix - 0 0
> 0 3 0
> 1d81c4380 ---P--- 47 informix - 0 0
> 1 240 442
> 1d81c4bd8 ---P--- 48 informix - 0 0
> 1 237 317
> 1d81c5430 ---P--- 49 informix - 0 0
> 1 197 329
> 1d81c5c88 ---P--- 50 informix - 0 0
> 1 165 552
> 1d81c64e0 ---P--- 51 informix - 0 0
> 2 202 252
> 1d81c6d38 ---P--- 52 informix - 0 0
> 1 304 76
> 1d81c7590 ---P--- 53 informix - 0 0
> 1 211 859
> 1d81c7de8 ---P--- 54 informix - 0 0
> 2 223 298
> 1d81c8640 ---P--- 55 informix - 0 0
> 1 204 414
> 1d81c8e98 ---P--- 56 informix - 0 0
> 1 263 329
> 1d81c96f0 ---P--- 57 informix - 0 0
> 2 237 364
> 1d81c9f48 ---P--- 58 informix - 0 0
> 2 302 174
> 1d81ca7a0 ---P--- 136 informix - 0 0
> 1 4 0
> 1d81caff8 S--P--- 137 informix - 1d9fbc8f0 0
> 1 186322 0
> 1d81cb850 ---P--- 138 informix - 0 0
> 0 272298 0
> 1d81cc0a8 ---P--- 139 informix - 0 0
> 1 44 0
> 1d81cc900 S--P--- 142 informix - 1dbb72ee0 0
> 0 0 211
> 1d81cd158 Y--P--- 144 informix - 1db2aadc8 0
> 1 9 23680
> 1d81cd9b0 S--P--- 4692 informix - 1dbb72ee0 0
> 2 3 4
> 1d81ce208 S--P--- 4531 informix - 1dbb72ee0 0
> 2 0 47
> 1d81cea60 Y--P--- 235 saapmgr - 1def80778 0
> 1 125 0
> 1d81cf2b8 S--P--- 4693 informix - 1dbb72ee0 0
> 2 3 1
> 1d81d0bc0 S--P--- 151 informix - 1dbb72ee0 0
> 2 13166 33392
> 1d81d1418 S--P--- 152 informix - 1dbb72ee0 0
> 2 14148 32374
> 1d81d1c70 S--P--- 153 informix - 1dbb72ee0 0
> 2 13705 31162
> 1d81d2d20 ---P--- 155 informix - 0 0
> 1 8890 0
> 1d81d3dd0 Y--P--- 157 informix - 1db2aaca8 0
> 2 13781 32324
> 1d81d4628 Y--P--- 4827 informix - 1db2aaca8 0
> 2 4 0
> 1d81d4e80 S--P--- 159 informix - 1dbb72ee0 0
> 2 13480 32593
> 1d81d56d8 S--P--- 160 informix - 1dbb72ee0 0
> 2 14200 32926
> 1d81d5f30 Y--P--- 2693 saapmgr - 1e5a69148 0
> 1 0 46
> 1d81d6788 Y--P--- 173 informix - 1db2aadc8 0
> 1 2 17280
> 1d81d6fe0 Y--P--- 238 saapmgr - 1e029cca8 0
> 1 0 0
> 1d81d7838 Y--P--- 239 saapmgr - 1e3846028 0
> 1 2 0
> 1d81d8090 Y--P--- 237 saapmgr - 1dc95d4c8 0
> 1 1 0
> 1d81d88e8 Y--P--- 260 saapmgr - 1dcaccd78 0
> 2 757 3042
> 1d81d9140 Y--P--- 244 saapmgr - 1dfffeb78 0
> 1 40 0
> 1d81d9998 Y--P--- 258 saapmgr - 1e3176be0 0
> 1 0 0
> 1d81da1f0 Y--P--- 236 saapmgr - 1de8514c8 0
> 1 1 0
> 1d81daa48 Y--P--- 241 saapmgr - 1e0d85c28 0
> 1 2 0
> 1d81db2a0 Y--P--- 262 saapmgr - 1dcb5d538 0
> 2 454 2717
> 1d81dbaf8 Y--P--- 242 saapmgr - 1dcadf148 0
> 1 0 0
> 1d81dc350 Y--P--- 240 saapmgr - 1de777308 0
> 1 1 0
> 1d81dcba8 Y--P--- 254 saapmgr - 1dcb2f788 0
> 1 3 0
> 1d81dd400 Y--P--- 243 saapmgr - 1dd7a1df8 0
> 1 1 0
> 1d81ddc58 Y--P--- 259 saapmgr - 1dcb5d1d8 0
> 2 820 2800
> 1d81de4b0 Y--P--- 256 saapmgr - 1ddc9ac68 0
> 1 9 0
> 1d81ded08 Y--P--- 2647 wwwmgr - 1dd5c9708 0
> 1 26 76
> 1d81df560 Y--P--- 261 saapmgr - 1dd7ec2f8 0
> 2 773 3510
> 1d81dfdb8 Y--P--- 263 saapmgr - 1e12b86c0 0
> 2 743 3245
> 1d81e0610 Y--P--- 246 saapmgr - 1dd7c64a8 0
> 1 0 0
> 1d81e0e68 Y--P--- 264 saapmgr - 1e12b8ca8 0
> 2 440 3917
> 1d81e16c0 Y--P--- 265 saapmgr - 1dea7b3d0 0
> 2 727 3044
> 1d81e1f18 Y--P--- 266 saapmgr - 1dea7ba48 0
> 1 460 3202
> 1d81e2770 Y--P--- 267 saapmgr - 1e12b8f78 0
> 2 651 3378
> 1d81e2fc8 Y--P--- 268 saapmgr - 1e04de580 0
> 1 891 3256
> 1d81e3820 Y--P--- 269 saapmgr - 1e04debf8 0
> 1 856 1164
> 1d81e4078 Y--P--- 270 saapmgr - 1de6233d0 0
> 1 2122 1273
> 1d81e48d0 Y--P--- 271 saapmgr - 1e00966e8 0
> 1 69 1045
> 1d81e5128 Y--P--- 272 saapmgr - 1de623df0 0
> 1 127 1460
> 1d81e5980 Y--P--- 273 saapmgr - 1df259318 0
> 1 63 1225
> 1d81e61d8 Y--P--- 274 saapmgr - 1df259900 0
> 1 65 1009
> 1d81e6a30 Y--P--- 275 saapmgr - 1e028fa88 0
> 1 86 1122
> 1d81e7288 Y--P--- 276 saapmgr - 1e0324740 0
> 1 73 1041
> 1d81e7ae0 Y--P--- 277 saapmgr - 1e0324d28 0
> 1 117 1273
> 1d81e8338 Y--P--- 278 saapmgr - 1dcb43958 0
> 1 69 1117
> 1d81e8b90 Y--P--- 279 saapmgr - 1e317a630 0
> 1 128223 247740
> 1d81e93e8 Y--P--- 2737 saapmgr - 1dbb7f268 0
> 1 0 26
> 1d81e9c40 Y--P--- 4588 saapmgr - 1e182d428 0
> 1 1 0
> 1d81ea498 Y--P--- 282 saapmgr - 1dd7f96e8 0
> 1 294 6235
> 1d81eacf0 Y--P--- 283 saapmgr - 1dd7f9808 0
> 1 186523 286151
> 1d81eb548 Y--P--- 284 saapmgr - 1dd7b6788 0
> 1 2224 337
> 1d81ebda0 Y--P--- 285 saapmgr - 1dcb9bda8 0
> 1 0 0
> 1d81ec5f8 S--P--- 4473 informix - 1dbb72ee0 0
> 6 4 140
> 1d81ece50 S--P--- 4500 informix - 1dbb72ee0 0
> 2 7 16
> 1d81ed6a8 Y--P--- 288 saapmgr - 1e1638e38 0
> 1 0 0
> 1d81edf00 Y--P--- 289 saapmgr - 1de7e7028 0
> 1 0 0
> 1d81ee758 Y--P--- 290 saapmgr - 1e0cf49b8 0
> 1 0 0
> 1d81eefb0 Y--P--- 291 saap
Thanks, Fernando!
Here it is, confused ...
We indeed have ER slow/ data queued issue now
Frank
informix@walter ~]$ onstat -g wmx
IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:54:05
-- 11847728 KbytesMutexes with waiters:
mid addr name holder lkcnt waiter
waittime
6543 1d9fbc8f0 rqmut_trg_send 5496 0 227 0
5504 0
98 0
196 0
107 0
5488 0
7605 1dbb72ee0 CDRDSn0022 4484 0 210 17
4430 17
4585 17
4754 17
4528 17
4481 17
4897 17
4527 17
4526 17
4608 17
4752 17
214 17
205 17
195 16
4554 16
206 16
213 16
4618 16
4695 16
4425 16
4619 16
4560 16
4428 16
4500 16
4448 16
4555 16
4603 16
4609 16
4584 16
4607 16
4502 16
4447 16
4604 16
4896 16
4602 16
4895 16
4753 16
204 16
4426 16
4634 16
Number of mutexes on VP free lists: 10735
Read/Write Locks
addr id name slocks xlock waiters
[informix@walter ~]$ onstat -g lmx
IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:54:17
-- 11847728 KbytesLocked mutexes:
mid addr name holder lkcnt waiter
waittime
6307 1d9c113d0 qtrMutex31 5488 0
6543 1d9fbc8f0 rqmut_trg_send 5496 0 5495 0
5488 0
6547 1d9fbcb90 rqrest_trg_send 5496 0
7605 1dbb72ee0 CDRDSn0022 4484 0 210 29
4430 29
4585 29
4754 29
4528 29
4481 29
4897 29
4527 29
4526 29
4608 29
4752 29
214 29
205 29
195 28
4554 28
206 28
213 28
4618 28
4695 28
4425 28
4619 28
4560 28
4428 28
4500 28
4448 28
4555 28
4603 28
4609 28
4584 28
4607 28
4502 28
4447 28
4604 28
4896 28
4602 28
4895 28
4753 28
204 28
4426 28
4634 28
Number of mutexes on VP free lists: 12879
Read/Write Locks
addr id name slocks xlock waiters
[informix@walter ~]$
On Wed, Aug 20, 2014 at 10:37 AM, Fernando Nunes <domusonline@gmail.com>
wrote:
> onstat -g wmx and onstat -g lmx to see what is the mutex...
> It can be hard to "see"...> Regards
>
> On Wed, Aug 20, 2014 at 3:34 PM, FRANK <yunyaoqu@gmail.com> wrote:
>
> > Folks,
> >
> > Normally I do not see S---- in output of onstat -u, but now I see.
> >
> > Any comments?
> >
> > Thanks
> > Frank
> >
> > RHEL 6
> >
> > [informix@walter ngdc_er_20140820]$ onstat -u
> > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:38:50
> > -- 11847728 Kbytes> > Userthreads
> > address flags sessid user tty wait tout
> > locks nreads nwrites
> > 1d81b3028 ---P--D 1 informix - 0 0
> > 0 398 1015
> > 1d81b3880 ---P--F 0 informix - 0 0
> > 0 0 12778
> > 1d81b40d8 ---P--F 0 informix - 0 0
> > 0 0 18875
> > 1d81b4930 ---P--F 0 informix - 0 0
> > 0 0 11092
> > 1d81b5188 ---P--F 0 informix - 0 0
> > 0 0 18161
> > 1d81b59e0 ---P--F 0 informix - 0 0
> > 0 0 26058
> > 1d81b6238 ---P--F 0 informix - 0 0
> > 0 0 35724
> > 1d81b6a90 ---P--F 0 informix - 0 0
> > 0 0 12501
> > 1d81b72e8 ---P--F 0 informix - 0 0
> > 0 0 36521
> > 1d81b7b40 ---P--F 0 informix - 0 0
> > 0 0 15588
> > 1d81b8398 ---P--F 0 informix - 0 0
> > 0 0 46021
> > 1d81b8bf0 ---P--F 0 informix - 0 0
> > 0 0 2964
> > 1d81b9448 ---P--F 0 informix - 0 0
> > 0 0 183203
> > 1d81b9ca0 ---P--F 0 informix - 0 0
> > 0 0 47100
> > 1d81ba4f8 ---P--F 0 informix - 0 0
> > 0 0 18092
> > 1d81bad50 ---P--F 0 informix - 0 0
> > 0 0 47221
> > 1d81bb5a8 ---P--F 0 informix - 0 0
> > 0 0 13200
> > 1d81bbe00 ---P--- 18 informix - 0 0
> > 0 0 63
> > 1d81bc658 ---P--B 19 informix - 0 0
> > 0 515817 4096
> > 1d81bceb0 Y--P--B 20 informix - 1d8cf0e50 0
> > 0 459361 3977
> > 1d81bdf60 ---P--D 24 informix - 0 0
> > 0 4 0
> > 1d81be7b8 ---P--D 23 informix - 0 0
> > 0 0 0
> > 1d81bf010 ---P--- 43 informix - 0 0
> > 1 240 21860
> > 1d81bf868 Y--P--- 143 informix - 1db2aadc8 0
> > 1 8 22528
> > 1d81c00c0 ---P--- 33 informix - 0 0
> > 0 0 0
> > 1d81c0918 Y--P--- 141 informix - 1db2ccb78 0
> > 1 9 3
> > 1d81c1170 ---P--- 35 informix - 0 0
> > 1 0 0
> > 1d81c19c8 Y--P--- 36 informix - 1d9910448 0
> > 3 0 0
> > 1d81c2220 Y--P--D 41 informix - 48bdf970 0
> > 0 0 0
> > 1d81c2a78 ---P--- 44 informix - 0 0
> > 1 666 32901
> > 1d81c32d0 ---P--- 45 informix - 0 0
> > 1 711 32575
> > 1d81c3b28 ---P--- 46 informix - 0 0
> > 0 3 0
> > 1d81c4380 ---P--- 47 informix - 0 0
> > 1 240 442
> > 1d81c4bd8 ---P--- 48 informix - 0 0
> > 1 237 317
> > 1d81c5430 ---P--- 49 informix - 0 0
> > 1 197 329
> > 1d81c5c88 ---P--- 50 informix - 0 0
> > 1 165 552
> > 1d81c64e0 ---P--- 51 informix - 0 0
> > 2 202 252
> > 1d81c6d38 ---P--- 52 informix - 0 0
> > 1 304 76
> > 1d81c7590 ---P--- 53 informix - 0 0
> > 1 211 859
> > 1d81c7de8 ---P--- 54 informix - 0 0
> > 2 223 298
> > 1d81c8640 ---P--- 55 informix - 0 0
> > 1 204 414
> > 1d81c8e98 ---P--- 56 informix - 0 0
> > 1 263 329
> > 1d81c96f0 ---P--- 57 informix - 0 0
> > 2 237 364
> > 1d81c9f48 ---P--- 58 informix - 0 0
> > 2 302 174
> > 1d81ca7a0 ---P--- 136 informix - 0 0
> > 1 4 0
> > 1d81caff8 S--P--- 137 informix - 1d9fbc8f0 0
> > 1 186322 0
> > 1d81cb850 ---P--- 138 informix - 0 0
> > 0 272298 0
> > 1d81cc0a8 ---P--- 139 informix - 0 0
> > 1 44 0
> > 1d81cc900 S--P--- 142 informix - 1dbb72ee0 0
> > 0 0 211
> > 1d81cd158 Y--P--- 144 informix - 1db2aadc8 0
> > 1 9 23680
> > 1d81cd9b0 S--P--- 4692 informix - 1dbb72ee0 0
> > 2 3 4
> > 1d81ce208 S--P--- 4531 informix - 1dbb72ee0 0
> > 2 0 47
> > 1d81cea60 Y--P--- 235 saapmgr - 1def80778 0
> > 1 125 0
> > 1d81cf2b8 S--P--- 4693 informix - 1dbb72ee0 0
> > 2 3 1
> > 1d81d0bc0 S--P--- 151 informix - 1dbb72ee0 0
> > 2 13166 33392
> > 1d81d1418 S--P--- 152 informix - 1dbb72ee0 0
> > 2 14148 32374
> > 1d81d1c70 S--P--- 153 informix - 1dbb72ee0 0
> > 2 13705 31162
> > 1d81d2d20 ---P--- 155 informix - 0 0
> > 1 8890 0
> > 1d81d3dd0 Y--P--- 157 informix - 1db2aaca8 0
> > 2 13781 32324
> > 1d81d4628 Y--P--- 4827 informix - 1db2aaca8 0
> > 2 4 0
> > 1d81d4e80 S--P--- 159 informix - 1dbb72ee0 0
> > 2 13480 32593
> > 1d81d56d8 S--P--- 160 informix - 1dbb72ee0 0
> > 2 14200 32926
> > 1d81d5f30 Y--P--- 2693 saapmgr - 1e5a69148 0
> > 1 0 46
> > 1d81d6788 Y--P--- 173 informix - 1db2aadc8 0
> > 1 2 17280
> > 1d81d6fe0 Y--P--- 238 saapmgr - 1e029cca8 0
> > 1 0 0
> > 1d81d7838 Y--P--- 239 saapmgr - 1e3846028 0
> > 1 2 0
> > 1d81d8090 Y--P--- 237 saapmgr - 1dc95d4c8 0
> > 1 1 0
> > 1d81d88
In 12.10xc4, we rewrote the way that the ER queues were locked. This
significantly reduced lock contention and improved ER performance by close
to 1200%
Sent from my iPad
> On Aug 20, 2014, at 8:49 AM, "FRANK" <yunyaoqu@gmail.com> wrote:
>
> Thanks, Fernando!
>
> Here it is, confused ...
> We indeed have ER slow/ data queued issue now
>
> Frank
>
> informix@walter ~]$ onstat -g wmx
> IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:54:05
> -- 11847728 Kbytes> Mutexes with waiters:
> mid addr name holder lkcnt waiter
> waittime
> 6543 1d9fbc8f0 rqmut_trg_send 5496 0 227 0
>
> 5504 0
>
> 98 0
>
> 196 0
>
> 107 0
>
> 5488 0
> 7605 1dbb72ee0 CDRDSn0022 4484 0 210 17
>
> 4430 17
>
> 4585 17
>
> 4754 17
>
> 4528 17
>
> 4481 17
>
> 4897 17
>
> 4527 17
>
> 4526 17
>
> 4608 17
>
> 4752 17
>
> 214 17
>
> 205 17
>
> 195 16
>
> 4554 16
>
> 206 16
>
> 213 16
>
> 4618 16
>
> 4695 16
>
> 4425 16
>
> 4619 16
>
> 4560 16
>
> 4428 16
>
> 4500 16
>
> 4448 16
>
> 4555 16
>
> 4603 16
>
> 4609 16
>
> 4584 16
>
> 4607 16
>
> 4502 16
>
> 4447 16
>
> 4604 16
>
> 4896 16
>
> 4602 16
>
> 4895 16
>
> 4753 16
>
> 204 16
>
> 4426 16
>
> 4634 16
> Number of mutexes on VP free lists: 10735
> Read/Write Locks
> addr id name slocks xlock waiters
> [informix@walter ~]$ onstat -g lmx
> IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:54:17
> -- 11847728 Kbytes> Locked mutexes:
> mid addr name holder lkcnt waiter
> waittime
> 6307 1d9c113d0 qtrMutex31 5488 0
> 6543 1d9fbc8f0 rqmut_trg_send 5496 0 5495 0
>
> 5488 0
> 6547 1d9fbcb90 rqrest_trg_send 5496 0
> 7605 1dbb72ee0 CDRDSn0022 4484 0 210 29
>
> 4430 29
>
> 4585 29
>
> 4754 29
>
> 4528 29
>
> 4481 29
>
> 4897 29
>
> 4527 29
>
> 4526 29
>
> 4608 29
>
> 4752 29
>
> 214 29
>
> 205 29
>
> 195 28
>
> 4554 28
>
> 206 28
>
> 213 28
>
> 4618 28
>
> 4695 28
>
> 4425 28
>
> 4619 28
>
> 4560 28
>
> 4428 28
>
> 4500 28
>
> 4448 28
>
> 4555 28
>
> 4603 28
>
> 4609 28
>
> 4584 28
>
> 4607 28
>
> 4502 28
>
> 4447 28
>
> 4604 28
>
> 4896 28
>
> 4602 28
>
> 4895 28
>
> 4753 28
>
> 204 28
>
> 4426 28
>
> 4634 28
> Number of mutexes on VP free lists: 12879
> Read/Write Locks
> addr id name slocks xlock waiters
> [informix@walter ~]$
>
> On Wed, Aug 20, 2014 at 10:37 AM, Fernando Nunes <domusonline@gmail.com>
> wrote:
>
> > onstat -g wmx and onstat -g lmx to see what is the mutex...
> > It can be hard to "see"...> > Regards
> >
> > On Wed, Aug 20, 2014 at 3:34 PM, FRANK <yunyaoqu@gmail.com> wrote:
> >
> > > Folks,
> > >
> > > Normally I do not see S---- in output of onstat -u, but now I see.
> > >
> > > Any comments?
> > >
> > > Thanks
> > > Frank
> > >
> > > RHEL 6
> > >
> > > [informix@walter ngdc_er_20140820]$ onstat -u
> > > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up
00:38:50
> > > -- 11847728 Kbytes> > > Userthreads
> > > address flags sessid user tty wait tout
> > > locks nreads nwrites
> > > 1d81b3028 ---P--D 1 informix - 0 0
> > > 0 398 1015
> > > 1d81b3880 ---P--F 0 informix - 0 0
> > > 0 0 12778
> > > 1d81b40d8 ---P--F 0 informix - 0 0
> > > 0 0 18875
> > > 1d81b4930 ---P--F 0 informix - 0 0
> > > 0 0 11092
> > > 1d81b5188 ---P--F 0 informix - 0 0
> > > 0 0 18161
> > > 1d81b59e0 ---P--F 0 informix - 0 0
> > > 0 0 26058
> > > 1d81b6238 ---P--F 0 informix - 0 0
> > > 0 0 35724
> > > 1d81b6a90 ---P--F 0 informix - 0 0
> > > 0 0 12501
> > > 1d81b72e8 ---P--F 0 informix - 0 0
> > > 0 0 36521
> > > 1d81b7b40 ---P--F 0 informix - 0 0
> > > 0 0 15588
> > > 1d81b8398 ---P--F 0 informix - 0 0
> > > 0 0 46021
> > > 1d81b8bf0 ---P--F 0 informix - 0 0
> > > 0 0 2964
> > > 1d81b9448 ---P--F 0 informix - 0 0
> > > 0 0 183203
> > > 1d81b9ca0 ---P--F 0 informix - 0 0
> > > 0 0 47100
> > > 1d81ba4f8 ---P--F 0 informix - 0 0
> > > 0 0 18092
> > > 1d81bad50 ---P--F 0 informix - 0 0
> > > 0 0 47221
> > > 1d81bb5a8 ---P--F 0 informix - 0 0
> > > 0 0 13200
> > > 1d81bbe00 ---P--- 18 informix - 0 0
> > > 0 0 63
> > > 1d81bc658 ---P--B 19 informix - 0 0
> > > 0 515817 4096
> > > 1d81bceb0 Y--P--B 20 informix - 1d8cf0e50 0
> > > 0 459361 3977
> > > 1d81bdf60 ---P--D 24 informix - 0 0
> > > 0 4 0
> > > 1d81be7b8 ---P--D 23 informix - 0 0
> > > 0 0 0
> > > 1d81bf010 ---P--- 43 informix - 0 0
> > > 1 240 21860
> > > 1d81bf868 Y--P--- 143 informix - 1db2aadc8 0
> > > 1 8 22528
> > > 1d81c00c0 ---P--- 33 informix - 0 0
> > > 0 0 0
> > > 1d81c0918 Y--P--- 141 informix - 1db2ccb78 0
> > > 1 9 3
> > > 1d81c1170 ---P--- 35 informix - 0 0
> > > 1 0 0
> > > 1d81c19c8 Y--P--- 36 informix - 1d9910448 0
> > > 3 0 0
> > > 1d81c2220 Y--P--D 41 informix - 48bdf970 0
> > > 0 0 0
> > > 1d81c2a78 ---P--- 44 informix - 0 0
> > > 1 666 32901
> > > 1d81c32d0 ---P--- 45 informix - 0 0
> > > 1 711 32575
> > > 1d81c3b28 ---P--- 46 informix - 0 0
> > > 0 3 0
> > > 1d81c4380 ---P--- 47 informix - 0 0
> > > 1 240 442
> > > 1d81c4bd8 ---P--- 48 informix - 0 0
> > > 1 237 317
> > > 1d81c5430 ---P--- 49 informix - 0 0
> > > 1 197 329
> > > 1d81c5c88 ---P--- 50 informix - 0 0
> > > 1 165 552
> > > 1d81c64e0 ---P--- 51 informix - 0 0
> > > 2 202 252
> > > 1d81c6d38 ---P--- 52 informix - 0 0
> > > 1 304 76
> > > 1d81c7590 ---P--- 53 informix - 0 0
> > > 1 211 859
> > > 1d81c7de8 ---P--- 54 informix - 0 0
> > > 2 223 298
> > > 1d81c8640 ---P--- 55 informix - 0 0
> > > 1 204 414
> > > 1d81c8e98 ---P--- 56 informix - 0 0
> > > 1 263 329
> > > 1d81c96f0 ---P--- 57 informix - 0 0
> > > 2 237 364
> > > 1d81c9f48 ---P--- 58 informix - 0 0
> > > 2 302 174
> > > 1d81ca7a0 ---P--- 136 informix - 0 0
> > > 1 4 0
> > > 1d81caff8 S--P--- 137 informix - 1d9fbc8f0 0
> > > 1 186322 0
> > > 1d81cb850 ---P--- 138 informix - 0 0
> > > 0 272298 0
> > > 1d81cc0a8 ---P--- 139 informix - 0 0
> > > 1 44 0
> > > 1d81cc900 S--P--- 142 informix - 1dbb72ee0 0
> > > 0 0 211
> > > 1d81cd158 Y--P--- 144 informix - 1db2aadc8 0
> > > 1 9 23680
> > > 1d81cd9b0 S--P--- 4692 informix - 1dbb72ee0 0
> > > 2 3 4
> > > 1d81ce208 S--P--- 4531 informix - 1dbb72ee0 0
> > > 2 0 47
> > > 1d81cea60 Y--P--- 235 saapmgr - 1def80778 0
> > > 1 125 0
> > > 1d81cf2b8 S--P--- 4693 informix - 1dbb72ee0 0
> > > 2 3 1
> > > 1d81d0bc0 S--P--- 151 informix - 1dbb72ee0 0
> > > 2 13166 33392@
So, Madison, you are suggesting that Frank upgrade?
Art
Art S. Kagel, Principal Consultant
ASK Database Management
Blog: http://informix-myview.blogspot.com/
Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on the IIUG, nor any other organization with which I am
associated either explicitly, implicitly, or by inference. Neither do
those opinions reflect those of other individuals affiliated with any
entity with which I am affiliated nor those of the entities themselves.
On Wed, Aug 20, 2014 at 11:05 AM, Madison Pruet <mpruet@us.ibm.com> wrote:
> In 12.10xc4, we rewrote the way that the ER queues were locked. This
> significantly reduced lock contention and improved ER performance by close
> to 1200%
>
> Sent from my iPad
>
> > On Aug 20, 2014, at 8:49 AM, "FRANK" <yunyaoqu@gmail.com> wrote:
> >
> > Thanks, Fernando!
> >
> > Here it is, confused ...
> > We indeed have ER slow/ data queued issue now
> >
> > Frank
> >
> > informix@walter ~]$ onstat -g wmx
> > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:54:05
> > -- 11847728 Kbytes> > Mutexes with waiters:
> > mid addr name holder lkcnt waiter
> > waittime
> > 6543 1d9fbc8f0 rqmut_trg_send 5496 0 227 0
> >
> > 5504 0
> >
> > 98 0
> >
> > 196 0
> >
> > 107 0
> >
> > 5488 0
> > 7605 1dbb72ee0 CDRDSn0022 4484 0 210 17
> >
> > 4430 17
> >
> > 4585 17
> >
> > 4754 17
> >
> > 4528 17
> >
> > 4481 17
> >
> > 4897 17
> >
> > 4527 17
> >
> > 4526 17
> >
> > 4608 17
> >
> > 4752 17
> >
> > 214 17
> >
> > 205 17
> >
> > 195 16
> >
> > 4554 16
> >
> > 206 16
> >
> > 213 16
> >
> > 4618 16
> >
> > 4695 16
> >
> > 4425 16
> >
> > 4619 16
> >
> > 4560 16
> >
> > 4428 16
> >
> > 4500 16
> >
> > 4448 16
> >
> > 4555 16
> >
> > 4603 16
> >
> > 4609 16
> >
> > 4584 16
> >
> > 4607 16
> >
> > 4502 16
> >
> > 4447 16
> >
> > 4604 16
> >
> > 4896 16
> >
> > 4602 16
> >
> > 4895 16
> >
> > 4753 16
> >
> > 204 16
> >
> > 4426 16
> >
> > 4634 16
> > Number of mutexes on VP free lists: 10735
> > Read/Write Locks
> > addr id name slocks xlock waiters
> > [informix@walter ~]$ onstat -g lmx
> > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:54:17
> > -- 11847728 Kbytes> > Locked mutexes:
> > mid addr name holder lkcnt waiter
> > waittime
> > 6307 1d9c113d0 qtrMutex31 5488 0
> > 6543 1d9fbc8f0 rqmut_trg_send 5496 0 5495 0
> >
> > 5488 0
> > 6547 1d9fbcb90 rqrest_trg_send 5496 0
> > 7605 1dbb72ee0 CDRDSn0022 4484 0 210 29
> >
> > 4430 29
> >
> > 4585 29
> >
> > 4754 29
> >
> > 4528 29
> >
> > 4481 29
> >
> > 4897 29
> >
> > 4527 29
> >
> > 4526 29
> >
> > 4608 29
> >
> > 4752 29
> >
> > 214 29
> >
> > 205 29
> >
> > 195 28
> >
> > 4554 28
> >
> > 206 28
> >
> > 213 28
> >
> > 4618 28
> >
> > 4695 28
> >
> > 4425 28
> >
> > 4619 28
> >
> > 4560 28
> >
> > 4428 28
> >
> > 4500 28
> >
> > 4448 28
> >
> > 4555 28
> >
> > 4603 28
> >
> > 4609 28
> >
> > 4584 28
> >
> > 4607 28
> >
> > 4502 28
> >
> > 4447 28
> >
> > 4604 28
> >
> > 4896 28
> >
> > 4602 28
> >
> > 4895 28
> >
> > 4753 28
> >
> > 204 28
> >
> > 4426 28
> >
> > 4634 28
> > Number of mutexes on VP free lists: 12879
> > Read/Write Locks
> > addr id name slocks xlock waiters
> > [informix@walter ~]$
> >
> > On Wed, Aug 20, 2014 at 10:37 AM, Fernando Nunes <domusonline@gmail.com>
> > wrote:
> >
> > > onstat -g wmx and onstat -g lmx to see what is the mutex...
> > > It can be hard to "see"...> > > Regards
> > >
> > > On Wed, Aug 20, 2014 at 3:34 PM, FRANK <yunyaoqu@gmail.com> wrote:
> > >
> > > > Folks,
> > > >
> > > > Normally I do not see S---- in output of onstat -u, but now I see.
> > > >
> > > > Any comments?
> > > >
> > > > Thanks
> > > > Frank
> > > >
> > > > RHEL 6
> > > >
> > > > [informix@walter ngdc_er_20140820]$ onstat -u
> > > > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up
> 00:38:50
> > > > -- 11847728 Kbytes> > > > Userthreads
> > > > address flags sessid user tty wait tout
> > > > locks nreads nwrites
> > > > 1d81b3028 ---P--D 1 informix - 0 0
> > > > 0 398 1015
> > > > 1d81b3880 ---P--F 0 informix - 0 0
> > > > 0 0 12778
> > > > 1d81b40d8 ---P--F 0 informix - 0 0
> > > > 0 0 18875
> > > > 1d81b4930 ---P--F 0 informix - 0 0
> > > > 0 0 11092
> > > > 1d81b5188 ---P--F 0 informix - 0 0
> > > > 0 0 18161
> > > > 1d81b59e0 ---P--F 0 informix - 0 0
> > > > 0 0 26058
> > > > 1d81b6238 ---P--F 0 informix - 0 0
> > > > 0 0 35724
> > > > 1d81b6a90 ---P--F 0 informix - 0 0
> > > > 0 0 12501
> > > > 1d81b72e8 ---P--F 0 informix - 0 0
> > > > 0 0 36521
> > > > 1d81b7b40 ---P--F 0 informix - 0 0
> > > > 0 0 15588
> > > > 1d81b8398 ---P--F 0 informix - 0 0
> > > > 0 0 46021
> > > > 1d81b8bf0 ---P--F 0 informix - 0 0
> > > > 0 0 2964
> > > > 1d81b9448 ---P--F 0 informix - 0 0
> > > > 0 0 183203
> > > > 1d81b9ca0 ---P--F 0 informix - 0 0
> > > > 0 0 47100
> > > > 1d81ba4f8 ---P--F 0 informix - 0 0
> > > > 0 0 18092
> > > > 1d81bad50 ---P--F 0 informix - 0 0
> > > > 0 0 47221
> > > > 1d81bb5a8 ---P--F 0 informix - 0 0
> > > > 0 0 13200
> > > > 1d81bbe00 ---P--- 18 informix - 0 0
> > > > 0 0 63
> > > > 1d81bc658 ---P--B 19 informix - 0 0
> > > > 0 515817 4096
> > > > 1d81bceb0 Y--P--B 20 informix - 1d8cf0e50 0
> > > > 0 459361 3977
> > > > 1d81bdf60 ---P--D 24 informix - 0 0
> > > > 0 4 0
> > > > 1d81be7b8 ---P--D 23 informix - 0 0
> > > > 0 0 0
> > > > 1d81bf010 ---P--- 43 informix - 0 0
> > > > 1 240 21860
> > > > 1d81bf868 Y--P--- 143 informix - 1db2aadc8 0
> > > > 1 8 22528
> > > > 1d81c00c0 ---P--- 33 informix - 0 0
> > > > 0 0 0
> > > > 1d81c0918 Y--P--- 141 informix - 1db2ccb78 0
> > > > 1 9 3
> > > > 1d81c1170 ---P--- 35 informix - 0 0
> > > > 1 0 0
> > > > 1d81c19c8 Y--P--- 36 informix - 1d9910448 0
> > > > 3 0 0
> > > > 1d81c2220 Y--P--D 41 informix - 48bdf970 0
> > > > 0 0 0
> > > > 1d81c2a78 ---P--- 44 informix - 0 0
> > > > 1 666 32901
> > > > 1d81c32d0 ---P--- 45 informix - 0 0
> > > > 1 711 32575
> > > > 1d81c3b28 ---P--- 46 informix - 0 0
> > > > 0 3 0
> > > > 1d81c4380 ---P--- 47 informix - 0 0
> > > > 1 240 442
> > > > 1d81c4bd8 ---P--- 48 informix - 0 0
> > > > 1 237 317
> > > > 1d81c5430 ---P--- 49 informix - 0 0
> > > > 1 197 329
> > > > 1d81c5c88 ---P--- 50 informix - 0 0
> > > > 1 165 552
> > > > 1d81c64e0 ---P--- 51 informix - 0 0
Original post:
Thanks, Fernando!
Here it is, confused ...
We indeed have ER slow/ data queued issue now
Frank
informix@walter ~]$ onstat -g wmx
IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:54:05
-- 11847728 KbytesMutexes with waiters:
mid addr name holder lkcnt waiter
waittime
6543 1d9fbc8f0 rqmut_trg_send 5496 0 227 0
5504 0
98 0
196 0
107 0
5488 0
7605 1dbb72ee0 CDRDSn0022 4484 0 210 17
4430 17
<stuff removed>
Number of mutexes on VP free lists: 10735
Read/Write Locks
addr id name slocks xlock waiters
[informix@walter ~]$ onstat -g lmx
IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:54:17
-- 11847728 KbytesLocked mutexes:
mid addr name holder lkcnt waiter
waittime
6307 1d9c113d0 qtrMutex31 5488 0
6543 1d9fbc8f0 rqmut_trg_send 5496 0 5495 0
5488 0
6547 1d9fbcb90 rqrest_trg_send 5496 0
7605 1dbb72ee0 CDRDSn0022 4484 0 210 29
4430 29
<stuff removed>
Response:
From the -g wmx/-g lmx along with the -u you posted makes it look like the
issue is with the CDRDSn0022 mutex. It has been held by thread 4484 for 29
seconds when this onstat data was captured. So is that thread waiting on
anything (I'd look at -g ath and -g stk <thread id>). I would say it's
probably not expected that a mutex to be held for ~ 30 seconds
1200%?! That's not a typo?
I have a customer that everytime I say something is faster he answers:
"They removed the 'sleeps'?!".
He may be right this time :)
On Wed, Aug 20, 2014 at 4:05 PM, Madison Pruet <mpruet@us.ibm.com> wrote:
> In 12.10xc4, we rewrote the way that the ER queues were locked. This
> significantly reduced lock contention and improved ER performance by close
> to 1200%
>
> Sent from my iPad
>
> > On Aug 20, 2014, at 8:49 AM, "FRANK" <yunyaoqu@gmail.com> wrote:
> >
> > Thanks, Fernando!
> >
> > Here it is, confused ...
> > We indeed have ER slow/ data queued issue now
> >
> > Frank
> >
> > informix@walter ~]$ onstat -g wmx
> > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:54:05
> > -- 11847728 Kbytes> > Mutexes with waiters:
> > mid addr name holder lkcnt waiter
> > waittime
> > 6543 1d9fbc8f0 rqmut_trg_send 5496 0 227 0
> >
> > 5504 0
> >
> > 98 0
> >
> > 196 0
> >
> > 107 0
> >
> > 5488 0
> > 7605 1dbb72ee0 CDRDSn0022 4484 0 210 17
> >
> > 4430 17
> >
> > 4585 17
> >
> > 4754 17
> >
> > 4528 17
> >
> > 4481 17
> >
> > 4897 17
> >
> > 4527 17
> >
> > 4526 17
> >
> > 4608 17
> >
> > 4752 17
> >
> > 214 17
> >
> > 205 17
> >
> > 195 16
> >
> > 4554 16
> >
> > 206 16
> >
> > 213 16
> >
> > 4618 16
> >
> > 4695 16
> >
> > 4425 16
> >
> > 4619 16
> >
> > 4560 16
> >
> > 4428 16
> >
> > 4500 16
> >
> > 4448 16
> >
> > 4555 16
> >
> > 4603 16
> >
> > 4609 16
> >
> > 4584 16
> >
> > 4607 16
> >
> > 4502 16
> >
> > 4447 16
> >
> > 4604 16
> >
> > 4896 16
> >
> > 4602 16
> >
> > 4895 16
> >
> > 4753 16
> >
> > 204 16
> >
> > 4426 16
> >
> > 4634 16
> > Number of mutexes on VP free lists: 10735
> > Read/Write Locks
> > addr id name slocks xlock waiters
> > [informix@walter ~]$ onstat -g lmx
> > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:54:17
> > -- 11847728 Kbytes> > Locked mutexes:
> > mid addr name holder lkcnt waiter
> > waittime
> > 6307 1d9c113d0 qtrMutex31 5488 0
> > 6543 1d9fbc8f0 rqmut_trg_send 5496 0 5495 0
> >
> > 5488 0
> > 6547 1d9fbcb90 rqrest_trg_send 5496 0
> > 7605 1dbb72ee0 CDRDSn0022 4484 0 210 29
> >
> > 4430 29
> >
> > 4585 29
> >
> > 4754 29
> >
> > 4528 29
> >
> > 4481 29
> >
> > 4897 29
> >
> > 4527 29
> >
> > 4526 29
> >
> > 4608 29
> >
> > 4752 29
> >
> > 214 29
> >
> > 205 29
> >
> > 195 28
> >
> > 4554 28
> >
> > 206 28
> >
> > 213 28
> >
> > 4618 28
> >
> > 4695 28
> >
> > 4425 28
> >
> > 4619 28
> >
> > 4560 28
> >
> > 4428 28
> >
> > 4500 28
> >
> > 4448 28
> >
> > 4555 28
> >
> > 4603 28
> >
> > 4609 28
> >
> > 4584 28
> >
> > 4607 28
> >
> > 4502 28
> >
> > 4447 28
> >
> > 4604 28
> >
> > 4896 28
> >
> > 4602 28
> >
> > 4895 28
> >
> > 4753 28
> >
> > 204 28
> >
> > 4426 28
> >
> > 4634 28
> > Number of mutexes on VP free lists: 12879
> > Read/Write Locks
> > addr id name slocks xlock waiters
> > [informix@walter ~]$
> >
> > On Wed, Aug 20, 2014 at 10:37 AM, Fernando Nunes <domusonline@gmail.com>
> > wrote:
> >
> > > onstat -g wmx and onstat -g lmx to see what is the mutex...
> > > It can be hard to "see"...> > > Regards
> > >
> > > On Wed, Aug 20, 2014 at 3:34 PM, FRANK <yunyaoqu@gmail.com> wrote:
> > >
> > > > Folks,
> > > >
> > > > Normally I do not see S---- in output of onstat -u, but now I see.
> > > >
> > > > Any comments?
> > > >
> > > > Thanks
> > > > Frank
> > > >
> > > > RHEL 6
> > > >
> > > > [informix@walter ngdc_er_20140820]$ onstat -u
> > > > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up
> 00:38:50
> > > > -- 11847728 Kbytes> > > > Userthreads
> > > > address flags sessid user tty wait tout
> > > > locks nreads nwrites
> > > > 1d81b3028 ---P--D 1 informix - 0 0
> > > > 0 398 1015
> > > > 1d81b3880 ---P--F 0 informix - 0 0
> > > > 0 0 12778
> > > > 1d81b40d8 ---P--F 0 informix - 0 0
> > > > 0 0 18875
> > > > 1d81b4930 ---P--F 0 informix - 0 0
> > > > 0 0 11092
> > > > 1d81b5188 ---P--F 0 informix - 0 0
> > > > 0 0 18161
> > > > 1d81b59e0 ---P--F 0 informix - 0 0
> > > > 0 0 26058
> > > > 1d81b6238 ---P--F 0 informix - 0 0
> > > > 0 0 35724
> > > > 1d81b6a90 ---P--F 0 informix - 0 0
> > > > 0 0 12501
> > > > 1d81b72e8 ---P--F 0 informix - 0 0
> > > > 0 0 36521
> > > > 1d81b7b40 ---P--F 0 informix - 0 0
> > > > 0 0 15588
> > > > 1d81b8398 ---P--F 0 informix - 0 0
> > > > 0 0 46021
> > > > 1d81b8bf0 ---P--F 0 informix - 0 0
> > > > 0 0 2964
> > > > 1d81b9448 ---P--F 0 informix - 0 0
> > > > 0 0 183203
> > > > 1d81b9ca0 ---P--F 0 informix - 0 0
> > > > 0 0 47100
> > > > 1d81ba4f8 ---P--F 0 informix - 0 0
> > > > 0 0 18092
> > > > 1d81bad50 ---P--F 0 informix - 0 0
> > > > 0 0 47221
> > > > 1d81bb5a8 ---P--F 0 informix - 0 0
> > > > 0 0 13200
> > > > 1d81bbe00 ---P--- 18 informix - 0 0
> > > > 0 0 63
> > > > 1d81bc658 ---P--B 19 informix - 0 0
> > > > 0 515817 4096
> > > > 1d81bceb0 Y--P--B 20 informix - 1d8cf0e50 0
> > > > 0 459361 3977
> > > > 1d81bdf60 ---P--D 24 informix - 0 0
> > > > 0 4 0
> > > > 1d81be7b8 ---P--D 23 informix - 0 0
> > > > 0 0 0
> > > > 1d81bf010 ---P--- 43 informix - 0 0
> > > > 1 240 21860
> > > > 1d81bf868 Y--P--- 143 informix - 1db2aadc8 0
> > > > 1 8 22528
> > > > 1d81c00c0 ---P--- 33 informix - 0 0
> > > > 0 0 0
> > > > 1d81c0918 Y--P--- 141 informix - 1db2ccb78 0
> > > > 1 9 3
> > > > 1d81c1170 ---P--- 35 informix - 0 0
> > > > 1 0 0
> > > > 1d81c19c8 Y--P--- 36 informix - 1d9910448 0
> > > > 3 0 0
> > > > 1d81c2220 Y--P--D 41 informix - 48bdf970 0
> > > > 0 0 0
> > > > 1d81c2a78 ---P--- 44 informix - 0 0
> > > > 1 666 32901
> > > > 1d81c32d0 ---P--- 45 informix - 0 0
> > > > 1 711 32575
> > > > 1d81c3b28 ---P--- 46 informix - 0 0
> > > > 0 3 0
> > > > 1d81c4380 ---P--- 47 informix - 0 0
> > > > 1 240 442
> > > > 1d81c4bd8 ---P--- 48 informix - 0 0
> > > > 1 237 317
> > > > 1d81c5430 ---P--- 49 informix - 0 0
> > > > 1 197 329
> > > > 1d81c5c88 ---P--- 50 informix - 0 0
> > > > 1 165 552
> > > > 1d81c64e0 ---P--- 51 informix - 0 0
> > > > 2 202 252
> > > > 1d81c6d38 ---P--- 52 informix - 0 0
> > > > 1 304 76
> > > > 1d81c7590 ---P--- 53 informix - 0 0
> > > > 1 211 859
> > > > 1d81c7de8 ---P--- 54 informix - 0 0
> > > > 2 223 298
> > > > 1d81c8640 ---P--- 55 informix - 0 0
> > > > 1 204 414
> > > > 1d81c8e98 ---P--- 56 informix - 0 0
> > > > 1 263 329
> > > > 1d81c96f0 ---P-
If they wasn't to eliminate the problem completely.
Sent from my iPad
> On Aug 20, 2014, at 9:24 AM, "Art Kagel" <art.kagel@gmail.com> wrote:
>
> So, Madison, you are suggesting that Frank upgrade?
>
> Art
>
> Art S. Kagel, Principal Consultant
> ASK Database Management
>
> Blog: http://informix-myview.blogspot.com/
>
> Disclaimer: Please keep in mind that my own opinions are my own opinions
> and do not reflect on the IIUG, nor any other organization with which I
am
> associated either explicitly, implicitly, or by inference. Neither do
> those opinions reflect those of other individuals affiliated with any
> entity with which I am affiliated nor those of the entities themselves.
>
> On Wed, Aug 20, 2014 at 11:05 AM, Madison Pruet <mpruet@us.ibm.com>
wrote:
>
> > In 12.10xc4, we rewrote the way that the ER queues were locked. This
> > significantly reduced lock contention and improved ER performance by
close
> > to 1200%
> >
> > Sent from my iPad
> >
> > > On Aug 20, 2014, at 8:49 AM, "FRANK" <yunyaoqu@gmail.com> wrote:
> > >
> > > Thanks, Fernando!
> > >
> > > Here it is, confused ...
> > > We indeed have ER slow/ data queued issue now
> > >
> > > Frank
> > >
> > > informix@walter ~]$ onstat -g wmx
> > > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up
00:54:05
> > > -- 11847728 Kbytes> > > Mutexes with waiters:
> > > mid addr name holder lkcnt waiter
> > > waittime
> > > 6543 1d9fbc8f0 rqmut_trg_send 5496 0 227 0
> > >
> > > 5504 0
> > >
> > > 98 0
> > >
> > > 196 0
> > >
> > > 107 0
> > >
> > > 5488 0
> > > 7605 1dbb72ee0 CDRDSn0022 4484 0 210 17
> > >
> > > 4430 17
> > >
> > > 4585 17
> > >
> > > 4754 17
> > >
> > > 4528 17
> > >
> > > 4481 17
> > >
> > > 4897 17
> > >
> > > 4527 17
> > >
> > > 4526 17
> > >
> > > 4608 17
> > >
> > > 4752 17
> > >
> > > 214 17
> > >
> > > 205 17
> > >
> > > 195 16
> > >
> > > 4554 16
> > >
> > > 206 16
> > >
> > > 213 16
> > >
> > > 4618 16
> > >
> > > 4695 16
> > >
> > > 4425 16
> > >
> > > 4619 16
> > >
> > > 4560 16
> > >
> > > 4428 16
> > >
> > > 4500 16
> > >
> > > 4448 16
> > >
> > > 4555 16
> > >
> > > 4603 16
> > >
> > > 4609 16
> > >
> > > 4584 16
> > >
> > > 4607 16
> > >
> > > 4502 16
> > >
> > > 4447 16
> > >
> > > 4604 16
> > >
> > > 4896 16
> > >
> > > 4602 16
> > >
> > > 4895 16
> > >
> > > 4753 16
> > >
> > > 204 16
> > >
> > > 4426 16
> > >
> > > 4634 16
> > > Number of mutexes on VP free lists: 10735
> > > Read/Write Locks
> > > addr id name slocks xlock waiters
> > > [informix@walter ~]$ onstat -g lmx
> > > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up
00:54:17
> > > -- 11847728 Kbytes> > > Locked mutexes:
> > > mid addr name holder lkcnt waiter
> > > waittime
> > > 6307 1d9c113d0 qtrMutex31 5488 0
> > > 6543 1d9fbc8f0 rqmut_trg_send 5496 0 5495 0
> > >
> > > 5488 0
> > > 6547 1d9fbcb90 rqrest_trg_send 5496 0
> > > 7605 1dbb72ee0 CDRDSn0022 4484 0 210 29
> > >
> > > 4430 29
> > >
> > > 4585 29
> > >
> > > 4754 29
> > >
> > > 4528 29
> > >
> > > 4481 29
> > >
> > > 4897 29
> > >
> > > 4527 29
> > >
> > > 4526 29
> > >
> > > 4608 29
> > >
> > > 4752 29
> > >
> > > 214 29
> > >
> > > 205 29
> > >
> > > 195 28
> > >
> > > 4554 28
> > >
> > > 206 28
> > >
> > > 213 28
> > >
> > > 4618 28
> > >
> > > 4695 28
> > >
> > > 4425 28
> > >
> > > 4619 28
> > >
> > > 4560 28
> > >
> > > 4428 28
> > >
> > > 4500 28
> > >
> > > 4448 28
> > >
> > > 4555 28
> > >
> > > 4603 28
> > >
> > > 4609 28
> > >
> > > 4584 28
> > >
> > > 4607 28
> > >
> > > 4502 28
> > >
> > > 4447 28
> > >
> > > 4604 28
> > >
> > > 4896 28
> > >
> > > 4602 28
> > >
> > > 4895 28
> > >
> > > 4753 28
> > >
> > > 204 28
> > >
> > > 4426 28
> > >
> > > 4634 28
> > > Number of mutexes on VP free lists: 12879
> > > Read/Write Locks
> > > addr id name slocks xlock waiters
> > > [informix@walter ~]$
> > >
> > > On Wed, Aug 20, 2014 at 10:37 AM, Fernando Nunes
<domusonline@gmail.com>
> > > wrote:
> > >
> > > > onstat -g wmx and onstat -g lmx to see what is the mutex...
> > > > It can be hard to "see"...> > > > Regards
> > > >
> > > > On Wed, Aug 20, 2014 at 3:34 PM, FRANK <yunyaoqu@gmail.com> wrote:
> > > >
> > > > > Folks,
> > > > >
> > > > > Normally I do not see S---- in output of onstat -u, but now I
see.
> > > > >
> > > > > Any comments?
> > > > >
> > > > > Thanks
> > > > > Frank
> > > > >
> > > > > RHEL 6
> > > > >
> > > > > [informix@walter ngdc_er_20140820]$ onstat -u
> > > > > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up
> > 00:38:50
> > > > > -- 11847728 Kbytes> > > > > Userthreads
> > > > > address flags sessid user tty wait tout
> > > > > locks nreads nwrites
> > > > > 1d81b3028 ---P--D 1 informix - 0 0
> > > > > 0 398 1015
> > > > > 1d81b3880 ---P--F 0 informix - 0 0
> > > > > 0 0 12778
> > > > > 1d81b40d8 ---P--F 0 informix - 0 0
> > > > > 0 0 18875
> > > > > 1d81b4930 ---P--F 0 informix - 0 0
> > > > > 0 0 11092
> > > > > 1d81b5188 ---P--F 0 informix - 0 0
> > > > > 0 0 18161
> > > > > 1d81b59e0 ---P--F 0 informix - 0 0
> > > > > 0 0 26058
> > > > > 1d81b6238 ---P--F 0 informix - 0 0
> > > > > 0 0 35724
> > > > > 1d81b6a90 ---P--F 0 informix - 0 0
> > > > > 0 0 12501
> > > > > 1d81b72e8 ---P--F 0 informix - 0 0
> > > > > 0 0 36521
> > > > > 1d81b7b40 ---P--F 0 informix - 0 0
> > > > > 0 0 15588
> > > > > 1d81b8398 ---P--F 0 informix - 0 0
> > > > > 0 0 46021
> > > > > 1d81b8bf0 ---P--F 0 informix - 0 0
> > > > > 0 0 2964
> > > > > 1d81b9448 ---P--F 0 informix - 0 0
> > > > > 0 0 183203
> > > > > 1d81b9ca0 ---P--F 0 informix - 0 0
> > > > > 0 0 47100
> > > > > 1d81ba4f8 ---P--F 0 informix - 0 0
> > > > > 0 0 18092
> > > > > 1d81bad50 ---P--F 0 informix - 0 0
> > > > > 0 0 47221
> > > > > 1d81bb5a8 ---P--F 0 informix - 0 0
> > > > > 0 0 13200
> > > > > 1d81bbe00 ---P--- 18 informix - 0 0
> > > > > 0 0 63
> > > > > 1d81bc658 ---P--B 19 informix - 0 0
> > > > > 0 515817 4096
> > > > > 1d81bceb0 Y--P--B 20 informix - 1d8cf0e50 0
> > > > > 0 459361 3977
> > > > > 1d81bdf60 ---P--D 24 informix - 0 0
> > > > > 0 4 0
> > > > > 1d81be7b8 ---P--D 23 informix - 0 0
> > > > > 0 0 0
> > > > > 1d81bf010 ---P--- 43 informix - 0 0
> > > > > 1 240 21860
> > > > > 1d81bf868 Y--P--- 143 informix - 1db2aadc8 0
> > > > > 1 8 22528
> > > > > 1d81c00c0 ---P--- 33 informix - 0 0
> > > > > 0 0 0
> > > > > 1d81c0918 Y--P---
Check onstat -g spi as well
Sent from my iPad
> On Aug 20, 2014, at 9:41 AM, "Jacques Renaut" <jrenaut@us.ibm.com> wrote:
>
> Original post:
>
> Thanks, Fernando!
>
> Here it is, confused ...
> We indeed have ER slow/ data queued issue now
>
> Frank
>
> informix@walter ~]$ onstat -g wmx
> IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:54:05
> -- 11847728 Kbytes> Mutexes with waiters:
> mid addr name holder lkcnt waiter
> waittime
> 6543 1d9fbc8f0 rqmut_trg_send 5496 0 227 0
>
> 5504 0
>
> 98 0
>
> 196 0
>
> 107 0
>
> 5488 0
> 7605 1dbb72ee0 CDRDSn0022 4484 0 210 17
>
> 4430 17
>
> <stuff removed>
>
> Number of mutexes on VP free lists: 10735
> Read/Write Locks
> addr id name slocks xlock waiters
> [informix@walter ~]$ onstat -g lmx
> IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:54:17
> -- 11847728 Kbytes> Locked mutexes:
> mid addr name holder lkcnt waiter
> waittime
> 6307 1d9c113d0 qtrMutex31 5488 0
> 6543 1d9fbc8f0 rqmut_trg_send 5496 0 5495 0
>
> 5488 0
> 6547 1d9fbcb90 rqrest_trg_send 5496 0
> 7605 1dbb72ee0 CDRDSn0022 4484 0 210 29
>
> 4430 29
>
> <stuff removed>
>
> Response:
>
> >From the -g wmx/-g lmx along with the -u you posted makes it look like
the
> issue is with the CDRDSn0022 mutex. It has been held by thread 4484 for
29
> seconds when this onstat data was captured. So is that thread waiting on
> anything (I'd look at -g ath and -g stk <thread id>). I would say it's
> probably not expected that a mutex to be held for ~ 30 seconds
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
Not a typo.
I changed the logic so that the consumers of the queues never have to r=
aise
a mutex lock, not even a read-write lock. The thread consuming the que=
ue
element no longer actually destroys the object, but posts it in a
to-be-destroyed list. The only threads that now lock the queue is one
adding something to the queue. While it has the queue locked, it scans=
the
to-be-destroyed list and frees the memory.
Also, I implemented a memory allocation scheme using a first find
allocation plan so as to reduce the memory pool spin contention.
Sent from my iPad
> On Aug 20, 2014, at 9:56 AM, "Fernando Nunes" <domusonline@gmail.com>=
wrote:
>
> 1200%?! That's not a typo?
> I have a customer that everytime I say something is faster he answers=
:
> "They removed the 'sleeps'?!".
> He may be right this time :)
>
> On Wed, Aug 20, 2014 at 4:05 PM, Madison Pruet <mpruet@us.ibm.com> wr=
ote:
>
> > In 12.10xc4, we rewrote the way that the ER queues were locked. Thi=
s
> > significantly reduced lock contention and improved ER performance b=
y
close
> > to 1200%
> >
> > Sent from my iPad
> >
> > > On Aug 20, 2014, at 8:49 AM, "FRANK" <yunyaoqu@gmail.com> wrote:
> > >
> > > Thanks, Fernando!
> > >
> > > Here it is, confused ...
> > > We indeed have ER slow/ data queued issue now
> > >
> > > Frank
> > >
> > > informix@walter ~]$ onstat -g wmx
> > > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up
00:54:05
> > > -- 11847728 Kbytes> > > Mutexes with waiters:
> > > mid addr name holder lkcnt waiter
> > > waittime
> > > 6543 1d9fbc8f0 rqmut_trg_send 5496 0 227 0
> > >
> > > 5504 0
> > >
> > > 98 0
> > >
> > > 196 0
> > >
> > > 107 0
> > >
> > > 5488 0
> > > 7605 1dbb72ee0 CDRDSn0022 4484 0 210 17
> > >
> > > 4430 17
> > >
> > > 4585 17
> > >
> > > 4754 17
> > >
> > > 4528 17
> > >
> > > 4481 17
> > >
> > > 4897 17
> > >
> > > 4527 17
> > >
> > > 4526 17
> > >
> > > 4608 17
> > >
> > > 4752 17
> > >
> > > 214 17
> > >
> > > 205 17
> > >
> > > 195 16
> > >
> > > 4554 16
> > >
> > > 206 16
> > >
> > > 213 16
> > >
> > > 4618 16
> > >
> > > 4695 16
> > >
> > > 4425 16
> > >
> > > 4619 16
> > >
> > > 4560 16
> > >
> > > 4428 16
> > >
> > > 4500 16
> > >
> > > 4448 16
> > >
> > > 4555 16
> > >
> > > 4603 16
> > >
> > > 4609 16
> > >
> > > 4584 16
> > >
> > > 4607 16
> > >
> > > 4502 16
> > >
> > > 4447 16
> > >
> > > 4604 16
> > >
> > > 4896 16
> > >
> > > 4602 16
> > >
> > > 4895 16
> > >
> > > 4753 16
> > >
> > > 204 16
> > >
> > > 4426 16
> > >
> > > 4634 16
> > > Number of mutexes on VP free lists: 10735
> > > Read/Write Locks
> > > addr id name slocks xlock waiters
> > > [informix@walter ~]$ onstat -g lmx
> > > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up
00:54:17
> > > -- 11847728 Kbytes> > > Locked mutexes:
> > > mid addr name holder lkcnt waiter
> > > waittime
> > > 6307 1d9c113d0 qtrMutex31 5488 0
> > > 6543 1d9fbc8f0 rqmut_trg_send 5496 0 5495 0
> > >
> > > 5488 0
> > > 6547 1d9fbcb90 rqrest_trg_send 5496 0
> > > 7605 1dbb72ee0 CDRDSn0022 4484 0 210 29
> > >
> > > 4430 29
> > >
> > > 4585 29
> > >
> > > 4754 29
> > >
> > > 4528 29
> > >
> > > 4481 29
> > >
> > > 4897 29
> > >
> > > 4527 29
> > >
> > > 4526 29
> > >
> > > 4608 29
> > >
> > > 4752 29
> > >
> > > 214 29
> > >
> > > 205 29
> > >
> > > 195 28
> > >
> > > 4554 28
> > >
> > > 206 28
> > >
> > > 213 28
> > >
> > > 4618 28
> > >
> > > 4695 28
> > >
> > > 4425 28
> > >
> > > 4619 28
> > >
> > > 4560 28
> > >
> > > 4428 28
> > >
> > > 4500 28
> > >
> > > 4448 28
> > >
> > > 4555 28
> > >
> > > 4603 28
> > >
> > > 4609 28
> > >
> > > 4584 28
> > >
> > > 4607 28
> > >
> > > 4502 28
> > >
> > > 4447 28
> > >
> > > 4604 28
> > >
> > > 4896 28
> > >
> > > 4602 28
> > >
> > > 4895 28
> > >
> > > 4753 28
> > >
> > > 204 28
> > >
> > > 4426 28
> > >
> > > 4634 28
> > > Number of mutexes on VP free lists: 12879
> > > Read/Write Locks
> > > addr id name slocks xlock waiters
> > > [informix@walter ~]$
> > >
> > > On Wed, Aug 20, 2014 at 10:37 AM, Fernando Nunes
<domusonline@gmail.com>
> > > wrote:
> > >
> > > > onstat -g wmx and onstat -g lmx to see what is the mutex...
> > > > It can be hard to "see"...> > > > Regards
> > > >
> > > > On Wed, Aug 20, 2014 at 3:34 PM, FRANK <yunyaoqu@gmail.com> wro=
te:
> > > >
> > > > > Folks,
> > > > >
> > > > > Normally I do not see S---- in output of onstat -u, but now I=
see.
> > > > >
> > > > > Any comments?
> > > > >
> > > > > Thanks
> > > > > Frank
> > > > >
> > > > > RHEL 6
> > > > >
> > > > > [informix@walter ngdc_er_20140820]$ onstat -u
> > > > > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- U=p
> > 00:38:50
> > > > > -- 11847728 Kbytes
> > > > > Userthreads
> > > > > address flags sessid user tty wait tout
> > > > > locks nreads nwrites
> > > > > 1d81b3028 ---P--D 1 informix - 0 0
> > > > > 0 398 1015
> > > > > 1d81b3880 ---P--F 0 informix - 0 0
> > > > > 0 0 12778
> > > > > 1d81b40d8 ---P--F 0 informix - 0 0
> > > > > 0 0 18875
> > > > > 1d81b4930 ---P--F 0 informix - 0 0
> > > > > 0 0 11092
> > > > > 1d81b5188 ---P--F 0 informix - 0 0
> > > > > 0 0 18161
> > > > > 1d81b59e0 ---P--F 0 informix - 0 0
> > > > > 0 0 26058
> > > > > 1d81b6238 ---P--F 0 informix - 0 0
> > > > > 0 0 35724
> > > > > 1d81b6a90 ---P--F 0 informix - 0 0
> > > > > 0 0 12501
> > > > > 1d81b72e8 ---P--F 0 informix - 0 0
> > > > > 0 0 36521
> > > > > 1d81b7b40 ---P--F 0 informix - 0 0
> > > > > 0 0 15588
> > > > > 1d81b8398 ---P--F 0 informix - 0 0
> > > > > 0 0 46021
> > > > > 1d81b8bf0 ---P--F 0 informix - 0 0
> > > > > 0 0 2964
> > > > > 1d81b9448 ---P--F 0 informix - 0 0
> > > > > 0 0 183203
> > > > > 1d81b9ca0 ---P--F 0 informix - 0 0
> > > > > 0 0 47100
> > > > > 1d81ba4f8 ---P--F 0 informix - 0 0
> > > > > 0 0 18092
> > > > > 1d81bad50 ---P--F 0 informix - 0 0
> > > > > 0 0 47221
> > > > > 1d81bb5a8 ---P--F 0 informix - 0 0
> > > > > 0 0 13200
> > > > > 1d81bbe00 ---P--- 18 informix - 0 0
> > > > > 0 0 63
> > > > > 1d81bc658 ---P--B 19 informix - 0 0
> > > > > 0 515817 4096
> > > > > 1d81bceb0 Y--P--B 20 informix - 1d8cf0e50 0
> > > > > 0 459361 3977
> > > > > 1d81bdf60 ---P--D 24 informix - 0 0
> > > > > 0 4 0
> > > > > 1d81be7b8 ---P--D 23 informix - 0 0
> > > > > 0 0 0
> > > > > 1d81bf010 ---P--- 43 informix - 0 0
> > > > > 1 240
Is this a consolidation model? How many nodes?
Sent from my iPad
> On Aug 20, 2014, at 8:49 AM, "FRANK" <yunyaoqu@gmail.com> wrote:
>
> Thanks, Fernando!
>
> Here it is, confused ...
> We indeed have ER slow/ data queued issue now
>
> Frank
>
> informix@walter ~]$ onstat -g wmx
> IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:54:05
> -- 11847728 Kbytes> Mutexes with waiters:
> mid addr name holder lkcnt waiter
> waittime
> 6543 1d9fbc8f0 rqmut_trg_send 5496 0 227 0
>
> 5504 0
>
> 98 0
>
> 196 0
>
> 107 0
>
> 5488 0
> 7605 1dbb72ee0 CDRDSn0022 4484 0 210 17
>
> 4430 17
>
> 4585 17
>
> 4754 17
>
> 4528 17
>
> 4481 17
>
> 4897 17
>
> 4527 17
>
> 4526 17
>
> 4608 17
>
> 4752 17
>
> 214 17
>
> 205 17
>
> 195 16
>
> 4554 16
>
> 206 16
>
> 213 16
>
> 4618 16
>
> 4695 16
>
> 4425 16
>
> 4619 16
>
> 4560 16
>
> 4428 16
>
> 4500 16
>
> 4448 16
>
> 4555 16
>
> 4603 16
>
> 4609 16
>
> 4584 16
>
> 4607 16
>
> 4502 16
>
> 4447 16
>
> 4604 16
>
> 4896 16
>
> 4602 16
>
> 4895 16
>
> 4753 16
>
> 204 16
>
> 4426 16
>
> 4634 16
> Number of mutexes on VP free lists: 10735
> Read/Write Locks
> addr id name slocks xlock waiters
> [informix@walter ~]$ onstat -g lmx
> IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:54:17
> -- 11847728 Kbytes> Locked mutexes:
> mid addr name holder lkcnt waiter
> waittime
> 6307 1d9c113d0 qtrMutex31 5488 0
> 6543 1d9fbc8f0 rqmut_trg_send 5496 0 5495 0
>
> 5488 0
> 6547 1d9fbcb90 rqrest_trg_send 5496 0
> 7605 1dbb72ee0 CDRDSn0022 4484 0 210 29
>
> 4430 29
>
> 4585 29
>
> 4754 29
>
> 4528 29
>
> 4481 29
>
> 4897 29
>
> 4527 29
>
> 4526 29
>
> 4608 29
>
> 4752 29
>
> 214 29
>
> 205 29
>
> 195 28
>
> 4554 28
>
> 206 28
>
> 213 28
>
> 4618 28
>
> 4695 28
>
> 4425 28
>
> 4619 28
>
> 4560 28
>
> 4428 28
>
> 4500 28
>
> 4448 28
>
> 4555 28
>
> 4603 28
>
> 4609 28
>
> 4584 28
>
> 4607 28
>
> 4502 28
>
> 4447 28
>
> 4604 28
>
> 4896 28
>
> 4602 28
>
> 4895 28
>
> 4753 28
>
> 204 28
>
> 4426 28
>
> 4634 28
> Number of mutexes on VP free lists: 12879
> Read/Write Locks
> addr id name slocks xlock waiters
> [informix@walter ~]$
>
> On Wed, Aug 20, 2014 at 10:37 AM, Fernando Nunes <domusonline@gmail.com>
> wrote:
>
> > onstat -g wmx and onstat -g lmx to see what is the mutex...
> > It can be hard to "see"...> > Regards
> >
> > On Wed, Aug 20, 2014 at 3:34 PM, FRANK <yunyaoqu@gmail.com> wrote:
> >
> > > Folks,
> > >
> > > Normally I do not see S---- in output of onstat -u, but now I see.
> > >
> > > Any comments?
> > >
> > > Thanks
> > > Frank
> > >
> > > RHEL 6
> > >
> > > [informix@walter ngdc_er_20140820]$ onstat -u
> > > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up
00:38:50
> > > -- 11847728 Kbytes> > > Userthreads
> > > address flags sessid user tty wait tout
> > > locks nreads nwrites
> > > 1d81b3028 ---P--D 1 informix - 0 0
> > > 0 398 1015
> > > 1d81b3880 ---P--F 0 informix - 0 0
> > > 0 0 12778
> > > 1d81b40d8 ---P--F 0 informix - 0 0
> > > 0 0 18875
> > > 1d81b4930 ---P--F 0 informix - 0 0
> > > 0 0 11092
> > > 1d81b5188 ---P--F 0 informix - 0 0
> > > 0 0 18161
> > > 1d81b59e0 ---P--F 0 informix - 0 0
> > > 0 0 26058
> > > 1d81b6238 ---P--F 0 informix - 0 0
> > > 0 0 35724
> > > 1d81b6a90 ---P--F 0 informix - 0 0
> > > 0 0 12501
> > > 1d81b72e8 ---P--F 0 informix - 0 0
> > > 0 0 36521
> > > 1d81b7b40 ---P--F 0 informix - 0 0
> > > 0 0 15588
> > > 1d81b8398 ---P--F 0 informix - 0 0
> > > 0 0 46021
> > > 1d81b8bf0 ---P--F 0 informix - 0 0
> > > 0 0 2964
> > > 1d81b9448 ---P--F 0 informix - 0 0
> > > 0 0 183203
> > > 1d81b9ca0 ---P--F 0 informix - 0 0
> > > 0 0 47100
> > > 1d81ba4f8 ---P--F 0 informix - 0 0
> > > 0 0 18092
> > > 1d81bad50 ---P--F 0 informix - 0 0
> > > 0 0 47221
> > > 1d81bb5a8 ---P--F 0 informix - 0 0
> > > 0 0 13200
> > > 1d81bbe00 ---P--- 18 informix - 0 0
> > > 0 0 63
> > > 1d81bc658 ---P--B 19 informix - 0 0
> > > 0 515817 4096
> > > 1d81bceb0 Y--P--B 20 informix - 1d8cf0e50 0
> > > 0 459361 3977
> > > 1d81bdf60 ---P--D 24 informix - 0 0
> > > 0 4 0
> > > 1d81be7b8 ---P--D 23 informix - 0 0
> > > 0 0 0
> > > 1d81bf010 ---P--- 43 informix - 0 0
> > > 1 240 21860
> > > 1d81bf868 Y--P--- 143 informix - 1db2aadc8 0
> > > 1 8 22528
> > > 1d81c00c0 ---P--- 33 informix - 0 0
> > > 0 0 0
> > > 1d81c0918 Y--P--- 141 informix - 1db2ccb78 0
> > > 1 9 3
> > > 1d81c1170 ---P--- 35 informix - 0 0
> > > 1 0 0
> > > 1d81c19c8 Y--P--- 36 informix - 1d9910448 0
> > > 3 0 0
> > > 1d81c2220 Y--P--D 41 informix - 48bdf970 0
> > > 0 0 0
> > > 1d81c2a78 ---P--- 44 informix - 0 0
> > > 1 666 32901
> > > 1d81c32d0 ---P--- 45 informix - 0 0
> > > 1 711 32575
> > > 1d81c3b28 ---P--- 46 informix - 0 0
> > > 0 3 0
> > > 1d81c4380 ---P--- 47 informix - 0 0
> > > 1 240 442
> > > 1d81c4bd8 ---P--- 48 informix - 0 0
> > > 1 237 317
> > > 1d81c5430 ---P--- 49 informix - 0 0
> > > 1 197 329
> > > 1d81c5c88 ---P--- 50 informix - 0 0
> > > 1 165 552
> > > 1d81c64e0 ---P--- 51 informix - 0 0
> > > 2 202 252
> > > 1d81c6d38 ---P--- 52 informix - 0 0
> > > 1 304 76
> > > 1d81c7590 ---P--- 53 informix - 0 0
> > > 1 211 859
> > > 1d81c7de8 ---P--- 54 informix - 0 0
> > > 2 223 298
> > > 1d81c8640 ---P--- 55 informix - 0 0
> > > 1 204 414
> > > 1d81c8e98 ---P--- 56 informix - 0 0
> > > 1 263 329
> > > 1d81c96f0 ---P--- 57 informix - 0 0
> > > 2 237 364
> > > 1d81c9f48 ---P--- 58 informix - 0 0
> > > 2 302 174
> > > 1d81ca7a0 ---P--- 136 informix - 0 0
> > > 1 4 0
> > > 1d81caff8 S--P--- 137 informix - 1d9fbc8f0 0
> > > 1 186322 0
> > > 1d81cb850 ---P--- 138 informix - 0 0
> > > 0 272298 0
> > > 1d81cc0a8 ---P--- 139 informix - 0 0
> > > 1 44 0
> > > 1d81cc900 S--P--- 142 informix - 1dbb72ee0 0
> > > 0 0 211
> > > 1d81cd158 Y--P--- 144 informix - 1db2aadc8 0
> > > 1 9 23680
> > > 1d81cd9b0 S--P--- 4692 informix - 1dbb72ee0 0
> > > 2 3 4
> > > 1d81ce208 S--P--- 4531 informix - 1dbb72ee0 0
> > > 2 0 47
> > > 1d81cea60 Y--P--- 235 saapmgr - 1def80778 0
> > > 1 125 0
> > > 1d81cf2b8 S--P--- 4693 informix - 1dbb72ee0 0
> > > 2 3 1
> > > 1d81d0bc0 S--P--- 151 informix - 1dbb72ee0 0
> > > 2 13166 33392
> > > 1d81d1418 S--P--- 152 informix - 1dbb72ee0 0
> > > 2 14148 32374
> > > 1d81d1c70 S--P--- 153 in
It is a three nodes update-any-where setup.
We can have a lot of busy updates in one site, which would be transferred
to others sites by ER. Actually, most times we only want to transfer " new
row just inserted" to other sites, we really do not care lot about
updates. So, it should help nus if we can easily define some replicates as
" replicate insert only".
Thanks
Frank
On Wed, Aug 20, 2014 at 12:27 PM, Madison Pruet <mpruet@us.ibm.com> wrote:
> Is this a consolidation model? How many nodes?
>
> Sent from my iPad
>
> > On Aug 20, 2014, at 8:49 AM, "FRANK" <yunyaoqu@gmail.com> wrote:
> >
> > Thanks, Fernando!
> >
> > Here it is, confused ...
> > We indeed have ER slow/ data queued issue now
> >
> > Frank
> >
> > informix@walter ~]$ onstat -g wmx
> > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:54:05
> > -- 11847728 Kbytes> > Mutexes with waiters:
> > mid addr name holder lkcnt waiter
> > waittime
> > 6543 1d9fbc8f0 rqmut_trg_send 5496 0 227 0
> >
> > 5504 0
> >
> > 98 0
> >
> > 196 0
> >
> > 107 0
> >
> > 5488 0
> > 7605 1dbb72ee0 CDRDSn0022 4484 0 210 17
> >
> > 4430 17
> >
> > 4585 17
> >
> > 4754 17
> >
> > 4528 17
> >
> > 4481 17
> >
> > 4897 17
> >
> > 4527 17
> >
> > 4526 17
> >
> > 4608 17
> >
> > 4752 17
> >
> > 214 17
> >
> > 205 17
> >
> > 195 16
> >
> > 4554 16
> >
> > 206 16
> >
> > 213 16
> >
> > 4618 16
> >
> > 4695 16
> >
> > 4425 16
> >
> > 4619 16
> >
> > 4560 16
> >
> > 4428 16
> >
> > 4500 16
> >
> > 4448 16
> >
> > 4555 16
> >
> > 4603 16
> >
> > 4609 16
> >
> > 4584 16
> >
> > 4607 16
> >
> > 4502 16
> >
> > 4447 16
> >
> > 4604 16
> >
> > 4896 16
> >
> > 4602 16
> >
> > 4895 16
> >
> > 4753 16
> >
> > 204 16
> >
> > 4426 16
> >
> > 4634 16
> > Number of mutexes on VP free lists: 10735
> > Read/Write Locks
> > addr id name slocks xlock waiters
> > [informix@walter ~]$ onstat -g lmx
> > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 00:54:17
> > -- 11847728 Kbytes> > Locked mutexes:
> > mid addr name holder lkcnt waiter
> > waittime
> > 6307 1d9c113d0 qtrMutex31 5488 0
> > 6543 1d9fbc8f0 rqmut_trg_send 5496 0 5495 0
> >
> > 5488 0
> > 6547 1d9fbcb90 rqrest_trg_send 5496 0
> > 7605 1dbb72ee0 CDRDSn0022 4484 0 210 29
> >
> > 4430 29
> >
> > 4585 29
> >
> > 4754 29
> >
> > 4528 29
> >
> > 4481 29
> >
> > 4897 29
> >
> > 4527 29
> >
> > 4526 29
> >
> > 4608 29
> >
> > 4752 29
> >
> > 214 29
> >
> > 205 29
> >
> > 195 28
> >
> > 4554 28
> >
> > 206 28
> >
> > 213 28
> >
> > 4618 28
> >
> > 4695 28
> >
> > 4425 28
> >
> > 4619 28
> >
> > 4560 28
> >
> > 4428 28
> >
> > 4500 28
> >
> > 4448 28
> >
> > 4555 28
> >
> > 4603 28
> >
> > 4609 28
> >
> > 4584 28
> >
> > 4607 28
> >
> > 4502 28
> >
> > 4447 28
> >
> > 4604 28
> >
> > 4896 28
> >
> > 4602 28
> >
> > 4895 28
> >
> > 4753 28
> >
> > 204 28
> >
> > 4426 28
> >
> > 4634 28
> > Number of mutexes on VP free lists: 12879
> > Read/Write Locks
> > addr id name slocks xlock waiters
> > [informix@walter ~]$
> >
> > On Wed, Aug 20, 2014 at 10:37 AM, Fernando Nunes <domusonline@gmail.com>
> > wrote:
> >
> > > onstat -g wmx and onstat -g lmx to see what is the mutex...
> > > It can be hard to "see"...> > > Regards
> > >
> > > On Wed, Aug 20, 2014 at 3:34 PM, FRANK <yunyaoqu@gmail.com> wrote:
> > >
> > > > Folks,
> > > >
> > > > Normally I do not see S---- in output of onstat -u, but now I see.
> > > >
> > > > Any comments?
> > > >
> > > > Thanks
> > > > Frank
> > > >
> > > > RHEL 6
> > > >
> > > > [informix@walter ngdc_er_20140820]$ onstat -u
> > > > IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up
> 00:38:50
> > > > -- 11847728 Kbytes> > > > Userthreads
> > > > address flags sessid user tty wait tout
> > > > locks nreads nwrites
> > > > 1d81b3028 ---P--D 1 informix - 0 0
> > > > 0 398 1015
> > > > 1d81b3880 ---P--F 0 informix - 0 0
> > > > 0 0 12778
> > > > 1d81b40d8 ---P--F 0 informix - 0 0
> > > > 0 0 18875
> > > > 1d81b4930 ---P--F 0 informix - 0 0
> > > > 0 0 11092
> > > > 1d81b5188 ---P--F 0 informix - 0 0
> > > > 0 0 18161
> > > > 1d81b59e0 ---P--F 0 informix - 0 0
> > > > 0 0 26058
> > > > 1d81b6238 ---P--F 0 informix - 0 0
> > > > 0 0 35724
> > > > 1d81b6a90 ---P--F 0 informix - 0 0
> > > > 0 0 12501
> > > > 1d81b72e8 ---P--F 0 informix - 0 0
> > > > 0 0 36521
> > > > 1d81b7b40 ---P--F 0 informix - 0 0
> > > > 0 0 15588
> > > > 1d81b8398 ---P--F 0 informix - 0 0
> > > > 0 0 46021
> > > > 1d81b8bf0 ---P--F 0 informix - 0 0
> > > > 0 0 2964
> > > > 1d81b9448 ---P--F 0 informix - 0 0
> > > > 0 0 183203
> > > > 1d81b9ca0 ---P--F 0 informix - 0 0
> > > > 0 0 47100
> > > > 1d81ba4f8 ---P--F 0 informix - 0 0
> > > > 0 0 18092
> > > > 1d81bad50 ---P--F 0 informix - 0 0
> > > > 0 0 47221
> > > > 1d81bb5a8 ---P--F 0 informix - 0 0
> > > > 0 0 13200
> > > > 1d81bbe00 ---P--- 18 informix - 0 0
> > > > 0 0 63
> > > > 1d81bc658 ---P--B 19 informix - 0 0
> > > > 0 515817 4096
> > > > 1d81bceb0 Y--P--B 20 informix - 1d8cf0e50 0
> > > > 0 459361 3977
> > > > 1d81bdf60 ---P--D 24 informix - 0 0
> > > > 0 4 0
> > > > 1d81be7b8 ---P--D 23 informix - 0 0
> > > > 0 0 0
> > > > 1d81bf010 ---P--- 43 informix - 0 0
> > > > 1 240 21860
> > > > 1d81bf868 Y--P--- 143 informix - 1db2aadc8 0
> > > > 1 8 22528
> > > > 1d81c00c0 ---P--- 33 informix - 0 0
> > > > 0 0 0
> > > > 1d81c0918 Y--P--- 141 informix - 1db2ccb78 0
> > > > 1 9 3
> > > > 1d81c1170 ---P--- 35 informix - 0 0
> > > > 1 0 0
> > > > 1d81c19c8 Y--P--- 36 informix - 1d9910448 0
> > > > 3 0 0
> > > > 1d81c2220 Y--P--D 41 informix - 48bdf970 0
> > > > 0 0 0
> > > > 1d81c2a78 ---P--- 44 informix - 0 0
> > > > 1 666 32901
> > > > 1d81c32d0 ---P--- 45 informix - 0 0
> > > > 1 711 32575
> > > > 1d81c3b28 ---P--- 46 informix - 0 0
> > > > 0 3 0
> > > > 1d81c4380 ---P--- 47 informix - 0 0
> > > > 1 240 442
> > > > 1d81c4bd8 ---P--- 48 informix - 0 0
> > > > 1 237 317
> > > > 1d81c5430 ---P--- 49 informix - 0 0
> > > > 1 197 329
> > > > 1d81c5c88 ---P--- 50 informix - 0 0
> > > > 1 165 552
> > > > 1d81c64e0 ---P--- 51 informix - 0 0
> > > > 2 202 252
> > > > 1d81c6d38 ---P--- 52 informix - 0 0
> > > > 1 304 76
> > > > 1d81c7590 ---P--- 53 informix - 0 0
> > > > 1 211 859
> > > > 1d81c7de8 ---P--- 54 informix - 0 0
> > > > 2 223 298
> > > > 1d81c8640 ---P--- 55