IDS random slow down
A user running IDS 10.00.UC6 on RHEL 3 reported the engine becoming unresponsive at random times, lasting minutes to hours, with normal performance otherwise. Responders asked for diagnostics taken during an episode (onstat -p, -P, -F, -R, -u, -d, -D, -g glo/iov/iof/act/wai, onstat -C and -C hot) plus the online log, and suggested tracing the running sqlexec thread via onstat -g act/-u/-g ses, checking what other sessions were waiting on (e.g. NETNORM) and for locks. No root cause was ever identified in the thread; the poster posted some onstat output, then reported the client had instead migrated to IDS 11.5 64-bit on RHEL 4 64-bit, which is now in production.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
I will apreciate any help. I have instaled Informix dynamic server 10.00 UC6 on RedHat Enterprise Linux 3 and I have the following situation. Aparently with out any reason and at random times of the day suddenly the engine becomes very unresponsive, this episodes can last for a few minutes to several houres. The rest of the day (or week)it performs normaly, so my question is: any of you have experimented this behabior?? and if yes, what do you do to cerrect it??
It happens, but there are always causes. What's happening on the server
during these "outages"? Post a section of the online log from the time from
before to after one of these episodes. Also, during an event try to
capture:
onstat -p
onstat -P
onstat -F
onstat -R
onstat -u
onstat -d
onstat -D
onstat -g glo
onstat -g iov
onstat -g iof
onstat -g act
onstat -g wai (if you have WAITSTATS enabled)
Art
Art S. Kagel
Oninit (www.oninit.com)
IIUG Board of Directors (art@iiug.org)
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Oninit, the IIUG, nor any other organization
with which I am associated either explicitly or implicitly. Neither do
those opinions reflect those of other individuals affiliated with any entity
with which I am affiliated nor those of the entities themselves.
2009/12/7 JAVIER HERNáNDEZ <javier_hdzt@yahoo.com.mx>
> I will apreciate any help.
>
> I have instaled Informix dynamic server 10.00 UC6 on RedHat Enterprise
> Linux 3
> and I have the following situation.
>
> Aparently with out any reason and at random times of the day suddenly the
> engine becomes very unresponsive, this episodes can last for a few minutes
> to
> several houres.
>
> The rest of the day (or week)it performs normaly, so my question is: any of
> you have experimented this behabior?? and if yes, what do you do to cerrect
> it??
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--000e0ce03d0e39ddc5047a2db214
JAVIER HERNáNDEZ wrote:
> I will apreciate any help.
>
> I have instaled Informix dynamic server 10.00 UC6 on RedHat Enterprise Linux
3
> and I have the following situation.
>
> Aparently with out any reason and at random times of the day suddenly the
> engine becomes very unresponsive, this episodes can last for a few minutes to
> several houres.
>
> The rest of the day (or week)it performs normaly, so my question is: any of
> you have experimented this behabior?? and if yes, what do you do to cerrect
> it??
Post the output from
onstat -C
onstat -C hot
--
Cheers,
Obnoxio The Clown
http://obotheclown.blogspot.com
I will now proceed to pleasure myself with this fish.
--
This message has been scanned for viruses and
dangerous content by OpenProtect(http://www.openprotect.com), and is
believed to be clean.
Thanks for your response, tomorrow Dec 9 will be in posibility of run the
onstat commands that you recomend and post the results.
Javier Hernández
Art,
This is the output of the commands that you recomended:
onstat -p
IBM Informix Dynamic Server Version 10.00.UC6 -- On-Line -- Up 1 days 14:53:17
-- 1351392 Kbytes
Profile
dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
993837 4671978 7455052730 99.99 2710588 2921594 26660379 89.84
isamtot open start read write rewrite delete commit rollbk
5056788046 4612495 7799340 4871519333 15011203 68912 136661 229651 8
gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
0 0 0 0 0 0 0
ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
0 0 0 34109.08 400.57 432 936
bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
88281 14 17112999194 0 0 72 53427 365600
ixda-RA idx-RA da-RA RA-pgsused lchwaits
40195 5858 550307 596318 59827
onstat -P
IBM Informix Dynamic Server Version 10.00.UC6 -- On-Line -- Up 1 days 14:53:17
-- 1351392 Kbytes
Buffer pool page size: 2048
partnum total btree data other dirty
0 11 1 6 4 1
1048578 3 1 1 1 0
1048579 13 5 8 0 0
1048580 19 10 9 0 0
1048581 3 2 1 0 0
1048582 9 4 5 0 0
1048583 4 1 3 0 0
1048584 7 3 4 0 0
1048589 1 1 0 0 0
1048592 1 1 0 0 0
1048596 1 1 0 0 0
1048598 1 1 0 0 0
1048599 1 1 0 0 0
1048600 1 1 0 0 0
1048602 1 1 0 0 0
1048603 3 3 0 0 0
1048611 3 2 1 0 0
1048623 4 3 1 0 0
1048624 3 3 0 0 0
4194305 2 0 0 2 1
4194306 93 42 49 2 5
4194307 158 73 83 2 5
4194308 102 11 91 0 0
4194309 43 21 20 2 4
4194310 2 1 1 0 0
4194311 12 3 7 2 5
4194312 2 1 1 0 0
4194313 5 2 1 2 4
4194315 1 1 0 0 0
4194316 104 18 86 0 0
4194317 12 7 5 0 0
4194319 1 1 0 0 0
4194320 31 19 12 0 0
4194321 90 6 83 1 0
4194322 11 6 5 0 0
4194323 8 6 2 0 0
4194324 3 2 1 0 0
4194325 10 5 5 0 0
4194326 1 1 0 0 0
4194327 1 1 0 0 0
4194329 2055 280 1775 0 0
4194330 108 21 87 0 0
4194331 1 1 0 0 0
4194332 4 3 1 0 0
4194335 2 1 0 1 0
4194336 1 1 0 0 0
4194338 4 3 1 0 0
4194341 1 1 0 0 0
4194347 1 1 0 0 0
4194350 215 139 76 0 0
4194351 3 3 0 0 0
4194356 6 0 6 0 0
4194357 3 3 0 0 0
4194358 1 0 1 0 0
4194359 1 1 0 0 0
4194360 7 0 6 1 0
4194361 1 1 0 0 0
4194366 4 0 3 1 0
4194368 6 0 6 0 0
4194369 2 2 0 0 0
4194370 1 0 0 1 0
4194372 11 0 10 1 0
4194373 4 4 0 0 0
4194374 1 0 1 0 0
4194375 1 1 0 0 0
4194376 1 0 1 0 0
4194377 1 1 0 0 0
4194378 1 0 1 0 0
4194379 1 1 0 0 0
4194380 1 0 1 0 0
4194381 1 1 0 0 0
4194382 1 0 1 0 0
4194384 1 0 1 0 0
4194385 1 1 0 0 0
4194386 1 0 1 0 0
4194387 1 1 0 0 0
4194388 3 0 2 1 0
4194389 1 1 0 0 0
4194392 143 0 142 1 0
4194393 49 49 0 0 0
4194394 541 0 540 1 0
4194395 37 37 0 0 0
4194396 59895 0 59877 18 20
4194397 2063 2061 0 2 3
4194398 2985 0 2984 1 0
4194400 1 0 1 0 0
4194402 2 0 2 0 2
4194403 5 5 0 0 0
4194404 1 0 1 0 0
4194406 1 0 1 0 0
4194407 1 1 0 0 0
4194412 1 0 1 0 0
4194414 9 0 7 2 1
4194415 3 3 0 0 0
4194416 2 0 1 1 0
4194417 2 2 0 0 0
4194418 1 0 0 1 0
4194419 3 3 0 0 0
4194420 1 0 0 1 0
4194421 2 2 0 0 0
4194424 13 0 13 0 0
4194425 2 2 0 0 0
4194426 16 0 14 2 1
4194427 27 27 0 0 0
4194428 1232 0 1230 2 0
4194429 20 20 0 0 0
4194430 77 0 75 2 0
4194431 4 4 0 0 0
4194432 3 0 1 2 0
4194433 3 3 0 0 0
4194436 1107 0 1105 2 0
4194437 36 34 0 2 0
4194444 366 0 365 1 0
4194445 103 103 0 0 0
4194448 2137 0 2135 2 0
4194449 128 125 0 3 0
4194452 16292 0 16286 6 2
4194453 621 619 0 2 2
4194454 1652 0 1650 2 3
4194455 88 88 0 0 1
4194456 14 0 12 2 1
4194457 29 27 0 2 0
4194458 53 0 51 2 0
4194459 92 90 0 2 0
4194460 3 0 1 2 0
4194461 3 3 0 0 0
4194466 1 0 1 0 0
4194467 1 1 0 0 0
4194468 3 0 2 1 0
4194469 1 1 0 0 0
4194470 3 0 2 1 0
4194471 1 1 0 0 0
4194472 4 0 2 2 0
4194473 5 5 0 0 0
4194474 1327 0 1325 2 0
4194475 3 3 0 0 0
4194476 4735 0 4732 3 0
4194477 6 4 0 2 0
4194478 1 0 1 0 1
4194479 1 1 0 0 0
4194480 14224 0 14219 5 1
4194481 20 20 0 0 0
4194482 3755 0 3751 4 0
4194483 783 781 0 2 0
4194484 2694 0 2692 2 0
4194485 6 6 0 0 0
4194486 1 0 1 0 0
4194487 1 0 1 0 0
4194488 1 1 0 0 0
4194489 1 0 1 0 0
4194490 1 1 0 0 0
4194491 1 0 1 0 0
4194493 9 0 8 1 0
4194494 4 4 0 0 0
4194495 14 0 14 0 0
4194496 15 15 0 0 0
4194499 13 0 12 1 0
4194500 2 2 0 0 0
4194501 1 0 1 0 0
4194502 1 1 0 0 0
4194503 33 0 32 1 0
4194504 38 38 0 0 0
4194507 1 0 1 0 0
4194508 1 1 0 0 0
4194509 1 0 1 0 0
4194510 1 1 0 0 0
4194513 125 0 123 2 1
4194514 20 20 0 0 0
4194515 23 0 23 0 0
4194516 4 4 0 0 0
4194517 21 0 20 1 0
4194518 6 6 0 0 0
4194519 45 0 45 0 0
4194520 16 16 0 0 0
4194521 26478 0 26462 16 1
4194522 27 27 0 0 0
4194523 3121 0 3118 3 5
4194524 5138 5136 0 2 5
4194525 14864 0 14848 16 0
4194526 44 44 0 0 0
4194527 6546 0 6542 4 0
4194528 31 31 0 0 0
4194529 310 0 307 3 0
4194530 55 55 0 0 0
4194531 38 0 36 2 0
4194532 4 4 0 0 0
4194535 37991 0 37968 23 0
4194536 14 14 0 0 0
4194544 8 0 7 1 0
4194545 11 0 9 2 1
4194546 23 21 0 2 0
4194547 79 0 77 2 0
4194548 7 5 0 2 0
4194551 5 0 3 2 0
4194552 8 6 0 2 0
4194553 1 0 1 0 0
4194554 1 1 0 0 0
4194555 1 0 1 0 0
4194556 1 1 0 0 0
4194557 1 0 1 0 0
4194558 1 1 0 0 0
4194559 1 0 0 1 0
4194561 1 0 1 0 0
4194562 1 1 0 0 0
4194563 3 0 2 1 0
4194564 1 1 0 0 0
4194565 9 0 9 0 0
4194567 1 0 1 0 0
4194575 1 0 1 0 0
4194576 1 1 0 0 0
4194579 3 0 2 1 0
4194581 1 0 1 0 0
4194582 1 1 0 0 0
4194583 390 0 389 1 0
4194584 19 19 0 0 0
4194585 17 0 16 1 0
4194586 40 40 0 0 0
4194587 140 0 139 1 0
4194589 236 0 234 2 2
4194590 36 36 0 0 1
4194591 1 0 1 0 0
4194592 1 1 0 0 0
4194593 35 0 34 1 0
4194595 593 0 591 2 2
4194596 178 178 0 0 1
4194597 7 0 5 2 2
4194598 5 5 0 0 1
4194599 109838 0 109809 29 4
4194600 25 23 0 2 1
4194601 84 0 82 2 2
4194602 53 51 0 2 1
4194603 4 0 2 2 0
4194604 2 2 0 0 0
4194605 3 0 1 2 0
4194606 4 4 0 0 0
4194607 3 0 1 2 0
4194608 3 2 0 1 0
4194609 18 0 17 1 0
4194621 3 0 1 2 2
4194622 3 3 0 0 1
4194623 1 0 1 0 1
4194624 1 1 0 0 1
4194625 5 0 3 2 1
4194626 15 15 0 0 0
4194627 5 0 3 2 1
4194628 3 3 0 0 0
4194629 1 0 1 0 0
4194630 1 1 0 0 0
4194633 216 0 216 0 0
4194634 105 105 0 0 0
4194653 2361 0 2359 2 1
4194654 3 3 0 0 0
4194655 961 0 959 2 0
4194656 197 197 0 0 0
4194657 13 0 11 2 0
4194658 10 10 0 0 0
4194659 1 0 1 0 0
4194660 1 1 0 0 0
4194661 1 0 1 0 0
4194662 1 1 0 0 0
4194663 8 0 7 1 0
4194664 4 4 0 0 0
4194665 1 0 1 0 0
4194667 1636 0 1634 2 3
4194668 106 104 0 2 1
4194669 55 0 53 2 0
4194670 14 14 0 0 0
4194671 7 0 5 2 0
4194672 3 3 0 0 0
4194673 7 0 5 2 1
4194674 3 3 0 0 0
4194675 1 0 0 1 0
4194676 3 3 0 0 0
4194677 4 0 2 2 0
4194678 4 3 0 1 0
4194679 26 0 25 1 0
4194680 1 1 0 0 0
4194681 19 0 18 1 0@
Obnoxio, I have included the output of the commands that you recomended in the response to Art.
Jumping in here...
I am guessing the diagnostics you posted were ran when the engine was in its
slow state right? I'd like to see a few things:
1) What is the sqlexec thread that is running actually doing?
You can get this information by looking at, in your diagnostics, the onstat -g
act output:
onstat -g act
IBM Informix Dynamic Server Version 10.00.UC6 -- On-Line -- Up 1 days 14:53:19
-- 1351392 Kbytes
Running threads:
tid tcb rstcb prty status vp-class name
89890 961fe9b0 95b2a478 2 running 1cpu sqlexec
The third column, rstcb, can be grepped out of an onstat -u. The onstat -u's
third column is session id. With session id known you can run onstat -g ses
<ses_id> to get details for the running session.
2) What are all of the other sessions waiting for?
My guess is you will see a condition of NETNORM when you grep the value of the
waiting column in the onstat -u out of an onstat -a output. Get that? Run an
onstat -a, redirect it to a file. Locate the onstat -u portion in the -aoutput and take note of the value in the waiting column. Grep for this value
in the onstat -a file and you should find a section that shows what the
waiting session is waiting for. I think you can also just run an onstat -g con
| grep <value in onstat -u wait column> as well.
3) Is that running sqlexec thread locking something it shouldn't?
I doubt this but check for the running sessions rstcb in the onstat -k output.
4) When you say things are slow - have you tried accessing the engine via
shared memory connections and tcp/ip connections?
MM
When I said that the engine slow down is because the users call to say so and
checking with dbaccess via tcpip it takes several seconds to display the list
of databases.
I'll make the checks that you are indicating and post results.
Thanks for your time.
To Art, Obnoxio, Mike, thanks for the advice. As a result of this problem the client finaly got convinced and we make the trancition to IDS 11.5 64-bit in RedHat 4 64-bit. The past days have been preparing and testing the instance and now is in full production. Javier Hernández
Great news, Javier! Glad we could help push them over the hump. They will be happy. Art Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) See you at the 2010 IIUG Informix Conference April 25-28, 2010 Overland Park (Kansas City), KS www.iiug.org/conf Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. 2009/12/15 JAVIER HERNáNDEZ <javier_hdzt@yahoo.com.mx> > To Art, Obnoxio, Mike, thanks for the advice. > > As a result of this problem the client finaly got convinced and we make the > trancition to IDS 11.5 64-bit in RedHat 4 64-bit. The past days have been > preparing and testing the instance and now is in full production. > > Javier Hernández > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --00151743f88473b653047ac7dc17