Locked Mutex session - freezing instance ✗ Answered: red (solid confidence) — Fernando Nunes suggests a probable known AIX bug related to reading in-flux sysmaster session structures and recommends opening a PMR; Art Kagel's follow-up is pure diagnostic questioning; the thread ends with the requested system details but no concrete answer to 'how to force-kill a hung session when onmode itself hangs' or what the session mutex is for.

Advisory only.

Hi AIX 5 IFX 10 FC4 We had a situation here where the instance froze in a way which commands ONMODE no longer worked and freeze the open connection of new sessions. Any ONMODE command executed, just freeze. After found the suspect session what freeze everybody , we just could not killed it because the onmode don't worked too. The solution was restart the server. My question is, situations like these, is there other way to force kill the session? The suspect session was appear be from our sentinel , where at this moment have a sqltrace active over a session. This session (sentinel) just keeped with running+reading state . The lock occur over the mutex "session" . For what is used this mutex? Moslty of the user access to the database is with IPCSHM and only few users with SOCTCP. But almost of users was locked on this mutex, just a couple of users keep running. Here is some output saved : > head -n 20 onstat.gwmx.out IBM Informix Dynamic Server Version 10.00.FC4 -- On-Line -- Up 6 days 01:03:28 -- 7663184 Kbytes Mutexes with waiters: mid addr name holder lkcnt waiter waittime 12 70000000009c030 session 2494402 0 2516947 1639 2506557 1637 16 1636 2495653 1635 2516831 1635 20 1635 2516844 1635 2516915 1633 18 1633 14 1630 2516958 1622 2516830 1615 2516866 1605 2490871 1603 2514053 1599 >grep 2494402 onstat.ath.out 2494402 7000001afde96c8 7000001bc336d68 2 running 5cpu sqlexec >grep 7000001bc336d68 onstat.u.out 7000001bc336d68 ---PR-- 262700 informix - 0 0 1 10 0 >cat onstat.gsql.out IBM Informix Dynamic Server Version 10.00.FC4 -- On-Line -- Up 6 days 01:17:58 -- 7663184 Kbytes Sess SQL Current Iso Lock SQL ISAM F.E. Id Stmt type Database Lvl Mode ERR ERR Vers Explain 262700 SELECT sysmaster CR Not Wait 0 0 9.03 Off Current statement name : _ifxc00000000003fx Current SQL statement : select t2.cbl_sessionid session_id, cbl_stmt[1,691] prpstm, t2.cbl_estcost sqs_estcst, t2.cbl_estrows sqs_estrws, t2.cbl_seqscan sqs_seqscn, t2.cbl_autoindex sqs_autoidx, t2.cbl_tempfile sqs_tempfls, cbl_sdbno seqnum from sysmaster:'informix'.sysconblock t2 where cbl_ismainblock='Y' AND length(cbl_stmt) between 190 AND 691 and t2.cbl_sessionid in (244926)

You must open a PMR I've found a very old issue on AIX (your system) fixed in 7.31.FC8... Maybe it's a regression. Your query is looking at session information, and this is an always changing structure. We need to take some measures to prevent the reading of dirty data (partially uninitialized structures). This includes some signal handling and apparently something could (can?!) go wrong. Only tech support will be able to help you. A stack trace of the session's thread would help tech support. Meanwhile you could try to reduce the number of times you run that (or similar) queries... So, it's apparently a bug, and it looks like one that should already been fixed... Regards. On Thu, Feb 3, 2011 at 4:15 PM, Cesar Inacio Martins < cesar_inacio_martins@yahoo.com.br> wrote: > Hi > > AIX 5 > IFX 10 FC4 > > We had a situation here where the instance froze in a way which commands > ONMODE no longer worked and freeze the open connection of new sessions. > Any ONMODE command executed, just freeze. > > After found the suspect session what freeze everybody , we just could not > killed it because the onmode don't worked too. > The solution was restart the server. > > My question is, situations like these, is there other way to force kill the > session? > > The suspect session was appear be from our sentinel , where at this moment > have a sqltrace active over a session. This session (sentinel) just keeped > with running+reading state . > The lock occur over the mutex "session" . For what is used this mutex? > > Moslty of the user access to the database is with IPCSHM and only few users > with SOCTCP. But almost of users was locked on this mutex, just a couple of > users keep running. > > Here is some output saved : > > > head -n 20 onstat.gwmx.out > > IBM Informix Dynamic Server Version 10.00.FC4 -- On-Line -- Up 6 days > 01:03:28 > -- 7663184 Kbytes> > Mutexes with waiters: > mid addr name holder lkcnt waiter waittime > 12 70000000009c030 session 2494402 0 2516947 1639 > > 2506557 1637 > > 16 1636 > > 2495653 1635 > > 2516831 1635 > > 20 1635 > > 2516844 1635 > > 2516915 1633 > > 18 1633 > > 14 1630 > > 2516958 1622 > > 2516830 1615 > > 2516866 1605 > > 2490871 1603 > > 2514053 1599 > > >grep 2494402 onstat.ath.out > 2494402 7000001afde96c8 7000001bc336d68 2 running 5cpu sqlexec > > >grep 7000001bc336d68 onstat.u.out > > 7000001bc336d68 ---PR-- 262700 informix - 0 0 1 10 0 > > >cat onstat.gsql.out > > IBM Informix Dynamic Server Version 10.00.FC4 -- On-Line -- Up 6 days > 01:17:58 > -- 7663184 Kbytes> > Sess SQL Current Iso Lock SQL ISAM F.E. > Id Stmt type Database Lvl Mode ERR ERR Vers Explain > 262700 SELECT sysmaster CR Not Wait 0 0 9.03 Off > > Current statement name : _ifxc00000000003fx > > Current SQL statement : > select t2.cbl_sessionid session_id, cbl_stmt[1,691] prpstm, t2.cbl_estcost > > sqs_estcst, t2.cbl_estrows sqs_estrws, t2.cbl_seqscan sqs_seqscn, > > t2.cbl_autoindex sqs_autoidx, t2.cbl_tempfile sqs_tempfls, cbl_sdbno > > seqnum from sysmaster:'informix'.sysconblock t2 where > > cbl_ismainblock='Y' AND length(cbl_stmt) between 190 AND 691 and > > t2.cbl_sessionid in (244926) > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --0015174c43c03bb231049b634bae

How many processor cores? How many CPU VPs configured? What are the NETTYPE, SERVERNAME, and DBSERVERNAME parameter settings from your ONCONFIG file? What are the relevant sqlhosts file entries for this server? Is Sentinel also running locally on the server? Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) IIUG Board of Directors (art@iiug.org) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Feb 3, 2011 at 11:15 AM, Cesar Inacio Martins < cesar_inacio_martins@yahoo.com.br> wrote: > Hi > > AIX 5 > IFX 10 FC4 > > We had a situation here where the instance froze in a way which commands > ONMODE no longer worked and freeze the open connection of new sessions. > Any ONMODE command executed, just freeze. > > After found the suspect session what freeze everybody , we just could not > killed it because the onmode don't worked too. > The solution was restart the server. > > My question is, situations like these, is there other way to force kill the > session? > > The suspect session was appear be from our sentinel , where at this moment > have a sqltrace active over a session. This session (sentinel) just keeped > with running+reading state . > The lock occur over the mutex "session" . For what is used this mutex? > > Moslty of the user access to the database is with IPCSHM and only few users > with SOCTCP. But almost of users was locked on this mutex, just a couple of > users keep running. > > Here is some output saved : > > > head -n 20 onstat.gwmx.out > > IBM Informix Dynamic Server Version 10.00.FC4 -- On-Line -- Up 6 days > 01:03:28 > -- 7663184 Kbytes> > Mutexes with waiters: > mid addr name holder lkcnt waiter waittime > 12 70000000009c030 session 2494402 0 2516947 1639 > > 2506557 1637 > > 16 1636 > > 2495653 1635 > > 2516831 1635 > > 20 1635 > > 2516844 1635 > > 2516915 1633 > > 18 1633 > > 14 1630 > > 2516958 1622 > > 2516830 1615 > > 2516866 1605 > > 2490871 1603 > > 2514053 1599 > > >grep 2494402 onstat.ath.out > 2494402 7000001afde96c8 7000001bc336d68 2 running 5cpu sqlexec > > >grep 7000001bc336d68 onstat.u.out > > 7000001bc336d68 ---PR-- 262700 informix - 0 0 1 10 0 > > >cat onstat.gsql.out > > IBM Informix Dynamic Server Version 10.00.FC4 -- On-Line -- Up 6 days > 01:17:58 > -- 7663184 Kbytes> > Sess SQL Current Iso Lock SQL ISAM F.E. > Id Stmt type Database Lvl Mode ERR ERR Vers Explain > 262700 SELECT sysmaster CR Not Wait 0 0 9.03 Off > > Current statement name : _ifxc00000000003fx > > Current SQL statement : > select t2.cbl_sessionid session_id, cbl_stmt[1,691] prpstm, t2.cbl_estcost > > sqs_estcst, t2.cbl_estrows sqs_estrws, t2.cbl_seqscan sqs_seqscn, > > t2.cbl_autoindex sqs_autoidx, t2.cbl_tempfile sqs_tempfls, cbl_sdbno > > seqnum from sysmaster:'informix'.sysconblock t2 where > > cbl_ismainblock='Y' AND length(cbl_stmt) between 190 AND 691 and > > t2.cbl_sessionid in (244926) > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --0015175cb2c6468e6d049b63be03

Hi Art and Fernando, So, unfortunatelly we could have time to generate a full dump at the moment, only a fews onstat are generates. We started onstat -a , onstat -g all , onstat -g stk all . But keep more the Art, The sentinel is in other machine (windows), accessing the database over SOCTCP . 12 CPUvps NETTYPE ipcshm,3,450,CPU # Configure poll thread(s) for nettype NETTYPE soctcp,3,450,NET # Configure poll thread(s) for nettype DBSERVERNAME xyz # Name of default database server DBSERVERALIASES abc_soc # List of alternate dbservernames xyz onipcshm host1 xyz_shm abc_soc onsoctcp host1 abc_soc At the moment we have no more of 500 sessions.. Cesar --- Em qui, 3/2/11, Art Kagel <art.kagel@gmail.com> escreveu: De: Art Kagel <art.kagel@gmail.com> Assunto: Re: Locked Mutex session - freezing instance [22696] Para: ids@iiug.org Data: Quinta-feira, 3 de Fevereiro de 2011, 15:03 How many processor cores? How many CPU VPs configured? What are the NETTYPE, SERVERNAME, and DBSERVERNAME parameter settings from your ONCONFIG file? What are the relevant sqlhosts file entries for this server? Is Sentinel also running locally on the server? Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) IIUG Board of Directors (art@iiug.org) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, Feb 3, 2011 at 11:15 AM, Cesar Inacio Martins < cesar_inacio_martins@yahoo.com.br> wrote: > Hi > > AIX 5 > IFX 10 FC4 > > We had a situation here where the instance froze in a way which commands > ONMODE no longer worked and freeze the open connection of new sessions. > Any ONMODE command executed, just freeze. > > After found the suspect session what freeze everybody , we just could not > killed it because the onmode don't worked too. > The solution was restart the server. > > My question is, situations like these, is there other way to force kill the > session? > > The suspect session was appear be from our sentinel , where at this moment > have a sqltrace active over a session. This session (sentinel) just keeped > with running+reading state . > The lock occur over the mutex "session" . For what is used this mutex? > > Moslty of the user access to the database is with IPCSHM and only few users > with SOCTCP. But almost of users was locked on this mutex, just a couple of > users keep running. > > Here is some output saved : > > > head -n 20 onstat.gwmx.out > > IBM Informix Dynamic Server Version 10.00.FC4 -- On-Line -- Up 6 days > 01:03:28 > -- 7663184 Kbytes> > Mutexes with waiters: > mid addr name holder lkcnt waiter waittime > 12 70000000009c030 session 2494402 0 2516947 1639 > > 2506557 1637 > > 16 1636 > > 2495653 1635 > > 2516831 1635 > > 20 1635 > > 2516844 1635 > > 2516915 1633 > > 18 1633 > > 14 1630 > > 2516958 1622 > > 2516830 1615 > > 2516866 1605 > > 2490871 1603 > > 2514053 1599 > > >grep 2494402 onstat.ath.out > 2494402 7000001afde96c8 7000001bc336d68 2 running 5cpu sqlexec > > >grep 7000001bc336d68 onstat.u.out > > 7000001bc336d68 ---PR-- 262700 informix - 0 0 1 10 0 > > >cat onstat.gsql.out > > IBM Informix Dynamic Server Version 10.00.FC4 -- On-Line -- Up 6 days > 01:17:58 > -- 7663184 Kbytes> > Sess SQL Current Iso Lock SQL ISAM F.E. > Id Stmt type Database Lvl Mode ERR ERR Vers Explain > 262700 SELECT sysmaster CR Not Wait 0 0 9.03 Off > > Current statement name : _ifxc00000000003fx > > Current SQL statement : > select t2.cbl_sessionid session_id, cbl_stmt[1,691] prpstm, t2.cbl_estcost > > sqs_estcst, t2.cbl_estrows sqs_estrws, t2.cbl_seqscan sqs_seqscn, > > t2.cbl_autoindex sqs_autoidx, t2.cbl_tempfile sqs_tempfls, cbl_sdbno > > seqnum from sysmaster:'informix'.sysconblock t2 where > > cbl_ismainblock='Y' AND length(cbl_stmt) between 190 AND 691 and > > t2.cbl_sessionid in (244926) > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --0015175cb2c6468e6d049b63be03 ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.