Stuck Transaction with BTS!
Mike Hoffman had a JDBC transaction updating a column in a BTS (basic text search) index hang on Informix 11.50.FC9/Solaris 10, leaving session 2826 holding 15 locks (including sysmaster) and causing -252 errors; onmode -z, -Z, -H and pushing logical logs all failed. The message log showed an assertion failure (MT_EX_OS, mem) and an exception during the MI_EVENT_END_XACT callback; stacks/pstack showed the thread stuck in bts_lock_write during rollback. Art Kagel advised bouncing the instance, which did clear the stuck transaction, but afterwards the BTS index gave BTS55 errors, which Michael suspected was related corruption (pursued in another thread).
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Hi All,
I am working remotely, so calling IBM is not a solution today. Hoping you all
can help instead.
Informix 11.50.FC9 running on Solaris 10
I have a transaction that was started over a JDBC connection, doing an update
on a column contained in a BTS index.
No error was returned but that transaction looked hung. So the user closed the
window.
Now, queries against the table return a 252 error (cannot get system info for
table). The sysmaster tables are locked.
onstat -x shows me the transaction is holding 15 locks. who-lock.sh shows thatsession 2826 is holding locks on the main table as well as some sysmaster
table.
onstat -u shows flags "--BP---" for the session.
onmode -z ses_id has been run several times, to no avail.
onmode -Z has no effect either -- returns "Transaction is busy"
onmode -H doesn't work because the transaction has not been commited.
There is no active user session on the Unix machine to kill.
What can I do to release those locks???
Thanks all!
Mike Hoffman
This is a development instance, with over 45 databases. So rebuilding the instance is an extremnely LAST resort. Even bouncing the index will cause ripples, but will be acceptable if nothing else will work. Thanks!
What does onstat -g ses for the session give?
What does onstat -g stk for the tid give?
Message in the online.log?
Message in os error log?
Regards,
David.
> On 29 October 2014 at 20:41 MICHAEL HOFFMAN <mrh@panix.com> wrote:
>
>
> This is a development instance, with over 45 databases. So rebuilding the
> instance is an extremnely LAST resort. Even bouncing the index will cause
> ripples, but will be acceptable if nothing else will work.
> Thanks!
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
Which sort of JDBC connection was it ? (Maybe XA protocol ?)
A way that could work is to execute
onmode -l until the session runs into long transaction.Then it should either rollback automatically or can be killed using onmode -H
Marcus Haarmann
----- Ursprüngliche Mail -----
Von: "MICHAEL HOFFMAN" <mrh@panix.com>
An: ids@iiug.org
Gesendet: Mittwoch, 29. Oktober 2014 21:40:09
Betreff: Stuck Transaction with BTS! [34055]
Hi All,
I am working remotely, so calling IBM is not a solution today. Hoping you all
can help instead.
Informix 11.50.FC9 running on Solaris 10
I have a transaction that was started over a JDBC connection, doing an update
on a column contained in a BTS index.
No error was returned but that transaction looked hung. So the user closed the
window.
Now, queries against the table return a 252 error (cannot get system info for
table). The sysmaster tables are locked.
onstat -x shows me the transaction is holding 15 locks. who-lock.sh shows thatsession 2826 is holding locks on the main table as well as some sysmaster
table.
onstat -u shows flags "--BP---" for the session.
onmode -z ses_id has been run several times, to no avail.
onmode -Z has no effect either -- returns "Transaction is busy"
onmode -H doesn't work because the transaction has not been commited.
There is no active user session on the Unix machine to kill.
What can I do to release those locks???
Thanks all!
Mike Hoffman
*******************************************************************************
Forum Note: Use "Reply" to post a response in the discussion forum.
I did push the logical logs passed the highwater mark, to no avail. :-(
In fact, onmode -l no longer moves to the next Logical Log.
1563eafa8 48 U-B---- 39241 1:249251 5000 5 0.10
1574c8028 47 U-B---- 39242 1:244251 5000 5 0.10
1574c8090 34 U---C-L 39243 1:179251 5000 11 0.22
1574c80f8 6 U-B---- 39165 1:27763 5000 5000 100.00
1574c8160 7 U-B---- 39166 1:32763 5000 5000 100.00
1574c81c8 8 U-B---- 39167 1:37763 5000 5000 100.00
1574c8230 9 U-B---- 39168 1:42763 5000 5000 100.00
1574c8298 10 U-B---- 39169 1:47763 5000 5000 100.00
The transaction in question starts in log #39187, much further down the list.
Mike
The most important thing would be to get onstat -g stk of the thread fo=
r
the txn.
From: "MICHAEL HOFFMAN" <mrh@panix.com>
To: ids@iiug.org
Date: 10/29/2014 05:23 PM
Subject: Re: Stuck Transaction with BTS! [34059]
Sent by: ids-bounces@iiug.org
I did push the logical logs passed the highwater mark, to no avail. :-(=
In fact, onmode -l no longer moves to the next Logical Log.
1563eafa8 48 U-B---- 39241 1:249251 5000 5 0.10
1574c8028 47 U-B---- 39242 1:244251 5000 5 0.10
1574c8090 34 U---C-L 39243 1:179251 5000 11 0.22
1574c80f8 6 U-B---- 39165 1:27763 5000 5000 100.00
1574c8160 7 U-B---- 39166 1:32763 5000 5000 100.00
1574c81c8 8 U-B---- 39167 1:37763 5000 5000 100.00
1574c8230 9 U-B---- 39168 1:42763 5000 5000 100.00
1574c8298 10 U-B---- 39169 1:47763 5000 5000 100.00
The transaction in question starts in log #39187, much further down the=
list.
Mike
***********************************************************************=
********
Forum Note: Use "Reply" to post a response in the discussion forum.
=
Hi David,
Ah... there was an Assery Violation buried way back in the message log (hours
ago!).
13:05:10 Assert Failed: Exception Caught. Type: MT_EX_OS, Context: mem
13:05:10 IBM Informix Dynamic Server Version 11.50.FC9
13:05:10 Who: Session(2826, javauser@dl093.dssc.sos.state.co.us, -1, 156687fe0
)
Thread(3380, sqlexec, 1566524c8, 3)
File: mtex.c Line: 417
13:05:10 Action: Please notify IBM Informix Technical Support.
13:05:10 stack trace for pid 1981 written to /tmp/af.111c3a66
13:05:11 See Also: /tmp/af.111c3a66, shmem.111c3a66.0
13:05:22 Exception Caught. Type: MT_EX_OS, Context: mem
13:05:22 Exception Caught. Type: MT_EX_OS, Context: mem
13:05:22 ERROR: Exception trap during MI_EVENT_END_XACT callback.
reason: mem
For the onmode -z commands, the message log *says* it worked, but really it
didn't:
13:25:20 sid 2826 username javauser@dl093.dssc.sos.state.co.us pid -1 terminate
d.
13:25:20 killed(MCMD_KILL)
13:25:31 sid 2826 username javauser@dl093.dssc.sos.state.co.us pid -1 terminate
d.
13:25:31 killed(MCMD_KILL)
13:29:55 Checkpoint Completed: duration was 6 seconds.
onstat -g ses shows no PID, and shows the transaction as active, but no values
changing.
onstat -u:
-----------
1566524c8 --BP--- 2826 javauser - 0 0 15 55305 16320
onstat -g ses 2826:
------------------------session effective #RSAM total used dynamic
id user user tty pid hostname threads memory memory explain
2826 blah12 - - -1 xxxxxx.ds 1 266240 179232 off
tid name rstcb flags curstk status
3380 sqlexec 1566524c8 --BP--- 9231 ready-
Memory pools count 2
name class addr totalsize freesize #allocfrag #freefrag
2826 V 157c33040 262144 86200 370 35
2826*O0 V 15cb1d040 4096 808 1 1
name free used name free used
overhead 0 6576 mtmisc 0 144
scb 0 144 opentable 0 16360
filetable 0 4352 ru 0 608
misc 0 136 log 0 16536
temprec 0 21664 partn 0 72
keys 0 1400 ralloc 0 43096
gentcb 0 1584 ostcb 0 2816
sort 0 152 sqscb 0 38184
sql 0 72 rdahead 0 1120
hashfiletab 0 552 osenv 0 2944
sqtcb 0 11208 fragman 0 4384
GenPg 0 856 sapi 0 296
SQL error 0 568 SAPI callback 0 944udr 0 464 sqlj 0 72
vii 0 1688
sqscb info
scb sqscb optofc pdqpriority optcompind directives
15b243218 1586d7028 0 0 0 1
Sess SQL Current Iso Lock SQL ISAM F.E.
Id Stmt type Database Lvl Mode ERR ERR Vers Explain
2826 char CR Not Wait -942 0 9.28 Off
Last parsed SQL statement :
update name set full_name = 'XXX YYY ZZZ', laorg_nm = 'AAA BBB CCC'
where name_id = 86887
onstat -g stk:
------------------------Stack for thread: 3380 sqlexecbase: 0x000000015b710000
len: 69632
pc: 0x0000000100e23ab4
tos: 0x000000015b71ebf1
state: running
vp: 3
0x100e23ab4 oninit :: yield_processor_mvp + 0x6d8 sp=0x15b71f3f0(0x162eedbc0,
0x101570, 0x10157d510, 0x10158b, 0x101400, 0x16053a028)
0xffffffff7b771b8c ?unknown? :: ?unknown? + 0x0 sp=0x15b71f520
delta_sp=304(0x165a29d00, 0x28000000, 0x165a39350, 0xffffffff7b91a498,
0x193a9c, 0xffffffff7b9064b8)
0xffffffff7b773884 ?unknown? :: ?unknown? + 0x0 sp=0x15b71f620
delta_sp=256(0x0, 0x28000000, 0x15b71f890, 0x165a39350, 0x14000, 0x0)
0xffffffff7b77e534 ?unknown? :: ?unknown? + 0x0 sp=0x15b71f6e0
delta_sp=192(0x0, 0x28000000, 0x15b71f890, 0x0, 0x165a43d00, 0x17000)
0xffffffff7b77b0b8 ?unknown? :: ?unknown? + 0x0 sp=0x15b71f7a0
delta_sp=192(0x0, 0x18b54c, 0x1586d7fe8, 0x165a1feb0, 0xffffffff7b9064b8,
0x165a43d00)
0x10027bbb0 oninit :: sqapi_callback_list_call + 0x984 sp=0x15b71f8d0
delta_sp=304(0x101315, 0x157995ed0, 0x101000, 0x69400, 0x159ccde40,
0x15cb21580)
0x10027ac58 oninit :: sqapi_call_callbacks + 0x160 sp=0x15b71fc40
delta_sp=880(0x0, 0xa, 0x1586d7fe8, 0x15b71fda8, 0x1400, 0x0)
0x10027c698 oninit :: sqapi_tx_callback + 0xac sp=0x15b71fcf0
delta_sp=176(0x15b71fe6c, 0x0, 0x2, 0x156687fe0, 0x10158b, 0x10157d4b8)
0x100659334 oninit :: rollbacktx + 0x40 sp=0x15b71fdb0 delta_sp=192(0x1, 0x0,
0x101400, 0x10157d4b8, 0x10157d4c0, 0x10157d)
0x100658eb8 oninit :: committx + 0x50 sp=0x15b71fe70 delta_sp=192(0x158c9bbc8,
0x1, 0x0, 0x1586d7028, 0x10157d, 0x10157d4b8)
0x10065fe9c oninit :: commitcmd + 0x140 sp=0x15b71ff30
delta_sp=192(0x158c9bbc8, 0x101400, 0x1, 0x0, 0x2, 0x101400)
0x10065c480 oninit :: excommand + 0x2940 sp=0x15b71ffe0
delta_sp=176(0x158c9bbc8, 0x159aec500, 0x10158b980, 0x2, 0x1, 0x1)
0x10052c4b4 oninit :: sq_execute + 0x1dc sp=0x15b720170
delta_sp=400(0x1586d7028, 0x6000000f, 0x10157d4c0, 0x10157d4b8, 0x10122ed70,
0x10e8)
0x1005dc46c oninit :: sqmain + 0xb28 sp=0x15b720280 delta_sp=272(0x1586d7028,
0x7, 0x1, 0x10122ed70, 0x20000, 0x10157d488)
0x100f00ee8 oninit :: listen_verify + 0x7d0 sp=0x15b720360
delta_sp=224(0x1566524c8, 0x158a45ed8, 0x10122ed70, 0x1388, 0x1005db944,
0x1566ba028)
0x100f00258 oninit :: spawn_thread + 0x1090 sp=0x15b7207f0
delta_sp=1168(0x101589, 0x0, 0x156f3ab28, 0x10157d498, 0x1566524c8,
0x1566ba028)
0x100e2ac74 oninit :: startup + 0x174 sp=0x15b720e50 delta_sp=1632(0xa,
0x10157d510, 0x100eff1c8, 0x1571fdba8, 0x1015b5be0, 0x1015701b8)
Madison,
I posted the stack thread in my other message, but to make it easier to see:
Stack for thread: 3380 sqlexecbase: 0x000000015b710000
len: 69632
pc: 0x0000000100e23ab4
tos: 0x000000015b71ebf1
state: ready
vp: 3
0x100e23ab4 oninit :: yield_processor_mvp + 0x6d8 sp=0x15b71f3f0(0x162eedbc0,
0x101570, 0x10157d510, 0x10158b, 0x101400, 0x16053a028)
0xffffffff7b771b8c ?unknown? :: ?unknown? + 0x0 sp=0x15b71f520
delta_sp=304(0x165a29d00, 0x28000000, 0x165a39350, 0xffffffff7b91a498,
0x193a9c, 0xffffffff7b9064b8)
0xffffffff7b773884 ?unknown? :: ?unknown? + 0x0 sp=0x15b71f620
delta_sp=256(0x0, 0x28000000, 0x15b71f890, 0x165a39350, 0x14000, 0x0)
0xffffffff7b77e534 ?unknown? :: ?unknown? + 0x0 sp=0x15b71f6e0
delta_sp=192(0x0, 0x28000000, 0x15b71f890, 0x0, 0x165a43d00, 0x17000)
0xffffffff7b77b0b8 ?unknown? :: ?unknown? + 0x0 sp=0x15b71f7a0
delta_sp=192(0x0, 0x18b54c, 0x1586d7fe8, 0x165a1feb0, 0xffffffff7b9064b8,
0x165a43d00)
0x10027bbb0 oninit :: sqapi_callback_list_call + 0x984 sp=0x15b71f8d0
delta_sp=304(0x101315, 0x157995ed0, 0x101000, 0x69400, 0x159ccde40,
0x15cb21580)
0x10027ac58 oninit :: sqapi_call_callbacks + 0x160 sp=0x15b71fc40
delta_sp=880(0x0, 0xa, 0x1586d7fe8, 0x15b71fda8, 0x1400, 0x0)
0x10027c698 oninit :: sqapi_tx_callback + 0xac sp=0x15b71fcf0
delta_sp=176(0x15b71fe6c, 0x0, 0x2, 0x156687fe0, 0x10158b, 0x10157d4b8)
0x100659334 oninit :: rollbacktx + 0x40 sp=0x15b71fdb0 delta_sp=192(0x1, 0x0,
0x101400, 0x10157d4b8, 0x10157d4c0, 0x10157d)
0x100658eb8 oninit :: committx + 0x50 sp=0x15b71fe70 delta_sp=192(0x158c9bbc8,
0x1, 0x0, 0x1586d7028, 0x10157d, 0x10157d4b8)
0x10065fe9c oninit :: commitcmd + 0x140 sp=0x15b71ff30
delta_sp=192(0x158c9bbc8, 0x101400, 0x1, 0x0, 0x2, 0x101400)
0x10065c480 oninit :: excommand + 0x2940 sp=0x15b71ffe0
delta_sp=176(0x158c9bbc8, 0x159aec500, 0x10158b980, 0x2, 0x1, 0x1)
0x10052c4b4 oninit :: sq_execute + 0x1dc sp=0x15b720170
delta_sp=400(0x1586d7028, 0x6000000f, 0x10157d4c0, 0x10157d4b8, 0x10122ed70,
0x10e8)
0x1005dc46c oninit :: sqmain + 0xb28 sp=0x15b720280 delta_sp=272(0x1586d7028,
0x7, 0x1, 0x10122ed70, 0x20000, 0x10157d488)
0x100f00ee8 oninit :: listen_verify + 0x7d0 sp=0x15b720360
delta_sp=224(0x1566524c8, 0x158a45ed8, 0x10122ed70, 0x1388, 0x1005db944,
0x1566ba028)
0x100f00258 oninit :: spawn_thread + 0x1090 sp=0x15b7207f0
delta_sp=1168(0x101589, 0x0, 0x156f3ab28, 0x10157d498, 0x1566524c8,
0x1566ba028)
0x100e2ac74 oninit :: startup + 0x174 sp=0x15b720e50 delta_sp=1632(0xa,
0x10157d510, 0x100eff1c8, 0x1571fdba8, 0x1015b5be0, 0x1015701b8)
What is the stack from /tmp/af.111c3a66?
Also this thread is running on vp 3
Run onstat -g glo to get the OS pid for VP 3 and then use pstack (or onmode -X)
to get the stack for the thread:
http://www-01.ibm.com/support/docview.wss?uid=swg21644662
Look like a callback related to the UDR - similar to
http://www-01.ibm.com/support/docview.wss?uid=swg1IC61977 but that is fixed in
your version.
Regards,
David.
> On 29 October 2014 at 22:34 MICHAEL HOFFMAN <mrh@panix.com> wrote:
>
>
> Hi David,
> Ah... there was an Assery Violation buried way back in the message log (hours
> ago!).
> 13:05:10 Assert Failed: Exception Caught. Type: MT_EX_OS, Context: mem
> 13:05:10 IBM Informix Dynamic Server Version 11.50.FC9
> 13:05:10 Who: Session(2826, javauser@dl093.dssc.sos.state.co.us, -1,
156687fe0
> )
>
> Thread(3380, sqlexec, 1566524c8, 3)
>
> File: mtex.c Line: 417
> 13:05:10 Action: Please notify IBM Informix Technical Support.
> 13:05:10 stack trace for pid 1981 written to /tmp/af.111c3a66
> 13:05:11 See Also: /tmp/af.111c3a66, shmem.111c3a66.0
> 13:05:22 Exception Caught. Type: MT_EX_OS, Context: mem
> 13:05:22 Exception Caught. Type: MT_EX_OS, Context: mem
> 13:05:22 ERROR: Exception trap during MI_EVENT_END_XACT callback.
>
> reason: mem
>
> For the onmode -z commands, the message log *says* it worked, but really it
> didn't:
> 13:25:20 sid 2826 username javauser@dl093.dssc.sos.state.co.us pid -1
> terminate
> d.
> 13:25:20 killed(MCMD_KILL)
> 13:25:31 sid 2826 username javauser@dl093.dssc.sos.state.co.us pid -1
> terminate
> d.
> 13:25:31 killed(MCMD_KILL)
> 13:29:55 Checkpoint Completed: duration was 6 seconds.
>
> onstat -g ses shows no PID, and shows the transaction as active, but novalues
> changing.
>
> onstat -u:
> -----------
> 1566524c8 --BP--- 2826 javauser - 0 0 15 55305 16320>
> onstat -g ses 2826:
> ------------------------> session effective #RSAM total used dynamic
> id user user tty pid hostname threads memory memory explain
> 2826 blah12 - - -1 xxxxxx.ds 1 266240 179232 off
>
> tid name rstcb flags curstk status
> 3380 sqlexec 1566524c8 --BP--- 9231 ready-
>
> Memory pools count 2
> name class addr totalsize freesize #allocfrag #freefrag
> 2826 V 157c33040 262144 86200 370 35
> 2826*O0 V 15cb1d040 4096 808 1 1
>
> name free used name free used
> overhead 0 6576 mtmisc 0 144
> scb 0 144 opentable 0 16360
> filetable 0 4352 ru 0 608
> misc 0 136 log 0 16536
> temprec 0 21664 partn 0 72
> keys 0 1400 ralloc 0 43096
> gentcb 0 1584 ostcb 0 2816
> sort 0 152 sqscb 0 38184
> sql 0 72 rdahead 0 1120
> hashfiletab 0 552 osenv 0 2944
> sqtcb 0 11208 fragman 0 4384
> GenPg 0 856 sapi 0 296
> SQL error 0 568 SAPI callback 0 944> udr 0 464 sqlj 0 72
> vii 0 1688
>
> sqscb info
> scb sqscb optofc pdqpriority optcompind directives
> 15b243218 1586d7028 0 0 0 1
>
> Sess SQL Current Iso Lock SQL ISAM F.E.
> Id Stmt type Database Lvl Mode ERR ERR Vers Explain
> 2826 char CR Not Wait -942 0 9.28 Off
>
> Last parsed SQL statement :
> update name set full_name = 'XXX YYY ZZZ', laorg_nm = 'AAA BBB CCC'>
> where name_id = 86887
>
> onstat -g stk:
> ------------------------> Stack for thread: 3380 sqlexec> base: 0x000000015b710000
> len: 69632
>
> pc: 0x0000000100e23ab4
> tos: 0x000000015b71ebf1
> state: running
>
> vp: 3
>
> 0x100e23ab4 oninit :: yield_processor_mvp + 0x6d8 sp=0x15b71f3f0(0x162eedbc0,
> 0x101570, 0x10157d510, 0x10158b, 0x101400, 0x16053a028)
> 0xffffffff7b771b8c ?unknown? :: ?unknown? + 0x0 sp=0x15b71f520
> delta_sp=304(0x165a29d00, 0x28000000, 0x165a39350, 0xffffffff7b91a498,
> 0x193a9c, 0xffffffff7b9064b8)
> 0xffffffff7b773884 ?unknown? :: ?unknown? + 0x0 sp=0x15b71f620
> delta_sp=256(0x0, 0x28000000, 0x15b71f890, 0x165a39350, 0x14000, 0x0)
> 0xffffffff7b77e534 ?unknown? :: ?unknown? + 0x0 sp=0x15b71f6e0
> delta_sp=192(0x0, 0x28000000, 0x15b71f890, 0x0, 0x165a43d00, 0x17000)
> 0xffffffff7b77b0b8 ?unknown? :: ?unknown? + 0x0 sp=0x15b71f7a0
> delta_sp=192(0x0, 0x18b54c, 0x1586d7fe8, 0x165a1feb0, 0xffffffff7b9064b8,
> 0x165a43d00)
> 0x10027bbb0 oninit :: sqapi_callback_list_call + 0x984 sp=0x15b71f8d0
> delta_sp=304(0x101315, 0x157995ed0, 0x101000, 0x69400, 0x159ccde40,
> 0x15cb21580)
> 0x10027ac58 oninit :: sqapi_call_callbacks + 0x160 sp=0x15b71fc40
> delta_sp=880(0x0, 0xa, 0x1586d7fe8, 0x15b71fda8, 0x1400, 0x0)
> 0x10027c698 oninit :: sqapi_tx_callback + 0xac sp=0x15b71fcf0
> delta_sp=176(0x15b71fe6c, 0x0, 0x2, 0x156687fe0, 0x10158b, 0x10157d4b8)
> 0x100659334 oninit :: rollbacktx + 0x40 sp=0x15b71fdb0 delta_sp=192(0x1, 0x0,
> 0x101400, 0x10157d4b8, 0x10157d4c0, 0x10157d)
> 0x100658eb8 oninit :: committx + 0x50 sp=0x15b71fe70
delta_sp=192(0x158c9bbc8,
> 0x1, 0x0, 0x1586d7028, 0x10157d, 0x10157d4b8)
> 0x10065fe9c oninit :: commitcmd + 0x140 sp=0x15b71ff30
> delta_sp=192(0x158c9bbc8, 0x101400, 0x1, 0x0, 0x2, 0x101400)
> 0x10065c480 oninit :: excommand + 0x2940 sp=0x15b71ffe0
> delta_sp=176(0x158c9bbc8, 0x159aec500, 0x10158b980, 0x2, 0x1, 0x1)
> 0x10052c4b4 oninit :: sq_execute + 0x1dc sp=0x15b720170
> delta_sp=400(0x1586d7028, 0x6000000f, 0x10157d4c0, 0x10157d4b8, 0x10122ed70,
> 0x10e8)
> 0x1005dc46c oninit :: sqmain + 0xb28 sp=0x15b720280 delta_sp=272(0x1586d7028,
> 0x7, 0x1, 0x10122ed70, 0x20000, 0x10157d488)
> 0x100f00ee8 oninit :: listen_verify + 0x7d0 sp=0x15b720360
> delta_sp=224(0x1566524c8, 0x158a45ed8, 0x10122ed70, 0x1388, 0x1005db944,
> 0x1566ba028)
> 0x100f00258 oninit :: spawn_thread + 0x1090 sp=0x15b7207f0
> delta_sp=1168(0x101589, 0x0, 0x156f3ab28, 0x10157d498, 0x1566524c8,
> 0x1566ba028)
> 0x100e2ac74 oninit :: startup + 0x174 sp=0x15b720e50 delta_sp=1632(0xa,
> 0x10157d510, 0x100eff1c8, 0x1571fdba8, 0x1015b5be0, 0x1015701b8)
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
Bounce the instance.
Art
Art S. Kagel, President and Principal Consultant
ASK Database Management
www.askdbmgt.com
Blog: http://informix-myview.blogspot.com/
Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on the IIUG, nor any other organization with which I am
associated either explicitly, implicitly, or by inference. Neither do
those opinions reflect those of other individuals affiliated with any
entity with which I am affiliated nor those of the entities themselves.
On Wed, Oct 29, 2014 at 1:40 PM, MICHAEL HOFFMAN <mrh@panix.com> wrote:
> Hi All,
> I am working remotely, so calling IBM is not a solution today. Hoping you
> all
> can help instead.
>
> Informix 11.50.FC9 running on Solaris 10
>
> I have a transaction that was started over a JDBC connection, doing an
> update
> on a column contained in a BTS index.
> No error was returned but that transaction looked hung. So the user closed
> the
> window.
> Now, queries against the table return a 252 error (cannot get system info
> for
> table). The sysmaster tables are locked.
>
> onstat -x shows me the transaction is holding 15 locks. who-lock.sh shows> that
> session 2826 is holding locks on the main table as well as some sysmaster
> table.
> onstat -u shows flags "--BP---" for the session.>
> onmode -z ses_id has been run several times, to no avail.
> onmode -Z has no effect either -- returns "Transaction is busy"
> onmode -H doesn't work because the transaction has not been commited.>
> There is no active user session on the Unix machine to kill.
>
> What can I do to release those locks???
>
> Thanks all!
> Mike Hoffman
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--089e0122930a933bc7050698cfb4
I had the sys admins run the pstack trace for VP 3 (bts vp; pid 1981) 1981: /soft/informix940/bin/oninit ----------------- lwp# 1 / thread# 1 -------------------- ffffffff7d6dbdf4 __time (15b71f5e0, 54524fa8, 101000, 156202c40, 101400, 0) + 8 ffffffff7b7718bc bts_lock_write (165a29d00, 28000000, 165a39350, ffffffff7b91a498, 193a9c, ffffffff7b9064b8) + 15c ffffffff7b773884 bts_lock_acquire_write_lock (0, 28000000, 15b71f890, 165a39350, 14000, 0) + ec ffffffff7b77e534 bts_init (0, 28000000, 15b71f890, 0, 165a43d00, 17000) + 23c ffffffff7b77b0b8 bts_xact_end_xact (0, 18b54c, 1586d7fe8, 165a1feb0, ffffffff7b9064b8, 165a43d00) + 150 000000010027bbb0 sqapi_callback_list_call (101315, 157995ed0, 101000, 69400, 159ccde40, 15cb21580) + 984 000000010027ac58 sqapi_call_callbacks (0, a, 1586d7fe8, 15b71fda8, 1400, 0) + 160 000000010027c698 sqapi_tx_callback (15b71fe6c, 0, 2, 156687fe0, 10158b, 10157d4b8) + ac 0000000100659334 rollbacktx (1, 0, 101400, 10157d4b8, 10157d4c0, 10157d) + 40 0000000100658eb8 committx (158c9bbc8, 1, 0, 1586d7028, 10157d, 10157d4b8) + 50 000000010065fe9c commitcmd (158c9bbc8, 101400, 1, 0, 2, 101400) + 140 000000010065c480 excommand (158c9bbc8, 159aec500, 10158b980, 2, 1, 1) + 2940 000000010052c4b4 sq_execute (1586d7028, 6000000f, 10157d4c0, 10157d4b8, 10122ed70, 10e8) + 1dc 00000001005dc46c sqmain (1586d7028, 7, 1, 10122ed70, 20000, 10157d488) + b28 0000000100f00ee8 listen_verify (1566524c8, 158a45ed8, 10122ed70, 1388, 1005db944, 1566ba028) + 7d0 0000000100f00258 spawn_thread (101589, 0, 156f3ab28, 10157d498, 1566524c8, 1566ba028) + 1090 0000000100e2ac74 startup (a, 10157d510, 100eff1c8, 1571fdba8, 1015b5be0, 1015701b8) + 174 0000000100e19578 mt_poll_yield (0, 0, 0, 0, 0, 0) + 114 ----------------- lwp# 2 / thread# 2 -------------------- ffffffff7d6dc4d0 kaio (6, 0, ffffffff7d84a300, 10, ffffffff7d84bf98, ffffffff7dd00a00) ffffffff7d6d8ad4 _lwp_start (0, 0, 0, 0, 0, 0)
Art, Bouncing is our last resort, but it looks like we'll go in that direction. Waiting to grab more stats first. Thanks, Mike
Hi Art, We finally did bounce the database, and that seemed to clear up the error. However, it brought on a new one (documented under the Datablade List forum) -- throwing a BTS55 error. I am now supposing that the stuck transaction has corrupted the BTS sysmaster tables and that is why we get the BTS55 error when trying to access the index. I am really getting the feeling that BTS 2.00 was 'not ready for prime time' release under Informix 11.5. Seems as though there is no stable version (we're on FC9) for it. :-( Thanks, Michael