Crash in informix 11.50FC8
Two Informix 11.50FC8 instances crashed on HP-UX 11i v1 after configuration changes (increasing PHYSFILE to 250000 and LOGBUFF to 64). Error: "Can't open /dev/async. KAIO disabled" with kaiothread() assertion failure. Responses suggested verifying KAIO kernel configuration and /dev/async device existence, and checking system KAIO resource limits. User restarted instances and they ran successfully afterward; root cause unclear.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Hi,
A few hours ago two instances informix did crash in one of our hosts. The
message is kaiothread error.
Before I made the following change in the instances:
1) Change the location of physical log and size by:
onparams -p -s 250000 -d physdbs -y (before rootdbs)
2)
PHYSFILE 250000 (before 128000)
3)
LOGBUFF 64 (before 32)
After the changes the instances were restarted. Both instances did crash after
some minutes. The relevant line in the log are:
...
09:54:32 HPUX KAIO Segment locked addr=0xb68dc000 size=49152000
09:54:32 HPUX KAIO Segment locked addr=0xd9ed2000 size=300138496
09:54:33 Can't open /dev/async. KAIO disabled
09:54:33 Assert Failed: kaiothread() ERROR
09:54:33 IBM Informix Dynamic Server Version 11.50.FC8
09:54:33 Who: Session(1, informix@luz005, 0, 0000000000000000)
Thread(237, kaio, 0, 14)
File: kaio.c Line: 2231
09:54:33 stack trace for pid 21320 written to /tmp/af.4d5ce29
09:54:33 See Also: /tmp/af.4d5ce29
...
This instances were migrated from Informix 7.31 to 11.50FC8 a year ago.
S.O = HP-UX 11i v1
Please, provide any clues according your experience.
Thanks in advance
Roger
Hello.
First issue: it seems your OS version is not certified, regarding your machine
notes doc:
OS Information : Built on HP-UX 11i (aka 11.11), Certified on 11i V2 (aka
11.23)
and 11i v3 (aka 11.31).Had you followed all the others machine notes
recommendations?
Check this section: 8. Kernel Asynchronous I/O (KAIO)Your system is saying
that your async device is not working right.
There, in machine notes, is a section explaining how you should do to avoid
this situation, check it out ok?
Here is your link:
http://publib.boulder.ibm.com/infocenter/idshelp/v115/topic/com.ibm.relnotes.doc
/ifx_1150xc8/mach/ids_machine_notes_11.50.hp64.html
Hope it helps.
Regards.
Alexandre Marini
> To: ids@iiug.org
> From: rvilca@luzdelsur.com.pe
> Subject: Crash in informix 11.50FC8 [27658]
> Date: Wed, 18 Jul 2012 13:50:26 -0400
>
> Hi,
> A few hours ago two instances informix did crash in one of our hosts. The
> message is kaiothread error.
>
> Before I made the following change in the instances:
>
> 1) Change the location of physical log and size by:
>
> onparams -p -s 250000 -d physdbs -y (before rootdbs)>
> 2)
> PHYSFILE 250000 (before 128000)>
> 3)
> LOGBUFF 64 (before 32)>
> After the changes the instances were restarted. Both instances did crash
after
> some minutes. The relevant line in the log are:
>
> .....
>
> 09:54:32 HPUX KAIO Segment locked addr=0xb68dc000 size=49152000
> 09:54:32 HPUX KAIO Segment locked addr=0xd9ed2000 size=300138496
> 09:54:33 Can't open /dev/async. KAIO disabled
> 09:54:33 Assert Failed: kaiothread() ERROR
> 09:54:33 IBM Informix Dynamic Server Version 11.50.FC8
> 09:54:33 Who: Session(1, informix@luz005, 0, 0000000000000000)
>
> Thread(237, kaio, 0, 14)
>
> File: kaio.c Line: 2231
> 09:54:33 stack trace for pid 21320 written to /tmp/af.4d5ce29
> 09:54:33 See Also: /tmp/af.4d5ce29
>
> .....
>
> This instances were migrated from Informix 7.31 to 11.50FC8 a year ago.
>
> S.O = HP-UX 11i v1
>
> Please, provide any clues according your experience.
>
> Thanks in advance
> Roger
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
Yes, the OS version is certified. HP-UX 11i = HP-UX 11i V1 = HP-UX 11.11 Yes, KAIO is configured. The instances have been working with no problems for a year. I'm surprised that instances have made crash after the change of parameters. Thanks, Roger
This is an error that will be produced upon the first raw I/O done on e=
ach
cpu virtual processor. it is basically saying that the /dev/async devi=
ce
is
not available. To correct this you will have to do two things:
1) Make sure kaio is enable in the kernel
2) Make the device /dev/async as root
/usr/sbin/mknod /dev/async c 101 0
For more detailed informaiton see the release notes.
http://publib.boulder.ibm.com/infocenter/idshelp/v117/index.jsp?topic=3D=
%2Fcom.ibm.relnotes.doc%2Fnotes%2Fifx_1170xc4%2Fmach%2Fids_machine_note=
s_11.70.hpia64.html
John F. Miller III
STSM, Embedability Architect
miller3@us.ibm.com
503-578-5645
IBM Informix Dynamic Server (IDS)
ids-bounces@iiug.org wrote on 07/18/2012 10:50:26 AM:
> From: "ROGER VILCA" <rvilca@luzdelsur.com.pe>
> To: ids@iiug.org,
> Date: 07/18/2012 11:18 AM
> Subject: Crash in informix 11.50FC8 [27658]
> Sent by: ids-bounces@iiug.org
>
> Hi,
> A few hours ago two instances informix did crash in one of our hosts.=
The
> message is kaiothread error.
>
> Before I made the following change in the instances:
>
> 1) Change the location of physical log and size by:
>
> onparams -p -s 250000 -d physdbs -y (before rootdbs)>
> 2)
> PHYSFILE 250000 (before 128000)>
> 3)
> LOGBUFF 64 (before 32)>
> After the changes the instances were restarted. Both instances did
> crash after
> some minutes. The relevant line in the log are:
>
> ....
>
> 09:54:32 HPUX KAIO Segment locked addr=3D0xb68dc000 size=3D49152000
> 09:54:32 HPUX KAIO Segment locked addr=3D0xd9ed2000 size=3D300138496
> 09:54:33 Can't open /dev/async. KAIO disabled
> 09:54:33 Assert Failed: kaiothread() ERROR
> 09:54:33 IBM Informix Dynamic Server Version 11.50.FC8
> 09:54:33 Who: Session(1, informix@luz005, 0, 0000000000000000)
>
> Thread(237, kaio, 0, 14)
>
> File: kaio.c Line: 2231
> 09:54:33 stack trace for pid 21320 written to /tmp/af.4d5ce29
> 09:54:33 See Also: /tmp/af.4d5ce29
>
> ....
>
> This instances were migrated from Informix 7.31 to 11.50FC8 a year ag=
o.
>
> S.O =3D HP-UX 11i v1
>
> Please, provide any clues according your experience.
>
> Thanks in advance
> Roger
>
>
>
***********************************************************************=
********
> Forum Note: Use "Reply" to post a response in the discussion forum.=
>=
Thanks John As I said to Alexandre, KAIO is configured. The instances have been working with no problems for a year. I'm surprised that instances have made crash after the change of parameters. I restarted the instances and they are running, I don't know why they crashed. Any clues ? PD. The instances crashed in different times. Roger
On HPUX KAIO needs to be carefully tuned. Check the system logs, it may be that your IO activity level has increased over time and you ran out of KAIO resources in the OS. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Wed, Jul 18, 2012 at 5:23 PM, ROGER VILCA <rvilca@luzdelsur.com.pe>wrote: > Thanks John > As I said to Alexandre, KAIO is configured. The instances have been working > with no problems for a year. I'm surprised that instances have made crash > after the change of parameters. > I restarted the instances and they are running, I don't know why they > crashed. > Any clues ? > > PD. The instances crashed in different times. > > Roger > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --bcaec51a758e07b7f404c52224f7