Serious Informix/Solaris 2.6 Problem

Long post warning. If anyone has encountered this problem (or a similar one), please contact me at tgirsch@arch.com or tgirsch@archcomm.com. We've encountered an obscure but serious problem with Informix OnLine Dynamic Server 7.2X running against Solaris 2.6. The problem is characterized by an engine hang. Upon further inspection, we found that the ADM VP is receiving a SEGV and dying. Results of this vary from the engine hanging to an outright panic. The message you'll see (if you get any) in the online.log file typically looks like this: 10:51:15 mt.c, line 10162, thread 17, proc id 15522, Unexpected virtualprocess or termination, pid = 15523, exit = 0x8a . 10:51:15 Assert Failed: Unexpected virtual processor termination, pid = 15523,exit = 0x8a 10:51:15 Who: Session(2, informix@, 0, 0) Thread(17, kaio, 0, 1) 10:51:15 Results: Fatal Internal Error requires system shutdown 10:51:15 Action: Restart OnLine 10:51:15 See Also: /informix/tmp/af.11dcc2, shmem.11dcc2.0, /informix/7.24.UC1 /bin/core 10:52:54 The Master Daemon Died 10:53:21 or this: 12:39:56 Assert Failed: Internal Error - Segmentation Violation 12:39:56 Who: Session(0, @, 0, 0) Thread(0, noname, 0, 0) 12:39:56 Results: OnLine must abort 12:39:56 Action: Reinitialize shared memory 12:39:56 See Also: /informix/tmp/af.f66b, shmem.f66b.0, /home/informix/core 12:39:56 or just this: 13:22:47 Logical Log 198 Complete. 13:22:48 The common element seems to be a time with no message following it. This is because the ADM VP is responsible for generating these messages, and it is the ADM VP that's dying. We can replicate this problem readily on one 2.6 system (an Ultra Enterprise 6000) and sporadically on another (an Ultra Enterprise 4000). After about two weeks of testing, up to and including the installation of a debug version of the Informix engine, we traced the problem down to a parameterless fork() call. From here, Sun determined that a "frame pointer" is getting clobbered, and proceeded to look into it further. The last word I received from Sun's advanced support is that this appears to be a kernel-level bug, involving context-switching. The outlook is that there probably won't be a patch at any time in the near future. I've not yet received a bug number from Sun. If anyone wants the Informix bug number (for tracking purposes only -- as far as we can ascertain, this isn't an Informix bug per se), feel free to contact me in private e-mail. We've encountered this problem in every version of Informix we've tested, from 7.23.UC4 through 7.24.UC1. The problem seems only to occur when the system is under a heavy write-intensive load, and the likelihood that the problem will be encountered seems to be temporarily reduced by a system reboot. Changing configuration parameters also seems to effect how often (but not whether) the problem occurs. Even after the ADM VP dies, processing continues until a checkpoint is required, so it's difficult to pinpoint the exact time at which the problem first surfaces. If you think you may have encountered this problem (or a similar one), _please_ contact me. If we can find more sites that have encountered this problem, we can expedite a fix. In the meantime, our only alternative is to downgrade to Solaris 2.5.1. Thanks for your attention and time. - Tom Girsch Database Systems Manager Arch Communications Group, Inc. tgirsch@arch.com (513) 530-2500 (USA)