Linux Operating System Limits/AF files missing

IDS Workgroup 11.50.UC9GE 65GB mem on box Two engines, one 27GB, one 20GB I believe we're running into mem issues (too many segs, etc.) which is causing the box or engine to crash. Not sure which just yet. Situation: The prod engine crashed due to ADM VP issue: 06/19/23 21:02:59 Fatal error in ADM VP at mt.c:14033 06/19/23 21:02:59Unexpected virtual processor termination, pid = 216374, exit = 0xb 06/19/23 21:02:59 PANIC: Attempting to bring system down The test engine died with no message, so I assume the box fell over. Q: there are no assertion failures files. Default location is $INFORMIXDIR/tmp - can't remember if that gets cleared on a box bounce (didn't think so). attempting to cap the test engine SHMTOTAL from 0 (of course). No matter what value I use for SHMTOTAL, the engine won't come up and msg log says "14:51:49 Value of SHMTOTAL exceeds operating system limits.". I used very small (which I expect to die since buffers is larger than what I used), and large as well, but same result. Pertinent kernel limits below: ------ Shared Memory Limits -------- max number of segments = 4096 max seg size (kbytes) = 18014398509481983 max total shared memory (kbytes) = 18014398509481980 min seg size (bytes) = 1 We've had a number of memory-related issues on this box. So I think something is awry in the kernel? Linux 4.12.14-lp151.27-default #1 SMP Fri May 10 14:13:15 UTC 2019 (862c838) x86_64 x86_64 x86_64 GNU/Linux Thanks in advance! Mark ------------------------------ Thanks! Mark Scranton The Mark Scranton Group mark@markscranton.com ------------------------------

I'd say "exit = 0xb" indicates this (adm) vp received a signal 11, i.e. SegV. Not clear how this would be related to (too) many shmem segments... is there an assertion and an af file? ------------------------------ Andreas Legner ------------------------------

Thanks Andreas ... No assertion failure files oddly enough. Not sure there were "too many segments" - just a behavior I've seen with this engine. Looking at data though it appears there wasn't a huge memory allocation on the box at the time. Mostly just stumped about Q2 in my post and the inability to cap mem with SHMTOTAL. Not suggesting this is the cause for this crash - but I need to be able to cap these engines w/ SHMTOTAL as I've seen as many as 91 segments allocated for their prod engine. I'll be changing SHMVIRTSIZE as soon as we bounce again of course to reduce the newly allocated segment number. Thanks! Mark mark@markscranton.com ------------------------------ Thanks! Mark Scranton The Mark Scranton Group mark@markscranton.com ------------------------------

Hi Mark, I can't answer the SHMTOTAL question, but it looks to me like the ADM VP may have been killed by the Linux out-of-memory killer. We see that fairly often when the engine dies with no associated assert failure. You can run "dmesg" to see if there were any incidents of a process being killed because the server was "Out of memory". Thanks, Barrie ------------------------------ Barrie Shaw ------------------------------

That I'd assume would result in "exit = 0x9", i.e. the process being sent a signal 9. ------------------------------ Andreas Legner ------------------------------

Good point. I stand corrected. Thanks, Barrie ------------------------------ Barrie Shaw ------------------------------