IDS 11.70 FC1 Crash
Hi, Please help us to find solution, informix 11.70 fc1 on HP-UX intanium at my customer sites always suddenly down/crash. But after we try start, informix can be successfully online. Below i print message log: 09:38:26 Dynamically allocated new virtual shared memory segment (size 16384KB) 09:38:26 Memory sizes:resident:113820 KB, virtual:156856 KB, no SHMTOTAL limit 09:42:11 Dynamically allocated new virtual shared memory segment (size 16384KB) 09:42:11 Memory sizes:resident:113820 KB, virtual:173240 KB, no SHMTOTAL limit 09:43:17 Checkpoint Completed: duration was 0 seconds. 09:43:17 Mon Apr 6 - loguniq 10228, logpos 0x5af018, timestamp: 0x661fee5c Interval: 387501 09:43:17 Maximum server connections 188 09:43:17 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 319, Llog used 160 09:43:20 stack trace for pid 2457 written to /tmp/af.5b5e4b7 09:43:20 Assert Failed: No Exception Handler 09:43:20 IBM Informix Dynamic Server Version 11.70.FC1 09:43:20 Who: Session(438, daemon@FPEuro, 11821, c0000000ed3a6698) Thread(461, sqlexec, c0000000e84b9b00, 1) File: mtex.c Line: 498 09:43:20 Results: Exception Caught. Type: MT_EX_OS, Context: mem 09:43:20 Action: Please notify IBM Informix Technical Support. 09:43:20 See Also: /tmp/af.5b5e4b7, shmem.5b5e4b7.0 09:43:25 mtex.c, line 498, thread 461, proc id 2457, No Exception Handler. 09:43:25 The Master Daemon Died 09:43:25 Fatal error in ADM VP at mt.c:14250 09:43:25 Unexpected virtual processor termination, pid = 2464, exit = 0x100 09:43:25 PANIC: Attempting to bring system down 09:49:45 IBM Informix Dynamic Server Initialized -- Shared Memory Initialized. 09:49:45 Started 1 B-tree scanners. 09:49:45 B-tree scanner threshold set at 5000. 09:49:45 B-tree scanner range scan size set to -1. 09:49:45 B-tree scanner ALICE mode set to 6. 09:49:45 B-tree scanner index compression level set to med. 09:49:45 Physical Recovery Started at Page (1:2529). 09:49:45 Physical Recovery Complete: 12 Pages Examined, 12 Pages Restored. 09:49:45 Logical Recovery Started. 09:49:45 10 recovery worker threads will be started. 09:49:46 Logical Recovery has reached the transaction cleanup phase. 09:49:46 Logical Recovery Complete. 2 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks 09:49:47 Dataskip is now OFF for all dbspaces 09:49:47 Global Row Counter(Reset) 760209211392 000000b100000000 09:49:47 SCHAPI: Started dbScheduler thread. 09:49:47 Checkpoint Completed: duration was 0 seconds. 09:49:47 Mon Apr 6 - loguniq 10228, logpos 0x5b4018, timestamp: 0x661fefa9 Interval: 387502 09:49:47 Maximum server connections 0 09:49:47 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 22, Llog used 2 09:49:47 On-Line Mode 09:49:47 Booting Language <spl> from module <> 09:49:47 Loading Module <SPLNULL> 09:49:47 SCHAPI: Started 2 dbWorker threads. 09:49:51 Performance Advisory: Based on the current workload, the physical log might be too small to accommodate the time it takes to flush the buffer pool. 09:49:51 Results: The server might block transactions during checkpoints. 09:49:51 Action: If transactions are blocked during the checkpoint, increase the size of the physical log to at least 14000 KB. 09:49:51 Performance Advisory: The physical log is too small for automatic checkpoints. 09:49:51 Results: Automatic checkpoints are disabled. 09:49:51 Action: To enable automatic checkpoints, increase the physical log to at least 14000 KB. 09:49:52 Defragmenter cleaner thread now running 09:49:52 Defragmenter cleaner thread cleaned:0 partitions 09:50:18 WARNING! Physical Log size 8192 is too small. Physical Log overflows may occur during peak activity. Recommended minimum Physical Log size is 160 times maximum concurrent user threads. 09:50:21 Dynamically allocated new virtual shared memory segment (size 16384KB) 09:50:21 Memory sizes:resident:113820 KB, virtual:91320 KB, no SHMTOTAL limit 09:52:42 Dynamically allocated new virtual shared memory segment (size 16384KB) 09:52:42 Memory sizes:resident:113820 KB, virtual:107704 KB, no SHMTOTAL limit 09:53:53 Dynamically allocated new virtual shared memory segment (size 16384KB) 09:53:53 Memory sizes:resident:113820 KB, virtual:124088 KB, no SHMTOTAL limit 09:54:50 Checkpoint Completed: duration was 0 seconds. 09:54:50 Mon Apr 6 - loguniq 10228, logpos 0x69f570, timestamp: 0x66208db7 Interval: 387503 09:54:50 Maximum server connections 121 09:54:50 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 374, Llog used 235 09:55:22 Dynamically allocated new virtual shared memory segment (size 16384KB) 09:55:22 Memory sizes:resident:113820 KB, virtual:140472 KB, no SHMTOTAL limit 09:57:03 Dynamically allocated new virtual shared memory segment (size 16384KB) 09:57:03 Memory sizes:resident:113820 KB, virtual:156856 KB, no SHMTOTAL limit Please help us? I already don't have idea to solve this issue, informix always down/crash after 2 or 3 days online.
Hi, Check the tech note bellow to see if it applies: http://www-01.ibm.com/support/ <http://www-01.ibm.com/support/docview.wss?uid=swg21423191>; You can read the af.* file for more details about the crash. Given the fact that you've got an AF I recommend that you open a PMR. Best regards. On Mon, 6 Apr 2015 at 03:09 MOHD FADZIL JUSOH <fadzil@isianpadu.com> wrote: > Hi, > > Please help us to find solution, informix 11.70 fc1 on HP-UX intanium at my > customer sites always suddenly down/crash. But after we try start, informix > can be successfully online. Below i print message log: > > 09:38:26 Dynamically allocated new virtual shared memory segment (size > 16384KB) > 09:38:26 Memory sizes:resident:113820 KB, virtual:156856 KB, no SHMTOTAL > limit > 09:42:11 Dynamically allocated new virtual shared memory segment (size > 16384KB) > 09:42:11 Memory sizes:resident:113820 KB, virtual:173240 KB, no SHMTOTAL > limit > 09:43:17 Checkpoint Completed: duration was 0 seconds. > 09:43:17 Mon Apr 6 - loguniq 10228, logpos 0x5af018, timestamp: 0x661fee5c > Interval: 387501 > > 09:43:17 Maximum server connections 188 > 09:43:17 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, > Plog used 319, Llog used 160 > > 09:43:20 stack trace for pid 2457 written to /tmp/af.5b5e4b7 > 09:43:20 Assert Failed: No Exception Handler > 09:43:20 IBM Informix Dynamic Server Version 11.70.FC1 > 09:43:20 Who: Session(438, daemon@FPEuro, 11821, c0000000ed3a6698) > > Thread(461, sqlexec, c0000000e84b9b00, 1) > > File: mtex.c Line: 498 > 09:43:20 Results: Exception Caught. Type: MT_EX_OS, Context: mem > 09:43:20 Action: Please notify IBM Informix Technical Support. > 09:43:20 See Also: /tmp/af.5b5e4b7, shmem.5b5e4b7.0 > 09:43:25 mtex.c, line 498, thread 461, proc id 2457, No Exception Handler. > 09:43:25 The Master Daemon Died > 09:43:25 Fatal error in ADM VP at mt.c:14250 > 09:43:25 Unexpected virtual processor termination, pid = 2464, exit = 0x100 > > 09:43:25 PANIC: Attempting to bring system down > 09:49:45 IBM Informix Dynamic Server Initialized -- Shared Memory > Initialized. > > 09:49:45 Started 1 B-tree scanners. > 09:49:45 B-tree scanner threshold set at 5000. > 09:49:45 B-tree scanner range scan size set to -1. > 09:49:45 B-tree scanner ALICE mode set to 6. > 09:49:45 B-tree scanner index compression level set to med. > 09:49:45 Physical Recovery Started at Page (1:2529). > 09:49:45 Physical Recovery Complete: 12 Pages Examined, 12 Pages Restored. > 09:49:45 Logical Recovery Started. > 09:49:45 10 recovery worker threads will be started. > 09:49:46 Logical Recovery has reached the transaction cleanup phase. > 09:49:46 Logical Recovery Complete. > > 2 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks > > 09:49:47 Dataskip is now OFF for all dbspaces > 09:49:47 Global Row Counter(Reset) 760209211392 000000b100000000 > 09:49:47 SCHAPI: Started dbScheduler thread. > 09:49:47 Checkpoint Completed: duration was 0 seconds. > 09:49:47 Mon Apr 6 - loguniq 10228, logpos 0x5b4018, timestamp: 0x661fefa9 > Interval: 387502 > > 09:49:47 Maximum server connections 0 > 09:49:47 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, > Plog used 22, Llog used 2 > 09:49:47 On-Line Mode > 09:49:47 Booting Language <spl> from module <> > 09:49:47 Loading Module <SPLNULL> > 09:49:47 SCHAPI: Started 2 dbWorker threads. > 09:49:51 Performance Advisory: Based on the current workload, the physical > log > might be too small to > accommodate the time it takes to flush the buffer pool. > 09:49:51 Results: The server might block transactions during checkpoints. > 09:49:51 Action: If transactions are blocked during the checkpoint, > increase > the size of the > physical log to at least 14000 KB. > 09:49:51 Performance Advisory: The physical log is too small for automatic > checkpoints. > 09:49:51 Results: Automatic checkpoints are disabled. > 09:49:51 Action: To enable automatic checkpoints, increase the physical > log to > at least 14000 KB. > 09:49:52 Defragmenter cleaner thread now running > 09:49:52 Defragmenter cleaner thread cleaned:0 partitions > 09:50:18 WARNING! Physical Log size 8192 is too small. > > Physical Log overflows may occur during peak activity. > > Recommended minimum Physical Log size is 160 times maximum > > concurrent user threads. > > 09:50:21 Dynamically allocated new virtual shared memory segment (size > 16384KB) > 09:50:21 Memory sizes:resident:113820 KB, virtual:91320 KB, no SHMTOTAL > limit > 09:52:42 Dynamically allocated new virtual shared memory segment (size > 16384KB) > 09:52:42 Memory sizes:resident:113820 KB, virtual:107704 KB, no SHMTOTAL > limit > 09:53:53 Dynamically allocated new virtual shared memory segment (size > 16384KB) > 09:53:53 Memory sizes:resident:113820 KB, virtual:124088 KB, no SHMTOTAL > limit > 09:54:50 Checkpoint Completed: duration was 0 seconds. > 09:54:50 Mon Apr 6 - loguniq 10228, logpos 0x69f570, timestamp: 0x66208db7 > Interval: 387503 > > 09:54:50 Maximum server connections 121 > 09:54:50 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, > Plog used 374, Llog used 235 > > 09:55:22 Dynamically allocated new virtual shared memory segment (size > 16384KB) > 09:55:22 Memory sizes:resident:113820 KB, virtual:140472 KB, no SHMTOTAL > limit > 09:57:03 Dynamically allocated new virtual shared memory segment (size > 16384KB) > 09:57:03 Memory sizes:resident:113820 KB, virtual:156856 KB, no SHMTOTAL > limit > > Please help us? I already don't have idea to solve this issue, informix > always > down/crash after 2 or 3 days online. > > > ************************************************************ > ******************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --047d7beba2169c92c0051308f31c
Mohd: Your client has several problems that I can see: 1- The physical log is VERY small at 8196 K. The engine is recommending making it nearly double that. I would go even further and make it around 30MB. 2- This is a very early release of 11.70 and has several bugs in it. Three in particular cause the engine to allocate more virtual memory over and over until the machine's memory is exhausted. One thing you can try immediately is to set SHMTOTAL to something around 70% of available memory. But the best option would be to upgrade to v11.70.xC9 or later or v12.10.xC5 or later. I would open a support case with IBM for this anyway. If your client is not on IBM support, you should get them to renew support. Art Art S. Kagel, President and Principal Consultant ASK Database Management www.askdbmgt.com Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Sun, Apr 5, 2015 at 10:09 PM, MOHD FADZIL JUSOH <fadzil@isianpadu.com> wrote: > Hi, > > Please help us to find solution, informix 11.70 fc1 on HP-UX intanium at my > customer sites always suddenly down/crash. But after we try start, informix > can be successfully online. Below i print message log: > > 09:38:26 Dynamically allocated new virtual shared memory segment (size > 16384KB) > 09:38:26 Memory sizes:resident:113820 KB, virtual:156856 KB, no SHMTOTAL > limit > 09:42:11 Dynamically allocated new virtual shared memory segment (size > 16384KB) > 09:42:11 Memory sizes:resident:113820 KB, virtual:173240 KB, no SHMTOTAL > limit > 09:43:17 Checkpoint Completed: duration was 0 seconds. > 09:43:17 Mon Apr 6 - loguniq 10228, logpos 0x5af018, timestamp: 0x661fee5c > Interval: 387501 > > 09:43:17 Maximum server connections 188 > 09:43:17 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, > Plog used 319, Llog used 160 > > 09:43:20 stack trace for pid 2457 written to /tmp/af.5b5e4b7 > 09:43:20 Assert Failed: No Exception Handler > 09:43:20 IBM Informix Dynamic Server Version 11.70.FC1 > 09:43:20 Who: Session(438, daemon@FPEuro, 11821, c0000000ed3a6698) > > Thread(461, sqlexec, c0000000e84b9b00, 1) > > File: mtex.c Line: 498 > 09:43:20 Results: Exception Caught. Type: MT_EX_OS, Context: mem > 09:43:20 Action: Please notify IBM Informix Technical Support. > 09:43:20 See Also: /tmp/af.5b5e4b7, shmem.5b5e4b7.0 > 09:43:25 mtex.c, line 498, thread 461, proc id 2457, No Exception Handler. > 09:43:25 The Master Daemon Died > 09:43:25 Fatal error in ADM VP at mt.c:14250 > 09:43:25 Unexpected virtual processor termination, pid = 2464, exit = 0x100 > > 09:43:25 PANIC: Attempting to bring system down > 09:49:45 IBM Informix Dynamic Server Initialized -- Shared Memory > Initialized. > > 09:49:45 Started 1 B-tree scanners. > 09:49:45 B-tree scanner threshold set at 5000. > 09:49:45 B-tree scanner range scan size set to -1. > 09:49:45 B-tree scanner ALICE mode set to 6. > 09:49:45 B-tree scanner index compression level set to med. > 09:49:45 Physical Recovery Started at Page (1:2529). > 09:49:45 Physical Recovery Complete: 12 Pages Examined, 12 Pages Restored. > 09:49:45 Logical Recovery Started. > 09:49:45 10 recovery worker threads will be started. > 09:49:46 Logical Recovery has reached the transaction cleanup phase. > 09:49:46 Logical Recovery Complete. > > 2 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks > > 09:49:47 Dataskip is now OFF for all dbspaces > 09:49:47 Global Row Counter(Reset) 760209211392 000000b100000000 > 09:49:47 SCHAPI: Started dbScheduler thread. > 09:49:47 Checkpoint Completed: duration was 0 seconds. > 09:49:47 Mon Apr 6 - loguniq 10228, logpos 0x5b4018, timestamp: 0x661fefa9 > Interval: 387502 > > 09:49:47 Maximum server connections 0 > 09:49:47 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, > Plog used 22, Llog used 2 > 09:49:47 On-Line Mode > 09:49:47 Booting Language <spl> from module <> > 09:49:47 Loading Module <SPLNULL> > 09:49:47 SCHAPI: Started 2 dbWorker threads. > 09:49:51 Performance Advisory: Based on the current workload, the physical > log > might be too small to > accommodate the time it takes to flush the buffer pool. > 09:49:51 Results: The server might block transactions during checkpoints. > 09:49:51 Action: If transactions are blocked during the checkpoint, > increase > the size of the > physical log to at least 14000 KB. > 09:49:51 Performance Advisory: The physical log is too small for automatic > checkpoints. > 09:49:51 Results: Automatic checkpoints are disabled. > 09:49:51 Action: To enable automatic checkpoints, increase the physical > log to > at least 14000 KB. > 09:49:52 Defragmenter cleaner thread now running > 09:49:52 Defragmenter cleaner thread cleaned:0 partitions > 09:50:18 WARNING! Physical Log size 8192 is too small. > > Physical Log overflows may occur during peak activity. > > Recommended minimum Physical Log size is 160 times maximum > > concurrent user threads. > > 09:50:21 Dynamically allocated new virtual shared memory segment (size > 16384KB) > 09:50:21 Memory sizes:resident:113820 KB, virtual:91320 KB, no SHMTOTAL > limit > 09:52:42 Dynamically allocated new virtual shared memory segment (size > 16384KB) > 09:52:42 Memory sizes:resident:113820 KB, virtual:107704 KB, no SHMTOTAL > limit > 09:53:53 Dynamically allocated new virtual shared memory segment (size > 16384KB) > 09:53:53 Memory sizes:resident:113820 KB, virtual:124088 KB, no SHMTOTAL > limit > 09:54:50 Checkpoint Completed: duration was 0 seconds. > 09:54:50 Mon Apr 6 - loguniq 10228, logpos 0x69f570, timestamp: 0x66208db7 > Interval: 387503 > > 09:54:50 Maximum server connections 121 > 09:54:50 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, > Plog used 374, Llog used 235 > > 09:55:22 Dynamically allocated new virtual shared memory segment (size > 16384KB) > 09:55:22 Memory sizes:resident:113820 KB, virtual:140472 KB, no SHMTOTAL > limit > 09:57:03 Dynamically allocated new virtual shared memory segment (size > 16384KB) > 09:57:03 Memory sizes:resident:113820 KB, virtual:156856 KB, no SHMTOTAL > limit > > Please help us? I already don't have idea to solve this issue, informix > always > down/crash after 2 or 3 days online. > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --001a1140f538b9d52b05130cee5a