Stored procedure failing after IDS4 server crash

I am trying to assist a friend in Africa with an IDS 9.4 FC3 problem on AIX 5.2 The server crashed with the following log information. 15:29:42 Assert Warning: I/O error, Primary Chunk '/prod_u02fs/physdbs' -- Offline 15:29:42 IBM Informix Dynamic Server Version 9.40.FC6 15:29:42 Who: Session(1, root@cscs_prod, 0, 700000030310028) Thread(7, main_loop(), 7000000302ce028, 3) File: rsbuff.c Line: 4605 15:29:42 Results: Chunk is now unusable 15:29:42 Action: Repair and restore from mirror or archive 15:29:42 stack trace for pid 725244 written to /tmp/af.3eff6d5 15:29:42 See Also: /tmp/af.3eff6d5 15:29:42 I/O error, Primary Chunk '/prod_u02fs/physdbs' -- Offline 15:29:43 Assert Failed: INFORMIX-OnLine Must ABORT Critical media failure. 15:29:43 IBM Informix Dynamic Server Version 9.40.FC6 15:29:43 Who: Session(1, root@cscs_prod, 0, 700000030310028) Thread(7, main_loop(), 7000000302ce028, 3) File: rsmirror.c Line: 1817 15:29:43 stack trace for pid 725244 written to /tmp/af.3eff6d5 15:29:43 See Also: /tmp/af.3eff6d5 15:29:45 rsmirror.c, line 1817, thread 7, proc id 725244,INFORMIX-OnLine Must ABORT Critical media failure.. 15:29:46 Fatal error in ADM VP at mt.c:12358 15:29:46 Unexpected virtual processor termination, pid = 725244, exit = 0x100 15:29:46 PANIC: Attempting to bring system down 15:41:43 IBM Informix Dynamic Server Started. 15:41:43 forced-resident shared memory not available 15:41:43 Shared memory segment 0x700000020000000 could not be forced resident. Mon Mar 27 15:41:43 2006 15:41:43 Event alarms enabled. ALARMPROG = '/opt/informix/product/9.4.0/etc/alarmprogram.sh' 15:41:43 Dynamically allocated new virtual shared memory segment (size 32768KB) 15:41:43 Booting Language <c> from module <> 15:41:43 Loading Module <CNULL> 15:41:43 Booting Language <builtin> from module <> 15:41:43 Loading Module <BUILTINNULL> 15:41:43 VP pid=745708 priority fixed at 60, former = 63 15:41:48 AIX MP latch code enabled 15:41:48 IBM Informix Dynamic Server Version 9.40.FC6 SoftwareSerial Number AAA#B000000 15:41:48 IBM Informix Dynamic Server Initialized -- Shared MemoryInitialized. 15:41:48 Physical Recovery Started at Page (4:96569). 15:41:48 Physical Recovery Complete: 692 Pages Examined, 53 Pages Restored. 15:41:48 Logical Recovery Started. 15:41:48 10 recovery worker threads will be started. 15:41:50 Fast Recovery Switching to Log 9115 15:41:52 Fast Recovery Switching to Log 9116 15:41:53 Fast Recovery Switching to Log 9117 15:41:55 Fast Recovery Switching to Log 9118 15:41:56 Fast Recovery Switching to Log 9119 15:42:00 Logical Recovery has reached the transaction cleanup phase. 15:42:00 Logical Recovery Complete. 109 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks 15:42:01 Dataskip is now OFF for all dbspaces 15:42:01 Checkpoint Completed: duration was 0 seconds. 15:42:01 Checkpoint loguniq 9119, logpos 0x44c018, timestamp:0x51044c07 15:42:01 Maximum server connections 0 15:42:01 On-Line Mode 15:42:16 Booting Language <spl> from module <> 15:42:16 Loading Module <SPLNULL> 15:43:57 Loading Module <$INFORMIXDIR/extend/web.4.13.FC3/web.bld> 15:43:57 The C Language Module </opt/informix/product/9.4.0/extend/web.4.13.FC3/web.bld> loaded 15:44:23 Loading Module <$INFORMIXDIR/extend/Node.1.0/Node.bld> 15:44:23 The C Language Module </opt/informix/product/9.4.0/extend/Node.1.0/Node.bld> loaded 15:50:58 Loading Module <$INFORMIXDIR/extend/ETX.1.30.FC8/ETX.bld> 15:50:58 The C Language Module </opt/informix/product/9.4.0/extend/ETX.1.30.FC8/ETX.bld> loaded 15:50:58 Dynamically allocated new virtual shared memory segment (size 32768KB) 15:50:58 Dynamically allocated new virtual shared memory segment (size 32768KB) 15:50:59 Dynamically allocated new virtual shared memory segment (size 32768KB) 15:52:03 Logical Log 9119 Complete, timestamp: 0x51065325. 15:52:30 Checkpoint Completed: duration was 0 seconds. 15:52:30 Checkpoint loguniq 9120, logpos 0x680018, timestamp:0x51071ae3 15:52:30 Maximum server connections 4 15:54:25 Assert Failed: No Exception Handler 15:54:25 IBM Informix Dynamic Server Version 9.40.FC6 15:54:25 Who: Session(50, informix@172.16.1.5, 3536, 700000030311fd8) Thread(75, sqlexec, 7000000302d5258, 1) File: mtex.c Line: 431 15:54:25 Results: Exception Caught. Type: MT_EX_OS, Context: mem 15:54:25 Action: Please notify IBM Informix Technical Support. 15:54:25 stack trace for pid 745708 written to /tmp/af.433fca0 15:54:25 See Also: /tmp/af.433fca0 15:54:27 mtex.c, line 431, thread 75, proc id 745708, No Exception Handler. 15:54:28 The Master Daemon Died 15:54:28 The Master Daemon Died 15:54:28 The Master Daemon Died 15:54:28 The Master Daemon Died 15:54:28 The Master Daemon Died 15:54:28 The Master Daemon Died 15:54:30 PANIC: Attempting to bring system down 09:24:13 IBM Informix Dynamic Server Started. 09:24:13 forced-resident shared memory not available 09:24:13 Shared memory segment 0x700000020000000 could not be forced resident. The next day, the database was restarted without any other actions. The physdbs dbspace came back online 09:24:13 Event alarms enabled. ALARMPROG = '/opt/informix/product/9.4.0/etc/alarmprogram.sh' 09:24:13 Dynamically allocated new virtual shared memory segment (size 32768KB) 09:24:13 Booting Language <c> from module <> 09:24:13 Loading Module <CNULL> 09:24:13 Booting Language <builtin> from module <> 09:24:13 Loading Module <BUILTINNULL> 09:24:13 VP pid=745644 priority fixed at 60, former = 63 09:24:18 AIX MP latch code enabled 09:24:18 IBM Informix Dynamic Server Version 9.40.FC6 SoftwareSerial Number AAA#B000000 09:24:18 IBM Informix Dynamic Server Initialized -- Shared MemoryInitialized. 09:24:18 Physical Recovery Started at Page (4:97789). 09:24:18 Physical Recovery Complete: 71 Pages Examined, 42 Pages Restored. 09:24:18 Logical Recovery Started. 09:24:18 10 recovery worker threads will be started. 09:24:21 Logical Recovery has reached the transaction cleanup phase. 09:24:21 Logical Recovery Complete. 10 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks 09:24:22 Dataskip is now OFF for all dbspaces 09:24:22 Checkpoint Completed: duration was 0 seconds. 09:24:22 Checkpoint loguniq 9120, logpos 0x9bc018, timestamp:0x51085e3e 09:24:22 Maximum server connections 0 09:24:22 On-Line Mode 09:24:28 Loading Module <$INFORMIXDIR/extend/web.4.13.FC3/web.bld> 09:24:28 The C Language Module </opt/informix/product/9.4.0/extend/web.4.13.FC3/web.bld> loaded 09:24:40 Booting Language <spl> from module <> 09:24:40 Loading Module <SPLNULL> 09:25:16 Loading Module <$INFORMIXDIR/extend/Node.1.0/Node.bld> 09:25:16 The C Language Module </opt/informix/product/9.4.0/extend/Node.1.0/Node.bld> loaded Now when they run a stored procedure called "loadfiles" through an ODBC connection, it fails with the following information. This stored procedure has been running for months. 10:25:03 Assert Failed: Exception Caught. Type: MT_EX_OS, Context: mem1

Gary wrote: > I am trying to assist a friend in Africa with an IDS 9.4 FC3 problem on > AIX 5.2 > > The server crashed with the following log information. > > 15:29:42 Assert Warning: I/O error, Primary Chunk > '/prod_u02fs/physdbs' -- Offline > 15:29:42 IBM Informix Dynamic Server Version 9.40.FC6 9.40.FC6 not 9.40.FC3. > 15:29:42 Who: Session(1, root@cscs_prod, 0, 700000030310028) > Thread(7, main_loop(), 7000000302ce028, 3) > File: rsbuff.c Line: 4605 > 15:29:42 Results: Chunk is now unusable > 15:29:42 Action: Repair and restore from mirror or archive > 15:29:42 stack trace for pid 725244 written to /tmp/af.3eff6d5 > 15:29:42 See Also: /tmp/af.3eff6d5 <snip>> > > Now when they run a stored procedure called "loadfiles" through an ODBC > connection, it fails with the following information. This stored > procedure has been running for months. > > > 10:25:03 Assert Failed: Exception Caught. Type: MT_EX_OS, Context: mem > 10:25:03 IBM Informix Dynamic Server Version 9.40.FC6 > 10:25:03 Who: Session(126, informix@172.16.1.2, 2696,> 700000030312248) > Thread(151, sqlexec, 7000000302d5258, 1) > File: mtex.c Line: 377 > 10:25:03 Action: Please notify IBM Informix Technical Support. > 10:25:03 stack trace for pid 745644 written to /tmp/af.47f00ef > 10:25:03 See Also: /tmp/af.47f00ef > 10:25:05 Exception Caught. Type: MT_EX_OS, Context: mem > 10:25:05 (-9791): ERROR: Routine execution trap -- > procname=<loadfiles> procid=903 > reason: mem > 10:25:21 Assert Warning: Memory block header corruption detected in > mt_shm_free 2 > 10:25:21 IBM Informix Dynamic Server Version 9.40.FC6 > 10:25:21 Who: Session(126, informix@172.16.1.2, 2696, I would suggest that your "friend" just drop and re-create the stored procedure before you go any further.

Thanks TBP. He re-created the stored procedure yesterday and it still failed.The stored procedure uploads files into sblobs. I got him to send me the files that needed to be uploaded and tested the procedure on my machine which has a copy of their database structures and it worked perfectly. they have uploaded +- 1 million files in the last two months. Is it possible that the memory has not been cleared since the initial problem. Could there be a job running that has to be killed before ODBC will work again ?

1/ 15:29:42 Assert Warning: I/O error, Primary Chunk '/prod_u02fs/physdbs' -- Offline 15:29:42 IBM Informix Dynamic Server Version 9.40.FC6 15:29:42 Who: Session(1, root@cscs_prod, 0, 700000030310028) Thread(7, main_loop(), 7000000302ce028, 3) File: rsbuff.c Line: 4605 15:29:42 Results: Chunk is now unusable 15:29:42 Action: Repair and restore from mirror or archive 15:29:42 stack trace for pid 725244 written to /tmp/af.3eff6d5 15:29:42 See Also: /tmp/af.3eff6d5 15:29:42 I/O error, Primary Chunk '/prod_u02fs/physdbs' -- Offline 15:29:43 Assert Failed: INFORMIX-OnLine Must ABORT Critical media failure. 15:29:43 IBM Informix Dynamic Server Version 9.40.FC6 15:29:43 Who: Session(1, root@cscs_prod, 0, 700000030310028) Thread(7, main_loop(), 7000000302ce028, 3) File: rsmirror.c Line: 1817 15:29:43 stack trace for pid 725244 written to /tmp/af.3eff6d5 15:29:43 See Also: /tmp/af.3eff6d5 15:29:45 rsmirror.c, line 1817, thread 7, proc id 725244,INFORMIX-OnLine Must ABORT Critical media failure.. They have had a device failure on a critical dbspace. It "repaired itself" overnight. Were any non critical dbspaces also affected by the same device failure? Have they perhaps not rectified themselves 2/ 10:27:25 Results: Unable to repair pool 10:27:25 Action: Please notify IBM Informix Technical Support. 10:27:25 stack trace for pid 253952 written to /tmp/af.3017d 10:27:25 See Also: /tmp/af.3017d I strongly recommend your friend follows this advice. I assume the have a valid support contract.

Hi scottishpoet. Thanks for your reply. I have advised them to log a call with their local IBM office. Thought I'd get a head start. If we look at the dbspaces in the admin tool, they are all online and there are no log entries for any other dbspace. The application that they are working on is running correctly (They are editing records etc) but we just cant get the procedure that gets called via ODBC to work again.. However, Im not convinced that the dbspace is working correctly. The physdbs that went down is the dbspace for the physical logs. The stored procedure works with sblobs which uses a sbspace which has logging set to on. If the physdbs is shown as online, does it mean that it definitely is ok. Also is there no job that runs on aix that may be blocking the ODBC functionality ? I also notice that the Shared memory segment could not be forced resident on startup. Thanks Gary