Informix crashing without a message

Hi, I had recently an event when informix server crashed without writing in online.log or OS logs IBM Informix Dynamic Server Version 12.10.FC7X9AEE -- On-Line -- Up 1 days 07:45:44 -- 10595476 Kbytes What I've never seen or noticed is the "openp free list override 1024" line at informix startup Some lines from online.log: 23:50:34 Maximum server connections 50 23:50:34 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 887, Llog used 4849 23:55:34 Checkpoint Completed: duration was 0 seconds. 23:55:34 Fri Jul 14 - loguniq 342736, logpos 0x16cdc018, timestamp: 0x1773d15f Interval: 236586 23:55:34 Maximum server connections 50 23:55:34 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 863, Llog used 4628 Sat Jul 15 00:00:34 2017 00:00:34 Checkpoint Completed: duration was 0 seconds. 00:00:34 Sat Jul 15 - loguniq 342736, logpos 0x17f09018, timestamp: 0x177435e0 Interval: 236587 00:00:34 Maximum server connections 50 00:00:34 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 1081, Llog used 4653 01:54:30 IBM Informix Dynamic Server Started. 01:54:30 openp free list override 1024 01:54:30 Requested shared memory segment size rounded from 4308KB to 4796KB Mon Jul 24 01:54:32 2017 01:54:32 Requested shared memory segment size rounded from 22459KB to 22460KB 01:54:32 Successfully added a bufferpool of page size 2K. 01:54:32 Requested shared memory segment size rounded from 82459KB to 82460KB 01:54:32 Successfully added a bufferpool of page size 8K. 01:54:33 Event alarms enabled. ALARMPROG = '/opt/IBM/informix_FC7/etc/alarmprogram.sh' 01:54:33 Booting Language <c> from module <> 01:54:33 Loading Module <CNULL> 01:54:33 Booting Language <builtin> from module <> 01:54:33 Loading Module <BUILTINNULL> 01:54:39 CCFLAGS2 value set to 0x200 01:54:39 SQL_FEAT_CTRL value set to 0x8008 01:54:39 SQL_DEF_CTRL value set to 0x4b0 01:54:39 DR: DRAUTO is 0 (Off) 01:54:39 DR: ENCRYPT_HDR is 0 (HDR encryption Disabled) 01:54:39 Event notification facility epoll enabled. 01:54:40 IBM Informix Dynamic Server Version 12.10.FC7X9AEE Software Serial Number AAA#B000000 01:54:45 IBM Informix Dynamic Server Initialized -- Shared Memory Initialized. 01:54:45 Started 1 B-tree scanners. 01:54:45 B-tree scanner threshold set at 5000. 01:54:45 B-tree scanner range scan size set to -1. 01:54:45 B-tree scanner ALICE mode set to 6. 01:54:45 B-tree scanner index compression level set to med. 01:54:45 Physical Recovery Started at Page (3:137088). 01:54:45 Physical Recovery Complete: 387 Pages Examined, 99 Pages Restored. 01:54:46 Logical Recovery Started. 01:54:46 24 recovery worker threads will be started. 01:54:51 Logical Recovery has reached the transaction cleanup phase. 01:54:51 Logical Recovery Complete. 1288 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks 01:54:52 Onconfig parameter RAS_LLOG_SPEED modified from 6119 to 3172. 01:54:52 Dataskip is now OFF for all dbspaces 01:54:53 Checkpoint Completed: duration was 0 seconds. 01:54:53 Mon Jul 24 - loguniq 342736, logpos 0x1840c0c0, timestamp: 0x177451a4 Interval: 236588 01:54:53 Maximum server connections 0 01:54:53 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, Plog used 489, Llog used 1 01:54:53 On-Line Mode 01:54:54 SCHAPI: Started dbScheduler thread. 01:54:55 Booting Language <spl> from module <> 01:54:55 Loading Module <SPLNULL> 01:54:55 Auto Registration is synced 01:54:55 SCHAPI: Started 2 dbWorker threads. 01:54:56 SCHAPI: last statement aus_refresh_stats(integer,integer) 01:54:56 SCHAPI: [Auto Update Statistics Refresh 41-236] Error -206 The specified table (informix.aus_cmd_info) is not in the database. 01:54:56 SCHAPI: [Auto Update Statistics Refresh 41-236] Error -111 ISAM error: no record found. 01:54:57 SCHAPI: [mongo_pam_auth 51-57] Error -217 Column (id) not found in any table in the query (or SLV is undefined). 01:54:57 Defra01:54:57 Defragmenter cleaner thread cleaned:0 partitions 01:54:59 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') 01:54:59 SCHAPI: [db_purge_tables 38-120873] Error -211 Cannot read system catalog (sysprocedures). 01:54:59 SCHAPI: [db_purge_tables 38-120873] Error -103 ISAM error: illegal key descriptor (too many parts or too long). 01:55:00 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') 01:55:00 SCHAPI: [db_purge_tables 38-120874] Error -211 Cannot read system catalog (sysprocedures). 01:55:00 SCHAPI: [db_purge_tables 38-120874] Error -103 ISAM error: illegal key descriptor (too many parts or too long). 01:55:10 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') 01:55:10 SCHAPI: [db_purge_tables 38-120876] Error -211 Cannot read system catalog (sysprocedures). 01:55:10 SCHAPI: [db_purge_tables 38-120876] Error -103 ISAM error: illegal key descriptor (too many parts or too long). 01:55:46 Logical Log 342736 Complete, timestamp: 0x1785167b. 01:55:48 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') 01:55:48 SCHAPI: [db_purge_tables 38-120878] Error -211 Cannot read system catalog (sysprocedures). 01:55:48 SCHAPI: [db_purge_tables 38-120878] Error -103 ISAM error: illegal key descriptor (too many parts or too long). 01:55:55 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') 01:55:55 SCHAPI: [db_purge_tables 38-120879] Error -211 Cannot read system catalog (sysprocedures). 01:55:55 SCHAPI: [db_purge_tables 38-120879] Error -103 ISAM error: illegal key descriptor (too many parts or too long). 01:55:56 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') 01:55:56 SCHAPI: [db_purge_tables 38-120880] Error -211 Cannot read system catalog (sysprocedures). 01:55:56 SCHAPI: [db_purge_tables 38-120880] Error -103 ISAM error: illegal key descriptor (too many parts or too long). 01:58:47 SCHAPI Estimate Compression for sysadmin:"informix".ph_run started 01:58:47 SCHAPI Estimate succeeded for table 'sysadmin:"informix".ph_run' partnum 1000c2. 01:58:47 admin_fragment_command('fragment estimate_compression ','1048770') succeeded 01:58:47 SCHAPI Estimate Compression for sysadmin:"informix".ix_ph_run_01 started 01:58:47 SCHAPI Estimate succeeded for index 'sysadmin:"informix".ix_ph_run_01' partnum 1000c3. 01:58:47 admin_fragment_command('fragment estimate_compression ','1048771') succeeded 01:58:47 SCHAPI Estimate Compression for sysadmin:"informix".ix_ph_run_02 started 01:58:47 SCHAPI Estimate succeeded for index 'sysadmin:"informix".ix_ph_run_02' partnum 1000c4. 01:58:47 admin_fragment_command('fragment estimate_compression ','1048772') succeeded 01:58:47 SCHAPI Estimate Compression for sysadmin:"informix".ix_ph_run_03 started 01:58:47 SCHAPI Estimate succeeded for index 'sysadmin:"informix".ix_ph_run_03' partnum 1000c5.gmenter cleaner thread now running

HI, 1. Version 12.10.FC7X9AEE - that would be a special build just for yhour organisation, what is different in that build? 2. Run "onstat -g env" Do you see any non-standard environment variables? 3. Run "onstat -c" - Do you see any non-standard onconfig settings? Otherwise open a PMR with IBM. Regards, David. > On 25 July 2017 at 15:45 R MICHAEL <mboncalo@gmail.com> wrote: > > > Hi, > I had recently an event when informix server crashed without writing in > online.log or OS logs > > IBM Informix Dynamic Server Version 12.10.FC7X9AEE -- On-Line -- Up 1 days > 07:45:44 -- 10595476 Kbytes> > What I've never seen or noticed is the "openp free list override 1024" line at > informix startup > Some lines from online.log: > > 23:50:34 Maximum server connections 50 > 23:50:34 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, > Plog used 887, Llog used 4849 > > 23:55:34 Checkpoint Completed: duration was 0 seconds. > 23:55:34 Fri Jul 14 - loguniq 342736, logpos 0x16cdc018, timestamp: 0x1773d15f > Interval: 236586 > > 23:55:34 Maximum server connections 50 > 23:55:34 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, > Plog used 863, Llog used 4628 > > Sat Jul 15 00:00:34 2017 > > 00:00:34 Checkpoint Completed: duration was 0 seconds. > 00:00:34 Sat Jul 15 - loguniq 342736, logpos 0x17f09018, timestamp: 0x177435e0 > Interval: 236587 > > 00:00:34 Maximum server connections 50 > 00:00:34 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, > Plog used 1081, Llog used 4653 > > 01:54:30 IBM Informix Dynamic Server Started. > 01:54:30 openp free list override 1024 > 01:54:30 Requested shared memory segment size rounded from 4308KB to 4796KB > > Mon Jul 24 01:54:32 2017 > > 01:54:32 Requested shared memory segment size rounded from 22459KB to 22460KB > 01:54:32 Successfully added a bufferpool of page size 2K. > > 01:54:32 Requested shared memory segment size rounded from 82459KB to 82460KB > 01:54:32 Successfully added a bufferpool of page size 8K. > > 01:54:33 Event alarms enabled. ALARMPROG = > '/opt/IBM/informix_FC7/etc/alarmprogram.sh' > 01:54:33 Booting Language <c> from module <> > 01:54:33 Loading Module <CNULL> > 01:54:33 Booting Language <builtin> from module <> > 01:54:33 Loading Module <BUILTINNULL> > 01:54:39 CCFLAGS2 value set to 0x200 > 01:54:39 SQL_FEAT_CTRL value set to 0x8008 > 01:54:39 SQL_DEF_CTRL value set to 0x4b0 > 01:54:39 DR: DRAUTO is 0 (Off) > 01:54:39 DR: ENCRYPT_HDR is 0 (HDR encryption Disabled) > 01:54:39 Event notification facility epoll enabled. > 01:54:40 IBM Informix Dynamic Server Version 12.10.FC7X9AEE Software Serial > Number AAA#B000000 > 01:54:45 IBM Informix Dynamic Server Initialized -- Shared Memory Initialized. > > 01:54:45 Started 1 B-tree scanners. > 01:54:45 B-tree scanner threshold set at 5000. > 01:54:45 B-tree scanner range scan size set to -1. > 01:54:45 B-tree scanner ALICE mode set to 6. > 01:54:45 B-tree scanner index compression level set to med. > 01:54:45 Physical Recovery Started at Page (3:137088). > 01:54:45 Physical Recovery Complete: 387 Pages Examined, 99 Pages Restored. > 01:54:46 Logical Recovery Started. > 01:54:46 24 recovery worker threads will be started. > 01:54:51 Logical Recovery has reached the transaction cleanup phase. > 01:54:51 Logical Recovery Complete. > > 1288 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks > > 01:54:52 Onconfig parameter RAS_LLOG_SPEED modified from 6119 to 3172. > 01:54:52 Dataskip is now OFF for all dbspaces > 01:54:53 Checkpoint Completed: duration was 0 seconds. > 01:54:53 Mon Jul 24 - loguniq 342736, logpos 0x1840c0c0, timestamp: 0x177451a4 > Interval: 236588 > > 01:54:53 Maximum server connections 0 > 01:54:53 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked 0, > Plog used 489, Llog used 1 > > 01:54:53 On-Line Mode > 01:54:54 SCHAPI: Started dbScheduler thread. > 01:54:55 Booting Language <spl> from module <> > 01:54:55 Loading Module <SPLNULL> > 01:54:55 Auto Registration is synced > 01:54:55 SCHAPI: Started 2 dbWorker threads. > 01:54:56 SCHAPI: last statement aus_refresh_stats(integer,integer) > 01:54:56 SCHAPI: [Auto Update Statistics Refresh 41-236] Error -206 The > specified table (informix.aus_cmd_info) is not in the database. > > 01:54:56 SCHAPI: [Auto Update Statistics Refresh 41-236] Error -111 ISAM > error: no record found. > > 01:54:57 SCHAPI: [mongo_pam_auth 51-57] Error -217 Column (id) not found in > any table in the query (or SLV is undefined). > > 01:54:57 Defra01:54:57 Defragmenter cleaner thread cleaned:0 partitions > 01:54:59 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > 01:54:59 SCHAPI: [db_purge_tables 38-120873] Error -211 Cannot read system > catalog (sysprocedures). > > 01:54:59 SCHAPI: [db_purge_tables 38-120873] Error -103 ISAM error: illegal > key descriptor (too many parts or too long). > > 01:55:00 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > 01:55:00 SCHAPI: [db_purge_tables 38-120874] Error -211 Cannot read system > catalog (sysprocedures). > > 01:55:00 SCHAPI: [db_purge_tables 38-120874] Error -103 ISAM error: illegal > key descriptor (too many parts or too long). > > 01:55:10 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > 01:55:10 SCHAPI: [db_purge_tables 38-120876] Error -211 Cannot read system > catalog (sysprocedures). > > 01:55:10 SCHAPI: [db_purge_tables 38-120876] Error -103 ISAM error: illegal > key descriptor (too many parts or too long). > > 01:55:46 Logical Log 342736 Complete, timestamp: 0x1785167b. > 01:55:48 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > 01:55:48 SCHAPI: [db_purge_tables 38-120878] Error -211 Cannot read system > catalog (sysprocedures). > > 01:55:48 SCHAPI: [db_purge_tables 38-120878] Error -103 ISAM error: illegal > key descriptor (too many parts or too long). > > 01:55:55 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > 01:55:55 SCHAPI: [db_purge_tables 38-120879] Error -211 Cannot read system > catalog (sysprocedures). > > 01:55:55 SCHAPI: [db_purge_tables 38-120879] Error -103 ISAM error: illegal > key descriptor (too many parts or too long). > > 01:55:56 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > 01:55:56 SCHAPI: [db_purge_tables 38-120880] Error -211 Cannot read system > catalog (sysprocedures). > > 01:55:56 SCHAPI: [db_purge_tables 38-120880] Error -103 ISAM error: illegal > key descriptor (too many parts or too long). > > 01:58:47 SCHAPI Estimate Compression for sysadmin:"informix".ph_run started > 01:58:47 SCHAPI Estimate succeeded for table 'sysadmin:"informix".ph_run' > partnum 1000c2. > 01:58:47 admin_fragment_command('fragment estimate_compression ','1048770') > succeeded > 01:58:47 SCHAPI Estimate Compression for sysadmin:"informix".ix_ph_run_01 > started > 01:58:47 SCHAPI Estimate succeeded for index > 'sysadmin:"informix".ix_ph_run_01' partnum 1000c3. > 01:58:47

Don't know about the "openp" message but I am worried about all of the messages your "db_purge_tables" procedure is getting about not being able to access sysprocedures and "illegal key descriptor". These indicate possible catalog corruption. I would oncheck -cc all of your databases (including the system databases) and see what's going on. Art Art S. Kagel, President and Principal Consultant ASK Database Management www.askdbmgt.com Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Tue, Jul 25, 2017 at 10:45 AM, R MICHAEL <mboncalo@gmail.com> wrote: > Hi, > I had recently an event when informix server crashed without writing in > online.log or OS logs > > IBM Informix Dynamic Server Version 12.10.FC7X9AEE -- On-Line -- Up 1 days > 07:45:44 -- 10595476 Kbytes> > What I've never seen or noticed is the "openp free list override 1024" > line at > informix startup > Some lines from online.log: > > 23:50:34 Maximum server connections 50 > 23:50:34 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, > Plog used 887, Llog used 4849 > > 23:55:34 Checkpoint Completed: duration was 0 seconds. > 23:55:34 Fri Jul 14 - loguniq 342736, logpos 0x16cdc018, timestamp: > 0x1773d15f > Interval: 236586 > > 23:55:34 Maximum server connections 50 > 23:55:34 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, > Plog used 863, Llog used 4628 > > Sat Jul 15 00:00:34 2017 > > 00:00:34 Checkpoint Completed: duration was 0 seconds. > 00:00:34 Sat Jul 15 - loguniq 342736, logpos 0x17f09018, timestamp: > 0x177435e0 > Interval: 236587 > > 00:00:34 Maximum server connections 50 > 00:00:34 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, > Plog used 1081, Llog used 4653 > > 01:54:30 IBM Informix Dynamic Server Started. > 01:54:30 openp free list override 1024 > 01:54:30 Requested shared memory segment size rounded from 4308KB to 4796KB > > Mon Jul 24 01:54:32 2017 > > 01:54:32 Requested shared memory segment size rounded from 22459KB to > 22460KB > 01:54:32 Successfully added a bufferpool of page size 2K. > > 01:54:32 Requested shared memory segment size rounded from 82459KB to > 82460KB > 01:54:32 Successfully added a bufferpool of page size 8K. > > 01:54:33 Event alarms enabled. ALARMPROG = > '/opt/IBM/informix_FC7/etc/alarmprogram.sh' > 01:54:33 Booting Language <c> from module <> > 01:54:33 Loading Module <CNULL> > 01:54:33 Booting Language <builtin> from module <> > 01:54:33 Loading Module <BUILTINNULL> > 01:54:39 CCFLAGS2 value set to 0x200 > 01:54:39 SQL_FEAT_CTRL value set to 0x8008 > 01:54:39 SQL_DEF_CTRL value set to 0x4b0 > 01:54:39 DR: DRAUTO is 0 (Off) > 01:54:39 DR: ENCRYPT_HDR is 0 (HDR encryption Disabled) > 01:54:39 Event notification facility epoll enabled. > 01:54:40 IBM Informix Dynamic Server Version 12.10.FC7X9AEE Software Serial > Number AAA#B000000 > 01:54:45 IBM Informix Dynamic Server Initialized -- Shared Memory > Initialized. > > 01:54:45 Started 1 B-tree scanners. > 01:54:45 B-tree scanner threshold set at 5000. > 01:54:45 B-tree scanner range scan size set to -1. > 01:54:45 B-tree scanner ALICE mode set to 6. > 01:54:45 B-tree scanner index compression level set to med. > 01:54:45 Physical Recovery Started at Page (3:137088). > 01:54:45 Physical Recovery Complete: 387 Pages Examined, 99 Pages Restored. > 01:54:46 Logical Recovery Started. > 01:54:46 24 recovery worker threads will be started. > 01:54:51 Logical Recovery has reached the transaction cleanup phase. > 01:54:51 Logical Recovery Complete. > > 1288 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks > > 01:54:52 Onconfig parameter RAS_LLOG_SPEED modified from 6119 to 3172. > 01:54:52 Dataskip is now OFF for all dbspaces > 01:54:53 Checkpoint Completed: duration was 0 seconds. > 01:54:53 Mon Jul 24 - loguniq 342736, logpos 0x1840c0c0, timestamp: > 0x177451a4 > Interval: 236588 > > 01:54:53 Maximum server connections 0 > 01:54:53 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, > Plog used 489, Llog used 1 > > 01:54:53 On-Line Mode > 01:54:54 SCHAPI: Started dbScheduler thread. > 01:54:55 Booting Language <spl> from module <> > 01:54:55 Loading Module <SPLNULL> > 01:54:55 Auto Registration is synced > 01:54:55 SCHAPI: Started 2 dbWorker threads. > 01:54:56 SCHAPI: last statement aus_refresh_stats(integer,integer) > 01:54:56 SCHAPI: [Auto Update Statistics Refresh 41-236] Error -206 The > specified table (informix.aus_cmd_info) is not in the database. > > 01:54:56 SCHAPI: [Auto Update Statistics Refresh 41-236] Error -111 ISAM > error: no record found. > > 01:54:57 SCHAPI: [mongo_pam_auth 51-57] Error -217 Column (id) not found in > any table in the query (or SLV is undefined). > > 01:54:57 Defra01:54:57 Defragmenter cleaner thread cleaned:0 partitions > 01:54:59 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > 01:54:59 SCHAPI: [db_purge_tables 38-120873] Error -211 Cannot read system > catalog (sysprocedures). > > 01:54:59 SCHAPI: [db_purge_tables 38-120873] Error -103 ISAM error: illegal > key descriptor (too many parts or too long). > > 01:55:00 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > 01:55:00 SCHAPI: [db_purge_tables 38-120874] Error -211 Cannot read system > catalog (sysprocedures). > > 01:55:00 SCHAPI: [db_purge_tables 38-120874] Error -103 ISAM error: illegal > key descriptor (too many parts or too long). > > 01:55:10 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > 01:55:10 SCHAPI: [db_purge_tables 38-120876] Error -211 Cannot read system > catalog (sysprocedures). > > 01:55:10 SCHAPI: [db_purge_tables 38-120876] Error -103 ISAM error: illegal > key descriptor (too many parts or too long). > > 01:55:46 Logical Log 342736 Complete, timestamp: 0x1785167b. > 01:55:48 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > 01:55:48 SCHAPI: [db_purge_tables 38-120878] Error -211 Cannot read system > catalog (sysprocedures). > > 01:55:48 SCHAPI: [db_purge_tables 38-120878] Error -103 ISAM error: illegal > key descriptor (too many parts or too long). > > 01:55:55 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > 01:55:55 SCHAPI: [db_purge_tables 38-120879] Error -211 Cannot read system > catalog (sysprocedures). > > 01:55:55 SCHAPI: [db_purge_tables 38-120879] Error -103 ISAM error: illegal > key descriptor (too many parts or too long). > > 01:55:56 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > 01:55:56 SCHAPI: [db_purge_tables 38-120880] Error -211 Cannot read system > catalog (sysprocedures). > > 01:55:56 SCHAPI: [db_purge_tables 38-120880] Error -103 ISAM error:

Stop the scheduler with execute function task ("scheduler stop"); Then run oncheck -cR oncheck -ce oncheck -cc oncheck -cID sysmaster oncheck -cID sysadmin then start the scheduler execute function task ("scheduler start"); Regards, David. > On 25 July 2017 at 16:54 Art Kagel <art.kagel@gmail.com> wrote: > > > Don't know about the "openp" message but I am worried about all of the > messages your "db_purge_tables" procedure is getting about not being able > to access sysprocedures and "illegal key descriptor". These indicate > possible catalog corruption. I would oncheck -cc all of your databases > (including the system databases) and see what's going on. > > Art > > Art S. Kagel, President and Principal Consultant > ASK Database Management > www.askdbmgt.com > > Blog: http://informix-myview.blogspot.com/ > > Disclaimer: Please keep in mind that my own opinions are my own opinions > and do not reflect on the IIUG, nor any other organization with which I am > associated either explicitly, implicitly, or by inference. Neither do > those opinions reflect those of other individuals affiliated with any > entity with which I am affiliated nor those of the entities themselves. > > On Tue, Jul 25, 2017 at 10:45 AM, R MICHAEL <mboncalo@gmail.com> wrote: > > > Hi, > > I had recently an event when informix server crashed without writing in > > online.log or OS logs > > > > IBM Informix Dynamic Server Version 12.10.FC7X9AEE -- On-Line -- Up 1 days > > 07:45:44 -- 10595476 Kbytes> > > > What I've never seen or noticed is the "openp free list override 1024" > > line at > > informix startup > > Some lines from online.log: > > > > 23:50:34 Maximum server connections 50 > > 23:50:34 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > > 0, > > Plog used 887, Llog used 4849 > > > > 23:55:34 Checkpoint Completed: duration was 0 seconds. > > 23:55:34 Fri Jul 14 - loguniq 342736, logpos 0x16cdc018, timestamp: > > 0x1773d15f > > Interval: 236586 > > > > 23:55:34 Maximum server connections 50 > > 23:55:34 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > > 0, > > Plog used 863, Llog used 4628 > > > > Sat Jul 15 00:00:34 2017 > > > > 00:00:34 Checkpoint Completed: duration was 0 seconds. > > 00:00:34 Sat Jul 15 - loguniq 342736, logpos 0x17f09018, timestamp: > > 0x177435e0 > > Interval: 236587 > > > > 00:00:34 Maximum server connections 50 > > 00:00:34 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > > 0, > > Plog used 1081, Llog used 4653 > > > > 01:54:30 IBM Informix Dynamic Server Started. > > 01:54:30 openp free list override 1024 > > 01:54:30 Requested shared memory segment size rounded from 4308KB to 4796KB > > > > Mon Jul 24 01:54:32 2017 > > > > 01:54:32 Requested shared memory segment size rounded from 22459KB to > > 22460KB > > 01:54:32 Successfully added a bufferpool of page size 2K. > > > > 01:54:32 Requested shared memory segment size rounded from 82459KB to > > 82460KB > > 01:54:32 Successfully added a bufferpool of page size 8K. > > > > 01:54:33 Event alarms enabled. ALARMPROG = > > '/opt/IBM/informix_FC7/etc/alarmprogram.sh' > > 01:54:33 Booting Language <c> from module <> > > 01:54:33 Loading Module <CNULL> > > 01:54:33 Booting Language <builtin> from module <> > > 01:54:33 Loading Module <BUILTINNULL> > > 01:54:39 CCFLAGS2 value set to 0x200 > > 01:54:39 SQL_FEAT_CTRL value set to 0x8008 > > 01:54:39 SQL_DEF_CTRL value set to 0x4b0 > > 01:54:39 DR: DRAUTO is 0 (Off) > > 01:54:39 DR: ENCRYPT_HDR is 0 (HDR encryption Disabled) > > 01:54:39 Event notification facility epoll enabled. > > 01:54:40 IBM Informix Dynamic Server Version 12.10.FC7X9AEE Software Serial > > Number AAA#B000000 > > 01:54:45 IBM Informix Dynamic Server Initialized -- Shared Memory > > Initialized. > > > > 01:54:45 Started 1 B-tree scanners. > > 01:54:45 B-tree scanner threshold set at 5000. > > 01:54:45 B-tree scanner range scan size set to -1. > > 01:54:45 B-tree scanner ALICE mode set to 6. > > 01:54:45 B-tree scanner index compression level set to med. > > 01:54:45 Physical Recovery Started at Page (3:137088). > > 01:54:45 Physical Recovery Complete: 387 Pages Examined, 99 Pages Restored. > > 01:54:46 Logical Recovery Started. > > 01:54:46 24 recovery worker threads will be started. > > 01:54:51 Logical Recovery has reached the transaction cleanup phase. > > 01:54:51 Logical Recovery Complete. > > > > 1288 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks > > > > 01:54:52 Onconfig parameter RAS_LLOG_SPEED modified from 6119 to 3172. > > 01:54:52 Dataskip is now OFF for all dbspaces > > 01:54:53 Checkpoint Completed: duration was 0 seconds. > > 01:54:53 Mon Jul 24 - loguniq 342736, logpos 0x1840c0c0, timestamp: > > 0x177451a4 > > Interval: 236588 > > > > 01:54:53 Maximum server connections 0 > > 01:54:53 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > > 0, > > Plog used 489, Llog used 1 > > > > 01:54:53 On-Line Mode > > 01:54:54 SCHAPI: Started dbScheduler thread. > > 01:54:55 Booting Language <spl> from module <> > > 01:54:55 Loading Module <SPLNULL> > > 01:54:55 Auto Registration is synced > > 01:54:55 SCHAPI: Started 2 dbWorker threads. > > 01:54:56 SCHAPI: last statement aus_refresh_stats(integer,integer) > > 01:54:56 SCHAPI: [Auto Update Statistics Refresh 41-236] Error -206 The > > specified table (informix.aus_cmd_info) is not in the database. > > > > 01:54:56 SCHAPI: [Auto Update Statistics Refresh 41-236] Error -111 ISAM > > error: no record found. > > > > 01:54:57 SCHAPI: [mongo_pam_auth 51-57] Error -217 Column (id) not found in > > any table in the query (or SLV is undefined). > > > > 01:54:57 Defra01:54:57 Defragmenter cleaner thread cleaned:0 partitions > > 01:54:59 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > > 01:54:59 SCHAPI: [db_purge_tables 38-120873] Error -211 Cannot read system > > catalog (sysprocedures). > > > > 01:54:59 SCHAPI: [db_purge_tables 38-120873] Error -103 ISAM error: illegal > > key descriptor (too many parts or too long). > > > > 01:55:00 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > > 01:55:00 SCHAPI: [db_purge_tables 38-120874] Error -211 Cannot read system > > catalog (sysprocedures). > > > > 01:55:00 SCHAPI: [db_purge_tables 38-120874] Error -103 ISAM error: illegal > > key descriptor (too many parts or too long). > > > > 01:55:10 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > > 01:55:10 SCHAPI: [db_purge_tables 38-120876] Error -211 Cannot read system > > catalog (sysprocedures). > > > > 01:55:10 SCHAPI: [db_purge_tables 38-120876] Error -103 ISAM error: illegal > > key descriptor (too many parts or too long). > > > > 01:55:46 Logical Log 342736 Complete, timestamp: 0x1785167b. > > 01:55:48 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > > 01:55:48 SCHAPI: [db_purge_tables 38-120878] Error -211 Cannot read system > > catalog (sysprocedures). >

Linux ? If so , check with dmesg command or in /var/log/messages or "journalctl -xb" if the **oom_killer** not kill them.. 2017-07-25 11:45 GMT-03:00 R MICHAEL <mboncalo@gmail.com>: > Hi, > I had recently an event when informix server crashed without writing in > online.log or OS logs > > IBM Informix Dynamic Server Version 12.10.FC7X9AEE -- On-Line -- Up 1 days > 07:45:44 -- 10595476 Kbytes> > What I've never seen or noticed is the "openp free list override 1024" > line at > informix startup > Some lines from online.log: > > 23:50:34 Maximum server connections 50 > 23:50:34 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, > Plog used 887, Llog used 4849 > > 23:55:34 Checkpoint Completed: duration was 0 seconds. > 23:55:34 Fri Jul 14 - loguniq 342736, logpos 0x16cdc018, timestamp: > 0x1773d15f > Interval: 236586 > > 23:55:34 Maximum server connections 50 > 23:55:34 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, > Plog used 863, Llog used 4628 > > Sat Jul 15 00:00:34 2017 > > 00:00:34 Checkpoint Completed: duration was 0 seconds. > 00:00:34 Sat Jul 15 - loguniq 342736, logpos 0x17f09018, timestamp: > 0x177435e0 > Interval: 236587 > > 00:00:34 Maximum server connections 50 > 00:00:34 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, > Plog used 1081, Llog used 4653 > > 01:54:30 IBM Informix Dynamic Server Started. > 01:54:30 openp free list override 1024 > 01:54:30 Requested shared memory segment size rounded from 4308KB to 4796KB > > Mon Jul 24 01:54:32 2017 > > 01:54:32 Requested shared memory segment size rounded from 22459KB to > 22460KB > 01:54:32 Successfully added a bufferpool of page size 2K. > > 01:54:32 Requested shared memory segment size rounded from 82459KB to > 82460KB > 01:54:32 Successfully added a bufferpool of page size 8K. > > 01:54:33 Event alarms enabled. ALARMPROG = > '/opt/IBM/informix_FC7/etc/alarmprogram.sh' > 01:54:33 Booting Language <c> from module <> > 01:54:33 Loading Module <CNULL> > 01:54:33 Booting Language <builtin> from module <> > 01:54:33 Loading Module <BUILTINNULL> > 01:54:39 CCFLAGS2 value set to 0x200 > 01:54:39 SQL_FEAT_CTRL value set to 0x8008 > 01:54:39 SQL_DEF_CTRL value set to 0x4b0 > 01:54:39 DR: DRAUTO is 0 (Off) > 01:54:39 DR: ENCRYPT_HDR is 0 (HDR encryption Disabled) > 01:54:39 Event notification facility epoll enabled. > 01:54:40 IBM Informix Dynamic Server Version 12.10.FC7X9AEE Software Serial > Number AAA#B000000 > 01:54:45 IBM Informix Dynamic Server Initialized -- Shared Memory > Initialized. > > 01:54:45 Started 1 B-tree scanners. > 01:54:45 B-tree scanner threshold set at 5000. > 01:54:45 B-tree scanner range scan size set to -1. > 01:54:45 B-tree scanner ALICE mode set to 6. > 01:54:45 B-tree scanner index compression level set to med. > 01:54:45 Physical Recovery Started at Page (3:137088). > 01:54:45 Physical Recovery Complete: 387 Pages Examined, 99 Pages Restored. > 01:54:46 Logical Recovery Started. > 01:54:46 24 recovery worker threads will be started. > 01:54:51 Logical Recovery has reached the transaction cleanup phase. > 01:54:51 Logical Recovery Complete. > > 1288 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks > > 01:54:52 Onconfig parameter RAS_LLOG_SPEED modified from 6119 to 3172. > 01:54:52 Dataskip is now OFF for all dbspaces > 01:54:53 Checkpoint Completed: duration was 0 seconds. > 01:54:53 Mon Jul 24 - loguniq 342736, logpos 0x1840c0c0, timestamp: > 0x177451a4 > Interval: 236588 > > 01:54:53 Maximum server connections 0 > 01:54:53 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked > 0, > Plog used 489, Llog used 1 > > 01:54:53 On-Line Mode > 01:54:54 SCHAPI: Started dbScheduler thread. > 01:54:55 Booting Language <spl> from module <> > 01:54:55 Loading Module <SPLNULL> > 01:54:55 Auto Registration is synced > 01:54:55 SCHAPI: Started 2 dbWorker threads. > 01:54:56 SCHAPI: last statement aus_refresh_stats(integer,integer) > 01:54:56 SCHAPI: [Auto Update Statistics Refresh 41-236] Error -206 The > specified table (informix.aus_cmd_info) is not in the database. > > 01:54:56 SCHAPI: [Auto Update Statistics Refresh 41-236] Error -111 ISAM > error: no record found. > > 01:54:57 SCHAPI: [mongo_pam_auth 51-57] Error -217 Column (id) not found in > any table in the query (or SLV is undefined). > > 01:54:57 Defra01:54:57 Defragmenter cleaner thread cleaned:0 partitions > 01:54:59 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > 01:54:59 SCHAPI: [db_purge_tables 38-120873] Error -211 Cannot read system > catalog (sysprocedures). > > 01:54:59 SCHAPI: [db_purge_tables 38-120873] Error -103 ISAM error: illegal > key descriptor (too many parts or too long). > > 01:55:00 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > 01:55:00 SCHAPI: [db_purge_tables 38-120874] Error -211 Cannot read system > catalog (sysprocedures). > > 01:55:00 SCHAPI: [db_purge_tables 38-120874] Error -103 ISAM error: illegal > key descriptor (too many parts or too long). > > 01:55:10 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > 01:55:10 SCHAPI: [db_purge_tables 38-120876] Error -211 Cannot read system > catalog (sysprocedures). > > 01:55:10 SCHAPI: [db_purge_tables 38-120876] Error -103 ISAM error: illegal > key descriptor (too many parts or too long). > > 01:55:46 Logical Log 342736 Complete, timestamp: 0x1785167b. > 01:55:48 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > 01:55:48 SCHAPI: [db_purge_tables 38-120878] Error -211 Cannot read system > catalog (sysprocedures). > > 01:55:48 SCHAPI: [db_purge_tables 38-120878] Error -103 ISAM error: illegal > key descriptor (too many parts or too long). > > 01:55:55 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > 01:55:55 SCHAPI: [db_purge_tables 38-120879] Error -211 Cannot read system > catalog (sysprocedures). > > 01:55:55 SCHAPI: [db_purge_tables 38-120879] Error -103 ISAM error: illegal > key descriptor (too many parts or too long). > > 01:55:56 SCHAPI: last statement EXECUTE PROCEDURE IFX_ALLOW_NEWLINE('T') > 01:55:56 SCHAPI: [db_purge_tables 38-120880] Error -211 Cannot read system > catalog (sysprocedures). > > 01:55:56 SCHAPI: [db_purge_tables 38-120880] Error -103 ISAM error: illegal > key descriptor (too many parts or too long). > > 01:58:47 SCHAPI Estimate Compression for sysadmin:"informix".ph_run started > 01:58:47 SCHAPI Estimate succeeded for table 'sysadmin:"informix".ph_run' > partnum 1000c2. > 01:58:47 admin_fragment_command('fragment estimate_compression > ','1048770') > succeeded > 01:58:47 SCHAPI Estimate Compression for sysadmin:"informix".ix_ph_run_01 > started > 01:58:47 SCHAPI Estimate succeeded for index > 'sysadmin:"informix".ix_ph_run_01' partnum 1000c3. > 01:58:47 admin_fragment_command('fragment estimate_compression > ','1048771') > succeeded > 01:58:47 SCHAPI Estimate Compression for sysadmin:"informix".ix_ph_run_02 > started >

@David, I think that build was release due to some functionality issues with client API and nothing is different regarding env variables or settings @Art, yes, I know about those messages, oncheck shows nothing so I think client did some changes there and messed up something but didn't complain about functionality @Cesar I checked that, but nothing can be found in the logs That's why I think it could be related to that openp message. I've read somewere that openp is referring to open processes. I checked and max open processes limit for informix is 1024 but it doesn't make sense for OS to kill informix because it reached the limit and how could informix open more than 1024 processes...so it's a dead end

That "openp free list override <n>", in all likelihood, originates from an env var settting: IFX_NOPENPS=<n> This system probably once had an "open tables" problem, this env var got set to address that problem. I don't think this has anything to do with such sudden vanishing of the database server. In case you're expecting/fearing a reoccurrence and need it explained, you might have a permanent terminal window to this box open somewhere and an 'onstat -i' running there. This onstat process outliving the database server would keep shared memory segments around and then could be used for further analysis, or a shared memory dump. HTH, Andreas From: "R MICHAEL" <mboncalo@gmail.com> To: ids@iiug.org Date: 07/26/2017 06:34 PM Subject: Re: Informix crashing without a message [39629] Sent by: ids-bounces@iiug.org @David, I think that build was release due to some functionality issues with client API and nothing is different regarding env variables or settings @Art, yes, I know about those messages, oncheck shows nothing so I think client did some changes there and messed up something but didn't complain about functionality @Cesar I checked that, but nothing can be found in the logs That's why I think it could be related to that openp message. I've read somewere that openp is referring to open processes. I checked and max open processes limit for informix is 1024 but it doesn't make sense for OS to kill informix because it reached the limit and how could informix open more than 1024 processes...so it's a dead end ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Hi Andreas, Yes, you're right about openp, thank you for clearing that out,it seems there is a variable set in the environment IFX_NOPENPS=1024 but still no clue on what happened..

Original post: Hi, I had recently an event when informix server crashed without writing in online.log or OS logs <lots of stuff removed> Response: The only 2 things that I've seen cause no message in the MSGPATH file would be, a machine crash (so 1 sec the machine was there the next it's not so there's no chance for the server to log anything before it's gone), or the machine was having either a machine wide file descriptor shortage or either a user or process level limit of file descriptors (in this case due to the fact that MSGPATH file is opened, written to, and then closed, if the server is unable to get a file descriptor for the open, we are unable to then write anything to the file and also likely can't create/open the associated af file as well). Jacques