Desperate, sudden cessation of unsupported database

Hi informix community We have a data base that has stopped with the errors below, the only thing that has changed is a new CPU has been added bringing the total to 13 . The first solution proposed has been to quickly migrate to another brand of database. So I'd hate to see an Informix database go. Any help would by highly appreciated Cheers Zanne Fagerstrom IBMGS (Australia) SunOS csun7621 5.6 Generic_105181-35 sun4u sparc SUNW,Ultra-Enterprise-10000 Informix 7.31 ******************************************************************************** * -25578 Network driver cannot disconnect from the network. A system call has failed. See the UNIX system administrator for assistance. ******************************************************************************** * 10:49:46 Logical Recovery Complete. 0 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks 10:49:47 Dataskip is now OFF for all dbspaces 10:49:47 On-Line Mode 10:49:47 Affinitied VP 3 to phys proc 37 10:49:47 Affinitied VP 4 to phys proc 38 10:49:47 Affinitied VP 7 to phys proc 41 10:49:47 Affinitied VP 8 to phys proc 42 10:49:47 Affinitied VP 9 to phys proc 43 10:49:47 Affinitied VP 1 to phys proc 36 10:49:47 Affinitied VP 5 to phys proc 39 10:49:47 Affinitied VP 6 to phys proc 40 10:49:47 Checkpoint Completed: duration was 0 seconds. 11:10:25 Assert Failed: No Exception Handler 11:10:25 Informix Dynamic Server Version 7.31.UC3X1 11:10:25 Who: Session(30, ccm_root@csun7621, 14554, 179292152) Thread(67, sqlexec, aad58b4, 8) File: mtex.c Line: 446 11:10:25 Results: Exception Caught. Type: MT_EX_OS, Context: mem 11:10:25 Action: Please notify Informix Technical Support. 11:10:29 See Also: /tmp/af.42b3970, shmem.42b3970.0 11:10:30 mtex.c, line 446, thread 67, proc id 12355, No Exception Handler. 11:10:30 PANIC: Attempting to bring system down #************************************************************************** # # INFORMIX SOFTWARE, INC. & Continuus Software Corporation # # Title: onconfig.std # Description: INFORMIX-OnLine Configuration Parameters # C/CM Server: rsun7351 # #************************************************************************** # Root Dbspace Configuration ROOTNAME rootdbs # Root dbspace name ROOTPATH /informix_db/rsun7351_root.dbs# Path for device containing root dbspace ROOTOFFSET 0 # Offset of root dbspace into device (Kbytes) ROOTSIZE 115000 # Size of root dbspace (Kbytes) # Disk Mirroring Configuration MIRROR 0 # Mirroring flag (Yes = 1, No = 0) MIRRORPATH # Path for device containing root dbspacemirror MIRROROFFSET 0 # Offset into mirror device (Kbytes) # Physical Log Configuration PHYSDBS rootdbs # Name of dbspace that contains physical log PHYSFILE 4096 # Physical log file size (Kbytes) # Logical Log Configuration LOGFILES 28 # Number of logical log files LOGSIZE 512 # Size of each logical log file (Kbytes) # Message Files MSGPATH /opt/ccm/informix/log/rsun7351.log # OnLine message log pathname CONSOLE /opt/ccm/informix/log/rsun7351.msg # System console message pathname # Archive Tape Device TAPEDEV /opt/ccm/informix/etc/rsun7351.tapedev # Archive tape device pathname TAPEBLK 16 # Archive tape block size (Kbytes) TAPESIZE 10240 # Max amount of data to put on tape (Kbytes) # Logical Log Backup Tape Device LTAPEDEV /dev/null # Logical log tape device pathname LTAPEBLK 16 # Logical log tape block size (Kbytes) LTAPESIZE 10240 # Max amount of data to put on log tape (Kbytes) # Optical STAGEBLOB # INFORMIX-OnLine/Optical staging area # System Configuration SERVERNUM 1 # Unique id associated with this OnLineinstance DBSERVERNAME rsun7351 # Unique name of this OnLine instance DBSERVERALIASES rsun7351_net # List of alternate dbservernames NETTYPE ipcshm,8,5,CPU # Configure poll thread(s) for nettype NETTYPE tlitcp,1,44,NET # Configure poll thread(s) for nettype DEADLOCK_TIMEOUT 60 # Max time to wait for lock in distributed env. RESIDENT 0 # Forced residency flag (Yes = 1, No = 0) MULTIPROCESSOR 1 # 0 for single-processor, 1 formulti-processor NUMCPUVPS 8 # Number of user (cpu) vps SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vps toone NOAGE 0 # Process aging AFF_SPROC 1 # Affinity start processor AFF_NPROCS 8 # Affinity number of processors # Shared Memory Parameters LOCKS 60000 # Maximum number of locks BUFFERS 2560 # Maximum number of shared memory buffers NUMAIOVPS # Number of IO vps PHYSBUFF 64 # Size of physical log buffers (Kbytes) LOGBUFF 512 # Size of logical log buffers (Kbytes)LOGSMAX 32 # Maximum number of logical log files CLEANERS 1 # Number of page-cleaner processes SHMBASE 0xa000000 # Shared memory base address SHMVIRTSIZE 8000 # initial virtual shared memory segmentsize SHMADD 8192 # Size of new shared memory segments (Kbytes) SHMTOTAL 0 # Total shared memory (Kbytes). 0=>unlimited CKPTINTVL 300 # Checkpoint interval (in seconds) LRUS 8 # Number of LRU queues LRU_MAX_DIRTY 60 # LRU modified begin-cleaning limit (percent) LRU_MIN_DIRTY 50 # LRU modified end-cleaning limit (percent) LTXHWM 50 # Long TX high-water mark (percent) LTXEHWM 60 # Long TX exclusive high-water mark (percent) TXTIMEOUT 0x12c # Transaction timeout (in seconds) STACKSIZE 32 # Stack size (Kbytes) # System Page Size # BUFFSIZE - OnLine no longer supports this configuration parameter. # To determine the page size used by OnLine on your platform # see the last line of output from the command, 'onstat -b'. # Machine- and Product-Specific Parameters # Recovery Variables # OFF_RECVRY_THREADS: # Number of parallel worker threads during fast recovery or an offline restore. # ON_RECVRY_THREADS: # Number of parallel worker threads during an online restore. OFF_RECVRY_THREADS 10 # Default number of offline workerthreads ON_RECVRY_THREADS 1 # Default number of online worker threads # Data Replication Variables # DRAUTO: 0 manual, 1 retain type, 2 reverse type DRAUTO 0 # DR automatic switchover DRINTERVAL 30 # DR max time between DR buffer flushes (in sec) DRTIMEOUT 30 # DR network timeout (in sec) DRLOSTFOUND /opt/ccm/informix/etc/dr.lostfound # DR lost+found filepath # CDR Variables CDR_LOGBUFFERS 2048 # size of log reading buffer pool (Kbytes) CDR_EVALTHREADS 1,2 # evaluator threads (per-cpu-vp,additional) CDR_DSLOCKWAIT 5 # DS lockwait timeout (seconds) CDR_QUEUEMEM 4096 # Maximum amount of memory for any CDR queue (Kbytes) # Backup/Restore variables BAR_ACT_LOG /tmp/bar_act.log BAR_MAX_BACKUP 0 BAR_RETRY 1 BAR_NB_XPORT_COUNT 10 BAR_XFER_BUF_SIZE 31 # Read Ahead Variables RA_PAGES # Number of pages to attempt to read ahead RA_THRESHOLD # Number of pages left before next group # DUMP*: # The following parameters control the type of diagnostics information which # is preserved when an unanticipated error condition (assertion failure) occurs # during OnLine operations. # For DUMPSHMEM, DUMPGCORE and DU

Zanne, This failure can have MANY reasons. Something inside the server caused server's internal memory violation. Please, take a look at file /tmp/af.42b3970 and look for two things: 1. Information about session 30 (output of onstat -g ses 30) 2. Information about the stack trace of session 30 (output of onstat -g stk 14554) SQL information about the problematic statement might be of interest to You. Stack information is needed to determine the actual problem ------------------------------------------ Alexey Sonkin Senior Database Administrator > -----Original Message----- > From: Zanne Fager.... [mailto:zfagerst@au1.ibm.com] > Sent: Wednesday, February 25, 2004 8:07 PM > To: ids@iiug.org > Subject: Desperate, sudden cessation of unsupported database [2574] > > > > > > > Hi informix community > > > We have a data base that has stopped with the errors below, the only > thing > that has changed is a new CPU has been added bringing the total to 13 . > The first solution proposed has been to quickly migrate to another brand > of database. So I'd hate to see an Informix database go. > > Any help would by highly appreciated > > Cheers > Zanne Fagerstrom > IBMGS (Australia) > > SunOS csun7621 5.6 Generic_105181-35 sun4u sparc > SUNW,Ultra-Enterprise-10000 > Informix 7.31 > > > ************************************************************************** > ******* > -25578 Network driver cannot disconnect from the network. > > A system call has failed. See the UNIX system administrator for > assistance. > ************************************************************************** > ******* > > 10:49:46 Logical Recovery Complete. > 0 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks > > 10:49:47 Dataskip is now OFF for all dbspaces > 10:49:47 On-Line Mode > 10:49:47 Affinitied VP 3 to phys proc 37 > 10:49:47 Affinitied VP 4 to phys proc 38 > 10:49:47 Affinitied VP 7 to phys proc 41 > 10:49:47 Affinitied VP 8 to phys proc 42 > 10:49:47 Affinitied VP 9 to phys proc 43 > 10:49:47 Affinitied VP 1 to phys proc 36 > 10:49:47 Affinitied VP 5 to phys proc 39 > 10:49:47 Affinitied VP 6 to phys proc 40 > 10:49:47 Checkpoint Completed: duration was 0 seconds. > 11:10:25 Assert Failed: No Exception Handler > 11:10:25 Informix Dynamic Server Version 7.31.UC3X1 > 11:10:25 Who: Session(30, ccm_root@csun7621, 14554, 179292152) > Thread(67, sqlexec, aad58b4, 8) > File: mtex.c Line: 446 > 11:10:25 Results: Exception Caught. Type: MT_EX_OS, Context: mem > 11:10:25 Action: Please notify Informix Technical Support. > 11:10:29 See Also: /tmp/af.42b3970, shmem.42b3970.0 > 11:10:30 mtex.c, line 446, thread 67, proc id 12355, No Exception > Handler. > 11:10:30 PANIC: Attempting to bring system down >

We went through hell and then some with the mtex.c assertion failure. I see you are on 7.31.... we recently patched to 7.31.UC4x4 we received from Informix. It might be a coincidence but since we patched we haven't had that assertion failure. "Zanne Fager...." <zfagerst@au1.ibm .com> To Sent by: ids@iiug.org forum.subscriber@ cc iiug.org Subject Desperate, sudden cessation of 02/25/2004 08:06 unsupported database [2574] PM Hi informix community We have a data base that has stopped with the errors below, the only thing that has changed is a new CPU has been added bringing the total to 13 . The first solution proposed has been to quickly migrate to another brand of database. So I'd hate to see an Informix database go. Any help would by highly appreciated Cheers Zanne Fagerstrom IBMGS (Australia) SunOS csun7621 5.6 Generic_105181-35 sun4u sparc SUNW,Ultra-Enterprise-10000 Informix 7.31 ******************************************************************************** * -25578 Network driver cannot disconnect from the network. A system call has failed. See the UNIX system administrator for assistance. ******************************************************************************** * 10:49:46 Logical Recovery Complete. 0 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks 10:49:47 Dataskip is now OFF for all dbspaces 10:49:47 On-Line Mode 10:49:47 Affinitied VP 3 to phys proc 37 10:49:47 Affinitied VP 4 to phys proc 38 10:49:47 Affinitied VP 7 to phys proc 41 10:49:47 Affinitied VP 8 to phys proc 42 10:49:47 Affinitied VP 9 to phys proc 43 10:49:47 Affinitied VP 1 to phys proc 36 10:49:47 Affinitied VP 5 to phys proc 39 10:49:47 Affinitied VP 6 to phys proc 40 10:49:47 Checkpoint Completed: duration was 0 seconds. 11:10:25 Assert Failed: No Exception Handler 11:10:25 Informix Dynamic Server Version 7.31.UC3X1 11:10:25 Who: Session(30, ccm_root@csun7621, 14554, 179292152) Thread(67, sqlexec, aad58b4, 8) File: mtex.c Line: 446 11:10:25 Results: Exception Caught. Type: MT_EX_OS, Context: mem 11:10:25 Action: Please notify Informix Technical Support. 11:10:29 See Also: /tmp/af.42b3970, shmem.42b3970.0 11:10:30 mtex.c, line 446, thread 67, proc id 12355, No Exception Handler. 11:10:30 PANIC: Attempting to bring system down #************************************************************************** # # INFORMIX SOFTWARE, INC. & Continuus Software Corporation # # Title: onconfig.std # Description: INFORMIX-OnLine Configuration Parameters # C/CM Server: rsun7351 # #************************************************************************** # Root Dbspace Configuration ROOTNAME rootdbs # Root dbspace name ROOTPATH /informix_db/rsun7351_root.dbs# Path for device containing root dbspace ROOTOFFSET 0 # Offset of root dbspace into device (Kbytes) ROOTSIZE 115000 # Size of root dbspace (Kbytes) # Disk Mirroring Configuration MIRROR 0 # Mirroring flag (Yes = 1, No = 0) MIRRORPATH # Path for device containing root dbspacemirror MIRROROFFSET 0 # Offset into mirror device (Kbytes) # Physical Log Configuration PHYSDBS rootdbs # Name of dbspace that contains physical log PHYSFILE 4096 # Physical log file size (Kbytes) # Logical Log Configuration LOGFILES 28 # Number of logical log files LOGSIZE 512 # Size of each logical log file (Kbytes) # Message Files MSGPATH /opt/ccm/informix/log/rsun7351.log # OnLine message log pathname CONSOLE /opt/ccm/informix/log/rsun7351.msg # System console message pathname # Archive Tape Device TAPEDEV /opt/ccm/informix/etc/rsun7351.tapedev # Archive tape device pathname TAPEBLK 16 # Archive tape block size (Kbytes) TAPESIZE 10240 # Max amount of data to put on tape (Kbytes) # Logical Log Backup Tape Device LTAPEDEV /dev/null # Logical log tape device pathname LTAPEBLK 16 # Logical log tape block size (Kbytes) LTAPESIZE 10240 # Max amount of data to put on log tape (Kbytes) # Optical STAGEBLOB # INFORMIX-OnLine/Optical staging area # System Configuration SERVERNUM 1 # Unique id associated with this OnLineinstance DBSERVERNAME rsun7351 # Unique name of this OnLine instance DBSERVERALIASES rsun7351_net # List of alternate dbservernames NETTYPE ipcshm,8,5,CPU # Configure poll thread(s) for nettype NETTYPE tlitcp,1,44,NET # Configure poll thread(s) for nettype DEADLOCK_TIMEOUT 60 # Max time to wait for lock in distributed env. RESIDENT 0 # Forced residency flag (Yes = 1, No = 0) MULTIPROCESSOR 1 # 0 for single-processor, 1 formulti-processor NUMCPUVPS 8 # Number of user (cpu) vps SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vps toone NOAGE 0 # Process aging AFF_SPROC 1 # Affinity start processor AFF_NPROCS 8 # Affinity number of processors # Shared Memory Parameters LOCKS 60000 # Maximum number of locks BUFFERS 2560 # Maximum number of shared memory buffers NUMAIOVPS # Number of IO vps PHYSBUFF 64 # Size of physical log buffers (Kbytes) LOGBUFF 512 # Size of logical log buffers (Kbytes)LOGSMAX 32 # Maximum number of logical log files CLEANERS 1 # Number of page-cleaner processes SHMBASE 0xa000000 # Shared memory base address SHMVIRTSIZE 8000 # initial virtual shared memory segmentsize SHMADD 8192 # Size of new shared memory segments (Kbytes) SHMTOTAL 0 # Total shared memory (Kbytes). 0=>unlimited CKPTINTVL 300 # Checkpoint interval (in seconds) LRUS 8 # Number of LRU queues LRU_MAX_DIRTY 60 # LRU modified begin-cleaning limit (percent) LRU_MIN_DIRTY 50 # LRU modified end-cleaning limit (percent) LTXHWM 50 # Long TX high-water mark (percent) LTXEHWM 60 # Long TX exclusive high-water mark (percent) TXTIMEOUT 0x12c # Transaction timeout (in seconds) STACKSIZE 32 # Stack size (Kbytes) # System Page Size # BUFFSIZE - OnLine no longer supports this configuration parameter. # To determine the page size used by OnLine on your platform # see the last line of output from the command, 'onstat -b'. # Machine- and Product-Specific Parameters # Recovery Variables # OFF_RECVRY_THREADS: # Number of parallel worker threads during fast recovery or an offline restore. # ON_RECVRY_THREADS: # Number of parallel worker threads during an online restore. OFF_RECVRY_THREADS 10 # Default number of offline workerthreads ON_RECVRY_THREADS 1 # Default number of online worker threads # Data Replication Variables # DRAUTO: 0 manual, 1 retain type, 2 reverse type DRAUTO 0 # DR automatic switchover DRINTERVAL 30 # DR max time between DR buffer flushes (in sec) DRTIMEOUT 30 # DR network timeout (in sec) DRLOSTFOUND /opt/ccm/informix/etc/dr.lostfound # DR lost+found filepath # CDR Variables CDR_LOGBUFFERS 2048 # size of log reading buffer pool (Kbytes) CDR_EVALTHREADS 1,2 # evaluator threads (per-cpu-vp,additional) CDR_DSLOCKWAIT 5 # DS lockwait timeout (seconds) CDR_QUEUEMEM 4096 # Maximum amount of memory for any CDR queue (Kbytes) # Backup/Restore variables BAR_ACT_L