Server fails to come online

All, My DB crashed with an error "THE MAIN DEAMON HAS DIEAD" since then it has refused to come online ... when i do oninit -v i get the following message: Checking group membership to determine server run mode...succeeded Reading configuration file '/home/informix/etc/onconfig.suse'...succeeded Creating /INFORMIXTMP/.infxdirs...succeeded Creating infos file "/home/informix/etc/.infos.suse_ows"...succeeded Linking conf file "/home/informix/etc/.conf.suse_ows"...succeeded Writing to infos file...succeeded Checking config parameters...succeeded Allocating and attaching to shared memory...succeeded Creating resident pool 95490 kbytes...succeeded Allocating 1200016 kbytes for buffer pool of 2K page size...succeeded Initializing rhead structure...succeeded Initializing ASF...succeeded Initializing Dictionary Cache and SPL Routine Cache...succeeded Bringing up ADM VP...succeeded Creating VP classes...succeeded Onlining 0 additional cpu vps...succeeded Onlining 2 IO vps...succeeded Initialization of Encryption...succeeded Forking main_loop thread...succeeded Initializing DR structures...succeeded Forking 1 'soctcp' listener threads...succeeded Forking 1 'ipcshm' listener threads...succeeded Starting tracing...succeeded Initializing 2 flushers...succeeded but when i run onstat -m i get the following message: shared memory not initialized for INFORMIXSERVER 'suse_shm' Message Log File: /home/informix/online.log 13:49:46 Event alarms enabled. ALARMPROG = '/home/informix/etc/alarmprogram.sh' 13:49:46 Booting Language <c> from module <> 13:49:46 Loading Module <CNULL> 13:49:46 Booting Language <builtin> from module <> 13:49:46 Loading Module <BUILTINNULL> 13:49:51 DR: DRAUTO is 0 (Off) 13:49:51 IBM Informix Dynamic Server Version 10.00.UC4 Software Serial Number AAA#B000000 13:49:51 requested number of KAIO events (32768) exceeds limit (16384). using 16384. 13:49:51 listener-thread: err = -25572: oserr = 98: errstr = : Network driver cannot bind a name to the port. System error = 98. 13:49:51 sql_listener: ASF_LISTEN failed 13:49:51 Attempting to bring listener thread down. 13:49:51 Server stopped. 13:49:51 IBM Informix Dynamic Server Stopped. 13:49:51 mt_shm_remove: WARNING: may not have removed all/correct segments Any idea whats happening here? Paul

Hi, Check your sqlhosts file if you have wrong entries for IP services. The bind command fails, which means that the IP listed there does not exist on this machine. Marcus -----Original Message----- From: PAUL GATHOGO [mailto:pgathogo@gmail.com] Sent: Wednesday, April 11, 2007 12:57 PM To: ids@iiug.org Subject: Server fails to come online [8821] All, My DB crashed with an error "THE MAIN DEAMON HAS DIEAD" since then it has refused to come online ... when i do oninit -v i get the following message: Checking group membership to determine server run mode...succeeded Reading configuration file '/home/informix/etc/onconfig.suse'...succeeded Creating /INFORMIXTMP/.infxdirs...succeeded Creating infos file "/home/informix/etc/.infos.suse_ows"...succeeded Linking conf file "/home/informix/etc/.conf.suse_ows"...succeeded Writing to infos file...succeeded Checking config parameters...succeeded Allocating and attaching to shared memory...succeeded Creating resident pool 95490 kbytes...succeeded Allocating 1200016 kbytes for buffer pool of 2K page size...succeeded Initializing rhead structure...succeeded Initializing ASF...succeeded Initializing Dictionary Cache and SPL Routine Cache...succeeded Bringing up ADM VP...succeeded Creating VP classes...succeeded Onlining 0 additional cpu vps...succeeded Onlining 2 IO vps...succeeded Initialization of Encryption...succeeded Forking main_loop thread...succeeded Initializing DR structures...succeeded Forking 1 'soctcp' listener threads...succeeded Forking 1 'ipcshm' listener threads...succeeded Starting tracing...succeeded Initializing 2 flushers...succeeded but when i run onstat -m i get the following message: shared memory not initialized for INFORMIXSERVER 'suse_shm' Message Log File: /home/informix/online.log 13:49:46 Event alarms enabled. ALARMPROG = '/home/informix/etc/alarmprogram.sh' 13:49:46 Booting Language <c> from module <> 13:49:46 Loading Module <CNULL> 13:49:46 Booting Language <builtin> from module <> 13:49:46 Loading Module <BUILTINNULL> 13:49:51 DR: DRAUTO is 0 (Off) 13:49:51 IBM Informix Dynamic Server Version 10.00.UC4 Software Serial Number AAA#B000000 13:49:51 requested number of KAIO events (32768) exceeds limit (16384). using 16384. 13:49:51 listener-thread: err = -25572: oserr = 98: errstr = : Network driver cannot bind a name to the port. System error = 98. 13:49:51 sql_listener: ASF_LISTEN failed 13:49:51 Attempting to bring listener thread down. 13:49:51 Server stopped. 13:49:51 IBM Informix Dynamic Server Stopped. 13:49:51 mt_shm_remove: WARNING: may not have removed all/correct segments Any idea whats happening here? Paul ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum.

On 11/04/07, PAUL GATHOGO <pgathogo@gmail.com> wrote: > All, > > My DB crashed with an error "THE MAIN DEAMON HAS DIEAD" since then it has > refused to come online ... when i do oninit -v i get the following message: > > Checking group membership to determine server run mode...succeeded > Reading configuration file '/home/informix/etc/onconfig.suse'...succeeded > Creating /INFORMIXTMP/.infxdirs...succeeded > Creating infos file "/home/informix/etc/.infos.suse_ows"...succeeded > Linking conf file "/home/informix/etc/.conf.suse_ows"...succeeded > Writing to infos file...succeeded > Checking config parameters...succeeded > Allocating and attaching to shared memory...succeeded > Creating resident pool 95490 kbytes...succeeded > Allocating 1200016 kbytes for buffer pool of 2K page size...succeeded > Initializing rhead structure...succeeded > Initializing ASF...succeeded > Initializing Dictionary Cache and SPL Routine Cache...succeeded > Bringing up ADM VP...succeeded > Creating VP classes...succeeded > Onlining 0 additional cpu vps...succeeded > Onlining 2 IO vps...succeeded > Initialization of Encryption...succeeded > Forking main_loop thread...succeeded > Initializing DR structures...succeeded > Forking 1 'soctcp' listener threads...succeeded > Forking 1 'ipcshm' listener threads...succeeded > Starting tracing...succeeded > Initializing 2 flushers...succeeded > > but when i run onstat -m i get the following message: > shared memory not initialized for INFORMIXSERVER 'suse_shm' > > Message Log File: /home/informix/online.log > > 13:49:46 Event alarms enabled. ALARMPROG = > '/home/informix/etc/alarmprogram.sh' > 13:49:46 Booting Language <c> from module <> > 13:49:46 Loading Module <CNULL> > 13:49:46 Booting Language <builtin> from module <> > 13:49:46 Loading Module <BUILTINNULL> > 13:49:51 DR: DRAUTO is 0 (Off) > 13:49:51 IBM Informix Dynamic Server Version 10.00.UC4 Software Serial Number > AAA#B000000 > 13:49:51 requested number of KAIO events (32768) exceeds limit (16384). using > 16384. > 13:49:51 listener-thread: err = -25572: oserr = 98: errstr = : Network driver > cannot bind a name to the port. > System error = 98. > 13:49:51 sql_listener: ASF_LISTEN failed > > 13:49:51 Attempting to bring listener thread down. > > 13:49:51 Server stopped. > > 13:49:51 IBM Informix Dynamic Server Stopped. > > 13:49:51 mt_shm_remove: WARNING: may not have removed all/correct segments > > Any idea whats happening here? > > Paul > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > Paul You probably have some sharedmemory segments still in existance at the OS level. Run ipcs -ma to see all segents and use ipcrm -m 'memory_id' to remove those belonging to informix (assuming you engine was statred by the informix user). Alternatively reboot the server !! Keith

Keith, ipcs -ma reveals no informix segments! Any ideas?

Probably some oninit process is "holding" port. Look for oninit processes and kill them (or as Keith said - reboot machine) HTH Hrvoje -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of PAUL GATHOGO Sent: Wednesday, April 11, 2007 2:11 PM To: ids@iiug.org Subject: Re: Server fails to come online [8824] Keith, ipcs -ma reveals no informix segments! Any ideas? **************************************************************************** *** Forum Note: Use "Reply" to post a response in the discussion forum.

On 11/04/07, Hrvoje Zokovic <hzokovic.iiug@gmail.com> wrote: > Probably some oninit process is "holding" port. Look for oninit processes > and > kill them (or as Keith said - reboot machine) > HTH > Hrvoje > > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of PAUL > GATHOGO > Sent: Wednesday, April 11, 2007 2:11 PM > To: ids@iiug.org > Subject: Re: Server fails to come online [8824] > > Keith, > > ipcs -ma reveals no informix segments! > > Any ideas? > > **************************************************************************** > *** > Forum Note: Use "Reply" to post a response in the discussion forum. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > Paul Look in /INFORMIXTMP. If there are no instances running there should not be any files and you can safely remove any. What was the cause of the original crash? There might be some issues with the KAIO settings as shown with the message. What OS are you on? Keith

Let's see. You've just migrated from 7.3X to 10. Now you're having multiple serious problems. - Did you follow all the steps suggested in the Migration Guide for moving from 7.3X to 10? - Did you allow it to complete all steps? I'm guessing your migration to 10 may not have worked or may not have completed successfully. Bob Roussey Unix / Informix Administration Spirit Airlines Robert.Roussey@SpiritAir.com -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of PAUL GATHOGO Sent: Wednesday, April 11, 2007 6:57 AM To: ids@iiug.org Subject: Server fails to come online [8821] All, My DB crashed with an error "THE MAIN DEAMON HAS DIEAD" since then it has refused to come online ... when i do oninit -v i get the following message: Checking group membership to determine server run mode...succeeded Reading configuration file '/home/informix/etc/onconfig.suse'...succeeded Creating /INFORMIXTMP/.infxdirs...succeeded Creating infos file "/home/informix/etc/.infos.suse_ows"...succeeded Linking conf file "/home/informix/etc/.conf.suse_ows"...succeeded Writing to infos file...succeeded Checking config parameters...succeeded Allocating and attaching to shared memory...succeeded Creating resident pool 95490 kbytes...succeeded Allocating 1200016 kbytes for buffer pool of 2K page size...succeeded Initializing rhead structure...succeeded Initializing ASF...succeeded Initializing Dictionary Cache and SPL Routine Cache...succeeded Bringing up ADM VP...succeeded Creating VP classes...succeeded Onlining 0 additional cpu vps...succeeded Onlining 2 IO vps...succeeded Initialization of Encryption...succeeded Forking main_loop thread...succeeded Initializing DR structures...succeeded Forking 1 'soctcp' listener threads...succeeded Forking 1 'ipcshm' listener threads...succeeded Starting tracing...succeeded Initializing 2 flushers...succeeded but when i run onstat -m i get the following message: shared memory not initialized for INFORMIXSERVER 'suse_shm' Message Log File: /home/informix/online.log 13:49:46 Event alarms enabled. ALARMPROG = '/home/informix/etc/alarmprogram.sh' 13:49:46 Booting Language <c> from module <> 13:49:46 Loading Module <CNULL> 13:49:46 Booting Language <builtin> from module <> 13:49:46 Loading Module <BUILTINNULL> 13:49:51 DR: DRAUTO is 0 (Off) 13:49:51 IBM Informix Dynamic Server Version 10.00.UC4 Software Serial Number AAA#B000000 13:49:51 requested number of KAIO events (32768) exceeds limit (16384). using 16384. 13:49:51 listener-thread: err = -25572: oserr = 98: errstr = : Network driver cannot bind a name to the port. System error = 98. 13:49:51 sql_listener: ASF_LISTEN failed 13:49:51 Attempting to bring listener thread down. 13:49:51 Server stopped. 13:49:51 IBM Informix Dynamic Server Stopped. 13:49:51 mt_shm_remove: WARNING: may not have removed all/correct segments Any idea whats happening here? Paul ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum.

Zokovic, I've looked & killed all oninit process, but when i again run oninit -v i get the following message: Checking group membership to determine server run mode...succeeded Reading configuration file '/home/informix/etc/onconfig.suse'...succeeded Creating /INFORMIXTMP/.infxdirs...succeeded Creating infos file "/home/informix/etc/.infos.suse_ows"...succeeded Linking conf file "/home/informix/etc/.conf.suse_ows"...succeeded Writing to infos file...succeeded Checking config parameters...succeeded Allocating and attaching to shared memory...succeeded Creating resident pool 95490 kbytes...succeeded Allocating 1200016 kbytes for buffer pool of 2K page size...succeeded Initializing rhead structure...succeeded Initializing ASF...succeeded Initializing Dictionary Cache and SPL Routine Cache...succeeded Bringing up ADM VP...succeeded Creating VP classes...succeeded Onlining 0 additional cpu vps...succeeded Onlining 2 IO vps...succeeded Initialization of Encryption...succeeded Forking main_loop thread...succeeded Initializing DR structures...succeeded Forking 1 'soctcp' listener threads...succeeded Forking 1 'ipcshm' listener threads...succeeded Starting tracing...succeeded Initializing 2 flushers...succeeded Initializing log/checkpoint information...succeeded Opening primary chunks...succeeded Opening mirror chunks...succeeded Initializing dbspaces...succeeded Validating chunks...succeeded Initialize Async Log Flusher...succeeded Forking btree cleaner...succeeded Initializing DBSPACETEMP list...succeeded SENDER IS NULL NO MAIL WILL BE SENT SENDER IS NULL NO MAIL WILL BE SENT SENDER IS NULL NO MAIL WILL BE SENT SENDER IS NULL NO MAIL WILL BE SENT then when i read the log file (onstat -m) i get the following: shared memory not initialized for INFORMIXSERVER 'suse_shm' Message Log File: /home/informix/online.log 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c ... any hints?

Robert, Sorry for using the word "migration" on that context, but what i ment is, i did a clean IDS10 installation on a clean suse10 box i then unloaded my data from the old IDS7 and loaded it to the new IDS10. Yes the IDS10 installation i did was clean and the database was ok, i only started having problems on one specific database (i have like 7 databases)... any hints? Paul

Hi, first you should have a look at the af-trace file - sometimes there is useful information in it: /home/informix/tmp/af.1d58c second, please provide all of the logfile since startup , not only the last few lines displayed by onstat -m . Regards, Andreas > ------------------------------------------- SPAR Österreichische Warenhandels-AG Hauptzentrale A - 5015 Salzburg, Europastrasse 3 FN 34170 a Tel: +43 662 4470 24223 Mobile: +43 664 6259575 E-Mail: Andreas.KUTSCHE@spar.at Internet: http://www.spar.at Wichtiger Hinweis: Der Inhalt dieser E-Mail kann vertrauliche und rechtlich geschützte Informationen, insbesondere Betriebs- oder Geschäftsgeheimnisse, enthalten, zu deren Geheimhaltung der Empfänger verpflichtet ist. Die Informationen in dieser E-Mail sind ausschließlich für den Adressaten bestimmt. Sollten Sie die E-Mail irrtümlich erhalten haben so ersuchen wir Sie, die Nachricht von Ihrem System zu löschen und sich mit uns in Verbindung zu setzen. Über das Internet versandte E-Mails können leicht manipuliert oder unter fremdem Namen erstellt werden. Daher schließen wir die rechtliche Verbindlichkeit der in dieser Nachricht enthaltenen Informationen aus. Der Inhalt der E-Mail ist nur rechtsverbindlich, wenn er von uns schriftlich bestätigt und gezeichnet wird. Sollte trotz der von uns verwendeten Virus-Schutzprogramme durch die Zusendung von E-Mails ein Virus in Ihre Systeme gelangen, haften wir nicht für evtl. hieraus entstehende Schäden. Wir danken für Ihr Verständnis. Important notice: The contents of this e-mail may contain confidential and legally protected information that is in particular related to operational and trade secrets, which the recipient is obliged to treat as confidential. The information in this e-mail is made available exclusively for use by the addressee. In the event that the e-mail may have been sent to you in error, we would ask you to kindly delete this communication from your system and to contact us. E-mails sent via the Internet can be easily manipulated or sent out under someone else's name. We therefore do not accept legal liability for the information contained in this communication. The contents of the e-mail are only legally binding if they have been confirmed and signed by us in writing. If, in spite of our using Antivirus protection software, a virus may have penetrated your system through the sending of this e-mail, we do not accept liability for any damage that may possibly arise as a result of this. We trust that you appreciate our position. ------------------------------------------- -----Ursprüngliche Nachricht----- > Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org]Im Auftrag von > PAUL GATHOGO > Gesendet: Mittwoch, 11. April 2007 14:43 > An: ids@iiug.org > Betreff: Re: RE: Server fails to come online [8828] > > > Zokovic, > > I've looked & killed all oninit process, but when i again run > oninit -v i get> the following message: > > Checking group membership to determine server run mode...succeeded > Reading configuration file > '/home/informix/etc/onconfig.suse'...succeeded > Creating /INFORMIXTMP/.infxdirs...succeeded > Creating infos file "/home/informix/etc/.infos.suse_ows"...succeeded > Linking conf file "/home/informix/etc/.conf.suse_ows"...succeeded > Writing to infos file...succeeded > Checking config parameters...succeeded > Allocating and attaching to shared memory...succeeded > Creating resident pool 95490 kbytes...succeeded > Allocating 1200016 kbytes for buffer pool of 2K page size...succeeded > Initializing rhead structure...succeeded > Initializing ASF...succeeded > Initializing Dictionary Cache and SPL Routine Cache...succeeded > Bringing up ADM VP...succeeded > Creating VP classes...succeeded > Onlining 0 additional cpu vps...succeeded > Onlining 2 IO vps...succeeded > Initialization of Encryption...succeeded > Forking main_loop thread...succeeded > Initializing DR structures...succeeded > Forking 1 'soctcp' listener threads...succeeded > Forking 1 'ipcshm' listener threads...succeeded > Starting tracing...succeeded > Initializing 2 flushers...succeeded > Initializing log/checkpoint information...succeeded > Opening primary chunks...succeeded > Opening mirror chunks...succeeded > Initializing dbspaces...succeeded > Validating chunks...succeeded > Initialize Async Log Flusher...succeeded > Forking btree cleaner...succeeded > Initializing DBSPACETEMP list...succeeded > SENDER IS NULL NO MAIL WILL BE SENT > SENDER IS NULL NO MAIL WILL BE SENT > SENDER IS NULL NO MAIL WILL BE SENT > SENDER IS NULL NO MAIL WILL BE SENT > > then when i read the log file (onstat -m) i get the following: > > shared memory not initialized for INFORMIXSERVER 'suse_shm' > > Message Log File: /home/informix/online.log > 15:33:17 stack trace for pid 9350 written to > /home/informix/tmp/af.1d58c > 15:33:17 stack trace for pid 9350 written to > /home/informix/tmp/af.1d58c > 15:33:17 stack trace for pid 9350 written to > /home/informix/tmp/af.1d58c > 15:33:17 stack trace for pid 9350 written to > /home/informix/tmp/af.1d58c > 15:33:17 stack trace for pid 9350 written to > /home/informix/tmp/af.1d58c > 15:33:17 stack trace for pid 9350 written to > /home/informix/tmp/af.1d58c > 15:33:17 stack trace for pid 9350 written to > /home/informix/tmp/af.1d58c > 15:33:17 stack trace for pid 9350 written to > /home/informix/tmp/af.1d58c > 15:33:17 stack trace for pid 9350 written to > /home/informix/tmp/af.1d58c > 15:33:17 stack trace for pid 9350 written to > /home/informix/tmp/af.1d58c > 15:33:17 stack trace for pid 9350 written to > /home/informix/tmp/af.1d58c > 15:33:17 stack trace for pid 9350 written to > /home/informix/tmp/af.1d58c > 15:33:17 stack trace for pid 9350 written to > /home/informix/tmp/af.1d58c > 15:33:17 stack trace for pid 9350 written to > /home/informix/tmp/af.1d58c > 15:33:17 stack trace for pid 9350 written to > /home/informix/tmp/af.1d58c > 15:33:17 stack trace for pid 9350 written to > /home/informix/tmp/af.1d58c > 15:33:17 stack trace for pid 9350 written to > /home/informix/tmp/af.1d58c > 15:33:17 stack trace for pid 9350 written to > /home/informix/tmp/af.1d58c > 15:33:17 stack trace for pid 9350 written to > /home/informix/tmp/af.1d58c > 15:33:17 stack trace for pid 9350 written to > /home/informix/tmp/af.1d58c > > .... any hints? > > > ************************************************************** > ***************** > Forum Note: Use "Reply" to post a response in the discussion forum. > >

Andreas, I've looked at the af* file, but am just wondering how do i get some usefull information from that file, it seems to have all sort of info! This is my online.log file (full version): 16:32:37 IBM Informix Dynamic Server Started. Wed Apr 11 16:32:39 2007 16:32:39 Event alarms enabled. ALARMPROG = '/home/informix/etc/alarmprogram.sh ' 16:32:39 Booting Language <c> from module <> 16:32:39 Loading Module <CNULL> 16:32:39 Booting Language <builtin> from module <> 16:32:39 Loading Module <BUILTINNULL> 16:32:44 DR: DRAUTO is 0 (Off) 16:32:45 IBM Informix Dynamic Server Version 10.00.UC4 Software Serial Numb er AAA#B000000 16:32:49 IBM Informix Dynamic Server Initialized -- Shared Memory Initialized. 16:32:49 Warning: Invalid (non-existent/blobspace/disabled) dbspace listed in DBSPACETEMP: '/tmp' 16:32:49 Physical Recovery Started at Page (1:45366). 16:32:49 Physical Recovery Complete: 1230 Pages Examined, 1230 Pages Restored. 16:32:49 Logical Recovery Started. 16:32:49 10 recovery worker threads will be started. 16:32:49 Dynamically allocated new virtual shared memory segment (size 16384KB) 16:33:10 Logical Recovery has reached the transaction cleanup phase. 16:33:10 Assert Failed: Rollback error 126 16:33:10 IBM Informix Dynamic Server Version 10.00.UC4 16:33:10 Who: Session(11, root@susesrvr, 0, 0xa54e8c30) Thread(46, fast_rec, a54bf9f4, 1) File: rstrans.c Line: 2583 16:33:10 Results: Log record (OLDRSAM:HINSERT) in log 6448, offset 0xb9518 was not rolled back 16:33:10 Action: Use 'onlog' to view the transaction and repair manually. 16:33:10 stack trace for pid 10654 written to /home/informix/tmp/af.416e396 16:33:10 See Also: /home/informix/tmp/af.416e396, shmem.416e396.0 16:33:15 Rollback error 126 16:33:15 Assert Failed: Rollback error 126 16:33:15 IBM Informix Dynamic Server Version 10.00.UC4 16:33:15 Who: Session(11, root@susesrvr, 0, 0xa54e8c30) Thread(46, fast_rec, a54bf9f4, 1) File: rsextlog.c Line: 1679 16:33:15 Results: Log record (OLDRSAM:HINSERT) in log 6448, offset 0xb9518 was not rolled back 16:33:15 Action: Use 'onlog' to view the transaction and repair manually. 16:33:15 stack trace for pid 10654 written to /home/informix/tmp/af.416e396 16:33:15 See Also: /home/informix/tmp/af.416e396 16:33:57 Rollback error 126 16:33:57 Assert Failed: Dynamic Server must abort 16:33:57 IBM Informix Dynamic Server Version 10.00.UC4 16:33:57 Who: Session(11, root@susesrvr, 0, 0xa54e8c30) Thread(46, fast_rec, a54bf9f4, 1) File: rslog.c Line: 3589 16:33:57 Results: Dynamic Server must abort 16:33:57 Action: Reinitialize shared memory 16:33:57 stack trace for pid 10654 written to /home/informix/tmp/af.416e396 16:33:57 See Also: /home/informix/tmp/af.416e396 16:33:59 rslog.c, line 3589, thread 46, proc id 10654, Dynamic Server must abor t. 16:33:59 PANIC: Attempting to bring system down 16:33:59 semctl: errno = 22 16:33:59 semctl: errno = 22 16:33:59 semctl: errno = 22 16:33:59 semctl: errno = 22 16:33:59 semctl: errno = 22 ... any ideas? Paul

Any possibility that your dbspaces or chunks are overlapping on disk? Are they cooked files or raw partitions of disk? Bob Roussey Unix / Informix Administration Spirit Airlines Robert.Roussey@SpiritAir.com -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of PAUL GATHOGO Sent: Wednesday, April 11, 2007 8:48 AM To: ids@iiug.org Subject: Re: RE: Server fails to come online [8829] Robert, Sorry for using the word "migration" on that context, but what i ment is, i did a clean IDS10 installation on a clean suse10 box i then unloaded my data from the old IDS7 and loaded it to the new IDS10. Yes the IDS10 installation i did was clean and the database was ok, i only started having problems on one specific database (i have like 7 databases)... any hints? Paul ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum.

Robert, The Primary chunks are on raw devices (via symbolic links ) and mirror chunks are cooked files, by the by this whole setup was working ok till i was loading some data from Ascii txt file when the server came crushing with the "MAIN DEAMON HAS DIED" error!! since then am un-able to respawn back to life ... Paul

You have a partially written logical log record. Some logical log records require multiple pages and if the record is incomplete on disk the engine will not startup. Looks like in the crash the engine was in the process of writing a multiple page log record. You'll have to contact tech support so they can truncate the logical log to a good record. Art S. Kagel ----- Original Message ----- From: Paul Gathogo <ids@iiug.org> At: 4/11 9:45:09 Andreas, I've looked at the af* file, but am just wondering how do i get some usefull information from that file, it seems to have all sort of info! This is my online.log file (full version): 16:32:37 IBM Informix Dynamic Server Started. Wed Apr 11 16:32:39 2007 16:32:39 Event alarms enabled. ALARMPROG = '/home/informix/etc/alarmprogram.sh ' 16:32:39 Booting Language <c> from module <> 16:32:39 Loading Module <CNULL> 16:32:39 Booting Language <builtin> from module <> 16:32:39 Loading Module <BUILTINNULL> 16:32:44 DR: DRAUTO is 0 (Off) 16:32:45 IBM Informix Dynamic Server Version 10.00.UC4 Software Serial Numb er AAA#B000000 16:32:49 IBM Informix Dynamic Server Initialized -- Shared Memory Initialized. 16:32:49 Warning: Invalid (non-existent/blobspace/disabled) dbspace listed in DBSPACETEMP: '/tmp' 16:32:49 Physical Recovery Started at Page (1:45366). 16:32:49 Physical Recovery Complete: 1230 Pages Examined, 1230 Pages Restored. 16:32:49 Logical Recovery Started. 16:32:49 10 recovery worker threads will be started. 16:32:49 Dynamically allocated new virtual shared memory segment (size 16384KB) 16:33:10 Logical Recovery has reached the transaction cleanup phase. 16:33:10 Assert Failed: Rollback error 126 16:33:10 IBM Informix Dynamic Server Version 10.00.UC4 16:33:10 Who: Session(11, root@susesrvr, 0, 0xa54e8c30) Thread(46, fast_rec, a54bf9f4, 1) File: rstrans.c Line: 2583 16:33:10 Results: Log record (OLDRSAM:HINSERT) in log 6448, offset 0xb9518 was not rolled back 16:33:10 Action: Use 'onlog' to view the transaction and repair manually. 16:33:10 stack trace for pid 10654 written to /home/informix/tmp/af.416e396 16:33:10 See Also: /home/informix/tmp/af.416e396, shmem.416e396.0 16:33:15 Rollback error 126 16:33:15 Assert Failed: Rollback error 126 16:33:15 IBM Informix Dynamic Server Version 10.00.UC4 16:33:15 Who: Session(11, root@susesrvr, 0, 0xa54e8c30) Thread(46, fast_rec, a54bf9f4, 1) File: rsextlog.c Line: 1679 16:33:15 Results: Log record (OLDRSAM:HINSERT) in log 6448, offset 0xb9518 was not rolled back 16:33:15 Action: Use 'onlog' to view the transaction and repair manually. 16:33:15 stack trace for pid 10654 written to /home/informix/tmp/af.416e396 16:33:15 See Also: /home/informix/tmp/af.416e396 16:33:57 Rollback error 126 16:33:57 Assert Failed: Dynamic Server must abort 16:33:57 IBM Informix Dynamic Server Version 10.00.UC4 16:33:57 Who: Session(11, root@susesrvr, 0, 0xa54e8c30) Thread(46, fast_rec, a54bf9f4, 1) File: rslog.c Line: 3589 16:33:57 Results: Dynamic Server must abort 16:33:57 Action: Reinitialize shared memory 16:33:57 stack trace for pid 10654 written to /home/informix/tmp/af.416e396 16:33:57 See Also: /home/informix/tmp/af.416e396 16:33:59 rslog.c, line 3589, thread 46, proc id 10654, Dynamic Server must abor t. 16:33:59 PANIC: Attempting to bring system down 16:33:59 semctl: errno = 22 16:33:59 semctl: errno = 22 16:33:59 semctl: errno = 22 16:33:59 semctl: errno = 22 16:33:59 semctl: errno = 22 .... any ideas? Paul ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Check the sqlhosts files and see what port is being used. Then check you /etc/services file to make sure that the port is defined similar to below: $INFORMIXDIR/etc/sqlhosts mydb onsoctcp 192.10.10.10 1526 /etc/services mydb 1526/tcp # Informix listener Lennie -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org]On Behalf Of PAUL GATHOGO Sent: Wednesday, April 11, 2007 5:57 AM To: ids@iiug.org Subject: Server fails to come online [8821] All, My DB crashed with an error "THE MAIN DEAMON HAS DIEAD" since then it has refused to come online ... when i do oninit -v i get the following message: Checking group membership to determine server run mode...succeeded Reading configuration file '/home/informix/etc/onconfig.suse'...succeeded Creating /INFORMIXTMP/.infxdirs...succeeded Creating infos file "/home/informix/etc/.infos.suse_ows"...succeeded Linking conf file "/home/informix/etc/.conf.suse_ows"...succeeded Writing to infos file...succeeded Checking config parameters...succeeded Allocating and attaching to shared memory...succeeded Creating resident pool 95490 kbytes...succeeded Allocating 1200016 kbytes for buffer pool of 2K page size...succeeded Initializing rhead structure...succeeded Initializing ASF...succeeded Initializing Dictionary Cache and SPL Routine Cache...succeeded Bringing up ADM VP...succeeded Creating VP classes...succeeded Onlining 0 additional cpu vps...succeeded Onlining 2 IO vps...succeeded Initialization of Encryption...succeeded Forking main_loop thread...succeeded Initializing DR structures...succeeded Forking 1 'soctcp' listener threads...succeeded Forking 1 'ipcshm' listener threads...succeeded Starting tracing...succeeded Initializing 2 flushers...succeeded but when i run onstat -m i get the following message: shared memory not initialized for INFORMIXSERVER 'suse_shm' Message Log File: /home/informix/online.log 13:49:46 Event alarms enabled. ALARMPROG = '/home/informix/etc/alarmprogram.sh' 13:49:46 Booting Language <c> from module <> 13:49:46 Loading Module <CNULL> 13:49:46 Booting Language <builtin> from module <> 13:49:46 Loading Module <BUILTINNULL> 13:49:51 DR: DRAUTO is 0 (Off) 13:49:51 IBM Informix Dynamic Server Version 10.00.UC4 Software Serial Number AAA#B000000 13:49:51 requested number of KAIO events (32768) exceeds limit (16384). using 16384. 13:49:51 listener-thread: err = -25572: oserr = 98: errstr = : Network driver cannot bind a name to the port. System error = 98. 13:49:51 sql_listener: ASF_LISTEN failed 13:49:51 Attempting to bring listener thread down. 13:49:51 Server stopped. 13:49:51 IBM Informix Dynamic Server Stopped. 13:49:51 mt_shm_remove: WARNING: may not have removed all/correct segments Any idea whats happening here? Paul **************************************************************************** *** Forum Note: Use "Reply" to post a response in the discussion forum.

Seems that it has some problem with the cleanup of temp space - Try using oninit -p Sunil. ________________________________ From: ids-bounces@iiug.org on behalf of PAUL GATHOGO Sent: Wed 4/11/2007 1:43 PM To: ids@iiug.org Subject: Re: RE: Server fails to come online [8828] Zokovic, I've looked & killed all oninit process, but when i again run oninit -v i get the following message: Checking group membership to determine server run mode...succeeded Reading configuration file '/home/informix/etc/onconfig.suse'...succeeded Creating /INFORMIXTMP/.infxdirs...succeeded Creating infos file "/home/informix/etc/.infos.suse_ows"...succeeded Linking conf file "/home/informix/etc/.conf.suse_ows"...succeeded Writing to infos file...succeeded Checking config parameters...succeeded Allocating and attaching to shared memory...succeeded Creating resident pool 95490 kbytes...succeeded Allocating 1200016 kbytes for buffer pool of 2K page size...succeeded Initializing rhead structure...succeeded Initializing ASF...succeeded Initializing Dictionary Cache and SPL Routine Cache...succeeded Bringing up ADM VP...succeeded Creating VP classes...succeeded Onlining 0 additional cpu vps...succeeded Onlining 2 IO vps...succeeded Initialization of Encryption...succeeded Forking main_loop thread...succeeded Initializing DR structures...succeeded Forking 1 'soctcp' listener threads...succeeded Forking 1 'ipcshm' listener threads...succeeded Starting tracing...succeeded Initializing 2 flushers...succeeded Initializing log/checkpoint information...succeeded Opening primary chunks...succeeded Opening mirror chunks...succeeded Initializing dbspaces...succeeded Validating chunks...succeeded Initialize Async Log Flusher...succeeded Forking btree cleaner...succeeded Initializing DBSPACETEMP list...succeeded SENDER IS NULL NO MAIL WILL BE SENT SENDER IS NULL NO MAIL WILL BE SENT SENDER IS NULL NO MAIL WILL BE SENT SENDER IS NULL NO MAIL WILL BE SENT then when i read the log file (onstat -m) i get the following: shared memory not initialized for INFORMIXSERVER 'suse_shm' Message Log File: /home/informix/online.log 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c 15:33:17 stack trace for pid 9350 written to /home/informix/tmp/af.1d58c .... any hints? ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum. ******************************************************************************** ******************* The information in this email is confidential and may be legally privileged Access to this email by anyone other than the intended addressee is unauthorized. If you are not the intended recipient of this message, any review, disclosure, copying, distribution, retention, or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you are not the intended recipient, please reply to or forward a copy of this message to the sender and delete the message, any attachments, and any copies thereof from your system. ******************************************************************************** *******************

Hi Paul, As you can see in one_of_the_usefull_parts of af file: - 16:32:49 Warning: Invalid (non-existent/blobspace/disabled) dbspace listed in DBSPACETEMP: '/tmp' you specified /tmp as DBSPACETEMP - this is not correct, this parameter specifies "logical" dbs names (i.e. you have to add dbspaces to your instance with onspaces) Also, at other_also_usefull_part of af file: - 16:32:49 Physical Recovery Complete: 1230 Pages Examined, 1230 Pages Restored. - 16:32:49 Logical Recovery Started. - 16:32:49 10 recovery worker threads will be started. - 16:32:49 Dynamically allocated new virtual shared memory segment (size 16384KB) - 16:33:10 Logical Recovery has reached the transaction cleanup phase. - 16:33:10 Assert Failed: Rollback error 126 - 16:33:10 IBM Informix Dynamic Server Version 10.00.UC4 - 16:33:10 Who: Session(11, root@susesrvr, 0, 0xa54e8c30) Thread(46, fast_rec, a54bf9f4, 1) File: rstrans.c Line: 2583 - 16:33:10 Results: Log record (OLDRSAM:HINSERT) in log 6448, offset 0xb9518 was not rolled back - 16:33:10 Action: Use 'onlog' to view the transaction and repair manually. - 16:33:10 stack trace for pid 10654 written to /home/informix/tmp/af.416e396 - 16:33:10 See Also: /home/informix/tmp/af.416e396, shmem.416e396.0 You can see that rollback error 126 occurs, fast recovery was stucked in log 6448, offset 0xb9518, and you can use onlog utility to view transactions in that log and to commit/rollback transactions manualy - of course this is extremely specialized method and it can be only done by specialist. As Art suggested, you can contact tech support to truncate logical log to last good record. BR Hrvoje -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of PAUL GATHOGO Sent: Wednesday, April 11, 2007 3:44 PM To: ids@iiug.org Subject: Re: AW: RE: Server fails to come online [8831] Andreas, I've looked at the af* file, but am just wondering how do i get some usefull information from that file, it seems to have all sort of info! This is my online.log file (full version): 16:32:37 IBM Informix Dynamic Server Started. Wed Apr 11 16:32:39 2007 16:32:39 Event alarms enabled. ALARMPROG = '/home/informix/etc/alarmprogram.sh ' 16:32:39 Booting Language <c> from module <> 16:32:39 Loading Module <CNULL> 16:32:39 Booting Language <builtin> from module <> 16:32:39 Loading Module <BUILTINNULL> 16:32:44 DR: DRAUTO is 0 (Off) 16:32:45 IBM Informix Dynamic Server Version 10.00.UC4 Software Serial Numb er AAA#B000000 16:32:49 IBM Informix Dynamic Server Initialized -- Shared Memory Initialized. 16:32:49 Warning: Invalid (non-existent/blobspace/disabled) dbspace listed in DBSPACETEMP: '/tmp' 16:32:49 Physical Recovery Started at Page (1:45366). 16:32:49 Physical Recovery Complete: 1230 Pages Examined, 1230 Pages Restored. 16:32:49 Logical Recovery Started. 16:32:49 10 recovery worker threads will be started. 16:32:49 Dynamically allocated new virtual shared memory segment (size 16384KB) 16:33:10 Logical Recovery has reached the transaction cleanup phase. 16:33:10 Assert Failed: Rollback error 126 16:33:10 IBM Informix Dynamic Server Version 10.00.UC4 16:33:10 Who: Session(11, root@susesrvr, 0, 0xa54e8c30) Thread(46, fast_rec, a54bf9f4, 1) File: rstrans.c Line: 2583 16:33:10 Results: Log record (OLDRSAM:HINSERT) in log 6448, offset 0xb9518 was not rolled back 16:33:10 Action: Use 'onlog' to view the transaction and repair manually. 16:33:10 stack trace for pid 10654 written to /home/informix/tmp/af.416e396 16:33:10 See Also: /home/informix/tmp/af.416e396, shmem.416e396.0 16:33:15 Rollback error 126 16:33:15 Assert Failed: Rollback error 126 16:33:15 IBM Informix Dynamic Server Version 10.00.UC4 16:33:15 Who: Session(11, root@susesrvr, 0, 0xa54e8c30) Thread(46, fast_rec, a54bf9f4, 1) File: rsextlog.c Line: 1679 16:33:15 Results: Log record (OLDRSAM:HINSERT) in log 6448, offset 0xb9518 was not rolled back 16:33:15 Action: Use 'onlog' to view the transaction and repair manually. 16:33:15 stack trace for pid 10654 written to /home/informix/tmp/af.416e396 16:33:15 See Also: /home/informix/tmp/af.416e396 16:33:57 Rollback error 126 16:33:57 Assert Failed: Dynamic Server must abort 16:33:57 IBM Informix Dynamic Server Version 10.00.UC4 16:33:57 Who: Session(11, root@susesrvr, 0, 0xa54e8c30) Thread(46, fast_rec, a54bf9f4, 1) File: rslog.c Line: 3589 16:33:57 Results: Dynamic Server must abort 16:33:57 Action: Reinitialize shared memory 16:33:57 stack trace for pid 10654 written to /home/informix/tmp/af.416e396 16:33:57 See Also: /home/informix/tmp/af.416e396 16:33:59 rslog.c, line 3589, thread 46, proc id 10654, Dynamic Server must abor t. 16:33:59 PANIC: Attempting to bring system down 16:33:59 semctl: errno = 22 16:33:59 semctl: errno = 22 16:33:59 semctl: errno = 22 16:33:59 semctl: errno = 22 16:33:59 semctl: errno = 22 .... any ideas? Paul **************************************************************************** *** Forum Note: Use "Reply" to post a response in the discussion forum.