Oninit cdi / Archive
Talk to an expert →

Re: IDS 9.4 Stuck at Fast Recovery

Posted in 2007

Topics: Storage & Space Management, Connectivity: ESQL/C, 4GL & Embedded SQL, Server Administration, Logging & Checkpoints, Platform-Specific Issues, Versions, Editions & End-of-Life

Keith Simmons — — source: Usenet: comp.databases.informix

On 10/11/2007, Marco Greco <marco@4glworks.com> wrote: > Keith Simmons wrote: > > On 10/11/2007, Marco Greco <marco@4glworks.com> wrote: > >> Keith Simmons wrote: > >>> Anyone > >>> > >>> I havea test instance (not being archived that regularly) that > >>> suffered some disk corruption (crossed chunks with another instance > >>> !!). It has bee running OK and Ihave been sorting issues with indexes > >>> as they appear (the crossed chunks were on a dbspace containing mainly > >>> detatched indexes). However there does appear to be a table in one of > >>> these chunks. oncheck -cI showed an issue and I sucessfully dropped > >>> theindexes but was unable to recreate them. The next time I bounced > >>> the instance it refused to start getting stuck at the Fast Recovery > >>> stage. I have tried with -p option to oninit but to no avail. > >>> Although this is only a test instance I really need to preserve most > >>> of the data if at all possible. I can recreate the odd table or two > >>> where required, but not all 200 Gb!! > >>> Any suggestions as to whether it is possible to force this engine into > >>> life by skipping the Fast Recovery in some way? > >>> IDS9.4 FC6 on AIX 5.3. > >>> onstat -p > >>> IBM Informix Dynamic Server Version 9.40.FC6 -- Fast Recovery -- > >>> Up 19:41:49 -- 3020880 Kbytes> >>> > >>> Profile > >>> dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached > >>> 0 0 0 0.00 0 0 0 0.00 > >>> > >>> isamtot open start read write rewrite delete commit rollbk > >>> 0 0 0 0 0 0 0 0 0 > >>> > >>> gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs > >>> 0 0 0 0 0 0 0 > >>> > >>> ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes > >>> 0 0 0 70734.67 19.47 0 0 > >>> > >>> bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans > >>> 0 0 0 0 0 0 0 0 > >>> > >>> ixda-RA idx-RA da-RA RA-pgsused lchwaits > >>> 0 0 0 0 52 > >>> > >>> Thanks > >>> > >>> Keith > >>> _______________________________________________ > >>> Informix-list mailing list > >>> Informix-list@iiug.org > >>> http://www.iiug.org/mailman/listinfo/informix-list > >>> > >> Chances are that the LSN is way behind, and in the next log that the engine is > >> trying to use. but without peeking around it's impossible to say. If it is > >> important to you, you know the drill... > >> -- > >> Ciao, > >> Marco > >> ______________________________________________________________________________ > >> Marco Greco /UK /IBM Standard disclaimers apply! > >> > >> Structured Query Scripting Language http://www.4glworks.com/sqsl.htm > >> 4glworks http://www.4glworks.com > >> Informix on Linux http://www.4glworks.com/ifmxlinux.htm > >> > > Marco > > > > Thanks. > > > > onstat -l > > 8187fb50 15 U-B---- 16818 3:525053 37500 37500 100.00 > > 8187fbb8 16 U-B---- 16819 3:562553 37500 37500 100.00 > > 8187fc20 17 U-B---- 16820 3:600053 37500 37500 100.00 > > 8187fc88 18 U---C-L 16821 3:637553 37500 22313 59.50 > > 8187fcf0 19 U-B---- 16792 3:675053 37500 37500 100.00 > > 8187fd58 20 U-B---- 16793 3:712553 37500 37500 100.00 > > 8187fdc0 21 U-B---- 16794 3:750053 37500 37500 100.00 > > 8187fe28 22 U-B---- 16795 3:787553 37500 37500 100.00 > > 8187fe90 23 U-B---- 16796 3:825053 37500 37500 100.00> > > > LTAPEDEV /dev/null> > > > last checkpoint in the log file was in llog 16821 and was a full > > checkpoint as part of onmode -yuck. > > > > Will do the required next week. > > > > Keith > > Nope, not the LSN. I take it there are no recovery afs in the logs, right? > if this is the case, kill the engine, set OFF_RECOVERY_THREADS to 1 and see > what happens > > -- > Ciao, > Marco > ______________________________________________________________________________ > Marco Greco /UK /IBM Standard disclaimers apply! > > Structured Query Scripting Language http://www.4glworks.com/sqsl.htm > 4glworks http://www.4glworks.com > Informix on Linux http://www.4glworks.com/ifmxlinux.htm > Marco Still no go, onstat -mSat Nov 10 20:51:43 2007 20:51:43 Event alarms enabled. ALARMPROG = '/usr/IDS9/etc/no_log.sh' 20:51:43 Booting Language <c> from module <> 20:51:43 Loading Module <CNULL> 20:51:43 Booting Language <builtin> from module <> 20:51:43 Loading Module <BUILTINNULL> 20:51:48 AIX MP latch code enabled 20:51:48 IBM Informix Dynamic Server Version 9.40.FC6 Software Serial Number AAA#B000000 20:51:49 IBM Informix Dynamic Server Initialized -- Shared Memory Initialized. 20:51:49 Physical Recovery Started at Page (2:3483). 20:51:49 Physical Recovery Complete: 0 Pages Examined, 0 Pages Restored. 20:51:49 Logical Recovery Started. 20:51:49 1 recovery worker threads will be started. 20:51:53 Logical Recovery has reached the transaction cleanup phase. been there 14 minutes and counting. Keith

↪ replying to Keith Simmons
david@smooth1.co.uk — — source: Usenet: comp.databases.informix

On 10 Nov, 21:06, "Keith Simmons" <smile...@googlemail.com> wrote: > On 10/11/2007, Marco Greco <ma...@4glworks.com> wrote: > > > > > Keith Simmons wrote: > > > On 10/11/2007, Marco Greco <ma...@4glworks.com> wrote: > > >> Keith Simmons wrote: > > >>> Anyone > > > >>> I havea test instance (not being archived that regularly) that > > >>> suffered some disk corruption (crossed chunks with another instance > > >>> !!). It has bee running OK and Ihave been sorting issues with indexes > > >>> as they appear (the crossed chunks were on a dbspace containing mainly > > >>> detatched indexes). However there does appear to be a table in one of > > >>> these chunks. oncheck -cI showed an issue and I sucessfully dropped > > >>> theindexes but was unable to recreate them. The next time I bounced > > >>> the instance it refused to start getting stuck at the Fast Recovery > > >>> stage. I have tried with -p option to oninit but to no avail. > > >>> Although this is only a test instance I really need to preserve most > > >>> of the data if at all possible. I can recreate the odd table or two > > >>> where required, but not all 200 Gb!! > > >>> Any suggestions as to whether it is possible to force this engine into > > >>> life by skipping the Fast Recovery in some way? > > >>> IDS9.4 FC6 on AIX 5.3. > > >>> onstat -p > > >>> IBM Informix Dynamic Server Version 9.40.FC6 -- Fast Recovery -- > > >>> Up 19:41:49 -- 3020880 Kbytes> > > >>> Profile > > >>> dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached > > >>> 0 0 0 0.00 0 0 0 0.00 > > > >>> isamtot open start read write rewrite delete commit rollbk > > >>> 0 0 0 0 0 0 0 0 0 > > > >>> gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs > > >>> 0 0 0 0 0 0 0 > > > >>> ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes > > >>> 0 0 0 70734.67 19.47 0 0 > > > >>> bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans > > >>> 0 0 0 0 0 0 0 0 > > > >>> ixda-RA idx-RA da-RA RA-pgsused lchwaits > > >>> 0 0 0 0 52 > > > >>> Thanks > > > >>> Keith > > >>> _______________________________________________ > > >>> Informix-list mailing list > > >>> Informix-l...@iiug.org > > >>>http://www.iiug.org/mailman/listinfo/informix-list > > > >> Chances are that the LSN is way behind, and in the next log that the engine is > > >> trying to use. but without peeking around it's impossible to say. If it is > > >> important to you, you know the drill... > > >> -- > > >> Ciao, > > >> Marco > > >> ___________________________________________________________________________'___ > > >> Marco Greco /UK /IBM Standard disclaimers apply! > > > >> Structured Query Scripting Language http://www.4glworks.com/sqsl.htm > > >> 4glworks http://www.4glworks.com > > >> Informix on Linux http://www.4glworks.com/ifmxlinux.htm > > > > Marco > > > > Thanks. > > > > onstat -l > > > 8187fb50 15 U-B---- 16818 3:525053 37500 37500 100.00 > > > 8187fbb8 16 U-B---- 16819 3:562553 37500 37500 100.00 > > > 8187fc20 17 U-B---- 16820 3:600053 37500 37500 100.00 > > > 8187fc88 18 U---C-L 16821 3:637553 37500 22313 59.50 > > > 8187fcf0 19 U-B---- 16792 3:675053 37500 37500 100.00 > > > 8187fd58 20 U-B---- 16793 3:712553 37500 37500 100.00 > > > 8187fdc0 21 U-B---- 16794 3:750053 37500 37500 100.00 > > > 8187fe28 22 U-B---- 16795 3:787553 37500 37500 100.00 > > > 8187fe90 23 U-B---- 16796 3:825053 37500 37500 100.00> > > > LTAPEDEV /dev/null> > > > last checkpoint in the log file was in llog 16821 and was a full > > > checkpoint as part of onmode -yuck. > > > > Will do the required next week. > > > > Keith > > > Nope, not the LSN. I take it there are no recovery afs in the logs, right? > > if this is the case, kill the engine, set OFF_RECOVERY_THREADS to 1 and see > > what happens > > > -- > > Ciao, > > Marco > > ___________________________________________________________________________'___ > > Marco Greco /UK /IBM Standard disclaimers apply! > > > Structured Query Scripting Language http://www.4glworks.com/sqsl.htm > > 4glworks http://www.4glworks.com > > Informix on Linux http://www.4glworks.com/ifmxlinux.htm > > Marco > > Still no go, > onstat -m> Sat Nov 10 20:51:43 2007 > > 20:51:43 Event alarms enabled. ALARMPROG = '/usr/IDS9/etc/no_log.sh' > 20:51:43 Booting Language <c> from module <> > 20:51:43 Loading Module <CNULL> > 20:51:43 Booting Language <builtin> from module <> > 20:51:43 Loading Module <BUILTINNULL> > 20:51:48 AIX MP latch code enabled > 20:51:48 IBM Informix Dynamic Server Version 9.40.FC6 Software Serial Numbe> r AAA#B000000 > 20:51:49 IBM Informix Dynamic Server Initialized -- Shared Memory Initialized. > > 20:51:49 Physical Recovery Started at Page (2:3483). > 20:51:49 Physical Recovery Complete: 0 Pages Examined, 0 Pages Restored. > 20:51:49 Logical Recovery Started. > 20:51:49 1 recovery worker threads will be started. > 20:51:53 Logical Recovery has reached the transaction cleanup phase.> > been there 14 minutes and counting. > > Keith- Hide quoted text - > > - Show quoted text - Run onstat -l, does it change? Run onstat -g ath what does onstat -g stk for the active threads give?

↪ replying to david@smooth1.co.uk
Keith Simmons — — source: Usenet: comp.databases.informix

On 11/11/2007, david@smooth1.co.uk <david@smooth1.co.uk> wrote: > On 10 Nov, 21:06, "Keith Simmons" <smile...@googlemail.com> wrote: > > On 10/11/2007, Marco Greco <ma...@4glworks.com> wrote: > > > > > > > > > Keith Simmons wrote: > > > > On 10/11/2007, Marco Greco <ma...@4glworks.com> wrote: > > > >> Keith Simmons wrote: > > > >>> Anyone > > > > > >>> I havea test instance (not being archived that regularly) that > > > >>> suffered some disk corruption (crossed chunks with another instance > > > >>> !!). It has bee running OK and Ihave been sorting issues with indexes > > > >>> as they appear (the crossed chunks were on a dbspace containing mainly > > > >>> detatched indexes). However there does appear to be a table in one of > > > >>> these chunks. oncheck -cI showed an issue and I sucessfully dropped > > > >>> theindexes but was unable to recreate them. The next time I bounced > > > >>> the instance it refused to start getting stuck at the Fast Recovery > > > >>> stage. I have tried with -p option to oninit but to no avail. > > > >>> Although this is only a test instance I really need to preserve most > > > >>> of the data if at all possible. I can recreate the odd table or two > > > >>> where required, but not all 200 Gb!! > > > >>> Any suggestions as to whether it is possible to force this engine into > > > >>> life by skipping the Fast Recovery in some way? > > > >>> IDS9.4 FC6 on AIX 5.3. > > > >>> onstat -p > > > >>> IBM Informix Dynamic Server Version 9.40.FC6 -- Fast Recovery -- > > > >>> Up 19:41:49 -- 3020880 Kbytes> > > > > >>> Profile > > > >>> dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached > > > >>> 0 0 0 0.00 0 0 0 0.00 > > > > > >>> isamtot open start read write rewrite delete commit rollbk > > > >>> 0 0 0 0 0 0 0 0 0 > > > > > >>> gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs > > > >>> 0 0 0 0 0 0 0 > > > > > >>> ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes > > > >>> 0 0 0 70734.67 19.47 0 0 > > > > > >>> bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans > > > >>> 0 0 0 0 0 0 0 0 > > > > > >>> ixda-RA idx-RA da-RA RA-pgsused lchwaits > > > >>> 0 0 0 0 52 > > > > > >>> Thanks > > > > > >>> Keith > > > >>> _______________________________________________ > > > >>> Informix-list mailing list > > > >>> Informix-l...@iiug.org > > > >>>http://www.iiug.org/mailman/listinfo/informix-list > > > > > >> Chances are that the LSN is way behind, and in the next log that the engine is > > > >> trying to use. but without peeking around it's impossible to say. If it is > > > >> important to you, you know the drill... > > > >> -- > > > >> Ciao, > > > >> Marco > > > >> ___________________________________________________________________________­___ > > > >> Marco Greco /UK /IBM Standard disclaimers apply! > > > > > >> Structured Query Scripting Language http://www.4glworks.com/sqsl.htm > > > >> 4glworks http://www.4glworks.com > > > >> Informix on Linux http://www.4glworks.com/ifmxlinux.htm > > > > > > Marco > > > > > > Thanks. > > > > > > onstat -l > > > > 8187fb50 15 U-B---- 16818 3:525053 37500 37500 100.00 > > > > 8187fbb8 16 U-B---- 16819 3:562553 37500 37500 100.00 > > > > 8187fc20 17 U-B---- 16820 3:600053 37500 37500 100.00 > > > > 8187fc88 18 U---C-L 16821 3:637553 37500 22313 59.50 > > > > 8187fcf0 19 U-B---- 16792 3:675053 37500 37500 100.00 > > > > 8187fd58 20 U-B---- 16793 3:712553 37500 37500 100.00 > > > > 8187fdc0 21 U-B---- 16794 3:750053 37500 37500 100.00 > > > > 8187fe28 22 U-B---- 16795 3:787553 37500 37500 100.00 > > > > 8187fe90 23 U-B---- 16796 3:825053 37500 37500 100.00> > > > > > LTAPEDEV /dev/null> > > > > > last checkpoint in the log file was in llog 16821 and was a full > > > > checkpoint as part of onmode -yuck. > > > > > > Will do the required next week. > > > > > > Keith > > > > > Nope, not the LSN. I take it there are no recovery afs in the logs, right? > > > if this is the case, kill the engine, set OFF_RECOVERY_THREADS to 1 and see > > > what happens > > > > > -- > > > Ciao, > > > Marco > > > ___________________________________________________________________________­___ > > > Marco Greco /UK /IBM Standard disclaimers apply! > > > > > Structured Query Scripting Language http://www.4glworks.com/sqsl.htm > > > 4glworks http://www.4glworks.com > > > Informix on Linux http://www.4glworks.com/ifmxlinux.htm > > > > Marco > > > > Still no go, > > onstat -m> > Sat Nov 10 20:51:43 2007 > > > > 20:51:43 Event alarms enabled. ALARMPROG = '/usr/IDS9/etc/no_log.sh' > > 20:51:43 Booting Language <c> from module <> > > 20:51:43 Loading Module <CNULL> > > 20:51:43 Booting Language <builtin> from module <> > > 20:51:43 Loading Module <BUILTINNULL> > > 20:51:48 AIX MP latch code enabled > > 20:51:48 IBM Informix Dynamic Server Version 9.40.FC6 Software Serial Numbe> > r AAA#B000000 > > 20:51:49 IBM Informix Dynamic Server Initialized -- Shared Memory Initialized. > > > > 20:51:49 Physical Recovery Started at Page (2:3483). > > 20:51:49 Physical Recovery Complete: 0 Pages Examined, 0 Pages Restored. > > 20:51:49 Logical Recovery Started. > > 20:51:49 1 recovery worker threads will be started. > > 20:51:53 Logical Recovery has reached the transaction cleanup phase.> > > > been there 14 minutes and counting. > > > > Keith- Hide quoted text - > > > > - Show quoted text - > > Run onstat -l, does it change? > > Run onstat -g ath what does onstat -g stk for the active threads give? > > > _______________________________________________ > Informix-list mailing list > Informix-list@iiug.org > http://www.iiug.org/mailman/listinfo/informix-list > > David onstat -l - no movement.onstta -g ath - poll threads running or waiting, btscanner sleeping 1 sec, fast_rec running, everything else sleeping forever. onstat -g stk fast_rec IBM Informix Dynamic Server Version 9.40.FC6 -- Fast Recovery -- Up 18:07:23 -- 3020880 Kbytes Stack for thread: 151 fast_rec base: 0x081b26000 len: 69632 pc: 0x0000000100039610 tos: 0x081b35f20state: running vp: 1 0x0000000100039610 (oninit)yield_processor_mvp(0x1000, 0x8123b998, 0x81b36160, 0 x5310083016c33000, 0x100054ec8, 0x81a67028, 0x1000, 0x1) 0x081b36110 (*nosymtab*)0x81b36110(0x5, 0x1, 0x81b362d0, 0x8123b998, 0x81b36240, 0x8

Related threads

Glossary