Rollforward of logs failing

We had a power outage that affected one of our Informix instances. We tried bringing it up, but the database seems to be stuck in Fast Recovery mode and I am getting the following error messages: 00:15:46 I/O read chunk 19, pagenum 836337, pagecnt 1 --> errno = 28 00:15:46 Assert Warning: pthdrpage:ptalloc:bad bfget 00:15:46 IBM Informix Dynamic Server Version 9.40.HC6X1 00:15:46 Who: Session(19, informix@spkwms01, 0, c844ccb8) Thread(78, xchg_1.8, c84271d4, 7) File: rspartn.c Line: 5899 00:15:46 Results: Cannot use TBLSpace page for TBLSpace 7341178 00:15:46 Action: Run 'oncheck -pt 7341178' 00:15:46 stack trace for pid 8370 written to /opt/informix/informixdumps/af.43632bf 00:15:47 See Also: /opt/informix/informixdumps/af.43632bf 00:15:48 pthdrpage:ptalloc:bad bfget 00:15:49 Rollforward of log record failed. iserrno = 172 00:15:49 Log Record: log = 1265071, pos = 0xf0684, type = OLDRSAM:ADDITEM(28), trans = 241 00:15:49 Rollforward of log record failed. iserrno = 172 00:15:49 Log Record: log = 1265071, pos = 0xf0684, type = OLDRSAM:ADDITEM(28), trans = 241 Is this an issue with the disk, or possibly the database itself is corrupted? Is there any way for me to get the instance online quickly so that we can take a look at which tables are affected by this? Does it look like the data is corrupted and we won't be able to get it back? Any insight to this would be helpful. Informix version: 9.40.HC6X1 OS Version: HP UX-11i Keith Schleicher IT Database Administrator Cell: 224-210-8358 Blackberry: 2242108358@messaging.sprintpcs.com<mailto:2242108358@messaging.sprintpcs.com> Page: 2242108358@sprint.skytel.com<mailto:2242108358@sprint.skytel.com> For more information, use our DBA Wiki page link below: http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Management#Dat abaseManagement<http://wiki.intra.sears.com/confluence/display/TechStrag/Databas e+Management> This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you.

Hi, did you execute the oncheck command which was in the log ? What's the output ? Marcus ----- Ursprüngliche Mail ----- Von: "Keith Schleicher" <Keith.Schleicher@searshc.com> An: ids@iiug.org Gesendet: Dienstag, 22. Januar 2013 09:23:04 Betreff: Rollforward of logs failing [29349] We had a power outage that affected one of our Informix instances. We tried bringing it up, but the database seems to be stuck in Fast Recovery mode and I am getting the following error messages: 00:15:46 I/O read chunk 19, pagenum 836337, pagecnt 1 --> errno = 28 00:15:46 Assert Warning: pthdrpage:ptalloc:bad bfget 00:15:46 IBM Informix Dynamic Server Version 9.40.HC6X1 00:15:46 Who: Session(19, informix@spkwms01, 0, c844ccb8) Thread(78, xchg_1.8, c84271d4, 7) File: rspartn.c Line: 5899 00:15:46 Results: Cannot use TBLSpace page for TBLSpace 7341178 00:15:46 Action: Run 'oncheck -pt 7341178' 00:15:46 stack trace for pid 8370 written to /opt/informix/informixdumps/af.43632bf 00:15:47 See Also: /opt/informix/informixdumps/af.43632bf 00:15:48 pthdrpage:ptalloc:bad bfget 00:15:49 Rollforward of log record failed. iserrno = 172 00:15:49 Log Record: log = 1265071, pos = 0xf0684, type = OLDRSAM:ADDITEM(28), trans = 241 00:15:49 Rollforward of log record failed. iserrno = 172 00:15:49 Log Record: log = 1265071, pos = 0xf0684, type = OLDRSAM:ADDITEM(28), trans = 241 Is this an issue with the disk, or possibly the database itself is corrupted? Is there any way for me to get the instance online quickly so that we can take a look at which tables are affected by this? Does it look like the data is corrupted and we won't be able to get it back? Any insight to this would be helpful. Informix version: 9.40.HC6X1 OS Version: HP UX-11i Keith Schleicher IT Database Administrator Cell: 224-210-8358 Blackberry: 2242108358@messaging.sprintpcs.com<mailto:2242108358@messaging.sprintpcs.com> Page: 2242108358@sprint.skytel.com<mailto:2242108358@sprint.skytel.com> For more information, use our DBA Wiki page link below: http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Management#Dat abaseManagement<http://wiki.intra.sears.com/confluence/display/TechStrag/Databas e+Management> This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you. ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Here is the output from "oncheck -pt 7341178": TBLspace Report for Unknown:Unknown.70047a ISAM error: no record found. Error opening TBLspace 70047a. Keith Schleicher IT Database Administrator Cell: 224-210-8358 Blackberry: 2242108358@messaging.sprintpcs.com Page: 2242108358@sprint.skytel.com=A0 For more information, use our DBA Wiki page link below: http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Managemen= t#DatabaseManagement -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Marcu= s Haarmann Sent: Tuesday, January 22, 2013 3:31 AM To: ids@iiug.org Subject: Re: Rollforward of logs failing [29350] Hi,=20 did you execute the oncheck command which was in the log ?=20 What's the output ?=20 Marcus=20 ----- Urspr=FCngliche Mail -----=20 Von: "Keith Schleicher" <Keith.Schleicher@searshc.com> An: ids@iiug.org Gesendet: Dienstag, 22. Januar 2013 09:23:04 Betreff: Rollforward of logs failing [29349]=20 We had a power outage that affected one of our Informix instances. We tried= bringing it up, but the database seems to be stuck in Fast Recovery mode a= nd I am getting the following error messages:=20 00:15:46 I/O read chunk 19, pagenum 836337, pagecnt 1 --> errno =3D 28 00:15:46 Assert Warning: pthdrpage:ptalloc:bad bfget 00:15:46 IBM Informix Dynamic Server Version 9.40.HC6X1 00:15:46 Who: Session(19, informix@spkwms01, 0, c844ccb8)=20 Thread(78, xchg_1.8, c84271d4, 7)=20 File: rspartn.c Line: 5899 00:15:46 Results: Cannot use TBLSpace page for TBLSpace 7341178 00:15:46 Action: Run 'oncheck -pt 7341178'=20 00:15:46 stack trace for pid 8370 written to /opt/informix/informixdumps/af= .43632bf 00:15:47 See Also: /opt/informix/informixdumps/af.43632bf 00:15:48 pthdrpage:ptalloc:bad bfget 00:15:49 Rollforward of log record failed. iserrno =3D 172 00:15:49 Log Record: log =3D 1265071, pos =3D 0xf0684, type =3D OLDRSAM:ADD= ITEM(28), trans =3D 241 00:15:49 Rollforward of log record failed. iserrno =3D 172 00:15:49 Log Record: log =3D 1265071, pos =3D 0xf0684, type =3D OLDRSAM:ADD= ITEM(28), trans =3D 241=20 Is this an issue with the disk, or possibly the database itself is corrupte= d?=20 Is there any way for me to get the instance online quickly so that we can t= ake a look at which tables are affected by this? Does it look like the data= is corrupted and we won't be able to get it back? Any insight to this woul= d be helpful.=20 Informix version: 9.40.HC6X1 OS Version: HP UX-11i=20 Keith Schleicher IT Database Administrator Cell: 224-210-8358 Blackberry:=20 2242108358@messaging.sprintpcs.com<mailto:2242108358@messaging.sprintpcs.co= m> Page: 2242108358@sprint.skytel.com<mailto:2242108358@sprint.skytel.com>=20 For more information, use our DBA Wiki page link below:=20 http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Managemen= t#DatabaseManagement<http://wiki.intra.sears.com/confluence/display/TechStr= ag/Database+Management>=20 This message, including any attachments, is the property of Sears Holdings = Corporation and/or one of its subsidiaries. It is confidential and may cont= ain proprietary or legally privileged information. If you are not the inten= ded recipient, please delete it without reading the contents. Thank you.=20 ***************************************************************************= **** Forum Note: Use "Reply" to post a response in the discussion forum.=20 ***************************************************************************= ****=20 Forum Note: Use "Reply" to post a response in the discussion forum.=20 This message, including any attachments, is the property of Sears Holdings = Corporation and/or one of its subsidiaries. It is confidential and may cont= ain proprietary or legally privileged information. If you are not the inten= ded recipient, please delete it without reading the contents. Thank you.

Hi, Then I would say you have a real issue which can be solved by IBM support only. (or by a restore including logs). I assume when the outage occurred, a checkpoint was in progress, so part of the data was written to disks and partly not. We once had a similar situation when a technical guy pulled a fibre channel cable while the DB was running. We were getting "missing logfile" and also hints that a table partition was not found. Only way to get out of this was to restore the instance. Hopefully you can try to salvage unsaved logs when restoring, which should give you the latest version before the crash. Good luck, Marcus ----- Ursprüngliche Mail ----- Von: "Keith Schleicher" <Keith.Schleicher@searshc.com> An: ids@iiug.org Gesendet: Dienstag, 22. Januar 2013 09:36:08 Betreff: RE: Rollforward of logs failing [29351] Here is the output from "oncheck -pt 7341178": TBLspace Report for Unknown:Unknown.70047a ISAM error: no record found. Error opening TBLspace 70047a. Keith Schleicher IT Database Administrator Cell: 224-210-8358 Blackberry: 2242108358@messaging.sprintpcs.com Page: 2242108358@sprint.skytel.com=A0 For more information, use our DBA Wiki page link below: http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Managemen= t#DatabaseManagement -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Marcu= s Haarmann Sent: Tuesday, January 22, 2013 3:31 AM To: ids@iiug.org Subject: Re: Rollforward of logs failing [29350] Hi,=20 did you execute the oncheck command which was in the log ?=20 What's the output ?=20 Marcus=20 ----- Urspr=FCngliche Mail -----=20 Von: "Keith Schleicher" <Keith.Schleicher@searshc.com> An: ids@iiug.org Gesendet: Dienstag, 22. Januar 2013 09:23:04 Betreff: Rollforward of logs failing [29349]=20 We had a power outage that affected one of our Informix instances. We tried= bringing it up, but the database seems to be stuck in Fast Recovery mode a= nd I am getting the following error messages:=20 00:15:46 I/O read chunk 19, pagenum 836337, pagecnt 1 --> errno =3D 28 00:15:46 Assert Warning: pthdrpage:ptalloc:bad bfget 00:15:46 IBM Informix Dynamic Server Version 9.40.HC6X1 00:15:46 Who: Session(19, informix@spkwms01, 0, c844ccb8)=20 Thread(78, xchg_1.8, c84271d4, 7)=20 File: rspartn.c Line: 5899 00:15:46 Results: Cannot use TBLSpace page for TBLSpace 7341178 00:15:46 Action: Run 'oncheck -pt 7341178'=20 00:15:46 stack trace for pid 8370 written to /opt/informix/informixdumps/af= ..43632bf 00:15:47 See Also: /opt/informix/informixdumps/af.43632bf 00:15:48 pthdrpage:ptalloc:bad bfget 00:15:49 Rollforward of log record failed. iserrno =3D 172 00:15:49 Log Record: log =3D 1265071, pos =3D 0xf0684, type =3D OLDRSAM:ADD= ITEM(28), trans =3D 241 00:15:49 Rollforward of log record failed. iserrno =3D 172 00:15:49 Log Record: log =3D 1265071, pos =3D 0xf0684, type =3D OLDRSAM:ADD= ITEM(28), trans =3D 241=20 Is this an issue with the disk, or possibly the database itself is corrupte= d?=20 Is there any way for me to get the instance online quickly so that we can t= ake a look at which tables are affected by this? Does it look like the data= is corrupted and we won't be able to get it back? Any insight to this woul= d be helpful.=20 Informix version: 9.40.HC6X1 OS Version: HP UX-11i=20 Keith Schleicher IT Database Administrator Cell: 224-210-8358 Blackberry:=20 2242108358@messaging.sprintpcs.com<mailto:2242108358@messaging.sprintpcs.co= m> Page: 2242108358@sprint.skytel.com<mailto:2242108358@sprint.skytel.com>=20 For more information, use our DBA Wiki page link below:=20 http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Managemen= t#DatabaseManagement<http://wiki.intra.sears.com/confluence/display/TechStr= ag/Database+Management>=20 This message, including any attachments, is the property of Sears Holdings = Corporation and/or one of its subsidiaries. It is confidential and may cont= ain proprietary or legally privileged information. If you are not the inten= ded recipient, please delete it without reading the contents. Thank you.=20 ***************************************************************************= **** Forum Note: Use "Reply" to post a response in the discussion forum.=20 ***************************************************************************= ****=20 Forum Note: Use "Reply" to post a response in the discussion forum.=20 This message, including any attachments, is the property of Sears Holdings = Corporation and/or one of its subsidiaries. It is confidential and may cont= ain proprietary or legally privileged information. If you are not the inten= ded recipient, please delete it without reading the contents. Thank you. ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

The database looks like it has finished its rolling forward, but I am getti= ng the message "oninit pid waiting for debugger." Is there any way to kick= this off manually? Keith Schleicher IT Database Administrator Cell: 224-210-8358 Blackberry: 2242108358@messaging.sprintpcs.com Page: 2242108358@sprint.skytel.com=A0 For more information, use our DBA Wiki page link below: http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Managemen= t#DatabaseManagement -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Marcu= s Haarmann Sent: Tuesday, January 22, 2013 4:12 AM To: ids@iiug.org Subject: Re: Rollforward of logs failing [29352] Hi,=20 Then I would say you have a real issue which can be solved by IBM support o= nly.=20 (or by a restore including logs).=20 I assume when the outage occurred, a checkpoint was in progress, so part of= the data was written to disks and partly not. We once had a similar situat= ion when a technical guy pulled a fibre channel cable while the DB was runn= ing. We were getting "missing logfile"=20 and also hints that a table partition was not found.=20 Only way to get out of this was to restore the instance. Hopefully you can = try to salvage unsaved logs when restoring, which should give you the lates= t version before the crash.=20 Good luck,=20 Marcus=20 ----- Urspr=FCngliche Mail -----=20 Von: "Keith Schleicher" <Keith.Schleicher@searshc.com> An: ids@iiug.org Gesendet: Dienstag, 22. Januar 2013 09:36:08 Betreff: RE: Rollforward of logs failing [29351]=20 Here is the output from "oncheck -pt 7341178":=20 TBLspace Report for Unknown:Unknown.70047a=20 ISAM error: no record found.=20 Error opening TBLspace 70047a.=20 Keith Schleicher IT Database Administrator Cell: 224-210-8358 Blackberry: 2242108358@messaging.sprintpcs.com Page: 2242108358@sprint.skytel.com=3DA0=20 For more information, use our DBA Wiki page link below:=20 http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Managemen= =3D t#DatabaseManagement=20 -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Marcu= =3D s Haarmann Sent: Tuesday, January 22, 2013 3:31 AM To: ids@iiug.org Subject: Re: Rollforward of logs failing [29350]=20 Hi,=3D20=20 did you execute the oncheck command which was in the log ?=3D20 What's the = output ?=3D20=20 Marcus=3D20=20 ----- Urspr=3DFCngliche Mail -----=3D20=20 Von: "Keith Schleicher" <Keith.Schleicher@searshc.com> An: ids@iiug.org Gesendet: Dienstag, 22. Januar 2013 09:23:04 Betreff: Rollforward of logs failing [29349]=3D20=20 We had a power outage that affected one of our Informix instances. We tried= =3D bringing it up, but the database seems to be stuck in Fast Recovery mod= e a=3D nd I am getting the following error messages:=3D20=20 00:15:46 I/O read chunk 19, pagenum 836337, pagecnt 1 --> errno =3D3D 28 00:15:46 Assert Warning: pthdrpage:ptalloc:bad bfget 00:15:46 IBM Informix Dynamic Server Version 9.40.HC6X1 00:15:46 Who: Session(19, informix@spkwms01, 0, c844ccb8)=3D20=20 Thread(78, xchg_1.8, c84271d4, 7)=3D20=20 File: rspartn.c Line: 5899 00:15:46 Results: Cannot use TBLSpace page for TBLSpace 7341178 00:15:46 Action: Run 'oncheck -pt 7341178'=3D20 00:15:46 stack trace for pid 8370 written to /opt/informix/informixdumps/af= =3D ...43632bf 00:15:47 See Also: /opt/informix/informixdumps/af.43632bf 00:15:48 pthdrpage:ptalloc:bad bfget 00:15:49 Rollforward of log record failed. iserrno =3D3D 172 00:15:49 Log Record: log =3D3D 1265071, pos =3D3D 0xf0684, type =3D3D OLDRS= AM:ADD=3D ITEM(28), trans =3D3D 241 00:15:49 Rollforward of log record failed. iserrno =3D3D 172 00:15:49 Log Record: log =3D3D 1265071, pos =3D3D 0xf0684, type =3D3D OLDRS= AM:ADD=3D ITEM(28), trans =3D3D 241=3D20=20 Is this an issue with the disk, or possibly the database itself is corrupte= =3D d?=3D20 Is there any way for me to get the instance online quickly so that we can t= =3D ake a look at which tables are affected by this? Does it look like the = data=3D is corrupted and we won't be able to get it back? Any insight to th= is woul=3D d be helpful.=3D20=20 Informix version: 9.40.HC6X1 OS Version: HP UX-11i=3D20=20 Keith Schleicher IT Database Administrator Cell: 224-210-8358 Blackberry:=3D20 2242108358@messaging.sprintpcs.com<mailto:2242108358@messaging.sprintpcs.co= =3D=20 m>=20 Page: 2242108358@sprint.skytel.com<mailto:2242108358@sprint.skytel.com>=3D2= 0=20 For more information, use our DBA Wiki page link below:=3D20=20 http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Managemen= =3D t#DatabaseManagement<http://wiki.intra.sears.com/confluence/display/TechStr= =3D ag/Database+Management>=3D20=20 This message, including any attachments, is the property of Sears Holdings = =3D Corporation and/or one of its subsidiaries. It is confidential and may = cont=3D ain proprietary or legally privileged information. If you are not t= he inten=3D ded recipient, please delete it without reading the contents. T= hank you.=3D20=20 ***************************************************************************= =3D **** Forum Note: Use "Reply" to post a response in the discussion forum.=3D20=20 ***************************************************************************= =3D ****=3D20 Forum Note: Use "Reply" to post a response in the discussion forum.=3D20=20 This message, including any attachments, is the property of Sears Holdings = =3D Corporation and/or one of its subsidiaries. It is confidential and may = cont=3D ain proprietary or legally privileged information. If you are not t= he inten=3D ded recipient, please delete it without reading the contents. T= hank you.=20 ***************************************************************************= **** Forum Note: Use "Reply" to post a response in the discussion forum.=20 ***************************************************************************= ****=20 Forum Note: Use "Reply" to post a response in the discussion forum.=20 This message, including any attachments, is the property of Sears Holdings = Corporation and/or one of its subsidiaries. It is confidential and may cont= ain proprietary or legally privileged information. If you are not the inten= ded recipient, please delete it without reading the contents. Thank you.

This is odd. The errno = 28 that is reported below means "no space left on device" on almost all UNIX variants including HPUX. What tha has to do with the read operation that apparently failed, I don't know. Check that all of the chunk file paths exist, that if they are links the target files or devices are there, that the privileges on the target files/devices are correct (0660), and that you can read them (us "dd if=<chunkpath> of=/dev/null bs=2k count=1000" to test the files/devices). If all is correctly mounted but you get errors from dd, then yes your disks are possibly hosed. Do you have an archive? Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Tue, Jan 22, 2013 at 3:23 AM, Schleicher, Keith < Keith.Schleicher@searshc.com> wrote: > We had a power outage that affected one of our Informix instances. We tried > bringing it up, but the database seems to be stuck in Fast Recovery mode > and I > am getting the following error messages: > > 00:15:46 I/O read chunk 19, pagenum 836337, pagecnt 1 --> errno = 28 > 00:15:46 Assert Warning: pthdrpage:ptalloc:bad bfget > 00:15:46 IBM Informix Dynamic Server Version 9.40.HC6X1 > 00:15:46 Who: Session(19, informix@spkwms01, 0, c844ccb8) > > Thread(78, xchg_1.8, c84271d4, 7) > > File: rspartn.c Line: 5899 > 00:15:46 Results: Cannot use TBLSpace page for TBLSpace 7341178 > 00:15:46 Action: Run 'oncheck -pt 7341178' > 00:15:46 stack trace for pid 8370 written to > /opt/informix/informixdumps/af.43632bf > 00:15:47 See Also: /opt/informix/informixdumps/af.43632bf > 00:15:48 pthdrpage:ptalloc:bad bfget > 00:15:49 Rollforward of log record failed. iserrno = 172 > 00:15:49 Log Record: log = 1265071, pos = 0xf0684, type = > OLDRSAM:ADDITEM(28), > trans = 241 > 00:15:49 Rollforward of log record failed. iserrno = 172 > 00:15:49 Log Record: log = 1265071, pos = 0xf0684, type = > OLDRSAM:ADDITEM(28), > trans = 241 > > Is this an issue with the disk, or possibly the database itself is > corrupted? > Is there any way for me to get the instance online quickly so that we can > take > a look at which tables are affected by this? Does it look like the data is > corrupted and we won't be able to get it back? Any insight to this would be > helpful. > > Informix version: 9.40.HC6X1 > OS Version: HP UX-11i > > Keith Schleicher > IT Database Administrator > Cell: 224-210-8358 > Blackberry: > 2242108358@messaging.sprintpcs.com<mailto: > 2242108358@messaging.sprintpcs.com> > Page: 2242108358@sprint.skytel.com<mailto:2242108358@sprint.skytel.com> > > For more information, use our DBA Wiki page link below: > > > http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Management#Dat abaseManagement > < > http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Management > > > > This message, including any attachments, is the property of Sears Holdings > Corporation and/or one of its subsidiaries. It is confidential and may > contain > proprietary or legally privileged information. If you are not the intended > recipient, please delete it without reading the contents. Thank you. > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --f46d0447882db7b3a404d3e1c732

We do have a backup, but we discovered that there were some mismatch errors= on the disk array. Once those were fixed the database instance came up wi= thout issue. Keith Schleicher IT Database Administrator Cell: 224-210-8358 Blackberry: 2242108358@messaging.sprintpcs.com Page: 2242108358@sprint.skytel.com=A0 For more information, use our DBA Wiki page link below: http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Managemen= t#DatabaseManagement -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Art K= agel Sent: Tuesday, January 22, 2013 9:56 AM To: ids@iiug.org Subject: Re: Rollforward of logs failing [29356] This is odd. The errno =3D 28 that is reported below means "no space left o= n device" on almost all UNIX variants including HPUX. What tha has to do wi= th the read operation that apparently failed, I don't know.=20 Check that all of the chunk file paths exist, that if they are links the ta= rget files or devices are there, that the privileges on the target files/de= vices are correct (0660), and that you can read them (us "dd if=3D<chunkpat= h> of=3D/dev/null bs=3D2k count=3D1000" to test the files/devices).=20 If all is correctly mounted but you get errors from dd, then yes your disks= are possibly hosed. Do you have an archive?=20 Art=20 Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/=20 Disclaimer: Please keep in mind that my own opinions are my own opinions an= d do not reflect on my employer, Advanced DataTools, the IIUG, nor any othe= r organization with which I am associated either explicitly, implicitly, or= by inference. Neither do those opinions reflect those of other individuals= affiliated with any entity with which I am affiliated nor those of the ent= ities themselves.=20 On Tue, Jan 22, 2013 at 3:23 AM, Schleicher, Keith < Keith.Schleicher@sears= hc.com> wrote:=20 > We had a power outage that affected one of our Informix instances. We=20 > tried bringing it up, but the database seems to be stuck in Fast=20 > Recovery mode and I am getting the following error messages: >=20 > 00:15:46 I/O read chunk 19, pagenum 836337, pagecnt 1 --> errno =3D 28 > 00:15:46 Assert Warning: pthdrpage:ptalloc:bad bfget > 00:15:46 IBM Informix Dynamic Server Version 9.40.HC6X1 > 00:15:46 Who: Session(19, informix@spkwms01, 0, c844ccb8) >=20 > Thread(78, xchg_1.8, c84271d4, 7) >=20 > File: rspartn.c Line: 5899 > 00:15:46 Results: Cannot use TBLSpace page for TBLSpace 7341178 > 00:15:46 Action: Run 'oncheck -pt 7341178'=20 > 00:15:46 stack trace for pid 8370 written to=20 > /opt/informix/informixdumps/af.43632bf > 00:15:47 See Also: /opt/informix/informixdumps/af.43632bf > 00:15:48 pthdrpage:ptalloc:bad bfget > 00:15:49 Rollforward of log record failed. iserrno =3D 172 > 00:15:49 Log Record: log =3D 1265071, pos =3D 0xf0684, type =3D=20 > OLDRSAM:ADDITEM(28), trans =3D 241 > 00:15:49 Rollforward of log record failed. iserrno =3D 172 > 00:15:49 Log Record: log =3D 1265071, pos =3D 0xf0684, type =3D=20 > OLDRSAM:ADDITEM(28), trans =3D 241 >=20 > Is this an issue with the disk, or possibly the database itself is=20 > corrupted? > Is there any way for me to get the instance online quickly so that we=20 > can take a look at which tables are affected by this? Does it look=20 > like the data is corrupted and we won't be able to get it back? Any=20 > insight to this would be helpful. >=20 > Informix version: 9.40.HC6X1 > OS Version: HP UX-11i >=20 > Keith Schleicher > IT Database Administrator > Cell: 224-210-8358 > Blackberry:=20 > 2242108358@messaging.sprintpcs.com<mailto:=20 > 2242108358@messaging.sprintpcs.com> > Page:=20 > 2242108358@sprint.skytel.com<mailto:2242108358@sprint.skytel.com> >=20 > For more information, use our DBA Wiki page link below:=20 >=20 >=20 >=20 http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Managemen= t#DatabaseManagement=20 > < > http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Mana > gement > >=20 >=20 > This message, including any attachments, is the property of Sears=20 > Holdings Corporation and/or one of its subsidiaries. It is=20 > confidential and may contain proprietary or legally privileged=20 > information. If you are not the intended recipient, please delete it=20 > without reading the contents. Thank you. >=20 >=20 >=20 >=20 ***************************************************************************= ****=20 > Forum Note: Use "Reply" to post a response in the discussion forum.=20 >=20 >=20 --f46d0447882db7b3a404d3e1c732=20 ***************************************************************************= **** Forum Note: Use "Reply" to post a response in the discussion forum.=20 This message, including any attachments, is the property of Sears Holdings = Corporation and/or one of its subsidiaries. It is confidential and may cont= ain proprietary or legally privileged information. If you are not the inten= ded recipient, please delete it without reading the contents. Thank you.

Great news! I suppose I don't have to have to tell you that 9.40 is VERY out-of-date and out-of-support. You REALLY should plan an upgrade to a supported release (11.50.xxx or 11.70.xxx). Not only are these versions supported and provide many new features that can improve performance for you, but they are faster by a significant margin than v9.40. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Tue, Jan 22, 2013 at 10:00 AM, Schleicher, Keith < Keith.Schleicher@searshc.com> wrote: > We do have a backup, but we discovered that there were some mismatch > errors= > on the disk array. Once those were fixed the database instance came up wi= > thout issue. > > Keith Schleicher > IT Database Administrator > Cell: 224-210-8358 > Blackberry: 2242108358@messaging.sprintpcs.com > Page: 2242108358@sprint.skytel.com=A0 > > For more information, use our DBA Wiki page link below: > > http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Managemen= > t#DatabaseManagement > > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Art > K= > agel > Sent: Tuesday, January 22, 2013 9:56 AM > To: ids@iiug.org > Subject: Re: Rollforward of logs failing [29356] > > This is odd. The errno =3D 28 that is reported below means "no space left > o= > n device" on almost all UNIX variants including HPUX. What tha has to do > wi= > th the read operation that apparently failed, I don't know.=20 > > Check that all of the chunk file paths exist, that if they are links the > ta= > rget files or devices are there, that the privileges on the target > files/de= > vices are correct (0660), and that you can read them (us "dd > if=3D<chunkpat= > h> of=3D/dev/null bs=3D2k count=3D1000" to test the files/devices).=20 > If all is correctly mounted but you get errors from dd, then yes your > disks= > are possibly hosed. Do you have an archive?=20 > > Art=20 > > Art S. Kagel > Advanced DataTools (www.advancedatatools.com) > Blog: http://informix-myview.blogspot.com/=20 > > Disclaimer: Please keep in mind that my own opinions are my own opinions > an= > d do not reflect on my employer, Advanced DataTools, the IIUG, nor any > othe= > r organization with which I am associated either explicitly, implicitly, > or= > by inference. Neither do those opinions reflect those of other individuals= > affiliated with any entity with which I am affiliated nor those of the ent= > ities themselves.=20 > > On Tue, Jan 22, 2013 at 3:23 AM, Schleicher, Keith < Keith.Schleicher@sears > = > hc.com> wrote:=20 > > > We had a power outage that affected one of our Informix instances. We=20 > > tried bringing it up, but the database seems to be stuck in Fast=20 > > Recovery mode and I am getting the following error messages: > >=20 > > 00:15:46 I/O read chunk 19, pagenum 836337, pagecnt 1 --> errno =3D 28 > > 00:15:46 Assert Warning: pthdrpage:ptalloc:bad bfget > > 00:15:46 IBM Informix Dynamic Server Version 9.40.HC6X1 > > 00:15:46 Who: Session(19, informix@spkwms01, 0, c844ccb8) > >=20 > > Thread(78, xchg_1.8, c84271d4, 7) > >=20 > > File: rspartn.c Line: 5899 > > 00:15:46 Results: Cannot use TBLSpace page for TBLSpace 7341178 > > 00:15:46 Action: Run 'oncheck -pt 7341178'=20 > > 00:15:46 stack trace for pid 8370 written to=20 > > /opt/informix/informixdumps/af.43632bf > > 00:15:47 See Also: /opt/informix/informixdumps/af.43632bf > > 00:15:48 pthdrpage:ptalloc:bad bfget > > 00:15:49 Rollforward of log record failed. iserrno =3D 172 > > 00:15:49 Log Record: log =3D 1265071, pos =3D 0xf0684, type =3D=20 > > OLDRSAM:ADDITEM(28), trans =3D 241 > > 00:15:49 Rollforward of log record failed. iserrno =3D 172 > > 00:15:49 Log Record: log =3D 1265071, pos =3D 0xf0684, type =3D=20 > > OLDRSAM:ADDITEM(28), trans =3D 241 > >=20 > > Is this an issue with the disk, or possibly the database itself is=20 > > corrupted? > > Is there any way for me to get the instance online quickly so that we=20 > > can take a look at which tables are affected by this? Does it look=20 > > like the data is corrupted and we won't be able to get it back? Any=20 > > insight to this would be helpful. > >=20 > > Informix version: 9.40.HC6X1 > > OS Version: HP UX-11i > >=20 > > Keith Schleicher > > IT Database Administrator > > Cell: 224-210-8358 > > Blackberry:=20 > > 2242108358@messaging.sprintpcs.com<mailto:=20 > > 2242108358@messaging.sprintpcs.com> > > Page:=20 > > 2242108358@sprint.skytel.com<mailto:2242108358@sprint.skytel.com> > >=20 > > For more information, use our DBA Wiki page link below:=20 > >=20 > >=20 > >=20 > > http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Managemen= > t#DatabaseManagement=20 > > < > > http://wiki.intra.sears.com/confluence/display/TechStrag/Database+Mana > > gement > > >=20 > >=20 > > This message, including any attachments, is the property of Sears=20 > > Holdings Corporation and/or one of its subsidiaries. It is=20 > > confidential and may contain proprietary or legally privileged=20 > > information. If you are not the intended recipient, please delete it=20 > > without reading the contents. Thank you. > >=20 > >=20 > >=20 > >=20 > > ***************************************************************************= > ****=20 > > Forum Note: Use "Reply" to post a response in the discussion forum.=20 > >=20 > >=20 > > --f46d0447882db7b3a404d3e1c732=20 > > > ***************************************************************************= > **** > Forum Note: Use "Reply" to post a response in the discussion forum.=20 > > This message, including any attachments, is the property of Sears Holdings > = > Corporation and/or one of its subsidiaries. It is confidential and may > cont= > ain proprietary or legally privileged information. If you are not the > inten= > ded recipient, please delete it without reading the contents. Thank you. > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --f46d0447f05e64b6c804d3e1f2a4