Unix administrator has just ruined my life !
A sysadmin reclaimed raw disk devices that Informix was using, so three dbspaces (including the one holding the logical logs) vanished and oninit failed with "Cannot Open Primary Chunk ... errno = 2" and a fatal shared-memory initialization error. Replies agreed that simply recreating an empty logspace chunk won't work: you must recreate all missing chunks with exactly the same paths (symbolic links make this easier), then restore from a level 0 archive; chunk layout info can be read from the ontape header. Side discussion warned against cooked files, which admins may delete. No confirmation from the poster that recovery succeeded is recorded.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Hello Potential Saviours,
It transpires that throughout today, our unix adminstrator was trying to
create some more space for, beleive it or not, a migration of (Ardent)
PiOpen from an old DG Box onto our spanking Sun Enterprise 3000. Prior to
last week the Sun box was mainly used to hold the informix engine.
Well, today, the lousy low down stinking s**t, decided that he would grab
some 'spare disk' from my lovely collection of raw disk, and at around
16:18, I started getting assert failures etc.
After him saying that he hadn't changed anything, he decided to reboot the
server, and therefor I had to bring informix down. In my stupidity, I had
the DUMPDIR set to /tmp - this has now changed, as before I could even copy
the shm failures and the assert failures to a different directory, the
system was on its way down.
So, my problem is, I have 3 dbspaces which were there, and are not there
now.
One of the dbspaces was for testing and therefor not really important, one
was data that had been live up to about 2 months ago, and finally the last
dbspace was my logspace.
All the rest of the 'important' dbspaces are ok, so how does one go about
bringing this thing back to life.
DATASKIP is off and ONDBSPACEDOWN is 0
from the online.log
18:25:06 Cannot Open Primary Chunk '/dev/vx/rdsk/datadg/vol02', errno = 2
18:25:06 Cannot Open Primary Chunk '/dev/vx/rdsk/datadg/vol01', errno = 2
18:25:06 Cannot Open Primary Chunk '/dev/vx/rdsk/datadg/vol04', errno = 2
from onmonitor
oninit: Cannot open chunk '/dev/vx/rdsk/datadg/vol02'. errno = 2
oninit: Cannot open chunk '/dev/vx/rdsk/datadg/vol01'. errno = 2
oninit: Cannot open chunk '/dev/vx/rdsk/datadg/vol04'. errno = 2
Can Not Open Primary Chunk /dev/vx/rdsk/datadg/vol02 ***** this is the one
where my inflogs are *******
oninit: Fatal error in shared memory initialization
Will it be ok to recreate a raw dbspace for my logspace inflogs with the
same path, and will this bring the engine back up ?
Please Help
Regards,
David
david.mcallister@soi.co.uk wrote:
> Hello Potential Saviours,
>
> It transpires that throughout today, our unix adminstrator was trying to
> create some more space for, beleive it or not, a migration of (Ardent)
> PiOpen from an old DG Box onto our spanking Sun Enterprise 3000. Prior to
> last week the Sun box was mainly used to hold the informix engine.
>
> Well, today, the lousy low down stinking s**t, decided that he would grab
> some 'spare disk' from my lovely collection of raw disk, and at around
> 16:18, I started getting assert failures etc.
>
> After him saying that he hadn't changed anything, he decided to reboot the
> server, and therefor I had to bring informix down. In my stupidity, I had
> the DUMPDIR set to /tmp - this has now changed, as before I could even copy
> the shm failures and the assert failures to a different directory, the
> system was on its way down.
>
> So, my problem is, I have 3 dbspaces which were there, and are not there
> now.
> One of the dbspaces was for testing and therefor not really important, one
> was data that had been live up to about 2 months ago, and finally the last
> dbspace was my logspace.
>
> All the rest of the 'important' dbspaces are ok, so how does one go about
> bringing this thing back to life.
> DATASKIP is off and ONDBSPACEDOWN is 0
>
> from the online.log
> 18:25:06 Cannot Open Primary Chunk '/dev/vx/rdsk/datadg/vol02', errno = 2
> 18:25:06 Cannot Open Primary Chunk '/dev/vx/rdsk/datadg/vol01', errno = 2
> 18:25:06 Cannot Open Primary Chunk '/dev/vx/rdsk/datadg/vol04', errno = 2>
> from onmonitor
> oninit: Cannot open chunk '/dev/vx/rdsk/datadg/vol02'. errno = 2
> oninit: Cannot open chunk '/dev/vx/rdsk/datadg/vol01'. errno = 2
> oninit: Cannot open chunk '/dev/vx/rdsk/datadg/vol04'. errno = 2
> Can Not Open Primary Chunk /dev/vx/rdsk/datadg/vol02 ***** this is the one
> where my inflogs are *******
> oninit: Fatal error in shared memory initialization
>
> Will it be ok to recreate a raw dbspace for my logspace inflogs with the
> same path, and will this bring the engine back up ?
No. You need to re-create all of the missing chunks and then do a restore from
your latest backup. Depending on your version, you can restore just the missing
dbspaces or all of them.
You DO have a backup, right?
Doug McAllister
>
>
> Please Help
>
> Regards,
>
> David
As Doug McAllister (your brother presumably?) says, you'll have to re-create
your chunks with the exact names you had previously (soft links, as
recommended in TFM, will make this straightforward), then restore from a
level 0 archive.
Neil Truby
Londis Stores
Hampton Hill, UK
david.mcallister@soi.co.uk wrote in message
<82h5gt$aap$1@news.xmission.com>...
>
>Hello Potential Saviours,
>
>It transpires that throughout today, our unix adminstrator was trying to
>create some more space for, beleive it or not, a migration of (Ardent)
>PiOpen from an old DG Box onto our spanking Sun Enterprise 3000. Prior to
>last week the Sun box was mainly used to hold the informix engine.
>
>Well, today, the lousy low down stinking s**t, decided that he would grab
>some 'spare disk' from my lovely collection of raw disk, and at around
>16:18, I started getting assert failures etc.
>
>After him saying that he hadn't changed anything, he decided to reboot the
>server, and therefor I had to bring informix down. In my stupidity, I had
>the DUMPDIR set to /tmp - this has now changed, as before I could even copy
>the shm failures and the assert failures to a different directory, the
>system was on its way down.
>
>So, my problem is, I have 3 dbspaces which were there, and are not there
>now.
>One of the dbspaces was for testing and therefor not really important, one
>was data that had been live up to about 2 months ago, and finally the last
>dbspace was my logspace.
>
>All the rest of the 'important' dbspaces are ok, so how does one go about
>bringing this thing back to life.
>DATASKIP is off and ONDBSPACEDOWN is 0
>
>from the online.log
>18:25:06 Cannot Open Primary Chunk '/dev/vx/rdsk/datadg/vol02', errno = 2
>18:25:06 Cannot Open Primary Chunk '/dev/vx/rdsk/datadg/vol01', errno = 2
>18:25:06 Cannot Open Primary Chunk '/dev/vx/rdsk/datadg/vol04', errno = 2>
>from onmonitor
>oninit: Cannot open chunk '/dev/vx/rdsk/datadg/vol02'. errno = 2
>oninit: Cannot open chunk '/dev/vx/rdsk/datadg/vol01'. errno = 2
>oninit: Cannot open chunk '/dev/vx/rdsk/datadg/vol04'. errno = 2
>Can Not Open Primary Chunk /dev/vx/rdsk/datadg/vol02 ***** this is the one
> where my inflogs are *******
>oninit: Fatal error in shared memory initialization
>
>Will it be ok to recreate a raw dbspace for my logspace inflogs with the
>same path, and will this bring the engine back up ?
>
>Please Help
>
>Regards,
>
>David
>
>
If its ontape the necessary info to recreate the chunks is in the
header of the ontape. We have scripts that read headers and build
instance shapes from them here.
When it's sorted I suggest you have a frank exchange of views with your
sysadmin.
mja
In article <82hg48$nep$1@taliesin.netcom.net.uk>,
"Neil Truby" <ntruby@netcomuk.co.uk> wrote:
> As Doug McAllister (your brother presumably?) says, you'll have to re-
create
> your chunks with the exact names you had previously (soft links, as
> recommended in TFM, will make this straightforward), then restore
from a
> level 0 archive.
>
> Neil Truby
> Londis Stores
> Hampton Hill, UK
>
> david.mcallister@soi.co.uk wrote in message
> <82h5gt$aap$1@news.xmission.com>...
> >
> >Hello Potential Saviours,
> >
> >It transpires that throughout today, our unix adminstrator was
trying to
> >create some more space for, beleive it or not, a migration of
(Ardent)
> >PiOpen from an old DG Box onto our spanking Sun Enterprise 3000.
Prior to
> >last week the Sun box was mainly used to hold the informix engine.
> >
> >Well, today, the lousy low down stinking s**t, decided that he would
grab
> >some 'spare disk' from my lovely collection of raw disk, and at
around
> >16:18, I started getting assert failures etc.
> >
> >After him saying that he hadn't changed anything, he decided to
reboot the
> >server, and therefor I had to bring informix down. In my stupidity,
I had
> >the DUMPDIR set to /tmp - this has now changed, as before I could
even copy
> >the shm failures and the assert failures to a different directory,
the
> >system was on its way down.
> >
> >So, my problem is, I have 3 dbspaces which were there, and are not
there
> >now.
> >One of the dbspaces was for testing and therefor not really
important, one
> >was data that had been live up to about 2 months ago, and finally
the last
> >dbspace was my logspace.
> >
> >All the rest of the 'important' dbspaces are ok, so how does one go
about
> >bringing this thing back to life.
> >DATASKIP is off and ONDBSPACEDOWN is 0
> >
> >from the online.log
> >18:25:06 Cannot Open Primary Chunk '/dev/vx/rdsk/datadg/vol02',
errno = 2
> >18:25:06 Cannot Open Primary Chunk '/dev/vx/rdsk/datadg/vol01',
errno = 2
> >18:25:06 Cannot Open Primary Chunk '/dev/vx/rdsk/datadg/vol04',
errno = 2> >
> >from onmonitor
> >oninit: Cannot open chunk '/dev/vx/rdsk/datadg/vol02'. errno = 2
> >oninit: Cannot open chunk '/dev/vx/rdsk/datadg/vol01'. errno = 2
> >oninit: Cannot open chunk '/dev/vx/rdsk/datadg/vol04'. errno = 2
> >Can Not Open Primary Chunk /dev/vx/rdsk/datadg/vol02 ***** this is
the one
> > where my inflogs are *******
> >oninit: Fatal error in shared memory initialization
> >
> >Will it be ok to recreate a raw dbspace for my logspace inflogs with
the
> >same path, and will this bring the engine back up ?
> >
> >Please Help
> >
> >Regards,
> >
> >David
> >
> >
>
>
Sent via Deja.com http://www.deja.com/
Before you buy.
martyn.ayshford@orange.co.uk wrote:
>
> If its ontape the necessary info to recreate the chunks is in the
> header of the ontape. We have scripts that read headers and build
> instance shapes from them here.
>
> When it's sorted I suggest you have a frank exchange of views with your
> sysadmin.
Pistols at 5 paces sounds good to me.
You know I had a very similar incident yesterday. On our Y2K test machine
I did not want to bother the sysadmins to create raw chunks for every
database that someone wanted me to add to the Y2K testing. It's just a
test machine after all and temporary at that. So I just got them to give
me directories on two different huge filesystems and used COOKED space.
Yesterday I discovered that 7 days ago someone deleted all of the chunk
files on one filesystem and 5 days ago they wiped out the other one.
Fortunately the instance has been up all this time and the sysadmins have
not rebooted to reset the system dates again so I was able to take an
archive, recreate the files, and restore. FHEW!
Anyway when it was over I was going to post this anecdote as a warning
to DBAs not to use cooked file space because of the danger of some
sysadmin deciding a filesystem is too full and deleting your HUGE file
which just MUST be some program's runaway output and mixed binary and text
at that! Then I spotted this post and just shook my head. Seems nothing
is sacred any more.
Art S. Kagel
"Art S. Kagel" wrote:
> martyn.ayshford@orange.co.uk wrote:
> >
> > If its ontape the necessary info to recreate the chunks is in the
> > header of the ontape. We have scripts that read headers and build
> > instance shapes from them here.
> >
> > When it's sorted I suggest you have a frank exchange of views with your
> > sysadmin.
>
> Pistols at 5 paces sounds good to me.
>
> You know I had a very similar incident yesterday. On our Y2K test machine
> I did not want to bother the sysadmins to create raw chunks for every
> database that someone wanted me to add to the Y2K testing. It's just a
> test machine after all and temporary at that. So I just got them to give
> me directories on two different huge filesystems and used COOKED space.
> Yesterday I discovered that 7 days ago someone deleted all of the chunk
> files on one filesystem and 5 days ago they wiped out the other one.
> Fortunately the instance has been up all this time and the sysadmins have
> not rebooted to reset the system dates again so I was able to take an
> archive, recreate the files, and restore. FHEW!
>
> Anyway when it was over I was going to post this anecdote as a warning
> to DBAs not to use cooked file space because of the danger of some
> sysadmin deciding a filesystem is too full and deleting your HUGE file
> which just MUST be some program's runaway output and mixed binary and text
> at that! Then I spotted this post and just shook my head. Seems nothing
> is sacred any more.
>
> Art S. Kagel
Maybe the system administrators have decided to launch a war against you guys,
so every body should be on the look out :-) .
--
Compliments of QueriX
--------------------------------------------------------------------------------------------------
QueriX 4GL Compilers are Informix 4GL Compatible, and Connection to other
RDBMS such as Oracle.
Hydra 4GL Compiler Compile once, run everywhere
Phoenix Windows GUI.
Chimera Java GUI The only GUI you will ever need...
Arachne Web Technology
For more details visit: http://www.querix.com/
---------------------------------------------------------------------------------------------------
QueriX 4GL / Mehdi wrote: > > "Art S. Kagel" wrote: > > > martyn.ayshford@orange.co.uk wrote: [SNIP] > Maybe the system administrators have decided to launch a war against you guys, > so every body should be on the look out :-) . I'm starting to like this guy. He's getting into the spirit of this group. Art S. Kagel