Symlinks and informix...
A DBA asked whether chunk symlinks can safely be deleted and recreated while IDS is online, assuming the engine only needs them at startup. Art Kagel explained that IDS opens and closes chunks periodically (not all VPs hold handles), so already-open files survive link removal, but any chunk open attempt while the links are missing would be disastrous. The real cause was an RPM packaging mistake: the DBSPACES directory lived under /usr/informix and was owned by the old version's RPM, so installing the new release deleted it. Advice was to keep dbspaces out of the binary directory (install binaries under a versioned path) and fix the RPM so it never removes the previously installed version, copying chunk links post-install instead. No confirmation of the final fix is recorded.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Hey all.
Now I know that creating symlinks for access to chunks is a really
really good idea and is a practice that we follow religiously.
I have a question though - when does the database engine make use of
them? Is it just at startup when the chunks are going through sanity
checking?
Obviously I'm not counting any interaction with onspaces or admin/task
operation that add/remove/manipulate allocated disk - required then.
The reason I ask is that we are looking at building a script that
destroys and then recreates them while the engine is running.
My assumption is that the engine creates the file handles it requires to
the chunks on startup and then doesn't need them again - so destroying
and recreating them won't cause a problem.
It would be better to do while the engine is off-line, but in this case
we are hoping to avoid the downtime.
We are on RedHat EL4 x86 (11.10.UC2W2 and 11.50.UC3).
Thanks
Jarrod Teale
Team Lead - Manufacturing Execution Systems
Automation & Process Control Group
NZ Technical
Fonterra
Fonterra Extn: 77525
DDI: +64 7 850 7525
Mobile: +64 21 968 364
fax: +64 7 849 7855
email: jarrod.teale@fonterra.com
<mailto:jarrod.teale@fonterra.com>
2009/1/26 Jarrod Teale <Jarrod.Teale@fonterra.com>:
> Hey all.
> Now I know that creating symlinks for access to chunks is a really
> really good idea and is a practice that we follow religiously.
> I have a question though - when does the database engine make use of
> them? Is it just at startup when the chunks are going through sanity
> checking?
> Obviously I'm not counting any interaction with onspaces or admin/task
> operation that add/remove/manipulate allocated disk - required then.
>
> The reason I ask is that we are looking at building a script that
> destroys and then recreates them while the engine is running.
> My assumption is that the engine creates the file handles it requires to
> the chunks on startup and then doesn't need them again - so destroying
> and recreating them won't cause a problem.
>
> It would be better to do while the engine is off-line, but in this case
> we are hoping to avoid the downtime.
>
> We are on RedHat EL4 x86 (11.10.UC2W2 and 11.50.UC3).
>
> Thanks
>
> Jarrod Teale
>
> Team Lead - Manufacturing Execution Systems
>
> Automation & Process Control Group
>
> NZ Technical
>
> Fonterra
>
> Fonterra Extn: 77525
>
> DDI: +64 7 850 7525
>
> Mobile: +64 21 968 364
> fax: +64 7 849 7855
> email: jarrod.teale@fonterra.com
> <mailto:jarrod.teale@fonterra.com>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
Jarrod
Sorry, I'm a bit confused here. You have started the engine accesing
these chunks, via symbolic links, containing all this nice data that
keeps your company running, and keeps you in a job, and then you want
to delete and re-create them ! Why? And you expect the engine to keep
running?
What is your aim, what are you trying to achive? That would be a
better qustion than:- I'm going to do this, will it put me out of a
job.
Keith
Jarrod,
IDS opens and closes the chunks periodically. If you run fuser against the
chunks you will often note that they are not all opened by all of the CPU
VPs. If a VP hasn't needed to perform IO against a chunk it will not have
th chunk open. Existing open files in the engine will not be affected if
you remove the links, and once the links have been reestablished to the
existing chunks (I assume you aren't going to link them elsewhere while the
engine is running) the engine should not notice that they were gone for a
bit, but if the engine tries to open the chunks while the links are down all
hell will break loose.
I agree with others. Why do this at all?
Art
On Mon, Jan 26, 2009 at 5:42 PM, Jarrod Teale <Jarrod.Teale@fonterra.com>wrote:
> Hey all.
> Now I know that creating symlinks for access to chunks is a really
> really good idea and is a practice that we follow religiously.
> I have a question though - when does the database engine make use of
> them? Is it just at startup when the chunks are going through sanity
> checking?
> Obviously I'm not counting any interaction with onspaces or admin/task
> operation that add/remove/manipulate allocated disk - required then.
>
> The reason I ask is that we are looking at building a script that
> destroys and then recreates them while the engine is running.
> My assumption is that the engine creates the file handles it requires to
> the chunks on startup and then doesn't need them again - so destroying
> and recreating them won't cause a problem.
>
> It would be better to do while the engine is off-line, but in this case
> we are hoping to avoid the downtime.
>
> We are on RedHat EL4 x86 (11.10.UC2W2 and 11.50.UC3).
>
> Thanks
>
> Jarrod Teale
>
> Team Lead - Manufacturing Execution Systems
>
> Automation & Process Control Group
>
> NZ Technical
>
> Fonterra
>
> Fonterra Extn: 77525
>
> DDI: +64 7 850 7525
>
> Mobile: +64 21 968 364
> fax: +64 7 849 7855
> email: jarrod.teale@fonterra.com
> <mailto:jarrod.teale@fonterra.com>
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--
Art S. Kagel
Oninit (www.oninit.com)
IIUG Board of Directors (art@iiug.org)
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Oninit, the IIUG, nor any other organization
with which I am associated either explicitly or implicitly. Neither do
those opinions reflect those of other individuals affiliated with any entity
with which I am affiliated nor those of the entities themselves.
--001636c5a1fb19ed9904616c2253
Okay - so the reason.
We deploy everything using the RedHat RPM system - including the base
setup of the informix engine.
When the package for 11.10.UC2W2 was built, the author added the
/usr/informix/DBSPACES directory to the package - making it owned by it.
If you then build a package to deploy 11.50.UC3 binaries and perform a
silent install into a different informix directory then you have the new
version of the engine installed and ready to pick up when the
INFORMIXDIR is changed and the instance cycled.
This is great - except that the upgrade to 11.50.UC3 removes the package
for 11.10.UC2W2 - along with all files it owns. 99% of these are just
the install media and scripts so harmless. But the DBSPACES directory is
really important as the engine is still running at the time.
The workaround we came up with was to push out a package that removed
ownership of the files from the RPM system - but to do that we have to
allow the RPM system to rip out the symlinks and then create them using
a post script so they are there, but not owned.
In brief:
11.10.UC2W2 is installed into /usr/informix11UC2W2
11.50.UC3 is being installed into /usr/informix115UC3
10.0.UC6 is also installed in /usr/informix10
The running engine has INFORMIXDIR set to /usr/informix
/usr/informix is a symlink to the installed set of binaries that we have
the engine running on.
Upgrade is to install the new version at the DBA's convienence then when
an outage can be achieved (may be months later) swap the /usr/informix
symlink to the new binary directory and let it migrate (testing all
completed ahead of time - there are 25ish of these identical systems).
Stupidly the directory with the chunks from the engines point of view is
in /usr/informix/DBSPACES - which is in the underlying binary directory
which is where the problem begins.
We all know that changing this now (moving the chunks) is not easy
(though not impossible).
The problem arises when the 11.50.UC3 package hits the server and
removes the content from the 11.10.UC2W2 directory (and the engine is
running).
Jarrod
-----Original Message-----
From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of
Art Kagel
Sent: Tuesday, 27 January 2009 1:43 p.m.
To: ids@iiug.org
Subject: Re: Symlinks and informix... [14637]
Jarrod,
IDS opens and closes the chunks periodically. If you run fuser against
the chunks you will often note that they are not all opened by all of
the CPU VPs. If a VP hasn't needed to perform IO against a chunk it will
not have th chunk open. Existing open files in the engine will not be
affected if you remove the links, and once the links have been
reestablished to the existing chunks (I assume you aren't going to link
them elsewhere while the engine is running) the engine should not notice
that they were gone for a bit, but if the engine tries to open the
chunks while the links are down all hell will break loose.
I agree with others. Why do this at all?
Art
On Mon, Jan 26, 2009 at 5:42 PM, Jarrod Teale
<Jarrod.Teale@fonterra.com>wrote:
> Hey all.
> Now I know that creating symlinks for access to chunks is a really
> really good idea and is a practice that we follow religiously.
> I have a question though - when does the database engine make use of
> them? Is it just at startup when the chunks are going through sanity
> checking?
> Obviously I'm not counting any interaction with onspaces or admin/task
> operation that add/remove/manipulate allocated disk - required then.
>
> The reason I ask is that we are looking at building a script that
> destroys and then recreates them while the engine is running.
> My assumption is that the engine creates the file handles it requires
> to the chunks on startup and then doesn't need them again - so
> destroying and recreating them won't cause a problem.
>
> It would be better to do while the engine is off-line, but in this
> case we are hoping to avoid the downtime.
>
> We are on RedHat EL4 x86 (11.10.UC2W2 and 11.50.UC3).
>
> Thanks
>
> Jarrod Teale
>
> Team Lead - Manufacturing Execution Systems
>
> Automation & Process Control Group
>
> NZ Technical
>
> Fonterra
>
> Fonterra Extn: 77525
>
> DDI: +64 7 850 7525
>
> Mobile: +64 21 968 364
> fax: +64 7 849 7855
> email: jarrod.teale@fonterra.com
> <mailto:jarrod.teale@fonterra.com>
>
>
>
>
************************************************************************
*******
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--
Art S. Kagel
Oninit (www.oninit.com)
IIUG Board of Directors (art@iiug.org)
Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on my employer, Oninit, the IIUG, nor any other
organization with which I am associated either explicitly or implicitly.
Neither do those opinions reflect those of other individuals affiliated
with any entity with which I am affiliated nor those of the entities
themselves.
--001636c5a1fb19ed9904616c2253
************************************************************************
*******
Forum Note: Use "Reply" to post a response in the discussion forum.
DISCLAIMER:
This email contains confidential information and may be legally privileged. If
you are not the intended recipient or have received this email in error,
please notify the sender immediately and destroy this email.
You may not use, disclose or copy this email or its attachments in any way.
Any opinions expressed in this email are those of the author and are not
necessarily those of the Fonterra Co-operative Group.
http://www.fonterra.com/
Jarrod Teale wrote: > > Stupidly the directory with the chunks from the engines point of view is > in /usr/informix/DBSPACES - which is in the underlying binary directory > which is where the problem begins. Actually, you're looking at this problem in the wrong way. Having your dbspaces in /usr/informix isn't a bad idea, but installing the binaries in /usr/informix is. You should really be installing the binaries into /usr/informix/<version> ... -- Cheers, Obnoxio The Clown http://obotheclown.blogspot.com -- This message has been scanned for viruses and dangerous content by OpenProtect(http://www.openprotect.com), and is believed to be clean.
OK, the RPM install should NOT be removing the existing software whether the
engine is online or not!
You should ALWAYS have two, and sometimes three, versions of the IDS
software on disk at all times:
1. The currently running version
2. The previous version in case you have to revert due to problems with
the newer software
3. Sometimes, between installation and starting the new version you'll
also have the next version on disk.
So, how to fix this. The RPM should not be removing /usr/informix or the
previous installed version. It should be trying to remove any previously
installed copy of the same new version. In this case it should be removing
any existing copy of 11.10.UC3 not 11.10.UC2. That solves your device
problems and the install script can copy the device links to the new install
directory post-install at the same time that it copies the sqlhosts and
ONCONFIG files. Then there's no conflict and no need to remove the existing
chunk links.
Art
On Mon, Jan 26, 2009 at 8:59 PM, Jarrod Teale <Jarrod.Teale@fonterra.com>wrote:
> Okay - so the reason.
>
> We deploy everything using the RedHat RPM system - including the base
> setup of the informix engine.
> When the package for 11.10.UC2W2 was built, the author added the
> /usr/informix/DBSPACES directory to the package - making it owned by it.
> If you then build a package to deploy 11.50.UC3 binaries and perform a
> silent install into a different informix directory then you have the new
> version of the engine installed and ready to pick up when the
> INFORMIXDIR is changed and the instance cycled.
> This is great - except that the upgrade to 11.50.UC3 removes the package
> for 11.10.UC2W2 - along with all files it owns. 99% of these are just
> the install media and scripts so harmless. But the DBSPACES directory is
> really important as the engine is still running at the time.
>
> The workaround we came up with was to push out a package that removed
> ownership of the files from the RPM system - but to do that we have to
> allow the RPM system to rip out the symlinks and then create them using
> a post script so they are there, but not owned.
>
> In brief:
> 11.10.UC2W2 is installed into /usr/informix11UC2W2
> 11.50.UC3 is being installed into /usr/informix115UC3
> 10.0.UC6 is also installed in /usr/informix10
> The running engine has INFORMIXDIR set to /usr/informix
> /usr/informix is a symlink to the installed set of binaries that we have
> the engine running on.
> Upgrade is to install the new version at the DBA's convienence then when
> an outage can be achieved (may be months later) swap the /usr/informix
> symlink to the new binary directory and let it migrate (testing all
> completed ahead of time - there are 25ish of these identical systems).
>
> Stupidly the directory with the chunks from the engines point of view is
> in /usr/informix/DBSPACES - which is in the underlying binary directory
> which is where the problem begins.
> We all know that changing this now (moving the chunks) is not easy
> (though not impossible).
>
> The problem arises when the 11.50.UC3 package hits the server and
> removes the content from the 11.10.UC2W2 directory (and the engine is
> running).
>
> Jarrod
> -----Original Message-----
> From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of
> Art Kagel
> Sent: Tuesday, 27 January 2009 1:43 p.m.
> To: ids@iiug.org
> Subject: Re: Symlinks and informix... [14637]
>
> Jarrod,
>
> IDS opens and closes the chunks periodically. If you run fuser against
> the chunks you will often note that they are not all opened by all of
> the CPU VPs. If a VP hasn't needed to perform IO against a chunk it will
> not have th chunk open. Existing open files in the engine will not be
> affected if you remove the links, and once the links have been
> reestablished to the existing chunks (I assume you aren't going to link
> them elsewhere while the engine is running) the engine should not notice
> that they were gone for a bit, but if the engine tries to open the
> chunks while the links are down all hell will break loose.
>
> I agree with others. Why do this at all?
>
> Art
>
> On Mon, Jan 26, 2009 at 5:42 PM, Jarrod Teale
> <Jarrod.Teale@fonterra.com>wrote:
>
> > Hey all.
> > Now I know that creating symlinks for access to chunks is a really
> > really good idea and is a practice that we follow religiously.
> > I have a question though - when does the database engine make use of
> > them? Is it just at startup when the chunks are going through sanity
> > checking?
> > Obviously I'm not counting any interaction with onspaces or admin/task
>
> > operation that add/remove/manipulate allocated disk - required then.
> >
> > The reason I ask is that we are looking at building a script that
> > destroys and then recreates them while the engine is running.
> > My assumption is that the engine creates the file handles it requires
> > to the chunks on startup and then doesn't need them again - so
> > destroying and recreating them won't cause a problem.
> >
> > It would be better to do while the engine is off-line, but in this
> > case we are hoping to avoid the downtime.
> >
> > We are on RedHat EL4 x86 (11.10.UC2W2 and 11.50.UC3).
> >
> > Thanks
> >
> > Jarrod Teale
> >
> > Team Lead - Manufacturing Execution Systems
> >
> > Automation & Process Control Group
> >
> > NZ Technical
> >
> > Fonterra
> >
> > Fonterra Extn: 77525
> >
> > DDI: +64 7 850 7525
> >
> > Mobile: +64 21 968 364
> > fax: +64 7 849 7855
> > email: jarrod.teale@fonterra.com
> > <mailto:jarrod.teale@fonterra.com>
> >
> >
> >
> >
> ************************************************************************
> *******
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
>
> --
> Art S. Kagel
> Oninit (www.oninit.com)
> IIUG Board of Directors (art@iiug.org)
>
> Disclaimer: Please keep in mind that my own opinions are my own opinions
> and do not reflect on my employer, Oninit, the IIUG, nor any other
> organization with which I am associated either explicitly or implicitly.
> Neither do those opinions reflect those of other individuals affiliated
> with any entity with which I am affiliated nor those of the entities
> themselves.
>
> --001636c5a1fb19ed9904616c2253
>
> ************************************************************************
> *******
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
> DISCLAIMER:
> This email contains confidential information and may be legally privileged.
> If
> you are not the intended recipient or have received this email in error,
> please notify the sender immediately and destroy this email.
> You may not use, disclose or copy this email or its attachments in any way.
> Any opinions expressed in this email are those of the author and are not
> necessarily those of the Fonterra Co-operative Group.
> http://www.fonterra.com/
>
>
>
>
*******************************************************************************
> F