All mirror chunks down!
IDS 7.24UC7, Solaris 2.6, Raw disks We've encountered a rather strange problem on one of our servers recently. Suddenly, all of our mirrored chunks were down without any notice from Informix. No entry in the online log for the downed chunks whatsoever!. None, nada, Zip! What I did was, turn off mirroring for those chunks and re-mirror them again. Re-mirroring them took way to long! (i.e more tha 15 mins. for a 1Gig chunk 90% empty). Any thoughts? By the way, we're using Volume Manage to define our chunks. Lyzander Marantal Bell Atlantic * Sent from RemarQ http://www.remarq.com The Internet's Discussion Network * The fastest and easiest way to search and participate in Usenet - Free!
Zandy <zandymNOzaSPAM@yahoo.com.invalid> wrote: > IDS 7.24UC7, Solaris 2.6, Raw disks > We've encountered a rather strange problem on one of our servers > recently. Suddenly, all of our mirrored chunks were down without any > notice from Informix. No entry in the online log for the downed chunks > whatsoever!. None, nada, Zip! This part is strange, normally a mesasge is entered in the logs. I assume that this next paragraph is your attempt to remedy the problem, correct? :) > What I did was, turn off mirroring for those chunks and re-mirror them > again. Re-mirroring them took way to long! (i.e more tha 15 mins. for a > 1Gig chunk 90% empty). > Any thoughts? The mirroring process is primarily affected by the *size* of the chunks, not how full they may or may not be. When Informix creates a mirror, it literally creates a bitmap copy of the primary to the mirror. This takes a while and, needless to say, can be affected by instance parameters as well as the physical machines configuration. To wit: are the drives on the same controller? Are you using KAIO or AIO? How many vps (CPU, AIO)? How many online recovery threads are configured? What is the concurrent use on the instance(s) or other applications the box is supporting?-it may be maxed out just trying to handle the "normal" workload without throwing a mirror creation process and all its I/O requests on top on it. Carlton ______________________________________________________________________ Carlton Doe DBA Resources, Inc. Salt Lake City, UT dbaresrc at xmission dot com http://www.xmission.com/~dbaresrc carlton at iiug dot org http://www.iiug.org
> To wit: are the drives on the same controller? Are you using > KAIO or AIO? > How many vps (CPU, AIO)? How many online recovery threads are > configured? > What is the concurrent use on the instance(s) or other > applications the box is > supporting?-it may be maxed out just trying to handle the "normal" > workload > without throwing a mirror creation process and all its I/O > requests on top on > it. > Carlton Thanks for your reply. I'm using KAIO. 3 CPU VPS on an 8 way SPARC E1000 2 AIOVPS 1 Online recovery thread (default) The disk subsystem is unusually slow. Since there are hardly any physical users on the system, (mostly cron jobs that aren't time critical), I didn't really notice it till this happened. I would really like to know a possible theory as to why this happened. Called Tech Support and they haven't got a clue ...:-( Lyzander Marantal Bell Atlantic * Sent from RemarQ http://www.remarq.com The Internet's Discussion Network * The fastest and easiest way to search and participate in Usenet - Free!
Are all the mirrors on the same SCSI channel? If so look for a problem on the card in /var/adm/messages, it could just be a card glitch. If the FS that the online.log is in is full then you wouldn't be able to write to the log. If the aio vp had died would you still be able to write to the log? and if you can't how do you know the aio vp has died?? AFAIK the speed of bringing the mirror on-line is purely a function of the size of the chunk and not it's current usage. If you re-activated all the mirroring at the same time it would take a while anyway. Which volume manager? Any particular reason you are using Informix mirroring and not LVM mirroring? Zandy wrote: > > IDS 7.24UC7, Solaris 2.6, Raw disks > > We've encountered a rather strange problem on one of our servers > recently. Suddenly, all of our mirrored chunks were down without any > notice from Informix. No entry in the online log for the downed chunks > whatsoever!. None, nada, Zip! > > What I did was, turn off mirroring for those chunks and re-mirror them > again. Re-mirroring them took way to long! (i.e more tha 15 mins. for a > 1Gig chunk 90% empty). > > Any thoughts? > > By the way, we're using Volume Manage to define our chunks. > > Lyzander Marantal > Bell Atlantic > > * Sent from RemarQ http://www.remarq.com The Internet's Discussion Network * > The fastest and easiest way to search and participate in Usenet - Free! -- Paul Watson # WF Software # If it was easy Tel ++44 1436 674729 # Everybody could do it Fax ++44 1436 678693 # www.wfsoftware.com/informix #
In article <387D8F6B.FCA98CA5@gmaccc.co.uk>, Paul Watson <Paul.Watson@gmaccc.co.uk> wrote: > Are all the mirrors on the same SCSI channel? No they're not. > If the FS that the online.log is in is full then you wouldn't be able > to write to the log. Nothing seems to be wrong from the online.log perspective. The checkpoint messages ang logical log messages were logged uninterruptibly. The informix filesystem which has the online log is only 68% full. Oh well, must be one for the X-files ... Thanks anyway, Lyzander * Sent from RemarQ http://www.remarq.com The Internet's Discussion Network * The fastest and easiest way to search and participate in Usenet - Free!