Logical Log File not found ..

Hello all, An informix instance on Secondary node of a Hardware Cluster is refusing to come up and giving following error. Wed Jun 26 04:10:30 2013 04:10:36 IBM Informix Dynamic Server Version 11.50.UC7 Software Serial Number AAA#B000000 04:10:37 Logical Log File not found. 04:10:37 oninit: Fatal error in shared memory initialization 04:10:37 IBM Informix Dynamic Server Stopped. 04:10:37 mt_shm_remove: WARNING: may not have removed all/correct segments By Hardware Cluster I mean, A virtual I/P to a Cluster which points to a Primary Node & upon Primary Node failure switches the package to Secondary Node. Also the Volume Group for raw devices is being shared and it's shifted to Secondary Node. So the problem is whenever Cluster is pointing to Primary node, everything seems working fine. But when the package is shifted and Cluster points to Secondary node then the same Informix instance refuses to come up with above error. Any body have any idea whats happening here ?? Regards, Akshay Polji.

Original post: Hello all, An informix instance on Secondary node of a Hardware Cluster is refusing to come up and giving following error. Wed Jun 26 04:10:30 2013 04:10:36 IBM Informix Dynamic Server Version 11.50.UC7 Software Serial Number AAA#B000000 04:10:37 Logical Log File not found. 04:10:37 oninit: Fatal error in shared memory initialization 04:10:37 IBM Informix Dynamic Server Stopped. 04:10:37 mt_shm_remove: WARNING: may not have removed all/correct segments By Hardware Cluster I mean, A virtual I/P to a Cluster which points to a Primary Node & upon Primary Node failure switches the package to Secondary Node. Also the Volume Group for raw devices is being shared and it's shifted to Secondary Node. So the problem is whenever Cluster is pointing to Primary node, everything seems working fine. But when the package is shifted and Cluster points to Secondary node then the same Informix instance refuses to come up with above error. Any body have any idea whats happening here ?? Regards, Akshay Polji. Response: That error basically means it can't find the checkpoint record for where physical and logical recovery is supposed to start from, so the server can't come up. It seems like there could be some problem when you go to shift the volume group/raw devices from the primary to the secondary something is going wrong. The servers code does writes in a particular sequence to make sure that the checkpoint record would get to disk prior to reserve page changes, but we also assume when a write call returns, it's completed and made it to disk. If something could be preventing some of the database writes to make it to disk or do them out of order, you could see some sort of error like this. Also, are you sure all the disks are getting over the secondary? If for some reason your root chunk/dbspace wasn't shared properly or didn't get over to the secondary but there was some disk there that was accessible but didn't have the updated reserve pages you might see some problem like that. You might want to do an oncheck -pr on your secondary and look at the checkpoint information for where the recovery checkpoint is and compare that to the logical log information. Then once you see the checkpoint location on the secondary, look at the last checkpoint completed message in MSGPATH of your primary, the checkpoint location should be either that checkpoint, or the checkpoint prior to that (in MSGPATH) Example if you aren't sure what I mean: info from oncheck -pr Validating PAGE_1CKPT & PAGE_2CKPT... Using check point page PAGE_1CKPT. Time stamp of checkpoint 0x4b02de Time of checkpoint 07/02/2013 12:48:49 Physical log begin address 1:263 Physical log size 25000 (p) Physical log position at Ckpt 13595 Logical log unique identifier 56 Logical log position at Ckpt 0xafd018 (Page 2813, byte 24) Checkpoint Interval 6335 So from that info on my system, the last checkpoint is logical log unique identifier 56, log position 0xafd018.... (more oncheck -pr output) Log file number 2 Unique identifier 56 Log contains last checkpoint: Page 2813, byte 24 Log file flags 0x3 Log file in use & Current log file Physical location 1:30263 Log size 5000 (p) So there I show that log file number 2 contains log unique identifier 56...so my log is present. On your system, I believe you will not find the log unique identifier (which is what that error means) Now I'll show the message from MSGPATH for that completed checkpoint 12:48:49 Checkpoint Completed: duration was 0 seconds. 12:48:49 Tue Jul 2 - loguniq 56, logpos 0xafd018, timestamp: 0x4b02e1 Interval: 6335 So you can see the loguniq (that's log unique identifier) is 56 and the log pos (log position) 0xafd018. Also, another thing you could check if possible is if you run oncheck -pr from the primary cluster before you switch over, that output should match the oncheck -pr output collected from your secondary cluster, and if itdoesn't...you've got some sort of problem. Jacques Renaut IBM Informix Advanced Support APD Team

Hello. So by meaning a shared cluster, you might be using one primary node, and the secondary must be an SDS secondary, ok? According to your case explanation, your secondary might even be a read-only node. Please confirm if that´s the way you implemented it, or not. Regards. Alexandre Marini IBM Informix Certified Professional v10 / v11.50 / v11.70 IBM Information Management Informix Technical Professional IBM Infosphere DataStage Technical Professional Informix Senior DBA - Orizon Brasil BRIUG website administrator Informix independent consultant > To: ids@iiug.org > From: akshuonline@gmail.com > Subject: Logical Log File not found .. [30736] > Date: Tue, 2 Jul 2013 13:32:19 -0400 > > Hello all, > > An informix instance on Secondary node of a Hardware Cluster is refusing to > come up and giving following error. > > Wed Jun 26 04:10:30 2013 > > 04:10:36 IBM Informix Dynamic Server Version 11.50.UC7 Software Serial Number > AAA#B000000 > 04:10:37 Logical Log File not found. > 04:10:37 oninit: Fatal error in shared memory initialization > 04:10:37 IBM Informix Dynamic Server Stopped. > 04:10:37 mt_shm_remove: WARNING: may not have removed all/correct segments > > By Hardware Cluster I mean, > > A virtual I/P to a Cluster which points to a Primary Node & upon Primary Node > failure switches the package to Secondary Node. Also the Volume Group for raw > devices is being shared and it's shifted to Secondary Node. > > So the problem is whenever Cluster is pointing to Primary node, everything > seems working fine. But when the package is shifted and Cluster points to > Secondary node then the same Informix instance refuses to come up with above > error. > > Any body have any idea whats happening here ?? > > Regards, > Akshay Polji. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >

Akshay: I would avoid using IP ghosting to provide HA failover and just use Informix MACH11 features. Setting up an HDR secondary server on the backup machine will only cost you the additional disks (as long as the secondary is only standby and not actively queries there is no additional license costs). If the secondaries are local and can actively share the drives attached to the primary, you can set up the secondary as an SDS (Shared Disk Secondary) server. In either case, users would either connect to the servers using an sqlhosts group entry name or using the Connection Manager. If the primary server does crash then the secondary, either SDS or HDR, can be promoted to primary almost instantly and users will reconnect to the new primary server. I have always found Informix HA failover technology to be far more reliable and simply faster than any external failover mechanism involving IP redirection. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Tue, Jul 2, 2013 at 1:32 PM, AKSHAY POLJI <akshuonline@gmail.com> wrote: > Hello all, > > An informix instance on Secondary node of a Hardware Cluster is refusing to > come up and giving following error. > > Wed Jun 26 04:10:30 2013 > > 04:10:36 IBM Informix Dynamic Server Version 11.50.UC7 Software Serial > Number > AAA#B000000 > 04:10:37 Logical Log File not found. > 04:10:37 oninit: Fatal error in shared memory initialization > 04:10:37 IBM Informix Dynamic Server Stopped. > 04:10:37 mt_shm_remove: WARNING: may not have removed all/correct segments > > By Hardware Cluster I mean, > > A virtual I/P to a Cluster which points to a Primary Node & upon Primary > Node > failure switches the package to Secondary Node. Also the Volume Group for > raw > devices is being shared and it's shifted to Secondary Node. > > So the problem is whenever Cluster is pointing to Primary node, everything > seems working fine. But when the package is shifted and Cluster points to > Secondary node then the same Informix instance refuses to come up with > above > error. > > Any body have any idea whats happening here ?? > > Regards, > Akshay Polji. > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --089e013d1742a2a03704e08c05be

Hi Alexandre, Thank you very much for your response. This is basically a fail-over mechanism designed purely on Hardware. Where you have a Virtual IP given to a Cluster which moves the Volume Groups from Primary to Secondary node in case of failure. Moreover both Primary & Secondary nodes are Read/Write enabled since the Volume Group is same. Regards, Akshay Polji.

Hi Art, I understand and appreciate your suggestions. Even I would personally prefer Informix HDR over IP ghosting or redirection. But the problem here is that the systems which are of concern were already setup by different vendors and we are just supporting it :| . Regards, Akshay Polji.

You are right on Money JACQUES :), I wanted to update the Reserved Pages details as well .. But instead I opted to keep it open for everyone to by their own understanding and to cross check if I m on right right. The problem here is with Reserved Pages. On Primary Node oncheck -pr Time stamp of checkpoint 0x3dc1ad1 Time of checkpoint 06/26/2013 10:15:11 Physical log begin address 19:53 Physical log size 250000 (p) Physical log position at Ckpt 234686 Logical log unique identifier 392 Logical log position at Ckpt 0x25b5018 (Page 9653, byte 24) Checkpoint Interval 76416 DBspace descriptor page 1:4 Chunk descriptor page 1:6 Mirror chunk descriptor page 1:8 On Secondary Node onchcek -pr Time stamp of checkpoint 0x3921824 Time of checkpoint 04/25/2013 16:01:22 Physical log begin address 19:53 Physical log size 250000 (p) Physical log position at Ckpt 110269 Logical log unique identifier 351 Logical log position at Ckpt 0x283f304 (Page 10303, byte 772) Checkpoint Interval 69207 DBspace descriptor page 1:5 Chunk descriptor page 1:7 Mirror chunk descriptor page 1:8 So to cut long story short, Reserved pages are indeed out of sync and that's the issue here. But the strange fact is that both Primary & Secondary are indeed using exactly the same rootchunk to read Reserved Pages... :| . One more important point to note here is that .. Few days back Secondary instance was crashing due to "Assert Failure error", So I requested for a reboot of that that single Unix Box i.e. Secondary Node. Interestingly, the instance which was not coming up earlier with "Assert Failure" has now started to fail during start with "Logical Log not found error" The only thing what I can think of is, if the reserved pages are loaded onto shared memory and that memory location is corrupt or something. Please share your views. Thanks & Regards, Akshay Polji.

Original post: You are right on Money JACQUES :), I wanted to update the Reserved Pages details as well .. But instead I opted to keep it open for everyone to by their own understanding and to cross check if I m on right right. The problem here is with Reserved Pages. <stuff cut> So to cut long story short, Reserved pages are indeed out of sync and that's the issue here. But the strange fact is that both Primary & Secondary are indeed using exactly the same rootchunk to read Reserved Pages... :| . One more important point to note here is that .. Few days back Secondary instance was crashing due to "Assert Failure error", So I requested for a reboot of that that single Unix Box i.e. Secondary Node. Interestingly, the instance which was not coming up earlier with "Assert Failure" has now started to fail during start with "Logical Log not found error" The only thing what I can think of is, if the reserved pages are loaded onto shared memory and that memory location is corrupt or something. Please share your views. Thanks & Regards, Akshay Polji. Response: Ok, well from the Informix perspective, also in the oncheck -pr output, you can compare/verify that the 2 different servers are looking at the same reserve page (each type of reserve page has 2 copies that we alternate back an for). So verify that each server is looking at the same checkpoint reserve page. Example: Validating PAGE_1CKPT & PAGE_2CKPT... Using check point page PAGE_2CKPT. So you'd want to make sure both -pr outputs showed they were looking at the same page (either PAGE_1CKPT or PAGE_2CKPT). After you confirm that, then I'd ask you are the logical volumns mirrored at the hardware level, or using Informix mirroring, or not at all. If you are using mirroring (with either informix or hardware) it could be the mirrors are out of sync. I have seen with hardware mirroring they can get out of sync and if the hardware either splits the IO requests across the primary and mirror, or if for some reason on 1 machine it favors 1 disk but on the 2nd machine it favors the other disk in the mirror, that could explain why on the 2 clusters you get different data when reading the same informix page (because the hardware mirrors are out of sync and the system isn't reading from the same devices between the 2 clusters). My guess would be this is some sort of mirror issue rather then not reading the same page of the reserve page pair, but it's still good to rule that out. Jacques Renaut IBM Informix Advanced Support APD Team

Yeah I can understand mirroring going out of sync could have damaged been the reason, but it's not the case in this scenario. There is no mirroring involved @ informix or hardware level. Here's the difference between "oncheck -pr" of Primary (working) & Secondary (failing) instance.. SECONDARY | PRIMARY | Validating PAGE_1CKPT & PAGE_2CKPT... | Validating PAGE_1CKPT & PAGE_2CKPT... Using check point page PAGE_2CKPT. | Using check point page PAGE_1CKPT. | Time stamp of checkpoint 0x3dc1ad1 | Time stamp of checkpoint 0x3921824 Time of checkpoint 06/26/2013 10:15:11| Time of checkpoint 04/25/2013 16:01:22 Physical log begin address 19:53 | Physical log begin address 19:53 Physical log size 250000 (p) | Physical log size 250000 (p) Physical log position at Ckpt 234686 | Physical log position at Ckpt 110269 Logical log unique identifier 392 | Logical log unique identifier 351 Logical log position at Ckpt 0x25b5018| Logical log position at Ckpt 0x283f304 (Page 10303, byte 772) | (Page 9653, byte 24) Checkpoint Interval 76416 | Checkpoint Interval 69207 DBspace descriptor page 1:4 | DBspace descriptor page 1:5 Chunk descriptor page 1:6 | Chunk descriptor page 1:7 Mirror chunk descriptor page 1:8 | Mirror chunk descriptor page 1:8 This is also case with other validation as well ... Validating PAGE_1DBSP & PAGE_2DBSP... | Validating PAGE_1DBSP & PAGE_2DBSP... Using DBspace page PAGE_2DBSP. | Using DBspace page PAGE_1DBSP. Validating PAGE_1PCHUNK & PAGE_2PCHUNK|Validating PAGE_1PCHUNK & PAGE_2PCHUNK Using primary chunk page PAGE_2PCHUNK.|Using primary chunk page PAGE_1PCHUNK. Validating PAGE_1ARCH & PAGE_2ARCH... |Validating PAGE_1ARCH & PAGE_2ARCH... And these pages seems to be out of sync. If my understandings are right then making sure the copies for Reserved Pages are in sync would solve the issue. But I have no idea how to do that. Can you or anybody help ? Thanks & Regards, Akshay Polji.