Parsing LOGICAL LOG files to fake replication?
I am currently working on a large data warehouse project that introduced it's own "replication" several years ago. (Before Enterprise Replication became available.) I'd LIKE to replace this with Enterprise Replication. We currently run 9.20uc3x4 (?) here. HOWEVER, after some benchmarking on some new E10000 (SunOS) that we are buying, it was determined that Enterprise Replication will not work for this project. (There apparently IS a problem with Enterprise Replication that Informix, as usual, has promised that the NEXT release will fix.) I can't wait. I CAN try to make an intermediate step that will get us halfway there. CURRENTLY when the client application does an insert/update/delete it sends data to a central "Replication" program that will then QUERY the database and put the new or changed record into a flat file for transmission to other groups that may have different architectures and/or databases. (The central Replication server is used so the ORDER in the file will be correct.) I DON"T LIKE THIS. There are TWO transactions for EVERY "real" transaction. (The middle man queries the database to get the data.... it is NOT sent from the client... only the key is sent.) ===================================== BUT I HAVE A NEW IDEA AND A QUESTION: ===================================== CAN I GRAB THE LOGICAL LOG FILES AND RUN THEM THROUGH A PROPRIETARY PARSER, EXTRACTING THE CHANGES TO THE TABLES THAT I NEED? I don't see any reason that I SHOULDN'T be able to do this..... but I thought I'd ask the question here... somebody else may have done this before and maybe I can get some ideas. (This method would allow me to cut out all of the code from the client applications that is in place for this "Replicatin" method. Then IF and when Enterprise Replication gets to working correctly.... we can use that.) Thanks for any ideas, ---klm
Keith L Morris wrote: > > I am currently working on a large data warehouse project that introduced > it's own "replication" several years ago. (Before Enterprise > Replication became available.) > > I'd LIKE to replace this with Enterprise Replication. > > We currently run 9.20uc3x4 (?) here. HOWEVER, after some benchmarking > on some new E10000 (SunOS) that we are buying, it was determined that > Enterprise Replication will not work for this project. (There > apparently IS a problem with Enterprise Replication that Informix, as > usual, has promised that the NEXT release will fix.) > > I can't wait. I CAN try to make an intermediate step that will get us > halfway there. > > CURRENTLY when the client application does an insert/update/delete it > sends data to a central "Replication" program that will then QUERY the > database and put the new or changed record into a flat file for > transmission to other groups that may have different architectures > and/or databases. (The central Replication server is used so the > ORDER in the file will be correct.) > > I DON"T LIKE THIS. There are TWO transactions for EVERY "real" > transaction. (The middle man queries the database to get the data.... > it is NOT sent from the client... only the key is sent.) > > ===================================== > BUT I HAVE A NEW IDEA AND A QUESTION: > ===================================== > > CAN I GRAB THE LOGICAL LOG FILES AND RUN THEM THROUGH A > PROPRIETARY PARSER, EXTRACTING THE CHANGES TO THE TABLES THAT I NEED? > > I don't see any reason that I SHOULDN'T be able to do this..... but I > thought I'd ask the question here... somebody else may have done this > before and maybe I can get some ideas. > > (This method would allow me to cut out all of the code from the client > applications that is in place for this "Replicatin" method. Then IF and > when Enterprise Replication gets to working correctly.... we can use > that.) Another alternative is create a UDF/UDR that can be called from triggers that will copy the transaction details to the replication server. Art S. Kagel
why won't ER work for the project? And what is version 9.20.UC3X4? Nona >Subject: Parsing LOGICAL LOG files to fake replication? >From: Keith L Morris keith.l.morris@wcom.com >Date: 10.07.00 19:34 W. Europe Daylight Time >Message-id: <396A0907.EF74615B@wcom.com> > >I am currently working on a large data warehouse project that introduced >it's own "replication" several years ago. (Before Enterprise >Replication became available.) > >I'd LIKE to replace this with Enterprise Replication. > >We currently run 9.20uc3x4 (?) here. HOWEVER, after some benchmarking >on some new E10000 (SunOS) that we are buying, it was determined that >Enterprise Replication will not work for this project. (There >apparently IS a problem with Enterprise Replication that Informix, as >usual, has promised that the NEXT release will fix.) > >I can't wait. I CAN try to make an intermediate step that will get us >halfway there. > >CURRENTLY when the client application does an insert/update/delete it >sends data to a central "Replication" program that will then QUERY the >database and put the new or changed record into a flat file for >transmission to other groups that may have different architectures >and/or databases. (The central Replication server is used so the >ORDER in the file will be correct.) > >I DON"T LIKE THIS. There are TWO transactions for EVERY "real" >transaction. (The middle man queries the database to get the data.... >it is NOT sent from the client... only the key is sent.) > >===================================== >BUT I HAVE A NEW IDEA AND A QUESTION: >===================================== > >CAN I GRAB THE LOGICAL LOG FILES AND RUN THEM THROUGH A >PROPRIETARY PARSER, EXTRACTING THE CHANGES TO THE TABLES THAT I NEED? > >I don't see any reason that I SHOULDN'T be able to do this..... but I >thought I'd ask the question here... somebody else may have done this >before and maybe I can get some ideas. > >(This method would allow me to cut out all of the code from the client >applications that is in place for this "Replicatin" method. Then IF and >when Enterprise Replication gets to working correctly.... we can use >that.) > > >Thanks for any ideas, >---klm > > > > > > > >
Watawinona wrote: > why won't ER work for the project? Not fast enough..... WAY too slow. The SENDING machine is okay... but then the RECEIVING machine is loading for 2 hours when the original load was only 20 minutes. BUT unfortuneately we load 24 hours a day.... so when would the receiving machine catch up? Now to be HONEST, I believe it must have been some kind of a configuration problem... BUT I'm not the DBA (anymore) I'm just a lowly developer. (In other words... keep my nose out of it. BUT I DO have to optimize the application. So if I'm told that ER will not work... I have to go on that premise.... even if I believe that to be incorrect.) Also.... the original plan had been to convert several other groups over to Informix... and several of the groups have bailed out... so they still want their proprietary format file. (And yes, it is ASCII. BLECH.) > And what is version 9.20.UC3X4? The version that we have here. Think of it as UC3 with a few added features for this company. (They turn off the btree cleaners... go figure...) Anyway, I would do the UDF/UDR route.. but this engine crashes about once a week...due to MEMORY ALLOCATION problems and I don't want the politics of the situation... pointing fingers at my cleanly written UDF/UDR. (That was with versions 7.23, 7.31 and now with 9.20.... still the same problem....) Besides... we WILL eventually get ER working correctly..... so this parsing of the logical logs would only be a "stop gap" until the ER gets the approval. But then there is the other group that has decided to stick with another database on another platform....... so we may need it anyway. OTOH, the system would become easier to maintain AND understand for new people coming into the group. (The old KISS idea.... keep it stupidly simple) Besides I would trust this parser method more than the method they use now...... it's coming DIRECTLY from the logical logs... and what they have now...... ick. ---klm > Nona > > >Subject: Parsing LOGICAL LOG files to fake replication? > >From: Keith L Morris keith.l.morris@wcom.com > >Date: 10.07.00 19:34 W. Europe Daylight Time > >Message-id: <396A0907.EF74615B@wcom.com> > > > >I am currently working on a large data warehouse project that introduced > >it's own "replication" several years ago. (Before Enterprise > >Replication became available.) > > > >I'd LIKE to replace this with Enterprise Replication. > > > >We currently run 9.20uc3x4 (?) here. HOWEVER, after some benchmarking > >on some new E10000 (SunOS) that we are buying, it was determined that > >Enterprise Replication will not work for this project. (There > >apparently IS a problem with Enterprise Replication that Informix, as > >usual, has promised that the NEXT release will fix.) > > > >I can't wait. I CAN try to make an intermediate step that will get us > >halfway there. > > > >CURRENTLY when the client application does an insert/update/delete it > >sends data to a central "Replication" program that will then QUERY the > >database and put the new or changed record into a flat file for > >transmission to other groups that may have different architectures > >and/or databases. (The central Replication server is used so the > >ORDER in the file will be correct.) > > > >I DON"T LIKE THIS. There are TWO transactions for EVERY "real" > >transaction. (The middle man queries the database to get the data.... > >it is NOT sent from the client... only the key is sent.) > > > >===================================== > >BUT I HAVE A NEW IDEA AND A QUESTION: > >===================================== > > > >CAN I GRAB THE LOGICAL LOG FILES AND RUN THEM THROUGH A > >PROPRIETARY PARSER, EXTRACTING THE CHANGES TO THE TABLES THAT I NEED? > > > >I don't see any reason that I SHOULDN'T be able to do this..... but I > >thought I'd ask the question here... somebody else may have done this > >before and maybe I can get some ideas. > > > >(This method would allow me to cut out all of the code from the client > >applications that is in place for this "Replicatin" method. Then IF and > >when Enterprise Replication gets to working correctly.... we can use > >that.) > > > > > >Thanks for any ideas, > >---klm > > > > > > > > > > > > > > > >