Re: Parsing LOGICAL LOG files to fake replication?

Keith L Morris wrote: > Watawinona wrote: > > > why won't ER work for the project? > > Not fast enough..... WAY too slow. The SENDING machine is okay... but then > the RECEIVING machine is loading for 2 hours when the original load was only > 20 minutes. I suspect that you are running into B120797 which would cause problems when there is only one source server for a given target. Yes - this has been corrected in 9.3 by a new feature. Also, I think that there is a simpler fix for 7.32 which can easily be applied to 9.2. I do not know if it has been scheduled yet or not. By applying multiple groups WITH the patch applied for 120797, you should maximize the functionality of the DataSync on the target system. The basic problem is that the 7.31/9.2x release was optimized for multiple source instances. This patch allow us to optimize the DataSync for a single source instance. The new feature for 9.3 is much more than this as it included many other DataSync performance enhancements. > > > BUT unfortuneately we load 24 hours a day.... so when would the receiving > machine catch up? > > Now to be HONEST, I believe it must have been some kind of a configuration > problem... BUT I'm not the DBA (anymore) I'm just a lowly developer. (In > other words... keep my nose out of it. BUT I DO have to optimize the > application. So if I'm told that ER will not work... I have to go on that > premise.... even if I believe that to be incorrect.) > > Also.... the original plan had been to convert several other groups over to > Informix... and several of the groups have bailed out... so they still want > their proprietary format file. (And yes, it is ASCII. BLECH.) > > > And what is version 9.20.UC3X4? > > The version that we have here. Think of it as UC3 with a few added features > for this company. (They turn off the btree cleaners... go figure...) > > Anyway, I would do the UDF/UDR route.. but this engine crashes about once a > week...due to MEMORY ALLOCATION problems and I don't want the politics of > the situation... pointing fingers at my cleanly written UDF/UDR. (That was > with versions 7.23, 7.31 and now with 9.20.... still the same problem....) > > Besides... we WILL eventually get ER working correctly..... so this parsing > of the logical logs would only be a "stop gap" until the ER gets the > approval. But then there is the other group that has decided to stick with > another database on another platform....... so we may need it anyway. > > OTOH, the system would become easier to maintain AND understand for new > people coming into the group. (The old KISS idea.... keep it stupidly > simple) > > Besides I would trust this parser method more than the method they use > now...... it's coming DIRECTLY from the logical logs... and what they have > now...... ick. > > ---klm > > > Nona > > > > >Subject: Parsing LOGICAL LOG files to fake replication? > > >From: Keith L Morris keith.l.morris@wcom.com > > >Date: 10.07.00 19:34 W. Europe Daylight Time > > >Message-id: <396A0907.EF74615B@wcom.com> > > > > > >I am currently working on a large data warehouse project that introduced > > >it's own "replication" several years ago. (Before Enterprise > > >Replication became available.) > > > > > >I'd LIKE to replace this with Enterprise Replication. > > > > > >We currently run 9.20uc3x4 (?) here. HOWEVER, after some benchmarking > > >on some new E10000 (SunOS) that we are buying, it was determined that > > >Enterprise Replication will not work for this project. (There > > >apparently IS a problem with Enterprise Replication that Informix, as > > >usual, has promised that the NEXT release will fix.) > > > > > >I can't wait. I CAN try to make an intermediate step that will get us > > >halfway there. > > > > > >CURRENTLY when the client application does an insert/update/delete it > > >sends data to a central "Replication" program that will then QUERY the > > >database and put the new or changed record into a flat file for > > >transmission to other groups that may have different architectures > > >and/or databases. (The central Replication server is used so the > > >ORDER in the file will be correct.) > > > > > >I DON"T LIKE THIS. There are TWO transactions for EVERY "real" > > >transaction. (The middle man queries the database to get the data.... > > >it is NOT sent from the client... only the key is sent.) > > > > > >===================================== > > >BUT I HAVE A NEW IDEA AND A QUESTION: > > >===================================== > > > > > >CAN I GRAB THE LOGICAL LOG FILES AND RUN THEM THROUGH A > > >PROPRIETARY PARSER, EXTRACTING THE CHANGES TO THE TABLES THAT I NEED? > > > > > >I don't see any reason that I SHOULDN'T be able to do this..... but I > > >thought I'd ask the question here... somebody else may have done this > > >before and maybe I can get some ideas. > > > > > >(This method would allow me to cut out all of the code from the client > > >applications that is in place for this "Replicatin" method. Then IF and > > >when Enterprise Replication gets to working correctly.... we can use > > >that.) > > > > > > > > >Thanks for any ideas, > > >---klm > > > > > > > > > > > > > > > > > > > > > > > >