Re: Wanna try SolidDB for IDS?
Not a troubleshooting thread but an argument about solidDB (the in-memory cache for IDS) and data-loss risk. Fernando Nunes argued it's a misconception that in-memory databases lose data because they're in memory: committed transactions are written to a log before the commit returns, so recovery is like any other RDBMS, though you can optionally turn logging off for speed. Ian Gumby insisted anything in RAM not yet flushed is lost on power failure, making such systems inherently riskier. Serge Rielau tried to mediate, noting both positions reduce to redundancy choices (duplication, independent power, distance). No agreement or resolution was reached; the exchange ended in personal sniping.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Ian Michael Gumby wrote: > Backpeddling are we? > > Lets face it you made an incorrect generalization and then tried to > explain to me why I was wrong when I wasn't. > > You need to understand risk and how to assess the risk and the > alternatives. Then decide whether or not its worth the price to be > risk adverse. > > > I have a production database sitting on a single SATA drive. It was a > proof of concept that ended up going live in production as a SAAS app. > > I now have to decide what I want to do to reduce the risk of a disk > failure and data loss. > I could mirror the drive. I could add a raid card and SAS drives then > create a new copy of the database. Maybe consider porting it to > Informix. But I'm not making enough money from the SAS to pay for the > IDS license. > > The reality is that each option reduces the risk(s) and increases the > probability of uptime. > But with each option there is a cost and I have to decide whether the > cost is enough to offset the risk. > > When you talk about an in-memory database, you are creating a db that > has fairly fast response time. But the risk is that if there is a > power failure, there is going to be data loss. There are ways to > mitigate this risk such that you can live with an in memory database. > Had IBM not purchased Informix from ASS-N-TAIL, the concept of in > memory tables would have been a logical extension to the work Kevin > Brown did with the real time loader. (So give credit where credit is > due.) What really sucks is that neither IBM nor Informix really got a > chance to market Financial Foundation. Terry was on the way and had > some really brilliant ideas in the UK. But alas, PG sold us out to > IBM... (Don't believe me, as Pickford) > > > On Aug 7, 4:58 pm, Fernando Nunes <domusonl...@gmail.com> wrote: >> Ian Michael Gumby wrote: >>> Sigh. >>> Lets try this again.... >>> You said: >>> " A usual misconception: Being in memory, you risk loss of data: simply >>> NOT TRUE." >>> And I will repeat what I've said... >>> You will always have the "risk" of data loss. >> Sure. But that's because what you say: We're dealing with bits and bytes. Not >> because we're using a in-memory database. So, hopefully, you'll understand why >> I said "a usual misconception". If you don't have that misconception fine. >> >>> You have something stored in RAM. You lose power to your RAM, you lost >>> what was in memory. >> Losing what is in memory is not losing data. >> >>> The fact is that an "in memory" system will face the issue of some data >>> loss. >> Exactly the same as in IDS or any other RDBMS. If the redundancy and the >> logging fails, you loose data. >> >>> The point is that with the RTL, the feed(s) are written to shared memory >>> for immediate use and then it gets buffered to disk. If you have a power >>> loss prior to the data being buffered to disk, you run the risk of >>> losing said data. The work around was to tee (a unix term) the data feed >>> and write to a flat file. >> There are several points here: >> 1- I didn't talk about RTL, neither did the OP. So that part of the >> conversation although very interesting is pointless to the fact that SolidDB >> looses data as any other RDBMS, and not a bit more because it's an in-memory >> database >> 2- You need to make a point that proves you're right. I really don't mind about >> that (not even care), but it's a bit tiring... >> >>> Looking at any design of an "in memory" database, you're still going to >>> write the data to disk at some point. Since there is a delta between >>> what's in memory and what's being buffered to disk, any time you have a >>> power loss, said delta will be lost. >> As other DBMS, you write committed transactions to a log file that will permit >> you to recover all your data (consistently). The commit return is sent after >> logging. >> >>> I really wish you would think about how things work, before you post the >>> dribble that you get in marketing pitches. >> I do... And I try to stick to the point. There is no marketing here. Only pure >> technological concepts, that as I said, I'm sure you're aware and know about >> them. It's a shame that you take this as a championship to see who is right, so >> you start wondering about "there is always space for failure", and "secondary >> sites" etc. That is all true BUT it's true for ALL RDBMS, including SolidDB, >> and not specially true for SolidDB, and not specially true because it's an >> in-memory RDBMS. >> The point here should be not to confuse other people. >> And the original point still stands: There is a misconception that in-memory >> databases loose data because they are in memory. This is simply not true. >> In SolidDB case (and I believe in TimesTen is the same, but Mark can answer for >> that), you CAN configure it to don't log... but it's your choice... As usual, >> performance vs security. >> >> Regards, >> >> -- >> Fernando Nunes >> Portugal >> >> http://informix-technology.blogspot.com >> My email works... but I don't check it frequently... > Simple yes or no question: Do you understand that SolidDB will not loose data anymore than any other RDBMS, because it's an in-memory database? -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently...
Fernando, Fernando, Fernando. You made a very erroneous statement. You're now trying to backpedal and try to say that the regurgitated marketing pitch was that "in memory " databases are not "riskier" than a regular database. Even here, you still fail to grasp the problem. Anything that is "in memory" and has not been flushed to disk is at risk in the event of a simple power loss. To answer your question ... "It depends". I am legally bound such that I can not go in to specifics, but what I can say that depending on the design of the in-memory "database", you have a greater chance of being at more risk. Less chance if you're in a distributed cloud... but that again deals with the design and spread of the cloud. Not all databases are created equal. Like I said, talk to Caldera or Payne about the real time loader and some of it's gaps. Or maybe even Paul from Onit might pipe up. But hey! What do I know? ;-) -G " A usual misconception: Being in memory, you risk loss of data: simply NOT TRUE." - Fernando Nunes, IBM employee extraordinaire > From: domusonline@gmail.com > Subject: Re: Wanna try SolidDB for IDS? > Date: Fri, 8 Aug 2008 01:03:38 +0100 > To: informix-list@iiug.org > > Ian Michael Gumby wrote: > > Backpeddling are we? > > > > Lets face it you made an incorrect generalization and then tried to > > explain to me why I was wrong when I wasn't. > > > > You need to understand risk and how to assess the risk and the > > alternatives. Then decide whether or not its worth the price to be > > risk adverse. > > > > > > I have a production database sitting on a single SATA drive. It was a > > proof of concept that ended up going live in production as a SAAS app. > > > > I now have to decide what I want to do to reduce the risk of a disk > > failure and data loss. > > I could mirror the drive. I could add a raid card and SAS drives then > > create a new copy of the database. Maybe consider porting it to > > Informix. But I'm not making enough money from the SAS to pay for the > > IDS license. > > > > The reality is that each option reduces the risk(s) and increases the > > probability of uptime. > > But with each option there is a cost and I have to decide whether the > > cost is enough to offset the risk. > > > > When you talk about an in-memory database, you are creating a db that > > has fairly fast response time. But the risk is that if there is a > > power failure, there is going to be data loss. There are ways to > > mitigate this risk such that you can live with an in memory database. > > Had IBM not purchased Informix from ASS-N-TAIL, the concept of in > > memory tables would have been a logical extension to the work Kevin > > Brown did with the real time loader. (So give credit where credit is > > due.) What really sucks is that neither IBM nor Informix really got a > > chance to market Financial Foundation. Terry was on the way and had > > some really brilliant ideas in the UK. But alas, PG sold us out to > > IBM... (Don't believe me, as Pickford) > > > > > > On Aug 7, 4:58 pm, Fernando Nunes <domusonl...@gmail.com> wrote: > >> Ian Michael Gumby wrote: > >>> Sigh. > >>> Lets try this again.... > >>> You said: > >>> " A usual misconception: Being in memory, you risk loss of data: simply > >>> NOT TRUE." > >>> And I will repeat what I've said... > >>> You will always have the "risk" of data loss. > >> Sure. But that's because what you say: We're dealing with bits and bytes. Not > >> because we're using a in-memory database. So, hopefully, you'll understand why > >> I said "a usual misconception". If you don't have that misconception fine. > >> > >>> You have something stored in RAM. You lose power to your RAM, you lost > >>> what was in memory. > >> Losing what is in memory is not losing data. > >> > >>> The fact is that an "in memory" system will face the issue of some data > >>> loss. > >> Exactly the same as in IDS or any other RDBMS. If the redundancy and the > >> logging fails, you loose data. > >> > >>> The point is that with the RTL, the feed(s) are written to shared memory > >>> for immediate use and then it gets buffered to disk. If you have a power > >>> loss prior to the data being buffered to disk, you run the risk of > >>> losing said data. The work around was to tee (a unix term) the data feed > >>> and write to a flat file. > >> There are several points here: > >> 1- I didn't talk about RTL, neither did the OP. So that part of the > >> conversation although very interesting is pointless to the fact that SolidDB > >> looses data as any other RDBMS, and not a bit more because it's an in-memory > >> database > >> 2- You need to make a point that proves you're right. I really don't mind about > >> that (not even care), but it's a bit tiring... > >> > >>> Looking at any design of an "in memory" database, you're still going to > >>> write the data to disk at some point. Since there is a delta between > >>> what's in memory and what's being buffered to disk, any time you have a > >>> power loss, said delta will be lost. > >> As other DBMS, you write committed transactions to a log file that will permit > >> you to recover all your data (consistently). The commit return is sent after > >> logging. > >> > >>> I really wish you would think about how things work, before you post the > >>> dribble that you get in marketing pitches. > >> I do... And I try to stick to the point. There is no marketing here. Only pure > >> technological concepts, that as I said, I'm sure you're aware and know about > >> them. It's a shame that you take this as a championship to see who is right, so > >> you start wondering about "there is always space for failure", and "secondary > >> sites" etc. That is all true BUT it's true for ALL RDBMS, including SolidDB, > >> and not specially true for SolidDB, and not specially true because it's an > >> in-memory RDBMS. > >> The point here should be not to confuse other people. > >> And the original point still stands: There is a misconception that in-memory > >> databases loose data because they are in memory. This is simply not true. > >> In SolidDB case (and I believe in TimesTen is the same, but Mark can answer for > >> that), you CAN configure it to don't log... but it's your choice... As usual, > >> performance vs security. > >> > >> Regards, > >> > >> -- > >> Fernando Nunes > >> Portugal > >> > >> http://informix-technology.blogspot.com > >> My email works... but I don't check it frequently... > > > > Simple yes or no question: > Do you understand that SolidDB will not loose data anymore than any other > RDBMS, because it's an in-memory database? > > -- > Fernando Nunes > Portugal > > http://informix-technology.blogspot.com > My email works... but I don't check it frequently... > _______________________________________________ > Informix-list mailing list > Informix-list@iiug.org > http://www.iiug.org/mailman/listinfo/informix-list _________________________________________________________________ Got Game? Win Prizes in the W
I should know better than to even try, but I think you two aren't positioned that far apart! At the crux of it all is redundancy. How you achieve redundancy and to which degree comes with a lot of choice. For an in memory database such as solidDB or objectGrid redundancy against a software glitch can be largely achieved by duplication. Against power loss you need independent power systems. Against disasters you need "distance" (the farther the better). The only difference between disk and volatile memory here is the relative immunity against power loss for the disk. Ironically the moment where distance is big enough the power loss problem goes away (as long as you are not in North America with its marode infrastructure where one fried squirrel darkens half the continent....) So where is either of you disagreeing? (Other than by principle) Cheers Serge -- Serge Rielau DB2 Solutions Development IBM Toronto Lab
Serge, You're just as slow as Fernado and just as willing to try and come up with an excuse when IBM marketing material paints a misleading picture. (Assuming of course that's where Fernando got his "risk" statement.) Look, Its quite simple really. You will always have risk when dealing with storing data. All you can do is to minimize that risk to a point. You have to determine where the costs of minimizing the risk exceeds the actual risk or exposure that you are willing to live with. In Memory databases will always be susceptible to power losses. More so than your traditional databases. You can design for this, in both your hardware and software, however, the general statement made by Fernando is plain wrong. > From: srielau@ca.ibm.com > Subject: Re: Wanna try SolidDB for IDS? > Date: Fri, 8 Aug 2008 13:36:17 -0400 > To: informix-list@iiug.org > > I should know better than to even try, but I think you two aren't > positioned that far apart! > > At the crux of it all is redundancy. > How you achieve redundancy and to which degree comes with a lot of choice. > > For an in memory database such as solidDB or objectGrid redundancy > against a software glitch can be largely achieved by duplication. > Against power loss you need independent power systems. > Against disasters you need "distance" (the farther the better). > > The only difference between disk and volatile memory here is the > relative immunity against power loss for the disk. Ironically the moment > where distance is big enough the power loss problem goes away (as long > as you are not in North America with its marode infrastructure where one > fried squirrel darkens half the continent....) > > So where is either of you disagreeing? (Other than by principle) > > Cheers > Serge > -- > Serge Rielau > DB2 Solutions Development > IBM Toronto Lab > _______________________________________________ > Informix-list mailing list > Informix-list@iiug.org > http://www.iiug.org/mailman/listinfo/informix-list _________________________________________________________________ Your PC, mobile phone, and online services work together like never before. http://clk.atdmt.com/MRT/go/108587394/direct/01/
Ian Michael Gumby wrote: > Serge, > > You're just as slow as Fernado and just as willing to try and come up > with an excuse when IBM marketing material paints a misleading picture. > (Assuming of course that's where Fernando got his "risk" statement.) Yup.. I should have known better... Well I tried. Cheers Serge PS: Thanks for the personal attack btw. You sure know how to make friends... -- Serge Rielau DB2 Solutions Development IBM Toronto Lab
On Aug 8, 1:41 pm, Serge Rielau <srie...@ca.ibm.com> wrote: > Ian Michael Gumby wrote: > PS: Thanks for the personal attack btw. You sure know how to make friends... > -- Now Serge... I think has you sent me the correspondence directly in stead of via C.D.I, then I would have responded on a more personal level. Since you and others have cc'd this group, there was nothing personal about it. Its sad really that you'd post something and then say that I made a personal attack when I chide you for taking the position you did. Lets face it. You have taken the position of an IBM apologist in the past.
Ian Michael Gumby wrote: > On Aug 8, 1:41 pm, Serge Rielau <srie...@ca.ibm.com> wrote: >> Ian Michael Gumby wrote: > >> PS: Thanks for the personal attack btw. You sure know how to make friends... >> -- > Now Serge... > > I think has you sent me the correspondence directly in stead of via > C.D.I, then I would have responded on a more personal level. > Since you and others have cc'd this group, there was nothing personal > about it. > > Its sad really that you'd post something and then say that I made a > personal attack when I chide you for taking the position you did. > > Lets face it. You have taken the position of an IBM apologist in the > past. "You're just as slow as Fernado" You are right calling Fernando and I "slow" in a public forum is a public personal attack, not a personal personal attack. My mistake. http://en.wikipedia.org/wiki/Libel Cheers Serge -- Serge Rielau DB2 Solutions Development IBM Toronto Lab