Re: Wanna try SolidDB for IDS?
Not a support question but an argument about whether SolidDB-style "in-memory" databases carry a greater risk of data loss on power failure than traditional disk-based engines like IDS. Kevin Cherkauer (IDS kernel developer) and Fernando Nunes argued that durability depends on write-ahead logging, not on how much data sits in memory (IDS with a huge buffer pool holds plenty in RAM too); the real difference is in the algorithms used. Ian Gumby disagreed, saying in-memory engines deprioritise flushing to disk, citing NDA limits. No agreement was reached, and the thread drifted into off-topic banter.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
"Ian Michael Gumby" <im_gumby@hotmail.com> wrote: > When you talk about an in-memory database, you are creating a db that > has fairly fast response time. But the risk is that if there is a > power failure, there is going to be data loss. Your statement is misleading, and your harping on Fernando misguided. You are implying above (and seem to have claimed explicitly in your exchanges with Fernando), that an "in-memory database" product has *more* risk of data loss than a traditional relational database. That is not true. Whether the database keeps all the data in memory (such as SolidDB) or only some of the data in memory (IDS, DB2, Oracle, SQL Server, MySQL...) is orthogonal to the amount of data loss that can occur when the power cord is kicked out and the RAM goes dark. If they are all using the same write-ahead logging algorithm to log changes disk, then they are all exactly equal in their risk of data loss due to loss of power to the RAM. The term "in-memory database" is more marking buzz-flash than a real qualitative category as far as risk of data loss is concerned. When a customer cranks up IDS with a 64 gigabyte buffer pool, potentially 64 GB of the database is "in memory," and insert-update-delete operations on those 64 GB of data are done in memory and not flushed to disk until later -- perhaps a long time later. (In fact, the longer the better, as far as performance is concerned.) It's the logging that is the key. As Fernando already said. Now it is true that if your in-memory database does no logging, you will lose data on power loss. But the same is true of traditional databases. It's the logging. It's not the "in-memoryness." We don't tell customers of IDS, "You need to use a smaller buffer pool to reduce the amount of data that is in memory to reduce your risk of data loss." That's got nothing to do with the risk of data loss. The real technical content of the term "in-memory database" versus traditional databases is that there is a difference in the algorithms used to organize and access data, because they are optimizing performance based on different assumptions. (Again, this is orthogonal to the risk of data loss.) An in-memory DB assumes it can fit all the data in memory and thus is free to use algorithms that are faster when this is the case, but which may run into problems if the data no longer can fit into memory. Ye Olde traditional databases assume they cannot fit all the data into memory, so they have different algorithms that optimize this case that are capable of handling much larger data sets, but turn out to have more overhead than an "in-memory" database's algorithms in the case where the data does, in fact, all fit in memory. -- Kevin Cherkauer Software Engineer IBM Informix Dynamic Server -- Database Kernel
Kevin Cherkauer wrote: > "Ian Michael Gumby" <im_gumby@hotmail.com> wrote: > > >> When you talk about an in-memory database, you are creating a db that >> has fairly fast response time. But the risk is that if there is a >> power failure, there is going to be data loss. >> > > Your statement is misleading, and your harping on Fernando misguided. You > are implying above (and seem to have claimed explicitly in your exchanges > with Fernando), that an "in-memory database" product has *more* risk of data > loss than a traditional relational database. That is not true. Whether the > database keeps all the data in memory (such as SolidDB) or only some of the > data in memory (IDS, DB2, Oracle, SQL Server, MySQL...) is orthogonal to the > amount of data loss that can occur when the power cord is kicked out and the > RAM goes dark. If they are all using the same write-ahead logging algorithm > to log changes disk, then they are all exactly equal in their risk of data > loss due to loss of power to the RAM. > > I'm looking forward to watching the Gumb wriggle out of this one. -- Cheers, Obnoxio the Clown http://obotheclown.blogspot.com
In article <mailman.1501.1218224672.20610.informix-list@iiug.org>, Obnoxio The Clown says... >I'm looking forward to watching the dumb wriggle out of this one. typo corrected.
Kevin Cherkauer wrote: > "Ian Michael Gumby" <im_gumby@hotmail.com> wrote: > >> When you talk about an in-memory database, you are creating a db that >> has fairly fast response time. But the risk is that if there is a >> power failure, there is going to be data loss. > > Your statement is misleading, and your harping on Fernando misguided. You > are implying above (and seem to have claimed explicitly in your exchanges > with Fernando), that an "in-memory database" product has *more* risk of data > loss than a traditional relational database. That is not true. Whether the > database keeps all the data in memory (such as SolidDB) or only some of the > data in memory (IDS, DB2, Oracle, SQL Server, MySQL...) is orthogonal to the > amount of data loss that can occur when the power cord is kicked out and the > RAM goes dark. If they are all using the same write-ahead logging algorithm > to log changes disk, then they are all exactly equal in their risk of data > loss due to loss of power to the RAM. > > The term "in-memory database" is more marking buzz-flash than a real > qualitative category as far as risk of data loss is concerned. When a > customer cranks up IDS with a 64 gigabyte buffer pool, potentially 64 GB of > the database is "in memory," and insert-update-delete operations on those 64 > GB of data are done in memory and not flushed to disk until later -- perhaps > a long time later. (In fact, the longer the better, as far as performance is > concerned.) > > It's the logging that is the key. As Fernando already said. > > Now it is true that if your in-memory database does no logging, you will > lose data on power loss. But the same is true of traditional databases. > > It's the logging. It's not the "in-memoryness." > > We don't tell customers of IDS, "You need to use a smaller buffer pool to > reduce the amount of data that is in memory to reduce your risk of data > loss." That's got nothing to do with the risk of data loss. > > The real technical content of the term "in-memory database" versus > traditional databases is that there is a difference in the algorithms used > to organize and access data, because they are optimizing performance based > on different assumptions. (Again, this is orthogonal to the risk of data > loss.) An in-memory DB assumes it can fit all the data in memory and thus is > free to use algorithms that are faster when this is the case, but which may > run into problems if the data no longer can fit into memory. Ye Olde > traditional databases assume they cannot fit all the data into memory, so > they have different algorithms that optimize this case that are capable of > handling much larger data sets, but turn out to have more overhead than an > "in-memory" database's algorithms in the case where the data does, in fact, > all fit in memory. > I really appreciate your effort, but this is definitely a waste of time and bandwidth. He will get back with RTL, and risk management etc... The real information is in the manuals and everything is available for testing... We have better uses for our time. It really is annoying when someone puts words in your mouth, trying to convince others that you said something you didn't... but after reading the thread again, I think that any rational being will understand what was said. Regards. -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently...
On Aug 8, 2:32 pm, "Kevin Cherkauer" <invalid_addr...@nowhere.com> wrote: > "Ian Michael Gumby" <im_gu...@hotmail.com> wrote: > > > When you talk about an in-memory database, you are creating a db that > > has fairly fast response time. But the risk is that if there is a > > power failure, there is going to be data loss. > > Your statement is misleading, and your harping on Fernando misguided. You > are implying above (and seem to have claimed explicitly in your exchanges > with Fernando), that an "in-memory database" product has *more* risk of data > loss than a traditional relational database. That is not true. Whether the > database keeps all the data in memory (such as SolidDB) or only some of the > data in memory (IDS, DB2, Oracle, SQL Server, MySQL...) is orthogonal to the > amount of data loss that can occur when the power cord is kicked out and the > RAM goes dark. If they are all using the same write-ahead logging algorithm > to log changes disk, then they are all exactly equal in their risk of data > loss due to loss of power to the RAM. > [SNIP] > -- > Kevin Cherkauer > Software Engineer > IBM Informix Dynamic Server -- Database Kernel My statement isn't misleading. Perhaps you've just haven't thought about the problem long enough to understand why there is more risk in an in memory database than in a traditional database. And also the fact that *all* databases have risk associated with data loss. Sigh, Lets make this simple shall we? If the engine's primary function is to support the i/o from memory, then your flushing to disk is a secondary feature. Meaning said task takes a lower priority and the engine is tuned as such. If your engine's primary function is to store data on disk, then you're going to still have a window of loss, however it is going to be much smaller of a window since the write to disk is going to take a higher priority. Let us not also forget that the use for "in memory" databases is for fast i/o of information. Not for the longer retention of data. Here, we're talking about the use case of the "in memory" database. So when you think about how to make the "in-memory" database redundant and viable for more uptime, you tend to look at higher bandwidth networks and use of persistence within a cloud. Log shipping to a secondary in-memory vs log-shipping to the physical database. You could then buffer the data via a secondary. But again, *I* have to be carefull in what *I* say on this topic. (I unlike others honor the terms of my NDAs) ;-) The bottom line, there is a higher risk of data loss for a couple of reasons when you look at in-memory vs the traditional gprdbms. I think the biggest problem is that you don't understand the problem.
Ian Michael Gumby wrote: > On Aug 8, 2:32 pm, "Kevin Cherkauer" <invalid_addr...@nowhere.com> > wrote: >> "Ian Michael Gumby" <im_gu...@hotmail.com> wrote: >> >>> When you talk about an in-memory database, you are creating a db that >>> has fairly fast response time. But the risk is that if there is a >>> power failure, there is going to be data loss. >> Your statement is misleading, and your harping on Fernando misguided. You >> are implying above (and seem to have claimed explicitly in your exchanges >> with Fernando), that an "in-memory database" product has *more* risk of data >> loss than a traditional relational database. That is not true. Whether the >> database keeps all the data in memory (such as SolidDB) or only some of the >> data in memory (IDS, DB2, Oracle, SQL Server, MySQL...) is orthogonal to the >> amount of data loss that can occur when the power cord is kicked out and the >> RAM goes dark. If they are all using the same write-ahead logging algorithm >> to log changes disk, then they are all exactly equal in their risk of data >> loss due to loss of power to the RAM. >> > [SNIP] >> -- >> Kevin Cherkauer >> Software Engineer >> IBM Informix Dynamic Server -- Database Kernel > > My statement isn't misleading. Perhaps you've just haven't thought > about the problem long enough to understand why there is more risk in > an in memory database than in a traditional database. And also the > fact that *all* databases have risk associated with data loss. > > Sigh, > > Lets make this simple shall we? > > If the engine's primary function is to support the i/o from memory, > then your flushing to disk is a secondary feature. Meaning said task > takes a lower priority and the engine is tuned as such. > > If your engine's primary function is to store data on disk, then > you're going to still have a window of loss, however it is going to be > much smaller of a window since the write to disk is going to take a > higher priority. > > Let us not also forget that the use for "in memory" databases is for > fast i/o of information. Not for the longer retention of data. Here, > we're talking about the use case of the "in memory" database. > > So when you think about how to make the "in-memory" database redundant > and viable for more uptime, you tend to look at higher bandwidth > networks and use of persistence within a cloud. Log shipping to a > secondary in-memory vs log-shipping to the physical database. You > could then buffer the data via a secondary. > > But again, *I* have to be carefull in what *I* say on this topic. (I > unlike others honor the terms of my NDAs) ;-) > > The bottom line, there is a higher risk of data loss for a couple of > reasons when you look at in-memory vs the traditional gprdbms. > > I think the biggest problem is that you don't understand the problem. > > No... the biggest and saddest problem is that you don't understand logging and it's role in the recovery process... ;) Regards. -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently...
> From: domusonline@gmail.com > > No... the biggest and saddest problem is that you don't understand logging and > it's role in the recovery process... ;) > Regards. > > -- I don't? I guess you didn't read my post. The point is that you don't have to "log to disk" when you have logging, nor do you have to recover from disk either. And again, I have to be very careful in what I can and can't say. Cloud and grid computing makes things interesting. Its important to point out that while there is this new "paradigm", IDS and XPS had a lot of the base technology that we will see "re-invented" over the next couple of years. I'm sure many here won't grok what I've said, but in time, you will. ;-) -G _________________________________________________________________ Get Windows Live and get whatever you need, wherever you are. Start here. http://www.windowslive.com/default.html?ocid=TXT_TAGLM_WL_Home_082008
You should have your own blog like the clown has done. 8o/ Might be more entertaining with some graphics too. Ian Michael Gumby wrote: > > > > From: domusonline@gmail.com > > > > > No... the biggest and saddest problem is that you don't understand > logging and > > it's role in the recovery process... ;) > > Regards. > > > > -- > I don't? > > I guess you didn't read my post. > The point is that you don't have to "log to disk" when you have logging, > nor do you have to recover from disk either. > > And again, I have to be very careful in what I can and can't say. > > Cloud and grid computing makes things interesting. > > Its important to point out that while there is this new "paradigm", IDS > and XPS had a lot of the base technology that we will see "re-invented" > over the next couple of years. I'm sure many here won't grok what I've > said, but in time, you will. ;-) > > -G > > > ------------------------------------------------------------------------ > Get Windows Live and get whatever you need, wherever you are. Start > here. > <http://www.windowslive.com/default.html?ocid=TXT_TAGLM_WL_Home_082008>;
> From: indeep@indeep.com > Subject: Re: Wanna try SolidDB for IDS? > Date: Tue, 12 Aug 2008 20:58:34 -0700 > To: informix-list@iiug.org > > You should have your own blog like the clown has done. > > 8o/ > > Might be more entertaining with some graphics too. > Sorry, no. I wonder how many people outside of c.d.i actually read the clown's blog. For the most part, I wonder how many people in c.d.i actively seek OTC's blog if he doesn't post that he updated it here first. But if I did a blog, I guess I could post about all of the gaffs made by Informix and IBM over the years. Unfortunately, I'm sure I'd get a call from IBM legal threatening me were I to share certain stories. ;-) Sorry, but I am really swamped right now. Its never fun when you have a 40 hour a week client and then a side project taking up your time. Of course the wife wants to kill me, but hey! That's married life for you. -G _________________________________________________________________ Get more from your digital life. Find out how. http://www.windowslive.com/default.html?ocid=TXT_TAGLM_WL_Home2_082008
"Ian Michael Gumby" <im_gumby@hotmail.com> wrote in message news:mailman.1507.1218642519.20610.informix-list@iiug.org... >> Sorry, but I am really swamped right now. >> Its never fun when you have a 40 hour a week client and then a side >> project taking up your time. >> Of course the wife wants to kill me ... Er, that's probably not just because you're busy, mate ...
> From: theharlequin36@hotmail.com> Subject: Re: Wanna try SolidDB for IDS?> Date: Wed, 13 Aug 2008 18:00:40 +0100> To: informix-list@iiug.org> > "Ian Michael Gumby" <im_gumby@hotmail.com> wrote in message > news:mailman.1507.1218642519.20610.informix-list@iiug.org...> > >> Sorry, but I am really swamped right now.> >> Its never fun when you have a 40 hour a week client and then a side > >> project taking up your time.> >> Of course the wife wants to kill me ...> > Er, that's probably not just because you're busy, mate ...> Nope. That pretty much sums it up. Although she doesn't like it when I watch either the Military Channel or the Science Channel all the time. ;-) Some people just don't find the value in watching historic dog fights or how the Universe was created.... She'd actually rather watch VH1's reality show programs. Now if anyone has any tips on how best to work with carbon fiber ... ;-) -G _________________________________________________________________ Get more from your digital life. Find out how. http://www.windowslive.com/default.html?ocid=TXT_TAGLM_WL_Home2_082008