Re: ISM and OnBar Users

David Williams wrote: > > > How can you do table level restore when most transactions update > >1 table? > This is an assumption for only one scenario, that being a transaction on more than one table. But let's think about a different scenario. Say for example I have a data warehouse, and every week I append data to specific tables, one at a time. I then apply updates to all the rows in one of the tables, and the table is 12 GB in size. The inserts and updates go beautifully and all is well. Then an analyst comes to me and tells me that this weeks cycle is trash can we restore up to last week on this table? Sure, I'll simply drop the table, recreate it. and reload last week's unload files. Since I don't want to bounce the engine because other users are using the machine and I don't want to play games with onmode, this is the fastest recovery. ( By the way, by doing a table-level unload, I can accomplish this to disk in an hour and a half. Onbar takes 7 hours to do the same thing to ADSM! ) Onbar doesn't allow me to recover just this table. The option desired here is to force a restore into a perfectly good table. I'd pull it in from ADSM, and voila, the table is back to last week's precycle data. Maybe I don't care about the logical logs for this table either, so there should be an option to ignore the logs. Onbar should allow me to do table-level backups to prepare for the ability to do table-level restores. If I have the table frozen periodically, this should work for OLTP or DSS systems. Currently I can specify which dbspaces are backed up at a time, or default to letting onbar decide what gets backed up first and with XPS I can specify dbslices as well. I can also tag each backup with a "queue" name which are supposedly jobnames that I can use to recover with, but I haven't tried this yet. Our site like most doesn't have the luxury of a multi-server test bed, and instead we must use the production system for most of our testing off hours. It has to done with care. Not knowing how onbar may or may not behave makes using it rather risky. Onbar has little to no reporting capability to really show me anything about the last backup or previous backups. <...> > Why not let the restore bring all 20 back?? > Maybe in the course of bringing things back we have problems with the machine, and can't bring all the tables back at one time. Maybe we want to bring back critical tables first, and get the system back up for critical queries, and then bring back the other tables after the system is recovered back to a usable state. Maybe we have managers screaming who want the system back, and they get in the way of things, and we need to satisfy their kindergarten brains by telling them the system is recovered for critical/core tables, the rest will come back by the next day. Maybe there are data marts that don't need to be recovered at the same time. Maybe maybe maybe... Too many variables. Using a "wholistic" approach to recovering data bases does not take into account the realities of using data bases or the realities of how systems are set up. The larger the instance the more drastic this becomes. If it takes me 7 hours to restore via onbar because I have to restore the whole instance, and it may only take me 3 hours to get a few of the tables back using reload files, which option do you think I'll take to get managers off my back? :-) <...> > >Onbar is required to make the engine do things, but in reality onbar is > >functionally useless because of the limited restore capabilities. What's > >the point of a backup if you can't restore with it? > > > Usually most people restore everything due to a system failure. E.g. > disk failure, accidental overwrite of a chunk. That is what onbar > is designed to do. > Until you have actually restored a system, and a multi-server XPS system at that, I would challenge this notion. I would never make assumptions about any of this, and defer to operating at the lowest level possible, which are the tables themselves. The larger the system becomes, the more critical a table level recovery becomes. When you restore a large system, this will become clear. > Informix have sent several questionnaires my way and even phoned me at > work asking how to improve their products. The problems you have seem > due to the fact that you are not running an OLTP system and hence do > need transactions restored but tables. > > Remember that onbar also runs against 7.x systems which are usually > purely OLTP systems and that the majority of Informix users run OLTP > systems. > This is probably true for now. However I would challenge the notion that XPS is or will be for DSS only in the near future. I've heard that there are some sites actually using XPS for OLTP. And from what I know about XPS there is nothing really wrong with doing this, it simply gets back to how the engine is tuned, and what the expectations are in terms of the more limited SQL available on XPS. As engines supposedly merge these issues may become moot. And... Keep in mind a table-level restore has been the number one requested feature for quite some time. It was the only feature that actually beat Linux in a user survey right before Phil White got the boot. Thanks, Tim -- - -- --- Tim Schaefer ---- tschaefe@mindspring.com --- http://www.inxutil.com -- -

In article <36D9627C.FFF7A887@mindspring.com>, Tim Schaefer <tschaefe@mindspring.com> writes >David Williams wrote: >> >> >> How can you do table level restore when most transactions update >> >1 table? >> > >This is an assumption for only one scenario, that being a transaction >on more than one table. But let's think about a different scenario. > >Say for example I have a data warehouse, and every week I append data >to specific tables, one at a time. I then apply updates to all the >rows in one of the tables, and the table is 12 GB in size. The inserts >and updates go beautifully and all is well. > >Then an analyst comes to me and tells me that this weeks cycle is trash >can we restore up to last week on this table? Sure, I'll simply drop >the table, recreate it. and reload last week's unload files. Since I >don't want to bounce the engine because other users are using the machine >and I don't want to play games with onmode, this is the fastest recovery. > >( By the way, by doing a table-level unload, I can accomplish this to >disk in an hour and a half. Onbar takes 7 hours to do the same thing >to ADSM! ) > >Onbar doesn't allow me to recover just this table. The option desired >here is to force a restore into a perfectly good table. I'd pull it in >from ADSM, and voila, the table is back to last week's precycle data. > >Maybe I don't care about the logical logs for this table either, so there >should be an option to ignore the logs. Onbar should allow me to do >table-level backups to prepare for the ability to do table-level restores. >If I have the table frozen periodically, this should work for OLTP or DSS >systems. > Sounds like a good feature. I just think that since Onbar is new that Informix have go the obvious and implemented the features which will be used most first. I hope that Onbar is enhanced to include other features.. PS Who co-ordinates feature requests from c.d.i? How about a list of requested features? I'll volunteer to keep it on my Web Site... I'd really like to help... <Snip> >This is probably true for now. However I would challenge the notion that >XPS is or will be for DSS only in the near future. I've heard that there >are some sites actually using XPS for OLTP. And from what I know about >XPS there is nothing really wrong with doing this, it simply gets back to >how the engine is tuned, and what the expectations are in terms of the >more limited SQL available on XPS. As engines supposedly merge these >issues may become moot. > I hope so, I have heard that XPS on 1 server can be faster than 7.x. Hopefully all the code streams will soon me merged certain 7.x and 9.x should be merged (You know the O-O bits added in) ... and bitwise indexes would be cool for the fields we need to index now.. Once this is done and stable (Plug for 7.3x with it's enhanced features in this area.. lots of nice debug info on a crash!) than 9.x and XPS should be the next merge. Hopefully the XPS guys can do lots of R&D playing with big datasets and find out what works fast.. oh for a nice toy to play with... Would be nice to have an XPS/7.x hybrid where each node is a 7.x engine so that multiple SMP machines can be join into one bug server. i.e. knowledge that this CPU is on this machine and can share memory space and it faster to communicate with. (visions of BGP and routing tables appearing in the engine.. (look the hop count to this node is lower hence... No ..No ..must stop drinking vodka!!)). >And... > >Keep in mind a table-level restore has been the number one requested feature >for quite some time. It was the only feature that actually beat Linux in a >user survey right before Phil White got the boot. > >Thanks, > >Tim -- David Williams

A rather lively and interesting exchange of ideas! In the hopes of solving the table-level restore issue, we are experimenting with BMC Datatools' SQL-Backtrack/OnBar. Not as robust as a full-featured storage manager, but offers decent software compression, table-level restore, restartable restore, and good reporting. IDS 7.24.UC5/Solaris 2.6. XPS version planned. Milton. David Williams wrote: > In article <36D9627C.FFF7A887@mindspring.com>, Tim Schaefer > <tschaefe@mindspring.com> writes > >David Williams wrote: > >> > >> > >> How can you do table level restore when most transactions update > >> >1 table? > >> > > > >This is an assumption for only one scenario, that being a transaction > >on more than one table. But let's think about a different scenario. > > > >Say for example I have a data warehouse, and every week I append data > >to specific tables, one at a time. I then apply updates to all the > >rows in one of the tables, and the table is 12 GB in size. The inserts > >and updates go beautifully and all is well. > > > >Then an analyst comes to me and tells me that this weeks cycle is trash > >can we restore up to last week on this table? Sure, I'll simply drop > >the table, recreate it. and reload last week's unload files. Since I > >don't want to bounce the engine because other users are using the machine > >and I don't want to play games with onmode, this is the fastest recovery. > > > >( By the way, by doing a table-level unload, I can accomplish this to > >disk in an hour and a half. Onbar takes 7 hours to do the same thing > >to ADSM! ) > > > >Onbar doesn't allow me to recover just this table. The option desired > >here is to force a restore into a perfectly good table. I'd pull it in > >from ADSM, and voila, the table is back to last week's precycle data. > > > >Maybe I don't care about the logical logs for this table either, so there > >should be an option to ignore the logs. Onbar should allow me to do > >table-level backups to prepare for the ability to do table-level restores. > >If I have the table frozen periodically, this should work for OLTP or DSS > >systems. > > > Sounds like a good feature. I just think that since Onbar is new that > Informix have go the obvious and implemented the features which will > be used most first. > > I hope that Onbar is enhanced to include other features.. > > PS Who co-ordinates feature requests from c.d.i? How about a list of > requested features? I'll volunteer to keep it on my Web Site... > I'd really like to help... > > <Snip> > >This is probably true for now. However I would challenge the notion that > >XPS is or will be for DSS only in the near future. I've heard that there > >are some sites actually using XPS for OLTP. And from what I know about > >XPS there is nothing really wrong with doing this, it simply gets back to > >how the engine is tuned, and what the expectations are in terms of the > >more limited SQL available on XPS. As engines supposedly merge these > >issues may become moot. > > > I hope so, I have heard that XPS on 1 server can be faster than 7.x. > Hopefully all the code streams will soon me merged certain 7.x and > 9.x should be merged (You know the O-O bits added in) ... and bitwise > indexes would be cool for the fields we need to index now.. > > Once this is done and stable (Plug for 7.3x with it's enhanced > features in this area.. lots of nice debug info on a crash!) > than 9.x and XPS should be the next merge. > > Hopefully the XPS guys can do lots of R&D playing with big datasets > and find out what works fast.. oh for a nice toy to play with... > > Would be nice to have an XPS/7.x hybrid where each node is a 7.x > engine so that multiple SMP machines can be join into one bug server. > i.e. knowledge that this CPU is on this machine and can share memory > space and it faster to communicate with. (visions of BGP and routing > tables appearing in the engine.. (look the hop count to this node is > lower hence... No ..No ..must stop drinking vodka!!)). > > >And... > > > > >Keep in mind a table-level restore has been the number one requested feature > >for quite some time. It was the only feature that actually beat Linux in a > >user survey right before Phil White got the boot. > > > >Thanks, > > > >Tim > > -- > David Williams

David Williams wrote: > > In article <36D9627C.FFF7A887@mindspring.com>, Tim Schaefer > <tschaefe@mindspring.com> writes > >David Williams wrote: > >> > >> > > > Sounds like a good feature. I just think that since Onbar is new that > Informix have go the obvious and implemented the features which will > be used most first. > > I hope that Onbar is enhanced to include other features.. > > PS Who co-ordinates feature requests from c.d.i? How about a list of > requested features? I'll volunteer to keep it on my Web Site... > I'd really like to help... > > David, you have the email addresses to many of the same folks I do if you want to coordinate with the IIUG. > <Snip> > >This is probably true for now. However I would challenge the notion that > >XPS is or will be for DSS only in the near future. I've heard that there > >are some sites actually using XPS for OLTP. And from what I know about > >XPS there is nothing really wrong with doing this, it simply gets back to > >how the engine is tuned, and what the expectations are in terms of the > >more limited SQL available on XPS. As engines supposedly merge these > >issues may become moot. > > > I hope so, I have heard that XPS on 1 server can be faster than 7.x. > Hopefully all the code streams will soon me merged certain 7.x and > 9.x should be merged (You know the O-O bits added in) ... and bitwise > indexes would be cool for the fields we need to index now.. > Well, bitmapped indexes are for a very limited use. There are several restrictions to using them, none of which I can remember off the top of my head, but you can read about them by downloading the XPS docs. Bitmapped indexes are not simply a new kind of index available to replace "regular" indexes. They are available as an option if your tables and query meet the stringent criteria. So their benefit is dubious at this point, even though the theory is nice. And I believe it was RedBrick who first implemented this technology. > Once this is done and stable (Plug for 7.3x with it's enhanced > features in this area.. lots of nice debug info on a crash!) > than 9.x and XPS should be the next merge. > I really am wondering how XPS can merge with the other engines. It really is the odd beast. 7.x and 9.x are more familiar spirits than 8.x. I think this is why the latest press releases regarding the TWO paths of engines is where things are really headed. The high performance loader in 8.x is so different from the 7.x loader--it is radically different actually. The things you do with 8.x are also so radically different. > Hopefully the XPS guys can do lots of R&D playing with big datasets > and find out what works fast.. oh for a nice toy to play with... > > Would be nice to have an XPS/7.x hybrid where each node is a 7.x > engine so that multiple SMP machines can be join into one bug server. See, this why XPS doesn't sell well. Until you have tasted a real multi-server, multi-cpu cluster, this is what folks want. "Cluster those 7.x boxes together and off we go.". :-) The real question is, once you cluster them together, now what? How do you manage them? To get to the end of the road you need to understand the difference between shared-nothing and shared-everything architectures. Clustering 7.x boxes together is only the beginning of the process. Oracle essentially does this, with a shared-everything architecture. This is what I believe Beowulf is for Linux, where there is a monolithic shared memory map across all the servers. But I've not investigated Beowulf to the degree that I should, so I could be wrong about it. I do believe Oracle will probably jump on the Beowulf bandwagon before too long and really push clustered data bases on Linux. My intuition never fails me. This is why Informix needs to pull their heads out, push shared-nothing, and of course XPS on Linux. Looking down the road, XPS on Linux makes a lot of sense. Even if the only pipe available is ethernet, and not proprietary switches like ESCON, XPS on Linux makes sense because XPS is available on NT, and this is how it works on NT. If Oracle gets their clustering pumped up on Linux this is where Informix needs to shine, and do it quick. Most Linux people will gobble up a clustered data base system as easily as anything else you throw at them. Informix has an excellent opportunity in front of them to take the ball and run with it. I know there are people inside who were ready last year to put XPS on Linux, they simply need to get it out. It may even be ported, simply awaiting the right time to release. You can read more about shared-nothing and parallel computing in the book "Parallel Systems in the Data Warehouse". Available at amazon.com. The architectural overview of the many different parallel systems alone is worth the price. > i.e. knowledge that this CPU is on this machine and can share memory > space and it faster to communicate with. (visions of BGP and routing Shared-nothing does not share memory across servers, shared-everything does. Stay non-monolithic and you'll be on your way. :-) > tables appearing in the engine.. (look the hop count to this node is > lower hence... No ..No ..must stop drinking vodka!!)). > You might prepare for spring with a switch to something more tropical such as a good rum. :-) > >And... > > > > >Keep in mind a table-level restore has been the number one requested feature > >for quite some time. It was the only feature that actually beat Linux in a > >user survey right before Phil White got the boot. > > > >Thanks, > > > >Tim > > -- > David Williams -- - -- --- Tim Schaefer ---- tschaefe@mindspring.com --- http://www.inxutil.com -- -

I forgot to add this URL: http://www.linuxworld.com/linuxworld/lw-1999-02/lw-02-clustering.html Bye! :-) Tim -- - -- --- Tim Schaefer ---- tschaefe@mindspring.com --- http://www.inxutil.com -- -