Practical difficulty with IDS 10-11.5 upgrade. Inspiration needed please ...!

IDS 10.0 on Solaris 10. We're upgrading from 10 -> 11.5 a customer who's HDR Primary is in the Northern Hemisphere and Secondary is in the Southern, with a consequent high latency on network connectivity. In this upgrade it's necessary to re-establish HDR from the top. The database server is about 300g in size and even the compressed Level 0 is 50g. This gives us a HUGE headache. If anyone has any smart suggestions we'd be very grateful, because this issue has so far prevented us from upgrading: My colleague's paraphrased summary is given below: "When we come to do the upgrade of the main London-Sydney replicated informix pair we're going to have to stop replication and then recover it from a full backup taken after the upgrade in London. This means getting a 50Gb backup file from London to Sydney quickly. Based on the previous estimates done with transferring this over the standard London-Sydney link, it would take anywhere from 2 to 3 days just to do the transfer so from the stopping of replication to its restart could be anything up to 4 days. The HDR data is used for for reporting and would not be updated for this period. The business could only do with the data being out of date for up to 20 hours so this approach does not seem viable. A slight variation would be ftp the backup file to an ftp site and download it to Sydney from there as this may prove quicker than the private London-Sydney link. Even if this is quicker however there are concerns about transmitting all this data via essentially public ftp. We could encrypt the backup file but this would add an overhead, in time and CPU, for this encryption/decryption. We also experimented with using rdiff to enable us to generate a smaller delta file that would allow us to only have to transfer changes in the backup file taken before the upgrade and the one after. This does work on the small scale I have tried it but it can only operate on uncompressed backup files and there simply isn't enough disk space to accommodate this (I estimate that uncompressed the file will be approximately 300Gb in size and two will be required). The servers do have USB ports so it may be possible to buy say a 1TB USB device and use that. I like the idea of this approach but it is quite hard and long winded to test and I doubt IBM would support it if it were to fail which could leave us with an even bigger outage. Another option may just be to use either a USB or tape based backup and have it couriered but again I guess the question of encryption may still arise and we'll have to establish the viability of the tape approach in terms of hardware. Do you have any thoughts on the above or possible alternatives? My primary concern is that once we start the upgrade process we're committed to restoring replication via a backup - pretty much whether we decide to remain on V11 or revert back to V10. Therefore if the transfer/restore is not reliable Sydney reporting could be affected for a far longer time than is said to be acceptable"

Wow, I would not have thought that HDR would work reliably with so long a WAN link! I have a client that had to have their WAN between Long Island and Manhattan cleaned and reconfigured by the Major Telecom vendor for six months before we could get an HDR secondary to stay online. Once they are up on 11.50 they should consider making the secondary an RSS server instead. The full-duplex and asynch communications would improve reliability. Anyway, that aside, maybe the real solution is to use ER instead of HDR/MACH11. If they have primary keys on their tables it's doable and upgrading can be done one server at a time! If they don't have primary keys and fear that adding them will break their applications, then I don't think they have a choice except to bite the bullet and transfer the archive as fast as possible. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Wed, Aug 4, 2010 at 7:36 PM, Neil Truby <neil.truby@ardenta.com> wrote: > IDS 10.0 on Solaris 10. > > We're upgrading from 10 -> 11.5 a customer who's HDR Primary is in the > Northern Hemisphere and Secondary is in the Southern, with a consequent > high > latency on network connectivity. > > In this upgrade it's necessary to re-establish HDR from the top. The > database server is about 300g in size and even the compressed Level 0 is > 50g. This gives us a HUGE headache. If anyone has any smart suggestions > we'd be very grateful, because this issue has so far prevented us from > upgrading: > > My colleague's paraphrased summary is given below: > > "When we come to do the upgrade of the main London-Sydney replicated > informix pair we're going to have to stop replication and then recover it > from a full backup taken after the upgrade in London. This means getting a > 50Gb backup file from London to Sydney quickly. > > Based on the previous estimates done with transferring this over the > standard London-Sydney link, it would take anywhere from 2 to 3 days just > to > do the transfer so from the stopping of replication to its restart could be > anything up to 4 days. The HDR data is used for for reporting and would not > be updated for this period. The business could only do with the data being > out of date for up to 20 hours so this approach does not seem viable. > > A slight variation would be ftp the backup file to an ftp site and download > it to Sydney from there as this may prove quicker than the private > London-Sydney link. Even if this is quicker however there are concerns > about transmitting all this data via essentially public ftp. We could > encrypt the backup file but this would add an overhead, in time and CPU, > for > this encryption/decryption. > > We also experimented with using rdiff to enable us to generate a smaller > delta file that would allow us to only have to transfer changes in the > backup file taken before the upgrade and the one after. This does work on > the small scale I have tried it but it can only operate on uncompressed > backup files and there simply isn't enough disk space to accommodate this > (I > estimate that uncompressed the file will be approximately 300Gb in size and > two will be required). The servers do have USB ports so it may be possible > to buy say a 1TB USB device and use that. I like the idea of this approach > but it is quite hard and long winded to test and I doubt IBM would support > it if it were to fail which could leave us with an even bigger outage. > > Another option may just be to use either a USB or tape based backup and > have > it couriered but again I guess the question of encryption may still arise > and we'll have to establish the viability of the tape approach in terms of > hardware. > > Do you have any thoughts on the above or possible alternatives? My primary > concern is that once we start the upgrade process we're committed to > restoring replication via a backup - pretty much whether we decide to > remain > on V11 or revert back to V10. Therefore if the transfer/restore is not > reliable Sydney reporting could be affected for a far longer time than is > said to be acceptable" > > _______________________________________________ > Informix-list mailing list > Informix-list@iiug.org > http://www.iiug.org/mailman/listinfo/informix-list >

On Aug 5, 7:36 am, "Neil Truby" <neil.tr...@ardenta.com> wrote: > IDS 10.0 on Solaris 10. > > We're upgrading from 10 -> 11.5 a customer who's HDR Primary is in the > Northern Hemisphere and Secondary is in the Southern, with a consequent high > latency on network connectivity. > > In this upgrade it's necessary to re-establish HDR from the top.  The > database server is about 300g in size and even the compressed Level 0 is > 50g.  This gives us a HUGE headache.  If anyone has any smart suggestions > we'd be very grateful, because this issue has so far prevented us from > upgrading: I know top posting is considered bad taste - so I'll 'middle post' Neil, I have to download bunches of data to my workstations on a fairly regular basis - and my down down loads have been painful at best because I work over a VPN - over public networks - and have to funnel through the IBM VPN hosts - and IBM rarely provides it's employees the same level of service as a customer. In a nut-shell - I ain't got no bandwidth. ;-) What I did to speed this process up is to write a front-end to accept input from a stream - and then split the input into multiple pieces and then ftp each of the pieces in parallel to to my system at home. Generally I can get the ftp's completed fairly close to the time that it takes to scan the source. In my case, the front end of the process is something like "tar .... | gzip .... | MyFTPSpliterManager". But it could also be somthing like "ontape -t STDIO | gzip ... | some encryption program | MyFTPSpliterManager". On my target side I just have to combine the pieces into a single file. I suspect that there are tools that could be purchased to do this type of stuff, but generally they expect the input to be a file and not a input stream. I'm in Malaysia this week and will be back in Dallas next week. If you are interested, I'd be glad to send you what I've got. M.P. > > My colleague's paraphrased summary is given below: > > "When we come to do the upgrade of the main London-Sydney replicated > informix pair we're going to have to stop replication and then recover it > from a full backup taken after the upgrade in London. This means getting a > 50Gb backup file from London to Sydney quickly. > > Based on the previous estimates done with  transferring this over the > standard London-Sydney link, it would take anywhere from 2 to 3 days just to > do the transfer so from the stopping of replication to its restart could be > anything up to 4 days. The HDR data is used for for reporting and would not > be updated for this period. The business could only do with the data being > out of date for up to 20 hours so this approach does not seem viable. > > A slight variation would be ftp the backup file to an ftp site and download > it to Sydney from there as this may prove quicker than the private > London-Sydney link.  Even if this is quicker however there are concerns > about transmitting all this data via essentially public ftp.  We could > encrypt the backup file but this would add an overhead, in time and CPU, for > this encryption/decryption. > > We also experimented with using rdiff to enable us to generate a smaller > delta file that would allow us to only have to transfer changes in the > backup file taken before the upgrade and the one after. This does work on > the small scale I have tried it but it can only operate on uncompressed > backup files and there simply isn't enough disk space to accommodate this (I > estimate that uncompressed the file will be approximately 300Gb in size and > two will be required). The servers do have USB ports so it may be possible > to buy say a 1TB USB device and use that.  I like the idea of this approach > but it is quite hard and long winded to test and I doubt IBM would support > it if it were to fail which could leave us with an even bigger outage. > > Another option may just be to use either a USB or tape based backup and have > it couriered but again I guess the question of encryption may still arise > and we'll have to establish the viability of the tape approach in terms of > hardware. > > Do you have any thoughts on the above or possible alternatives? My primary > concern is that once we start the upgrade process we're committed to > restoring replication via a backup - pretty much whether we decide to remain > on V11 or revert back to V10. Therefore if the transfer/restore is not > reliable Sydney reporting could be affected for a far longer time than is > said to be acceptable"

On Aug 5, 7:36 am, "Neil Truby" <neil.tr...@ardenta.com> wrote: > IDS 10.0 on Solaris 10. > > We're upgrading from 10 -> 11.5 a customer who's HDR Primary is in the > Northern Hemisphere and Secondary is in the Southern, with a consequent high > latency on network connectivity. Oh yes... Compress it before you encrypt it. If you encrypt it first, it's not going to compress very well. > > In this upgrade it's necessary to re-establish HDR from the top.  The > database server is about 300g in size and even the compressed Level 0 is > 50g.  This gives us a HUGE headache.  If anyone has any smart suggestions > we'd be very grateful, because this issue has so far prevented us from > upgrading: > > My colleague's paraphrased summary is given below: > > "When we come to do the upgrade of the main London-Sydney replicated > informix pair we're going to have to stop replication and then recover it > from a full backup taken after the upgrade in London. This means getting a > 50Gb backup file from London to Sydney quickly. > > Based on the previous estimates done with  transferring this over the > standard London-Sydney link, it would take anywhere from 2 to 3 days just to > do the transfer so from the stopping of replication to its restart could be > anything up to 4 days. The HDR data is used for for reporting and would not > be updated for this period. The business could only do with the data being > out of date for up to 20 hours so this approach does not seem viable. > > A slight variation would be ftp the backup file to an ftp site and download > it to Sydney from there as this may prove quicker than the private > London-Sydney link.  Even if this is quicker however there are concerns > about transmitting all this data via essentially public ftp.  We could > encrypt the backup file but this would add an overhead, in time and CPU, for > this encryption/decryption. > > We also experimented with using rdiff to enable us to generate a smaller > delta file that would allow us to only have to transfer changes in the > backup file taken before the upgrade and the one after. This does work on > the small scale I have tried it but it can only operate on uncompressed > backup files and there simply isn't enough disk space to accommodate this (I > estimate that uncompressed the file will be approximately 300Gb in size and > two will be required). The servers do have USB ports so it may be possible > to buy say a 1TB USB device and use that.  I like the idea of this approach > but it is quite hard and long winded to test and I doubt IBM would support > it if it were to fail which could leave us with an even bigger outage. > > Another option may just be to use either a USB or tape based backup and have > it couriered but again I guess the question of encryption may still arise > and we'll have to establish the viability of the tape approach in terms of > hardware. > > Do you have any thoughts on the above or possible alternatives? My primary > concern is that once we start the upgrade process we're committed to > restoring replication via a backup - pretty much whether we decide to remain > on V11 or revert back to V10. Therefore if the transfer/restore is not > reliable Sydney reporting could be affected for a far longer time than is > said to be acceptable"

Hi Neil, Not sure this approach work for you or not: 1) Stop HDR 2) Upgrade Primary Server to 11.50 3) Couriered Level 0 backup to Secondary Server site 4a) Level 0 restore at Secondary Server 4b) At Primary Server site, on going FTP Logical log files Backup to Secondary Server site 5) Lastly bring up RSS to catch up the image Happy trying. On Thu, Aug 5, 2010 at 7:36 AM, Neil Truby <neil.truby@ardenta.com> wrote: > IDS 10.0 on Solaris 10. > > We're upgrading from 10 -> 11.5 a customer who's HDR Primary is in the > Northern Hemisphere and Secondary is in the Southern, with a consequent > high > latency on network connectivity. > > In this upgrade it's necessary to re-establish HDR from the top. The > database server is about 300g in size and even the compressed Level 0 is > 50g. This gives us a HUGE headache. If anyone has any smart suggestions > we'd be very grateful, because this issue has so far prevented us from > upgrading: > > My colleague's paraphrased summary is given below: > > "When we come to do the upgrade of the main London-Sydney replicated > informix pair we're going to have to stop replication and then recover it > from a full backup taken after the upgrade in London. This means getting a > 50Gb backup file from London to Sydney quickly. > > Based on the previous estimates done with transferring this over the > standard London-Sydney link, it would take anywhere from 2 to 3 days just > to > do the transfer so from the stopping of replication to its restart could be > anything up to 4 days. The HDR data is used for for reporting and would not > be updated for this period. The business could only do with the data being > out of date for up to 20 hours so this approach does not seem viable. > > A slight variation would be ftp the backup file to an ftp site and download > it to Sydney from there as this may prove quicker than the private > London-Sydney link. Even if this is quicker however there are concerns > about transmitting all this data via essentially public ftp. We could > encrypt the backup file but this would add an overhead, in time and CPU, > for > this encryption/decryption. > > We also experimented with using rdiff to enable us to generate a smaller > delta file that would allow us to only have to transfer changes in the > backup file taken before the upgrade and the one after. This does work on > the small scale I have tried it but it can only operate on uncompressed > backup files and there simply isn't enough disk space to accommodate this > (I > estimate that uncompressed the file will be approximately 300Gb in size and > two will be required). The servers do have USB ports so it may be possible > to buy say a 1TB USB device and use that. I like the idea of this approach > but it is quite hard and long winded to test and I doubt IBM would support > it if it were to fail which could leave us with an even bigger outage. > > Another option may just be to use either a USB or tape based backup and > have > it couriered but again I guess the question of encryption may still arise > and we'll have to establish the viability of the tape approach in terms of > hardware. > > Do you have any thoughts on the above or possible alternatives? My primary > concern is that once we start the upgrade process we're committed to > restoring replication via a backup - pretty much whether we decide to > remain > on V11 or revert back to V10. Therefore if the transfer/restore is not > reliable Sydney reporting could be affected for a far longer time than is > said to be acceptable" > > _______________________________________________ > Informix-list mailing list > Informix-list@iiug.org > http://www.iiug.org/mailman/listinfo/informix-list >

"KN Liew Liew" <liewkn@gmail.com> wrote in message news:mailman.392.1280984482.1071.informix-list@iiug.org... >> Hi Neil, Not sure this approach work for you or not: 1) Stop HDR 2) Upgrade Primary Server to 11.50 3) Couriered Level 0 backup to Secondary Server site 4a) Level 0 restore at Secondary Server 4b) At Primary Server site, on going FTP Logical log files Backup to Secondary Server site >> 5) Lastly bring up RSS to catch up the image Hi there. Thanks for the interest. It's HDR, not RSS (though we may change this but don't have the option on 10.0). This is more-or-less exactly what we will do. But this step ... 3) Couriered Level 0 backup to Secondary Server site ... is likely to take 40 hours at best. And we "cannot" take 40 hours (plus backup, compress, encrypt, decrypt, uncompress and restore time) outage. regards Neil

"mpruet" <mpruet1@verizon.net> wrote in message news:0c7399a5-b8b4-4735-b0ab-35348a632e58@d8g2000yqf.googlegroups.com... On Aug 5, 7:36 am, "Neil Truby" <neil.tr...@ardenta.com> wrote: Thanks Madison, that would ge good. I think you may have mentioned this at IIUG actually, come to think of it ... regards

Wouldn't it be possible to create a temporary secondary server on London to allow business to get it's reports while the "real" secondary is being built on Sydney? This would of course need additional hardware.. Apart from that... and this would be tricky and maybe risky... Change the secondary into a standard server and use ER to replicate the tables needed by the reports...Than in parallel create the secondary server on Sydney... In any case, write a press release... I want to see our competition doing that kind of replication across hemispheres :) Regards. On Thu, Aug 5, 2010 at 12:36 AM, Neil Truby <neil.truby@ardenta.com> wrote: > IDS 10.0 on Solaris 10. > > We're upgrading from 10 -> 11.5 a customer who's HDR Primary is in the > Northern Hemisphere and Secondary is in the Southern, with a consequent > high > latency on network connectivity. > > In this upgrade it's necessary to re-establish HDR from the top. The > database server is about 300g in size and even the compressed Level 0 is > 50g. This gives us a HUGE headache. If anyone has any smart suggestions > we'd be very grateful, because this issue has so far prevented us from > upgrading: > > My colleague's paraphrased summary is given below: > > "When we come to do the upgrade of the main London-Sydney replicated > informix pair we're going to have to stop replication and then recover it > from a full backup taken after the upgrade in London. This means getting a > 50Gb backup file from London to Sydney quickly. > > Based on the previous estimates done with transferring this over the > standard London-Sydney link, it would take anywhere from 2 to 3 days just > to > do the transfer so from the stopping of replication to its restart could be > anything up to 4 days. The HDR data is used for for reporting and would not > be updated for this period. The business could only do with the data being > out of date for up to 20 hours so this approach does not seem viable. > > A slight variation would be ftp the backup file to an ftp site and download > it to Sydney from there as this may prove quicker than the private > London-Sydney link. Even if this is quicker however there are concerns > about transmitting all this data via essentially public ftp. We could > encrypt the backup file but this would add an overhead, in time and CPU, > for > this encryption/decryption. > > We also experimented with using rdiff to enable us to generate a smaller > delta file that would allow us to only have to transfer changes in the > backup file taken before the upgrade and the one after. This does work on > the small scale I have tried it but it can only operate on uncompressed > backup files and there simply isn't enough disk space to accommodate this > (I > estimate that uncompressed the file will be approximately 300Gb in size and > two will be required). The servers do have USB ports so it may be possible > to buy say a 1TB USB device and use that. I like the idea of this approach > but it is quite hard and long winded to test and I doubt IBM would support > it if it were to fail which could leave us with an even bigger outage. > > Another option may just be to use either a USB or tape based backup and > have > it couriered but again I guess the question of encryption may still arise > and we'll have to establish the viability of the tape approach in terms of > hardware. > > Do you have any thoughts on the above or possible alternatives? My primary > concern is that once we start the upgrade process we're committed to > restoring replication via a backup - pretty much whether we decide to > remain > on V11 or revert back to V10. Therefore if the transfer/restore is not > reliable Sydney reporting could be affected for a far longer time than is > said to be acceptable" > > _______________________________________________ > Informix-list mailing list > Informix-list@iiug.org > http://www.iiug.org/mailman/listinfo/informix-list > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently...

"Art Kagel" <art.kagel@gmail.com> wrote in message news:mailman.390.1280966291.1071.informix-list@iiug.org... >> Wow, I would not have thought that HDR would work reliably with so long a >> WAN link! I have a client that had to have their WAN between Long Island >> and Manhattan cleaned and reconfigured by the Major Telecom vendor for >> six months before we could get an HDR secondary to stay online. Yes, it is quite imperssive. The link has network accelerators for the HDR port traffic at both ends, and we get longish (~5s) checkpoints, but it's still impressive. And the data volumes were muich smaller of course when we began 3 years ago ... >> Once they are up on 11.50 they should consider making the secondary an >> RSS server instead. The full-duplex and asynch communications would >> improve reliability. Indeed. If we can get on to 11.5 at all ... ;-)

"Fernando Nunes" <domusonline@gmail.com> wrote in message news:mailman.393.1280999508.1071.informix-list@iiug.org... >> Wouldn't it be possible to create a temporary secondary server on London >> to allow business to get it's reports while the "real" secondary is being >> built on Sydney? Doesn't really help, because the actual requirement is not to have an HDR secodary for reslience per se, but to have a near-synchronous copy of live data for reporting in Sydney. We may be able to satisfy the requirement by pointing the reporting apps at London for the duration, but has as-yet-unknown performance implications ... >> In any case, write a press release... I want to see our competition doing >> that kind of replication across hemispheres :) Unfortunately this customer is in one of those industries of which IBM officially disapproves, so you wouldn't be able to cite it much!

> > Do you have any thoughts on the above or possible alternatives? My primary > concern is that once we start the upgrade process we're committed to > restoring replication via a backup - pretty much whether we decide to remain > on V11 or revert back to V10. Therefore if the transfer/restore is not > reliable Sydney reporting could be affected for a far longer time than is > said to be acceptable" Neil Whilst I have not tried this across a version gap as big as this (and you will therefore need to experiment) if you use gzip --rsyncable to compress the archives and use rsync to transfer the 11.5 archive over the top of an already transferred 10.0 archive - you MAY be surprised at the transfer time. Others closer to the internals of an ontape archive may be able to blow this out the water... If indeed it doesn't help enough can you do an upgrade in UK on a test machine just to get a valid archive and get that transferred to rsync over the top of? It might even be faster, once you have transferred such a reference archive to rsync the final archive uncompressed, whilst using rsyncs own on the fly compression. -- Clive -- This message has been scanned for viruses and dangerous content by OpenProtect(http://www.openprotect.com), and is believed to be clean.

> From: neil.truby@ardenta.com > Subject: Re: Practical difficulty with IDS 10-11.5 upgrade. Inspiration neededplease ...! > Date: Thu, 5 Aug 2010 10:15:48 +0100 > To: informix-list@iiug.org > > Unfortunately this customer is in one of those industries of which IBM > officially disapproves, so you wouldn't be able to cite it much! > Porn? Now you're venturing in to Timmy's old Florida gig territory... :-)