Backup/Restore

Hello, We're running on AIX 4.3, ids 7.31. The backup of the system lasts for about 30 minutes, but the restore(on a duplicate test-system) lasts 3 hours. I heard that the restore may last 2 or 3 times the backup time, but 3 hours is too long. Nobody could find anything in soft- or hardware. Does anybody know this problem on aix-systems, is there a known problem? I'll be delighted if someone can help us. Thanks in advance, Paul Hartman Netherlands

Hi, the information you're giving is quite general. Can you be more specific ? E.g. regarding following considerations: - what backup/restore method are you using ? (ON-Bar, ontape, onarchive ?) - how big is the system that you're backing up ? (onstat -d output helps counting used pages vs. free pages ...) - are you restoring logical logs also ? - is the system very busy while the backup is taken ? (re-playing many log records takes some time.) - are there open transactions when the backup is taken ? Are they rather long ? (even when not re-playing logical logs, open transactions need to be rolled back upon restore. If they were long at backup time, rollback will take considerable time.) - are you using cooked files on the restore-system ? (if they have "size of zero" at restore time, they need to be "blown up" to the necessary size. On some systems this can be rather slow.) - check your OnLine Message Log file of the restored system. This will tell more specifically, which phase of the restore takes long. (If you're using ON-Bar, also check the ON-Bar Activity Log file.) I hope this gives you some ideas. Regards, Martin -- Martin Fuerderer IBM Informix Development Munich Data Management Solutions "Paul Hartman " <Heartware@freeler.nl> Sent by: forum.subscriber@iiug.org 22.09.2003 09:03 To: ids@iiug.org cc: Subject: Backup/Restore [1908] Hello, We're running on AIX 4.3, ids 7.31. The backup of the system lasts for about 30 minutes, but the restore(on a duplicate test-system) lasts 3 hours. I heard that the restore may last 2 or 3 times the backup time, but 3 hours is too long. Nobody could find anything in soft- or hardware. Does anybody know this problem on aix-systems, is there a known problem? I'll be delighted if someone can help us. Thanks in advance, Paul Hartman Netherlands

We had a what a similar problem and finally discovered that the head on the tape drive which was doing the restore were not aligned the same as the backup tape drive. The restore would complete but very, very slowly. A new tape drive solved the problem. HTH, -Darryl -----Original Message----- From: Paul Hartman [mailto:Heartware@freeler.nl] Sent: Monday, September 22, 2003 3:03 AM To: ids@iiug.org Subject: Backup/Restore [1908] Hello, We're running on AIX 4.3, ids 7.31. The backup of the system lasts for about 30 minutes, but the restore(on a duplicate test-system) lasts 3 hours. I heard that the restore may last 2 or 3 times the backup time, but 3 hours is too long. Nobody could find anything in soft- or hardware. Does anybody know this problem on aix-systems, is there a known problem? I'll be delighted if someone can help us. Thanks in advance, Paul Hartman Netherlands ____________________________________________________________________________ The information contained in this communication may be confidential, is intended only for the use of the recipient named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please re-send this communication to the sender and delete the original message and any copy of it from your computer system. Thank you. For more information please visit us at http://www.piperrudnick.com ____________________________________________________________________________

Hello, Let me answer all questions and give some future answers: We're using ontape -r, with a blocksize of 128 Kb The system is about 8 Gb I'm NOT restoring any logical logs The system is NOT busy. sar is about 95% idle. And the time I'm talking about is just the time for restoring the chunks from tape. The chunks aren't cooked. We use 29 raw chunks. The log file just describes the start and the end of the restoration. Only checkpoints are showed. There can be open transactions. The snapshot (level 0 tape backup) occurs in bright daylight. The Physical and Logical Recovery takes about 10 minutes. ACX (Access Control from CA) runs also on the machine as a kernel extension. But the party that rules that application(Security) doesn't see any violations. Maybe someone does ring a bell? Thx in advance, Paul Hartman -----Oorspronkelijk bericht----- Van: Martin Fuerderer [mailto:MARTINFU@de.ibm.com] Verzonden: maandag 22 september 2003 11:57 Aan: Paul Hartman CC: forum.subscriber@iiug.org; ids@iiug.org Onderwerp: Re: Backup/Restore [1908] Hi, the information you're giving is quite general. Can you be more specific ? E.g. regarding following considerations: - what backup/restore method are you using ? (ON-Bar, ontape, onarchive ?) - how big is the system that you're backing up ? (onstat -d output helps counting used pages vs. free pages ...) - are you restoring logical logs also ? - is the system very busy while the backup is taken ? (re-playing many log records takes some time.) - are there open transactions when the backup is taken ? Are they rather long ? (even when not re-playing logical logs, open transactions need to be rolled back upon restore. If they were long at backup time, rollback will take considerable time.) - are you using cooked files on the restore-system ? (if they have "size of zero" at restore time, they need to be "blown up" to the necessary size. On some systems this can be rather slow.) - check your OnLine Message Log file of the restored system. This will tell more specifically, which phase of the restore takes long. (If you're using ON-Bar, also check the ON-Bar Activity Log file.) I hope this gives you some ideas. Regards, Martin -- Martin Fuerderer IBM Informix Development Munich Data Management Solutions "Paul Hartman " <Heartware@freeler.nl> Sent by: forum.subscriber@iiug.org 22.09.2003 09:03 To: ids@iiug.org cc: Subject: Backup/Restore [1908] Hello, We're running on AIX 4.3, ids 7.31. The backup of the system lasts for about 30 minutes, but the restore(on a duplicate test-system) lasts 3 hours. I heard that the restore may last 2 or 3 times the backup time, but 3 hours is too long. Nobody could find anything in soft- or hardware. Does anybody know this problem on aix-systems, is there a known problem? I'll be delighted if someone can help us. Thanks in advance, Paul Hartman Netherlands

Hello , This weekend we restored the database and there was a normal restore time. I really don't know why. Maybe because a third party has made some impriovement to the network. Is it possible that a bad network connection,a bad connection from the tty-terminal with the machine and e.g. a bad connection from ontape with the terminal can cause performance problems? That day we had a network-dip and we lost connection with the tty, and ontape too. In that situation ended ontape his actions , freed his memory and rew the tape. Could it be that ontape every second watches if the connection exist? Thanxs in advance, Paul Hartman -----Oorspronkelijk bericht----- Van: forum.subscriber@iiug.org [mailto:forum.subscriber@iiug.org]Namens Paul Hartman Verzonden: dinsdag 23 september 2003 21:28 Aan: ids@iiug.org Onderwerp: RE: Backup/Restore [1915] Hello, Let me answer all questions and give some future answers: We're using ontape -r, with a blocksize of 128 Kb The system is about 8 Gb I'm NOT restoring any logical logs The system is NOT busy. sar is about 95% idle. And the time I'm talking about is just the time for restoring the chunks from tape. The chunks aren't cooked. We use 29 raw chunks. The log file just describes the start and the end of the restoration. Only checkpoints are showed. There can be open transactions. The snapshot (level 0 tape backup) occurs in bright daylight. The Physical and Logical Recovery takes about 10 minutes. ACX (Access Control from CA) runs also on the machine as a kernel extension. But the party that rules that application(Security) doesn't see any violations. Maybe someone does ring a bell? Thx in advance, Paul Hartman -----Oorspronkelijk bericht----- Van: Martin Fuerderer [mailto:MARTINFU@de.ibm.com] Verzonden: maandag 22 september 2003 11:57 Aan: Paul Hartman CC: forum.subscriber@iiug.org; ids@iiug.org Onderwerp: Re: Backup/Restore [1908] Hi, the information you're giving is quite general. Can you be more specific ? E.g. regarding following considerations: - what backup/restore method are you using ? (ON-Bar, ontape, onarchive ?) - how big is the system that you're backing up ? (onstat -d output helps counting used pages vs. free pages ...) - are you restoring logical logs also ? - is the system very busy while the backup is taken ? (re-playing many log records takes some time.) - are there open transactions when the backup is taken ? Are they rather long ? (even when not re-playing logical logs, open transactions need to be rolled back upon restore. If they were long at backup time, rollback will take considerable time.) - are you using cooked files on the restore-system ? (if they have "size of zero" at restore time, they need to be "blown up" to the necessary size. On some systems this can be rather slow.) - check your OnLine Message Log file of the restored system. This will tell more specifically, which phase of the restore takes long. (If you're using ON-Bar, also check the ON-Bar Activity Log file.) I hope this gives you some ideas. Regards, Martin -- Martin Fuerderer IBM Informix Development Munich Data Management Solutions "Paul Hartman " <Heartware@freeler.nl> Sent by: forum.subscriber@iiug.org 22.09.2003 09:03 To: ids@iiug.org cc: Subject: Backup/Restore [1908] Hello, We're running on AIX 4.3, ids 7.31. The backup of the system lasts for about 30 minutes, but the restore(on a duplicate test-system) lasts 3 hours. I heard that the restore may last 2 or 3 times the backup time, but 3 hours is too long. Nobody could find anything in soft- or hardware. Does anybody know this problem on aix-systems, is there a known problem? I'll be delighted if someone can help us. Thanks in advance, Paul Hartman Netherlands

Hi, since it was the physical restore that took such a long time before, I would rather think that ontape's reading from the tape device was slow. Have you configured the TAPEDEV as a remote tape device on the machine where you restore ? I.e. the tape device from the production machine where you take the archive as a remote tape device on the restore machine ? I would suspect this to be the slow link in the chain ... You could check for this by writing/reading from the tape device on the 2 machines via Unix system utility "dd" and measure the throughput. If you're just taking the tape from the archive machine to the restore machine (using local tape devices), then a different blocking factor of the tape devices might still cause you problems. Some tape devices even "decide" to rewind automatically as soon as the application (here ontape) intermediately stops an ongoing read or write operation. In that case continuing the reading will first require wind forward to where it paused before. This can sometimes slow very fast tape devices to a crawl ... I do not think that a slow connection from ontape to the controlling terminal really is the problem, because after the physical restore has started there are (as far as I know) no outputs from ontape to the terminal until the physical restore is done ... Regards, Martin -- Martin Fuerderer IBM Informix Development Munich Data Management Solutions "Paul Hartman" <Heartware@freeler.nl> Sent by: forum.subscriber@iiug.org 28.09.2003 20:54 To: ids@iiug.org cc: Subject: RE: Backup/Restore [1945] Hello , This weekend we restored the database and there was a normal restore time. I really don't know why. Maybe because a third party has made some impriovement to the network. Is it possible that a bad network connection,a bad connection from the tty-terminal with the machine and e.g. a bad connection from ontape with the terminal can cause performance problems? That day we had a network-dip and we lost connection with the tty, and ontape too. In that situation ended ontape his actions , freed his memory and rew the tape. Could it be that ontape every second watches if the connection exist? Thanxs in advance, Paul Hartman -----Oorspronkelijk bericht----- Van: forum.subscriber@iiug.org [mailto:forum.subscriber@iiug.org]Namens Paul Hartman Verzonden: dinsdag 23 september 2003 21:28 Aan: ids@iiug.org Onderwerp: RE: Backup/Restore [1915] Hello, Let me answer all questions and give some future answers: We're using ontape -r, with a blocksize of 128 Kb The system is about 8 Gb I'm NOT restoring any logical logs The system is NOT busy. sar is about 95% idle. And the time I'm talking about is just the time for restoring the chunks from tape. The chunks aren't cooked. We use 29 raw chunks. The log file just describes the start and the end of the restoration. Only checkpoints are showed. There can be open transactions. The snapshot (level 0 tape backup) occurs in bright daylight. The Physical and Logical Recovery takes about 10 minutes. ACX (Access Control from CA) runs also on the machine as a kernel extension. But the party that rules that application(Security) doesn't see any violations. Maybe someone does ring a bell? Thx in advance, Paul Hartman -----Oorspronkelijk bericht----- Van: Martin Fuerderer [mailto:MARTINFU@de.ibm.com] Verzonden: maandag 22 september 2003 11:57 Aan: Paul Hartman CC: forum.subscriber@iiug.org; ids@iiug.org Onderwerp: Re: Backup/Restore [1908] Hi, the information you're giving is quite general. Can you be more specific ? E.g. regarding following considerations: - what backup/restore method are you using ? (ON-Bar, ontape, onarchive ?) - how big is the system that you're backing up ? (onstat -d output helps counting used pages vs. free pages ...) - are you restoring logical logs also ? - is the system very busy while the backup is taken ? (re-playing many log records takes some time.) - are there open transactions when the backup is taken ? Are they rather long ? (even when not re-playing logical logs, open transactions need to be rolled back upon restore. If they were long at backup time, rollback will take considerable time.) - are you using cooked files on the restore-system ? (if they have "size of zero" at restore time, they need to be "blown up" to the necessary size. On some systems this can be rather slow.) - check your OnLine Message Log file of the restored system. This will tell more specifically, which phase of the restore takes long. (If you're using ON-Bar, also check the ON-Bar Activity Log file.) I hope this gives you some ideas. Regards, Martin -- Martin Fuerderer IBM Informix Development Munich Data Management Solutions "Paul Hartman " <Heartware@freeler.nl> Sent by: forum.subscriber@iiug.org 22.09.2003 09:03 To: ids@iiug.org cc: Subject: Backup/Restore [1908] Hello, We're running on AIX 4.3, ids 7.31. The backup of the system lasts for about 30 minutes, but the restore(on a duplicate test-system) lasts 3 hours. I heard that the restore may last 2 or 3 times the backup time, but 3 hours is too long. Nobody could find anything in soft- or hardware. Does anybody know this problem on aix-systems, is there a known problem? I'll be delighted if someone can help us. Thanks in advance, Paul Hartman Netherlands

Hi brother Try this: If you want to regenerate the instance in another box, backup to filesystem using pipes and gzip Then restore the same way. We've used this method for years and believe me it WORKS GREAT AND FAST ! (for example, we can restore some 7 GB instance in less than 10 minutes...) If you need the implementation details I can hand them for sure FYI: hw = intel, os = linux, ids = 9.40 All the best

I think this would be useful to share with the group. Mike Badar ESRI-Denver One International Ct. Broomfield, CO 80021-3200 303-449-7779 mbadar@esri.com www.esri.com > -----Original Message----- > From: ESTEBAN RIEZNIK [mailto:erieznikr@repsolypf.com] > Sent: Tuesday, September 30, 2003 4:08 PM > To: ids@iiug.org > Subject: Re: RE: Backup/Restore [1957] > > > Hi brother > > Try this: If you want to regenerate the instance in another > box, backup to filesystem using pipes and gzip > > Then restore the same way. We've used this method for years > and believe me it WORKS GREAT AND FAST ! (for example, we can > restore some 7 GB instance in less than 10 minutes...) > > If you need the implementation details I can hand them for sure > > FYI: hw = intel, os = linux, ids = 9.40 > > All the best > > >

Hi, right. This works for scenarios where the server instance to be "duplicated" is down during the time of the backup. Alternatively, it is possible to do such backup, which we call "External Backup" (since it's done with methods external to the database) inbetween the following commands: onmode -c block ... do your backup stuff ... onmode -c unblockThese commands ensure that there's no ongoing activity on the system, so the backup will be consistent. For more information about external backup and restore please read the manual "Backup and Restore Guide". But on a system with ongoing user activity and without using above described External Backup method, it is not really possible to create a consistent backup by external means. If you do so and you are then able to restore it to a consistent state, then you were lucky by coincidence. It may work quite often on an instance where there's not much activity, but that does not mean that it will always work on all systems/instances. Regards, Martin -- Martin Fuerderer IBM Informix Development Munich Data Management Solutions "ESTEBAN RIEZNIK" <erieznikr@repsolypf.com> Sent by: forum.subscriber@iiug.org 01.10.2003 00:07 To: ids@iiug.org cc: Subject: Re: RE: Backup/Restore [1957] Hi brother Try this: If you want to regenerate the instance in another box, backup to filesystem using pipes and gzip Then restore the same way. We've used this method for years and believe me it WORKS GREAT AND FAST ! (for example, we can restore some 7 GB instance in less than 10 minutes...) If you need the implementation details I can hand them for sure FYI: hw = intel, os = linux, ids = 9.40 All the best

Backup/Restore method to filesystem (pipes & gzip, unix only!) 1) Create a pipe at any FS, say mknod /foo/bar chmod 660 /foo/bar 2) Set LTAPEDEV to /foo/bar in onconfig file Also set, TAPESIZE to a huge value, e.g. 7168000 - or more! you milleage may vary - and TAPEBLK can be anything e.g. 16 3) create a backup script, something like: ### ontape -s -L0 <<EOF 1>/tmp/log 2>&1 & <<<--- leave a blank line!EOF cat /foo/bar | gzip -c > backup.dat.gz ### This can also be done in two sessions (first runing intape, second running gzip) How to restore: 1) DB must be off-line 2) open one session (e.g. telnet), run: ontape -r 3) open a second session and run: gzip -c /foo/backup.dat.gz > /foo/bar 4) When ontape asks a few questions (e.g. backup logical logs and stuff) don't worry, go back to previous step 3, and the restore will continue Best performance if run in brand new hardware (specially Intel-based boxen) Try it out an check your results Hope it helps.

That is a good mechanism, unless one of your unloads exceed 2GB. Rob -----Original Message----- From: ESTEBAN RIEZNIK [mailto:erieznikr@repsolypf.com] Sent: Tuesday, September 30, 2003 6:08 PM To: ids@iiug.org Subject: Re: RE: Backup/Restore [1957] Hi brother Try this: If you want to regenerate the instance in another box, backup to filesystem using pipes and gzip Then restore the same way. We've used this method for years and believe me it WORKS GREAT AND FAST ! (for example, we can restore some 7 GB instance in less than 10 minutes...) If you need the implementation details I can hand them for sure FYI: hw = intel, os = linux, ids = 9.40 All the best

Hi brothers, here are some corrections 1)Create a pipe at any FS, say mknod /foo/bar p (the "p" at the end for a pipe special file) 2)Set TAPEDEV to /foo/bar in onconfig file (Must be the TAPEDEV backup device) 3) Creat a backup script ontape -s -L 0 <<EOF 1>/tmp/log 2>&1 & (add a space between "-L" and "0") I apologize for the mistakes. Sorry.

So, When you perform a restore from one of these external backups, can you perform a recovery of logical logs? If so, how does Informix know which logical log to start with, and how do you get Informix into a state where it will accept a logical recovery? --John Bejarano --- "Martin Fuer...." <MARTINFU@de.ibm.com> wrote: > Hi, > > right. This works for scenarios where the server > instance to be > "duplicated" is down during the time of the backup. > > Alternatively, it is possible to do such backup, > which we call > "External Backup" (since it's done with methods > external to the > database) inbetween the following commands: > onmode -c block > ... do your backup stuff ... > onmode -c unblock> These commands ensure that there's no ongoing > activity on > the system, so the backup will be consistent. > For more information about external backup and > restore please > read the manual "Backup and Restore Guide". > > But on a system with ongoing user activity and > without using above > described External Backup method, it is not really > possible to create > a consistent backup by external means. If you do so > and you are then > able to restore it to a consistent state, then you > were lucky by > coincidence. It may work quite often on an instance > where there's not > much activity, but that does not mean that it will > always work on all > systems/instances. > > Regards, > Martin > -- > Martin Fuerderer > IBM Informix Development Munich > Data Management Solutions > > > > > > "ESTEBAN RIEZNIK" <erieznikr@repsolypf.com> > Sent by: forum.subscriber@iiug.org > 01.10.2003 00:07 > > > To: ids@iiug.org > cc: > Subject: Re: RE: Backup/Restore > [1957] > > > > Hi brother > > Try this: If you want to regenerate the instance in > another box, backup to > filesystem using pipes and gzip > > Then restore the same way. We've used this method > for years and believe me > it WORKS GREAT AND FAST ! (for example, we can > restore some 7 GB instance > in less than 10 minutes...) > > If you need the implementation details I can hand > them for sure > > FYI: hw = intel, os = linux, ids = 9.40 > > All the best > > > > > >

ESTEBAN RIEZNIK wrote: ... > ... backup to filesystem using pipes and gzip ... ... What do you mean? It sounds interesting. -- Atte, Jesús Antonio Santos Giraldo jeansagi@myrealbox.com jeansagi@netscape.net