ontape -r suddenly very slow

I have 2 Informix 7.31UD9 servers. One live server and one for backup. Every night I do a full backup to tape and then a full restore to the backup server with ontape -r. This has worked fine for years, but suddenly this restore is very slow. I noticed that the disks are 99% busy during this restore. (sar -d) But I don't know why. I already replace the HD and copying large files isn't a problem. How does ontape -r works? Does it use tempory files to restore chunks? I recently added a chunk to an excisting dbspace. Could this cause troubles? Any ideas what the problem might be or where to search? Thanks

Are you change the TAPEBLK parameter in your onconfig? Juan Jorge Cruces Fernández Accelya ----- Original Message ----- From: "ROELAND MOORS" <roeland@falcon.be> To: <ids@iiug.org> Sent: Tuesday, November 25, 2008 1:39 PM Subject: ontape -r suddenly very slow [14104] >I have 2 Informix 7.31UD9 servers. > One live server and one for backup. > Every night I do a full backup to tape and then a full restore to the > backup > server with ontape -r. > This has worked fine for years, but suddenly this restore is very slow. > > I noticed that the disks are 99% busy during this restore. (sar -d) > But I don't know why. I already replace the HD and copying large files > isn't a > problem. > > How does ontape -r works? Does it use tempory files to restore chunks? > I recently added a chunk to an excisting dbspace. Could this cause > troubles? > > Any ideas what the problem might be or where to search? > > Thanks > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > Este mensaje se dirige exclusivamente a su destinatario, contiene información CONFIDENCIAL sometida a secreto profesional y su divulgación está prohibida por ley. Si ha recibido este mensaje por error, su copia y uso están prohibidos, rogándole que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción. El correo electrónico no garantiza la confidencialidad de los mensajes ni su integridad o correcta recepción. La compañía ACCELYA no asume responsabilidad por estas circunstancias. Si el destinatario de este mensaje no consintiera la utilización del correo electrónico y la grabación de los mensajes, rogamos nos lo notifique de forma inmediata. This message is intended exclusively for its addressee. It contains information that is CONFIDENTIAL and protected by a professional privilege or whose disclosure is prohibited by law. If this message has been received in error, you should know that it is forbidden to copy or use it. Please immediately notify us via e-mail and delete it. Internet e-mail neither guarantees the confidentiality nor the integrity or proper receipt of the messages sent. ACCELYA company assumes no liability or responsibility for any error caused by matters beyond our control. If the addressee of this message does not consent to the use of Internet e-mail and message recording, please notify us immediately.

No, I didn't change it. The settings are still the same on both servers: Live-server: TAPEDEV /dev/rmt/0cb TAPEBLK 16 TAPESIZE 72000000 Backup-server: TAPEDEV s3:/dev/rmt/0cb TAPEBLK 16 TAPESIZE 72000000

You didn't say what flavor of tape device you have or what your OS is, but I'd suggest bumping up your TAPEBLK pretty substantially. I found a sweet spot on our systems (IDS 9.40.FC8, HP-UX 11.11, DDS3 DAT tape) with TAPEBLK set at 256. If your system isn't busy during your backup period, you could probably go higher. --EEM > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > ROELAND MOORS > Sent: Tuesday, November 25, 2008 7:46 AM > To: ids@iiug.org > Subject: Re: ontape -r suddenly very slow [14106] > > No, I didn't change it. > The settings are still the same on both servers: > > Live-server: > TAPEDEV /dev/rmt/0cb > TAPEBLK 16 > TAPESIZE 72000000> > Backup-server: > TAPEDEV s3:/dev/rmt/0cb > TAPEBLK 16 > TAPESIZE 72000000> > > ************************************************************************ ** > ***** > Forum Note: Use "Reply" to post a response in the discussion forum.

Tape device: Sun DAT72 OS backup-server: Solaris 2.6 OS live-server: Solaris 9 I could try changing the blocksize. The strange thing is that it used to be 5 hours for a complete restore. Now it's about 15 hours. It's also strange that the disks are for 99% busy when the network connection between the servers is not that fast. (100Mbit/s)

During the restore, using onstat -D, see if you can determine a specific dbspace/chunk that is having issues. Mike

If nothing else it should help. Bumping my block size up dropped my backup time from almost two hours down to about 45 minutes. > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > ROELAND MOORS > Sent: Tuesday, November 25, 2008 8:47 AM > To: ids@iiug.org > Subject: Re: RE: ontape -r suddenly very slow [14110] > > Tape device: Sun DAT72 > OS backup-server: Solaris 2.6 > OS live-server: Solaris 9 > > I could try changing the blocksize. > > The strange thing is that it used to be 5 hours for a complete restore. > Now it's about 15 hours. > > It's also strange that the disks are for 99% busy when the network > connection > between the servers is not that fast. (100Mbit/s) > > > ************************************************************************ ** > ***** > Forum Note: Use "Reply" to post a response in the discussion forum.

Can you do a little test: Try to tar some files to same tape unit (DO NOT USE BACKUP TAPES, use spare) Measure time spent for tar command and calculate speed of tape unit (in MB/s). If this speed is correlated to Informix ontape backup speed, then you have some hardware problem. HTH Hrvoje > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On > Behalf Of ROELAND MOORS > Sent: Tuesday, November 25, 2008 3:47 PM > To: ids@iiug.org > Subject: Re: RE: ontape -r suddenly very slow [14110] > > Tape device: Sun DAT72 > OS backup-server: Solaris 2.6 > OS live-server: Solaris 9 > > I could try changing the blocksize. > > The strange thing is that it used to be 5 hours for a > complete restore. > Now it's about 15 hours. > > It's also strange that the disks are for 99% busy when the > network connection between the servers is not that fast. (100Mbit/s) > > > ************************************************************** > ***************** > Forum Note: Use "Reply" to post a response in the discussion forum. >

You may have hit the "old pages" feature. If the timestamp for your pages being backed up gets "too far behind" the current timestamp (they haven't had any inserts, deletes or updates), ontape will update the timestamp on those pages before backing them up. Still a 'feature' in IDS 10 as I remember also. -- Bob -------------- Original message -------------- From: "ROELAND MOORS" <roeland@falcon.be> > I have 2 Informix 7.31UD9 servers. > One live server and one for backup. > Every night I do a full backup to tape and then a full restore to the backup > server with ontape -r. > This has worked fine for years, but suddenly this restore is very slow. > > I noticed that the disks are 99% busy during this restore. (sar -d) > But I don't know why. I already replace the HD and copying large files isn't a > problem. > > How does ontape -r works? Does it use tempory files to restore chunks? > I recently added a chunk to an excisting dbspace. Could this cause troubles? > > Any ideas what the problem might be or where to search? > > Thanks > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >

Are you restoring to cooked files? If so, you may have write cache disabled at disk sub-system level. Stuart McCann Integrated Spatial Services Unit Information Communication & Technology Department of Lands, Bathurst Phone: (02) 63328284 stuart.mccann@lands.nsw.gov.au -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of ROELAND MOORS Sent: Tuesday, 25 November 2008 11:39 PM To: ids@iiug.org Subject: ontape -r suddenly very slow [14104] I have 2 Informix 7.31UD9 servers. One live server and one for backup. Every night I do a full backup to tape and then a full restore to the backup server with ontape -r. This has worked fine for years, but suddenly this restore is very slow. I noticed that the disks are 99% busy during this restore. (sar -d) But I don't know why. I already replace the HD and copying large files isn't a problem. How does ontape -r works? Does it use tempory files to restore chunks? I recently added a chunk to an excisting dbspace. Could this cause troubles? Any ideas what the problem might be or where to search? Thanks ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum. *************************************************************** This message is intended for the addressee named and may contain confidential information. If you are not the intended recipient, please delete it and notify the sender. Views expressed in this message are those of the individual sender, and are not necessarily the views of the Department of Lands. This email message has been swept by MIMEsweeper for the presence of computer viruses. *************************************************************** Please consider the environment before printing this email.

This doesn't affect restores... Regards. On Tue, Nov 25, 2008 at 3:12 PM, rroussey@comcast.net <rroussey@comcast.net>wrote: > You may have hit the "old pages" feature. If the timestamp for your pages > being backed up gets "too far behind" the current timestamp (they haven't > had > any inserts, deletes or updates), ontape will update the timestamp on those > pages before backing them up. Still a 'feature' in IDS 10 as I remember > also. > > -- > Bob > > -------------- Original message -------------- > From: "ROELAND MOORS" <roeland@falcon.be> > > > I have 2 Informix 7.31UD9 servers. > > One live server and one for backup. > > Every night I do a full backup to tape and then a full restore to the > backup > > server with ontape -r. > > This has worked fine for years, but suddenly this restore is very slow. > > > > I noticed that the disks are 99% busy during this restore. (sar -d) > > But I don't know why. I already replace the HD and copying large files > isn't > a > > problem. > > > > How does ontape -r works? Does it use tempory files to restore chunks? > > I recently added a chunk to an excisting dbspace. Could this cause > troubles? > > > > Any ideas what the problem might be or where to search? > > > > Thanks > > > > > > > > ******************************************************************************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently...

On the other hand, if you're using RAW, confirm they're pointing to character devices... Regards On Tue, Nov 25, 2008 at 9:28 PM, Stuart McCann < Stuart.McCann@lands.nsw.gov.au> wrote: > Are you restoring to cooked files? If so, you may have write cache > disabled at disk sub-system level. > > Stuart McCann > Integrated Spatial Services Unit > Information Communication & Technology > Department of Lands, Bathurst > > Phone: (02) 63328284 > stuart.mccann@lands.nsw.gov.au > > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of > ROELAND MOORS > Sent: Tuesday, 25 November 2008 11:39 PM > To: ids@iiug.org > Subject: ontape -r suddenly very slow [14104] > > I have 2 Informix 7.31UD9 servers. > One live server and one for backup. > Every night I do a full backup to tape and then a full restore to the > backup > server with ontape -r. > This has worked fine for years, but suddenly this restore is very slow. > > I noticed that the disks are 99% busy during this restore. (sar -d) > But I don't know why. I already replace the HD and copying large files > isn't a > problem. > > How does ontape -r works? Does it use tempory files to restore chunks? > I recently added a chunk to an excisting dbspace. Could this cause > troubles? > > Any ideas what the problem might be or where to search? > > Thanks > > ************************************************************************ > ******* > Forum Note: Use "Reply" to post a response in the discussion forum. > > *************************************************************** > This message is intended for the addressee named and may contain > confidential > information. If you are not the intended recipient, please delete it and > notify the sender. Views expressed in this message are those of the > individual > sender, and are not necessarily the views of the Department of Lands. This > email message has been swept by MIMEsweeper for the presence of computer > viruses. > *************************************************************** > Please consider the environment before printing this email. > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently...

I did the test: --- BACKUP LIVESERER TO TAPE --- Wed Nov 26 09:17:07 CET 2008 mkfile 5000m 5gb Wed Nov 26 09:19:20 CET 2008 tar cvf /dev/rmt/0cb ./5gb a ./5gb 10240000 tape blocks Wed Nov 26 09:27:50 CET 2008 -> 9.8 Mb/s --- RESTORE FROM TAPE TO BACKUPSERVER --- Wed Nov 26 09:28:27 MET 2008 rsh -n s3 dd if=/dev/rmt/0cb bs=16384 | tar xvBfb - 16384 x ./5gb, 5242880000 bytes, 10240000 tape blocks 0+512001 records in 0+512001 records out Wed Nov 26 10:01:32 MET 2008 -> 2.52 Mb/s It's slower because it's a restore over the network, but it's fast enough. The restore would take less than 5 hours at this speed.

Hi, I've not gone thru all the messages in this thread (just back from vacation), but what I've not seen yet is info as to which phase of the restore takes how long. There's: - "physical restore" when the data of dbspaces is copied from backup to disk, - "logical restore" when logical log file data is copied from backup to disk and the log records get applied, i.e. rolled forward, - "logical log cleanup phase" when after logical restore there are open transactions that need to be rolled back. It often was experienced that due to changing system activity (during backup time), the last phase can greatly vary in time for restores of different backups. So please check (e.g. in IDS message log). TIA, Martin -- Martin Fuerderer IBM Informix Development Munich, Germany Information Management IBM Deutschland Research & Development GmbH Chairman of the Supervisory Board: Martin Jetter Board of Management: Erich Baier Corporate Seat: Boeblingen, Germany Reg.-Gericht: Amtsgericht Stuttgart, HRB 243294 ids-bounces@iiug.org wrote on 26.11.2008 10:24:56: > I did the test: > > --- BACKUP LIVESERER TO TAPE --- > Wed Nov 26 09:17:07 CET 2008 > mkfile 5000m 5gb > Wed Nov 26 09:19:20 CET 2008 > tar cvf /dev/rmt/0cb ./5gb > a ./5gb 10240000 tape blocks > Wed Nov 26 09:27:50 CET 2008 > -> 9.8 Mb/s > > --- RESTORE FROM TAPE TO BACKUPSERVER --- > Wed Nov 26 09:28:27 MET 2008 > rsh -n s3 dd if=/dev/rmt/0cb bs=16384 | tar xvBfb - 16384 > x ./5gb, 5242880000 bytes, 10240000 tape blocks > 0+512001 records in > 0+512001 records out > Wed Nov 26 10:01:32 MET 2008 > -> 2.52 Mb/s > > It's slower because it's a restore over the network, but it's fast enough. > The restore would take less than 5 hours at this speed. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >

You asked a question about ontape but (I think) no one answered, so let me try: ontape is restoring XXspace one by one, and chunk inside each dbspace one by one. So, you can watch your restore process in several ways: onstat -u => watch "nwrites" column of ontape process, it should increase onstat -g iof => here you can see how ontape is "filling" chunks you can see if it is not reading tape/writing chunks with the same speed, there is some issue (disk?) HTH Hrvoje > -----Original Message----- > From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On > Behalf Of ROELAND MOORS > Sent: Wednesday, November 26, 2008 10:25 AM > To: ids@iiug.org > Subject: Re: RE: RE: ontape -r suddenly very slow [14126] > > I did the test: > > --- BACKUP LIVESERER TO TAPE --- > Wed Nov 26 09:17:07 CET 2008 > mkfile 5000m 5gb > Wed Nov 26 09:19:20 CET 2008 > tar cvf /dev/rmt/0cb ./5gb > a ./5gb 10240000 tape blocks > Wed Nov 26 09:27:50 CET 2008 > -> 9.8 Mb/s > > --- RESTORE FROM TAPE TO BACKUPSERVER --- Wed Nov 26 09:28:27 > MET 2008 rsh -n s3 dd if=/dev/rmt/0cb bs=16384 | tar xvBfb - > 16384 x ./5gb, 5242880000 bytes, 10240000 tape blocks > 0+512001 records in > 0+512001 records out > Wed Nov 26 10:01:32 MET 2008 > -> 2.52 Mb/s > > It's slower because it's a restore over the network, but it's > fast enough. > The restore would take less than 5 hours at this speed. > > > ************************************************************** > ***************** > Forum Note: Use "Reply" to post a response in the discussion forum. >

It's during the physical restore

I'm using cooked files. If restoring from a tar on the tape is working fine on the same disk, this should be no problem? (I don't believe solaris 2.6 has softpartitions like solaris 9)