Speeding HDR ontape archive-restore
The poster streams an ontape level-0 archive from an HDR primary over the network to seed a secondary (HP-UX 11.23, IDS 11.50.FC3) and asks whether the ONCONFIG TAPE parameters matter when no real tape is involved, and how to speed things up. Replies: yes, TAPEBLK/block size still affects throughput — ideally match the filesystem stripe size, and for network transfers tune it to what the network drivers handle (larger packets reduce header overhead). Another suggestion was piping the archive through the parallel compressor pigz over ssh into ontape -p, which cut one site's run from 30 to 7 minutes, though compiling pigz on HP-UX was untested. No feedback from the original poster on results.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
I am streaming an ontape archive from an HDR primary server over my
network and restoring it to a secondary server to start replication.
Do the onconfig file's TAPE-related parameters affect the speed of
the
transfer even though a physical tape isn't in use?
Are there any other steps I can take to improve the performance of the
archive-restore?
Using HPUX 11.23 and IDS 11.50.FC3.
Block size seems to have an effect on archive/restore performance even
to/from files, though obviously not as severe as when performing IO to/from
tape. The ideal block size will depend on how the filesystem the files
lives on was built. Matching the stripe size of the filesystem would be
ideal.
Art
Art S. Kagel
Advanced DataTools (www.advancedatatools.com)
IIUG Board of Directors (art@iiug.org)
Blog: http://informix-myview.blogspot.com/
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Advanced DataTools, the IIUG, nor any other
organization with which I am associated either explicitly, implicitly, or by
inference. Neither do those opinions reflect those of other individuals
affiliated with any entity with which I am affiliated nor those of the
entities themselves.
On Tue, Dec 7, 2010 at 2:05 PM, red_valsen <red_valsen@yahoo.com> wrote:
> I am streaming an ontape archive from an HDR primary server over my
> network and restoring it to a secondary server to start replication.
> Do the onconfig file's TAPE-related parameters affect the speed of
> the
> transfer even though a physical tape isn't in use?
>
> Are there any other steps I can take to improve the performance of the
> archive-restore?
>
> Using HPUX 11.23 and IDS 11.50.FC3.
> _______________________________________________
> Informix-list mailing list
> Informix-list@iiug.org
> http://www.iiug.org/mailman/listinfo/informix-list
>
Maybe this isn't valid for your environment...
What I already used to improve performance over archive/restore to
configure HDR/RSS and I get great improve over total execution time (30
minutes to 7 minutes), is using the pigz utility, where it will
parallelize your CPU , force more I/O throughput (ontape reading) and
will optimize your network transfer.
http://www.zlib.net/pigz/
- It isn't an Informix utility
- I don't know if is easy to compile it for HP-UX (I used it only with
Linux)
- Will help only if you have multicore machine and great I/O throughput.
Regards
Cesar
On 12/07/2010 05:05 PM, red_valsen wrote:
> I am streaming an ontape archive from an HDR primary server over my
> network and restoring it to a secondary server to start replication.
> Do the onconfig file's TAPE-related parameters affect the speed of
> the
> transfer even though a physical tape isn't in use?
>
> Are there any other steps I can take to improve the performance of the
> archive-restore?
>
> Using HPUX 11.23 and IDS 11.50.FC3.
> _______________________________________________
> Informix-list mailing list
> Informix-list@iiug.org
> http://www.iiug.org/mailman/listinfo/informix-list
I forgot, the command will be something like:
ontape -s -d -v -L 0 -t STDIO | pgiz -1c | \\
ssh informix@otherserver ". myenvs.sh ; pigz -cd | ontape -p -t STDIO"
On 12/08/2010 11:07 AM, Cesar Inacio Martins wrote:
> Maybe this isn't valid for your environment...
>
> What I already used to improve performance over archive/restore to
> configure HDR/RSS and I get great improve over total execution time (30
> minutes to 7 minutes), is using the pigz utility, where it will
> parallelize your CPU , force more I/O throughput (ontape reading) and
> will optimize your network transfer.
>
> http://www.zlib.net/pigz/
>
> - It isn't an Informix utility
> - I don't know if is easy to compile it for HP-UX (I used it only with
> Linux)
> - Will help only if you have multicore machine and great I/O throughput.
>
>
> Regards
> Cesar
>
> On 12/07/2010 05:05 PM, red_valsen wrote:
>> I am streaming an ontape archive from an HDR primary server over my
>> network and restoring it to a secondary server to start replication.
>> Do the onconfig file's TAPE-related parameters affect the speed of
>> the
>> transfer even though a physical tape isn't in use?
>>
>> Are there any other steps I can take to improve the performance of the
>> archive-restore?
>>
>> Using HPUX 11.23 and IDS 11.50.FC3.
>> _______________________________________________
>> Informix-list mailing list
>> Informix-list@iiug.org
>> http://www.iiug.org/mailman/listinfo/informix-list
>> "Art Kagel" <art.kagel@gmail.com> wrote in message >> news:mailman.677.1291749356.1071.informix-list@iiug.org... Block size seems to have an effect on archive/restore performance even to/from files, though obviously not as severe as when performing IO to/from tape. The ideal block size will depend on how the filesystem the files lives on was built. Matching the stripe size of the filesystem would be ideal. I don't know the answer to this, so it's a genuine question: might not the outbound block size have at least as much impact on the network performance as the stripe size for restore ...?
If you are backing up over the network, then yes, blocksize should be coordinated with the system's network drivers. IP packets can be from 2K to 32K but most drivers can't handle 32K packets and break them up into smaller packets. That said, the larger the actual packets that are sent are the less waste the IP and TCP packet headers will represent and the better the throughput. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) IIUG Board of Directors (art@iiug.org) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Wed, Dec 8, 2010 at 7:39 PM, Neil Truby <neil.truby@ardenta.com> wrote: > >> "Art Kagel" <art.kagel@gmail.com> wrote in message > >> news:mailman.677.1291749356.1071.informix-list@iiug.org... > Block size seems to have an effect on archive/restore performance even > to/from files, though obviously not as severe as when performing IO to/from > tape. The ideal block size will depend on how the filesystem the files > lives on was built. Matching the stripe size of the filesystem would be > ideal. > > > I don't know the answer to this, so it's a genuine question: might not the > outbound block size have at least as much impact on the network performance > as the stripe size for restore ...? > > > _______________________________________________ > Informix-list mailing list > Informix-list@iiug.org > http://www.iiug.org/mailman/listinfo/informix-list >