Tip: ontape + gzip parallel compress

Posted in 2010

Topics: Backup & Restore, Platform-Specific Issues

Cesar Inacio Martins — Tue, 28 Sep 2010 12:59:23 — source: IIUG Forums & Mailing Lists

Hi people, This is a tip for who work with multi-core Linux machines, have a good I/O (fast) and want try improve the ontape time of your backups. I executed some tests with the PIGZ - parallel implementation of gzip . http://www.zlib.net/pigz/ This program use the advantage of multi-cores to compacting . In a test environment, where I have a 8 cores with 2 threads each (16 cpus from linux point of view) running red hat, with emc storage where I got I/O average of 280MB/s and an archive without compact of 280GB . Running just with ontape : 45 minutes Running with ontape + gzip -1 (fast compact): 1h15m Running with ontape + pigz -1 (fast compact),: 35 minutes (using the 16 cores and blocks of 1MB) time ontape -s -L 0 -v -t STDIO | pigz -1c -b 1024 > archive.L0.gz .... real 35m21.294s The generated file is 100% gzip , with size of 67 GB . Regards Cesar

↪ replying to Cesar Inacio Martins

Ivan Zavis DBA — Wed, 29 Sep 2010 07:10:57 — source: IIUG Forums & Mailing Lists

Yes, it is great, I'm using this too! But, when you do restore - unpigz does not use all cores, and I had some problem with unpigz when I did restore logical log until I set following: TAPESIZE 0 LTAPEBLK 1024 LTAPESIZE 102400000 Did you try full restore (physical + logical) with unpigz filter ? On 28.09.2010 14:59, Cesar Inacio Martins wrote: > Hi people, > > This is a tip for who work with multi-core Linux machines, have a good I/O > (fast) and want try improve the ontape time of your backups. > > I executed some tests with the PIGZ - parallel implementation of gzip . > http://www.zlib.net/pigz/ > > This program use the advantage of multi-cores to compacting . > > In a test environment, where I have a 8 cores with 2 threads each (16 cpus > from linux point of view) running red hat, with emc storage where I got I/O > average of 280MB/s and an archive without compact of 280GB . > > Running just with ontape : 45 minutes > Running with ontape + gzip -1 (fast compact): 1h15m > Running with ontape + pigz -1 (fast compact),: 35 minutes > (using the 16 cores and blocks of 1MB) > > time ontape -s -L 0 -v -t STDIO | pigz -1c -b 1024> archive.L0.gz > ..... > real 35m21.294s > > The generated file is 100% gzip , with size of 67 GB . > > Regards > Cesar > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > -- Ivan Zaviç System& DB Administrator Mobile:+381-64-846-99-08 mail: ivan.zavis@mi-system.co.rs _________________________________________________________ M&I SYSTEMS CO. Cirila i Metodija 13/a, 21000 Novi Sad, Serbia Tel/Fax: +381-(0)21-68-98-600 Mail: info@mi-system.co.rs, URL: http://www.mi-system.co.rs

↪ replying to Cesar Inacio Martins

Ivan Zavis DBA — Wed, 29 Sep 2010 07:17:44 — source: IIUG Forums & Mailing Lists

BTW I'm using this with BACKUP filter in $ONCONFIG .... (IDS 11.50) BACKUP_FILTER /usr/bin/pigz RESTORE_FILTER /usr/bin/unpigz On 28.09.2010 14:59, Cesar Inacio Martins wrote: > Hi people, > > This is a tip for who work with multi-core Linux machines, have a good I/O > (fast) and want try improve the ontape time of your backups. > > I executed some tests with the PIGZ - parallel implementation of gzip . > http://www.zlib.net/pigz/ > > This program use the advantage of multi-cores to compacting . > > In a test environment, where I have a 8 cores with 2 threads each (16 cpus > from linux point of view) running red hat, with emc storage where I got I/O > average of 280MB/s and an archive without compact of 280GB . > > Running just with ontape : 45 minutes > Running with ontape + gzip -1 (fast compact): 1h15m > Running with ontape + pigz -1 (fast compact),: 35 minutes > (using the 16 cores and blocks of 1MB) > > time ontape -s -L 0 -v -t STDIO | pigz -1c -b 1024> archive.L0.gz > ..... > real 35m21.294s > > The generated file is 100% gzip , with size of 67 GB . > > Regards > Cesar > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > -- Ivan Zaviç System& DB Administrator Mobile:+381-64-846-99-08 mail: ivan.zavis@mi-system.co.rs _________________________________________________________ M&I SYSTEMS CO. Cirila i Metodija 13/a, 21000 Novi Sad, Serbia Tel/Fax: +381-(0)21-68-98-600 Mail: info@mi-system.co.rs, URL: http://www.mi-system.co.rs

↪ replying to Ivan Zavis DBA

Cesar Inacio Martins — Wed, 29 Sep 2010 10:26:28 — source: IIUG Forums & Mailing Lists

Hi Ivan, Yes, for decompress, they don't use all cores. This is a documented limitation. They creates only some threads for the I/O process. With logical log, I don't execute any tests. At first I don't need for this, because logical logs are smaller and the gzip solve the problem. Just a note , here I'm using into a test environment . Not production... I'm still "validating" the and testing the file created with pigz. Cesar --- Em qua, 29/9/10, Ivan Zavis DBA <ivan.zavis@mi-system.co.rs> escreveu: De: Ivan Zavis DBA <ivan.zavis@mi-system.co.rs> Assunto: Re: Tip: ontape + gzip parallel compress [21481] Para: ids@iiug.org Data: Quarta-feira, 29 de Setembro de 2010, 4:10 Yes, it is great, I'm using this too! But, when you do restore - unpigz does not use all cores, and I had some problem with unpigz when I did restore logical log until I set following: TAPESIZE 0 LTAPEBLK 1024 LTAPESIZE 102400000 Did you try full restore (physical + logical) with unpigz filter ? On 28.09.2010 14:59, Cesar Inacio Martins wrote: > Hi people, > > This is a tip for who work with multi-core Linux machines, have a good I/O > (fast) and want try improve the ontape time of your backups. > > I executed some tests with the PIGZ - parallel implementation of gzip . > http://www.zlib.net/pigz/ > > This program use the advantage of multi-cores to compacting . > > In a test environment, where I have a 8 cores with 2 threads each (16 cpus > from linux point of view) running red hat, with emc storage where I got I/O > average of 280MB/s and an archive without compact of 280GB . > > Running just with ontape : 45 minutes > Running with ontape + gzip -1 (fast compact): 1h15m > Running with ontape + pigz -1 (fast compact),: 35 minutes > (using the 16 cores and blocks of 1MB) > > time ontape -s -L 0 -v -t STDIO | pigz -1c -b 1024> archive.L0.gz > ..... > real 35m21.294s > > The generated file is 100% gzip , with size of 67 GB . > > Regards > Cesar > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > -- Ivan Zaviç System& DB Administrator Mobile:+381-64-846-99-08 mail: ivan.zavis@mi-system.co.rs _________________________________________________________ M&I SYSTEMS CO. Cirila i Metodija 13/a, 21000 Novi Sad, Serbia Tel/Fax: +381-(0)21-68-98-600 Mail: info@mi-system.co.rs, URL: http://www.mi-system.co.rs ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Glossary

ontape