Integrated Backup Encryption and compression

We're in the process of planning a move from 11.50 on HP-UX to 14.10 on Linux, and I have a question about the Integrated Backup Encryption feature in 14.10. Can the archives/backups be compressed? I'm asking because the compression usually only works well on unencrypted data. Once encrypted, the data usually does not have enough repeating character strings to give good compression results. From a recent test with an 11.50 instance, I can do a level 1 archive of a dev instance that takes 530 MB. If I compress that archive, I can get it down under 100 MB, a reduction of a bit over 82%. But if I PGP encrypt the archive before attempting to compress (and I disable the compress option in PGP), the encryption adds a small amount of overhead, increasing the file size by just a few MB, but that encrypted archive then does not compress at all. In fact, compressing the encrypted file adds yet another MB or two. So, does the Integrated Backup Encryption allow the archives and/or backups to be compressed? What configuration options need to be tweaked to make that happen? Are the BACKUP_FILTER / RESTORE_FILTER parameters still used in 14.10 when using the backup encryption? For those who have implemented Integrated Backup Encryption along with compression (assuming it's feasible), what kind of compression ratios are you seeing? Thanks in advance. ------------------------------ Mark Collins ------------------------------ #Informix

Hi Mark, I have tested doing ontape backup using local encryption key. When creating a local encryption key, I referenced the article below. https://www.ibm.com/support/knowledgecenter/SSGU8G_14.1.0/com.ibm.bar.doc/ids_bar_local_encryption_key.htm The results show that the encrypted backup file is compressed well. -- 1. Perform ontape backup without integrated backup encryption setting [informix@db2 backups]$ onstat -c | egrep '^BACKUP_FILTER|^RESTORE_FILTER|^BAR_ENC' BACKUP_FILTER RESTORE_FILTERBAR_ENCRYPTION [informix@db2 backups]$ ontape -s -L 0 10 percent done. 20 percent done. 30 percent done. 40 percent done. 100 percent done. File created: /work1/informix/ids1410fc3/backups/db2_0_L0 Please label this tape as number 1 in the arc tape sequence. This tape contains the following logical logs: 247 Program over. [informix@db2 backups]$ ls -la total 266788 drwxrwxr-x. 3 informix informix 33 Mar 31 09:54 . drwxr-xr-x. 28 informix informix 4096 Mar 21 22:30 .. -rw-rw----. 1 informix informix 273186816 Mar 31 09:54 db2_0_L0 drwxrwxr-x. 2 informix informix 182 Mar 31 09:51 tmp -- 2. Perform ontape backup with integrated backup encryption setting [informix@db2 backups]$ openssl rand -base64 24 > /work1/informix/ids1410fc3/etc/l_key192 [informix@db2 backups]$ vi $INFORMIXDIR/etc/$ONCONFIG [informix@db2 backups]$ onstat -c | egrep '^BACKUP_FILTER|^RESTORE_FILTER|^BAR_ENC' BACKUP_FILTER RESTORE_FILTER BAR_ENCRYPTION keyfile=/work1/informix/ids1410fc3/etc/l_key192,cipher=aes192 [informix@db2 backups]$ ontape -s -L 0The backup volume will be encrypted. 10 percent done. 20 percent done. 30 percent done. 40 percent done. 100 percent done. File created: /work1/informix/ids1410fc3/backups/db2_0_L0 Please label this tape as number 1 in the arc tape sequence. This tape contains the following logical logs: 247 Program over. [informix@db2 backups]$ ls -la total 533572 drwxrwxr-x. 3 informix informix 65 Mar 31 09:55 . drwxr-xr-x. 28 informix informix 4096 Mar 21 22:30 .. -rw-rw----. 1 informix informix 273186816 Mar 31 09:54 db2_0_20200331_095432_L0 -rw-rw----. 1 informix informix 273186816 Mar 31 09:55 db2_0_L0 drwxrwxr-x. 2 informix informix 182 Mar 31 09:51 tmp -- 3. Check backup file contents [informix@db2 backups]$ strings db2_0_20200331_095432_L0 | head -30 Archive Backup TapeIBM Informix Dynamic Server Version 14.10.FC3DETue Mar 31 09:54:32 2020informix/dev/pts/0 /work1/informix/ids1410fc3/backups/ rootdbs plog llog datadbs1 datadbs2 datadbs3 data8dbs1 data8dbs2 data8dbs3 sbspace1 IBM Informix Dynamic Server Copyright 2001, 2019 IBM Corporation 14.10.xC2 F"01-2020 ROOTNAME rootdbsROOTPATH /work1/informix/ids1410fc3/storage/rootdbs ROOTOFFSET 0 ROOTSIZE 157696MIRROR 0 MIRRORPATH /work1/informix/ids1410fc3/tmp/demo_on.root_mirror MIRROROFFSET 0 DBSERVERNAME ol_informix1410 SERVERNUM 0 MSGPATH /work1/informix/ids1410fc3/ol_informix1410.logTAPEDEV /work1/informix/ids1410fc3/backups TAPESIZE 0 TAPEBLK 32 LTAPEDEV /dev/null LTAPESIZE 0 LTAPEBLK 32 PHYSFILE 65430 PHYSBUFF 512 LOGFILES 20 LOGSIZE 6144 LOGBUFF 256 DYNAMIC_LOGS 2 LTXHWM 70 LTXEHWM 80 [informix@db2 backups]$ strings db2_0_L0 | head -30Archive Backup TapeIBM Informix Dynamic Server Version 14.10.FC3DETue Mar 31 09:55:07 2020informix/dev/pts/0 /work1/informix/ids1410fc3/backups/ aes192 DQZQ QX9p a5b( H%*g wecI )rO0" z\\$b #_t; azXX aW`g $9s~f {Rus T<.u (;*& [!wS {'dT g9c% k#|~ G)9)1 >X2B CnSq ]SJ~ kmt9 ol!n >JMsy b!9~` [C4R -- 4. Perform ontape backup with BACKUP_FILTER and integrated backup encryption settings [informix@db2 backups]$ vi $INFORMIXDIR/etc/$ONCONFIG [informix@db2 backups]$ onstat -c | egrep '^BACKUP_FILTER|^RESTORE_FILTER|^BAR_ENC' BACKUP_FILTER /bin/gzip RESTORE_FILTER /bin/gunzip BAR_ENCRYPTION keyfile=/work1/informix/ids1410fc3/etc/l_key192,cipher=aes192 [informix@db2 backups]$ ontape -s -L 0The backup volume will be encrypted. Using the backup and restore filter /bin/gzip. 10 percent done. 20 percent done. 30 percent done. 40 percent done. 100 percent done. File created: /work1/informix/ids1410fc3/backups/db2_0_L0 Please label this tape as number 1 in the arc tape sequence. This tape contains the following logical logs: 247 Program over. [informix@db2 backups]$ ls -la total 548100 drwxrwxr-x. 3 informix informix 97 Mar 31 09:56 . drwxr-xr-x. 28 informix informix 4096 Mar 21 22:30 .. -rw-rw----. 1 informix informix 273186816 Mar 31 09:54 db2_0_20200331_095432_L0 -rw-rw----. 1 informix informix 273186816 Mar 31 09:55 db2_0_20200331_095507_L0 -rw-rw----. 1 informix informix 14876672 Mar 31 09:56 db2_0_L0 drwxrwxr-x. 2 informix informix 182 Mar 31 09:51 tmp ------------------------------ SangGyu Jeong Software Engineer Infrasoft Seoul Korea, Republic of ------------------------------

SangGyu, Thank you for testing this and posting your results. That is a big relief. We're currently in the process of setting up a couple of sandbox environments where I'll be playing with this, along with other functions, as we prepare for our upgrade. ------------------------------ Mark Collins ------------------------------

The upgrade from version 11.50 to version 14.10 may have quite a lot to consider. I hope you have a smooth upgrade! ------------------------------ SangGyu Jeong Software Engineer Infrasoft Seoul Korea, Republic of ------------------------------

Hello Mark, Adding to SangGyu's reply, I would highly recommend you use "gzip -1" as your backup filter, as the difference in compression between compression 1 (which is the minimum and faster compression) and gzip's default compression of 6 (out of 9 which is the maximum but slow compression) is negligible, but the difference in performance is substantial. YMMV. Cheers! ------------------------------ Ramon Rey Any opinions I express here are my own and not the views of my employer. ------------------------------

Ramon, Thanks for that suggestion. I'll add that to the list of things to test. ------------------------------ Mark Collins ------------------------------

Also consider "pigz" which is multi-threaded and therefore much faster. It uses "zlib" so is inter-operable with "gzip". You can compile it from the source code here: https://zlib.net/pigz ------------------------------ Doug Lawry ------------------------------