Compressing log files 'on the fly'
Posted in 1995
I have seen compress reduce log files 7:1, making a 400MB log file only 60MB in size. (We only use compress when our system is running out of disk space). Because compress requires enough space to hold both the original and compressed file during part of it's cycle, we are getting ready to set up our system to stop logging once an hour, compress the old file, change the log file name (add 1), restart logging, and append the compressed file to tape. This will reduce the impact of compress, allow us to archive once a day instead of twice, and maintain our 'no single point of failure' philosophy. However, compress is *extremly* cpu intensive, and I would rather not max out our 8 servers for 2 minutes once an hour if I can avoid it. Question -- I know that compress will accept pipe input (cat <file> | compress >file.Z). Could you utilize this feature to compress log files as they are being created?? Is there a way to create a 'pipe' file in Unix (HP9000/800) that could be redirected into compress (cat -f <file> | compress >file.Z) in such a way that the input file would remain small and the process could be finished in a controlled fashion?? I remember using pipe files on Honeywell DPS-6 (GCOS MOD400) that worked in such a way that as long as an input process had the pipe file opened for output, the output pipe process would continue to process data as it appeared. However, once all input pipe processes had been terminated, the output pipe process would act as if an <EOF> had been received. ----------------------------------------------------------------------------- >On Tue, 14 Nov 1995, Steve Weiland wrote: > >} Yes, I have done this successfully in both HPUX and SCO environments. >} In fact, I even automatically compress the archives after its done, and >} always keep 7 days worth on disk. I do the same with the logical logs. >} This permits pretty fast restore/recovery capabilities. It also has >} the benefit of not having to dedicate tape drives to informix.