CDR_NIFCOMPRESS setting for ER
A user planning to scale Enterprise Replication to ~120M rows/hour (~80 Mbps of traffic) asked how much bandwidth CDR_NIFCOMPRESS would save and whether a formula exists for picking a level. Madison Pruet (IBM) replied that most of the gain comes from level 1: ER sends rows in external format, full of spaces and zeros, so it compresses far better than typical binary data — closer to Art Kagel's 75–90% text figure than 30–50%. Higher levels (up to 9) compress a bit more but cost extra CPU and memory and add replication latency. Eric Rowell suggested benchmarking by unloading sample data and timing gzip at levels 1–9 to find the best trade-off. No single formula was offered.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Hi All,
We have an enterprise replication environment that currently is lightly used,
but there is a plan to change this dramatically. We want to support
replicating up to 120 million rows of data per hour (where each row is 300
bytes). This will be done 24/7, so there would be no lull in activity. I
figured this to be about 80 Mbits of data stored every second. If this data is
replicated without compression, then it seems that we would need at least 80
Mbps of network bandwidth between the primary and the backup. We need to
figure out what effect setting the CDR_NIFCOMPRESS onconfig parameter to a
value greater than 0 will have. Is there a formula somewhere that can be used
to determine this, or do we just have to rely on trial and error? Does anybody
know what the maximum expected compression factor (approx.) if we set
CDR_NIFCOMPRESS to 9?
Any information on this would be appreciated.
Thanks,
Jeff
The biggest percentage of reduction can be had by just using 1.
From: "JEFFREY GENEGA" <jeffrey.genega@spirent.com>
To: ids@iiug.org,
Date: 09/11/2012 10:00 AM
Subject: CDR_NIFCOMPRESS setting for ER [28282]
Sent by: ids-bounces@iiug.org
Hi All,
We have an enterprise replication environment that currently is lightly=
used,
but there is a plan to change this dramatically. We want to support
replicating up to 120 million rows of data per hour (where each row is =
300
bytes). This will be done 24/7, so there would be no lull in activity. =
I
figured this to be about 80 Mbits of data stored every second. If this =
data
is
replicated without compression, then it seems that we would need at lea=
st
80
Mbps of network bandwidth between the primary and the backup. We need t=
o
figure out what effect setting the CDR_NIFCOMPRESS onconfig parameter t=
o a
value greater than 0 will have. Is there a formula somewhere that can b=
e
used
to determine this, or do we just have to rely on trial and error? Does
anybody
know what the maximum expected compression factor (approx.) if we set
CDR_NIFCOMPRESS to 9?
Any information on this would be appreciated.
Thanks,
Jeff
***********************************************************************=
********
Forum Note: Use "Reply" to post a response in the discussion forum.
=
IB that the compression is LZ type compression which compresses text at
between 75 and 90% and binary data between 30 & 50%. I tend to use that as
a rule of thumb. Maybe Madison can give us a better handle.
Art
Art S. Kagel
Advanced DataTools (www.advancedatatools.com)
Blog: http://informix-myview.blogspot.com/
Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on my employer, Advanced DataTools, the IIUG, nor any
other organization with which I am associated either explicitly,
implicitly, or by inference. Neither do those opinions reflect those of
other individuals affiliated with any entity with which I am affiliated nor
those of the entities themselves.
On Tue, Sep 11, 2012 at 10:58 AM, JEFFREY GENEGA <jeffrey.genega@spirent.com
> wrote:
> Hi All,
>
> We have an enterprise replication environment that currently is lightly
> used,
> but there is a plan to change this dramatically. We want to support
> replicating up to 120 million rows of data per hour (where each row is 300
> bytes). This will be done 24/7, so there would be no lull in activity. I
> figured this to be about 80 Mbits of data stored every second. If this
> data is
> replicated without compression, then it seems that we would need at least
> 80
> Mbps of network bandwidth between the primary and the backup. We need to
> figure out what effect setting the CDR_NIFCOMPRESS onconfig parameter to a
> value greater than 0 will have. Is there a formula somewhere that can be
> used
> to determine this, or do we just have to rely on trial and error? Does
> anybody
> know what the maximum expected compression factor (approx.) if we set
> CDR_NIFCOMPRESS to 9?>
> Any information on this would be appreciated.
>
> Thanks,
> Jeff
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--14dae93412633b7ca904c96f5978
While the typical binary data doesn't compress that well, database rows=
(i.e. what ER sends) compresses at a much higher ratio.
The data that ER sends across the wire is in external row image format =
-
which is quite a bit larger than the internal format. That means that =
it
is much easier to find patterns in the data stream due to all of the ze=
ros
and spaces. So a lower level of compress can give a much higher
compression reduction ratio than would otherwise be expected.
I'd expect the compression level 1 to be closer to the 75-90% reduction=
than the 34-50%.
M.P.
From: "Art Kagel" <art.kagel@gmail.com>
To: ids@iiug.org,
Date: 09/11/2012 11:14 AM
Subject: Re: CDR_NIFCOMPRESS setting for ER [28284]
Sent by: ids-bounces@iiug.org
IB that the compression is LZ type compression which compresses text at=
between 75 and 90% and binary data between 30 & 50%. I tend to use that=
as
a rule of thumb. Maybe Madison can give us a better handle.
Art
Art S. Kagel
Advanced DataTools (www.advancedatatools.com)
Blog: http://informix-myview.blogspot.com/
Disclaimer: Please keep in mind that my own opinions are my own opinion=
s
and do not reflect on my employer, Advanced DataTools, the IIUG, nor an=
y
other organization with which I am associated either explicitly,
implicitly, or by inference. Neither do those opinions reflect those of=
other individuals affiliated with any entity with which I am affiliated=
nor
those of the entities themselves.
On Tue, Sep 11, 2012 at 10:58 AM, JEFFREY GENEGA
<jeffrey.genega@spirent.com
> wrote:
> Hi All,
>
> We have an enterprise replication environment that currently is light=
ly
> used,
> but there is a plan to change this dramatically. We want to support
> replicating up to 120 million rows of data per hour (where each row i=
s
300
> bytes). This will be done 24/7, so there would be no lull in activity=
. I
> figured this to be about 80 Mbits of data stored every second. If thi=
s
> data is
> replicated without compression, then it seems that we would need at l=
east
> 80
> Mbps of network bandwidth between the primary and the backup. We need=
to
> figure out what effect setting the CDR_NIFCOMPRESS onconfig parameter=
to
a
> value greater than 0 will have. Is there a formula somewhere that can=
be
> used
> to determine this, or do we just have to rely on trial and error? Doe=
s
> anybody
> know what the maximum expected compression factor (approx.) if we set=
> CDR_NIFCOMPRESS to 9?>
> Any information on this would be appreciated.
>
> Thanks,
> Jeff
>
>
>
>
***********************************************************************=
********
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--14dae93412633b7ca904c96f5978
***********************************************************************=
********
Forum Note: Use "Reply" to post a response in the discussion forum.
=
Thanks for the info Madison. However, I am a little confused about why a
compression value of 1 is better than 9. The 11.70 Administration manual has
the following:
--- begin ---
CDR_NIFCOMPRESS Configuration Parameter
Specifies the level of compression the database server uses before sending
data from the source database server to the target database server.
onconfig.std value
0
range of values
-1 specifies no compression
0 specifies to compress only if the target server expects compression
1 - 9 specifies increasing levels of compression
The compression values determine how much memory can be used to store
information while compressing, as follows:
0 = no additional memory
1 = 128k + 1k = 129k
2 = 128k + 2k = 130k
...
6 = 128k + 32k = 160k
...
8 = 128k + 128k = 256k
9 = 128k + 256k = 384k
Higher levels of CDR_NIFCOMPRESS cause greater compression.
--- end ---
From the documentation, I would think that a value of 9 would cause greater
compression than a value of 1. Is there more to it than this?
Thanks,
Jeff
The higher the number, the more CPU cycles it takes to compress.
A value of 9 will compress better than 1, but at the cost of more CPU
resources.
Also - it takes longer to compress at 9 than 1 - and that will increase=
the
latency of replication. While the increased time is negligible for a
single transaction, for the totality of transactions over the course of=
a
day - it could be significant.
From: "JEFFREY GENEGA" <jeffrey.genega@spirent.com>
To: ids@iiug.org,
Date: 09/11/2012 12:04 PM
Subject: Re: CDR_NIFCOMPRESS setting for ER [28286]
Sent by: ids-bounces@iiug.org
Thanks for the info Madison. However, I am a little confused about why =
a
compression value of 1 is better than 9. The 11.70 Administration manua=
l
has
the following:
--- begin ---
CDR_NIFCOMPRESS Configuration Parameter
Specifies the level of compression the database server uses before send=
ing
data from the source database server to the target database server.
onconfig.std value
0
range of values
-1 specifies no compression
0 specifies to compress only if the target server expects compression
1 - 9 specifies increasing levels of compression
The compression values determine how much memory can be used to store
information while compressing, as follows:
0 =3D no additional memory
1 =3D 128k + 1k =3D 129k
2 =3D 128k + 2k =3D 130k
....
6 =3D 128k + 32k =3D 160k
....
8 =3D 128k + 128k =3D 256k
9 =3D 128k + 256k =3D 384k
Higher levels of CDR_NIFCOMPRESS cause greater compression.
--- end ---
>From the documentation, I would think that a value of 9 would cause
greater
compression than a value of 1. Is there more to it than this?
Thanks,
Jeff
***********************************************************************=
********
Forum Note: Use "Reply" to post a response in the discussion forum.
=
It is similar to what happens with other compression programs. Try
this: unload the data you plan to replicate into a directory. At
least 100MB but I like a GB so there is a nice long runtime. Run gzip
on the file at each of its compression rates(1-9) to different files.
Using time and sar you can see the used resources for each run.
gzip file file_1.gz
gzip -9 file file_9.gz
Chat the results and you should find the best bang for the CPU is at
the lower compress rates.
I have done this test with gzip, compress, pkzip, and bzip2. I would
expect this to be the same for this compression method.
Sent from my iPhone
Eric B. Rowell
On Sep 11, 2012, at 1:02 PM, JEFFREY GENEGA <jeffrey.genega@spirent.com> wrote:
> Thanks for the info Madison. However, I am a little confused about why a
> compression value of 1 is better than 9. The 11.70 Administration manual has
> the following:
>
> --- begin ---
>
> CDR_NIFCOMPRESS Configuration Parameter>
> Specifies the level of compression the database server uses before sending
> data from the source database server to the target database server.
>
> onconfig.std value
>
> 0
> range of values
>
> -1 specifies no compression
>
> 0 specifies to compress only if the target server expects compression
>
> 1 - 9 specifies increasing levels of compression
>
> The compression values determine how much memory can be used to store
> information while compressing, as follows:
>
> 0 = no additional memory
> 1 = 128k + 1k = 129k
> 2 = 128k + 2k = 130k
> ....
> 6 = 128k + 32k = 160k
> ....
> 8 = 128k + 128k = 256k
> 9 = 128k + 256k = 384k
>
> Higher levels of CDR_NIFCOMPRESS cause greater compression.
>
> --- end ---
>
>> From the documentation, I would think that a value of 9 would cause greater
> compression than a value of 1. Is there more to it than this?
>
> Thanks,
> Jeff
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>