IDS Performance Problem
Dear All,
I am trying to fine tune my IDS. My system configuration is mentioned below. I am worried about XXX, I have 4 GB physical memory on IBM-P5 machine. I have set BUFFERS to 8000, # of LRUs to 8, # of CLEANERS to 1, lru_min_dirty to 50% & lru_max_dirty to 60%. But still I can see bufwaits=1506913. Can anyone guide me.
Further, please give me some comments, how can I improve my IDS performance.
############################################
# System Summary
############################################
4 - CPUs currently
4 - CPUs configured
1900 - MHz CPU clock rate
PowerPC_POWER5 - Processor
64 bit - Hardware
32 bit - Kernel
Memory - 4 GB
Dynamic - Logical Partition
5.3.0.40 ML04 - AIX Kernel Version
############################################
# result of onstat -p
############################################
IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line -- Up 1 days 05:13:12 -- 83728 Kbytes
Profile
dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
7670584 9375803 159269260 95.22 1231936 2288175 11155217 88.96
isamtot open start read write rewrite delete commit rollbk
114686642 4314954 5106448 67991599 604319 6379231 534288 75865 0
gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
0 0 0 0 0 0 0
ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
0 0 0 1251.98 429.56 353 706
bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
1506913 0 32132874 0 0 211 128130 192953
ixda-RA idx-RA da-RA RA-pgsused lchwaits
609522 121903 4765665 5495588 676
############################################
# Shared Memory Parameters in ONCONFIG file
############################################
LOCKS 8000 # Maximum number of locks
NUMAIOVPS # Number of IO vps
PHYSBUFF 32 # Physical log buffer size (Kbytes)
LOGBUFF 32 # Logical log buffer size (Kbytes)#CLEANERS 1 # Number of buffer cleaner processes
CLEANERS 1 # Number of buffer cleaner processes
SHMBASE 0x40000000L # Shared memory base address
SHMVIRTSIZE 8192 # initial virtual shared memory segment size
SHMADD 8192 # Size of new shared memory segments (Kbytes)
EXTSHMADD 8192 # Size of new extension shared memory segments (Kbytes)
SHMTOTAL 0 # Total shared memory (Kbytes). 0=>unlimited
CKPTINTVL 300 # Check point interval (in sec)
TXTIMEOUT 300 # Transaction timeout (in sec)
STACKSIZE 64
# Read Ahead Variables
RA_PAGES # Number of pages to attempt to read ahead
RA_THRESHOLD # Number of pages left before next group
# The following parameters are related to the buffer pool
#BUFFERPOOL default,buffers=1000,lrus=8,lru_min_dirty=50,lru_max_dirty=60
BUFFERPOOL size=4k,buffers=8000,lrus=8,lru_min_dirty=50,lru_max_dirty=60
Regards,
Navaid Arif
On Feb 7, 6:25 pm, "Navaid Arif" <navaid.a...@access.net.pk> wrote:
> Dear All,
> I am trying to fine tune my IDS. My system configuration is mentioned below. I am worried about XXX, I have 4 GB physical memory on IBM-P5 machine. I have set BUFFERS to 8000, # of LRUs to 8, # of CLEANERS to 1, lru_min_dirty to 50% & lru_max_dirty to 60%. But still I can see bufwaits=1506913. Can anyone guide me.
>
> Further, please give me some comments, how can I improve my IDS performance.
>
> ############################################
> # System Summary
> ############################################
>
> 4 - CPUs currently
> 4 - CPUs configured
> 1900 - MHz CPU clock rate
> PowerPC_POWER5 - Processor
> 64 bit - Hardware
> 32 bit - Kernel
> Memory - 4 GB
> Dynamic - Logical Partition
> 5.3.0.40 ML04 - AIX Kernel Version
>
> ############################################
> # result of onstat -p
> ############################################
>
> IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line -- Up 1 days 05:13:12 -- 83728 Kbytes>
> Profile
> dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
> 7670584 9375803 159269260 95.22 1231936 2288175 11155217 88.96
>
> isamtot open start read write rewrite delete commit rollbk
> 114686642 4314954 5106448 67991599 604319 6379231 534288 75865 0
>
> gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
> 0 0 0 0 0 0 0
>
> ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
> 0 0 0 1251.98 429.56 353 706
>
> bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
> 1506913 0 32132874 0 0 211 128130 192953
>
> ixda-RA idx-RA da-RA RA-pgsused lchwaits
> 609522 121903 4765665 5495588 676
>
> ############################################
> # Shared Memory Parameters in ONCONFIG file
> ############################################
>
> LOCKS 8000 # Maximum number of locks
> NUMAIOVPS # Number of IO vps
> PHYSBUFF 32 # Physical log buffer size (Kbytes)
> LOGBUFF 32 # Logical log buffer size (Kbytes)> #CLEANERS 1 # Number of buffer cleaner processes
> CLEANERS 1 # Number of buffer cleaner processes
> SHMBASE 0x40000000L # Shared memory base address
> SHMVIRTSIZE 8192 # initial virtual shared memory segment size
> SHMADD 8192 # Size of new shared memory segments (Kbytes)
> EXTSHMADD 8192 # Size of new extension shared memory segments (Kbytes)
> SHMTOTAL 0 # Total shared memory (Kbytes). 0=>unlimited
> CKPTINTVL 300 # Check point interval (in sec)
> TXTIMEOUT 300 # Transaction timeout (in sec)
> STACKSIZE 64>
> # Read Ahead Variables
> RA_PAGES # Number of pages to attempt to read ahead
> RA_THRESHOLD # Number of pages left before next group>
> # The following parameters are related to the buffer pool
> #BUFFERPOOL default,buffers=1000,lrus=8,lru_min_dirty=50,lru_max_dirty=60
> BUFFERPOOL size=4k,buffers=8000,lrus=8,lru_min_dirty=50,lru_max_dirty=60
>
> Regards,
>
> Navaid Arif
Look at many previous posts in this newsgroup with this topic. There
are very good advices from Art Kagel, Neil Truby, Obnoxio The Clown
and others.
If you don't want to do that, you can try the following.
Disclaimer: I am yet to use IDS 10 in production, so my comments are
based on experience with versions 7 and 9.
With that much RAM, you should substantially increase values of some
parameters:
SHMVIRTSIZE to at least several hundreds of MB,
SHMADD to 20-40 % of SHMVIRTSIZE,LOCKS to several hundreds of thousands,
LRUS to say, 65 or 127CLEANERS to at least several (say, 15), up to the number of LRUS
You may need to decrease LRU_MIN_DIRTY and LRU_MAX_DIRTY depending on
the duration of your checkpoint.
You have powerfull processors. Could you use 64-bit AIX and Informix
and add more RAM? Anyway, you may need to set NUMCPUVPS to a value
equal to multiple of the number of CPUs that you will use for IDS. Try
and see what suits you.
Good luck,
Darko Krstic
darko wrote:
> On Feb 7, 6:25 pm, "Navaid Arif" <navaid.a...@access.net.pk> wrote:
>> Dear All,
>> I am trying to fine tune my IDS. My system configuration is mentioned below. I am worried about XXX, I have 4 GB physical memory on IBM-P5 machine. I have set BUFFERS to 8000, # of LRUs to 8, # of CLEANERS to 1, lru_min_dirty to 50% & lru_max_dirty to 60%. But still I can see bufwaits=1506913. Can anyone guide me.
>>
>> Further, please give me some comments, how can I improve my IDS performance.
>>
>> ############################################
>> # System Summary
>> ############################################
>>
>> 4 - CPUs currently
>> 4 - CPUs configured
>> 1900 - MHz CPU clock rate
>> PowerPC_POWER5 - Processor
>> 64 bit - Hardware
>> 32 bit - Kernel
>> Memory - 4 GB
>> Dynamic - Logical Partition
>> 5.3.0.40 ML04 - AIX Kernel Version
>>
>> ############################################
>> # result of onstat -p
>> ############################################
>>
>> IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line -- Up 1 days 05:13:12 -- 83728 Kbytes>>
>> Profile
>> dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
>> 7670584 9375803 159269260 95.22 1231936 2288175 11155217 88.96
>>
>> isamtot open start read write rewrite delete commit rollbk
>> 114686642 4314954 5106448 67991599 604319 6379231 534288 75865 0
>>
>> gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
>> 0 0 0 0 0 0 0
>>
>> ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
>> 0 0 0 1251.98 429.56 353 706
>>
>> bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
>> 1506913 0 32132874 0 0 211 128130 192953
>>
>> ixda-RA idx-RA da-RA RA-pgsused lchwaits
>> 609522 121903 4765665 5495588 676
>>
>> ############################################
>> # Shared Memory Parameters in ONCONFIG file
>> ############################################
>>
>> LOCKS 8000 # Maximum number of locks
>> NUMAIOVPS # Number of IO vps
>> PHYSBUFF 32 # Physical log buffer size (Kbytes)
>> LOGBUFF 32 # Logical log buffer size (Kbytes)>> #CLEANERS 1 # Number of buffer cleaner processes
>> CLEANERS 1 # Number of buffer cleaner processes
>> SHMBASE 0x40000000L # Shared memory base address
>> SHMVIRTSIZE 8192 # initial virtual shared memory segment size
>> SHMADD 8192 # Size of new shared memory segments (Kbytes)
>> EXTSHMADD 8192 # Size of new extension shared memory segments (Kbytes)
>> SHMTOTAL 0 # Total shared memory (Kbytes). 0=>unlimited
>> CKPTINTVL 300 # Check point interval (in sec)
>> TXTIMEOUT 300 # Transaction timeout (in sec)
>> STACKSIZE 64>>
>> # Read Ahead Variables
>> RA_PAGES # Number of pages to attempt to read ahead
>> RA_THRESHOLD # Number of pages left before next group>>
>> # The following parameters are related to the buffer pool
>> #BUFFERPOOL default,buffers=1000,lrus=8,lru_min_dirty=50,lru_max_dirty=60
>> BUFFERPOOL size=4k,buffers=8000,lrus=8,lru_min_dirty=50,lru_max_dirty=60
>>
>> Regards,
>>
>> Navaid Arif
>
> Look at many previous posts in this newsgroup with this topic. There
> are very good advices from Art Kagel, Neil Truby, Obnoxio The Clown
> and others.
>
> If you don't want to do that, you can try the following.
>
> Disclaimer: I am yet to use IDS 10 in production, so my comments are
> based on experience with versions 7 and 9.
>
> With that much RAM, you should substantially increase values of some
> parameters:
> SHMVIRTSIZE to at least several hundreds of MB,
> SHMADD to 20-40 % of SHMVIRTSIZE,> LOCKS to several hundreds of thousands,
> LRUS to say, 65 or 127> CLEANERS to at least several (say, 15), up to the number of LRUS
>
> You may need to decrease LRU_MIN_DIRTY and LRU_MAX_DIRTY depending on
> the duration of your checkpoint.
>
> You have powerfull processors. Could you use 64-bit AIX and Informix
> and add more RAM? Anyway, you may need to set NUMCPUVPS to a value
> equal to multiple of the number of CPUs that you will use for IDS. Try
> and see what suits you.
>
> Good luck,
> Darko Krstic
>
And increase number of buffers. I have a little development server and
use 40000 buffers (Linux, xeon, suse 9, 1.5 GB ram).
As mentioned above, read specific posts in this newsgroup.
HTH
Michael
On Feb 7, 2:09 pm, "darko" <darko.krs...@gmail.com> wrote:
> On Feb 7, 6:25 pm, "Navaid Arif" <navaid.a...@access.net.pk> wrote:
>
>
>
> > Dear All,
> > I am trying to fine tune my IDS. My system configuration is mentioned below. I am worried about XXX, I have 4 GB physical memory on IBM-P5 machine. I have set BUFFERS to 8000, # of LRUs to 8, # of CLEANERS to 1, lru_min_dirty to 50% & lru_max_dirty to 60%. But still I can see bufwaits=1506913. Can anyone guide me.
>
> > Further, please give me some comments, how can I improve my IDS performance.
>
> > ############################################
> > # System Summary
> > ############################################
>
> > 4 - CPUs currently
> > 4 - CPUs configured
> > 1900 - MHz CPU clock rate
> > PowerPC_POWER5 - Processor
> > 64 bit - Hardware
> > 32 bit - Kernel
> > Memory - 4 GB
> > Dynamic - Logical Partition
> > 5.3.0.40 ML04 - AIX Kernel Version
>
> > ############################################
> > # result of onstat -p
> > ############################################
>
> > IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line -- Up 1 days 05:13:12 -- 83728 Kbytes>
> > Profile
> > dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
> > 7670584 9375803 159269260 95.22 1231936 2288175 11155217 88.96
>
> > isamtot open start read write rewrite delete commit rollbk
> > 114686642 4314954 5106448 67991599 604319 6379231 534288 75865 0
>
> > gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
> > 0 0 0 0 0 0 0
>
> > ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
> > 0 0 0 1251.98 429.56 353 706
>
> > bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
> > 1506913 0 32132874 0 0 211 128130 192953
>
> > ixda-RA idx-RA da-RA RA-pgsused lchwaits
> > 609522 121903 4765665 5495588 676
>
> > ############################################
> > # Shared Memory Parameters in ONCONFIG file
> > ############################################
>
> > LOCKS 8000 # Maximum number of locks
> > NUMAIOVPS # Number of IO vps
> > PHYSBUFF 32 # Physical log buffer size (Kbytes)
> > LOGBUFF 32 # Logical log buffer size (Kbytes)> > #CLEANERS 1 # Number of buffer cleaner processes
> > CLEANERS 1 # Number of buffer cleaner processes
> > SHMBASE 0x40000000L # Shared memory base address
> > SHMVIRTSIZE 8192 # initial virtual shared memory segment size
> > SHMADD 8192 # Size of new shared memory segments (Kbytes)
> > EXTSHMADD 8192 # Size of new extension shared memory segments (Kbytes)
> > SHMTOTAL 0 # Total shared memory (Kbytes). 0=>unlimited
> > CKPTINTVL 300 # Check point interval (in sec)
> > TXTIMEOUT 300 # Transaction timeout (in sec)
> > STACKSIZE 64>
> > # Read Ahead Variables
> > RA_PAGES # Number of pages to attempt to read ahead
> > RA_THRESHOLD # Number of pages left before next group>
> > # The following parameters are related to the buffer pool
> > #BUFFERPOOL default,buffers=1000,lrus=8,lru_min_dirty=50,lru_max_dirty=60
> > BUFFERPOOL size=4k,buffers=8000,lrus=8,lru_min_dirty=50,lru_max_dirty=60
>
> > Regards,
>
> > Navaid Arif
>
> Look at many previous posts in this newsgroup with this topic. There
> are very good advices from Art Kagel, Neil Truby, Obnoxio The Clown
> and others.
>
> If you don't want to do that, you can try the following.
>
> Disclaimer: I am yet to use IDS 10 in production, so my comments are
> based on experience with versions 7 and 9.
>
> With that much RAM, you should substantially increase values of some
> parameters:
> SHMVIRTSIZE to at least several hundreds of MB,
> SHMADD to 20-40 % of SHMVIRTSIZE,> LOCKS to several hundreds of thousands,
> LRUS to say, 65 or 127> CLEANERS to at least several (say, 15), up to the number of LRUS
>
> You may need to decrease LRU_MIN_DIRTY and LRU_MAX_DIRTY depending on
> the duration of your checkpoint.
>
> You have powerfull processors. Could you use 64-bit AIX and Informix
> and add more RAM? Anyway, you may need to set NUMCPUVPS to a value
> equal to multiple of the number of CPUs that you will use for IDS. Try
> and see what suits you.
>
> Good luck,
> Darko Krstic
For 10 I also like setting this parameter high:
DS_NONPDQ_QUERY_MEM 65536 # Non PDQ query memory (Kbytes)
It can only be 25% of DS_TOTAL_MEMORY. Yes, you need more LRU queues and more Buffers.
Check the application you seem to have a good number of sequential
scans make sure that you don't have scans happening when using an
index would be better.
You should add the kind of system OLTP or reporting and a full copy of
the ONCONFIG file.
On Feb 7, 8:17 pm, Michael Krzepkowski <mkrzepkow...@hotmail.com>
wrote:
> darko wrote:
> > On Feb 7, 6:25 pm, "Navaid Arif" <navaid.a...@access.net.pk> wrote:
> >> Dear All,
> >> I am trying to fine tune my IDS. My system configuration is mentioned below. I am worried about XXX, I have 4 GB physical memory on IBM-P5 machine. I have set BUFFERS to 8000, # of LRUs to 8, # of CLEANERS to 1, lru_min_dirty to 50% & lru_max_dirty to 60%. But still I can see bufwaits=1506913. Can anyone guide me.
>
> >> Further, please give me some comments, how can I improve my IDS performance.
>
> >> ############################################
> >> # System Summary
> >> ############################################
>
> >> 4 - CPUs currently
> >> 4 - CPUs configured
> >> 1900 - MHz CPU clock rate
> >> PowerPC_POWER5 - Processor
> >> 64 bit - Hardware
> >> 32 bit - Kernel
> >> Memory - 4 GB
> >> Dynamic - Logical Partition
> >> 5.3.0.40 ML04 - AIX Kernel Version
>
> >> ############################################
> >> # result of onstat -p
> >> ############################################
>
> >> IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line -- Up 1 days 05:13:12 -- 83728 Kbytes>
> >> Profile
> >> dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
> >> 7670584 9375803 159269260 95.22 1231936 2288175 11155217 88.96
>
> >> isamtot open start read write rewrite delete commit rollbk
> >> 114686642 4314954 5106448 67991599 604319 6379231 534288 75865 0
>
> >> gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
> >> 0 0 0 0 0 0 0
>
> >> ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
> >> 0 0 0 1251.98 429.56 353 706
>
> >> bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
> >> 1506913 0 32132874 0 0 211 128130 192953
>
> >> ixda-RA idx-RA da-RA RA-pgsused lchwaits
> >> 609522 121903 4765665 5495588 676
>
> >> ############################################
> >> # Shared Memory Parameters in ONCONFIG file
> >> ############################################
>
> >> LOCKS 8000 # Maximum number of locks
> >> NUMAIOVPS # Number of IO vps
> >> PHYSBUFF 32 # Physical log buffer size (Kbytes)
> >> LOGBUFF 32 # Logical log buffer size (Kbytes)> >> #CLEANERS 1 # Number of buffer cleaner processes
> >> CLEANERS 1 # Number of buffer cleaner processes
> >> SHMBASE 0x40000000L # Shared memory base address
> >> SHMVIRTSIZE 8192 # initial virtual shared memory segment size
> >> SHMADD 8192 # Size of new shared memory segments (Kbytes)
> >> EXTSHMADD 8192 # Size of new extension shared memory segments (Kbytes)
> >> SHMTOTAL 0 # Total shared memory (Kbytes). 0=>unlimited
> >> CKPTINTVL 300 # Check point interval (in sec)
> >> TXTIMEOUT 300 # Transaction timeout (in sec)
> >> STACKSIZE 64>
> >> # Read Ahead Variables
> >> RA_PAGES # Number of pages to attempt to read ahead
> >> RA_THRESHOLD # Number of pages left before next group>
> >> # The following parameters are related to the buffer pool
> >> #BUFFERPOOL default,buffers=1000,lrus=8,lru_min_dirty=50,lru_max_dirty=60
> >> BUFFERPOOL size=4k,buffers=8000,lrus=8,lru_min_dirty=50,lru_max_dirty=60
>
> >> Regards,
>
> >> Navaid Arif
>
> > Look at many previous posts in this newsgroup with this topic. There
> > are very good advices from Art Kagel, Neil Truby, Obnoxio The Clown
> > and others.
>
> > If you don't want to do that, you can try the following.
>
> > Disclaimer: I am yet to use IDS 10 in production, so my comments are
> > based on experience with versions 7 and 9.
>
> > With that much RAM, you should substantially increase values of some
> > parameters:
> > SHMVIRTSIZE to at least several hundreds of MB,
> > SHMADD to 20-40 % of SHMVIRTSIZE,> > LOCKS to several hundreds of thousands,
> > LRUS to say, 65 or 127> > CLEANERS to at least several (say, 15), up to the number of LRUS
>
> > You may need to decrease LRU_MIN_DIRTY and LRU_MAX_DIRTY depending on
> > the duration of your checkpoint.
>
> > You have powerfull processors. Could you use 64-bit AIX and Informix
> > and add more RAM? Anyway, you may need to set NUMCPUVPS to a value
> > equal to multiple of the number of CPUs that you will use for IDS. Try
> > and see what suits you.
>
> > Good luck,
> > Darko Krstic
>
> And increase number of buffers. I have a little development server and
> use 40000 buffers (Linux, xeon, suse 9, 1.5 GB ram).
>
> As mentioned above, read specific posts in this newsgroup.
>
> HTH
>
> Michael
By all means. I remember that I had a feeling that I had missed
something in my previous post :-) Buffers!
Darko Krstic