onstat -g rea almot 100 queue there !!!

Dear All... I've a IDS 7.31 in a RHEL 3 server , with 4 dual core cpu there , I set cpuvps to 8 and NETTYPE ipcshm,1,20,CPU soctcp,8,30,NET , and onstat -g ses | wc -l showes 220 ... I have noticed that top showes cpus' idle are about 30%~35% , and onstat -g rea showes about 80~100 threads queue over there , sometimes maybe 30~50 , but more often 80~100 !! so I decide to onmode -p +8 cpu , the cpu idle down to 18%~23% , and onstat -g rea showes 30~50 threads over there , and the ap runs faster than while only 8 cpu vps in there ... I noticed the onstat -g rea , the ready queue for cpu , srvinfx took 2/3 , sqlexec took 1/3 , srvinfx manage the sessions the ap connect to IDS A and process like "select * from db@idsB:tablex" to this IDS engine , sqlexec manage the sessions which connect to this IDS and do something directly ~~ I have 16 cpu vps now and still have 30~50 queue for cpu , and cpu idle get 18%~23% , is that normal ?! Should I add 8 more cpu vps to 24 ?! or I have something to do ?! Do I miss something so that even the cpu is idle , I still get threads queue over there ~~ Thanks for any comment ..

You have idle time and items in the queue concurrently because the CPU VPs sometimes have to wait for resources. There is lots more information we need to try to help you: - How many LRUs do you have configured? - How much cache is configured? If you do not have enough LRUs configured there will be serious contention for aged buffers to be overwritten. If you do not have enough cache there will be contention because of the increased need to overwrite buffers with new data constantly. - What is the current BR (Bufwaits Ratio)? - What is the BTR (Buffer Turnover Rate)? - What is the RAU (Read Ahead Utilization)? These are metrics that show the levels of contention for IO resources like buffers, LRUs, etc. - How many IO operations per second are being thrown at your disk subsystems per chunk (onstat -g iof)? - Can the 'disk' and channel handle that level of IO? - Are you beating hell out of one structure while other disk spindles are quiescent much of the time? - Are you using RAW disk or at least DIRECT_IO? - Have you avoided RAID5 like the plague that it is? Will more CPU VPs help? Depends. As you saw, some of the wait is just that, and having more hands doing work will put more requests in the queue faster so that there is, ultimately, less waiting and more doing. However, you may reach a point where you are experiencing increased contention for resources including LRU queues, buffers, IO channel bandwidth, disk bandwidth, CPU cycles. Do you have the free cycles available to run three CPU VPs per core instead of two? It seems like it. My rule of thumb is you need about 450-500MHZ available per CPU VP (~700-750MHZ for Intel cores - sorry). From your idle time, I'm guessing you are running 2.2 GHZ cores, but that's just a guess. If that's about right, then if they are not Intel you should be able to support up to 4 CPU VPs per core - three if they are Intel. Of course, my guess could be wrong, you didn't post your processor speed. There are those who disagree with me that you can run more than one CPU VP per core and gain throughput, but as you have seen, it works. That said, if your other resources are actually being limiting factors, then more VPs will not solve anything until you resolve those resources as well. Art On Wed, Jul 9, 2008 at 9:10 PM, MARS CHEN <mars@jsun.com> wrote: > Dear All... > > I've a IDS 7.31 in a RHEL 3 server , with 4 dual core cpu there , > I set cpuvps to 8 and NETTYPE ipcshm,1,20,CPU soctcp,8,30,NET , > and onstat -g ses | wc -l showes 220 ... > > I have noticed that top showes cpus' idle are about 30%~35% , > and onstat -g rea showes about 80~100 threads queue over there , > sometimes maybe 30~50 , but more often 80~100 !! > > so I decide to onmode -p +8 cpu , the cpu idle down to 18%~23% , > and onstat -g rea showes 30~50 threads over there , > and the ap runs faster than while only 8 cpu vps in there ... > > I noticed the onstat -g rea , the ready queue for cpu , > srvinfx took 2/3 , sqlexec took 1/3 , srvinfx manage the sessions > the ap connect to IDS A and process like "select * from db@idsB:tablex" > to this IDS engine , sqlexec manage the sessions which connect to > this IDS and do something directly ~~ > > I have 16 cpu vps now and still have 30~50 queue for cpu , > and cpu idle get 18%~23% , is that normal ?! > Should I add 8 more cpu vps to 24 ?! or I have something to do ?! > Do I miss something so that even the cpu is idle , > I still get threads queue over there ~~ > > Thanks for any comment .. > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves.

Thanks Art Kagel ... The following is what I get in the server : >- How many LRUs do you have configured? LRUS 8 LRU_MAX_DIRTY 2 LRU_MIN_DIRTY 1 >- How much cache is configured? Sorry..I don't know what this means ... Buffers ?! Physical Log Buffer Size [ 128] K Logical Log Buffer Size [ 128] K Max # of Logical Logs [ 200] Max # of Locks [ 2000000] Max # of Buffers [ 100000] Resident Shared Memory size [ 317542] Kbytes >- What is the current BR (Bufwaits Ratio)? >- What is the BTR (Buffer Turnover Rate)? >- What is the RAU (Read Ahead Utilization)? post onstat -p : Profile dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached 36850 43972 3714504778 100.00 103209 346874 11912950 99.13 usercpu syscpu numckpts flushes 47787.69 5838.60 427 856 bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans 104026 4253 981169431 0 0 1301 563 6533000 ixda-RA idx-RA da-RA RA-pgsused lchwaits 1327 2753 25041 29045 16257541 And ... I didnot set these 2 fields about Read Ahead ~~ Num of Read Ahead Pages [ ] Read Ahead Threshold [ ] >- How many IO operations per second are being thrown at your disk > subsystems per chunk (onstat -g iof)? >- Can the 'disk' and channel handle that level of IO? >- Are you beating hell out of one structure while other disk spindles are > quiescent much of the time? >- Are you using RAW disk or at least DIRECT_IO? >- Have you avoided RAID5 like the plague that it is? onstat -g iof showes :AIO global files: gfd pathname totalops dskread dskwrite io/s 3 raw1 8629 38 8591 0.1 4 raw2 49352 12930 36422 0.8 5 raw3 1 1 0 0.0 6 raw4 1 1 0 0.0 7 raw5 1 1 0 0.0 onstat -g iov showes :class/vp s io/s totalops dskread dskwrite dskcopy wakeups io/wup errors msc 0 i 0.1 9225 0 0 0 9226 1.0 0 aio 0 i 6.5 417654 164344 156802 0 413630 1.0 0 aio 1 i 0.1 9496 2880 5630 0 6529 1.5 0 aio 2 i 0.1 4553 575 3886 0 1364 3.3 0 aio 3 i 0.1 4106 266 3767 0 989 4.2 0 aio 4 i 0.1 3909 174 3668 0 913 4.3 0 aio 5 i 0.1 3807 132 3626 0 834 4.6 0 aio 6 i 0.1 3760 129 3577 0 808 4.7 0 aio 7 i 0.1 3660 96 3509 0 799 4.6 0 aio 8 i 0.1 3668 105 3514 0 788 4.7 0 aio 9 i 0.1 3710 81 3576 0 777 4.8 0 pio 0 i 0.0 538 0 538 0 539 1.0 0 lio 0 i 0.1 5782 0 5782 0 5783 1.0 0 onstat -g ioq showes : AIO I/O queues:q name/id len maxlen totalops dskread dskwrite dskcopy adt 0 0 0 0 0 0 0 msc 0 0 1 9291 0 0 0 aio 0 0 8 101442 2994 2 0 pio 0 0 1 541 0 541 0 lio 0 0 1 5817 0 5817 0 gfd 3 0 3 3190 38 3152 0 gfd 4 0 150 136343 36844 99499 0 gfd 5 0 1 1 1 0 0 gfd 6 0 1 1 1 0 0 gfd 7 0 1 1 1 0 0 gfd 9 0 1 16 8 8 0 gfd 11 0 1 10 2 8 0 And I use raw device as my chunk : Chunks address chk/dbs offset size free bpages flags pathname 5768e210 1 1 0 900000 666899 PO- /opt/informix/raw1 5768ea90 2 2 0 900000 529275 PO- /opt/informix/raw2 5768eb90 3 2 0 900000 899997 PO- /opt/informix/raw3 5768ec90 4 2 0 900000 899997 PO- /opt/informix/raw4 5768ed90 5 2 0 900000 899997 PO- /opt/informix/raw5 I noticed that chuck 2 contained all the tables since chunk 3,4,5 are empty, so I just create those busy tables to chunk 3,4,5 will make I/o more smooth, I think ~~~ My cpu information : /proc/cpuinfo I list processor 0 , but it is 0 ~ 7 ... to save the time , list processor 0 : processor : 0 vendor_id : GenuineIntel cpu family : 15 model : 4 model name : Intel(R) Xeon(TM) MP CPU 3.66GHz stepping : 1 cpu MHz : 3657.666 cache size : 1024 KB physical id : 0 siblings : 2 runqueue : 0 fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm nx lm bogomips : 7287.60 And for RAID ? ... I try to post /dmesg , I don't know whether it is raid 5?! 000001 ....... : Boot DT : 1 .... IRQ redirection table: NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect: 00 000 00 1 0 0 0 0 0 0 00 01 000 00 1 0 0 0 0 0 0 00 02 000 00 1 0 0 0 0 0 0 00 03 000 00 1 0 0 0 0 0 0 00 04 000 00 1 0 0 0 0 0 0 00 05 000 00 1 0 0 0 0 0 0 00 06 000 00 1 0 0 0 0 0 0 00 07 000 00 1 0 0 0 0 0 0 00 08 000 00 1 0 0 0 0 0 0 00 09 000 00 1 0 0 0 0 0 0 00 0a 000 00 1 0 0 0 0 0 0 00 0b 000 00 1 0 0 0 0 0 0 00 0c 000 00 1 0 0 0 0 0 0 00 0d 000 00 1 0 0 0 0 0 0 00 0e 000 00 1 0 0 0 0 0 0 00 0f 000 00 1 0 0 0 0 0 0 00 10 000 00 1 0 0 0 0 0 0 00 11 000 00 1 0 0 0 0 0 0 00 12 000 00 1 0 0 0 0 0 0 00 13 000 00 1 0 0 0 0 0 0 00 14 000 00 1 0 0 0 0 0 0 00 15 000 00 1 0 0 0 0 0 0 00 16 000 00 1 0 0 0 0 0 0 00 17 000 00 1 0 0 0 0 0 0 00 IO APIC #12...... .... register #00: 0C000000 ....... : physical APIC id: 0C ....... : Delivery Type: 0 ....... : LTS : 0 .... register #01: 00178020 ....... : max redirection entries: 0017 ....... : PRQ implemented: 1 ....... : IO APIC version: 0020 .... register #02: 0C000000 ....... : arbitration: 0C .... register #03: 00000001 ....... : Boot DT : 1 .... IRQ redirection table: NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect: 00 000 00 1 0 0 0 0 0 0 00 01 000 00 1 0 0 0 0 0 0 00 02 000 00 1 0 0 0 0 0 0 00 03 000 00 1 0 0 0 0 0 0 00 04 0FF 0F 1 1 0 1 0 1 1 C1 05 000 00 1 0 0 0 0 0 0 00 06 000 00 1 0 0 0 0 0 0 00 07 000 00 1 0 0 0 0 0 0 00 08 000 00 1 0 0 0 0 0 0 00 09 000 00 1 0 0 0 0 0 0 00 0a 000 00 1 0 0 0 0 0 0 00 0b 000 00 1 0 0 0 0 0 0 00 0c 000 00 1 0 0 0 0 0 0 00 0d 000 00 1 0 0 0 0 0 0 00 0e 000 00 1 0 0 0 0 0 0 00 0f 000 00 1 0 0 0 0 0 0 00 10 000 00 1 0 0 0 0 0 0 00 11 000 00 1 0 0 0 0 0 0 00 12 000 00 1 0 0 0 0 0 0 00 13 000 00 1 0 0 0 0 0 0 00 14 000 00 1 0 0 0 0 0 0 00 15 000 00 1 0 0 0 0 0 0 00 16 000 00 1 0 0 0 0 0 0 00 17 000 00 1 0 0 0 0 0 0 00 IRQ to pin mappings: IRQ0 -> 0:0 IRQ1 -> 0:1 IRQ2 -> 0:2 IRQ3 -> 0:3 IRQ4 -> 0:4 IRQ5 -> 0:5 IRQ6 -> 0:6 IRQ7 -> 0:7 IRQ8 -> 0:8 IRQ12 -> 0:12 IRQ13 -> 0:13 IRQ14 -> 0:14 IRQ15 -> 0:15 IRQ16 -> 0:16 IRQ18 -> 0:18 IRQ19 -> 0:19 IRQ23 -> 0:23 IRQ48 -> 2:0 IRQ49 -> 2:1 IRQ100 -> 4:4 .................................... done. Using local APIC timer interrupts. calibrating APIC timer ... ..... CPU clock speed is 3657.6601 MHz. ..... host bus clock speed is 166.2571 MHz. cpu: 0, clocks: 1662571, slice: 184730 CPU0<T0:1662560,T1:1477824,D:6,S:184730,C:1662571> cpu: 1, clocks: 1662571, slice: 184730 cpu: 2, clocks: 1662571, slice: 184730 cpu: 3, clocks: 1662571, slice: 184730 cpu: 7, clocks: 1662571, slice: 184730 cpu: 5, clocks: 1662571, slice: 184730 cpu: 6, clocks: 1662571, slice: 184730 cpu: 4, clocks: 1662571, slice: 184730 CPU2<T0:1662560,T1:1108368,D:2,S:184730,C:1662571> CPU4<T0:1662560,T1:738896,D:14,S:184730,C:1662571> CPU5<T0:1662560,T1:554176,D:4,S:184730,C:1662571> CPU6<T0:1662560,T1:369440,D:10,S:184730,C:1662571> CPU7<T0:1662560,T1:184720,D:0,S:184730,C:1662571> CPU1<T0:16

sorry.... onstat -p looks so disorder , try again : dskreads 36850 pagreads 43972 bufreads 3714504778 %cached 100.00 dskwrits 103209 pagwrits 346874 bufwrits 11912950 %cached 99.13 usercpu 47787.69 syscpu 5838.60 numckpts 427 flushes 856 bufwaits 104026 lokwaits 4253 lockreqs 981169431 deadlks 0 dltouts 0 ckpwaits 1301 compress 563 seqscans 6533000 ixda-RA 1327 idx-RA 2753 da-RA 25041 RA-pgsused 29045 lchwaits 16257541 "Read Utilization (UR): ",((RApgsused /( ixdaRA + idxRA + daRA) )*100),"%"; = 29045 / (1327 + 2753 + 25041) * 100 = 99.73% "BufWaits Ratio(BR): ", ((bufwaits/(pagreads + bufwrits)) *100),"%"; = = 104026 / (43972 + 11912950) * 100 = 0.87%

Mars, See below: On Wed, Jul 9, 2008 at 11:23 PM, MARS CHEN <mars@jsun.com> wrote: > Thanks Art Kagel ... > > The following is what I get in the server : > > >- How many LRUs do you have configured? > LRUS 8 > LRU_MAX_DIRTY 2 > LRU_MIN_DIRTY 1 OK, looks low, lets see what the metrics below say, read on... > > > >- How much cache is configured? > Sorry..I don't know what this means ... Buffers ?! Yes, BUFFERS or bufferpool in the ONCONFIG file or 'Max # of Buffers in the onmonitor utility. So you have 100,000 buffers. Also sounds low. Continue... > > Physical Log Buffer Size [ 128] K > Logical Log Buffer Size [ 128] K > Max # of Logical Logs [ 200] > Max # of Locks [ 2000000] > Max # of Buffers [ 100000] > > Resident Shared Memory size [ 317542] Kbytes > > >- What is the current BR (Bufwaits Ratio)? > >- What is the BTR (Buffer Turnover Rate)? > >- What is the RAU (Read Ahead Utilization)? These three metrics are derived from the onstat -p output see the formulae and results below: BR = (bufwaits / (pagreads + bufwrits)) * 100.00 values below 7 are good, 7-10 indicate some LRU or buffer contention and means your server is slowing down, >10 is death mode, all of your users are likely complaining about performance at this point. BR = (104026 / (43972 + 11912950)) * 100.00 = 0.870 -- Looks good BTR = ((pagreads + bufwrits) / BUFFERS) / (fractional hours since the stats were last zero'd - or startup) This is an estimate the number of times per hour that you are turning over, or replacing, your entire buffer cache. Values < 10 are OK, > 10 means you probably need more buffers. Keep in mind, if your BTR is 10 you are replacing the entire contents of your buffer cache every 6 minutes or you are seriously thrashing a subset of the cache far mroe frequently than that. You did not post your onstat output heading lines, so I don't know when your server was bounced last or if you've run onstat -z to reset the statistics since startup, so I cannot calculate this one for you. You'll have to do that yourself. But here's what I do have: BTR = ((43972 + 11912950) / 100000) / ??? = 119.569 / ??? -- So, you've turned this cache over almost 120 times since the stats were last zero'd out (maybe at startup?). If that was fewer than 11.9 hours prior to running this onstat -p report, you have too few buffers. RAU = (RA-pgsused / (ixda-RA + idx-RA + da-RA)) * 100.0 This indicates the percentage of read ahead that you are actually using. It should be as close as possible to 100.00, even 99.5 is suspect if the absolute number of RA pages represented by that 1/2% represents a significant portion of your buffer cache. RAU = (29045 / (1327 + 2753 + 25041)) * 100.0 = 99.73 -- Not too bad for the default settings and the absolute numbers are low. > > post onstat -p : > > Profile > dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached > 36850 43972 3714504778 100.00 103209 346874 11912950 99.13 > > usercpu syscpu numckpts flushes > 47787.69 5838.60 427 856 > > bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans > 104026 4253 981169431 0 0 1301 563 6533000 > > ixda-RA idx-RA da-RA RA-pgsused lchwaits > 1327 2753 25041 29045 16257541 > > And ... I didnot set these 2 fields about Read Ahead ~~ > > Num of Read Ahead Pages [ ] > Read Ahead Threshold [ ] So these are defaulting to 8 & 4 respectively. Probably no ideal, but not excessive. > > > >- How many IO operations per second are being thrown at your disk > > subsystems per chunk (onstat -g iof)? > >- Can the 'disk' and channel handle that level of IO? > >- Are you beating hell out of one structure while other disk spindles are > > quiescent much of the time? > >- Are you using RAW disk or at least DIRECT_IO? > >- Have you avoided RAID5 like the plague that it is? > > onstat -g iof showes :> AIO global files: > gfd pathname totalops dskread dskwrite io/s > 3 raw1 8629 38 8591 0.1 > 4 raw2 49352 12930 36422 0.8 > 5 raw3 1 1 0 0.0 > 6 raw4 1 1 0 0.0 > 7 raw5 1 1 0 0.0 > > onstat -g iov showes :> class/vp s io/s totalops dskread dskwrite dskcopy wakeups io/wup errors > msc 0 i 0.1 9225 0 0 0 9226 1.0 0 > aio 0 i 6.5 417654 164344 156802 0 413630 1.0 0 > aio 1 i 0.1 9496 2880 5630 0 6529 1.5 0 > aio 2 i 0.1 4553 575 3886 0 1364 3.3 0 > aio 3 i 0.1 4106 266 3767 0 989 4.2 0 > aio 4 i 0.1 3909 174 3668 0 913 4.3 0 > aio 5 i 0.1 3807 132 3626 0 834 4.6 0 > aio 6 i 0.1 3760 129 3577 0 808 4.7 0 > aio 7 i 0.1 3660 96 3509 0 799 4.6 0 > aio 8 i 0.1 3668 105 3514 0 788 4.7 0 > aio 9 i 0.1 3710 81 3576 0 777 4.8 0 > pio 0 i 0.0 538 0 538 0 539 1.0 0 > lio 0 i 0.1 5782 0 5782 0 5783 1.0 0 It looks like either you have KAIO turned off, or all of your chunks are COOKED files or Block devices. That means that your AIO VPs are doing all of the IO work and there are not enough of them. All of you AIO VPs show from 1.0 to 4.8 IOs per wake up (io/wup). You should have at least one of these with an io/wup that shows less than 1.0. I'm estimating that you need to increase from 10 to 20 AIO VPs or to move to RAW chunks an KAIO. Right now, on average, 4 out of 5 IO requests is waiting at least some time before being serviced. > > > onstat -g ioq showes : > AIO I/O queues:> q name/id len maxlen totalops dskread dskwrite dskcopy > adt 0 0 0 0 0 0 0 > msc 0 0 1 9291 0 0 0 > aio 0 0 8 101442 2994 2 0 > pio 0 0 1 541 0 541 0 > lio 0 0 1 5817 0 5817 0 > gfd 3 0 3 3190 38 3152 0 > gfd 4 0 150 136343 36844 99499 0 This device, gfd4, has a max wait queue of 150 requests. For whatever reason (aio vps are busy, the disk or controller is slow, there is contention for the disk or controller with other applications sharing the resource) this device is slow. > > gfd 5 0 1 1 1 0 0 > gfd 6 0 1 1 1 0 0 > gfd 7 0 1 1 1 0 0 > gfd 9 0 1 16 8 8 0 > gfd 11 0 1 10 2 8 0 > > And I use raw device as my chunk : Just because you named them 'raw1' etc. doesn't make them RAW. Are these Character devices or Block devices? Only Character devices are RAW, filesystem files and block devices are all COOKED. If you are running on Linux or a later version of Solaris you can turn on O_DIRECT IO which can use KAIO mode IO which is faster than using the AIO VPs to simulate Kernel IO. Right now you are using AIO VPs which can ONLY happen if the chunks are COOKED or if you set KAIOOFF=1 in the server's environment (or if you are running under HPUX you did not set KAIOON=1 in the environment - HPUX is the ONLY platform that defaults to KAOOFF). Sounds like you are running on Linux. You can turn on O_DIRECT IO by adding the parameter DIRECT_IO 1 into your ONCONFIG file. > > Chunks > address chk/dbs offset size free bpages flags pathname > 5768e210 1 1 0 900000 666899 PO- /opt/informix/raw1 > 5768ea90 2 2 0 900000 529275 PO- /opt/informix/raw2 > 5768eb90 3 2 0 900000 899997 PO- /opt/informix/raw3 > 5768ec90 4 2 0 900000 899997 PO- /opt/informix/raw4 > 5768ed90 5 2 0 900000 899997 PO- /opt/informix/raw5 > > I noticed that

Dear Art Kagel ... Thanks for your kind response ~~ My case is the IDS restart every 24 hours , the very busy is about 5 hours , and not very busy is 3 hours , and 16 hours without any sessions , so I'll try to onstat -z at tomorrow morning just before rush hours to come , to figure out the correct number from onstat -p ... Meanwhile , I think I use the raw device , I check : #cat /etc/sysconfig/rawdevices /dev/raw/raw1 /dev/sda11 /dev/raw/raw2 /dev/sda12 /dev/raw/raw3 /dev/sda13 /dev/raw/raw4 /dev/sda14 /dev/raw/raw5 /dev/sda15 and fdisk /dev/sda , I create those partiton : /dev/sda11 6374 6628 2048256 83 Linux /dev/sda12 6629 6883 2048256 83 Linux /dev/sda13 6884 7138 2048256 83 Linux /dev/sda14 7139 7393 2048256 83 Linux /dev/sda15 7394 7648 2048256 83 Linux and symbolic link to /dev/raw : # ls -l /opt/informix lrwxrwxrwx 1 informix informix 13 Sep 2 2005 raw1 -> /dev/raw/raw1 lrwxrwxrwx 1 informix informix 13 Sep 2 2005 raw2 -> /dev/raw/raw2 lrwxrwxrwx 1 informix informix 13 Sep 2 2005 raw3 -> /dev/raw/raw3 lrwxrwxrwx 1 informix informix 13 Sep 2 2005 raw4 -> /dev/raw/raw4 lrwxrwxrwx 1 informix informix 13 Sep 2 2005 raw5 -> /dev/raw/raw5 ls -l /dev/raw/raw1 crw-rw---- 1 informix informix 162, 1 Jun 25 2004 /dev/raw/raw1 [root@linux06 root]# ls -l /dev/raw/raw2 crw-rw---- 1 informix informix 162, 2 Jun 25 2004 /dev/raw/raw2 [root@linux06 root]# ls -l /dev/raw/raw3 crw-rw---- 1 informix informix 162, 3 Jun 25 2004 /dev/raw/raw3 [root@linux06 root]# ls -l /dev/raw/raw4 crw-rw---- 1 informix informix 162, 4 Jun 25 2004 /dev/raw/raw4 [root@linux06 root]# ls -l /dev/raw/raw5 crw-rw---- 1 informix informix 162, 5 Jun 25 2004 /dev/raw/raw5 as preceeding showes , I think I use raw devices , but I don't know how to enable KAIO , it is in a RedHat Enterprise Linux 3.x OS , If I am sure that chunk are all raw devices , still I should set DIRECT_IO 1 in onconfig ?!... it works at 7.31 ?! I used to think , onstat -g rea has threads queue over there , it must be the cpu power not enough , but as your information , It can be something else like I/O , Buffers,.... , not just easily thinking it should be the cpu vps issue , so my concept is wrong !!.... if your cpus are very powerful , very fast , but if you have a very slow I/O , or you have a very small Buffers , it will still cause the onstat -g rea queue a lot of threads , and make OS cpu idle also ... I think that what you want to tell me~ Thanks again for your kind help ~

Sorry ... to post again ~~~ According to http://www.informixfaq.com/faq.php?id=6 , "LRUs and BUFFERs" , the recommended LRUS is set 128 , I just set this 8 ... that is a huge gap !! I've 4G RAM in the server , so if I modify LRU , 8 -> 128 , I also need to change Buffers , say from 100000 -> 200000 , to get the good effects of LRUs , Am i correct ?!

Yes, you've got it. You are correct, your chunks certainly do look like they are RAW and so KAIO should be working for you, but, as I said, it's not. Your onstat -g iov output does not show any kio threads running. KAIO is on by default on all platforms except HPUX (because HPUX doesn't come with KAIO installed, it's a separate package). However, check out the engine startup script, the environment variable KAIOOFF can be used to disable KAIO, maybe someone put it in ther by mistake. Also, no IDS 7.31 does not have O_DIRECT support, it's in 10.00 and later and the most recent releases of 9.40. Art On Thu, Jul 10, 2008 at 2:28 AM, MARS CHEN <mars@jsun.com> wrote: > Dear Art Kagel ... > > Thanks for your kind response ~~ > > My case is the IDS restart every 24 hours , the very busy is about 5 hours > , > and not very busy is 3 hours , and 16 hours without any sessions , > so I'll try to onstat -z at tomorrow morning just before rush hours to come > , > to figure out the correct number from onstat -p ... > > Meanwhile , I think I use the raw device , I check : > #cat /etc/sysconfig/rawdevices > > /dev/raw/raw1 /dev/sda11 > /dev/raw/raw2 /dev/sda12 > /dev/raw/raw3 /dev/sda13 > /dev/raw/raw4 /dev/sda14 > /dev/raw/raw5 /dev/sda15 > > and fdisk /dev/sda , I create those partiton : > > /dev/sda11 6374 6628 2048256 83 Linux > /dev/sda12 6629 6883 2048256 83 Linux > /dev/sda13 6884 7138 2048256 83 Linux > /dev/sda14 7139 7393 2048256 83 Linux > /dev/sda15 7394 7648 2048256 83 Linux > > and symbolic link to /dev/raw : > # ls -l /opt/informix> > lrwxrwxrwx 1 informix informix 13 Sep 2 2005 raw1 -> /dev/raw/raw1 > lrwxrwxrwx 1 informix informix 13 Sep 2 2005 raw2 -> /dev/raw/raw2 > lrwxrwxrwx 1 informix informix 13 Sep 2 2005 raw3 -> /dev/raw/raw3 > lrwxrwxrwx 1 informix informix 13 Sep 2 2005 raw4 -> /dev/raw/raw4 > lrwxrwxrwx 1 informix informix 13 Sep 2 2005 raw5 -> /dev/raw/raw5 > > ls -l /dev/raw/raw1 > crw-rw---- 1 informix informix 162, 1 Jun 25 2004 /dev/raw/raw1 > [root@linux06 root]# ls -l /dev/raw/raw2 > crw-rw---- 1 informix informix 162, 2 Jun 25 2004 /dev/raw/raw2 > [root@linux06 root]# ls -l /dev/raw/raw3 > crw-rw---- 1 informix informix 162, 3 Jun 25 2004 /dev/raw/raw3 > [root@linux06 root]# ls -l /dev/raw/raw4 > crw-rw---- 1 informix informix 162, 4 Jun 25 2004 /dev/raw/raw4 > [root@linux06 root]# ls -l /dev/raw/raw5 > crw-rw---- 1 informix informix 162, 5 Jun 25 2004 /dev/raw/raw5 > > as preceeding showes , I think I use raw devices , but I don't know > how to enable KAIO , it is in a RedHat Enterprise Linux 3.x OS , > If I am sure that chunk are all raw devices , still I should set > DIRECT_IO 1 in onconfig ?!... it works at 7.31 ?!> > I used to think , onstat -g rea has threads queue over there , > it must be the cpu power not enough , but as your information , > It can be something else like I/O , Buffers,.... , > not just easily thinking it should be the cpu vps issue , > so my concept is wrong !!.... if your cpus are very powerful , very fast , > but if you have a very slow I/O , or you have a very small Buffers , > it will still cause the onstat -g rea queue a lot of threads , > and make OS cpu idle also ... I think that what you want to tell me~ > > Thanks again for your kind help ~ > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves.

the more LRUS the merrier, but if you did not have enough LRU queues your BR would be sky high. I wouldn't worry until you have many more users running. Art On Thu, Jul 10, 2008 at 3:33 AM, MARS CHEN <mars@jsun.com> wrote: > Sorry ... to post again ~~~ > > According to http://www.informixfaq.com/faq.php?id=6 , > "LRUs and BUFFERs" , the recommended LRUS is set 128 , > I just set this 8 ... that is a huge gap !! > > I've 4G RAM in the server , so if I modify LRU , 8 -> 128 , > I also need to change Buffers , say from 100000 -> 200000 , > to get the good effects of LRUs , Am i correct ?! > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves.

You say : ----------------------------------------------------------------------------- the more LRUS the merrier, but if you did not have enough LRU queues your BR would be sky high. I wouldn't worry until you have many more users running. ----------------------------------------------------------------------------- Sorry , my english is really poor , "the more LRUS the merrier" , it is "the more BUFFERs the merrier" ?! forgive me for my poor english ...I didn't catch your point.

>Yes, you've got it. You are correct, your chunks certainly do look like >they are RAW and so KAIO should be working for you, but, as I said, it's >not. Your onstat -g iov output does not show any kio threads running. KAIO >is on by default on all platforms except HPUX (because HPUX doesn't come >with KAIO installed, it's a separate package). However, check out the >engine startup script, the environment variable KAIOOFF can be used to >disable KAIO, maybe someone put it in ther by mistake. Also, no IDS 7.31 >does not have O_DIRECT support, it's in 10.00 and later and the most recent >releases of 9.40. Thanks for your kind response ~~ I set KAIOON=1 , and then oninit immediately , the result of onstat -g iov still showes aio , not kaio .... Maybe I forget to config something for IDS 7.31 or RedHat Enterprise Linux 3, the kernel or else ~~

Your English is fine. You asked about changing the setting for LRUS from 8 to 128 per some recommendation. I replied that having more LRUS isn't bad thing and mostly a good thing, however, in your particular case you are not experiencing any contention between sessions for access to the LRU queues. Such contention is the main cause of increased BR (Buffwaits Ratio) values - not enough buffers can have some effect also because it can cause excessive buffer thrashing which requires more LRU latching than a system that's not thrashing. However, your BR values are low which is good and indicates that you have enough LRU queues to prevent contention for that resource. Art On Thu, Jul 10, 2008 at 3:58 AM, MARS CHEN <mars@jsun.com> wrote: > You say : > > > ----------------------------------------------------------------------------- > the more LRUS the merrier, but if you did not have enough LRU queues your > BR > would be sky high. I wouldn't worry until you have many more users running. > > ----------------------------------------------------------------------------- > > Sorry , my english is really poor , "the more LRUS the merrier" , > it is "the more BUFFERs the merrier" ?! > > forgive me for my poor english ...I didn't catch your point. > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves.

AHH! RH3 did not support O_DIRECT may not have supported KAIO (Kernel Asynchronous IO) or even REAL RAW devices (Linus was against RAW device support for a LOOOONG time, even now he's trying to get rid of RAW now that O_DIRECT is almost as good for databases. That may be your problem. Check that out on the RH support groups, I'm no RH expert. Art On Thu, Jul 10, 2008 at 4:07 AM, MARS CHEN <mars@jsun.com> wrote: > >Yes, you've got it. You are correct, your chunks certainly do look like > >they are RAW and so KAIO should be working for you, but, as I said, it's > >not. Your onstat -g iov output does not show any kio threads running. KAIO > >is on by default on all platforms except HPUX (because HPUX doesn't come > >with KAIO installed, it's a separate package). However, check out the > >engine startup script, the environment variable KAIOOFF can be used to > >disable KAIO, maybe someone put it in ther by mistake. Also, no IDS 7.31 > >does not have O_DIRECT support, it's in 10.00 and later and the most > recent > >releases of 9.40. > > Thanks for your kind response ~~ > > I set KAIOON=1 , and then oninit immediately , the result of onstat -g iov > still showes aio , not kaio .... > > Maybe I forget to config something for IDS 7.31 or RedHat Enterprise Linux > 3, > the kernel or else ~~ > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves.

I seem to recall that FC3 supported Raw, but when I moved to FC4 the support was no longer there. How that maps to RH3 I am unsure. j. On Thu, Jul 10, 2008 at 11:01 AM, Art Kagel wrote: > AHH! RH3 did not support O_DIRECT may not have supported KAIO (Kernel Asynchronous IO) or even REAL RAW devices (Linus was against RAW device support for a LOOOONG time, even now he's trying to get rid of RAW now that O_DIRECT is almost as good for databases. That may be your problem. Check that out on the RH support groups, I'm no RH expert. Art On Thu, Jul 10, 2008 at 4:07 AM, MARS CHEN <mars@jsun.com <mailto:mars@jsun.com> <mailto:mars@jsun.com> > wrote: >> Yes, you've got it. You are correct, your chunks certainly do look >> like they are RAW and so KAIO should be working for you, but, as I >> said, it's not. Your onstat -g iov output does not show any kio >> threads running. KAIO is on by default on all platforms except HPUX >> (because HPUX doesn't come with KAIO installed, it's a separate >> package). However, check out the engine startup script, the >> environment variable KAIOOFF can be used to disable KAIO, maybe >> someone put it in ther by mistake. Also, no IDS 7.31 does not have >> O_DIRECT support, it's in 10.00 and later and the most > recent >> releases of 9.40. > > Thanks for your kind response ~~ > I set KAIOON=1 , and then oninit immediately , the result of onstat -g > iov still showes aio , not kaio .... > Maybe I forget to config something for IDS 7.31 or RedHat Enterprise > Linux 3, the kernel or else ~~ > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > -- Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) <mailto:(art@iiug.org)> <mailto:(art@iiug.org)> Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Thanks for Jack and Art ... I'll try 11.50 IDS and RedHat 5 , to see whethere raw or O_DIRECT can help me to improve the performance , thanks a lot ~~~

Thank you , Art ... Now I got your point , and thanks a lot these days ~ Mars

We are using kaio on RHEL3, you need to have the libaio rpm installed on your machine. The Informix chunks must be bound using the raw utility, as follows. $ cat /etc/sysconfig/rawdevices# raw device bindings # format: <rawdev> <major> <minor> # <rawdev> <blockdev> # example: /dev/raw/raw1 /dev/sda1 # /dev/raw/raw2 8 5 /dev/raw/raw1 /dev/sdc1 /dev/raw/raw2 /dev/sdc2 $ Then as root execute /etc/init.d/rawdevices restart to bind the new devices and give the Informix user the permissions on the devices chown informix:informix /dev/raw/raw[12] (devices as in example above). Use /dev/raw/raw1 or links that point to these devices in the Informix configuration. The KAIOON variable controls the number of requests allocated to each CPU VP thus it should be set to around 10000, if need be tune /proc/sys/fs/aio-max-nr. Regards, Kenneth Penza -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Art Kagel Sent: 10 July 2008 17:01 To: ids@iiug.org Subject: Re: onstat -g rea almot 100 queue there !!! [12629] AHH! RH3 did not support O_DIRECT may not have supported KAIO (Kernel Asynchronous IO) or even REAL RAW devices (Linus was against RAW device support for a LOOOONG time, even now he's trying to get rid of RAW now that O_DIRECT is almost as good for databases. That may be your problem. Check that out on the RH support groups, I'm no RH expert. Art On Thu, Jul 10, 2008 at 4:07 AM, MARS CHEN <mars@jsun.com> wrote: > >Yes, you've got it. You are correct, your chunks certainly do look like > >they are RAW and so KAIO should be working for you, but, as I said, it's > >not. Your onstat -g iov output does not show any kio threads running. KAIO > >is on by default on all platforms except HPUX (because HPUX doesn't come > >with KAIO installed, it's a separate package). However, check out the > >engine startup script, the environment variable KAIOOFF can be used to > >disable KAIO, maybe someone put it in ther by mistake. Also, no IDS 7.31 > >does not have O_DIRECT support, it's in 10.00 and later and the most > recent > >releases of 9.40. > > Thanks for your kind response ~~ > > I set KAIOON=1 , and then oninit immediately , the result of onstat -g iov > still showes aio , not kaio .... > > Maybe I forget to config something for IDS 7.31 or RedHat Enterprise Linux > 3, > the kernel or else ~~ > > > > ************************************************************************ ******* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum.

RE: Set up for raw spaces in Red Hat Here is what we use to set up raw space. The f_partition is run only once. The script sits in /etc/init.d and is run on reboot or when we make a change. John ====================== ... function f_partition # custom partitioning using fdisk to match the SAN. # Failover system may have the same setup. { echo f_partition for i in sddz sdea sdeb sdec sded sdee \\\\ sdef sdeg sdeh sdei do sfdisk /dev/${i} << EOF unit: sectors /dev/${i}: start= 16065, size= 4176900, Id=83 /dev/${i}: start= 4192965, size= 4176900, Id=83 /dev/${i}: start= 8369865, size= 4176900, Id=83 /dev/${i}: start= 12546765, size= 29382885, Id=85 /dev/${i}: start= 12562830, size= 4176900, Id=83 /dev/${i}: start= 16755795, size= 4176900, Id=83 /dev/${i}: start= 20948760, size= 4176900, Id=83 /dev/${i}: start= 25141725, size= 4176900, Id=83 /dev/${i}: start= 29334690, size= 4176900, Id=83 /dev/${i}: start= 33527655, size= 4176900, Id=83 /dev/${i}: start= 37720620, size= 4176900, Id=83 EOF done } function f_LUN_raw_block # Create the Informix chunk spaces. Update /etc/init.d/informix_LUN # with changes made here. Note the altenating nature of the # block names. sdb1 is controller 1, sdc1 is controller 2, etc. { echo f_LUN_raw_block case $( id -un ) in root ) ;; * ) echo "Run as root" ; exit ;; esac while read raw block chunk junk do case $raw in '#' ) continue ;; esac if [ ! -L /dbspaces/$chunk ] ; then #su - informix -c "ln -s $raw /dbspaces/$chunk" ln -s $raw /dbspaces/$chunk chown informix:informix $raw chown informix:informix $block fi raw $raw $block done <<- ! # # Change this to create spaces across the SAN LUNs sddz - sdei # Each LUN has partitions 1 - 11 # Do not use extended partition 4 ( sddz4, etc ) # /mclogs is on /dev/sdej1, /h in on /dev/sdej2 # and /dpsr is on /dev/sdej3 which are not in this group of partitions # # Check performance of LUN # iostat -x /dev/sddz1 -x /dev/sdea1 -x /dev/sdeb1 5 # # WARNING: if you change the order of the items in this list you # probably need to delete the corresponding link in /dbspaces. # /dev/raw/raw1 /dev/sddz1 rootdbs /dev/raw/raw2 /dev/sdea1 ifmx_sys_0 # physical & logical /dev/raw/raw3 /dev/sdeb1 ifmx_sys_1 # logs /dev/raw/raw4 /dev/sdec1 ifmx_sys_2 /dev/raw/raw5 /dev/sded1 mdmetc_d00 # metcast (d)bspace /dev/raw/raw6 /dev/sdee1 mdmetc_b00 # metcast (b)lobspace /dev/raw/raw7 /dev/sdef1 mdchnl_d00 # channels (d)bspace /dev/raw/raw8 /dev/sdeg1 mdchnl_b00 # channels (b)lobspace /dev/raw/raw9 /dev/sdeh1 mdchnl_s00 # channels (s)mart blob /dev/raw/raw10 /dev/sdei1 mdimg_d00 # image (d)bspace /dev/raw/raw11 /dev/sddz2 mdimg_b00 # image (b)lobspace /dev/raw/raw12 /dev/sdea2 mdimg_b01 # image (b)lobspace # /dev/raw/raw13 /dev/sdeb2 mdtxt_d00 # txt # /dev/raw/raw14 /dev/sdec2 mdtxt_b00 # /dev/raw/raw15 /dev/sded2 mdrem_d00 # rem # /dev/raw/raw16 /dev/sdee2 mdrem_b00 # /dev/raw/raw17 /dev/sdef2 # future use /dev/raw/raw18 /dev/sdeg2 opars_d00 # opars /dev/raw/raw19 /dev/sdeh2 mdgrid_d00 # grid (d)bspace /dev/raw/raw20 /dev/sddi2 obs_fep_00 /dev/raw/raw21 /dev/sddz3 obs_fep_01 /dev/raw/raw22 /dev/sdea3 obs_fep_02 /dev/raw/raw23 /dev/sdeb3 obs_fep_03 /dev/raw/raw24 /dev/sdec3 obs_fep_04 /dev/raw/raw25 /dev/sded3 obs_fep_05 # /dev/raw/raw26 /dev/sdee3 # future use /dev/raw/raw27 /dev/sdef3 mdgrid_b00 # grid (b)lobspace /dev/raw/raw28 /dev/sdeg3 mdgrid_b01 /dev/raw/raw29 /dev/sdeh3 mdgrid_b02 /dev/raw/raw30 /dev/sdei3 mdgrid_b03 # # # 4 is extended partition # /dev/raw/raw31 /dev/sddz5 mdgrid_b04 /dev/raw/raw32 /dev/sdea5 mdgrid_b05 /dev/raw/raw33 /dev/sdeb5 mdgrid_b06 ... /dev/raw/raw98 /dev/sdeg11 mdgrid_b71 /dev/raw/raw99 /dev/sdeh11 mdgrid_b72 /dev/raw/raw100 /dev/sdei11 mdgrid_b73 # # # # There is a limit of 255 raw partitions # # ! } #--- main ---------------------------------------------------------- # root space for the Informix db space links. [[ -d /dbspaces ]] || { mkdir /dbspaces chown informix:informix /dbspaces chmod 775 /dbspaces } # case $1 in start ) f_LUN_raw_block ;; stop ) echo "Stopping informix_LUN" ;; partition ) f_partition ;; esac

Thanks for all who help me to solve my problem ~~~~ My IDS 7.31 running in RedHat Enterprise linux 3 , rpm -qa | grep "libaio*" showes libaio-devel-0.3.96-5 and libaio-0.3.96-5 ... I wonder if this is the case I need to upgrade libaio's version ?! Here is how I set raw devices in RHEL 3 : 1. fdisk /dev/sda to add partition !!! /dev/sda11 6373 6629 2064321 83 Linux /dev/sda12 6630 6886 2064321 83 Linux /dev/sda13 6887 7143 2064321 83 Linux /dev/sda14 7144 7400 2064321 83 Linux /dev/sda15 7401 7657 2064321 83 Linux 2. vi /etc/sysconfig/rawdevices to map the partition !! /dev/raw/raw1 /dev/sda11 /dev/raw/raw2 /dev/sda12 /dev/raw/raw3 /dev/sda13 /dev/raw/raw4 /dev/sda14 /dev/raw/raw5 /dev/sda15 3. check if it is raw : ls -l /dev/raw/raw1 crw-rw---- 1 informix informix 162, 1 Jun 25 2004 /dev/raw/raw1 ls -l /dev/raw/raw2 crw-rw---- 1 informix informix 162, 1 Jun 25 2004 /dev/raw/raw2 ls -l /dev/raw/raw3 crw-rw---- 1 informix informix 162, 1 Jun 25 2004 /dev/raw/raw3 ls -l /dev/raw/raw4 crw-rw---- 1 informix informix 162, 1 Jun 25 2004 /dev/raw/raw4 ls -l /dev/raw/raw5 crw-rw---- 1 informix informix 162, 1 Jun 25 2004 /dev/raw/raw5 4. make symbolic link : ls -l /opt/informix/raw* lrwxrwxrwx 1 informix informix 13 Sep 15 2005 raw1 -> /dev/raw/raw1 lrwxrwxrwx 1 informix informix 13 Sep 15 2005 raw2 -> /dev/raw/raw2 lrwxrwxrwx 1 informix informix 13 Sep 15 2005 raw3 -> /dev/raw/raw3 lrwxrwxrwx 1 informix informix 13 Sep 15 2005 raw4 -> /dev/raw/raw4 lrwxrwxrwx 1 informix informix 13 Sep 15 2005 raw5 -> /dev/raw/raw5 and then ... I still can not have kio in onstat -g iov , cat /proc/slabinfof | grep "kio" showes : kioctx 0 0 128 0 0 1 : 1008 252 kiocb 0 0 128 0 0 1 : 1008 252 kiobuf 60 60 128 2 2 1 : 1008 252 And then , I have another env IDS 11.50 developer's edition running in RHEL 5 , which have libaio version libaio-0.3.106-3.2 , and then I set raw device in /etc/udev/rules.d/6-=raw.rules , and make the symbolic link to those raw device as usual , this time in IDS 11.50 , onstat -g iov will see kio threads in there ~~~ So far ... I wonder it is the libaio's version problem , but not sure yet ...