Re: Kernel Asynchronous IO on HP-UX

Here is the real story behind Informix/HP KAIO over the years. It so long so those of you that make it to the end I have include some HP Kaio tuning tips. There have been three different version of HP KAIO over the years which are basically the same except their polling medthods. You can tell which version of KAIO by the line put in the online.log at startup Version 1 "HPUX Version XXX -> Using select based KAIO" Version 2 "HPUX Version XXX -> Using flag style KAIO" Version 3 "HPUX Version XXX ->Using flag/select style KAIO" Version 1 It uses the select() system call to poll the OS for IO completions status. This causes the kernel to use excess amounts of system time due to the expensive nature of select() system call on HP. Advantages: Very fast Disadvantages: As the system gets busy the select() system call gets very expensive. Online Versions 7.1, 7.2, 7.3 Version 2 It use a memory address registered with the kernel to poll the OS for IO completions status. When the server issues an I/O request through KAIO, it must poll the HP-UX asynchronous driver to determine when the request has been fulfilled. If a request has not yet been filled, the I/O thread may choose to go to sleep for 10 milliseconds before checking the request status again. Up through ODS 7.1, the I/O thread would sleep using a select() system call. This was changed in 7.2 to use the newer and lighter nanosleep() system call. Performance tests showed this was lighter on the CPU, and still gave us the desired sleep behavior. Sometime after 7.2 was released (we believe), nanosleep() was changed to correctly conform to its IEEE standard. One of the impacts is that we can no longer sleep for just 10 milliseconds. On average, the I/O thread is now sleeping for almost twice that long (when it does not need to) and this is why we believe KAIO performance has gotten so much worse. Advantages: Signigicant reduction in resources Disadvantages: When the system is not busy the kaio poll thread will wait for I/O an extra 6 to 8 ms (Defect #92132) Not supported prior to HPUX 10.10 Online Versions 7.2, 7.3 Version 3 It use the memory address registered with the kernel to poll the OS for I/O completions status when the system is busy, but when the system goes idle and waits on I/O completions the system will not use nanosleep as in version 2, but will use the select() system call to wait. Advantages: Signigicant reduction in resources Very fast. In direct response to Defect #92132 Disadvantages: None found to date. Online Versions 7.30.UC3 and above and 9.14.UC1 and above Question: Using the select() system call won't this cause the same performance issues as kaio version 1? Answer: No it will not. The problem with version one was that as the system became busier the amount of I/O inceased which caused an increase in select() calls to be made. In the new method (version 3) we only use the select system call if the system is idle and is wait on I/O. In addition there is a benefit of using the select system call which is the ability to interupt the timer if and I/O completes before the timer expires. This will give early notification of the I/O. Tune Tips for HP KAIO RESIDENCY In version 7.30 for best KAIO performance set RESIDENT flag in the onconfig to -1. In versions prior to version 7.30 set the resident flag to 1. This will reduce a significant amount of kernel locking that must be done on non-resident segments. NOTE: Not setting the RESIDENT flag in the onconfig will cause an additional 8KB of memory to be allocated because the online system will require assurance that this flag is on its on OS memory page. If the RESIDENCY flag is set this assurance does not have to be done. IFMX_HPKAIO_NUM_REQ This specifies the maximum number of simultaneous KAIO request online can process at one time. This can currently be set between HPKAIO_MAX_CONCURRENT (5000) and HPKAIO_MIN_CONCURRENT (10) with a default value of HPKAIO_DEF_CONCURRENT 1000. The only draw back to setting this higher is memory. Below is the formula to determine how much memory will be used: bytes of memory = (# of cpuvps)*((12*IFMX_HPKAIO_NUM_REQ)+4) The default setting should take about 12KB per cpu vp. If you see the error "KAIO: out of OS resources, errno = %d, pid = %d" then you should consider raising IFMX_HPKAIO_NUM_REQ. Hope this helps and answers alot of your questions, ---jmiller John Miller Pete Smith wrote: > Defect 92132 is one such reported problem: > It reads thus: > > HP KAIO HAS A SIGNIFICANT I/O BOTTLENECK WHICH CAN CAUSE 30% TO 75% IN > PERFORMANCE > The general symptom of the problem is that HP KAIO (native kernel > async I/O) is *much* slower than Informix AIO. The impact is > anywhere from 30% to >100% slowdown in disk I/O. > The problem lies in the server's I/O polling mechanism. When the > server issues an I/O request through KAIO, it must poll the HP-UX > asynchronous driver to determine when the request has been > fulfilled. If a request has not yet been filled, the I/O thread may > choose to go to sleep for 10 milliseconds before checking the request > status again. Up through ODS 7.1, the I/O thread would sleep using a > select() system call. This was changed in 7.2 to use the newer and > lighter nanosleep() system call. Performance tests showed this was > lighter on the CPU, and still gave us the desired sleep behavior. > Sometime after 7.2 was released (we believe), nanosleep() was changed > to correctly conform to its IEEE standard. One of the impacts is > that we can no longer sleep for just 10 milliseconds. On average, > the I/O thread is now sleeping for almost twice that long (when it > does not need to) and this is why we believe KAIO performance has > gotten so much worse. > > I know of one of customer's running into this when upgrading from 7.13 to > 7.24.UC5. > We don't use KAIO on our HP kit because of the performance hit. > I'm not aware of any release which has fixed this, either > > HTH > > Pete Smith > DBA > Aethos Communication Systems > > EsquivelR wrote: > > > I have two Unix servers which run Online 7.23.UC7 and Online 7.14.UD6. Both > > servers are running HP-UX 10.20. Has anyone out there had any experience using@@NL@