Re: Raw Disk still preferred for 11.5?

Superboer wrote: > Hello Fernando, > > does DIRECT_IO bypasses the filesystem cache??? using cooked can kill > performance because linux/unix is being hit > (to) hard. Yes, but look for further comments below. > one example is do a restore on char mode raw dev and cooked specially > on Sun. Cooked restore is a factor 2/3 or more slower!!! > besides this the machine is suffering when the restore is done. > On linux this is no different and one can not limit the amount of > memory used for fs cache... well not that i am aware of like > hp does with > DBC_MAX_PCT and DBC_MIN_PCT. I didn't test this recently, but I recall a situation on Tru64 using advfs where the file space allocation was terribly slow. When the restore was running, when it started a new chunk it spend an enormous amount of time just to allocate the chunk size. I really didn't try to figure out if this was a problem in Informix or in the advfs file system. But it didn't happen with ufs with the same IDS version (this doesn't necessarily means that IDS wasn't doing something not recommend). If we repeat the restore it would be much faster just because the files were already fully allocated. As a workaround I run a simple dd to create the files and tried to keep the dd ahead of the restore ;) A real race for time :P > > Also on aix i could do 50 mb/sec writes to char mode raw (yeah yeah 7 > years ago...) while the aix teacher did 40 mb/sec writing to the > filesystem using dd.....and that was really the best he could do.... Using the same block size? > > So if filesystem caches etc are not seriously improved since the last > 7 years i still would gofor char mode raw always. > and not being lazy Now the complex stuff...: What DIRECT_IO does really depends on your file system supported features. Accordingly to the manual: "Direct I/O bypasses the use of the file system buffers, and therefore is more efficient for reads and writes that go to disk (as opposed to those reads and writes that merely access the file system buffers). Direct I/O generally requires data to be aligned on disk sector boundaries. Direct I/O also generally allows the use of kernel asynchronous I/O (KAIO), which provides a further boost to performance. By using direct I/O and KAIO where available, the performance of cooked files used for dbspace chunks can approach the performance of raw devices." And... "If your file system supports direct I/O for the page size used for the dbspace chunk, Dynamic Server operates as follows: - Does not use direct I/O by default. - Uses direct I/O if the DIRECT_IO configuration parameter is set to 1. - Uses KAIO (if the file system supports it) with direct I/O by default. - Does not use KAIO with direct I/O if the environment variable KAIOOFF is set. - Does not use direct I/O for temporary dbspaces." So, setting DIRECT_IO makes IDS try to use some of the underlying fs features. The first and more generally available is to bypass filesystem cache. Some filesystems set this as a general property, others allow different files to use or not use buffer caching. This will depend on the flags used for the I/O primitives. Besides skipping buffer cache, it will try to do KAIO. I'm not an expert on this, but if you use KAIO you'll have less "mode switching" (from privilege/kernel mode to user mode) Also, and I'm not 100% sure about this, in certain file systems it will also try to use CIO or similar. CIO means concurrent I/O. Basically, even using DIO, there is still one big bottleneck: filesystem inode locking. Some FS allow the application to specify that it should not manage inode locking (making it theoretically possible to have two processes writing to the same file which in normal circumstances would cause corruption). With databases, only one process should have the files opened, so using CIO would be perfectly safe... I was amazed to see a study from IBM (it's somewhere in developerworks) where it's showed that only by activating CIO they would approach RAW performance... I would not think that this inode locking would have such a great impact on performance... So... I would like to see more details coming out from R&D about this, but I understand it may be difficult... It's very techie stuff, and it's not only platform dependent but also FS dependent... That's why all this manuals, papers, blog articles etc. all include a big "TEST IT"... That's why I would recommend: 1- TEST IT ;) 2- If performance is absolutely critical and you can accept the RAW management overhead, go to RAW. 3- If you can concede a bit of I/O performance and you prefer to use COOKED, go to cooked... 4- If performance is absolutely critical and you prefer to use COOKED, then go to 1) :P In recent situations, a customer was used to RAW, and didn't want to test, so I advised RAW for production and Cooked for development/quality (if the processes run slow in quality they will try to improve them, which will be good for the production environment ;) Anyway, I would love to hear feedback from real systems in different platforms/fs Regards. -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently...