Question about Linux VM tuning for IDS

Hi all! IDS 10.00.FC5 Red Hat Enterprise Linux AS release 4 (Nahant Update 4) (2.6.9-42.ELsmp x86_64) I'm wondering if anyone knows of any best practise on how to configure memory handling on Linux 2.6 kernel for a *dedicated* IDS box. On other operating systems there are ways to protect the shared memory segments allocated to applications (IDS in this case) from being swapped/paged out to disk. Even though we're using the RESIDENT functionality in IDS for the buffer pool and it does appear as resident in "onstat -g seg", it's still part of the "cached" part of memory in Linux (seen from "free - m" or in "/proc/meminfo"). My understanding is that everything in the "cached" part of memory can get swapped if not accessed frequently enough and if other processes need memory allocated. Does the RESIDENT functionality actually do what it's supposed to on Linux (kernel 2.4 and 2.6) and if not, is there anyway of protecting the IDS shared memory (or just any shared memory segments) from being swapped? I couldn't anything on this in the machine notes for IDS on that OS. In all my searches, the "swappiness" kernel parameter came up but that parameter will affect any type of memory segments and not just the ones we are interested in. We have seen a box starting to use plenty of swap space when creating an L0 ontape backup o a 30+ GB instance, piping it through gzip and placing it on disk. With no way of protecting the memory pages allocated to the database server, these were probably put out to disk. Another way to prevent this from happening would perhaps be to limit the amount of memory that can be used for caching filesystems data. Does anyone know how to do this and does anyone have any experience with it? Is the best answer really "leave it as it is"? Any ideas? RoB Some outputs: onstat -g seg IBM Informix Dynamic Server Version 10.00.FC5 -- On-Line (Prim) -- Up 1 days 05:10:28 -- 5544624 Kbytes Segment Summary: id key addr size ovhd class blkused blkfree 917506 1381451777 b000000 3370270720 526768 R* 822817 3 <---- 3.14GB RESIDENT BUFFER POOL 950275 1381451778 d3e24000 2097152000 64736 V 315980 196020 983044 1381451779 150e24000 557056 760 M 135 1 1015824 1381451780 150eac000 209715200 7136 V 11721 39479 Total: - - 5677694976 - - 1150653 235503 (* segment locked in memory) cat /proc/meminfo; free -m MemTotal: 10073484 kB <---- TOTAL AVAILABLE MEMORY MemFree: 81276 kB Buffers: 50364 kB Cached: 9608560 kB <---- THIS IS MUCH TO LARGE NOT TO INCLUDE THE BUFFER POOL! SwapCached: 408 kB Active: 5497940 kB Inactive: 4277888 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 10073484 kB LowFree: 81276 kB SwapTotal: 10482372 kB SwapFree: 10468780 kB Dirty: 552 kB Writeback: 0 kB Mapped: 5485448 kB Slab: 70928 kB CommitLimit: 15519112 kB Committed_AS: 6492512 kB PageTables: 93724 kB VmallocTotal: 536870911 kB VmallocUsed: 7516 kB VmallocChunk: 536862715 kB HugePages_Total: 0 HugePages_Free: 0 Hugepagesize: 2048 kB free -m total used free shared buffers cached Mem: 9837 9758 79 0 49 9383 -/+ buffers/cache: 325 9511 Swap: 10236 13 10223 cat /proc/sys/vm/swappiness 60

RoB wrote: > Hi all! > > IDS 10.00.FC5 > Red Hat Enterprise Linux AS release 4 (Nahant Update 4) > (2.6.9-42.ELsmp x86_64) > > I'm wondering if anyone knows of any best practise on how to configure > memory handling on Linux 2.6 kernel for a *dedicated* IDS box. > > On other operating systems there are ways to protect the shared memory > segments allocated to applications (IDS in this case) from being > swapped/paged out to disk. > Even though we're using the RESIDENT functionality in IDS for the > buffer pool and it does appear as resident in "onstat -g seg", it's > still part of the "cached" part of memory in Linux (seen from "free - > m" or in "/proc/meminfo"). My understanding is that everything in the > "cached" part of memory can get swapped if not accessed frequently > enough and if other processes need memory allocated. > > Does the RESIDENT functionality actually do what it's supposed to on > Linux (kernel 2.4 and 2.6) and if not, is there anyway of protecting > the IDS shared memory (or just any shared memory segments) from being > swapped? I couldn't anything on this in the machine notes for IDS on > that OS. I can't answer your questions without spending a lot of time on it so here's one practical suggestion. Setting the resident value to -1 will flag the virtual portion of shared memory as resident as well as the buffer pool. See if this helps you. Regards, Ben.

Rob, To answer your question... Memory Residency is an OS thing. Informix has a flag/switch that if set, it will attempt to perform memory residency *if* and I mean *iff* the underlying OS does support this feature. Going from memory... mlock() is the c function which will lock a set of pages in memory. (Ie forced residency) Not sure if Linux does support this, however it appears that it does. But back to your question... you have a linux box *dedicated* to IDS. If its dedicated, then there's nothing else that would cause IDS to swap out pages from main memory to its system cache. So you really don't need forced residency. If you do have other things but you are really using your IDS, then it will stay in memory. (Caching is based on a least used algorithm.) Look at it this way... If your system is constantly busy with IDS, then its pages will be in memory and not cache. If your system isn't using IDS but has other things going on, then it should swap out IDS pages to cache. Keeping those pages locked effectively will reduce the memory available to your system with no real performance gain. So the bottom line, its a *feature* that really has minimal value. >From: "RoB" <plumas4u@yahoo.com> >Hi all! > >IDS 10.00.FC5 >Red Hat Enterprise Linux AS release 4 (Nahant Update 4) >(2.6.9-42.ELsmp x86_64) > >I'm wondering if anyone knows of any best practise on how to configure >memory handling on Linux 2.6 kernel for a *dedicated* IDS box. > >On other operating systems there are ways to protect the shared memory >segments allocated to applications (IDS in this case) from being >swapped/paged out to disk. >Even though we're using the RESIDENT functionality in IDS for the >buffer pool and it does appear as resident in "onstat -g seg", it's >still part of the "cached" part of memory in Linux (seen from "free - >m" or in "/proc/meminfo"). My understanding is that everything in the >"cached" part of memory can get swapped if not accessed frequently >enough and if other processes need memory allocated. > >Does the RESIDENT functionality actually do what it's supposed to on >Linux (kernel 2.4 and 2.6) and if not, is there anyway of protecting >the IDS shared memory (or just any shared memory segments) from being >swapped? I couldn't anything on this in the machine notes for IDS on >that OS. > >In all my searches, the "swappiness" kernel parameter came up but that >parameter will affect any type of memory segments and not just the >ones we are interested in. > > >We have seen a box starting to use plenty of swap space when creating >an L0 ontape backup o a 30+ GB instance, piping it through gzip and >placing it on disk. With no way of protecting the memory pages >allocated to the database server, these were probably put out to >disk. > >Another way to prevent this from happening would perhaps be to limit >the amount of memory that can be used for caching filesystems data. >Does anyone know how to do this and does anyone have any experience >with it? > >Is the best answer really "leave it as it is"? Any ideas? > >RoB > > >Some outputs: > >onstat -g seg> >IBM Informix Dynamic Server Version 10.00.FC5 -- On-Line (Prim) -- >Up 1 days 05:10:28 -- 5544624 Kbytes> >Segment Summary: >id key addr size ovhd class >blkused blkfree >917506 1381451777 b000000 3370270720 526768 R* >822817 3 <---- 3.14GB RESIDENT BUFFER POOL >950275 1381451778 d3e24000 2097152000 64736 V >315980 196020 >983044 1381451779 150e24000 557056 760 M >135 1 >1015824 1381451780 150eac000 209715200 7136 V >11721 39479 >Total: - - 5677694976 - - >1150653 235503 > > (* segment locked in memory) > >cat /proc/meminfo; free -m > >MemTotal: 10073484 kB <---- TOTAL AVAILABLE >MEMORY >MemFree: 81276 kB >Buffers: 50364 kB >Cached: 9608560 kB <---- THIS IS MUCH TO >LARGE NOT TO INCLUDE THE BUFFER POOL! >SwapCached: 408 kB >Active: 5497940 kB >Inactive: 4277888 kB >HighTotal: 0 kB >HighFree: 0 kB >LowTotal: 10073484 kB >LowFree: 81276 kB >SwapTotal: 10482372 kB >SwapFree: 10468780 kB >Dirty: 552 kB >Writeback: 0 kB >Mapped: 5485448 kB >Slab: 70928 kB >CommitLimit: 15519112 kB >Committed_AS: 6492512 kB >PageTables: 93724 kB >VmallocTotal: 536870911 kB >VmallocUsed: 7516 kB >VmallocChunk: 536862715 kB >HugePages_Total: 0 >HugePages_Free: 0 >Hugepagesize: 2048 kB > >free -m > total used free shared buffers >cached >Mem: 9837 9758 79 0 49 >9383 >-/+ buffers/cache: 325 9511 >Swap: 10236 13 10223 > >cat /proc/sys/vm/swappiness >60 > >_______________________________________________ >Informix-list mailing list >Informix-list@iiug.org >http://www.iiug.org/mailman/listinfo/informix-list _________________________________________________________________ Don't miss your chance to WIN 10 hours of private jet travel from Microsoft' Office Live http://clk.atdmt.com/MRT/go/mcrssaub0540002499mrt/direct/01/

-->>So the bottom line, its a *feature* that really has minimal value. do not understimate this!!!!!!!!!!!! linux may kick out pages etc due to filesystem cache!!!!!!!!! Superboer.

On Mar 1, 7:59 am, "Superboer" <superbo...@t-online.de> wrote: > -->>So the bottom line, its a *feature* that really has minimal > value. > > do not understimate this!!!!!!!!!!!! linux may kick out pages etc due > to filesystem cache!!!!!!!!! > > Superboer. I agree completely with Superboer as performing a L0 backup to disk makes the box swap and probably swapped out memory allocated to the server. The backup image will be a very large file (compared to the size of total physical memory) sitting on a filesystem and that will be partly or completely cached in memory and cause havoc if the file itself is large. This will produce a really cr@ppy performance for the database server if the OS has to swap in any type of memory that the instance wants to access. To answer Ian's post: backups will be performed on dedicated database server and they are sometimes performed to files on filesystems, just as in this case. Using RESIDENT for all memory segments (i.e. set to -1) could be tested but I just wanted to hear if anyone actually *knows* that the RESIDENT functionality really pins the memory segments in the physical memory, I cannot find any documentation stating that it will. So as Superboer states: "This is not a thing to underestimate!" RoB

Ok, look at it this way... Informix memory residency will use memlock() if its supported by the OS. If its there. Fine. If not. Oh well. Now did RHat implement memlock correctly? I don't know. Is this a make or break issue on performance? No. Not really. Now that's a loaded statement. Going back to the OP, he's building a dedicated IDS slab. Ok... does he mean nothing but basic OS functions and IDS or is he running something else? (Shut down IDS and then run top -s1 command to see the load and what else is running.) How much memory does the slab have? (Note: Memory requirements will vary based on intended use of IDS.) What sort of disks does the slab have? SATA, PATA, SCSI, or SAS? Any raid? So you see, there are a couple of design factors which will have a serious impact on the amount of swapping and the speed of the swap. (Memory type/system architecture/ disk type all will impact this number). So going back to the initial point. If you force memory residency, then your system must function with less actual memory. If you don't, then if IDS is in use, it will stay resident. Remember least used pages get cached. If you don't force memory residency, and IDS is hardly used and there is other activity on the system, then you will swap and the delay of the swap is going to impact the first function. Then the page is in memory, and the next function will run without that initial delay. AGAIN NOTE: The delay is going to be based on the hardware choices made. Make and type of mother board will drive which type of memory you can use, which type of hard drive controller and hard drives you use. Raid or not Raid. (Or rather do you swap to a disk that is part of an LVM or raid volume) Bottom line. its pretty much a useless feature. BTW, pages get kicked out based on least used and last used queue. So if you're not using Informix, then it should be kicked out. >From: "Superboer" <superboer7@t-online.de> >-->>So the bottom line, its a *feature* that really has minimal >value. > >do not understimate this!!!!!!!!!!!! linux may kick out pages etc due >to filesystem cache!!!!!!!!! > > >Superboer. > >_______________________________________________ >Informix-list mailing list >Informix-list@iiug.org >http://www.iiug.org/mailman/listinfo/informix-list _________________________________________________________________ Play Flexicon: the crossword game that feeds your brain. PLAY now for FREE.' http://zone.msn.com/en/flexicon/default.htm?icid=flexicon_hmtagline

If Informix acts the way you say, either one of two things. 1) You don't have enough memory. 2) Informix was poorly designed. The size of the level 0 back up should not effect the amount of memory being used to the point that it will swap out the IDS engine. What you're implying is that during the IO process, you're filling up all the pages in memory and forcing the system to swap as it writes to disk. That should not or never be the case if you have enough memory for the system the way you've tuned your system. Note that you can see a slowdown of performance due to I/O issues as you read from disk and write to tape. If you're I/O bound, that will have a negative hit on performance. Does that make sense? >From: "RoB" <plumas4u@yahoo.com> 0A] > >On Mar 1, 7:59 am, "Superboer" <superbo...@t-online.de> wrote: > > -->>So the bottom line, its a *feature* that really has minimal > > value. > > > > do not understimate this!!!!!!!!!!!! linux may kick out pages etc due > > to filesystem cache!!!!!!!!! > > > > Superboer. > >I agree completely with Superboer as performing a L0 backup to disk >makes the box swap and probably swapped out memory allocated to the >server. The backup image will be a very large file (compared to the >size of total physical memory) sitting on a filesystem and that will >be partly or completely cached in memory and cause havoc if the file >itself is large. This will produce a really cr@ppy performance for the >database server if the OS has to swap in any type of memory that the >instance wants to access. > >To answer Ian's post: backups will be performed on dedicated database >server and they are sometimes performed to files on filesystems, just >as in this case. > >Using RESIDENT for all memory segments (i.e. set to -1) could be >tested but I just wanted to hear if anyone actually *knows* that the >RESIDENT functionality really pins the memory segments in the physical >memory, I cannot find any documentation stating that it will. > >So as Superboer states: "This is not a thing to underestimate!" > >RoB > >_______________________________________________ >Informix-list mailing list >Informix-list@iiug.org >http://www.iiug.org/mailman/listinfo/informix-list _________________________________________________________________ Rates near 39yr lows! $430K Loan for $1,399/mo - Paying Too Much? Calculate new payment http://www.lowermybills.com/lre/index.jsp?sourceid=lmb-9632-18226&moid=7581