Check if buffers are in swap?

Hello, maybe last question of 2023 :-) I noticed slow downs during last week in our single instance on RHEL8 - i suspect storage (SAN) speed slowdowns, but not totally sure. Also maybe swap oocured with oninit processes ? How can i check to verify this ? Thanks and good start for 2024! ------------------------------ Marc Demhartner ------------------------------

Look at the memfree and swaptotal versus swapfree lines in: $ cat /proc/meminfo ------------------------------Art S. Kagel, President and Principal Consultant ASK Database Management Corp. www.askdbmgt.com ------------------------------

Hello, you can find swap usage by process with (as root ofc): for x in /proc/*/status; do grep -q VmSwap $x && awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $x ; done | sort -k 2 -n -r | less Informix processes are oninit so all of those should have 0 usage. HTH Hrvoje ------------------------------ Hrvoje Zokovic ------------------------------

Thanks. Looks like its the case.. i have many entries there with oninit 4784 KB How to prevent oninit to get swapped? ------------------------------ Marc Demhartner ------------------------------

Marc, Is vmstat showing any swapping activity? ------------------------------ Sincerely, Dennis ------------------------------

It's typically not oninit process memory that's getting swapped, but rather allocated shared memory, and this you can protect from swapping by setting it resident - this is the purpose of the RESIDENT onconfig parameter. HTH, Andreas ------------------------------ Andreas Legner ------------------------------

Yes, RESIDENT 1 is set. I also think buffers were in swap. I deactiaved swap totally and its even faster as before now. Not much, but a bit. Problem is solved, but strange situation. ------------------------------ Marc Demhartner ------------------------------

Marc: You should be setting RESIDENT to -1. Setting it to '1' only sets the initial resident segment and the first virtual segment to be non-swappable while the message and buffer cache segments will still be swappable. Setting RESIDENT to '-1' will set all shared memory segments to non-swabbable including buffer cache segments and any added virtual segments (message segments, unfortunately, are not marked as non-swappable which can slow local shared memory connections). Art ------------------------------ Art S. Kagel, President and Principal Consultant ASK Database Management Corp. www.askdbmgt.com ------------------------------

Art - I have a client whose boxes end up with "out of swap" condition(s) often. Haven't tracked down (yet) what's in swap, but realized they have RESIDENT = 0 on all engines. Changing that and bouncing the engine: WARNING: Shared-memory residency is enabled while direct I/O and KAIO (kernel asynchronous I/O) are active. This configuration could lead to runtime KAIO errors, which might shut down the instance or set chunks offline. Set the RESIDENT configuration parameter to 0 to turn off shared-memory residency, and then restart the database server. Refer to the machine notes and APAR IC76872 for more information. Platform is AIX 7.2, IDS 12.10.FC12W1. Thanks! Mark ------------------------------ Thanks! Mark Scranton The Mark Scranton Group mark@markscranton.com ------------------------------

Hi Mark, as you might guess from https://www.ibm.com/support/pages/apar/IC76872, this can result, on AIX, from Informix resident segments competing with OS KAIO for 'pinnable' memory, i.e. memory that can be fixed in RAM (I think it mainly needs to match certain alignment criteria). Since Informix has no way to anticipate those pinnable memory resources (there might be other Informix instances, other shared memory users, other activities using pinned memory, etc....), it warns the user that ... bad things have already happened due to running out of this resource (AIX apparently doing a sub-optimal job at making sure it reserves was it needs itself). If, on the other hand, you're confident your AIX lpar always has sufficient memory reserves, you should be safe using Informix RESIDENT setting. Also, remember this isn't a boolean (ON or OFF), but you always can say "this many shmem segments resident, please". Then, I also remember having been told that AIX is good enough at memory management and won't even need setting shared memory segments resident - but the experience you're citing seems to indicate to the contrary? Lastly, maybe going for large/huge memory pages would avoid this whole discussion (as such memory would be exempt from swapping anyway.) And very lastly: you haven't even indicated that the swapping incurred by those systems affects Informix (performance wise) or even occurs to Informix shared memory? What else could be consuming memory? What exactly is incurring "out of swap"? BR, Andreas ------------------------------ Andreas Legner ------------------------------

Mark: You have a quandary. AIX is based on BSD code rather than System V code. So AIX's handling of swap is different from System V based memory management like that in Solaris and even Linux. It does not wait for physical memory to be used up before it begins to swap segments out to the swap device(s). AIX does not perform particularly better with non-swappable memory segments, however, because of how it does memory management. however, swapping can be aggressive and so making Informix's memory "resident" is recommended. The alternative would be to increase the amount of swap space to that it can handle all of shared memory plus private application memory without running low/out. That all said, I agree with Andreas, if you know that you have enough physical memory on the box that it should not need to swap. Art ------------------------------ Art S. Kagel, President and Principal Consultant ASK Database Management Corp. www.askdbmgt.com ------------------------------

Andres/Art - Thanks guys. And yes Andres - I need to start gathering (again) what's in swap when it's being consumed. This doesn't happen in prod much but more frequently in test/dev ... and you know how we typically handle those boxes ... "when we get around to it." But this has been a prod issue years ago, and mostly just bugs me and I'm curious for future use perhaps. I've built some sniffers for swap - now I gotta do some collection/analysis. Thanks for the detailed responses guys - always appreciated. Mark ------------------------------ Thanks! Mark Scranton The Mark Scranton Group mark@markscranton.com ------------------------------

Even when setting it to RESIDENT -1 on RHEL8 it keeps swapping oninit proccesses oninit 5084 kB oninit 5076 kB oninit 5068 kB oninit 5056 kB oninit 5056 kB oninit 5056 kB oninit 5048 kB oninit 5040 kB oninit 5028 kB oninit 5008 kB oninit 5008 kB oninit 4976 kB oninit 4976 kB oninit 4972 kB oninit 4964 kB oninit 4964 kB oninit 4960 kB oninit 4956 kB oninit 4948 kB oninit 4948 kB ------------------------------ Marc Demhartner ------------------------------

Linux has a swap management system that push unused memory segments to swap even if it's not required to allow more memory for disk buffers, etc. Open the file /etc/sysctl.conf in a text editor and enter the vm.swappiness parameter of your choice. Example: vm.swappiness = 0 Restart the operating system to load the changes.