IFX_LARGE_PAGES experiences?

According to the documentation, IFX_LARGE_PAGES can have " significant performance benefits in large memory configurations". What is considered "large" here? We have around 400GB shared memory, but some small testing did not show any gain. Is this still too small to have a benefit? Are anyone using this? Experiences? ------------------------------ Øyvind Gjerstad Developer/Architect PostNord AS ------------------------------ #Informix

Hi Øyvind, I wrote this seven years ago: https://informixdba.wordpress.com/2013/03/08/huge-pages/ One clear benefit is that the size of the operating system's page table will be significantly smaller. I would say 400 GB qualifies as large and you don't say how big your buffer pools are which can also go in huge pages. Most systems use Workgroup Edition which limits the size of an instance well below this figure (still only 32 GB with 14.10, 16 GB in earlier editions). In my view huge pages work well. The only things to be aware of are: * Instances leaving memory segments behind which means the next time you start up there are insufficient pages and in some situations it is possible for the instance to start using standard pages, which is bad when you've allocated huge pages to it. (A similar problem exists without huge pages too.) * Dynamically allocated segments cannot take advantage of huge pages. Ben. ------------------------------ Benjamin Thompson ------------------------------

Hi Øyvind. What platform are you running on? We have around 700GB of shared memory and it really makes a difference. Have you enabled HugePages at the OS level (cat /proc/meminfo)? When starting the engine you should see in online.log: 5/20/20 22:55:08 Shared memory segment will use huge pages. 05/20/20 22:55:08 Segment locked: addr=0x3a5000000, size=715290312704 05/20/20 22:55:15 Successfully added a bufferpool of page size 8K. Best regards, -Snorri ------------------------------ Snorri Bergmann ------------------------------

Yes, I read this article already. It was good, but didn't provide any insight into what sizes are considered "large". ------------------------------ Øyvind Gjerstad Developer/Architect PostNord AS ------------------------------

We are on Linux (was RHEL6 during testing, is RHEL7 now). Hugepages was enabled at OS level, yes. And it was logged during startup as well. We had 12.10 at the time, are on 14.10 now, but I guess that shouldn't make any difference. What difference do you notice? Less system time used (less overhead in kernel), or less user time by CP VPs? ------------------------------ Øyvind Gjerstad Developer/Architect PostNord AS ------------------------------

We did not do any extensive testing, except running some heavy queries. The queries run faster (can't recall how much, > 5 years ago) with HugePages enabled. CPU load on the VP's was around 100% for each CPU VP so the engine did more useful work with HugePages. Regs, -S ------------------------------ Snorri Bergmann ------------------------------

Hi Øyvind, the large pages in most Informix deployments is like icing on the cake, you have to get other areas tuned, after that you'll see most likely 10% or so improvement from using "large" (on Linux "huge") pages (and maybe face couple of interesting issues - if you also bind Informix to specific node(s) for the best affinity). As other folks already mentioned, the savings mostly come from better translation of the (operating system's) virtual memory address to physical, which uses table (called TLB or similar) of limited size (note that size is hardware-specific). When application (Informix) is using default operating system pages, usually 4k on Linux (should not be confused with Informix configurable page size) - the translation table is likely to overflow around / after few GB of RAM, so having 400GB on the system (and using fair portion of it for Informix) has a good chance of seeing benefit from using large pages. The Informix for Linux has IFX_LARGE_PAGES enabled by defaults, as long as you configured enough of them (on Linux the size of a huge page is usually 2GB), and, I think another requirement is for Informix to have RESIDENT flag set - they will be used. Here is a simple example using system with 128GB of RAM and Informix total memory footprint not exceeding 100GB: -- with Informix stopped -- # cat /proc/meminfoMemTotal: 131764620 kB MemFree: 127360576 kB # sysctl -w vm.nr_hugepages=50000 # cat /proc/meminfoMemTotal: 131764620 kB MemFree: 28652856 kB .... HugePages_Total: 50000 HugePages_Free: 50000 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Note that 100GB are now not available for general use (MemFree) - so if other running apps needed (now or in future) more than (MemTotal - allocated.LP.size) - it'll cause performance issues. The remaining question - is it enough for Informix to see performance benefit from using large pages? Unfortunately the answer is no. One of the additional requirements is otherwise tuned and CPU bound workload. Another is workload type and the size of the "working set". While Informix may allocate/reserve tens or hundreds GBs of RAM - it's possible that only small portion of it is ACTIVELY used. If the "working set" size is small enough to not cause TLB misses - no effect will be seen from using large pages. The "best" workload for seeing effect of the large pages is the TPC-C benchmark, since it accesses data in UNIFORM manner. However even there there will be dependency from the workload pattern. As an example here is the data from several sets of (CPU-bound) OLTP benchmarks (TPC-C like) where each set uses different number of active client sessions, from 1 to 1200, number of sessions is the first column: default large CL 4k - page 2G - page diff % -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - 1 56991.38 60144.81 5.5 % 2 109120.38 114728.69 5.1 % 3 156527.69 165304.43 5.6 % 4 203524.18 216079.75 6.2 % 5 249224.44 266030.82 6.7 % 6 290745.63 311187.94 7.0 % 8 364152.18 393351.18 8.0 % 10 403662.70 441657.76 9.4 % 12 438587.99 479961.70 9.4 % 14 468131.94 520049.46 11.1 % 16 501562.53 560274.93 11.7 % 20 502320.43 561788.31 11.8 % 24 503480.26 565137.04 12.2 % 28 503165.67 568734.12 13.0 % 32 501205.86 568941.13 13.5 % 40 499063.01 567008.11 13.6 % 48 496270.85 564567.48 13.8 % 56 492001.89 561357.70 14.1 % 64 487212.29 557250.38 14.4 % 80 476395.25 547373.80 14.9 % 100 464257.84 535203.78 15.3 % 200 440156.05 508607.34 15.6 % 400 433493.65 499677.93 15.3 % 800 420687.47 485471.21 15.4 % 1000 407470.80 476033.15 16.8 % 1200 397504.27 466980.94 17.5 % -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - Second column is the throughput (transactions per minute) with default (4k) pages, next column is the same but with large (2GB) pages, and last column is the difference in %. Working set size was about 32GB worth of pages (Informix data + index) and everything was fully cached in the bufferpool. One benefit that can be derived from this information (I hope) is the conclusion that if there is no difference with/without large pages - the configuration can be further tuned for performance... :-) Hope it helps a bit, --Vladimir ------------------------------ Vladimir Kolobrodov ------------------------------

I guess IBM has it's own notion on how to format "code segments" making them totally useless... Here is the table which will hopefully make more sense: default large CL 4k-page 2G-page diff% ------------------------------ 1 56991.38 60144.81 5.5% 2 109120.38 114728.69 5.1% 3 156527.69 165304.43 5.6% 4 203524.18 216079.75 6.2% 5 249224.44 266030.82 6.7% 6 290745.63 311187.94 7.0% 8 364152.18 393351.18 8.0% 10 403662.70 441657.76 9.4% 12 438587.99 479961.70 9.4% 14 468131.94 520049.46 11.1% 16 501562.53 560274.93 11.7% 20 502320.43 561788.31 11.8% 24 503480.26 565137.04 12.2% 28 503165.67 568734.12 13.0% 32 501205.86 568941.13 13.5% 40 499063.01 567008.11 13.6% 48 496270.85 564567.48 13.8% 56 492001.89 561357.70 14.1% 64 487212.29 557250.38 14.4% 80 476395.25 547373.80 14.9% 100 464257.84 535203.78 15.3% 200 440156.05 508607.34 15.6% 400 433493.65 499677.93 15.3% 800 420687.47 485471.21 15.4% 1000 407470.80 476033.15 16.8% 1200 397504.27 466980.94 17.5% ------------------------------ .... ------------------------------ Vladimir Kolobrodov ------------------------------