Hugepages

| view attached (8) Hello, I'm using Informix as a container into a kubernetes cluster. I use an IBM cloud K8S cluster and build the Informix container with the offical scripts ( https://github.com/informix/informix-server-dockerfiles ). All works well but I would like to enable hugepage to get better performances. I have followed the IBM cloud documentation to enable hugepages on the node supporting Informix ( https://cloud.ibm.com/docs/containers?topic=containers-kernel and https://kubernetes.io/docs/tasks/manage-hugepages/scheduling-hugepages/ ). The yaml scripts used to launch Informix on the k8s cluster are attached to this post. The Informix pod is launched but the Informix instance refuse to start and don't show any particular issue. Without the hugepages, with the same configuration, Informix works well. I attached to this post the following informations comming from the pod: envvar.txt : the environment variables; meminfo.txt : the content of "/proc/meminfo"; online.log.txt : the Informix log; oninit.txt : the output of the "oninit -ivy" command; onconfig.txt : the content of the "onconfig" file; ipcs.txt : the result of the "ipcs" command. It seems that Informix correcly see the hugepages, try to use it but has a problem to allocate shared memory. However, I have to little traces to understand the root cause of the problem. I also tried the same configuration on my laptop with microk8s cluster and all is working fine: Informix is able to start and uses the hugepages. Could you please help me? Is it possible to enable more debug traces on Informix? Thanks a lot for your help. Regards, Renaud ------------------------------ Renaud Demarneffe ------------------------------ #Informix

Anything in onstat –m Cheers Paul

Did you shut down properly, before attempting to restart? This is your (current) problem: 19:58:58 Requested shared memory segment size rounded from 8996KB to 10240KB 19:58:58 Shared memory segment will use huge pages. 19:58:58 shmget: [EEXIST][17]: key 52564801: shared memory already exists A simple 'onmode -ky' might remove this leftover segment (and others). Doesn't immediately look like a huge pages problem, yet might be the result of your real problem So I'd suggest a retry after removing those existing segments, then re-posting online.log. ------------------------------ Andreas Legner ------------------------------

| view attached Hi Paul, Here is the output of "onstat -m". For me, there is no more information. Regards, Renaud ------------------------------ Renaud Demarneffe ------------------------------

Does this mean there was no output ?

| view attached (2) Hi Andreas, Here is the new online.log after removing the shm segments. The problem is still there. The shm segment appears after the failed "oninit". Regards, Renaud ------------------------------ Renaud Demarneffe ------------------------------

No, we have only this. informix@informixdb-hp-0:~$ onstat -m Your evaluation license will expire on 2019-12-15 11:32:57 shared memory not initialized for INFORMIXSERVER 'informix' Message Log File: /opt/ibm/data/logs/online.log 21:24:49 Parameter's user-configured value was adjusted. (VPCLASS) 21:24:49 IBM Informix Dynamic Server Started. 21:24:49 Requested shared memory segment size rounded from 8996KB to 10240KB 21:24:49 Shared memory segment will use huge pages. informix@informixdb-hp-0:~$ ------------------------------ Renaud Demarneffe ------------------------------

Hi Renaud, there should be many more messages in the online.log if the server didn't start. Are you sure that the server isn't running. Please have a look on your oninit processes. What does onstat without any parameter say? It is possible you get the warning from oninit (WARNING: server initialization failed or timed out.) even though the server is running. We experience this frequently due to a long lasting check of thousands of chunks during startup of the server. Regards, Reinhard. ------------------------------ ARZ Emmendingen Support ------------------------------

Hi Reinhard, Sorry for the late response. I retested the solution on a new k8s node with a new pod. I got the same error and I have no more information on the online.log. I let the pod start for more than 1 hour before reading the logs and executing the onstat cmd. Here is the content of the online.log file: informix@informixdb-hp-0:~$ cat /opt/ibm/data/logs/online.log 21:24:49 Parameter's user-configured value was adjusted. (VPCLASS) 21:24:49 IBM Informix Dynamic Server Started. 21:24:49 Requested shared memory segment size rounded from 8996KB to 10240KB 21:24:49 Shared memory segment will use huge pages. 15:54:22 Parameter's user-configured value was adjusted. (VPCLASS) 15:54:22 IBM Informix Dynamic Server Started. 15:54:22 Requested shared memory segment size rounded from 8996KB to 10240KB 15:54:22 Shared memory segment will use huge pages. informix@informixdb-hp-0:~$The output of the onstat command is the following: informix@informixdb-hp-0:~$ onstat Your evaluation license will expire on 2019-12-15 11:32:57 shared memory not initialized for INFORMIXSERVER 'informix' informix@informixdb-hp-0:~$ I also checked with "ps -aux" the state of the "oninit" process and it's defunct. informix@informixdb-hp-0:~$ ps -aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND informix 1 0.0 0.0 4084 692 ? Ss Nov07 0:00 /opt/ibm/scripts/dinit /opt/ibm/scripts/informix_entry.sh 0x73 /opt/ibm/scripts/informix_stop.sh informix 8 0.0 0.0 4340 812 ? S Nov07 0:00 sh -c /opt/ibm/scripts/informix_entry.sh 0x73 informix 9 0.0 0.0 20620 3512 ? S Nov07 0:00 /bin/bash /opt/ibm/scripts/informix_entry.sh 0x73 root 225 0.0 0.0 0 0 ? Z Nov07 0:00 [oninit] < defunct > informix 305 0.0 0.0 4276 760 ? S Nov07 0:03 tail -f /opt/ibm/data/logs/online.log informix 318 0.0 0.0 20288 3296 pts/0 Ss 16:48 0:00 /bin/bash informix 335 0.0 0.0 17508 2132 pts/0 R+ 16:50 0:00 ps -aux informix@informixdb-hp-0:~$ I don't understand why Informix got into timeout and why the process is killed. Regards, Renaud ------------------------------ Renaud Demarneffe ------------------------------

This is what this normally should be looking like: 11/05/19 17:19:17 Requested shared memory segment size rounded from 2302208KB to 3145728KB 11/05/19 17:19:17 Shared memory segment will use huge pages. 11/05/19 17:19:17 Segment locked: addr=0x140000000, size=3221225472 11/05/19 17:19:17 Successfully added a bufferpool of page size 2K. 11/05/19 17:19:17 Requested shared memory segment size rounded from 2148608KB to 3145728KB 11/05/19 17:19:17 Shared memory segment will use huge pages. 11/05/19 17:19:18 Segment locked: addr=0x200000000, size=3221225472 11/05/19 17:19:18 Successfully added a bufferpool of page size 8K. So are you saying, above, that you're missing the "Segment locked" and "Successfully added" messages and nothing more gets written to the message log? Huge pages memory allocation can take a while, yet it this doesn't move at all, something might be wrong on OS level? If this really is just hanging there, can you show us 'ipcs -m' output at this stage? can you retry using strace, as user root: strace -o /tmp/oninit.trc oninit then post the oninit.trc file? ------------------------------ Andreas Legner ------------------------------

| view attached Hi Andreas, Yes, I'm missing the "Segment locked" and "Successfully added" messages and nothing more gets written to the message log. The output of the "ipcs -m" command, after the oninit hangup is: informix@informixdb-hp-0:~$ ipcs -m ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x52564801 262144 root 660 10485760 0 The oninit.trc is attached to the post. As the Informix is installed as a non-root installation (executed by informix user), I need to hack in order to attach to the right process as soon as it started ( oninit -ivwy >> $INIT_LOG & oninitpid=$(pidof oninit) ; sudo strace -s 65536 -o /tmp/oninit.trc -p $oninitpid ).Thanks for your help. Regards, Renaud ------------------------------ Renaud Demarneffe ------------------------------

Huge pages and a non-root install doesn't work - or I have never been able to get it to work, change to a roof installation with no config changes and it should just work Cheers Paul Paul Watson Oninit LLC +1-913-387-7529 www.oninit.com Oninit® is a registered trademark of Oninit LLC

| view attached (3) Hi Paul, I made a mistake into my previous response. The Informix instance is a root one. After some investigations, I think the problem comes from the K8S cluster that does not allow access to huge pages even if it mount it into the pod. Indeed, the same docker running on my PC with a microk8s cluster is working while it is not running on an IKS cluster (public IBM K8S cluster). I created a docker with a small program that tests access to huge pages via shm (shmget function). This docker runs smoothly on microk8s but not on IKS. On IKS, it returns the following error: --- SIGBUS {si_signo=SIGBUS, si_code=BUS_ADRERR, si_addr=0x7f7f71400000} --- +++ killed by SIGBUS (core dumped) +++ The files for this test are attached. The error of this program is the same as the Informix one. I think the problem comes from this sentence that is in the Kubernetes documentation: Applications that consume huge pages via shmget() with SHM_HUGETLB must run with a supplemental group that matches proc/sys/vm/hugetlb_shm_group. Or comes from a security option of the IKS. But I don't understand how to solve it: * I'm running the "hugepage-shm" program as root user (uid=0, gid=0); * The content of "/proc/sys/vm/hugetlb_shm_group" is "0". Maybe I need to adapt the pod's scuritycontext or apply a particular Pod Security Policy but I do not see which one? I opened a support ticket on IBM. Let's see what they will say. Regards, Renaud ------------------------------ Renaud Demarneffe ------------------------------

Support ticket directed to which part of IBM? I guess not Informix support? Could you also post output of your test prog when it fails? I'm wondering where exactly it fails - with SIGBUS? ------------------------------ Andreas Legner ------------------------------

| view attached The ticket is directed to Kubernetes (Containers) support . The error is when the program tries to write into the huge page. I join the strace output. ------------------------------ Renaud Demarneffe ------------------------------

Hello, I found the solution. In addition to configuring the worker node to create the huge pages in the kernel, I have to modify the cgroup permissions to allow the pod to write in the huge pages. This is done into the daemonset applied on the worker node (file worker-node-kernel-settings.yaml.txt) where the command to apply on the worker node is now: sysctl -w vm.nr_hugepages=800; echo 9223372036854771712 | tee /sys/fs/cgroup/hugetlb/kubepods/hugetlb.2MB.limit_in_bytes; ------------------------------ Renaud Demarneffe ------------------------------