identify what is allocating memory

Hi , How identify what/who is consuming memory in a specific virtual memory segment ? With onstat -g ses , there is no session with high memory allocated. The amount of users sessions is keeping at regular average... No buffers pool was allocated automatically (I check with onstat -g buf) Not sure if how I doing is correct or accurate. I looking with 'onstat -g mem' , looking for the base of memory address and adding the totalsize column. Below is part of 'onstat -g seg' , with last segments . Where the engine was allocated at last 4 hours. I choose the "7000029f" to check with onstat -g mem , which have 512MB and almost no free blocks. =========== IBM Informix Dynamic Server Version 12.10.FC8W1 -- On-Line -- Up 51 days 12:42:34 -- 176212368 KbytesSegment Summary: id key addr size ovhd class blkused blkfree . . . 1048588 52574813 700002920000000 268435456 3147384 V 65514 22 1048587 52574814 700002930000000 268435456 3147384 V 65530 6 24 52574815 700002940000000 268435456 3147384 V 65514 22 25 52574816 700002950000000 268435456 3147384 V 65522 14 26 52574817 700002960000000 268435456 3147384 V 65516 20 27 52574818 700002970000000 536870912 6293160 V 131052 20 28 52574819 700002990000000 536870912 6293160 V 131035 37 29 5257481a 7000029b0000000 536870912 6293160 V 131040 32 30 5257481b 7000029d0000000 536870912 6293160 V 131020 52 31 5257481c 7000029f0000000 536870912 6293160 V 131008 64 32 5257481d 700002a10000000 536870912 6293160 V 130951 121 33 5257481e 700002a30000000 536870912 6293160 V 89679 41393 Total: - - 180441464832 - - 43985827 67265 =============== Running my script, where it execute 'onstat -g mem' , filter using grep and adding the 4 column of all output (Pool + Blkpool) Doesn't reach 100MB (73MB at the total) $ ifx.mem.sh 7000029f Pool Summary: - - - - - 0 MB 0 MB name class addr totalsize freesize #allocfrag #freefrag 0 MB 0 MB Blkpool Summary: - - - - - 0 MB 0 MB name class addr size #blks - - 0 MB 0 MB 4149288*O0 V 7000029f0bd1040 4096 768 1 1 0 MB 0 MB 4149352*O0 V 7000029fe1aa040 4096 768 1 1 0 MB 0 MB 4149370*O0 V 7000029f8ade040 4096 768 1 1 0 MB 0 MB ... 4150008 V 7000029f9bec040 2199552 60320 1765 62 43 MB 2 MB 4151843 V 7000029f2b9a040 2285568 95584 1766 50 46 MB 2 MB 4149410 V 7000029fb921040 2437120 79944 1820 82 48 MB 2 MB 4149299 V 7000029f643b040 2482176 42592 1796 61 50 MB 2 MB 4149357 V 7000029fe8fa040 3330048 82608 3000 65 53 MB 3 MB 4149544 V 7000029f54b0040 4034560 25760 3975 29 57 MB 4 MB 4149799 V 7000029f06b5040 6234112 17256 7376 23 63 MB 6 MB 4149484 V 7000029fcb7b040 10416128 155432 9314 119 73 MB 10 MB is correct sum this way ? they don't should match with 'onstat -g seg' ? filtering the base of memory address to match with the segments at 'onstat -g seg' ? the onstat -g mem doesn't show all entries which use the virtual memory ? (I've notice this version doesn't show the Buffer pool any more, but as I said before, there is no change at buffers)

Not easy to answer, but what is the reason for tracking a specific memory consumption ? I assume the shared mem is growing and you cannot find a reason for this. We had a similar problem with 12.10FC8W1. The instance was upgraded to FC8W2 and the issue is gone. Might be this bug: http://www-01.ibm.com/support/docview.wss?uid=swg27049673 IT19869 HUGE MEMORY LEAK IN MT POOL AT MTMUTEX Marcus Haarmann Von: "Cesar Martins" <cesar.inacio.martins@gmail.com> An: "ids" <ids@iiug.org> Gesendet: Freitag, 28. Juli 2017 16:02:05 Betreff: identify what is allocating memory [39635] Hi , How identify what/who is consuming memory in a specific virtual memory segment ? With onstat -g ses , there is no session with high memory allocated. The amount of users sessions is keeping at regular average... No buffers pool was allocated automatically (I check with onstat -g buf) Not sure if how I doing is correct or accurate. I looking with 'onstat -g mem' , looking for the base of memory address and adding the totalsize column. Below is part of 'onstat -g seg' , with last segments . Where the engine was allocated at last 4 hours. I choose the "7000029f" to check with onstat -g mem , which have 512MB and almost no free blocks. =========== IBM Informix Dynamic Server Version 12.10.FC8W1 -- On-Line -- Up 51 days 12:42:34 -- 176212368 KbytesSegment Summary: id key addr size ovhd class blkused blkfree .. . . 1048588 52574813 700002920000000 268435456 3147384 V 65514 22 1048587 52574814 700002930000000 268435456 3147384 V 65530 6 24 52574815 700002940000000 268435456 3147384 V 65514 22 25 52574816 700002950000000 268435456 3147384 V 65522 14 26 52574817 700002960000000 268435456 3147384 V 65516 20 27 52574818 700002970000000 536870912 6293160 V 131052 20 28 52574819 700002990000000 536870912 6293160 V 131035 37 29 5257481a 7000029b0000000 536870912 6293160 V 131040 32 30 5257481b 7000029d0000000 536870912 6293160 V 131020 52 31 5257481c 7000029f0000000 536870912 6293160 V 131008 64 32 5257481d 700002a10000000 536870912 6293160 V 130951 121 33 5257481e 700002a30000000 536870912 6293160 V 89679 41393 Total: - - 180441464832 - - 43985827 67265 =============== Running my script, where it execute 'onstat -g mem' , filter using grep and adding the 4 column of all output (Pool + Blkpool) Doesn't reach 100MB (73MB at the total) $ ifx.mem.sh 7000029f Pool Summary: - - - - - 0 MB 0 MB name class addr totalsize freesize #allocfrag #freefrag 0 MB 0 MB Blkpool Summary: - - - - - 0 MB 0 MB name class addr size #blks - - 0 MB 0 MB 4149288*O0 V 7000029f0bd1040 4096 768 1 1 0 MB 0 MB 4149352*O0 V 7000029fe1aa040 4096 768 1 1 0 MB 0 MB 4149370*O0 V 7000029f8ade040 4096 768 1 1 0 MB 0 MB .... 4150008 V 7000029f9bec040 2199552 60320 1765 62 43 MB 2 MB 4151843 V 7000029f2b9a040 2285568 95584 1766 50 46 MB 2 MB 4149410 V 7000029fb921040 2437120 79944 1820 82 48 MB 2 MB 4149299 V 7000029f643b040 2482176 42592 1796 61 50 MB 2 MB 4149357 V 7000029fe8fa040 3330048 82608 3000 65 53 MB 3 MB 4149544 V 7000029f54b0040 4034560 25760 3975 29 57 MB 4 MB 4149799 V 7000029f06b5040 6234112 17256 7376 23 63 MB 6 MB 4149484 V 7000029fcb7b040 10416128 155432 9314 119 73 MB 10 MB is correct sum this way ? they don't should match with 'onstat -g seg' ? filtering the base of memory address to match with the segments at 'onstat -g seg' ? the onstat -g mem doesn't show all entries which use the virtual memory ? (I've notice this version doesn't show the Buffer pool any more, but as I said before, there is no change at buffers) ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

original post: Hi , How identify what/who is consuming memory in a specific virtual memory segment ? With onstat -g ses , there is no session with high memory allocated. The amount of users sessions is keeping at regular average... No buffers pool was allocated automatically (I check with onstat -g buf) Not sure if how I doing is correct or accurate. I looking with 'onstat -g mem' , looking for the base of memory address and adding the totalsize column. Below is part of 'onstat -g seg' , with last segments . Where the engine was allocated at last 4 hours. I choose the "7000029f" to check with onstat -g mem , which have 512MB and almost no free blocks. <stuff removed> Response: Well, you can't do what you are trying with onstat -g mem to try and see what's using specific memory in a specific segment. The pool address from onstat -g seg is just the address of the pool header structure and thatstructure could be allocated in any piece of virtual memory. Then at that point, 4Kb blocks get added to the pool, and those 4Kb blocks could then come from any virtual segment. So there isn't really any relationship from the pool address reported in onstat -g mem to where the memory for that pool (so it's total size from the onstat -g mem) is coming from. You can track all the individual blocks allocated to a pool via onstat -g afr <pool name>. But there isn't a way to easily identify which pools are using blocks in a specific virtual segment. Jacques

> On 28 July 2017 at 15:21 JACQUES RENAUT <jacques.renaut@hcl.com> wrote: > > > original post: > > Hi , > > How identify what/who is consuming memory in a specific virtual memory > segment ? > With onstat -g ses , there is no session with high memory allocated. > The amount of users sessions is keeping at regular average... > No buffers pool was allocated automatically (I check with onstat -g buf) > > Not sure if how I doing is correct or accurate. > I looking with 'onstat -g mem' , looking for the base of memory address and > adding the totalsize column. > > Below is part of 'onstat -g seg' , with last segments . Where the engine > was allocated at last 4 hours. > > I choose the "7000029f" to check with onstat -g mem , which have 512MB and > almost no free blocks. > > <stuff removed> > > Response: > > Well, you can't do what you are trying with onstat -g mem to try and see > what's using specific memory in a specific segment. The pool address from > onstat -g seg is just the address of the pool header structure and that> structure could be allocated in any piece of virtual memory. Then at that > point, 4Kb blocks get added to the pool, and those 4Kb blocks could then come > from any virtual segment. So there isn't really any relationship from the pool > address reported in onstat -g mem to where the memory for that pool (so it's > total size from the onstat -g mem) is coming from. You can track all the > individual blocks allocated to a pool via onstat -g afr <pool name>. But there > isn't a way to easily identify which pools are using blocks in a specific > virtual segment. > > Jacques > onstat -g afr in includes the column "addr (hexadecimal) Memory address of thepool fragment". Does the memory address of the pool fragment tell you which virtual segment contains the fragment? Regards, David. https://www.ibm.com/support/knowledgecenter/en/SSGU8G_12.1.0/com.ibm.adref.doc/i ds_adr_0512.htm > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >

original post: <stuff removed> onstat -g afr in includes the column "addr (hexadecimal) Memory address of thepool fragment". Does the memory address of the pool fragment tell you which virtual segment contains the fragment? Regards, David. Response: Yes it does. If you look at onstat -g seg output, it includes the "addr" and "size" columns. The addr + size columns gives you the range of addresses that would be in any particular segment. So once you have the start and end address range, when you look at the 1st column in onstat -g afr (which is the addr column). If the address of that line in the onstat -g afr output is contained in the address range of the addr+size of the onstat -g seg output, then that block of memory from the afr output would be in that particular segment. So from a tiny instance I have up currently I have this in my onstat -g seg (it's only got 1 virtual segment but just for illustration) Segment Summary: id key addr size ovhd class blkused blkfree 73531403 525f4801 44000000 911104 495784 R 1199 0 73564177 525f4802 444af000 135839744 1593528 V 10462 22702 73596946 525f4803 4c63b000 113287168 1 B 27658 0 So my 1 virtual segment range would be from 0x444af000 - 0x4c63b000 (which you can see it is contiguous with the following Buffer pool segment that starts at 0x4c63b000...this is normally what you would see, but on some platforms there can be gaps between the segments reported in onstat -g seg). So then from an onstat -g afr on a random pool we have Allocations for pool name rascron: addr size memid fileid location 46635000 3328 overhead 301 mtshpool.c:606 46635d00 232 TRACE 16 ph_collector.c:428 46635de8 152 tQ 3974 rsbufq.c:166 so looking at the 1st column the addr 0x46635000 is an address that would fall in the range of 0x444af000 - 0x4c63b000, so that 3328 bytes of memory (or what I refer to generally as a block or the doc is referring to as pool fragment) would be in the virtual segment who's seg id was "73564177". Jacques

@Marcus , The reason to try identify who is using determined segment is because the engine just start allocate segments and do not stop. Itried identify who needed all this memory. I check the defect that you mentioned and I suspect could be it , because the mt pool was ~2GB of total size. (this do not match with +9GB segments allocated , but since we are talking about a defect, always consider everything is possible here) So, we already delayed our upgrade from FC8W1 to FC9 few days ago. With this situation , we should probably need to bounce our instance soon or it would run out of memory, so I've stopped it and upgrade to FC9 . @Jacques , Thank you for your explanation. Please, can you answer my question below? If I get the sum of the column totalsize of all lines on 'onstat -g mem' (Pool+blkpool), should be close value of the used memory for the segments at 'onstat -g seg' ? (respecting the type of the segment, vitual, reserved...) Ask that because the situation I had here the sum of everything at 'onstat -g mem' was close of 8.3GB and the used segments at 'onstat -g seg' was ~16G (blkused*4096). Which is to huge discrepancy... I open an PMR for this, but unfortunately we do not have time enough to wait a feedback to collect more information and decide bounce and upgrade the instance since we was into a window for maintanece where this will be allowed. Regards Cesar 2017-07-28 12:09 GMT-03:00 JACQUES RENAUT <jacques.renaut@hcl.com>: > original post: > > <stuff removed> > > onstat -g afr in includes the column "addr (hexadecimal) Memory address of> the > pool fragment". > > Does the memory address of the pool fragment tell you which virtual segment > contains the fragment? > > Regards, > David. > > Response: > > Yes it does. If you look at onstat -g seg output, it includes the "addr" > and > "size" columns. The addr + size columns gives you the range of addresses > that > would be in any particular segment. So once you have the start and end > address > range, when you look at the 1st column in onstat -g afr (which is the addr > column). If the address of that line in the onstat -g afr output is > contained > in the address range of the addr+size of the onstat -g seg output, then > that > block of memory from the afr output would be in that particular segment. > > So from a tiny instance I have up currently I have this in my onstat -g seg > (it's only got 1 virtual segment but just for illustration) > > Segment Summary: > id key addr size ovhd class blkused blkfree > 73531403 525f4801 44000000 911104 495784 R 1199 0 > 73564177 525f4802 444af000 135839744 1593528 V 10462 22702 > 73596946 525f4803 4c63b000 113287168 1 B 27658 0 > > So my 1 virtual segment range would be from 0x444af000 - 0x4c63b000 (which > you > can see it is contiguous with the following Buffer pool segment that > starts at > 0x4c63b000...this is normally what you would see, but on some platforms > there > can be gaps between the segments reported in onstat -g seg). > > So then from an onstat -g afr on a random pool we have > > Allocations for pool name rascron: > addr size memid fileid location > 46635000 3328 overhead 301 mtshpool.c:606 > 46635d00 232 TRACE 16 ph_collector.c:428 > 46635de8 152 tQ 3974 rsbufq.c:166 > > so looking at the 1st column the addr 0x46635000 is an address that would > fall > in the range of 0x444af000 - 0x4c63b000, so that 3328 bytes of memory (or > what > I refer to generally as a block or the doc is referring to as pool > fragment) > would be in the virtual segment who's seg id was "73564177". > > Jacques > > > ************************************************************ > ******************* > Forum Note: Use "Reply" to post a response in the discussion forum. > >

original post: <some stuff deleted> @Jacques , Thank you for your explanation. Please, can you answer my question below? If I get the sum of the column totalsize of all lines on 'onstat -g mem' (Pool+blkpool), should be close value of the used memory for the segments at 'onstat -g seg' ? (respecting the type of the segment, vitual, reserved...) Ask that because the situation I had here the sum of everything at 'onstat -g mem' was close of 8.3GB and the used segments at 'onstat -g seg' was ~16G (blkused*4096). Which is to huge discrepancy... I open an PMR for this, but unfortunately we do not have time enough to wait a feedback to collect more information and decide bounce and upgrade the instance since we was into a window for maintanece where this will be allowed. Regards Cesar Response: There is a couple things that are missing. First off, in onstat -g seg, if you are using 12.x and your buffer pools are in there own segments of class "B" then the blkused for those segments is not accounted for by a corresponding pool that is reported in onstat -g mem. So if you have buffer pools in their own segments, you would want to no include those blkused counts in your comparison of memory used via the sum of total size from onstat -g mem for pools and block pools and the blkused in onstat -g seg. Second item would be if your server uses VP_MEMORY_CACHE_KB or not. If your server has VP_MEMORY_CACHE_KB = 0 (so it's turned off) then I would expect that the sum of all the total size for pools and block pools in onstat -g mem to equal all of the used blocks portion of onstat -g seg (excluding blkused for segments of class "B"). However, if VP_MEMORY_CACHE_KB is a non-zero value and you don't have it set to STATIC then the size of the vp memory cache can vary and in that case you have to look at onstat -g vpcache to see how much memory each oninit vp has in it's vp cache....in that case then the sum of all blocks in the vp caches + sum of all the total size of all the pools and block pools from onstat -g mem should be roughly the size of the used blocks in onstat -g seg (excluding blkused for class "B" segments). At least that is what I would expect. I don't know that I'm certain enough to say this is 100% true, but it's my feeling on what I think would be the behavior. I know for sure the buffer pool segment blks are not accounted for in the onstat -g mem and I know for sure blocks in the vpcache are not accounted for in onstat -g mem...I'm just not 100% sure that there isn't anything else that wouldn't be accounted for in the onstat -g mem. Jacques

Jacques has beaten me to it but the most likely candidate is the CPU VP cache (VP_MEMORY_CACHE_KB) not draining quickly enough, which presumably you're running in DYNAMIC mode. "onstat -g vpcache" needs to be used to see the memory usage and it is hard to decipher without the aid of a script. https://informixdba.wordpress.com/2013/11/11/monitoring-virtual-segment-usage-an d-the-cpu-vp-caches/ If it does turn out to be this there is an RFE you can vote for asking for "onstat -g mem" to give a complete picture of memory usage by an instance: http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=39174 Ben.

I believe the latest code will show the underpinning function under onstat -g agr (??) instead of just sapimem.c If there is no symbol table available i.e. Custom UDRs you will still get sapimem Cheers Paul Paul Watson Oninit www.oninit.com +1 913 387 7529 Oninit® is a Registered Trademark of Oninit LLC > On Jul 29, 2017, at 14:29, BENJAMIN THOMPSON <benjamin.thompson@skybettingandgaming.com> wrote: > > Jacques has beaten me to it but the most likely candidate is the CPU VP cache > (VP_MEMORY_CACHE_KB) not draining quickly enough, which presumably you're > running in DYNAMIC mode. "onstat -g vpcache" needs to be used to see the > memory usage and it is hard to decipher without the aid of a script. > > > https://informixdba.wordpress.com/2013/11/11/monitoring-virtual-segment-usage-an d-the-cpu-vp-caches/ > > If it does turn out to be this there is an RFE you can vote for asking for > "onstat -g mem" to give a complete picture of memory usage by an instance: > http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=39174 > > Ben. > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum.