SHMTOTAL

IDS 10.0FC5 on RHEL AS 4 We have a system where, as ever, we have set SHMTOTAL to zero. Having run fine for months, the system has sporadically started taking, in a very short space of time, multiple shared memory segments. eg Saturday at 0800, fine with one V segment, by 1000: IBM Informix Dynamic Server Version 10.00.FC5 -- On-Line (Prim) -- Up 14 days 05:07:31 -- 4883156 Kbytes Segment Summary: id key addr size ovhd class blkused blkfree 622592 1381451777 44000000 1139298304 458688 R* 278146 3 655361 1381451778 87e85000 2518319104 77592 V 538129 76695 688130 1381451779 11e02d000 557056 760 M 135 1 720899 1381451780 11e0b5000 67108864 2784 V 963 15421 753668 1381451781 1220b5000 67108864 2784 V 1181 15203 786437 1381451782 1260b5000 67108864 2784 V 631 15753 819206 1381451783 12a0b5000 67108864 2784 V 834 15550 851975 1381451784 12e0b5000 67108864 2784 V 234 16150 884744 1381451785 1320b5000 67108864 2784 V 415 15969 917513 1381451786 1360b5000 67108864 2784 V 443 15941 950282 1381451787 13a0b5000 67108864 2784 V 308 16076 983051 1381451788 13e0b5000 67108864 2784 V 1091 15293 1015820 1381451789 1420b5000 67108864 2784 V 734 15650 1048589 1381451790 1460b5000 67108864 2784 V 350 16034 1081358 1381451791 14a0b5000 67108864 2784 V 351 16033 1114127 1381451792 14e0b5000 67108864 2784 V 395 15989 1146896 1381451793 1520b5000 67108864 2784 V 628 15756 1179665 1381451794 1560b5000 67108864 2784 V 576 15808 1212434 1381451795 15a0b5000 67108864 2784 V 313 16071 1245203 1381451796 15e0b5000 67108864 2784 V 275 16109 1277972 1381451797 1620b5000 67108864 2784 V 149 16235 1310741 1381451798 1660b5000 67108864 2784 V 89 16295 1343510 1381451799 16a0b5000 67108864 2784 V 79 16305 Total: - - 5000351744 - - 826449 394340 The application supplier wants us to cap SHMTOTAL, and says that his app will respond well to a refusal for extra memory. I'm not so sure: what are other's experiences with capping total memory? This server only has 4g of physical mem so we need to do something ... thanks Neil

Hi, hmm. This answer is not so easy. In general I think the handling should be better when SHMTOTAL is set to a value other than zero. Of course it needs to be set to a sensible number that corresponds to memory availability on the machine. Setting SHMTOTAL has the advantage that IDS knows in advance what the limit is and can act (or not act) accordingly. I.e. if memory space tightens, IDS can act with pre-cautions that will avoid catastrophic scenarios. If SHMTOTAL is set to zero, it means unlimited memory supply. This of course is never real, but always hypothetical. If IDS hits the limit (and the OS returns an error upon the request for more memory), then this error can be quite unexpected at the time. The underlying problem is, for what action or purpose is the memory needed. If it is needed for a user session (e.g. sorting, joining, etc.), then IDS will handle a memory shortage "gracefully", i.e. the respective session will get a corresponding error. But IDS will continue to run and function. If memory is needed for a new session, then the connect request will be rejected if no more memory is available - but IDS will continue to run. However, if some memory is needed for an IDS internal and vital function, then an out-of-memory error can be fatal for the functioning of IDS. And this is where the difference between SHMTOTAL set to zero or some (sensible) number comes into play. If SHMTOTAL is set to a sensible number, then IDS knows the limit in advance and can take care that such a fatal "out-of-memory error" does not happen. But if SHMTOTAL is set to zero, then an out-of-memory error from the OS will be unexpected - and under said circumstances can be fatal, because IDS relies on un-limited supply. I hope that this is somewhat understandable. :) Regards, Martin -- Martin Fuerderer IBM Informix Development Munich, Germany Information Management Informix URLs list: http://home.arcor.de/mfu1/informix/urls.html informix-list-bounces@iiug.org wrote on 29.01.2007 13:41:16: > IDS 10.0FC5 on RHEL AS 4 > > We have a system where, as ever, we have set SHMTOTAL to zero. > > Having run fine for months, the system has sporadically started taking, in a > very short space of time, multiple shared memory segments. eg Saturday at > 0800, fine with one V segment, by 1000: > > IBM Informix Dynamic Server Version 10.00.FC5 -- On-Line (Prim) --Up 14 > days 05:07:31 -- 4883156 Kbytes > > Segment Summary: > id key addr size ovhd class blkused > blkfree > 622592 1381451777 44000000 1139298304 458688 R* 278146 > 3 > 655361 1381451778 87e85000 2518319104 77592 V 538129 > 76695 > 688130 1381451779 11e02d000 557056 760 M 135 > 1 > 720899 1381451780 11e0b5000 67108864 2784 V 963 > 15421 > 753668 1381451781 1220b5000 67108864 2784 V 1181 > 15203 > 786437 1381451782 1260b5000 67108864 2784 V 631 > 15753 > 819206 1381451783 12a0b5000 67108864 2784 V 834 > 15550 > 851975 1381451784 12e0b5000 67108864 2784 V 234 > 16150 > 884744 1381451785 1320b5000 67108864 2784 V 415 > 15969 > 917513 1381451786 1360b5000 67108864 2784 V 443 > 15941 > 950282 1381451787 13a0b5000 67108864 2784 V 308 > 16076 > 983051 1381451788 13e0b5000 67108864 2784 V 1091 > 15293 > 1015820 1381451789 1420b5000 67108864 2784 V 734 > 15650 > 1048589 1381451790 1460b5000 67108864 2784 V 350 > 16034 > 1081358 1381451791 14a0b5000 67108864 2784 V 351 > 16033 > 1114127 1381451792 14e0b5000 67108864 2784 V 395 > 15989 > 1146896 1381451793 1520b5000 67108864 2784 V 628 > 15756 > 1179665 1381451794 1560b5000 67108864 2784 V 576 > 15808 > 1212434 1381451795 15a0b5000 67108864 2784 V 313 > 16071 > 1245203 1381451796 15e0b5000 67108864 2784 V 275 > 16109 > 1277972 1381451797 1620b5000 67108864 2784 V 149 > 16235 > 1310741 1381451798 1660b5000 67108864 2784 V 89 > 16295 > 1343510 1381451799 16a0b5000 67108864 2784 V 79 > 16305 > Total: - - 5000351744 - - 826449 > 394340 > > The application supplier wants us to cap SHMTOTAL, and says that his app > will respond well to a refusal for extra memory. I'm not so sure: what are > other's experiences with capping total memory? This server only has 4g of > physical mem so we need to do something ... > > thanks > Neil > > > _______________________________________________ > Informix-list mailing list > Informix-list@iiug.org > http://www.iiug.org/mailman/listinfo/informix-list

On 29 Jan, 15:17, Martin Fuerderer <MARTI...@de.ibm.com> wrote: > Hi, > > hmm. This answer is not so easy. > > In general I think the handling should be better when > SHMTOTAL is set to a value other than zero. Of course it needs > to be set to a sensible number that corresponds to memory availability > on the machine. > > Setting SHMTOTAL has the advantage that IDS knows in advance > what the limit is and can act (or not act) accordingly. I.e. if memory > space tightens, IDS can act with pre-cautions that will avoid > catastrophic scenarios. > CUSTOMER-REPORTED BUGS FIXED IN IBM INFORMIX DYNAMIC SERVER 10.00.xC5 bug_number 175251 description AFTER SIZE OF RESIDENT + VIRTUAL > SHMTOTAL IS DETECTED, ENGINE CRASHES product_code ONLINE component_code ASF Neil is lucky that he is on 10.00.xC5 already! I do not believe all possible "out of memory" conditions are handled gracefully in the code. surely IDS contains a single underlying function to allocate memory? Whether the function checks SHMTOTAL and returns an error or does a shmget and returns an error code should make no difference for stability. I assume whatever the function is it has been debugging preety throughly by now? I do not use SHMTOTAL since why have the application fail (normally with error 208 "Memory allocation failed in query processing") rather than allocate more memory and succeed? Whatever the application claims to do it if SHMTOTAL is reachedthe application cannot process the sql and get the desired result since there is not enough memory to complete the operation! > If SHMTOTAL is set to zero, it means unlimited memory supply. > This of course is never real, but always hypothetical. If IDS hits > the limit (and the OS returns an error upon the request for more > memory), then this error can be quite unexpected at the time. > Memory allocation should be handled by one or a few memory allocation functions that been thoughly debugged. Perhaps you guys should run a static code checker that understands the IDS memory allocation functions and can audit for code that does not check the return code of the function? I assume IDS it still written in C? I here Coverity have a good product to do static code analysis that could be adapted to check the IDS code. Do you guys run Purify over the code?

<david@smooth1.co.uk> wrote in message news:1170203435.318464.63850@a75g2000cwd.googlegroups.com... > On 29 Jan, 15:17, Martin Fuerderer <MARTI...@de.ibm.com> wrote: > CUSTOMER-REPORTED BUGS FIXED IN IBM INFORMIX DYNAMIC SERVER 10.00.xC5 > > bug_number 175251 > description AFTER SIZE OF RESIDENT + VIRTUAL > SHMTOTAL IS > DETECTED, ENGINE CRASHES > product_code ONLINE > component_code ASF > > > Neil is lucky that he is on 10.00.xC5 already! Hmm. I'd dispute that it has anything to do with luck; more assiduous planning and an aggressive upgrade programme! But thanks for the warning. Obviously I never read release notes or documentation as that would be a sign of weakness ... ;-) My concern is that this is an application in use at a number oif customers:, and at another customer site the memory allocation is failing and the application is not taking it at all welll. Albeit at the second location the reason for the mem alloc failure is that IDS is bumping up against a 2.7g addressibility limit of 32-bit Linux, so maybe the error returned is different from the -208 which, I presume as you do, will result from the SHMTOTAL limit.

Hi David, right. I did say that this answer is not so easy ... and I kind of anticipated a reply/comment like yours. :) OK, here it goes. a) I was not saying that with SHMTOTAL set to a value other than zero there will not be any problem at all. In other words, I did not say that all out-of-memory "occurances" would be handled gracefully. Such a statement would actually amount to a claim that the software (or even part of it) is free of any bug. And I'm too long in software development for daring to make such a claim, especially when the software is evolving with (lots of) new features (i.e. new code) being added regularly ... :) b) The problem (I think) is not so much the code (functions/subroutines) that do the actual memory allocation. After all, (as far as I know) out-of-memory errors are normally reported correctly (which means that these functions do check for this and other errors and give it back). c) Unfortunately, graceful handling of an error condition is not just about checking a functions return code. If that would do the trick, then IDS would be on the safe side, because out-of-memory errors are detected, reported and (in extreme cases that we are here talking about), are handled by bringing down the server. [ We would not handle the error if we would assume that (despite an ignored error) we got memory anyway and then simpy try to use it ... that would cause something like a SEGV - later on in the code. And that is a different scenario. Disclaimer for David :) : with this I do not say that a SEGV can never happen in the IDS server. This would amount to the "bug-free software" thing - see above. But I'm rather convinced that in IDS we do not get SEGVs because we didn't handle an out-of-memory error (that happened earlier). Reasons for possible SEGVs in IDS are different problems. ] Back to graceful handling of out-of-memory errors: Graceful handling of such errors would mean that the error condition is handled in a way that affects the overall operation of the server only minimally or not at all. And to achieve that is a very difficult thing, especially in a software product with > 1 million lines of code. It needs to be done on all levels of the function stack (upwards from these "memory allocation routines"). At any point you'd have to prepare for failure due out-of-memory and provide "alternative". E.g. you'd have to expect that you can't get a single byte of memory to simply report the error condition itself (e.g. in an constructed message with some detailed info to be sent to the user). You won't be able to construct the message, because you will not be able to get the memory for the string ... Disclaimer: with the following I'm not suggesting in any way, that QA for the IDS product is not a major focus in all the cycles of the product's life time. It was, is and remains a major focus and quality is improved continually. Now, I don't want to say that it is not possible to achieve such gracefulness. But I doubt that any customer would want to pay the price (in dollars and/or in waiting time) for such a software product. :-( [ I know that on one side this is a sad thing - and now it goes off into the philosophical area ... - but on the other side this is customers' fault as well ...: It used to take several years to develop a new (makeover) model for a car. After the development a prototype was built and tested many months. After that it still took a long time until you actually could buy that new model from your dealer. All this has been cut short dramatically, not the least with the help of lots of software (CAD, CAS, CAM, ...). Time-to-market is very important these days. And with software itself it is much more important. Turnover cycles are expected to be a year only, maybe 2 (but that would be the maximum). If it takes any longer ... your product is considered dead. So, what do you expect? Software would have had exponential price hikes if the quality assurance would be as rigorous as it used to be in some other industries (where it deteriorated as well). In one form or another we all (as consumers of software in one fway or another, directly or indirectly) have to pay the price. If not in dollars or waiting time ... ] Hmm. I hope this brightens up your day! ;) Cheers, Martin -- Martin Fuerderer IBM Informix Development Munich, Germany Information Management Informix URLs list: http://home.arcor.de/mfu1/informix/urls.html informix-list-bounces@iiug.org wrote on 31.01.2007 01:30:35: > On 29 Jan, 15:17, Martin Fuerderer <MARTI...@de.ibm.com> wrote: > > Hi, > > > > hmm. This answer is not so easy. > > > > In general I think the handling should be better when > > SHMTOTAL is set to a value other than zero. Of course it needs > > to be set to a sensible number that corresponds to memory availability > > on the machine. > > > > > Setting SHMTOTAL has the advantage that IDS knows in advance > > what the limit is and can act (or not act) accordingly. I.e. if memory > > space tightens, IDS can act with pre-cautions that will avoid > > catastrophic scenarios. > > > > CUSTOMER-REPORTED BUGS FIXED IN IBM INFORMIX DYNAMIC SERVER 10.00.xC5 > > bug_number 175251 > description AFTER SIZE OF RESIDENT + VIRTUAL > SHMTOTAL IS > DETECTED, ENGINE CRASHES > product_code ONLINE > component_code ASF > > > Neil is lucky that he is on 10.00.xC5 already! > > I do not believe all possible "out of memory" conditions are handled > gracefully in the code. > > surely IDS contains a single underlying function to allocate memory? > > Whether the function checks SHMTOTAL and returns an error or does a > shmget and returns an error code should make no difference for > stability. I assume whatever the function is it has been debugging > preety throughly by now? > > I do not use SHMTOTAL since why have the application fail (normally > with error 208 "Memory allocation failed in query processing") > rather than allocate more memory and succeed? Whatever the application > claims to do it if SHMTOTAL is reachedthe > application cannot process the sql and get the desired result since > there is not enough memory to complete the operation! > > > If SHMTOTAL is set to zero, it means unlimited memory supply. > > This of course is never real, but always hypothetical. If IDS hits > > the limit (and the OS returns an error upon the request for more > > memory), then this error can be quite unexpected at the time. > > > > Memory allocation should be handled by one or a few memory allocation > functions that been thoughly debugged. Perhaps you guys should > run a static code checker that understands the IDS memory allocation > functions and can audit for code that does not check the return code > of the function? > > I assume IDS it still written in C? I here Coverity have a good > product to do static co