shared memory size limit on 32-bit vs 64 bit IDS

Dear all, I've googled until my fingers hurt on this subject and still can't get it. In the new feature of IDS 10,it said we can specify shared memory size greater than 4G. So in my understanding, in previous version the maximum memory size is 4G. However I read some article show the maximum memory size for 32bit is 2G. Did I misunderstand what it said? what is the maximum you can allocate for one instance (or all instances)? is it different between different platform ? is there any official document in IBM web site or other place can help me with this problem? thanks in advance, Fan

Different on different platforms. It depends on the OS and the size and location of shared libraries, also. I have a test app that I'll send to your email address directly. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) IIUG Board of Directors (art@iiug.org) See you at the 2010 IIUG Informix Conference April 25-28, 2010 Overland Park (Kansas City), KS www.iiug.org/conf Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Thu, May 6, 2010 at 6:01 AM, FAN YANG <fania.yang@gmail.com> wrote: > Dear all, > > I've googled until my fingers hurt on this subject and still can't get it. > > In the new feature of IDS 10,it said we can specify shared memory size > greater > than 4G. So in my understanding, in previous version the maximum memory > size > is 4G. However I read some article show the maximum memory size for 32bit > is > 2G. Did I misunderstand what it said? what is the maximum you can allocate > for > one instance (or all instances)? is it different between different platform > ? > is there any official document in IBM web site or other place can help me > with > this problem? > > thanks in advance, > Fan > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --00504502ad00bf20160485ea3663

Hi, a "word" is used to address the memory. Hence the size of a word determines how much memory is adressable. This addressable memory also is referred to as "address space". The size of a word on a 32-bit architecture is 32 bit, on a 64-bit architecture it is 64 bit. (Actually this size of the word is giving the names "32-bit" or "64-bit".) With a 32-bit word, 4 GB of memory can be addressed. With a 64-bit word, ... ... much more than normally physically available (in these days) can be addressed. This concept of address space is valid for each process running on a system. So much for the general theory. In practice however, the address space get's fragmented by different pieces (segments) of memory used for different purposes. Usually there is a heap, a stack, somewhere are shared libraries, maybe there is a "text segment" (containing the programs code), etc. All these things are mapped into a processes address space, at different places within this space. So the "free memory" that can be used by an application is limited by how much these other things take up - and by how much the remaining space is fragmented. And this is where in general the OS specifics come into the picture. Each OS has a different way of doing the reservation for and mapping of these different segments. Often both, reservation and mapping can somehow be influenced or configured, sometimes per process, sometimes system wide (e.g. by kernel parameters). IDS specific in this picture is, that IDS mostly (almost exclusively) uses shared memory. This is also mapped to segments in the address space of a process. The config parameter SHMBASE determines, where in the address space the first shared memory segment will be mapped to. Following shared memory segments are mapped consecutively whenever possible. But in the end the OS decides about this mapping. However, one shared memory segment of IDS (see also "onstat -g seg") can consist of several OS shared memory segments (see "ipcs -m"). In that case, all OS shared memory segments of a single IDS shared memory segment are expected (by IDS) to be mapped adjacently. But if the OS decides to do this in a different way (e.g. because a shared lib segment "is in the way"), then things will not work out. Hence such circumstances will limit the amount of memory that IDS can use (as shared memory). If e.g. the shared libraries are mapped into the middle of the address space, with the benevolent idea of the OS to leave space for heap on one side and for stack on the other, then this means that IDS is likely to use only half of the available memory, on either side of "the barrier" of shared libs segments. The other half will go pretty much unused, because IDS makes little to no use of stack and heap. So, there are many parameters that influence these things. Therefore it is almost impossible to give any "fixed numbers" about maximum possible memory use. I do not know of "an official IBM web site or other place" giving such numbers. But with the above explanation and some basic understanding of memory management ... you can figure out the scenario on your system and the resulting limitations. Regards, Martin -- Martin Fuerderer IBM Informix Development Munich, Germany Information Management IBM Deutschland Research & Development GmbH Chairman of the Supervisory Board: Martin Jetter Board of Management: Dirk Wittkopp Corporate Seat: Boeblingen, Germany Reg.-Gericht: Amtsgericht Stuttgart, HRB 243294 ids-bounces@iiug.org wrote on 05/06/2010 12:01:14 PM: > Dear all, > > I've googled until my fingers hurt on this subject and still can't get it. > > In the new feature of IDS 10,it said we can specify shared memory size greater > than 4G. So in my understanding, in previous version the maximum memory size > is 4G. However I read some article show the maximum memory size for 32bit is > 2G. Did I misunderstand what it said? what is the maximum you can allocate for > one instance (or all instances)? is it different between different platform ? > is there any official document in IBM web site or other place can help me with > this problem? > > thanks in advance, > Fan > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >

hi, Martin thanks for your reply However,I read about the feature nots in the machine nots for IDS 9.40 FC1. The larger memory addressability (LMA) provides the ability to support system configurations with greater than 4GB of RAM. And it shows that the maximum size of any shared memory segment is 4278190080 bytes. In the other side, the new feature in IDS 10, it also shows it can support shared memory size greater than 4G.So now I am confused and I really can't figure out what they talk about. Fan *********************************************************************** Hi, a "word" is used to address the memory. Hence the size of a word determines how much memory is adressable. This addressable memory also is referred to as "address space". The size of a word on a 32-bit architecture is 32 bit, on a 64-bit architecture it is 64 bit. (Actually this size of the word is giving the names "32-bit" or "64-bit".) With a 32-bit word, 4 GB of memory can be addressed. With a 64-bit word, ... ... much more than normally physically available (in these days) can be addressed. This concept of address space is valid for each process running on a system. So much for the general theory. In practice however, the address space get's fragmented by different pieces (segments) of memory used for different purposes. Usually there is a heap, a stack, somewhere are shared libraries, maybe there is a "text segment" (containing the programs code), etc. All these things are mapped into a processes address space, at different places within this space. So the "free memory" that can be used by an application is limited by how much these other things take up - and by how much the remaining space is fragmented. And this is where in general the OS specifics come into the picture. Each OS has a different way of doing the reservation for and mapping of these different segments. Often both, reservation and mapping can somehow be influenced or configured, sometimes per process, sometimes system wide (e.g. by kernel parameters). IDS specific in this picture is, that IDS mostly (almost exclusively) uses shared memory. This is also mapped to segments in the address space of a process. The config parameter SHMBASE determines, where in the address space the first shared memory segment will be mapped to. Following shared memory segments are mapped consecutively whenever possible. But in the end the OS decides about this mapping. However, one shared memory segment of IDS (see also "onstat -g seg") can consist of several OS shared memory segments (see "ipcs -m"). In that case, all OS shared memory segments of a single IDS shared memory segment are expected (by IDS) to be mapped adjacently. But if the OS decides to do this in a different way (e.g. because a shared lib segment "is in the way"), then things will not work out. Hence such circumstances will limit the amount of memory that IDS can use (as shared memory). If e.g. the shared libraries are mapped into the middle of the address space, with the benevolent idea of the OS to leave space for heap on one side and for stack on the other, then this means that IDS is likely to use only half of the available memory, on either side of "the barrier" of shared libs segments. The other half will go pretty much unused, because IDS makes little to no use of stack and heap. So, there are many parameters that influence these things. Therefore it is almost impossible to give any "fixed numbers" about maximum possible memory use. I do not know of "an official IBM web site or other place" giving such numbers. But with the above explanation and some basic understanding of memory management ... you can figure out the scenario on your system and the resulting limitations. Regards, Martin -- Martin Fuerderer IBM Informix Development Munich, Germany Information Management IBM Deutschland Research & Development GmbH Chairman of the Supervisory Board: Martin Jetter Board of Management: Dirk Wittkopp Corporate Seat: Boeblingen, Germany Reg.-Gericht: Amtsgericht Stuttgart, HRB 243294 ids-bounces@iiug.org wrote on 05/06/2010 12:01:14 PM: > Dear all, > > I've googled until my fingers hurt on this subject and still can't get it. > > In the new feature of IDS 10,it said we can specify shared memory size greater > than 4G. So in my understanding, in previous version the maximum memory size > is 4G. However I read some article show the maximum memory size for 32bit is > 2G. Did I misunderstand what it said? what is the maximum you can allocate for > one instance (or all instances)? is it different between different platform ? > is there any official document in IBM web site or other place can help me with > this problem? > > thanks in advance, > Fan > > > *******************************************************************************

Some 32 bit architectures use a 64 bit address space internally so they can have more than 4GB of memory addressable by the OS. However, 32bit applications will still only have a 32bit address space, it's just that the starting physical address can be greater than 4GB. IB that's what the notes are referring to. To be absolutely clear, no 32 bit instance of Informix can address more than 4GB of memory and on most platforms they are practically limited by the stack, heap, and shared library attach addresses to something closed to 3.2GB. On some platforms like AIX that can be as log as 2.2GB. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) IIUG Board of Directors (art@iiug.org) See you at the 2010 IIUG Informix Conference April 25-28, 2010 Overland Park (Kansas City), KS www.iiug.org/conf Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Fri, May 7, 2010 at 5:40 AM, FAN YANG <fania.yang@gmail.com> wrote: > hi, Martin > > thanks for your reply > > However,I read about the feature nots in the machine nots for IDS 9.40 FC1. > The larger memory addressability (LMA) provides the ability to support > system > configurations with greater than 4GB of RAM. And it shows that the maximum > size of any shared memory segment is 4278190080 bytes. In the other side, > the > new feature in IDS 10, it also shows it can support shared memory size > greater > than 4G.So now I am confused and I really can't figure out what they talk > about. > > Fan > > *********************************************************************** > > Hi, > > a "word" is used to address the memory. Hence the size of a > word determines how much memory is adressable. This > addressable memory also is referred to as "address space". > The size of a word on a 32-bit architecture is 32 bit, on a > 64-bit architecture it is 64 bit. (Actually this size of the word > is giving the names "32-bit" or "64-bit".) With a 32-bit word, > 4 GB of memory can be addressed. With a 64-bit word, ... > .... much more than normally physically available (in these days) > can be addressed. This concept of address space is valid for > each process running on a system. So much for the general theory. > > In practice however, the address space get's fragmented by > different pieces (segments) of memory used for different purposes. > Usually there is a heap, a stack, somewhere are shared libraries, > maybe there is a "text segment" (containing the programs code), > etc. All these things are mapped into a processes address space, > at different places within this space. So the "free memory" that > can be used by an application is limited by how much these other > things take up - and by how much the remaining space is fragmented. > And this is where in general the OS specifics come into the picture. > Each OS has a different way of doing the reservation for and mapping > of these different segments. Often both, reservation and mapping can > somehow be influenced or configured, sometimes per process, > sometimes system wide (e.g. by kernel parameters). > > IDS specific in this picture is, that IDS mostly (almost exclusively) > uses shared memory. This is also mapped to segments in the > address space of a process. The config parameter SHMBASE > determines, where in the address space the first shared memory > segment will be mapped to. Following shared memory segments > are mapped consecutively whenever possible. But in the end the > OS decides about this mapping. However, one shared memory > segment of IDS (see also "onstat -g seg") can consist of several > OS shared memory segments (see "ipcs -m"). In that case, all > OS shared memory segments of a single IDS shared memory > segment are expected (by IDS) to be mapped adjacently. But if > the OS decides to do this in a different way (e.g. because a > shared lib segment "is in the way"), then things will not work out. > Hence such circumstances will limit the amount of memory that > IDS can use (as shared memory). If e.g. the shared libraries are > mapped into the middle of the address space, with the benevolent > idea of the OS to leave space for heap on one side and for stack > on the other, then this means that IDS is likely to use only half of > the available memory, on either side of "the barrier" of shared libs > segments. The other half will go pretty much unused, because IDS > makes little to no use of stack and heap. > > So, there are many parameters that influence these things. > Therefore it is almost impossible to give any "fixed numbers" about > maximum possible memory use. I do not know of "an official IBM > web site or other place" giving such numbers. But with the above > explanation and some basic understanding of memory management ... > you can figure out the scenario on your system and the resulting > limitations. > > Regards, Martin > -- > Martin Fuerderer > IBM Informix Development Munich, Germany > Information Management > > IBM Deutschland Research & Development GmbH > Chairman of the Supervisory Board: Martin Jetter > Board of Management: Dirk Wittkopp > Corporate Seat: Boeblingen, Germany > Reg.-Gericht: Amtsgericht Stuttgart, HRB 243294 > > ids-bounces@iiug.org wrote on 05/06/2010 12:01:14 PM: > > Dear all, > > > > I've googled until my fingers hurt on this subject and still can't get > it. > > > > In the new feature of IDS 10,it said we can specify shared memory size > greater > > than 4G. So in my understanding, in previous version the maximum memory > size > > is 4G. However I read some article show the maximum memory size for > 32bit is > > 2G. Did I misunderstand what it said? what is the maximum you can > allocate for > > one instance (or all instances)? is it different between different > platform ? > > is there any official document in IBM web site or other place can help > me with > > this problem? > > > > thanks in advance, > > Fan > > > > > > > > > ******************************************************************************* > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --000e0cd6085459f1f10485fead16

Hi, as a rule of thumb i would say: A 32bit-IDS instance can use about 2 GB of shared memory, a 64bit-IDS instance can allocate as much memory as you want (if the OS can provide it). The difference between IDS9 and IDS10/11 is that IDS9 allocates its shared memory with multiple segments of 4 GB, even if the OS can provide larger shared memory segments. IDS10/11 try to allocate the virtual shared memory (SHMVIRTSIZE) and the bufferpool (BUFFERS/BUFFERPOOL) each in one large shared memory segment. For a real installation there is a huge dependancy between IDS and OS (as Martin already tried to explain to you): With HPUX (even 64bit/PA-Risc) and IDS (32bit) for example there are special limitations about the maximum shared memory for all 32bit applications running on a 64bit-system together (<2 GB), unless you are using shmmagic executables, which will give you some other problems. Therefore the shared memory limit (<2 GB) holds for all 32bit instances together in practice. On Linux (even on 32bit) the limit of about 2 GB holds only for each instance, I think (I'm not 100% sure). Together they can allocate more. With HPUX (64bit/IA64) and IDS (64bit) you can allocate almost all of the available physical memory for the IDS instance (reserve about 20% for the OS with standard configuration, about 10-15% with large base_pagesize). On HPUX you need enough real swap space (devices), on Linux not. Some OS limit the part of real memory, that you can use as shared memory. Therefore please provide specific informations about physical machine, OS (and kernel configuration) and IDS version, if you need more specific answers. Regards, Andreas > ------------------------------------------- SPAR Österreichische Warenhandels-AG Hauptzentrale A - 5015 Salzburg, Europastrasse 3 FN 34170 a Tel: +43 662 4470 84923 Fax: Mobile: +43 664 6259575 E-Mail: Andreas.KUTSCHE@spar.at Internet: http://www.spar.at Wichtiger Hinweis: Der Inhalt dieser E-Mail kann vertrauliche und rechtlich geschützte Informationen, insbesondere Betriebs- oder Geschäftsgeheimnisse, enthalten, zu deren Geheimhaltung der Empfänger verpflichtet ist. Die Informationen in dieser E-Mail sind ausschließlich für den Adressaten bestimmt. Sollten Sie die E-Mail irrtümlich erhalten haben so ersuchen wir Sie, die Nachricht von Ihrem System zu löschen und sich mit uns in Verbindung zu setzen. Über das Internet versandte E-Mails können leicht manipuliert oder unter fremdem Namen erstellt werden. Daher schließen wir die rechtliche Verbindlichkeit der in dieser Nachricht enthaltenen Informationen aus. Der Inhalt der E-Mail ist nur rechtsverbindlich, wenn er von uns schriftlich bestätigt und gezeichnet wird. Sollte trotz der von uns verwendeten Virus-Schutzprogramme durch die Zusendung von E-Mails ein Virus in Ihre Systeme gelangen, haften wir nicht für evtl. hieraus entstehende Schäden. Wir danken für Ihr Verständnis. Important notice: The contents of this e-mail may contain confidential and legally protected information that is in particular related to operational and trade secrets, which the recipient is obliged to treat as confidential. The information in this e-mail is made available exclusively for use by the addressee. In the event that the e-mail may have been sent to you in error, we would ask you to kindly delete this communication from your system and to contact us. E-mails sent via the Internet can be easily manipulated or sent out under someone else's name. We therefore do not accept legal liability for the information contained in this communication. The contents of the e-mail are only legally binding if they have been confirmed and signed by us in writing. If, in spite of our using Antivirus protection software, a virus may have penetrated your system through the sending of this e-mail, we do not accept liability for any damage that may possibly arise as a result of this. We trust that you appreciate our position. ------------------------------------------- -----Ursprüngliche Nachricht----- > Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im Auftrag von FAN > YANG > Gesendet: Freitag, 7. Mai 2010 11:40 > An: ids@iiug.org > Betreff: Re: shared memory size limit on 32-bit vs 64 bit I [20066] > > hi, Martin > > thanks for your reply > > However,I read about the feature nots in the machine nots for IDS 9.40 > FC1. > The larger memory addressability (LMA) provides the ability to support > system > configurations with greater than 4GB of RAM. And it shows that the maximum > size of any shared memory segment is 4278190080 bytes. In the other side, > the > new feature in IDS 10, it also shows it can support shared memory size > greater > than 4G.So now I am confused and I really can't figure out what they talk > about. > > Fan > > *********************************************************************** > > Hi, > > a "word" is used to address the memory. Hence the size of a > word determines how much memory is adressable. This > addressable memory also is referred to as "address space". > The size of a word on a 32-bit architecture is 32 bit, on a > 64-bit architecture it is 64 bit. (Actually this size of the word > is giving the names "32-bit" or "64-bit".) With a 32-bit word, > 4 GB of memory can be addressed. With a 64-bit word, ... > .... much more than normally physically available (in these days) > can be addressed. This concept of address space is valid for > each process running on a system. So much for the general theory. > > In practice however, the address space get's fragmented by > different pieces (segments) of memory used for different purposes. > Usually there is a heap, a stack, somewhere are shared libraries, > maybe there is a "text segment" (containing the programs code), > etc. All these things are mapped into a processes address space, > at different places within this space. So the "free memory" that > can be used by an application is limited by how much these other > things take up - and by how much the remaining space is fragmented. > And this is where in general the OS specifics come into the picture. > Each OS has a different way of doing the reservation for and mapping > of these different segments. Often both, reservation and mapping can > somehow be influenced or configured, sometimes per process, > sometimes system wide (e.g. by kernel parameters). > > IDS specific in this picture is, that IDS mostly (almost exclusively) > uses shared memory. This is also mapped to segments in the > address space of a process. The config parameter SHMBASE > determines, where in the address space the first shared memory > segment will be mapped to. Following shared memory segments > are mapped consecutively whenever possible. But in the end the > OS decides about this mapping. However, one shared memory > segment of IDS (see also "onstat -g seg") can consist of several > OS shared memory segments (see "ipcs -m"). In that case, all > OS shared memory segments of a single IDS shared memory > segment are expected (by IDS) to be mapped adjacently. But if > the OS