Running out of virtual mem segments

This problem has been a constant issue on our production server for years, according to my previous manager, Informix support has been called to solve this issue years ago, but was unable to provide a resolution. What we have been doing to prevent this is to re-start the unix process that calls the 4gl code several time a week. Here is a general description of the problem: Environment : HPUX 11, IDS 7.31.Uc6AXF, 4gl 7.30.HC4 Program structure: 1) Parent unix shell - this shell runs (constantly) on the background, sleeps for 30 seconds, then checks if there are files to be processed in directory a. If there are no files, sleeps again for 30 seconds, and awakes to check. If there are files, calls child unix shell script using "exec child.sh" and then exits 2) Child shell - this script processes the files by calling a 4gl program and after files are processed, calls the parent shell to check for files again, using "exec parent.sh" 3) 4GL program - uses dbload to load the files into table A then re formats the data for insertion to table B Problem: After several days of the scripts and 4gl running, there comes a point when the system will run out of virtual mem segments. At this point, we find that the 4gl program is hanging, in the middle of processing a file, with no apparent error or reason to hang (if we re-process the file it loads ok). And also, that the 4gl program has requested hundreds of segments until online runs out. We clear the issue by killing the hanging processes, and then the segments are automatically freed. We would like to know if others have had a similar setup, and may provde suggestions before we change the process logic, we are thinking of using a cron instead of a unix daemon that is constantly running. Regards, MCruz

My answer does not address the issue of why there are so many segs being allocated but how to prevent it from causing a hang - hopefully. Can you reconfigure your SHMVIRTSIZE $ONCONFIG parameter such that the database server has more virtual memory at startup and does not need to allocate additional segments? Is the hang occuring because you are running out of shared memory or because the engine is prevented from allocating another segment? The later would be caused by a kernal parameter that controls the # of segments a process can attach to - the former would be just simply running out of memory, and if SHMTOTAL is tuned to the highest value possible there would be nothing for you to do on the Informix side. Mike

HP-UX has always been tricky with SHM segments. Like someone else suggested, try making SHMADD in your $ONCONFIG much larger so that you don't have a lot of added-on V segments. Also, IDS 7.31 is probably no longer supported. Getting to at least IDS 10 (or 11) makes a DBA's life so much better! Bob Roussey Unix / Informix Administration Spirit Airlines Robert.Roussey@SpiritAir.com -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of MARGARITA CRUZ Sent: Wednesday, April 30, 2008 3:37 PM To: ids@iiug.org Subject: Running out of virtual mem segments [11939] This problem has been a constant issue on our production server for years, according to my previous manager, Informix support has been called to solve this issue years ago, but was unable to provide a resolution. What we have been doing to prevent this is to re-start the unix process that calls the 4gl code several time a week. Here is a general description of the problem: Environment : HPUX 11, IDS 7.31.Uc6AXF, 4gl 7.30.HC4 Program structure: 1) Parent unix shell - this shell runs (constantly) on the background, sleeps for 30 seconds, then checks if there are files to be processed in directory a. If there are no files, sleeps again for 30 seconds, and awakes to check. If there are files, calls child unix shell script using "exec child.sh" and then exits 2) Child shell - this script processes the files by calling a 4gl program and after files are processed, calls the parent shell to check for files again, using "exec parent.sh" 3) 4GL program - uses dbload to load the files into table A then re formats the data for insertion to table B Problem: After several days of the scripts and 4gl running, there comes a point when the system will run out of virtual mem segments. At this point, we find that the 4gl program is hanging, in the middle of processing a file, with no apparent error or reason to hang (if we re-process the file it loads ok). And also, that the 4gl program has requested hundreds of segments until online runs out. We clear the issue by killing the hanging processes, and then the segments are automatically freed. We would like to know if others have had a similar setup, and may provde suggestions before we change the process logic, we are thinking of using a cron instead of a unix daemon that is constantly running. Regards, MCruz ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum. See you at the IIUG Informix 2008 Conference The Power Conference for Informix Professionals April 27 - 30, 2008 Marriott Overland Park (Kansas City), Kansas http://www.iiug.org/conf Registration Now Open!!

Wonder if all those segments are really beging used. Might try this as a script and run it as user informix if informix owns the segments. ------------ #!/bin/ksh TMP=${0}.tmp ipcs -mob | grep "0 200" | grep -v "root" | grep -v "informix" > ${TMP} count=0 while read ipcs_val do segment=$(echo ${ipcs_val} | awk '{print $2}') ipcrm -m ${segment} (( count = count + 1 )) done < ${TMP} echo "\\ ${count} unattached shared memory segments removed.\\ " rm -f ${TMP} ---------------- I've seen issues where the number of segments can cause a problem. As others have suggested, take the number of additonal segmentes times SHMADD and add this to SHMVIRTSIZE. You might consider doubling the size of SHMADD to reduce the number of segments. Might be interesing to know which memory pool is growing, RSAM?

Margarita, I am suspecting an error in the 4gl code - it starts a sub process (and dies) which requests a memory segment in which to operate, processes the batch, re-starts the other process and then dies - without releasing the previously requested segment... rinse, lather, repeat. Pretty soon all the memory will be locked up. -ScottM. -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of MARGARITA CRUZ Sent: Wednesday, April 30, 2008 1:37 PM To: ids@iiug.org Subject: Running out of virtual mem segments [11939] This problem has been a constant issue on our production server for years, according to my previous manager, Informix support has been called to solve this issue years ago, but was unable to provide a resolution. What we have been doing to prevent this is to re-start the unix process that calls the 4gl code several time a week. Here is a general description of the problem: Environment : HPUX 11, IDS 7.31.Uc6AXF, 4gl 7.30.HC4 Program structure: 1) Parent unix shell - this shell runs (constantly) on the background, sleeps for 30 seconds, then checks if there are files to be processed in directory a. If there are no files, sleeps again for 30 seconds, and awakes to check. If there are files, calls child unix shell script using "exec child.sh" and then exits 2) Child shell - this script processes the files by calling a 4gl program and after files are processed, calls the parent shell to check for files again, using "exec parent.sh" 3) 4GL program - uses dbload to load the files into table A then re formats the data for insertion to table B Problem: After several days of the scripts and 4gl running, there comes a point when the system will run out of virtual mem segments. At this point, we find that the 4gl program is hanging, in the middle of processing a file, with no apparent error or reason to hang (if we re-process the file it loads ok). And also, that the 4gl program has requested hundreds of segments until online runs out. We clear the issue by killing the hanging processes, and then the segments are automatically freed. We would like to know if others have had a similar setup, and may provde suggestions before we change the process logic, we are thinking of using a cron instead of a unix daemon that is constantly running. Regards, MCruz ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum. See you at the IIUG Informix 2008 Conference The Power Conference for Informix Professionals April 27 - 30, 2008 Marriott Overland Park (Kansas City), Kansas http://www.iiug.org/conf Registration Now Open!!

Good script mate. RALPH GENTRY <gentrym@staples.com> wrote: Wonder if all those segments are really beging used. Might try this as a script and run it as user informix if informix owns the segments. ------------ #!/bin/ksh TMP=${0}.tmp ipcs -mob | grep "0 200" | grep -v "root" | grep -v "informix" > ${TMP} count=0 while read ipcs_val do segment=$(echo ${ipcs_val} | awk '{print $2}') ipcrm -m ${segment} (( count = count + 1 )) done < ${TMP} echo "\\ ${count} unattached shared memory segments removed.\\ " rm -f ${TMP} ---------------- I've seen issues where the number of segments can cause a problem. As others have suggested, take the number of additonal segmentes times SHMADD and add this to SHMVIRTSIZE. You might consider doubling the size of SHMADD to reduce the number of segments. Might be interesing to know which memory pool is growing, RSAM? ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum. See you at the IIUG Informix 2008 Conference The Power Conference for Informix Professionals April 27 - 30, 2008 Marriott Overland Park (Kansas City), Kansas http://www.iiug.org/conf Registration Now Open!! --------------------------------- Get the name you always wanted with the new y7mail email address.

On 30/04/2008, MARGARITA CRUZ <margarita.cruz@dhl.com> wrote: > This problem has been a constant issue on our production server for years, > according to my previous manager, Informix support has been called to solve > this issue years ago, but was unable to provide a resolution. What we have > been doing to prevent this is to re-start the unix process that calls the 4gl > code several time a week. Here is a general description of the problem: > > Environment : HPUX 11, IDS 7.31.Uc6AXF, 4gl 7.30.HC4 > Program structure: > 1) Parent unix shell - this shell runs (constantly) on the background, sleeps > for 30 seconds, then checks if there are files to be processed in directory a. > If there are no files, sleeps again for 30 seconds, and awakes to check. If > there are files, calls child unix shell script using "exec child.sh" and then > exits > 2) Child shell - this script processes the files by calling a 4gl program and > after files are processed, calls the parent shell to check for files again, > using "exec parent.sh" > 3) 4GL program - uses dbload to load the files into table A then re formats > the data for insertion to table B > > Problem: > After several days of the scripts and 4gl running, there comes a point when > the system will run out of virtual mem segments. At this point, we find that > the 4gl program is hanging, in the middle of processing a file, with no > apparent error or reason to hang (if we re-process the file it loads ok). And > also, that the 4gl program has requested hundreds of segments until online > runs out. > > We clear the issue by killing the hanging processes, and then the segments are > automatically freed. > > We would like to know if others have had a similar setup, and may provde > suggestions before we change the process logic, we are thinking of using a > cron instead of a unix daemon that is constantly running. > > Regards, > MCruz > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > See you at the IIUG Informix 2008 Conference > The Power Conference for Informix Professionals > April 27 - 30, 2008 Marriott Overland Park (Kansas City), Kansas > http://www.iiug.org/conf > Registration Now Open!! > MARGARITA Many others have suggested looking at shmem, increasing it, removing unused segments. Might I suggest looking at the scripts themselves. Running a parent which execs a child which execs the parent which execs the child etc.etc. does not seem particularly good. I would suggest it would be better to leave a single parent running continuously and check for files every 30 seconds, but then simply call the child, which calls the 4GL and then returns rather than exec'ing the parent. I think the long chain of execs is where you are losing memory. Keith