11.5 UC7E crashes

Hi, we have an Informix Server 11.5 UC7E running under Debian Linux. The system ran fine in no-logging-mode for several years until we switch to logging-mode to support a new application. Since then we experience crashes - the database dies with the following messages from the online.log: 18:09:34 Assert fehlgeschlagen: Bedingung nicht erfüllt (Bad block pointer 0x893fe2c in pool 0x28632028), In (mt_shm_free) 18:09:34 IBM Informix Dynamic Server Version 11.50.UC7E 18:09:34 Wer: Session(137915, hoffmann@tefix, 6734, 0x38e64e80) Thread(141348, sqlexec, 3ede9f40, 1) File: mtshpool.c Line: 4193 18:09:34 Aktion: Bitte informieren Sie die technische Unterstützung für IBM Informix. 18:09:34 stack trace for pid 8244 written to /opt/informix/tmp/af.2c0cc14d 18:09:34 Siehe auch: /opt/informix/tmp/af.2c0cc14d 18:09:36 Bedingung nicht erfüllt (Bad block pointer 0x893fe2c in pool 0x28632028), In (mt_shm_free) 18:09:36 stack trace for pid 8244 written to /opt/informix/tmp/af.2c0cc14d 18:09:36 Assert fehlgeschlagen: No Exception Handler 18:09:36 IBM Informix Dynamic Server Version 11.50.UC7E 18:09:36 Wer: Session(137915, hoffmann@tefix, 6734, 0x38e64e80) Thread(141348, sqlexec, 3ede9f40, 1) File: mtex.c Line: 491 18:09:36 Ergebnisse: Exception Caught. Type: MT_EX_OS, Context: mem 18:09:36 Aktion: Bitte informieren Sie die technische Unterstützung für IBM Informix. 18:09:36 Siehe auch: /opt/informix/tmp/af.2c0cc14d 18:09:38 Starting crash time check of: 18:09:38 1. memory block headers 18:09:38 2. stacks 18:09:38 Found bad memory block; address:29acfc88 18:09:38 mtex.c, line 491, thread 141348, proc id 8244, No Exception Handler. 18:09:39 Der Master-Dämon ist nicht mehr aktiv 18:09:39 PANIC: Attempting to bring system down I am not an Informix expert, but responsible for the system. The internet is not helping with these error messages. I don´t know if this is important or has anything to do with the problem, but we have this message regularly: 08:28:44 Contiguous shared memory segment allocation failed at 0xb78ee000. Allocation successful at 0x43800000. Check SHMBASE is consistent with the value in $INFORMIXDIR/etc/onconfig.std. If you are using the correct SHMBASE value in your ONCONFIG file, then consider this message informational only. 08:28:44 Dynamically allocated new virtual shared memory segment (size 8192KB) 08:28:44 Memory sizes:resident:1761624 KB, virtual:139872 KB, no SHMTOTAL limit Short extract from our onconfig-file: SHMBASE 0x44000000L SHMVIRTSIZE 32656 SHMADD 8192 EXTSHMADD 8192 SHMTOTAL 0 SHMVIRT_ALLOCSEG 0.000000 SHMNOACCESS Can anybody help me or can give me an advice how to solve this problem? Or is anybody experiencing similiar problems? Thanks for reading my long post :-) Klaus

Hallo Klaus, locks a bit like your IDS instance is running out of memory and has problems allocating new segments. I suggest you call your Informix vendor or our friends from IBM Munich, as your onconfig file locks pretty much like the one out of the box without any adjustments. Our IIUG friends from the states can´t help much, because of german messages in your log file. At first I suggest to start the engine with much more memory allocated, this way the engine does not have to search for free continuous pieces of free memory all the time. How much memory has your machine? Good suggestion is also the IDS Reference by Eric Herber, ISBN 978-3868020120 as a pocket reference. Mit freundlichen Grüßen Joerg Volz ----------------------------------------------------------------------------- -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of KLAUS GLAHN Sent: Tuesday, December 20, 2011 8:16 PM To: ids@iiug.org Subject: 11.5 UC7E crashes [25681] Hi, we have an Informix Server 11.5 UC7E running under Debian Linux. The system ran fine in no-logging-mode for several years until we switch to logging-mode to support a new application. Since then we experience crashes - the database dies with the following messages from the online.log: 18:09:34 Assert fehlgeschlagen: Bedingung nicht erfüllt (Bad block pointer 0x893fe2c in pool 0x28632028), In (mt_shm_free) 18:09:34 IBM Informix Dynamic Server Version 11.50.UC7E 18:09:34 Wer: Session(137915, hoffmann@tefix, 6734, 0x38e64e80) Thread(141348, sqlexec, 3ede9f40, 1) File: mtshpool.c Line: 4193 18:09:34 Aktion: Bitte informieren Sie die technische Unterstützung für IBM Informix. 18:09:34 stack trace for pid 8244 written to /opt/informix/tmp/af.2c0cc14d 18:09:34 Siehe auch: /opt/informix/tmp/af.2c0cc14d 18:09:36 Bedingung nicht erfüllt (Bad block pointer 0x893fe2c in pool 0x28632028), In (mt_shm_free) 18:09:36 stack trace for pid 8244 written to /opt/informix/tmp/af.2c0cc14d 18:09:36 Assert fehlgeschlagen: No Exception Handler 18:09:36 IBM Informix Dynamic Server Version 11.50.UC7E 18:09:36 Wer: Session(137915, hoffmann@tefix, 6734, 0x38e64e80) Thread(141348, sqlexec, 3ede9f40, 1) File: mtex.c Line: 491 18:09:36 Ergebnisse: Exception Caught. Type: MT_EX_OS, Context: mem 18:09:36 Aktion: Bitte informieren Sie die technische Unterstützung für IBM Informix. 18:09:36 Siehe auch: /opt/informix/tmp/af.2c0cc14d 18:09:38 Starting crash time check of: 18:09:38 1. memory block headers 18:09:38 2. stacks 18:09:38 Found bad memory block; address:29acfc88 18:09:38 mtex.c, line 491, thread 141348, proc id 8244, No Exception Handler. 18:09:39 Der Master-Dämon ist nicht mehr aktiv 18:09:39 PANIC: Attempting to bring system down I am not an Informix expert, but responsible for the system. The internet is not helping with these error messages. I don´t know if this is important or has anything to do with the problem, but we have this message regularly: 08:28:44 Contiguous shared memory segment allocation failed at 0xb78ee000. Allocation successful at 0x43800000. Check SHMBASE is consistent with the value in $INFORMIXDIR/etc/onconfig.std. If you are using the correct SHMBASE value in your ONCONFIG file, then consider this message informational only. 08:28:44 Dynamically allocated new virtual shared memory segment (size 8192KB) 08:28:44 Memory sizes:resident:1761624 KB, virtual:139872 KB, no SHMTOTAL limit Short extract from our onconfig-file: SHMBASE 0x44000000L SHMVIRTSIZE 32656 SHMADD 8192 EXTSHMADD 8192 SHMTOTAL 0 SHMVIRT_ALLOCSEG 0.000000 SHMNOACCESS Can anybody help me or can give me an advice how to solve this problem? Or is anybody experiencing similiar problems? Thanks for reading my long post :-) Klaus ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum. IT Handel und Beratung Jörg Volz Bernhard-Früh-Str. 7 77855 Achern GERMANY Tel: +49 (0)7841-681651 Fax: +49 (0)7841-681654 Mobil: +49 (0)170-2989757 VAT-ID: DE201383541 http://www.it-volz.de

Hi, looks like a shared memory problem. The max SHMEM addressable under Debian (I assume 32bit) is about 2.7 GB. If you run into problems below this mark, check your shared memory parameters. It also depends on the running kernel. If you are using a bigmem kernel, there can be more shared memory than in a standard kernel. Running the 32bit version in a 64bit env with ia32 is also possible and gives you about 3.4 GB of shared memory adressable because the shared libs are loaded in a higher segment there. This leaves more space for shared memory. I assume you have done a transaction which requires lots of memory, otherwise your memory would not increase to 1.3 GB. Maybe you get into the problem because the number of segments is too high. Try the following (depending on your machine configuration, should be 3GB min): SHMVIRTSIZE 800000 # initial virtual shared memory segmentsize SHMADD 150000 # Size of new shared memory segments (Kbytes) that way the machine allocates 800MB initially and increases shared memory in 150MB steps. (your setup is 32K initial and 8K increase, which leads to a lot of segments when 1.3GB is filled). The reached memory limit address looks like the begin of the shared libraries on a standard kernel. You can also try to lower the SHMBASE parameter to get more free mem for shared memory. Lowest is 0x10000000, which we hav done on a 64bit system running ia32. This should be tried first ! (look in /proc/<processid of first oninit/maps to see what is allocatable) You can see from the error messages that IDS tries to find mem below the given SHMBASE. (0x43800000) Check on a separate system how memory size can be allocated with a dummy instance (start system given the parameters above and allocate shared memory with onmode -a 150000until you get an error message). Check onstat -g seg in your active instance. It is not forbidden to have multiple SHM Segments but you should monitor them. Warning: if you set too high shared memory initially, we once ran into a problem with backup via shared memory (might be solved). For us, 800000 is a working value. Hope this helps, Marcus -----Original Message----- From: KLAUS GLAHN [mailto:glahn@berlinale.de] Sent: Tuesday, December 20, 2011 8:07 PM To: ids@iiug.org Subject: 11.5 UC7E crashes [25681] Hi, we have an Informix Server 11.5 UC7E running under Debian Linux. The system ran fine in no-logging-mode for several years until we switch to logging-mode to support a new application. Since then we experience crashes - the database dies with the following messages from the online.log: 18:09:34 Assert fehlgeschlagen: Bedingung nicht erfllt (Bad block pointer 0x893fe2c in pool 0x28632028), In (mt_shm_free) 18:09:34 IBM Informix Dynamic Server Version 11.50.UC7E 18:09:34 Wer: Session(137915, hoffmann@tefix, 6734, 0x38e64e80) Thread(141348, sqlexec, 3ede9f40, 1) File: mtshpool.c Line: 4193 18:09:34 Aktion: Bitte informieren Sie die technische Untersttzung fr IBM Informix. 18:09:34 stack trace for pid 8244 written to /opt/informix/tmp/af.2c0cc14d 18:09:34 Siehe auch: /opt/informix/tmp/af.2c0cc14d 18:09:36 Bedingung nicht erfllt (Bad block pointer 0x893fe2c in pool 0x28632028), In (mt_shm_free) 18:09:36 stack trace for pid 8244 written to /opt/informix/tmp/af.2c0cc14d 18:09:36 Assert fehlgeschlagen: No Exception Handler 18:09:36 IBM Informix Dynamic Server Version 11.50.UC7E 18:09:36 Wer: Session(137915, hoffmann@tefix, 6734, 0x38e64e80) Thread(141348, sqlexec, 3ede9f40, 1) File: mtex.c Line: 491 18:09:36 Ergebnisse: Exception Caught. Type: MT_EX_OS, Context: mem 18:09:36 Aktion: Bitte informieren Sie die technische Untersttzung fr IBM Informix. 18:09:36 Siehe auch: /opt/informix/tmp/af.2c0cc14d 18:09:38 Starting crash time check of: 18:09:38 1. memory block headers 18:09:38 2. stacks 18:09:38 Found bad memory block; address:29acfc88 18:09:38 mtex.c, line 491, thread 141348, proc id 8244, No Exception Handler. 18:09:39 Der Master-Dmon ist nicht mehr aktiv 18:09:39 PANIC: Attempting to bring system down I am not an Informix expert, but responsible for the system. The internet is not helping with these error messages. I dont know if this is important or has anything to do with the problem, but we have this message regularly: 08:28:44 Contiguous shared memory segment allocation failed at 0xb78ee000. Allocation successful at 0x43800000. Check SHMBASE is consistent with the value in $INFORMIXDIR/etc/onconfig.std. If you are using the correct SHMBASE value in your ONCONFIG file, then consider this message informational only. 08:28:44 Dynamically allocated new virtual shared memory segment (size 8192KB) 08:28:44 Memory sizes:resident:1761624 KB, virtual:139872 KB, no SHMTOTAL limit Short extract from our onconfig-file: SHMBASE 0x44000000L SHMVIRTSIZE 32656 SHMADD 8192 EXTSHMADD 8192 SHMTOTAL 0 SHMVIRT_ALLOCSEG 0.000000 SHMNOACCESS Can anybody help me or can give me an advice how to solve this problem? Or is anybody experiencing similiar problems? Thanks for reading my long post :-) Klaus ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum.

Hello, thank you for your answers. I already assumed that it´s a memory problem. But with my limited knowledge (and I have the Eric-Herber-book ;-) it´s already difficult to ask the right questions. We have a test instance and I tried SHMVIRTSIZE 800000 SHMADD 150000 but when I do oninit I get an error message 20:20:33 IBM Informix Dynamic Server gestartet. 20:20:33 shmget: [EEXIST][17]: key 52574802: shared memory already exists 20:20:33 mt_shm_init: Segment resident kann nicht angelegt werden After that the instance couldn´t start until I changed the SERVERNUM-Parameter to another number (thanks to Google :-) I don´t know how to clean this up, I tried onclean - but this doesn´t help. Is reboot the only solution? As far as I know (I didn´t install the machine, there is little documentation) it´s a 32-Bit-Informix running on Debian with Bigmem-Extension. The crashes seem real random - but we have approx. 100 users working with the database with different applications - and some of them using MS-Access and can do large transactions. /proc/<pid>/maps is empty for all my oninit-processes Should I try lower values for SHMVIRTSIZE? I don´t want to experiment, because I don´t want to change the SERVERNUM after each failed test. Sorry, I am really a newbie to the Informix internals. The last years everything was fine - until we wanted logging ... Thanks again for your help. All the best Klaus

Running "onmode -ky" will usually destroy any left over shared memory segments from the original instance that crashed (assuming you don't change the SERVERNUM of course). Also in the most recent releases (don't know the first one) there is a script, onclean, in $INFORMIXDIR/bin that basically does that then uses ipcrm to physically remove any segments that happen to survive. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Wed, Dec 21, 2011 at 2:40 PM, KLAUS GLAHN <glahn@berlinale.de> wrote: > Hello, > > thank you for your answers. I already assumed that it´s a memory problem. > But > with my limited knowledge (and I have the Eric-Herber-book ;-) it´s already > difficult to ask the right questions. > > We have a test instance and I tried > > SHMVIRTSIZE 800000 > SHMADD 150000> > but when I do oninit I get an error message > > 20:20:33 IBM Informix Dynamic Server gestartet. > 20:20:33 shmget: [EEXIST][17]: key 52574802: shared memory already exists > 20:20:33 mt_shm_init: Segment resident kann nicht angelegt werden > > After that the instance couldn´t start until I changed the > SERVERNUM-Parameter > to another number (thanks to Google :-) I don´t know how to clean this up, > I > tried onclean - but this doesn´t help. Is reboot the only solution? > > As far as I know (I didn´t install the machine, there is little > documentation) > it´s a 32-Bit-Informix running on Debian with Bigmem-Extension. The crashes > seem real random - but we have approx. 100 users working with the database > with different applications - and some of them using MS-Access and can do > large transactions. > > /proc/<pid>/maps is empty for all my oninit-processes > > Should I try lower values for SHMVIRTSIZE? I don´t want to experiment, > because > I don´t want to change the SERVERNUM after each failed test. > > Sorry, I am really a newbie to the Informix internals. The last years > everything was fine - until we wanted logging ... Thanks again for your > help. > > All the best > Klaus > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --bcaec51a8672aa816f04b49f92ff

Hallo Klaus, sometimes the oninit process does not terminate correct or fast enough. = Take a look with ps for running informix processes, may be you need to = kill them, or reboot the machine (always the cleanest solution). How much memory has the machine? A 100 users is not so much, but Access = as client is a bitch, especially when users kill Access because they = have to wait for longer transactions. Mfg Joerg Volz Am 21.12.2011 um 20:46 schrieb "KLAUS GLAHN" <glahn@berlinale.de>: > Hello,=20 >=20 > thank you for your answers. I already assumed that it=C3=82=C2=B4s a = memory problem. But=20 > with my limited knowledge (and I have the Eric-Herber-book ;-) = it=C3=82=C2=B4s already=20 > difficult to ask the right questions.=20 >=20 > We have a test instance and I tried=20 >=20 > SHMVIRTSIZE 800000=20 > SHMADD 150000=20 >=20> but when I do oninit I get an error message=20 >=20 > 20:20:33 IBM Informix Dynamic Server gestartet.=20 > 20:20:33 shmget: [EEXIST][17]: key 52574802: shared memory already = exists=20 > 20:20:33 mt_shm_init: Segment resident kann nicht angelegt werden=20 >=20 > After that the instance couldn=C3=82=C2=B4t start until I changed the = SERVERNUM-Parameter=20 > to another number (thanks to Google :-) I don=C3=82=C2=B4t know how to = clean this up, I=20 > tried onclean - but this doesn=C3=82=C2=B4t help. Is reboot the only = solution?=20 >=20 > As far as I know (I didn=C3=82=C2=B4t install the machine, there is = little documentation)=20 > it=C3=82=C2=B4s a 32-Bit-Informix running on Debian with = Bigmem-Extension. The crashes=20 > seem real random - but we have approx. 100 users working with the = database=20 > with different applications - and some of them using MS-Access and can = do=20 > large transactions.=20 >=20 > /proc/<pid>/maps is empty for all my oninit-processes=20 >=20 > Should I try lower values for SHMVIRTSIZE? I don=C3=82=C2=B4t want to = experiment, because=20 > I don=C3=82=C2=B4t want to change the SERVERNUM after each failed = test.=20 >=20 > Sorry, I am really a newbie to the Informix internals. The last years=20 > everything was fine - until we wanted logging ... Thanks again for = your help.=20 >=20 > All the best=20 > Klaus=20 >=20 >=20 > = *************************************************************************= ******=20 > Forum Note: Use "Reply" to post a response in the discussion forum.=20 >=20 >=20 >=20 IT Handel und Beratung J=C3=B6rg Volz Bernhard-Fr=C3=BCh-Str. 7 77855 Achern GERMANY Tel: +49 (0)7841-681651 Fax: +49 (0)7841-681654 Mobil: +49 (0)170-2989757 VAT-ID: DE201383541 http://www.it-volz.de

'onclean' sounds like a nice utility to be included. If it's not there, the old manual method is to check for leftover SHM segments by running 'ipcs' as root. If there are any segments that are (co-) owned by informix, they need to be removed with 'ipcrm'. As far as /proc/<pid>/maps goes, I think you only need to look at it for the first (adm cpu) oninit. If Debian doesn't do that (Redhat does) then you are stuck experimenting with different values for SHMBASE to juggle where all the runtime libraries get loaded. As long as you clean up the SHM segments, there is no need to keep changing SERVERNUM after each test or crash. *** WARNING *** There are different SHM segments associated with each SERVERNUM. If you have more than one instance on a server, you don't want to kill the wrong segments! Bob ----- Original Message ----- From: "Art Kagel" <art.kagel@gmail.com> To: ids@iiug.org Sent: Wednesday, December 21, 2011 2:56:05 PM Subject: Re: 11.5 UC7E crashes [25701] Running "onmode -ky" will usually destroy any left over shared memory segments from the original instance that crashed (assuming you don't change the SERVERNUM of course). Also in the most recent releases (don't know the first one) there is a script, onclean, in $INFORMIXDIR/bin that basically does that then uses ipcrm to physically remove any segments that happen to survive. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Wed, Dec 21, 2011 at 2:40 PM, KLAUS GLAHN <glahn@berlinale.de> wrote: > Hello, > > thank you for your answers. I already assumed that its a memory problem. > But > with my limited knowledge (and I have the Eric-Herber-book ;-) its already > difficult to ask the right questions. > > We have a test instance and I tried > > SHMVIRTSIZE 800000 > SHMADD 150000> > but when I do oninit I get an error message > > 20:20:33 IBM Informix Dynamic Server gestartet. > 20:20:33 shmget: [EEXIST][17]: key 52574802: shared memory already exists > 20:20:33 mt_shm_init: Segment resident kann nicht angelegt werden > > After that the instance couldnt start until I changed the > SERVERNUM-Parameter > to another number (thanks to Google :-) I dont know how to clean this up, > I > tried onclean - but this doesnt help. Is reboot the only solution? > > As far as I know (I didnt install the machine, there is little > documentation) > its a 32-Bit-Informix running on Debian with Bigmem-Extension. The crashes > seem real random - but we have approx. 100 users working with the database > with different applications - and some of them using MS-Access and can do > large transactions. > > /proc/<pid>/maps is empty for all my oninit-processes > > Should I try lower values for SHMVIRTSIZE? I dont want to experiment, > because > I dont want to change the SERVERNUM after each failed test. > > Sorry, I am really a newbie to the Informix internals. The last years > everything was fine - until we wanted logging ... Thanks again for your > help. > > All the best > Klaus > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --bcaec51a8672aa816f04b49f92ff ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

Hi Joerg, reboot is not an option tonight :-( There are other applications running on the server, which I can´t interrupt now. I tried onmode -ky: didn´t help I tried onclean: found no problems and didn´t help I tried ipcs: no informix segments I tried ps -e: no oninit processes but when I change my SERVERNUM back to the original value 0, I can´t start the instance: 23:35:19 IBM Informix Dynamic Server gestartet. 23:35:19 shmget: [EEXIST][17]: key 52564801: shared memory already exists 23:35:19 mt_shm_init: Segment resident kann nicht angelegt werden With ipcs I can see the segment: 0x52564801 11272195 root 660 33554432 0 but it belongs to root? I guess I have to reboot tomorrow. And then I can switch back to the original problem ... Thanks again :-) Klaus

Hey Bob, hey Art we have only one instance, so this is no problem :-) As I wrote here: http://www.iiug.org/forums/ids/index.cgi/read/25704 I didn´t get Informix to clean up everything. I tried to switch back to the original SERVERNUM, but I can´t start the instance than. Perhaps reboot is my only choice. But this I can only do tomorrow night. Thanks to the both of you Klaus

>With ipcs I can see the segment: >0x52564801 11272195 root 660 33554432 0 >but it belongs to root? Who is starting the original instance? If IDS comes up during boot time (/etc/rc.d/informix or some mechanism like that) it´s likely that the "oldest" IDS process is owned by root and not informix. Mit freundlichen Grüßen Joerg Volz ----------------------------------------------------------------------------- -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of KLAUS GLAHN Sent: Thursday, December 22, 2011 12:00 AM To: ids@iiug.org Subject: Re: 11.5 UC7E crashes [25704] Hi Joerg, reboot is not an option tonight :-( There are other applications running on the server, which I can´t interrupt now. I tried onmode -ky: didn´t help I tried onclean: found no problems and didn´t help I tried ipcs: no informix segments I tried ps -e: no oninit processes but when I change my SERVERNUM back to the original value 0, I can´t start the instance: 23:35:19 IBM Informix Dynamic Server gestartet. 23:35:19 shmget: [EEXIST][17]: key 52564801: shared memory already exists 23:35:19 mt_shm_init: Segment resident kann nicht angelegt werden With ipcs I can see the segment: 0x52564801 11272195 root 660 33554432 0 but it belongs to root? I guess I have to reboot tomorrow. And then I can switch back to the original problem ... Thanks again :-) Klaus ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum. IT Handel und Beratung Jörg Volz Bernhard-Früh-Str. 7 77855 Achern GERMANY Tel: +49 (0)7841-681651 Fax: +49 (0)7841-681654 Mobil: +49 (0)170-2989757 VAT-ID: DE201383541 http://www.it-volz.de

Did you change SERVERNUM back BEFORE you tried onmode & onclean? Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) Blog: http://informix-myview.blogspot.com/ Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Wed, Dec 21, 2011 at 5:50 PM, KLAUS GLAHN <glahn@berlinale.de> wrote: > Hi Joerg, > > reboot is not an option tonight :-( There are other applications running on > the server, which I can´t interrupt now. > > I tried onmode -ky: didn´t help > I tried onclean: found no problems and didn´t help > I tried ipcs: no informix segments > I tried ps -e: no oninit processes > > but when I change my SERVERNUM back to the original value 0, I can´t start > the > instance: > > 23:35:19 IBM Informix Dynamic Server gestartet. > 23:35:19 shmget: [EEXIST][17]: key 52564801: shared memory already exists > 23:35:19 mt_shm_init: Segment resident kann nicht angelegt werden > > With ipcs I can see the segment: > 0x52564801 11272195 root 660 33554432 0 > but it belongs to root? > > I guess I have to reboot tomorrow. > > And then I can switch back to the original problem ... > > Thanks again :-) > Klaus > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > --14dae9340f613b97d104b4a3442e

Hi Klaus, If you are trying to start the informix engine again, it will not start if one or more of the shared memory segments is still used (check ipcs). Can happen after a crash and the the segments have been removed when the engine stopped. Each instance of informix is based on the value of SERVERNUM (value between 0 and 255 possbile). So either change the value of SERVERNUM and start you engine with "oninit -v" until the next reboot of your server ;You will be wasting a segment of memory or run ipcrm -m <shmid> using the shared memory id of the segments used for that instance: *need to be root to run ipcrm* If you have severval shared memory segments , you can run ipcrm -m for each segment or ipcrm -m <shmid> -m <shmid of the second segment> ... etc Note: the shared memory id is NOT the key (0x52564801) but the shmid (11272195) You do not have to reboot if you can remove your shared memory segments via ipcrm being root I hope that this helps you start you engine again. Cordialement, Regards, Khaled Bentebal Directeur Général - ConsultiX Président UGIF - User Group Informix France IIUG - Board of Directors Tél: 33 (0) 1 39 12 18 00 Fax: 33 (0) 1 39 12 18 18 Mobile: 33 (0) 6 07 78 41 97 Email: khaled.bentebal@consult-ix.fr Site Web: www.consult-ix.fr Le 21/12/11 23:50, KLAUS GLAHN a écrit : > Hi Joerg, > > reboot is not an option tonight :-( There are other applications running on > the server, which I can´t interrupt now. > > I tried onmode -ky: didn´t help > I tried onclean: found no problems and didn´t help > I tried ipcs: no informix segments > I tried ps -e: no oninit processes > > but when I change my SERVERNUM back to the original value 0, I can´t start the > instance: > > 23:35:19 IBM Informix Dynamic Server gestartet. > 23:35:19 shmget: [EEXIST][17]: key 52564801: shared memory already exists > 23:35:19 mt_shm_init: Segment resident kann nicht angelegt werden > > With ipcs I can see the segment: > 0x52564801 11272195 root 660 33554432 0 > but it belongs to root? > > I guess I have to reboot tomorrow. > > And then I can switch back to the original problem ... > > Thanks again :-) > Klaus > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > >

On Thu, Dec 22, 2011 at 3:04 AM, Khaled Bentebal < khaled.bentebal@consult-ix.fr> wrote: > Hi Klaus, > > If you are trying to start the informix engine again, it will not start > if one or more of the shared memory segments is still used (check ipcs). > Can happen after a crash and the the segments have been removed when the > engine stopped. > > Each instance of informix is based on the value of SERVERNUM (value > between 0 and 255 possbile). > > So either change the value of SERVERNUM and start you engine with > "oninit -v" until the next reboot of your server ;You will be wasting a > segment of memory > I'm sorry if I'm missing the context, since I haven't been following the thread, but this can be a VERY DANGEROUS thing to do... I've seen customers destroying instances because they used this "trick". If by any chance the old oninits are still there, and you change the SERVERNUM you may end up with two running instances on top of the same data. This will corrupt the data for sure. Before the listeners could be started/stop (around 11.50.FC5) the "second" instance would not come up, because it would not be able to grab the TCP ports. But currently that's not the case, and the instance will come online even without the listerner(s) So, thonk several times before you change the SERVERNUM to solve anything. It can be done if you make sure no more oninits or other informix tools are running. Regards. or > run ipcrm -m <shmid> using the shared memory id of the segments used for > that instance: *need to be root to run ipcrm* > > If you have severval shared memory segments , you can run ipcrm -m for > each segment or ipcrm -m <shmid> -m <shmid of the second segment> ... etc > > Note: the shared memory id is NOT the key (0x52564801) but the shmid > (11272195) > > You do not have to reboot if you can remove your shared memory segments > via ipcrm being root > > I hope that this helps you start you engine again. > > Cordialement, Regards, > > Khaled Bentebal > Directeur Général - ConsultiX > Président UGIF - User Group Informix France > IIUG - Board of Directors > Tél: 33 (0) 1 39 12 18 00 > Fax: 33 (0) 1 39 12 18 18 > Mobile: 33 (0) 6 07 78 41 97 > Email: khaled.bentebal@consult-ix.fr > Site Web: www.consult-ix.fr > > Le 21/12/11 23:50, KLAUS GLAHN a écrit : > > Hi Joerg, > > > > reboot is not an option tonight :-( There are other applications running > on > > the server, which I can´t interrupt now. > > > > I tried onmode -ky: didn´t help > > I tried onclean: found no problems and didn´t help > > I tried ipcs: no informix segments > > I tried ps -e: no oninit processes > > > > but when I change my SERVERNUM back to the original value 0, I can´t > start > the > > instance: > > > > 23:35:19 IBM Informix Dynamic Server gestartet. > > 23:35:19 shmget: [EEXIST][17]: key 52564801: shared memory already exists > > 23:35:19 mt_shm_init: Segment resident kann nicht angelegt werden > > > > With ipcs I can see the segment: > > 0x52564801 11272195 root 660 33554432 0 > > but it belongs to root? > > > > I guess I have to reboot tomorrow. > > > > And then I can switch back to the original problem ... > > > > Thanks again :-) > > Klaus > > > > > > > > ******************************************************************************* > > Forum Note: Use "Reply" to post a response in the discussion forum. > > > > > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > > -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently... --20cf3074b1b416a23b04b4a5e306

Hallo Klaus, you got already some (new) answers, so here I'll only write down some of my thoughts that may not have been covered by those previous answers ... > ... tried onclean, but this doesn't help. ... I didn't check out the source code of this utility, but there can't be too much magic for what it is doing (or trying to do). Does the onclean show any error or warning message? > /proc/<pid>/maps is empty for all my oninit-processes What exactly do you mean by "empty" ? After the server crashed, when the oninit processes have died, there should not even be a /proc/<pid> directory anymore for those processes, as they don't exist anymore. If a process (still) exists, its maps file in /proc/<pid> should never be empty (i.,e. contain nothing at all), as any process will at least occupy some memory, if it is only the TEXT portion (i.e. the program code itself, that is loaded into memory). These two statements from you ... somehow do not fit the picture, that is expected of a Linux system ... hmm. My guess is, that when there is the situation of "shmget: [EEXIST][17]: key 52574802: shared memory already exists", there is still some (old) process "hanging around" and still attached to that shared memory segment. In this case, the onclean utility will not be able to remove the SHM segment, because Linux doesn't allow it. Also as user "root" using the ipcrm system utility you will not be able to remove an SHM segment to which some process is still attached. Using "ipcs -m" as user root, you should be able to identify that SHM segment (using the key). The output of "ipcs -m" usually has a column named "nattch" which means "number of processes attached". It is likely non-zero (otherwise onclean would have removed it already). Possibly the column "status" says something like "destroyed", which means that the Linux OS will automatically remove it once the last process has detached from it. Unfortunately, "ipcs" doesn't tell, which process (pid) is still attached. :-( So the task is to find and kill that process (or those processes). There are two possibilities: a) There is an (old) oninit process left over, that is still attached to the SHM segment. If this Informix Server instance is the only one running on this machine, then simply do a command like "ps -efw | grep oninit" to see, if there is an oninit process still (half) alive. If you identify some oninit processes that should not be there anymore: kill them as user "root": "kill -9 <pid> [ <pid> ... ]" In case you have other Informix Server instances running on this machine, set the respective environment for each instance and do "onstat -g sch". The output will include a listing of all the associated oninit processes with their PID. These PIDs should not be killed with above "kill -9 ..." command ... :) b) Some client process (i.e. not an oninit process) is still running and attached to the SHM segment. This could e.g. be an onbar process (try "ps -efw | grep onbar"). In case you have configured SHM communication between clients and Informix Server, it could be any local client process that has connected to the server using SHM for communication. Check in the sqlhosts file, whether there is an entry with "onipcshm" as the second parameter. Check for any client process (you have to know or figure out, what could be their name ... usually "dbaccess" is one candidate for running ad hoc test queries), using the same "ps -efw | grep ..." method. If you find something ... kill it as well, using "kill -9 <pid>". Once all process are gone (have been killed), the SHM segments should have been removed. In a rare case where a segment could still be left over, again running onclean should do the job. Regards, Martin -- Martin Fuerderer IBM Informix Development Munich, Germany Information Management Read about the Informix Warehouse Accelerator: http://tinyurl.com/the-iwa-blog IBM Deutschland Research & Development GmbH Chairman of the Supervisory Board: Martina Koederitz Board of Management: Dirk Wittkopp Corporate Seat: Boeblingen, Germany Reg.-Gericht: Amtsgericht Stuttgart, HRB 243294 ids-bounces@iiug.org wrote on 12/21/2011 08:40:45 PM: > Hello, > > thank you for your answers. I already assumed that it´s a memory problem. But > with my limited knowledge (and I have the Eric-Herber-book ;-) it´s already > difficult to ask the right questions. > > We have a test instance and I tried > > SHMVIRTSIZE 800000 > SHMADD 150000> > but when I do oninit I get an error message > > 20:20:33 IBM Informix Dynamic Server gestartet. > 20:20:33 shmget: [EEXIST][17]: key 52574802: shared memory already exists > 20:20:33 mt_shm_init: Segment resident kann nicht angelegt werden > > After that the instance couldn´t start until I changed the SERVERNUM-Parameter > to another number (thanks to Google :-) I don´t know how to clean this up, I > tried onclean - but this doesn´t help. Is reboot the only solution? > > As far as I know (I didn´t install the machine, there is little documentation) > it´s a 32-Bit-Informix running on Debian with Bigmem-Extension. The crashes > seem real random - but we have approx. 100 users working with the database > with different applications - and some of them using MS-Access and can do > large transactions. > > /proc/<pid>/maps is empty for all my oninit-processes > > Should I try lower values for SHMVIRTSIZE? I don´t want to experiment, because > I don´t want to change the SERVERNUM after each failed test. > > Sorry, I am really a newbie to the Informix internals. The last years > everything was fine - until we wanted logging ... Thanks again for your help. > > All the best > Klaus > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. >

You can get shared memory id for your particular server number by below= calculation. example : my servernum : 105 ( onstat -c | grep SERVERNUM) hexadecimal of 105 =3D 69 Shared memory id's for server num 105 is Hexadecimal add ( 69+56) =3D B= F user informix ipcs -m | grep "bf" 0x52bf4801 184582176 root 660 33554432 12 0x52bf4802 184614945 root 660 33554432 12 0x52bf4803 184647714 root 660 33554432 12 0x52bf4804 184680483 root 660 18694144 12 0x52bf4805 184713252 root 660 33439744 12 0x52bf4806 184746021 root 660 8388608 12 Here you get shared memory allocated. based on this you can remove. user root ipcrm -m <shared memory id> Thanks, Harsha From: "Martin Fuerderer" <MARTINFU@de.ibm.com> To: ids@iiug.org Date: 12/22/2011 03:48 PM Subject: Re: 11.5 UC7E crashes [25711] Sent by: ids-bounces@iiug.org Hallo Klaus, you got already some (new) answers, so here I'll only write down some of my thoughts that may not have been covered by those previous answers ... > ... tried onclean, but this doesn't help. ... I didn't check out the source code of this utility, but there can't be too much magic for what it is doing (or trying to do). Does the onclean show any error or warning message? > /proc/<pid>/maps is empty for all my oninit-processes What exactly do you mean by "empty" ? After the server crashed, when the oninit processes have died, there should not even be a /proc/<pid> directory anymore for those processes, as they don't exist anymore. If a process (still) exists, its maps file in /proc/<pid> should never be empty (i.,e. contain nothing at all), as any process will at least occupy some memory, if it is only the TEXT portion (i.e. the program code itself, that is loaded into memory). These two statements from you ... somehow do not fit the picture, that is expected of a Linux system ... hmm. My guess is, that when there is the situation of "shmget: [EEXIST][17]: key 52574802: shared memory already exists", there is still some (old) process "hanging around" and still attached to that shared memory segment. In this case, the onclean utility will not be able to remove the SHM segment, because Linux doesn't allow it. Also as user "root" using the ipcrm system utility you will not be able to remove an SHM segment to which some process is still attached. Using "ipcs -m" as user root, you should be able to identify that SHM segment (using the key). The output of "ipcs -m" usually has a column named "nattch" which means "number of processes attached". It is likely non-zero (otherwise onclean would have removed it already). Possibly the column "status" says something like "destroyed", which means that the Linux OS will automatically remove it once the last process has detached from it. Unfortunately, "ipcs" doesn't tell, which process (pid) is still attached. :-( So the task is to find and kill that process (or those processes). There are two possibilities: a) There is an (old) oninit process left over, that is still attached to the SHM segment. If this Informix Server instance is the only one running on this machine, then simply do a command like "ps -efw | grep oninit" to see, if there is an oninit process still (half) alive. If you identify some oninit processes that should not be there anymore: kill them as user "root": "kill -9 <pid> [ <pid> ... ]" In case you have other Informix Server instances running on this machine, set the respective environment for each instance and do "onstat -g sch". The output will include a listing of all the associated oninit processes with their PID. These PIDs should not be killed with above "kill -9 ..." command ... :) b) Some client process (i.e. not an oninit process) is still running and attached to the SHM segment. This could e.g. be an onbar process (try "ps -efw | grep onbar"). In case you have configured SHM communication between clients and Informix Server, it could be any local client process that has connected to the server using SHM for communication. Check in the sqlhosts file, whether there is an entry with "onipcshm" as the second parameter. Check for any client process (you have to know or figure out, what could be their name ... usually "dbaccess" is one candidate for running ad hoc test queries), using the same "ps -efw | grep ..." method. If you find something ... kill it as well, using "kill -9 <pid>". Once all process are gone (have been killed), the SHM segments should have been removed. In a rare case where a segment could still be left over, again running onclean should do the job. Regards, Martin -- Martin Fuerderer IBM Informix Development Munich, Germany Information Management Read about the Informix Warehouse Accelerator: http://tinyurl.com/the-iwa-blog IBM Deutschland Research & Development GmbH Chairman of the Supervisory Board: Martina Koederitz Board of Management: Dirk Wittkopp Corporate Seat: Boeblingen, Germany Reg.-Gericht: Amtsgericht Stuttgart, HRB 243294 ids-bounces@iiug.org wrote on 12/21/2011 08:40:45 PM: > Hello, > > thank you for your answers. I already assumed that it=B4s a memory problem. But > with my limited knowledge (and I have the Eric-Herber-book ;-) it=B4s= already > difficult to ask the right questions. > > We have a test instance and I tried > > SHMVIRTSIZE 800000 > SHMADD 150000> > but when I do oninit I get an error message > > 20:20:33 IBM Informix Dynamic Server gestartet. > 20:20:33 shmget: [EEXIST][17]: key 52574802: shared memory already exists > 20:20:33 mt_shm_init: Segment resident kann nicht angelegt werden > > After that the instance couldn=B4t start until I changed the SERVERNUM-Parameter > to another number (thanks to Google :-) I don=B4t know how to clean t= his up, I > tried onclean - but this doesn=B4t help. Is reboot the only solution?= > > As far as I know (I didn=B4t install the machine, there is little documentation) > it=B4s a 32-Bit-Informix running on Debian with Bigmem-Extension. The= crashes > seem real random - but we have approx. 100 users working with the database > with different applications - and some of them using MS-Access and ca= n do > large transactions. > > /proc/<pid>/maps is empty for all my oninit-processes > > Should I try lower values for SHMVIRTSIZE? I don=B4t want to experime= nt, because > I don=B4t want to change the SERVERNUM after each failed test. > > Sorry, I am really a newbie to the Informix internals. The last years= > everything was fine - until we wanted logging ... Thanks again for yo= ur help. > > All the best > Klaus > > > ***********************************************************************= ******** > Forum Note: Use "Reply" to post a response in the discussion forum. > *********************************

*In this case there is one instance having the problem and none of the oninits are running.* On the the other hand, if you start another instance based on a different SERVERNUM, it will start ONLY if the new SERVERNUM is unique since it tries to use the *shmget* function which has the followinf signature or definition*(int shmget(key_t key, size_t size, int shmflg).* If *shmflg* specifies both IPC_CREAT and IPC_EXCL and a shared memory segment already exists for key, then shmget() fails with errno set to EEXIST. (This is analogous to the effect of the combination O_CREAT | O_EXCL for open()) The new instance cannot come up if you use the same SERVERNUM. Informix uses an algorithm that calculates the key based on the hexadecimal value for "RVH" (the initials of the architect of Informix Turbo for information) and uses the value of SERVERNUM to calculate the first 3 bytes of the Key. The remaining byte is 01, 02 , 03, etc which is the 1st segment, 2nd segment, etc for that instance. So the new memory segments cannot be the same. You cannot work on the same data. For safety reasons, I always make sure to get rid of all of the segments related to an instance if possible and for sure all of the oninits related to that instance. Cordialement, Regards, Khaled Bentebal Directeur Général - ConsultiX Président UGIF - User Group Informix France IIUG - Board of Directors Tél: 33 (0) 1 39 12 18 00 Fax: 33 (0) 1 39 12 18 18 Mobile: 33 (0) 6 07 78 41 97 Email: khaled.bentebal@consult-ix.fr Site Web: www.consult-ix.fr Le 22/12/11 04:28, Fernando Nunes a écrit : > On Thu, Dec 22, 2011 at 3:04 AM, Khaled Bentebal< > khaled.bentebal@consult-ix.fr> wrote: > >> Hi Klaus, >> >> If you are trying to start the informix engine again, it will not start >> if one or more of the shared memory segments is still used (check ipcs). >> Can happen after a crash and the the segments have been removed when the >> engine stopped. >> >> Each instance of informix is based on the value of SERVERNUM (value >> between 0 and 255 possbile). >> >> So either change the value of SERVERNUM and start you engine with >> "oninit -v" until the next reboot of your server ;You will be wasting a >> segment of memory >> > I'm sorry if I'm missing the context, since I haven't been following the > thread, but this can be a VERY DANGEROUS thing to do... > I've seen customers destroying instances because they used this "trick". If > by any chance the old oninits are still there, and you change the SERVERNUM > you may end up with two running instances on top of the same data. This > will corrupt the data for sure. > > Before the listeners could be started/stop (around 11.50.FC5) the "second" > instance would not come up, because it would not be able to grab the TCP > ports. But currently that's not the case, and the instance will come online > even without the listerner(s) > > So, thonk several times before you change the SERVERNUM to solve anything. > It can be done if you make sure no more oninits or other informix tools are > running. > > Regards. > > or >> run ipcrm -m<shmid> using the shared memory id of the segments used for >> that instance: *need to be root to run ipcrm* >> >> If you have severval shared memory segments , you can run ipcrm -m for >> each segment or ipcrm -m<shmid> -m<shmid of the second segment> ... etc >> >> Note: the shared memory id is NOT the key (0x52564801) but the shmid >> (11272195) >> >> You do not have to reboot if you can remove your shared memory segments >> via ipcrm being root >> >> I hope that this helps you start you engine again. >> >> Cordialement, Regards, >> >> Khaled Bentebal >> Directeur Général - ConsultiX >> Président UGIF - User Group Informix France >> IIUG - Board of Directors >> Tél: 33 (0) 1 39 12 18 00 >> Fax: 33 (0) 1 39 12 18 18 >> Mobile: 33 (0) 6 07 78 41 97 >> Email: khaled.bentebal@consult-ix.fr >> Site Web: www.consult-ix.fr >> >> Le 21/12/11 23:50, KLAUS GLAHN a écrit : >>> Hi Joerg, >>> >>> reboot is not an option tonight :-( There are other applications running >> on >>> the server, which I can´t interrupt now. >>> >>> I tried onmode -ky: didn´t help >>> I tried onclean: found no problems and didn´t help >>> I tried ipcs: no informix segments >>> I tried ps -e: no oninit processes >>> >>> but when I change my SERVERNUM back to the original value 0, I can´t >> start >> the >>> instance: >>> >>> 23:35:19 IBM Informix Dynamic Server gestartet. >>> 23:35:19 shmget: [EEXIST][17]: key 52564801: shared memory already exists >>> 23:35:19 mt_shm_init: Segment resident kann nicht angelegt werden >>> >>> With ipcs I can see the segment: >>> 0x52564801 11272195 root 660 33554432 0 >>> but it belongs to root? >>> >>> I guess I have to reboot tomorrow. >>> >>> And then I can switch back to the original problem ... >>> >>> Thanks again :-) >>> Klaus >>> >>> >>> >> > ******************************************************************************* >>> Forum Note: Use "Reply" to post a response in the discussion forum. >>> >>> >> >> >> > ******************************************************************************* >> Forum Note: Use "Reply" to post a response in the discussion forum. >> >>

Again thanks to everybody ... at least I know now how to resolve the "instance is not starting"-issue. When I mess with SHMVIRTSIZE and SHMADD and start the instance, it crashed. The memory segments (lots of them) got stuck and they didn´t belong to "informix" but to "root". So onclean couldn´t clean them up. Because I couldn´t deceide which segment belongs to informix, I booted the machine. After that everything was ok - of course. With a little testing I found out, that all mem segments that informix allocates are assigned to user "root". When I repeat the whole process, I just copy all ids for the root mem segments and make a quick ipcrm -m <id> script for all those segments. With this I can clean up manually. Now back to my original problem: the crashes. What I was about to find out, was if it helps when I increase the SHMVIRTSIZE and SHMADD parameters like I was advised by Marcus. His suggested values didn´t work for our test instance. 800000 was much to high. The maximum value I could use was 131072. I only find complex formulas how to find the correct value. I hope there are some rules of thumb like: "when you have 3GB for the database take x for SHMVIRTSIZE and y for SHMADD" ... Is there a good ressource available with information about tuning this parameter? best wishes Klaus

Hallo Klaus, can you post your onconfig and machine data (Memory, Processor, OS Releae/Kernel) of the test and the real machine? It would make things easier. There is no explicit listing of "take this value if that one is..." Some you find in the setup manuals, some in the performance manuals. Another good resource are the Informix newsletters by IBM Munich: Anmeldung / Abmeldung / Anmerkung Der Newsletter wird ausschließlich an angemeldete Adressen verschickt. Die Anmeldung erfolgt, indem Sie eine Email mit dem Betreff "ANMELDUNG" an ifmxnews@de.ibm.com senden. Im Falle einer Abmeldung senden Sie "ABMELDUNG" an diese Adresse. Das Archiv der bisherigen Ausgaben finden Sie zum Beispiel unter: http://www.iug.de/index.php?option=com_content&task=view&id=95&Itemid=149 http://www.informix-zone.com/informix-german-newsletter http://www.drap.de/link/informix http://www.nsi.de/informix/newsletter http://www.bytec.de/de/software/ibm_software/newsletter/ http://www.cursor-distribution.de/index.php/aktuelles/informix-newsletter http://www.listec.de/Informix_Newsletter/ http://www.bereos.eu/software/informix/newsletter/ Merry Christmas to all! Mit freundlichen Grüßen Joerg Volz ----------------------------------------------------------------------------- -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of KLAUS GLAHN Sent: Friday, December 23, 2011 1:00 PM To: ids@iiug.org Subject: Re: 11.5 UC7E crashes [25716] Again thanks to everybody ... at least I know now how to resolve the "instance is not starting"-issue. When I mess with SHMVIRTSIZE and SHMADD and start the instance, it crashed. The memory segments (lots of them) got stuck and they didn´t belong to "informix" but to "root". So onclean couldn´t clean them up. Because I couldn´t deceide which segment belongs to informix, I booted the machine. After that everything was ok - of course. With a little testing I found out, that all mem segments that informix allocates are assigned to user "root". When I repeat the whole process, I just copy all ids for the root mem segments and make a quick ipcrm -m <id> script for all those segments. With this I can clean up manually. Now back to my original problem: the crashes. What I was about to find out, was if it helps when I increase the SHMVIRTSIZE and SHMADD parameters like I was advised by Marcus. His suggested values didn´t work for our test instance. 800000 was much to high. The maximum value I could use was 131072. I only find complex formulas how to find the correct value. I hope there are some rules of thumb like: "when you have 3GB for the database take x for SHMVIRTSIZE and y for SHMADD" ... Is there a good ressource available with information about tuning this parameter? best wishes Klaus ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum. IT Handel und Beratung Jörg Volz Bernhard-Früh-Str. 7 77855 Achern GERMANY Tel: +49 (0)7841-681651 Fax: +49 (0)7841-681654 Mobil: +49 (0)170-2989757 VAT-ID: DE201383541 http://www.it-volz.de

Hello Joerg, production and test systems are identical. Linux-Version: Linux tefix 2.6.26-2-686-bigmem #1 SMP Fri Aug 14 01:52:30 UTC 2009 i686 GNU/Linux Linux version 2.6.26-2-686-bigmem (Debian 2.6.26-17lenny2) (dannf@debian.org) (gcc version 4.1.3 20080704 (prerelease) (Debian 4.1.2-25)) #1 SMP Fri Aug 14 0 1:52:30 UTC 2009 Memory: top - 15:10:35 up 1 day, 2:31, 3 users, load average: 0.00, 0.01, 0.00 Tasks: 197 total, 2 running, 195 sleeping, 0 stopped, 0 zombie Cpu(s): 0.1%us, 0.1%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 16634528k total, 2123260k used, 14511268k free, 514016k buffers Swap: 19526996k total, 0k used, 19526996k free, 1018000k cached CPU: processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 23 model name : Intel(R) Xeon(R) CPU E5420 @ 2.50GHz stepping : 10 cpu MHz : 2500.020 cache size : 6144 KB physical id : 0 siblings : 4 core id : 0 cpu cores : 4 apicid : 0 initial apicid : 0 fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm bogomips : 5004.18 clflush size : 64 power management: Now the onconfig ... I deleted all comments: ROOTNAME rootdbs ROOTOFFSET 0MIRROR 0 #MIRRORPATH $INFORMIXDIR/tmp/demo_on.root_mirror MIRROROFFSET 0 a PHYSFILE 1000000 # 26.11.11 BG PLOG_OVERFLOW_PATH /opt/informix/tmp PHYSBUFF 1024 # 28.11.2011 LOGFILES 6 LOGSIZE 1000000 # 13.06.10 BG * 10 DYNAMIC_LOGS 2 LOGBUFF 3000 # 13.06.2010 BG erhöht LTXHWM 70 LTXEHWM 80 CONSOLE /opt/informix/tmp/online.con TBLTBLFIRST 0 TBLTBLNEXT 0 TBLSPACE_STATS 1 DBSPACETEMP SBSPACETEMP SBSPACENAME SYSSBSPACENAME ONDBSPACEDOWN 2 NETTYPE ipcshm,1,100,CPU NETTYPE onsoctcp,1,100,CPU # ergänzt LISTEN_TIMEOUT 60 MAX_INCOMPLETE_CONNECTIONS 1024 FASTPOLL 1 MULTIPROCESSOR 1 VP_MEMORY_CACHE_KB 0 SINGLE_CPU_VP 0 CLEANERS 8AUTO_AIOVPS 1 DIRECT_IO 0 LOCKS 200000 DEF_TABLE_LOCKMODE row RESIDENT 0 SHMBASE 0x44000000L SHMVIRTSIZE 32656 SHMADD 8192 EXTSHMADD 8192 SHMTOTAL 0 SHMVIRT_ALLOCSEG 0.000000 SHMNOACCESS CKPTINTVL 300AUTO_CKPTS 1 RTO_SERVER_RESTART 0 BLOCKTIMEOUT 3600 TXTIMEOUT 300 DEADLOCK_TIMEOUT 60 HETERO_COMMIT 0 TAPEBLK 32 TAPESIZE 0 LTAPEBLK 32 LTAPESIZE 0 BAR_DEBUG 0 BAR_MAX_BACKUP 0 BAR_RETRY 1 BAR_NB_XPORT_COUNT 20 BAR_XFER_BUF_SIZE 31 RESTARTABLE_RESTORE on BAR_PROGRESS_FREQ 0 BAR_BSALIB_PATH BACKUP_FILTER RESTORE_FILTER BAR_PERFORMANCE 0 ISM_DATA_POOL ISMData ISM_LOG_POOL ISMLogs DD_HASHSIZE 31 DD_HASHMAX 10 DS_HASHSIZE 31 DS_POOLSIZE 127 PC_HASHSIZE 31 PC_POOLSIZE 127 STMT_CACHE 0 STMT_CACHE_HITS 0 STMT_CACHE_SIZE 512 STMT_CACHE_NOLIMIT 0 STMT_CACHE_NUMPOOL 1 USEOSTIME 0 STACKSIZE 32 ALLOW_NEWLINE 0 USELASTCOMMITTED "COMMITTED READ" FILLFACTOR 90 MAX_FILL_DATA_PAGES 0 ONLIDX_MAXMEM 5120 MAX_PDQPRIORITY 100 DS_MAX_QUERIES DS_TOTAL_MEMORY DS_MAX_SCANS 1048576 DS_NONPDQ_QUERY_MEM 128 DATASKIP off OPTCOMPIND 2 DIRECTIVES 1 EXT_DIRECTIVES 0 OPT_GOAL -1 IFX_FOLDVIEW 0AUTO_REPREPARE 1 RA_PAGES 64 RA_THRESHOLD 16 EXPLAIN_STAT 1 IFX_EXTEND_ROLE 1 SECURITY_LOCALCONNECTION 0 UNSECURE_ONSTAT 0 ADMIN_USER_MODE_WITH_DBSA 0 SSL_KEYSTORE_LABEL PLCY_POOLSIZE 127 PLCY_HASHSIZE 31 USRC_POOLSIZE 127 USRC_HASHSIZE 31 STAGEBLOB OPCACHEMAX 0 ON_RECVRY_THREADS 1 OFF_RECVRY_THREADS 10 DUMPSHMEM 0 DUMPGCORE 0 DUMPCORE 0 DUMPCNT 1 ALRM_ALL_EVENTS 0 STORAGE_FULL_ALARM 600,3 RAS_PLOG_SPEED 41666 RAS_LLOG_SPEED 3175 EILSEQ_COMPAT_MODE 0 QSTATS 0 WSTATS 0 JVPJAVALIB /bin JVPJAVAVM jvm AUTO_LRU_TUNING 1 ROOTPATH /dev/informix-root MSGPATH /opt/informix/online.log ROOTSIZE 46000000 # 46.000.000 = 46 GB TAPEDEV /dev/null LTAPEDEV /dev/null DBSERVERNAME filmfest DBSERVERALIASES SERVERNUM 0 ALARMPROGRAM /opt/informix/etc/alarmprogram.sh DRLOSTFOUND /opt/informix/etc/dr.lostfound BAR_ACT_LOG /opt/informix/bar_act.log BAR_DEBUG_LOG /opt/informix/bar_dbug.log SYSALARMPROGRAM /opt/informix/etc/evidence.sh DUMPDIR /opt/informix/tmp JVPJAVAHOME /opt/informix/extend/krakatoa/jre JVPHOME /opt/informix/extend/krakatoa/ JVPPROPFILE /opt/informix/extend/krakatoa/.jvpprops JVPLOGFILE /opt/informix/jvp.log JVPCLASSPATH /opt/informix/extend/krakatoa/krakatoa.jar:/opt/informix/extend/krakatoa/jdbc.jar VPCLASS cpu,num=1,noage BUFFERPOOL size=2K,buffers=800000,lrus=4,lru_min_dirty=50.000000,lru_max_dirty=60.000000 BTSCANNER num=1,threshold=5000,rangesize=-1,alice=6,compression=default MIRRORPATH ADMIN_MODE_USERS CONVERSION_GUARD 2 RESTORE_POINT_DIR /opt/informix/tmp BATCHEDREAD_TABLE 0 Merry Christmas :-) Klaus

Hi Klaus, I bet your problem is system limited shared memory parameters. What is your content in /etc/sysctl.conf ? My settings on a comparable system for these parameters is: kernel.shmmax=4398046511104 kernel.shmmni=4096 kernel.shmall=8388608 vm.min_free_kbytes=128000 vm.swappiness=1 Then you should be able to allocate much more shared memory. BTW: you can change the paramters in the file and activate using sysctl -p Happy Christmas, Marcus -----Original Message----- From: KLAUS GLAHN [mailto:glahn@berlinale.de] Sent: Friday, December 23, 2011 3:55 PM To: ids@iiug.org Subject: Re: RE: 11.5 UC7E crashes [25719] Hello Joerg, production and test systems are identical. Linux-Version: Linux tefix 2.6.26-2-686-bigmem #1 SMP Fri Aug 14 01:52:30 UTC 2009 i686 GNU/Linux Linux version 2.6.26-2-686-bigmem (Debian 2.6.26-17lenny2) (dannf@debian.org) (gcc version 4.1.3 20080704 (prerelease) (Debian 4.1.2-25)) #1 SMP Fri Aug 14 0 1:52:30 UTC 2009 Memory: top - 15:10:35 up 1 day, 2:31, 3 users, load average: 0.00, 0.01, 0.00 Tasks: 197 total, 2 running, 195 sleeping, 0 stopped, 0 zombie Cpu(s): 0.1%us, 0.1%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 16634528k total, 2123260k used, 14511268k free, 514016k buffers Swap: 19526996k total, 0k used, 19526996k free, 1018000k cached CPU: processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 23 model name : Intel(R) Xeon(R) CPU E5420 @ 2.50GHz stepping : 10 cpu MHz : 2500.020 cache size : 6144 KB physical id : 0 siblings : 4 core id : 0 cpu cores : 4 apicid : 0 initial apicid : 0 fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm bogomips : 5004.18 clflush size : 64 power management: Now the onconfig ... I deleted all comments: ROOTNAME rootdbs ROOTOFFSET 0MIRROR 0 #MIRRORPATH $INFORMIXDIR/tmp/demo_on.root_mirror MIRROROFFSET 0 a PHYSFILE 1000000 # 26.11.11 BG PLOG_OVERFLOW_PATH /opt/informix/tmp PHYSBUFF 1024 # 28.11.2011 LOGFILES 6 LOGSIZE 1000000 # 13.06.10 BG * 10 DYNAMIC_LOGS 2 LOGBUFF 3000 # 13.06.2010 BG erhht LTXHWM 70 LTXEHWM 80 CONSOLE /opt/informix/tmp/online.con TBLTBLFIRST 0 TBLTBLNEXT 0 TBLSPACE_STATS 1 DBSPACETEMP SBSPACETEMP SBSPACENAME SYSSBSPACENAME ONDBSPACEDOWN 2 NETTYPE ipcshm,1,100,CPU NETTYPE onsoctcp,1,100,CPU # ergnzt LISTEN_TIMEOUT 60 MAX_INCOMPLETE_CONNECTIONS 1024 FASTPOLL 1 MULTIPROCESSOR 1 VP_MEMORY_CACHE_KB 0 SINGLE_CPU_VP 0 CLEANERS 8AUTO_AIOVPS 1 DIRECT_IO 0 LOCKS 200000 DEF_TABLE_LOCKMODE row RESIDENT 0 SHMBASE 0x44000000L SHMVIRTSIZE 32656 SHMADD 8192 EXTSHMADD 8192 SHMTOTAL 0 SHMVIRT_ALLOCSEG 0.000000 SHMNOACCESS CKPTINTVL 300AUTO_CKPTS 1 RTO_SERVER_RESTART 0 BLOCKTIMEOUT 3600 TXTIMEOUT 300 DEADLOCK_TIMEOUT 60 HETERO_COMMIT 0 TAPEBLK 32 TAPESIZE 0 LTAPEBLK 32 LTAPESIZE 0 BAR_DEBUG 0 BAR_MAX_BACKUP 0 BAR_RETRY 1 BAR_NB_XPORT_COUNT 20 BAR_XFER_BUF_SIZE 31 RESTARTABLE_RESTORE on BAR_PROGRESS_FREQ 0 BAR_BSALIB_PATH BACKUP_FILTER RESTORE_FILTER BAR_PERFORMANCE 0 ISM_DATA_POOL ISMData ISM_LOG_POOL ISMLogs DD_HASHSIZE 31 DD_HASHMAX 10 DS_HASHSIZE 31 DS_POOLSIZE 127 PC_HASHSIZE 31 PC_POOLSIZE 127 STMT_CACHE 0 STMT_CACHE_HITS 0 STMT_CACHE_SIZE 512 STMT_CACHE_NOLIMIT 0 STMT_CACHE_NUMPOOL 1 USEOSTIME 0 STACKSIZE 32 ALLOW_NEWLINE 0 USELASTCOMMITTED "COMMITTED READ" FILLFACTOR 90 MAX_FILL_DATA_PAGES 0 ONLIDX_MAXMEM 5120 MAX_PDQPRIORITY 100 DS_MAX_QUERIES DS_TOTAL_MEMORY DS_MAX_SCANS 1048576 DS_NONPDQ_QUERY_MEM 128 DATASKIP off OPTCOMPIND 2 DIRECTIVES 1 EXT_DIRECTIVES 0 OPT_GOAL -1 IFX_FOLDVIEW 0AUTO_REPREPARE 1 RA_PAGES 64 RA_THRESHOLD 16 EXPLAIN_STAT 1 IFX_EXTEND_ROLE 1 SECURITY_LOCALCONNECTION 0 UNSECURE_ONSTAT 0 ADMIN_USER_MODE_WITH_DBSA 0 SSL_KEYSTORE_LABEL PLCY_POOLSIZE 127 PLCY_HASHSIZE 31 USRC_POOLSIZE 127 USRC_HASHSIZE 31 STAGEBLOB OPCACHEMAX 0 ON_RECVRY_THREADS 1 OFF_RECVRY_THREADS 10 DUMPSHMEM 0 DUMPGCORE 0 DUMPCORE 0 DUMPCNT 1 ALRM_ALL_EVENTS 0 STORAGE_FULL_ALARM 600,3 RAS_PLOG_SPEED 41666 RAS_LLOG_SPEED 3175 EILSEQ_COMPAT_MODE 0 QSTATS 0 WSTATS 0 JVPJAVALIB /bin JVPJAVAVM jvm AUTO_LRU_TUNING 1 ROOTPATH /dev/informix-root MSGPATH /opt/informix/online.log ROOTSIZE 46000000 # 46.000.000 = 46 GB TAPEDEV /dev/null LTAPEDEV /dev/null DBSERVERNAME filmfest DBSERVERALIASES SERVERNUM 0ALARMPROGRAM /opt/informix/etc/alarmprogram.sh DRLOSTFOUND /opt/informix/etc/dr.lostfound BAR_ACT_LOG /opt/informix/bar_act.log BAR_DEBUG_LOG /opt/informix/bar_dbug.log SYSALARMPROGRAM /opt/informix/etc/evidence.sh DUMPDIR /opt/informix/tmp JVPJAVAHOME /opt/informix/extend/krakatoa/jre JVPHOME /opt/informix/extend/krakatoa/ JVPPROPFILE /opt/informix/extend/krakatoa/.jvpprops JVPLOGFILE /opt/informix/jvp.log JVPCLASSPATH /opt/informix/extend/krakatoa/krakatoa.jar:/opt/informix/extend/krakatoa /jdbc.jar VPCLASS cpu,num=1,noage BUFFERPOOL size=2K,buffers=800000,lrus=4,lru_min_dirty=50.000000,lru_max_dirty=60.000000 BTSCANNER num=1,threshold=5000,rangesize=-1,alice=6,compression=default MIRRORPATH ADMIN_MODE_USERS CONVERSION_GUARD 2 RESTORE_POINT_DIR /opt/informix/tmp BATCHEDREAD_TABLE 0 Merry Christmas :-) Klaus ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum.

Hello again, today I changed the shared memory parameters and everything looks fine ... so far. The problem with random crashes: you´ll never can be sure you fix them. I increased the kernel-parameters for Debian and then could take much larger values for SHMVIRTSIZE (512 MB) and SHMADD (65 MB). After one day I had no warnings ("contiguous shared memory segment allocation failed" was a frequent visitor in online.log) and no crashes. So I want to say thanks again to everybody and have a happy new year. Best wishes Klaus

Hi, the system crashed again :-( 15:13:36 stack trace for pid 2565 written to /opt/informix/tmp/af.3f185e8f 15:13:36 Assert fehlgeschlagen: No Exception Handler 15:13:36 IBM Informix Dynamic Server Version 11.50.UC7E 15:13:36 Wer: Session(78763, read@kbbws1112064.kbb.intern, 4032, 0x2e76b3b8) Thread(80688, sqlexec, 2f0841c0, 1) File: mtex.c Line: 491 15:13:36 Ergebnisse: Exception Caught. Type: MT_EX_OS, Context: mem 15:13:36 Aktion: Bitte informieren Sie die technische Unterstützung für IBM Informix. 15:13:36 Siehe auch: /opt/informix/tmp/af.3f185e8f 15:13:38 Starting crash time check of: 15:13:38 1. memory block headers 15:13:38 2. stacks 15:13:38 Found bad memory block; address:34403fa8 15:13:38 mtex.c, line 491, thread 80688, proc id 2565, No Exception Handler. 15:13:38 Der Master-Dämon ist nicht mehr aktiv 15:13:38 PANIC: Attempting to bring system down No problems since 6 days and the "Contiguous shared memory segment allocation failed"-message has disappeared from online.log. The system looked healthy (online.log looked ok, no warnings, just logical protocol and checkpoint-messages - good performance) - and then out of nowhere: Assertion failed Any further ideas? Thanks Klaus

Hi, You should send the af trace to IBM support or your local distributor. I assume you ran into a bug, which might be known. It all depends on what is inside the af trace, and only IBM will be able to tell what kind of operation has triggered the bug. MT_EX_OS can be anything .... Marcus -----Original Message----- From: KLAUS GLAHN [mailto:glahn@berlinale.de] Sent: Wednesday, January 04, 2012 3:40 PM To: ids@iiug.org Subject: crash again [25808] Hi, the system crashed again :-( 15:13:36 stack trace for pid 2565 written to /opt/informix/tmp/af.3f185e8f 15:13:36 Assert fehlgeschlagen: No Exception Handler 15:13:36 IBM Informix Dynamic Server Version 11.50.UC7E 15:13:36 Wer: Session(78763, read@kbbws1112064.kbb.intern, 4032, 0x2e76b3b8) Thread(80688, sqlexec, 2f0841c0, 1) File: mtex.c Line: 491 15:13:36 Ergebnisse: Exception Caught. Type: MT_EX_OS, Context: mem 15:13:36 Aktion: Bitte informieren Sie die technische Untersttzung fr IBM Informix. 15:13:36 Siehe auch: /opt/informix/tmp/af.3f185e8f 15:13:38 Starting crash time check of: 15:13:38 1. memory block headers 15:13:38 2. stacks 15:13:38 Found bad memory block; address:34403fa8 15:13:38 mtex.c, line 491, thread 80688, proc id 2565, No Exception Handler. 15:13:38 Der Master-Dmon ist nicht mehr aktiv 15:13:38 PANIC: Attempting to bring system down No problems since 6 days and the "Contiguous shared memory segment allocation failed"-message has disappeared from online.log. The system looked healthy (online.log looked ok, no warnings, just logical protocol and checkpoint-messages - good performance) - and then out of nowhere: Assertion failed Any further ideas? Thanks Klaus ************************************************************************ ******* Forum Note: Use "Reply" to post a response in the discussion forum.