Oninit cdi / Archive
Talk to an expert →

onmode -ky hangs

Posted in 2006

Summary

A bulk LOAD/INSERT on IDS 10.00.UC5 (32-bit, AIX 5.2) consumed locks in huge numbers, forcing IDS to allocate virtual shared memory segments until it hit the OS limit ('out of virtual shared memory', shmat EMFILE); the server then hung, blocked on checkpoint, accepting no connections, and even 'onmode -ky' hung (since onmode itself must connect and needs memory). The poster recovered only by killing the oninit processes and restarting. Suggested preventions: lock the table (exclusive/share) or set page-level locking before loading, drop indexes, and commit every few thousand rows instead of one giant transaction; also check OS SHMSEG/shared-memory limits (32-bit AIX allows ~11 segments per process) and consider 64-bit. No vendor fix beyond these workarounds is recorded.

Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.

Topics: Server Administration, Logging & Checkpoints, Platform-Specific Issues, Internationalization & Character Sets

FRANK — — source: IIUG Forums & Mailing Lists

We had a bulk LOAD -INSERT and caused the IDS hangs. see below log info. It remains hanging after I issued "onmode -ky". What should we do? Thanks, Frank ====== 12:29:38 dynamically allocated 100000 locks 12:29:47 dynamically allocated 100000 locks 12:30:01 Checkpoint Completed: duration was 4 seconds. 12:30:01 Checkpoint loguniq 23, logpos 0x4f32018, timestamp: 0x1717af75 12:30:01 Maximum server connections 3 12:30:01 Contiguous shared memory segment allocation failed at 0xd0000000. Allocation successful at 0xe0000000. Check SHMBASE is consistent with the value in $INFORMIXDIR/etc/onconfig.std. If you are using the correct SHMBASE value in your ONCONFIG file, then consider this message informational only. 12:30:01 Dynamically allocated new virtual shared memory segment (size 32768KB) 12:30:01 dynamically allocated 100000 locks 12:30:10 dynamically allocated 100000 locks 12:30:20 dynamically allocated 100000 locks 12:30:29 shmat: [EMFILE][24]: out of shared memory segments, check system SHMSEG 12:30:29 out of virtual shared memory ======= Program Name: onstat Build Version: 10.00.UC5 Build Number: N202 Build Host: ibm6c1b Build OS: AIX 5.2 Build Date: Thu May 18 00:09:08 CDT 2006 GLS Version: glslib-4.00.UC8

↪ replying to FRANK
Julio Gerardo Flores Olvera — — source: IIUG Forums & Mailing Lists

You are exceeded your limit for locks. I recommend you use one of this tips: - Change de lock mode of your table to 'PAGE'. - Drop indexes and change your table to 'row' mode, then load the data. - Before the load, lock your table in share mode. -----Mensaje original----- De: FRANK [mailto:yunyaoqu@gmail.com] Enviado el: Lunes, 16 de Octubre de 2006 09:38 a.m. Para: ids@iiug.org Asunto: onmode -ky hangs [7630] We had a bulk LOAD -INSERT and caused the IDS hangs. see below log info. It remains hanging after I issued "onmode -ky". What should we do? Thanks, Frank ====== 12:29:38 dynamically allocated 100000 locks 12:29:47 dynamically allocated 100000 locks 12:30:01 Checkpoint Completed: duration was 4 seconds. 12:30:01 Checkpoint loguniq 23, logpos 0x4f32018, timestamp: 0x1717af75 12:30:01 Maximum server connections 3 12:30:01 Contiguous shared memory segment allocation failed at 0xd0000000. Allocation successful at 0xe0000000. Check SHMBASE is consistent with the value in $INFORMIXDIR/etc/onconfig.std. If you are using the correct SHMBASE value in your ONCONFIG file, then consider this message informational only. 12:30:01 Dynamically allocated new virtual shared memory segment (size 32768KB) 12:30:01 dynamically allocated 100000 locks 12:30:10 dynamically allocated 100000 locks 12:30:20 dynamically allocated 100000 locks 12:30:29 shmat: [EMFILE][24]: out of shared memory segments, check system SHMSEG 12:30:29 out of virtual shared memory ======= Program Name: onstat Build Version: 10.00.UC5 Build Number: N202 Build Host: ibm6c1b Build OS: AIX 5.2 Build Date: Thu May 18 00:09:08 CDT 2006 GLS Version: glslib-4.00.UC8 **************************************************************************** *** Forum Note: Use "Reply" to post a response in the discussion forum.

↪ replying to FRANK
Mark Jamison — — source: IIUG Forums & Mailing Lists

Hi Frank, Could you post an onstat -g ath when this happens? It's definitely a result of running out of memory while acquiring locks, but that shoulld not necessarily cause an engine hang. -------------------------------------------------------------------------------- ---- Praise the Lord oh my soul, let my inmost being praise His holy name. -Psalm 103:1 ----- Original Message ---- From: FRANK <yunyaoqu@gmail.com> To: ids@iiug.org Sent: Monday, October 16, 2006 9:38:25 AM Subject: onmode -ky hangs [7630] We had a bulk LOAD -INSERT and caused the IDS hangs. see below log info. It remains hanging after I issued "onmode -ky". What should we do? Thanks, Frank ====== 12:29:38 dynamically allocated 100000 locks 12:29:47 dynamically allocated 100000 locks 12:30:01 Checkpoint Completed: duration was 4 seconds. 12:30:01 Checkpoint loguniq 23, logpos 0x4f32018, timestamp: 0x1717af75 12:30:01 Maximum server connections 3 12:30:01 Contiguous shared memory segment allocation failed at 0xd0000000. Allocation successful at 0xe0000000. Check SHMBASE is consistent with the value in $INFORMIXDIR/etc/onconfig.std. If you are using the correct SHMBASE value in your ONCONFIG file, then consider this message informational only. 12:30:01 Dynamically allocated new virtual shared memory segment (size 32768KB) 12:30:01 dynamically allocated 100000 locks 12:30:10 dynamically allocated 100000 locks 12:30:20 dynamically allocated 100000 locks 12:30:29 shmat: [EMFILE][24]: out of shared memory segments, check system SHMSEG 12:30:29 out of virtual shared memory ======= Program Name: onstat Build Version: 10.00.UC5 Build Number: N202 Build Host: ibm6c1b Build OS: AIX 5.2 Build Date: Thu May 18 00:09:08 CDT 2006 GLS Version: glslib-4.00.UC8 ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

↪ replying to Mark Jamison
FRANK — — source: IIUG Forums & Mailing Lists

HI, Mark, We already killed all the oninit processes and restarted IDS. The following is some info I caught before we killed it, it may or may not be usefull. By the way, we can successfully finish it by using "lock table in exclusive mode" Hanging there, it did not accept any connections, commands,..... and onstat -K show millions locks. Thanks, Frank informix@erika $ onstat -g mem IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line (CKPT REQ) -- Up 2 days 21:03:56 -- 2008880 Kbytes Blocked:CKPT Pool Summary: name class addr totalsize freesize #allocfrag #freefrag afpool V b000d020 4096 2128 2 1 tpcpool V b0756020 8192 3648 4 2 seqpool V b0771020 4096 2400 2 1 pnlpool V b0759020 8192 2536 5 2 sbtlist V b009f020 12288 5600 4 3 dstpool V b0755020 4096 1512 2 1 sqcrypto V b08ff020 4096 2272 2 1 ampool V b076b020 4096 1832 12 1 shmcon M 30001020 536576 6520 2 2 main_loop() V b0907020 372736 66224 83 40 sb_delundoq V b00bd020 28672 8424 4 3 XTF_mem V b07ad020 565248 7744 4 3 bf_prioswee V b0d57020 24576 4256 13 3 onmode_mon V b0d39020 24576 4256 13 3 lgflushpool V b0c59020 4096 2184 5 1 2 V b091b020 12288 4120 13 2 rsam V b012f020 308527104 84336 889 41 sscpool0 V b074f020 8192 2680 2 2 3 V b0938020 12288 4120 13 2 4 V b0955020 12288 3544 14 2 5 V b0962020 12288 2896 19 3 6 V b097d020 12288 2896 19 3 7 V b0994020 94208 5160 213 3 aslogflush V b0c5a020 24576 4256 13 3 aio V b0334020 5316608 364912 411 115 9 V b099b020 12288 4120 13 2 opcinstpool V b0758020 12288 1512 10 2 dbausrpool V b0775020 4096 1952 6 1 smartblob V b009e020 36864 13128 16 5 gls V b0749020 28672 5544 348 4 res-buff0 R 442bd020 1638424576 6520 2 2 DefConvWrit V b07ab020 8192 1016 102 2 resroutpool V b075b020 4096 1512 2 1 btscanner_0 V b0cf3020 24576 1376 21 4 sb_delq V b00a2020 28672 8424 4 3 resident R 40042020 69709824 6520 2 2 dictpool V b0750020 131072 1784 62 1 7*O0 V b0dd2020 4096 2448 1 1 mt V b000e020 1540096 5232 9060 77 10 V b09a7020 12288 2896 19 3 aggpool V b075a020 4096 1512 2 1 73 V b0c4d020 49152 5800 80 7 93 V b0998020 45056 4216 62 5 94 V b0de0020 12288 4120 13 2 procpool V b0752020 16384 2928 26 2 95 V b0df3020 12288 4120 13 2 extdpool V b0772020 4096 1256 4 1 69 V b099a020 12288 4120 13 2 global V b0003020 7315456 38120 1294 14 ctcpool V b0757020 4096 488 3 1 tmxapool V b0774020 4096 2384 3 1 sb_arcspace V b012c020 4096 416 8 1 sb_loheader V b00d8020 262144 25944 413 60 inhpool V b076d020 4096 1520 16 1 opcpool V b076c020 4096 376 34 1 Blkpool Summary: name class addr size #blks mt V b000fe10 1982464 44 global V b00036c0 0 0 informix@erika $ onstat -p IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line (CKPT REQ) -- Up 2 days 21:04:35 -- 2008880 Kbytes Blocked:CKPT Profile dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached 1194011 3221512 32323277 96.31 305009 378823 6796070 95.51 isamtot open start read write rewrite delete commit rollbk 32089058 960 3208713 9630965 3207687 493 0 45 0 gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs 16 0 0 0 0 0 8 ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes 0 0 0 1578.56 178.35 12 1615 bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans 13561 0 19276754 0 0 4 13 15 ixda-RA idx-RA da-RA RA-pgsused lchwaits 23 14848 0 14870 290 informix@erika $ onstat -g seg IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line (CKPT REQ) -- Up 2 days 21:04:56 -- 2008880 Kbytes Blocked:CKPT Segment Summary: id key addr size ovhd class blkused blkfree 7340036 1382041603 30000000 540672 672 M 132 0 9437187 1382041601 40000000 1708425216 269028 R* 417091 5 9437189 1382041602 b0000000 314572800 10248 V 73279 3521 6 1382041604 e0000000 33554432 1672 V 6482 1710 Total: - - 2057093120 - - 496984 5236 (* segment locked in memory) informix@erika $ onstat -g rbm IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line (CKPT REQ) -- Up 2 days 21:05:19 -- 2008880 Kbytes Blocked:CKPT Block bitmap for resident segment address 0x40000000: address = 0x40000288, size(bits) = 417096 used = 417091, largest_free = 4294967295 0: startsearch = 17086, lastfree = -1 1: startsearch = 0, lastfree = 0 2: startsearch = 0, lastfree = 0 3: startsearch = 0, lastfree = 0 4: startsearch = 0, lastfree = 0 5: startsearch = 417091, lastfree = -1 0:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 256:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 512:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 768:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 1024:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 1280:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 1536:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 1792:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 2048:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 2304:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 2560:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 2816:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 3072:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 3328:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff i informix@erika $ onstat -g nbm IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line (CKPT REQ) -- Up 2 days 21:05:25 -- 2008880 Kbytes Blocked:CKPT Block bitmap for virtual segment address 0xb0000000: address = 0xb0000288, size(bits) = 76800 used = 73279, largest_free = 4294967295 0: startsearch = 3574, lastfree = 3797 1: startsearch = 3801, lastfree = -1 2: startsearch = 3795, lastfree = -1 3: startsearch = 3808, lastfree = 3904 4: startsearch = 3808, lastfree = 3895 5: startsearch = 76800, lastfree = -1 0:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 256:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 512:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 768:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 1024:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 1280:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 1536:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 1792:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 2048:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 2304:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 2560:ffff

↪ replying to FRANK
FRANK — — source: IIUG Forums & Mailing Lists

HI, Mark, We already killed all the oninit processes and restarted IDS. The following is some info I caught before we killed it, it may or may not be usefull. By the way, we can successfully finish it by using "lock table in exclusive mode" Hanging there, it did not accept any connections, commands,..... and onstat -K show millions locks. Thanks, Frank informix@erika $ onstat -g mem IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line (CKPT REQ) -- Up 2 days 21:03:56 -- 2008880 Kbytes Blocked:CKPT Pool Summary: name class addr totalsize freesize #allocfrag #freefrag afpool V b000d020 4096 2128 2 1 tpcpool V b0756020 8192 3648 4 2 seqpool V b0771020 4096 2400 2 1 pnlpool V b0759020 8192 2536 5 2 sbtlist V b009f020 12288 5600 4 3 dstpool V b0755020 4096 1512 2 1 sqcrypto V b08ff020 4096 2272 2 1 ampool V b076b020 4096 1832 12 1 shmcon M 30001020 536576 6520 2 2 main_loop() V b0907020 372736 66224 83 40 sb_delundoq V b00bd020 28672 8424 4 3 XTF_mem V b07ad020 565248 7744 4 3 bf_prioswee V b0d57020 24576 4256 13 3 onmode_mon V b0d39020 24576 4256 13 3 lgflushpool V b0c59020 4096 2184 5 1 2 V b091b020 12288 4120 13 2 rsam V b012f020 308527104 84336 889 41 sscpool0 V b074f020 8192 2680 2 2 3 V b0938020 12288 4120 13 2 4 V b0955020 12288 3544 14 2 5 V b0962020 12288 2896 19 3 6 V b097d020 12288 2896 19 3 7 V b0994020 94208 5160 213 3 aslogflush V b0c5a020 24576 4256 13 3 aio V b0334020 5316608 364912 411 115 9 V b099b020 12288 4120 13 2 opcinstpool V b0758020 12288 1512 10 2 dbausrpool V b0775020 4096 1952 6 1 smartblob V b009e020 36864 13128 16 5 gls V b0749020 28672 5544 348 4 res-buff0 R 442bd020 1638424576 6520 2 2 DefConvWrit V b07ab020 8192 1016 102 2 resroutpool V b075b020 4096 1512 2 1 btscanner_0 V b0cf3020 24576 1376 21 4 sb_delq V b00a2020 28672 8424 4 3 resident R 40042020 69709824 6520 2 2 dictpool V b0750020 131072 1784 62 1 7*O0 V b0dd2020 4096 2448 1 1 mt V b000e020 1540096 5232 9060 77 10 V b09a7020 12288 2896 19 3 aggpool V b075a020 4096 1512 2 1 73 V b0c4d020 49152 5800 80 7 93 V b0998020 45056 4216 62 5 94 V b0de0020 12288 4120 13 2 procpool V b0752020 16384 2928 26 2 95 V b0df3020 12288 4120 13 2 extdpool V b0772020 4096 1256 4 1 69 V b099a020 12288 4120 13 2 global V b0003020 7315456 38120 1294 14 ctcpool V b0757020 4096 488 3 1 tmxapool V b0774020 4096 2384 3 1 sb_arcspace V b012c020 4096 416 8 1 sb_loheader V b00d8020 262144 25944 413 60 inhpool V b076d020 4096 1520 16 1 opcpool V b076c020 4096 376 34 1 Blkpool Summary: name class addr size #blks mt V b000fe10 1982464 44 global V b00036c0 0 0 informix@erika $ onstat -p IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line (CKPT REQ) -- Up 2 days 21:04:35 -- 2008880 Kbytes Blocked:CKPT Profile dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached 1194011 3221512 32323277 96.31 305009 378823 6796070 95.51 isamtot open start read write rewrite delete commit rollbk 32089058 960 3208713 9630965 3207687 493 0 45 0 gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs 16 0 0 0 0 0 8 ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes 0 0 0 1578.56 178.35 12 1615 bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans 13561 0 19276754 0 0 4 13 15 ixda-RA idx-RA da-RA RA-pgsused lchwaits 23 14848 0 14870 290 informix@erika $ onstat -g seg IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line (CKPT REQ) -- Up 2 days 21:04:56 -- 2008880 Kbytes Blocked:CKPT Segment Summary: id key addr size ovhd class blkused blkfree 7340036 1382041603 30000000 540672 672 M 132 0 9437187 1382041601 40000000 1708425216 269028 R* 417091 5 9437189 1382041602 b0000000 314572800 10248 V 73279 3521 6 1382041604 e0000000 33554432 1672 V 6482 1710 Total: - - 2057093120 - - 496984 5236 (* segment locked in memory) informix@erika $ onstat -g rbm IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line (CKPT REQ) -- Up 2 days 21:05:19 -- 2008880 Kbytes Blocked:CKPT Block bitmap for resident segment address 0x40000000: address = 0x40000288, size(bits) = 417096 used = 417091, largest_free = 4294967295 0: startsearch = 17086, lastfree = -1 1: startsearch = 0, lastfree = 0 2: startsearch = 0, lastfree = 0 3: startsearch = 0, lastfree = 0 4: startsearch = 0, lastfree = 0 5: startsearch = 417091, lastfree = -1 0:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 256:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 512:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 768:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 1024:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 1280:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 1536:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 1792:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 2048:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 2304:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 2560:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 2816:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 3072:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 3328:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff i informix@erika $ onstat -g nbm IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line (CKPT REQ) -- Up 2 days 21:05:25 -- 2008880 Kbytes Blocked:CKPT Block bitmap for virtual segment address 0xb0000000: address = 0xb0000288, size(bits) = 76800 used = 73279, largest_free = 4294967295 0: startsearch = 3574, lastfree = 3797 1: startsearch = 3801, lastfree = -1 2: startsearch = 3795, lastfree = -1 3: startsearch = 3808, lastfree = 3904 4: startsearch = 3808, lastfree = 3895 5: startsearch = 76800, lastfree = -1 0:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 256:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 512:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 768:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 1024:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 1280:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 1536:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 1792:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 2048:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 2304:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 2560:ffff

↪ replying to FRANK
dharmendra sharma — — source: IIUG Forums & Mailing Lists

To avoid this kind of situation, you should try one of the following; 1) Do not run your INSERT statement within a TRANSACTION i.e. BEGIN AND COMMIT WORK. 2) If you must have a TRANSACTION, then make sure you execute COMMIT WORK after every 1000 records (just to avoid lock overflow). Thanks, ______________________________________________________________ From: "FRANK" <yunyaoqu@gmail.com> Reply-To: ids@iiug.org To: ids@iiug.org Subject: onmode -ky hangs [7630] Date: Mon, 16 Oct 2006 10:38:25 -0400 (EDT) Received: from perform.iiug.org ([216.177.38.211]) by bay0-mc6-f9.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.2444); Mon, 16 Oct 2006 07:45:47 -0700 Received: by perform.iiug.org (Postfix, from userid 60001)id 89108A13E; Mon, 16 Oct 2006 10:38:39 -0400 (EDT) Received: from perform.iiug.org (localhost [127.0.0.1])by perform.iiug.org (Postfix) with ESMTP id AB2FDA134;Mon, 16 Oct 2006 10:38:33 -0400 (EDT) Received: by perform.iiug.org (Postfix, from userid 60001)id 5CE48A134; Mon, 16 Oct 2006 10:38:25 -0400 (EDT) >We had a bulk LOAD -INSERT and caused the IDS hangs. see below log info. > >It remains hanging after I issued "onmode -ky". > >What should we do? > >Thanks, >Frank > >====== >12:29:38 dynamically allocated 100000 locks >12:29:47 dynamically allocated 100000 locks >12:30:01 Checkpoint Completed: duration was 4 seconds. >12:30:01 Checkpoint loguniq 23, logpos 0x4f32018, timestamp: 0x1717af75 > >12:30:01 Maximum server connections 3 >12:30:01 Contiguous shared memory segment allocation failed at 0xd0000000. >Allocation successful at 0xe0000000. >Check SHMBASE is consistent with the value in $INFORMIXDIR/etc/onconfig.std. >If you are using the correct SHMBASE value in your ONCONFIG file, then >consider this message informational only. > >12:30:01 Dynamically allocated new virtual shared memory segment (size >32768KB) >12:30:01 dynamically allocated 100000 locks >12:30:10 dynamically allocated 100000 locks >12:30:20 dynamically allocated 100000 locks >12:30:29 shmat: [EMFILE][24]: out of shared memory segments, check system >SHMSEG >12:30:29 out of virtual shared memory > >======= >Program Name: onstat >Build Version: 10.00.UC5 >Build Number: N202 >Build Host: ibm6c1b >Build OS: AIX 5.2 >Build Date: Thu May 18 00:09:08 CDT 2006 >GLS Version: glslib-4.00.UC8 > > >****************************************************************** ************* > Forum Note: Use "Reply" to post a response in the discussion forum. > _________________________________________________________________ [1]Use your PC to make calls at very low rates References 1. http://g.msn.com/8HMBENUS/2749??PS=47575

↪ replying to FRANK
SANJIT CHAKRABORTY — — source: IIUG Forums & Mailing Lists

Hello Frank The onmode is nothing but a client application. Whenever, you execute an onmode command, it will make a new connection to the database server and each new connection uses some virtual memory. During the load job you added several virtual memory segments and consumed all of them, eventually reached to maximum shared memory segments that can create system wide and Informix couldn’t able to add any farther virtual shared memory segment. -Sanjit

↪ replying to SANJIT CHAKRABORTY
FRANK — — source: IIUG Forums & Mailing Lists

Hi, Sanjit, When this happens, Informix hangs, so the only way is to kill oninit processes to restart our ilfe, is this right? Thanks, Frank On 10/18/06, SANJIT CHAKRABORTY <c_sanjit@lycos.com> wrote: > > > Hello Frank > > The onmode is nothing but a client application. Whenever, you execute an > onmode command, it will make a new connection to the database server and > each > new connection uses some virtual memory. > > During the load job you added several virtual memory segments and consumed > all > of them, eventually reached to maximum shared memory segments that can > create > system wide and Informix couldn't able to add any farther virtual shared > memory segment. > > -Sanjit > > > > ******************************************************************************* > Forum Note: Use "Reply" to post a response in the discussion forum. > >

↪ replying to FRANK
Sebastian, Norma J. — — source: IIUG Forums & Mailing Lists

Hello Frank, You are on IDS 10.00.UC5 (32 bit) Your onstat -g seg showed : IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line (CKPT REQ) -- Up 2 days 21:04:56 -- 2008880 KbytesSegment Summary: id key addr size ovhd class blkused blkfree 7340036 1382041603 30000000 540672 672 M 132 0 9437187 1382041601 40000000 1708425216 269028 R* 417091 5 9437189 1382041602 b0000000 314572800 10248 V 73279 3521 6 1382041604 e0000000 33554432 1672 V 6482 1710 Total: - - 2,057,093,120 - - 496984 5236 I don't know much about AIX other than hearing an Informix DBA say something about a limit of 11 memory segments which isn't your problem as shown above you have 4. Back when we ran 32 bit IDS on 32 bit Solaris, IDS would crash when it hit the max memory allowed for 32bit app on 32bit o/s. We upgraded to 64bit. Check this out: http://publib16.boulder.ibm.com/pseries/en_US/aixprggd/genprogc/ipc_limi ts.htm some info from this link: The limit that is the most confusing is the maximum number of shared memory segments that can be attached simultaneously per process. For 64-bit processes, the maximum number of shared memory segments is 268435456. For 32-bit processes, the maximum number of shared memory segments is 11, unless the extended shmat capability is used. For more information on extending shmat, see Understanding Memory Mapping. ......... The following tables summarize the shared memory limits on IPC mechanisms. Shared Memory 4.3.0 4.3.1 4.3.2 5.1 5.2 Maximum segment size (32-bit) 256 MB 2 GB 2 GB 2 GB 2 GB Maximum segment size (64-bit) 256 MB 2 GB 2 GB 64 GB 1 TB Minimum segment size 1 1 1 1 1 Maximum number of shared memory IDs 4096 4096 131072 131072 131072 Maximum number of segments per process (32-bit) 11 11 11 11 11 Good luck, Norma Jean ============================================================ The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any reproduction, dissemination or distribution of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. Thank you. Tellabs ============================================================

↪ replying to Sebastian, Norma J.
Martin Fuerderer — — source: IIUG Forums & Mailing Lists

Hi, one IDS's SHM segments can consist of several OS SHM segments. Looking at "onstat -g seg" output gives you the view of IDS. Use "ipcs-m" (as user root) to see the OS view of SHM things. It may give you a different picture. [ If IDS requests from the OS an SHM segment that is too big, it will not get it from the OS. In that case IDS is smart enough to request several smaller segments from the OS. Condition for this to work is that they are mapped consecutivly in the private memory space of the requesting process (also see earlier e-mail from me on that mapping part). With the segments then appearing consecutivly in the oninit processes' private memory space, IDS can treat these segments as one big segment. Just as if it would have received one big segment from the OS. And this is what "onstat -g seg" will show, rather than the many smaller segments that it actually had to request from the OS. ] Regards, Martin -- Martin Fuerderer IBM Informix Development Munich, Germany Information Management Informix URLs list: http://home.arcor.de/mfu1/informix/urls.html "Sebastian, Norma J." <NormaJean.Sebastian@tellabs.com> Sent by: ids-bounces@iiug.org 22.10.2006 04:55 Please respond to ids@iiug.org To ids@iiug.org cc Subject Re: onmode -ky hangs [7659] Hello Frank, You are on IDS 10.00.UC5 (32 bit) Your onstat -g seg showed : IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line (CKPT REQ) -- Up 2 days 21:04:56 -- 2008880 KbytesSegment Summary: id key addr size ovhd class blkused blkfree 7340036 1382041603 30000000 540672 672 M 132 0 9437187 1382041601 40000000 1708425216 269028 R* 417091 5 9437189 1382041602 b0000000 314572800 10248 V 73279 3521 6 1382041604 e0000000 33554432 1672 V 6482 1710 Total: - - 2,057,093,120 - - 496984 5236 I don't know much about AIX other than hearing an Informix DBA say something about a limit of 11 memory segments which isn't your problem as shown above you have 4. Back when we ran 32 bit IDS on 32 bit Solaris, IDS would crash when it hit the max memory allowed for 32bit app on 32bit o/s. We upgraded to 64bit. Check this out: http://publib16.boulder.ibm.com/pseries/en_US/aixprggd/genprogc/ipc_limi ts.htm some info from this link: The limit that is the most confusing is the maximum number of shared memory segments that can be attached simultaneously per process. For 64-bit processes, the maximum number of shared memory segments is 268435456. For 32-bit processes, the maximum number of shared memory segments is 11, unless the extended shmat capability is used. For more information on extending shmat, see Understanding Memory Mapping. .......... The following tables summarize the shared memory limits on IPC mechanisms. Shared Memory 4.3.0 4.3.1 4.3.2 5.1 5.2 Maximum segment size (32-bit) 256 MB 2 GB 2 GB 2 GB 2 GB Maximum segment size (64-bit) 256 MB 2 GB 2 GB 64 GB 1 TB Minimum segment size 1 1 1 1 1 Maximum number of shared memory IDs 4096 4096 131072 131072 131072 Maximum number of segments per process (32-bit) 11 11 11 11 11 Good luck, Norma Jean ============================================================ The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any reproduction, dissemination or distribution of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. Thank you. Tellabs ============================================================ ******************************************************************************* Forum Note: Use "Reply" to post a response in the discussion forum.

↪ replying to SANJIT CHAKRABORTY
Jonathan Leffler — — source: IIUG Forums & Mailing Lists

On 10/18/06, SANJIT CHAKRABORTY <c_sanjit@lycos.com> wrote: > The onmode is nothing but a client application. Whenever, you execute an > onmode command, it will make a new connection to the database server and each > new connection uses some virtual memory. Well, ON-Mode is not quite the same as a regular ESQL/C application, but it does connect to the database. OTOH, when it finishes, it should close the connection. Some space is used by each connection - but not all connections cause a server to allocate more memory from the system, and when a connection ends, the space should be available for reuse by other connections. -- Jonathan Leffler #include <disclaimer.h> Email: jleffler@earthlink.net, jleffler@us.ibm.com Guardian of DBD::Informix v2005.02 -- http://dbi.perl.org/

Related threads

Glossary