Ontape Backup Problem
An ontape archive on IDS 9.40 (Tru64) failed with "ISAM error: cannot allocate memory", with the online.log showing shmat ENOMEM and "out of virtual shared memory". Advice: check onstat -g seg for extra V segments, review machine-specific release notes/kernel params (MAXMEM), and roll the added virtual segments into a larger SHMVIRTSIZE (plus bigger SHMADD) then restart; Art Kagel noted the shmat failure meant no valid address space, not exhausted memory. Raising SHMVIRTSIZE to 455680 stopped the server from starting, so the user reverted to 327680; after the restart only one V segment existed and backups worked again. The underlying cause (likely kernel tuning) was never confirmed.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
I am getting following error message.
Archive Failed - ISAM error: cannot allocate memory.
The entry in online.log file.
03:17:43 Checkpoint loguniq 606145, logpos 0x232018, timestamp: -1361032356
03:17:43 Maximum server connections 505
03:17:43 shmat: [ENOMEM][12]: out of available data space, check system MAXMEM
03:17:43 out of virtual shared memory
03:17:44 Assert Warning: ISAM error: cannot allocate memory.
03:17:44 IBM Informix Dynamic Server Version 9.40.FC2
03:17:44 Who: Session(78398, informix@es45-1.pasco.gov, 77781, 22c4bfcc8)
Thread(79050, ontape, 22bac68e8, 3)
File: rsarcutl.c Line: 119
03:17:44 Action: init_arcbu()/SCBMALLOC()
03:17:44 stack trace for pid 922 written to /InformixBackup/tmp/af.38b2ada7
03:17:44 See Also: /InformixBackup/tmp/af.38b2ada7
03:17:44 ISAM error: cannot allocate memory.
03:22:49 Fuzzy Checkpoint Completed: duration was 0 seconds, 2 buffers not
flushed,
timestamp: -1360931041.
Informix Version: IBM Informix Dynamic Server Version 9.40.FC2
OS Version: Compaq Tru64 UNIX V5.1A (Rev. 1885)
We have not change any parameters or system hardware.
It was working fine until yesterday.
Thank you
Take a look at your onstat -g seg output and see if it has created multiple
virtual (V) segments. Some servers (may be old news) can only handle up to 10
"V" segments before it will stop allocating segments. If this is the case, you
will need to incorporate those extra segments and redefine your SHMVIRTSIZE
(within your $ONCONFIG) and reboot. You may want to increase the $ONCONFIG
SHMADD parameter as well.
Take care.
Clifton
> To: ids@iiug.org
> From: kbakarania@pascoclerk.com
> Subject: Ontape Backup Problem [18147]
> Date: Wed, 18 Nov 2009 09:06:04 -0500
>
> I am getting following error message.
>
> Archive Failed - ISAM error: cannot allocate memory.
>
> The entry in online.log file.
>
> 03:17:43 Checkpoint loguniq 606145, logpos 0x232018, timestamp: -1361032356
>
> 03:17:43 Maximum server connections 505
> 03:17:43 shmat: [ENOMEM][12]: out of available data space, check system
MAXMEM
>
> 03:17:43 out of virtual shared memory
>
> 03:17:44 Assert Warning: ISAM error: cannot allocate memory.
>
> 03:17:44 IBM Informix Dynamic Server Version 9.40.FC2
> 03:17:44 Who: Session(78398, informix@es45-1.pasco.gov, 77781, 22c4bfcc8)
>
> Thread(79050, ontape, 22bac68e8, 3)
>
> File: rsarcutl.c Line: 119
> 03:17:44 Action: init_arcbu()/SCBMALLOC()
> 03:17:44 stack trace for pid 922 written to /InformixBackup/tmp/af.38b2ada7
> 03:17:44 See Also: /InformixBackup/tmp/af.38b2ada7
> 03:17:44 ISAM error: cannot allocate memory.
>
> 03:22:49 Fuzzy Checkpoint Completed: duration was 0 seconds, 2 buffers not
> flushed,
> timestamp: -1360931041.
>
> Informix Version: IBM Informix Dynamic Server Version 9.40.FC2
>
> OS Version: Compaq Tru64 UNIX V5.1A (Rev. 1885)
>
> We have not change any parameters or system hardware.
> It was working fine until yesterday.
>
> Thank you
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
_________________________________________________________________
Windows 7: It works the way you want. Learn more.
http://www.microsoft.com/Windows/windows-7/default.aspx?ocid=PID24727::T:WLMTAGL
:ON:WL:en-US:WWL_WIN_evergreen:112009v2
I have only three multiple Virtual segments (V), one (M) and one (*R) segments.
Here is the output for onstat -g seg
IBM Informix Dynamic Server Version 9.40.FC2 -- On-Line -- Up 6 days 11:45:3
3 -- 999424 Kbytes
Segment Summary:
id key addr size ovhd class blkused blkfree
0 1381386241 200000000 595591168 437760 R* 144776 632
1 1381386242 223800000 335544320 10752 V 68212 13708
2 1381386243 237800000 8388608 768 M 1231 817
3 1381386244 238000000 41943040 1792 V 8273 1967
4 1381386245 23a800000 41943040 1792 V 8206 2034
Total: - - 1023410176 - - 230698 19158
Thank you
I would suggest looking at your machine-specific release notes and make sure
the kernel parameters are set as they should be.
Take care.
Clifton
> To: ids@iiug.org
> From: kbakarania@pascoclerk.com
> Subject: Re: RE: Ontape Backup Problem [18151]
> Date: Wed, 18 Nov 2009 10:13:18 -0500
>
> I have only three multiple Virtual segments (V), one (M) and one (*R)
> segments.
> Here is the output for onstat -g seg
>
> IBM Informix Dynamic Server Version 9.40.FC2 -- On-Line -- Up 6 days 11:45:3
> 3 -- 999424 Kbytes>
> Segment Summary:
> id key addr size ovhd class blkused blkfree
> 0 1381386241 200000000 595591168 437760 R* 144776 632
> 1 1381386242 223800000 335544320 10752 V 68212 13708
> 2 1381386243 237800000 8388608 768 M 1231 817
> 3 1381386244 238000000 41943040 1792 V 8273 1967
> 4 1381386245 23a800000 41943040 1792 V 8206 2034
> Total: - - 1023410176 - - 230698 19158
>
> Thank you
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
_________________________________________________________________
Windows 7: It works the way you want. Learn more.
http://www.microsoft.com/Windows/windows-7/default.aspx?ocid=PID24727::T:WLMTAGL
:ON:WL:en-US:WWL_WIN_evergreen:112009v2
OK, errno 12 is 'Cannot allocate memory' which is as massively unhelpful as
errno's 5 & 6 'read error' and 'write error'. However, I've seen this
before, both in IDS and in my own systems level coding. The fact that it
was returned from shmat() rather than shmget() means that you haven't run
out of memory, but there was no valid address location in the oninit
executable to which the OS could attach a shared memory segment the size of
the one that was allocated by shmget(). This is a common problem on AIX,
though it occurs on other UNIX releases as well. What to do? You have to
adjust shared memory base address in the ONCONFIG file or load more shared
memory up front when the address space of the oninit processes are cleaner.
Try increasing the SHMVIRTSIZE to include all additional virtual segments
that have been added to the instance since startup and restart the
instance. That should help for a while.
If you have a C programmer or systems programmer on staff who's good at
debugging, he/she may be able to help you determine a different value for
SHMBASE. If not, contact IBM tech support and they should be able to help.
Art
Art S. Kagel
Oninit (www.oninit.com)
IIUG Board of Directors (art@iiug.org)
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Oninit, the IIUG, nor any other organization
with which I am associated either explicitly or implicitly. Neither do
those opinions reflect those of other individuals affiliated with any entity
with which I am affiliated nor those of the entities themselves.
On Wed, Nov 18, 2009 at 9:06 AM, KANTI BAKARANIA
<kbakarania@pascoclerk.com>wrote:
> I am getting following error message.
>
> Archive Failed - ISAM error: cannot allocate memory.
>
> The entry in online.log file.
>
> 03:17:43 Checkpoint loguniq 606145, logpos 0x232018, timestamp: -1361032356
>
> 03:17:43 Maximum server connections 505
> 03:17:43 shmat: [ENOMEM][12]: out of available data space, check system
> MAXMEM
>
> 03:17:43 out of virtual shared memory
>
> 03:17:44 Assert Warning: ISAM error: cannot allocate memory.
>
> 03:17:44 IBM Informix Dynamic Server Version 9.40.FC2
> 03:17:44 Who: Session(78398, informix@es45-1.pasco.gov, 77781, 22c4bfcc8)
>
> Thread(79050, ontape, 22bac68e8, 3)
>
> File: rsarcutl.c Line: 119
> 03:17:44 Action: init_arcbu()/SCBMALLOC()
> 03:17:44 stack trace for pid 922 written to /InformixBackup/tmp/af.38b2ada7
> 03:17:44 See Also: /InformixBackup/tmp/af.38b2ada7
> 03:17:44 ISAM error: cannot allocate memory.
>
> 03:22:49 Fuzzy Checkpoint Completed: duration was 0 seconds, 2 buffers not
> flushed,
> timestamp: -1360931041.
>
> Informix Version: IBM Informix Dynamic Server Version 9.40.FC2
>
> OS Version: Compaq Tru64 UNIX V5.1A (Rev. 1885)
>
> We have not change any parameters or system hardware.
> It was working fine until yesterday.
>
> Thank you
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--00151747965a238ab40478a7732e
OK, so increase SHMVIRTSIZE by 125MB to allow for the third virtual segment
that couldn't be attached and restart the server.
Art
Art S. Kagel
Oninit (www.oninit.com)
IIUG Board of Directors (art@iiug.org)
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Oninit, the IIUG, nor any other organization
with which I am associated either explicitly or implicitly. Neither do
those opinions reflect those of other individuals affiliated with any entity
with which I am affiliated nor those of the entities themselves.
On Wed, Nov 18, 2009 at 10:13 AM, KANTI BAKARANIA <kbakarania@pascoclerk.com
> wrote:
> I have only three multiple Virtual segments (V), one (M) and one (*R)
> segments.
> Here is the output for onstat -g seg
>
> IBM Informix Dynamic Server Version 9.40.FC2 -- On-Line -- Up 6 days
> 11:45:3
> 3 -- 999424 Kbytes>
> Segment Summary:
> id key addr size ovhd class blkused blkfree
> 0 1381386241 200000000 595591168 437760 R* 144776 632
> 1 1381386242 223800000 335544320 10752 V 68212 13708
> 2 1381386243 237800000 8388608 768 M 1231 817
> 3 1381386244 238000000 41943040 1792 V 8273 1967
> 4 1381386245 23a800000 41943040 1792 V 8206 2034
> Total: - - 1023410176 - - 230698 19158
>
> Thank you
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--0015174769c85e66780478a7bb50
I changed SHMVIRTSIZE to 455680 from 327680. Try to reboot and I got the
following message.
19:11:59 Checkpoint loguniq 606440, logpos 0x1f9364, timestamp: -1329617785
19:11:59 Maximum server connections 505
19:18:34 IBM Informix Dynamic Server Started.
19:18:35 Segment locked: addr=0x200000000, size=595591168
19:18:35 Requested shared memory segment size rounded from 455680KB to 458752KB
19:18:35 shmat: [ENOMEM][12]: out of available data space, check system MAXMEM
19:18:35 mt_shm_init: can't create virtual segment
Manchine does not come up. I changed back the value for SHMVIRTSIZE to 327680
& reboot.
Informix came up (Online). The ontape backup also work.
onstat -g seg output after reboot is as follows.
# onstat -g seg
IBM Informix Dynamic Server Version 9.40.FC2 -- On-Line -- Up 13:26:35 -- 917504 Kbytes
Segment Summary:
id key addr size ovhd class blkused bl
kfree
0 1381386241 200000000 595591168 437760 R* 144776 63
2
1 1381386242 223800000 335544320 10752 V 34846 47
074
2 1381386243 237800000 8388608 768 M 1231 81
7
Total: - - 939524096 - - 180853 48
523
(* segment locked in memory)
Now only one V segment is there.
I don't know what was the problen but after rebooting the machine ontape
backup worked.
Thank you all
Kanti
This really points to a kernel issue, one of whose parameters is likely to be
MAXMEM or some variant thereof.
Do read the machine-specific release notes and make the suggested changes
(items listed in those notes are minimums, so if any of your numbers are
higher, best leave the higher number.
Take care.
Clifton
> To: ids@iiug.org
> From: kbakarania@pascoclerk.com
> Subject: Re: RE: Ontape Backup Problem [18167]
> Date: Thu, 19 Nov 2009 09:07:45 -0500
>
> I changed SHMVIRTSIZE to 455680 from 327680. Try to reboot and I got the
> following message.
>
> 19:11:59 Checkpoint loguniq 606440, logpos 0x1f9364, timestamp: -1329617785
>
> 19:11:59 Maximum server connections 505
> 19:18:34 IBM Informix Dynamic Server Started.
> 19:18:35 Segment locked: addr=0x200000000, size=595591168
> 19:18:35 Requested shared memory segment size rounded from 455680KB to
> 458752KB
> 19:18:35 shmat: [ENOMEM][12]: out of available data space, check system
MAXMEM
>
> 19:18:35 mt_shm_init: can't create virtual segment
>
> Manchine does not come up. I changed back the value for SHMVIRTSIZE to 327680
> & reboot.
>
> Informix came up (Online). The ontape backup also work.
>
> onstat -g seg output after reboot is as follows.>
> # onstat -g seg>
> IBM Informix Dynamic Server Version 9.40.FC2 -- On-Line -- Up 13:26:35 -- 91> 7504 Kbytes
>
> Segment Summary:
> id key addr size ovhd class blkused bl
> kfree
> 0 1381386241 200000000 595591168 437760 R* 144776 63
> 2
> 1 1381386242 223800000 335544320 10752 V 34846 47
> 074
> 2 1381386243 237800000 8388608 768 M 1231 81
> 7
> Total: - - 939524096 - - 180853 48
> 523
>
> (* segment locked in memory)
>
> Now only one V segment is there.
>
> I don't know what was the problen but after rebooting the machine ontape
> backup worked.
>
> Thank you all
>
> Kanti
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
_________________________________________________________________
Hotmail: Trusted email with Microsoft's powerful SPAM protection.
http://clk.atdmt.com/GBL/go/177141664/direct/01/
http://clk.atdmt.com/GBL/go/177141664/direct/01/
Related threads
- Posting from the Informix-list
- Migrating from IDS 9.40.UC6 to 11.50.UC3
- Ip for a network session
- questions onstat -g