Dynamic Server 2000 crash
Posted in 2000
A user running Informix Dynamic Server 2000 on Caldera Linux saw the engine panic ('Master Daemon Died', repeated dynamic memory segment allocations, semctl errno 22) while a PHP script did many inserts of long text into lvarchar columns. Replies blamed shared-memory limits: raise Linux kernel shmmax (plus file-max, bdflush, overcommit_memory), check segment usage with ipcs/onstat -g seg, add swap, and enlarge SHMVIRTSIZE/SHMADD. After applying these the user still hit -208 memory allocation errors and posted his full ONCONFIG, kernel settings and script; no fix is recorded, though he noted dumping rows to a file and loading with dbload worked.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Topics: Error Codes & Troubleshooting, Data Types & Schema Design, Logging & Checkpoints, Platform-Specific Issues
I was using IF 2000 on a Caldera Linux system. I wrote a program that did lots of inserts into tables -- specifically large lines of text into lvarchar fields. During this process, the server crashed. The database holding the table that I was using uses No Logging, so I assume that the problem wasn't that I filled my logical logs. The Informix error log held the messages: dynamically allocating new memory segment (size: 16777216) -- repeats 10 times The Master Daemon Died invoke_alarm(): mt_exec failed, status 32256, errno 0 PANIC: Attempting to bring system down semctl: errno = 22 -- repeats a number of times Does anyone have an idea of what might be doing this, or how I can diagnose it? I have 512 Mb of memory, so it doesn't seem like I should have run out.
Check kernel parameters I am not sure you can have more than 10 shared memory segments under Linux. Increase SHMVIRtSIZE in your ONCONFIG file. Also check the kernel parameters for semaphores. Richard Puchalsky wrote in message <1AZm5.61869$RG6.3235859@bgtnsc05-news.ops.worldnet.att.net>... >I was using IF 2000 on a Caldera Linux system. I wrote a program that did >lots of inserts into tables -- specifically large lines of text into >lvarchar fields. During this process, the server crashed. The database >holding the table that I was using uses No Logging, so I assume that the >problem wasn't that I filled my logical logs. > >The Informix error log held the messages: > >dynamically allocating new memory segment (size: 16777216) -- repeats 10 >times >The Master Daemon Died >invoke_alarm(): mt_exec failed, status 32256, errno 0 >PANIC: Attempting to bring system down >semctl: errno = 22 -- repeats a number of times > >Does anyone have an idea of what might be doing this, or how I can diagnose >it? I have 512 Mb of memory, so it doesn't seem like I should have run >out. > >
Hi Richard, play with the following (for Linux 2.2.x). echo -n "Setting kernel parameters for Informix" echo "16384" > /proc/sys/fs/file-max echo "5 128 64 256 15 3000 500 1884 2" > /proc/sys/vm/bdflush echo "0" > /proc/sys/vm/overcommit_memory echo "400000000" > /proc/sys/kernel/shmmax It looks as if you have not enough shared memory available. The linux kernel standard shmem is much too low for larger Informix tasks. The above shmmax entry is for a system with 512MB of RAM. Peter Richard Puchalsky wrote: > > I was using IF 2000 on a Caldera Linux system. I wrote a program that did > lots of inserts into tables -- specifically large lines of text into > lvarchar fields. During this process, the server crashed. The database > holding the table that I was using uses No Logging, so I assume that the > problem wasn't that I filled my logical logs. > > The Informix error log held the messages: > > dynamically allocating new memory segment (size: 16777216) -- repeats 10 > times > The Master Daemon Died > invoke_alarm(): mt_exec failed, status 32256, errno 0 > PANIC: Attempting to bring system down > semctl: errno = 22 -- repeats a number of times > > Does anyone have an idea of what might be doing this, or how I can diagnose > it? I have 512 Mb of memory, so it doesn't seem like I should have run > out.
"Peter Eckhardt" <Peter.Eckhardt@transcom.de> wrote: > Hi Richard, > > play with the following (for Linux 2.2.x). > > echo -n "Setting kernel parameters for Informix" > echo "16384" > /proc/sys/fs/file-max > echo "5 128 64 256 15 3000 500 1884 2" > /proc/sys/vm/bdflush > echo "0" > /proc/sys/vm/overcommit_memory > echo "400000000" > /proc/sys/kernel/shmmax > > It looks as if you have not enough shared memory available. The linux > kernel standard shmem is much too low for larger Informix tasks. The above > shmmax entry is for a system with 512MB of RAM. Thanks, I'll try it. I note that there is also a /proc/sys/kernel/shmall in my Linux; any idea what that should be?
"Peter Eckhardt" <Peter.Eckhardt@transcom.de> wrote: > play with the following (for Linux 2.2.x). > > echo -n "Setting kernel parameters for Informix" > echo "16384" > /proc/sys/fs/file-max > echo "5 128 64 256 15 3000 500 1884 2" > /proc/sys/vm/bdflush > echo "0" > /proc/sys/vm/overcommit_memory > echo "400000000" > /proc/sys/kernel/shmmax > > It looks as if you have not enough shared memory available. The linux > kernel standard shmem is much too low for larger Informix tasks. The above > shmmax entry is for a system with 512MB of RAM. Well, I tried this, and I still get a -208 memory allocation error on one of my insert statements (a different one each time I try the program). Does anyone have any other ideas?
Hi Richard,
maybe you have already allocated too much shared memory segments.
Check with ipcs.
If this is a problem you can either try to allow more segments in
the linux kernel (don't know if linux allows that, check in the
kernel documentation). Or have a larger initial shared memory
segment, and larger chunks of shared mem in informix. We use:
SHMVIRTSIZE 16000 # initial virtual shared memory segment size
SHMADD 32000 # Size of new shared memory segments (Kbytes)
Peter
Richard Puchalsky wrote:
>
> "Peter Eckhardt" <Peter.Eckhardt@transcom.de> wrote:
> > play with the following (for Linux 2.2.x).
> >
> > echo -n "Setting kernel parameters for Informix"
> > echo "16384" > /proc/sys/fs/file-max
> > echo "5 128 64 256 15 3000 500 1884 2" > /proc/sys/vm/bdflush
> > echo "0" > /proc/sys/vm/overcommit_memory
> > echo "400000000" > /proc/sys/kernel/shmmax
> >
> > It looks as if you have not enough shared memory available. The linux
> > kernel standard shmem is much too low for larger Informix tasks. The above
> > shmmax entry is for a system with 512MB of RAM.
>
> Well, I tried this, and I still get a -208 memory allocation error on one of
> my insert statements (a different one each time I try the program). Does
> anyone have any other ideas?
Add more swap space.
You are running out of memory.
What are the BUFFERS,SHMVIRTSIZE,SHMADD entries
in $INFORMIXDIR/etc/$ONCONFIG set to?
What does onstat -g seg give?
Richard Puchalsky wrote in message ...
>
>"Peter Eckhardt" <Peter.Eckhardt@transcom.de> wrote:
>> play with the following (for Linux 2.2.x).
>>
>> echo -n "Setting kernel parameters for Informix"
>> echo "16384" > /proc/sys/fs/file-max
>> echo "5 128 64 256 15 3000 500 1884 2" > /proc/sys/vm/bdflush
>> echo "0" > /proc/sys/vm/overcommit_memory
>> echo "400000000" > /proc/sys/kernel/shmmax
>>
>> It looks as if you have not enough shared memory available. The linux
>> kernel standard shmem is much too low for larger Informix tasks. The
above
>> shmmax entry is for a system with 512MB of RAM.
>
>Well, I tried this, and I still get a -208 memory allocation error on one
of
>my insert statements (a different one each time I try the program). Does
>anyone have any other ideas?
>
>
>
I had a crash a while back and my posts seem to have gotten forwarded to the
linux-informix mailing list, from which I got some list responses. I'm not
sure how the gatewaying works between comp.databases.informix and there, so
I'm just going to post to c.d.i. and trust that it will find its way to
wherever it needs to go.
Last time I tried the program in question, I got a --208 memory allocation
error. People wanted more information, so I'm going to dump a lot of it.
Thanks to everyone who has tried to help me.
My machine is a dual-CPU Xeon with 512 Mb RAM. running Caldera eServer 2.3
Linux and IF 2000. I think I set aside 2 Gb of disk space for swap space.
First, my onconfig settings (the remotely relevant ones):
# Physical Log Configuration
PHYSDBS rootdbs # Location (dbspace) of physical log
PHYSFILE 4000 # Physical log file size (Kbytes)
# Logical Log Configuration
LOGFILES 6 # Number of logical log files
LOGSIZE 5000 # Logical log size (Kbytes)
# System Configuration
SERVERNUM 1 # Unique id corresponding to a OnLineinstance
DBSERVERNAME grserver # Name of default database server
DBSERVERALIASES # List of alternate dbservernames
NETTYPE ipcshm,2,100,CPU # Configure poll thread(s) for nettype
DEADLOCK_TIMEOUT 60 # Max time to wait of lock in distributed
env.
RESIDENT 1 # Forced residency flag (Yes = 1, No = 0)
MULTIPROCESSOR 1 # 0 for single-processor, 1 formulti-processor
SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vps toone
# Shared Memory Parameters
LOCKS 2000 # Maximum number of locks
BUFFERS 50000 # Maximum number of shared buffers
PHYSBUFF 32 # Physical log buffer size (Kbytes)
LOGBUFF 32 # Logical log buffer size (Kbytes)LOGSMAX 6 # Maximum number of logical log files
CLEANERS 3 # Number of buffer cleaner processes
SHMBASE 0x10000000 #define SHMBASE 0x10000000L # Sharedmemory base address
SHMVIRTSIZE 8000 # initial virtual shared memory segment size
SHMADD 32000 # Size of new shared memory segments
(Kbytes)
SHMTOTAL 320000 # Total shared memory (Kbytes). 0=>unlimited
CKPTINTVL 9000 # Check point interval (in sec)
LRUS 8 # Number of LRU queues
LRU_MAX_DIRTY 60 # LRU percent dirty begin cleaning limit
LRU_MIN_DIRTY 50 # LRU percent dirty end cleaning limit
LTXHWM 50 # Long transaction high water markpercentage
LTXEHWM 60 # Long transaction high water mark
(exclusive)
TXTIMEOUT 0x12c # Transaction timeout (in sec)
STACKSIZE 32 # Stack size (Kbytes)
Next, my Linux kernel settings. I have no idea where the semaphore controls
are located:
/proc/sys/fs/file-max: 16384
/proc/sys/vm/bdflush : 40 500 64 256 500 3000 500 1884 2
/proc/sys/vm/overcommit_memory: 0
/proc/sys/kernel/shmmax: 400000000
/proc/sys/kernel/shmall: 400000000
Next, the program I was running was in 3.0.16 PHP. It's the first PHP
program I've ever written, so don't be too hard on me. It is intended to be
invoked from the shell, and to read lines from a table that has a sequence
number filed and an lvarchar field, find lines in the lvarchar that have
header numbers (numbers followed by tabs), and re-write the table into a
another table with the header numbers in seperate fields, and used to label
all rows. I realize that I could do this in some other programming
language, but I was trying to test PHP. It is:
#!/usr/local/bin/php
<?php
$conn1 = ifx_pconnect ("gr","informix","password") ;
set_time_limit(0);
if (! $conn1) die ("No connection!\\n") ;
$qid1 = ifx_query ("select * from rmp_text_tmp order by seq", $conn1) ;
if (! $qid1) die ("No retrieval!\\n") ;
$count = 0 ;
$count2 = 0 ;
$execsum = 0;
$rmp_id = 0 ;
$seq_num = 0 ;
while (true) {
$row = ifx_fetch_row ($qid1) ;
if ($count > 100 ) break ;
if (is_array($row)) {
$count++ ;
if ($posn=strpos($row["line"]," ")) {
$count2++ ;
$pieces = explode (" ",$row["line"]) ;
$execsum = $pieces [0] ;
$rmp_id = $pieces [1] ;
$line = str_replace('"','""', chop ($pieces [2])) ;
}
else {
$line = str_replace('"','""',chop($row ["line"])) ;
}
$seq_num = $row ["seq"] ;
$ins = 'insert into rmp_execsum values (' . $execsum . ',' . $rmp_id
. ',' . $seq_num . ',"' . $line . '")' ;
$prep1 = ifx_query ($ins,$conn1) ;
if (! $prep1) die ("bad insert: $ins\\n") ;
}
else
break ;
}
echo "$count\\n" ;
echo "$count2\\n" ;
?>
The above PHP program works fine if I just dump the lines to a file and use
dbload to load them back into Informix, rather than trying to insert them
directly.