IDS 10.00.UC4 stalls
A user running IDS 10.00.UC4 on SUSE Linux (4GB RAM, 2 dual-core CPUs) reported intermittent server stalls under heavy load: no new connections accepted, CPU pegged near 100%, and the engine repeatedly adding shared-memory segments until the OS limit was hit, with even onmode -ky unresponsive so the process had to be killed. Suggestions were to capture diagnostics in a loop (onstat -u, -g ath, -g ses, -g mem, plus top/ps) or poll syssqlcurses/syssessions via a Perl/DBI script from cron to catch the offending SQL before the hang, and to set SHMTOTAL just below the failure point (and VP_MEMORY_CACHE_KB 0). The poster noted he can't connect once the stall occurs, and no confirmed root cause or resolution is recorded in the thread.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Hi,
we are runing IDS 10 UC4 on a Novell Linux box with 4 GB Ram and 2 Dualcore
CPUs. There are severall applications driven by the IDS as some J2EE stuff
on apache Webserver, legacy Datablade applications and even some
PowerBuilder 3rd party application.
From time to time we found the situation that IDS stalls, which we notice
having no more connections allowed, CPU load is 99,9% and within online log
i find repeated events of allocation of new shared memory segments. Usually
the IDS is running with an initial segment of 512MB, but when this problem
occurs it's addionally allocation one segment after the other in 2 or 3
hours - untill there are no more segments allowed by the OS.
I even can not stop the DB with onmode -ky (as it does not respond for some
minutes), so i have to kill the process.
I am sure there is some application running crazy - as this all occurs
during havy workload phases - but i don't know how to find out which process
is resposible for this situation. I already tried all my best with onstat -
but no success so far.
I would be very thankful fo any help or hint.
Kind Regards
Marcus
Marcus,
The following is a ugly PERL (run on AIX) script that can print out
session's running SQL activities.
You can loop it in a frequency and pipe the output to a file.....
Thanks,
Frank
#!/usr/local/bin/perl -w
#
#-------------------------------------------------------------
use DBI;
$connect = 'dbi:Informix:sysmaster';
$dbh = DBI->connect($connect) or die "could not connect to database\\
";
$dbh->{ChopBlanks} = 1;
$output = *STDOUT;
$cnt=0;
$stmt= " SELECT a.scs_sessionid, a.scs_currdb, b.uid,
b.username,b.pid,b.hostname, a.scs_sqlstatement FRO
M syssqlcurses a, syssessions b where a.scs_sessionid=b.sid";
$reptab = $dbh->prepare($stmt);
$reptab->bind_columns(undef,\\\\$sid,\\\\$dbname,\\\\$uid,\\\\$username, \\\\$pid,
\\\\$hostname,\\\\$sqs_statement);
$reptab->execute;
printf $output "\\
%16s %10s %10s %10s %16s %18s \\
", "sid",
"dbname", "uid", "username", "pid", "hos
tname" ;
while ($reptab->fetch) {
if (defined $sqs_statement) {
printf $output "\\
%16s %10s %10s %10s %16s %18s \\
", $sid,
$dbname, $uid, $username, $pid,
$hostname;
print $output " $sqs_statement \\
";
$cnt++;
}
}
$theTime=`date`;
print $output "\\
$theTime \\
total sessions: $cnt \\
";
On Thu, Jul 3, 2008 at 11:54 AM, Marcus Delp <marcus.delp@globit.com> wrote:
> Hi,
>
> we are runing IDS 10 UC4 on a Novell Linux box with 4 GB Ram and 2 Dualcore
> CPUs. There are severall applications driven by the IDS as some J2EE stuff
> on apache Webserver, legacy Datablade applications and even some
> PowerBuilder 3rd party application.
>
> >From time to time we found the situation that IDS stalls, which we notice
> having no more connections allowed, CPU load is 99,9% and within online log
> i find repeated events of allocation of new shared memory segments. Usually
> the IDS is running with an initial segment of 512MB, but when this problem
> occurs it's addionally allocation one segment after the other in 2 or 3
> hours - untill there are no more segments allowed by the OS.
>
> I even can not stop the DB with onmode -ky (as it does not respond for some
> minutes), so i have to kill the process.
>
> I am sure there is some application running crazy - as this all occurs
> during havy workload phases - but i don't know how to find out which
> process
> is resposible for this situation. I already tried all my best with onstat -
> but no success so far.
>
> I would be very thankful fo any help or hint.
>
> Kind Regards
> Marcus
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
Marcus Delp wrote:
> Hi,
>
> we are runing IDS 10 UC4 on a Novell Linux box with 4 GB Ram and 2 Dualcore
> CPUs. There are severall applications driven by the IDS as some J2EE stuff
> on apache Webserver, legacy Datablade applications and even some
> PowerBuilder 3rd party application.
>
>>From time to time we found the situation that IDS stalls, which we notice
> having no more connections allowed, CPU load is 99,9% and within online log
> i find repeated events of allocation of new shared memory segments. Usually
> the IDS is running with an initial segment of 512MB, but when this problem
> occurs it's addionally allocation one segment after the other in 2 or 3
> hours - untill there are no more segments allowed by the OS.
>
> I even can not stop the DB with onmode -ky (as it does not respond for some
> minutes), so i have to kill the process.
>
> I am sure there is some application running crazy - as this all occurs
> during havy workload phases - but i don't know how to find out which process
> is resposible for this situation. I already tried all my best with onstat -
> but no success so far.
>
> I would be very thankful fo any help or hint.
>
If you can run onstat when the issues occurrs, start with some usefull
onstats in a loop and save the output to a file it may serve as a
starter for investigation.
You might also want to add some UNIX commands to the loop
e.g:
---------------
#!/bin/bash
DATE=`date +%d.%b.%Y_%T`
OUTFILE=ids_hung_onstats_"$DATE"
[ -f "$OUTFILE" ] && rm "$OUTFILE"
while (true)
do
date '+%d %b %Y -- %T %Z' >>"$OUTFILE"
echo "" >> "$OUTFILE"
echo "onstat -u" >>"$OUTFILE"
onstat -u >>"$OUTFILE"
echo "----------------------" >> "$OUTFILE"
echo "onstat -g ath" >>"$OUTFILE"
onstat -g ath >>"$OUTFILE"
echo "----------------------" >> "$OUTFILE"
echo "onstat -g ses 0" >>"$OUTFILE"
onstat -g ses 0 >>"$OUTFILE"
echo "----------------------" >> "$OUTFILE"
echo "onstat -g mem" >>"$OUTFILE"
onstat -g mem >>"$OUTFILE"
echo "----------------------" >> "$OUTFILE"
echo "top -b -n 1" >>"$OUTFILE"
top -b -n 1 >>"$OUTFILE"
echo "----------------------" >> "$OUTFILE"
echo "ps axu" >>"$OUTFILE"
ps axu >>"$OUTFILE"
echo "" >>"$OUTFILE"
echo "================--------------- - - - - - - - " >>"$OUTFILE"sleep 10
done
----------------
HTH
Tilman
Hi Frank,
i think that will not work, as IDS does not allow any connection when this
situation occurs. Also, any SQL i start, for example in dbaccess or SQL
Editor does not respond - so this will probably be the same for sysmaster, i
guess?
Marcus
> -----Ursprüngliche Nachricht-----
> Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im
> Auftrag von FRANK
> Gesendet: Donnerstag, 3. Juli 2008 21:57
> An: ids@iiug.org
> Betreff: Re: IDS 10.00.UC4 stalls [12574]
>
> Marcus,
>
> The following is a ugly PERL (run on AIX) script that can print out
> session's running SQL activities.
> You can loop it in a frequency and pipe the output to a file.....
>
> Thanks,
> Frank
>
> #!/usr/local/bin/perl -w
> #
> #-------------------------------------------------------------
>
> use DBI;
>
> $connect = 'dbi:Informix:sysmaster';
>
> $dbh = DBI->connect($connect) or die "could not connect to
> database\\
";
>
> $dbh->{ChopBlanks} = 1;
>
> $output = *STDOUT;
>
> $cnt=0;
>
> $stmt= " SELECT a.scs_sessionid, a.scs_currdb, b.uid,
> b.username,b.pid,b.hostname, a.scs_sqlstatement FRO
> M syssqlcurses a, syssessions b where a.scs_sessionid=b.sid";
>
> $reptab = $dbh->prepare($stmt);
>
> $reptab->bind_columns(undef,\\\\$sid,\\\\$dbname,\\\\$uid,\\\\$username, \\\\$pid,
> \\\\$hostname,\\\\$sqs_statement);
>
> $reptab->execute;
>
> printf $output "\\
%16s %10s %10s %10s %16s %18s \\
", "sid",
> "dbname", "uid", "username", "pid", "hos
> tname" ;
>
> while ($reptab->fetch) {
>
> if (defined $sqs_statement) {
>
> printf $output "\\
%16s %10s %10s %10s %16s %18s \\
", $sid,
> $dbname, $uid, $username, $pid,
> $hostname;
>
> print $output " $sqs_statement \\
";
>
> $cnt++;
>
> }
>
> }
>
> $theTime=`date`;
>
> print $output "\\
$theTime \\
total sessions: $cnt \\
";
>
> On Thu, Jul 3, 2008 at 11:54 AM, Marcus Delp
> <marcus.delp@globit.com> wrote:
>
> > Hi,
> >
> > we are runing IDS 10 UC4 on a Novell Linux box with 4 GB
> Ram and 2 Dualcore
> > CPUs. There are severall applications driven by the IDS as
> some J2EE stuff
> > on apache Webserver, legacy Datablade applications and even some
> > PowerBuilder 3rd party application.
> >
> > >From time to time we found the situation that IDS stalls,
> which we notice
> > having no more connections allowed, CPU load is 99,9% and
> within online log
> > i find repeated events of allocation of new shared memory
> segments. Usually
> > the IDS is running with an initial segment of 512MB, but
> when this problem
> > occurs it's addionally allocation one segment after the
> other in 2 or 3
> > hours - untill there are no more segments allowed by the OS.
> >
> > I even can not stop the DB with onmode -ky (as it does not
> respond for some
> > minutes), so i have to kill the process.
> >
> > I am sure there is some application running crazy - as this
> all occurs
> > during havy workload phases - but i don't know how to find
> out which
> > process
> > is resposible for this situation. I already tried all my
> best with onstat -
> > but no success so far.
> >
> > I would be very thankful fo any help or hint.
> >
> > Kind Regards
> > Marcus
> >
> >
> >
> >
> **************************************************************
> *****************
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
>
>
> **************************************************************
> *****************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
Set SHMTOTAL to one segment less than where it's stalling right now.
Also, add the following to your $ONCONFIG file just in case you have a version
with this UC6 feature:
VP_MEMORY_CACHE_KB 0
Bob Roussey
Unix / Informix Administration
Spirit Airlines
Robert.Roussey@SpiritAir.com
-----Original Message-----
From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Marcus
Delp
Sent: Friday, July 04, 2008 3:30 AM
To: ids@iiug.org
Subject: AW: IDS 10.00.UC4 stalls [12578]
Hi Frank,
i think that will not work, as IDS does not allow any connection when this
situation occurs. Also, any SQL i start, for example in dbaccess or SQL
Editor does not respond - so this will probably be the same for sysmaster, i
guess?
Marcus
> -----Ursprüngliche Nachricht-----
> Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im
> Auftrag von FRANK
> Gesendet: Donnerstag, 3. Juli 2008 21:57
> An: ids@iiug.org
> Betreff: Re: IDS 10.00.UC4 stalls [12574]
>
> Marcus,
>
> The following is a ugly PERL (run on AIX) script that can print out
> session's running SQL activities.
> You can loop it in a frequency and pipe the output to a file.....
>
> Thanks,
> Frank
>
> #!/usr/local/bin/perl -w
> #
> #-------------------------------------------------------------
>
> use DBI;
>
> $connect = 'dbi:Informix:sysmaster';
>
> $dbh = DBI->connect($connect) or die "could not connect to
> database\\
";
>
> $dbh->{ChopBlanks} = 1;
>
> $output = *STDOUT;
>
> $cnt=0;
>
> $stmt= " SELECT a.scs_sessionid, a.scs_currdb, b.uid,
> b.username,b.pid,b.hostname, a.scs_sqlstatement FRO
> M syssqlcurses a, syssessions b where a.scs_sessionid=b.sid";
>
> $reptab = $dbh->prepare($stmt);
>
> $reptab->bind_columns(undef,\\\\$sid,\\\\$dbname,\\\\$uid,\\\\$username, \\\\$pid,
> \\\\$hostname,\\\\$sqs_statement);
>
> $reptab->execute;
>
> printf $output "\\
%16s %10s %10s %10s %16s %18s \\
", "sid",
> "dbname", "uid", "username", "pid", "hos
> tname" ;
>
> while ($reptab->fetch) {
>
> if (defined $sqs_statement) {
>
> printf $output "\\
%16s %10s %10s %10s %16s %18s \\
", $sid,
> $dbname, $uid, $username, $pid,
> $hostname;
>
> print $output " $sqs_statement \\
";
>
> $cnt++;
>
> }
>
> }
>
> $theTime=`date`;
>
> print $output "\\
$theTime \\
total sessions: $cnt \\
";
>
> On Thu, Jul 3, 2008 at 11:54 AM, Marcus Delp
> <marcus.delp@globit.com> wrote:
>
> > Hi,
> >
> > we are runing IDS 10 UC4 on a Novell Linux box with 4 GB
> Ram and 2 Dualcore
> > CPUs. There are severall applications driven by the IDS as
> some J2EE stuff
> > on apache Webserver, legacy Datablade applications and even some
> > PowerBuilder 3rd party application.
> >
> > >From time to time we found the situation that IDS stalls,
> which we notice
> > having no more connections allowed, CPU load is 99,9% and
> within online log
> > i find repeated events of allocation of new shared memory
> segments. Usually
> > the IDS is running with an initial segment of 512MB, but
> when this problem
> > occurs it's addionally allocation one segment after the
> other in 2 or 3
> > hours - untill there are no more segments allowed by the OS.
> >
> > I even can not stop the DB with onmode -ky (as it does not
> respond for some
> > minutes), so i have to kill the process.
> >
> > I am sure there is some application running crazy - as this
> all occurs
> > during havy workload phases - but i don't know how to find
> out which
> > process
> > is resposible for this situation. I already tried all my
> best with onstat -
> > but no success so far.
> >
> > I would be very thankful fo any help or hint.
> >
> > Kind Regards
> > Marcus
> >
> >
> >
> >
> **************************************************************
> *****************
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
>
>
> **************************************************************
> *****************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
*******************************************************************************
Forum Note: Use "Reply" to post a response in the discussion forum.
Marcus,
If you run it as a cron , it can catch the SQLs before situation occurs, so
you can know what was running when the problem is started.
Frank
On Fri, Jul 4, 2008 at 3:29 AM, Marcus Delp <marcus.delp@globit.com> wrote:
> Hi Frank,
>
> i think that will not work, as IDS does not allow any connection when this
> situation occurs. Also, any SQL i start, for example in dbaccess or SQL
> Editor does not respond - so this will probably be the same for sysmaster,
> i
> guess?
>
> Marcus
>
> > -----Ursprüngliche Nachricht-----
> > Von: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] Im
> > Auftrag von FRANK
> > Gesendet: Donnerstag, 3. Juli 2008 21:57
> > An: ids@iiug.org
> > Betreff: Re: IDS 10.00.UC4 stalls [12574]
> >
> > Marcus,
> >
> > The following is a ugly PERL (run on AIX) script that can print out
> > session's running SQL activities.
> > You can loop it in a frequency and pipe the output to a file.....
> >
> > Thanks,
> > Frank
> >
> > #!/usr/local/bin/perl -w
> > #
> > #-------------------------------------------------------------
> >
> > use DBI;
> >
> > $connect = 'dbi:Informix:sysmaster';
> >
> > $dbh = DBI->connect($connect) or die "could not connect to
> > database\\
";
> >
> > $dbh->{ChopBlanks} = 1;
> >
> > $output = *STDOUT;
> >
> > $cnt=0;
> >
> > $stmt= " SELECT a.scs_sessionid, a.scs_currdb, b.uid,
> > b.username,b.pid,b.hostname, a.scs_sqlstatement FRO
> > M syssqlcurses a, syssessions b where a.scs_sessionid=b.sid";
> >
> > $reptab = $dbh->prepare($stmt);
> >
> > $reptab->bind_columns(undef,\\\\$sid,\\\\$dbname,\\\\$uid,\\\\$username, \\\\$pid,
> > \\\\$hostname,\\\\$sqs_statement);
> >
> > $reptab->execute;
> >
> > printf $output "\\
%16s %10s %10s %10s %16s %18s \\
", "sid",
> > "dbname", "uid", "username", "pid", "hos
> > tname" ;
> >
> > while ($reptab->fetch) {
> >
> > if (defined $sqs_statement) {
> >
> > printf $output "\\
%16s %10s %10s %10s %16s %18s \\
", $sid,
> > $dbname, $uid, $username, $pid,
> > $hostname;
> >
> > print $output " $sqs_statement \\
";
> >
> > $cnt++;
> >
> > }
> >
> > }
> >
> > $theTime=`date`;
> >
> > print $output "\\
$theTime \\
total sessions: $cnt \\
";
> >
> > On Thu, Jul 3, 2008 at 11:54 AM, Marcus Delp
> > <marcus.delp@globit.com> wrote:
> >
> > > Hi,
> > >
> > > we are runing IDS 10 UC4 on a Novell Linux box with 4 GB
> > Ram and 2 Dualcore
> > > CPUs. There are severall applications driven by the IDS as
> > some J2EE stuff
> > > on apache Webserver, legacy Datablade applications and even some
> > > PowerBuilder 3rd party application.
> > >
> > > >From time to time we found the situation that IDS stalls,
> > which we notice
> > > having no more connections allowed, CPU load is 99,9% and
> > within online log
> > > i find repeated events of allocation of new shared memory
> > segments. Usually
> > > the IDS is running with an initial segment of 512MB, but
> > when this problem
> > > occurs it's addionally allocation one segment after the
> > other in 2 or 3
> > > hours - untill there are no more segments allowed by the OS.
> > >
> > > I even can not stop the DB with onmode -ky (as it does not
> > respond for some
> > > minutes), so i have to kill the process.
> > >
> > > I am sure there is some application running crazy - as this
> > all occurs
> > > during havy workload phases - but i don't know how to find
> > out which
> > > process
> > > is resposible for this situation. I already tried all my
> > best with onstat -
> > > but no success so far.
> > >
> > > I would be very thankful fo any help or hint.
> > >
> > > Kind Regards
> > > Marcus
> > >
> > >
> > >
> > >
> > **************************************************************
> > *****************
> > > Forum Note: Use "Reply" to post a response in the discussion forum.
> > >
> > >
> >
> >
> > **************************************************************
> > *****************
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
>
>
>
>
*******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>