RE: System Freezes

Thanks Fernando. The UNIX system logs and the Informix logs show nothing out of the usual. The only thing that changed in our system, and it corresponded with these freezes, is that we rolled out a group of perl CGI web pages along with a handful of new tables to store the data for this new system. We have used Cognos Impromtu and have had a person or two use Microsoft Access to run queries from time to time and I have experienced these types of freezes when those applications have run some pretty large queries. I am also surprised that the OS seems locked by database activity, but I have seen it. I have top and dbtop open, but those freeze too for 20-30 seconds so I cannot see what is going on during the freeze. When the system again becomes responsive, the output from those tools seems normal. There shouldn't be anything going on in our system this week related to the filesystem that wasn't occurring last week. Again, the only difference is a handful of CGI's that are running SELECT's, INSERT's, and UPDATE's to our database. From: Fernando Nunes [mailto:domusonline@gmail.com] Sent: Tuesday, January 11, 2011 3:07 PM To: Rubinstein, James Cc: informix-list@iiug.org Subject: Re: System Freezes On Tue, Jan 11, 2011 at 9:29 PM, Rubinstein, James <JRUBIN@midwestern.edu> wrote: I'm running IDS 11.50.FC6 on HPUX 11.31. We recently rolled out an internally developed web-based system (Apache perl/mod_perl) and immediately started noticing that our HPUX system becomes completely unresponsive, for 20-30 seconds at a time, many times throughout the day. My first thought was network problems, but we have pretty much ruled this out since I can connect to a twin HPUX server which seems fine during the outages. During these system freezes, any connections to the database fail and the system is completely unresponsive to the point that I cannot even type any commands at the shell. I have seen this behavior in the past when the oninit processes use a lot of CPU resources, but it is usually pretty easy to track these down to some bad SQL/report writing. In this case, I'm trying to figure out what may be causing the system freezes. I have top and the dbtop utility from IIUG, but those don't refresh during or freezes. I am also unable to type any onstat commands until the system comes back. By that time, everything looks pretty normal with low load averages and our oninit processes at normal levels. I'm looking at the various system reports in OAT, but don't see anything that jumps out as the culprit. I'd appreciate any troubleshooting ideas. It may look as I'm defending Informix, but I find it very hard to believe that the symptoms you describe can be caused by any kind of bad SQL. I'd start looking at memory usage etc. Check the OS ratios for filesystem cache vs program memory. Check you memory usage. Try to keep a "top" or similar tool open and see if you notice something. Take frequent snapshots of paging status (so that you can compare before/after counters. Also check your system logs. Last time I saw something similar (not on HP) it was the filesystem/program memory ratios. If froze the machine whenever a filesystem intensive operation was run. By no means I'm insinuating it's everything ok with Informix, but whatever happens with it should not cause that effect. Regards. -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently...