errors: -104 (too many files) and -242 (could not open database table), among others

hello all - we are on version 11.5.FC7 (from v10.fc5) running HDR on AIX 5.3 about 5 days after upgrading, the developers started getting an error report from their cold fusion servers with several errors in the debugging/catch information : Description: Error Executing Database Query. Cannot open system catalog (sysviews). ErrorCode -311 Message Cannot open system catalog (sysviews). ErrorCode -104 Message ISAM error: too many files open. other times the error will read as : ErrorCode -104 Message ISAM error: too many files open. -242 Could not open database table table-name (and the table name is sometimes a system table and other times a regular database table or will say Could not open database table (%s) the Cold Fusion debugging info shows the data source as usually being the Secondary server in the HDR pair. We have increased the nofiles (in ulimit) from 2000 to unlimited on the secondary server and also increased the nofiles on the app server as well but that did not resolve the error. On our dev tests, we are able to trap for the -242 error on the db server (and send info to support) but the trap will not fire off on the db server for the -104 error. the DB message log does not ever show an error nor does the OS show any errors in logging, debugging, etc Also, we went from AIO to using KAIO about a month before the v11 upgrade. Trying to get support from Adobe for Cold fusion is about as painful (and as helpful) as sticking needles in your eye as we are exploring what could be wrong in all areas since we really don't understand where the problem is emanating from. any ideas are appreciated. thanks, tom

On Nov 5, 3:21 pm, Tom Lehr <tomc...@gmail.com> wrote: > hello all - > > we are on version 11.5.FC7 (from v10.fc5) running HDR on AIX 5.3 > > about 5 days after upgrading, the developers started getting an error > report from their cold fusion servers with several errors in the > debugging/catch information : > > Description:    Error Executing Database Query. Cannot open system > catalog (sysviews). > > ErrorCode       -311 > Message Cannot open system catalog (sysviews). > > ErrorCode       -104 > Message ISAM error: too many files open. > > other times the error will read as : > > ErrorCode       -104 > Message ISAM error: too many files open. > > -242 > Could not open database table table-name > (and the table name is sometimes a system table and other times a > regular database table > or will say > Could not open database table (%s) > > the Cold Fusion debugging info shows the data source as usually being > the Secondary server in the HDR pair. > > We have increased the nofiles (in ulimit) from 2000 to unlimited on > the secondary server > and also increased the nofiles on the app server as well but that did > not resolve the error. > > On our dev tests, we are able to trap for the -242 error on the db > server (and send info to support) but the trap will not fire off on > the db server for the -104 error. > > the DB message log does not ever show an error nor does the OS show > any errors in logging, debugging, etc > > Also, we went from AIO to using KAIO about a month before the v11 > upgrade. > > Trying to get support from Adobe for Cold fusion is about as painful > (and as helpful) as sticking needles in your eye as we are exploring > what could be wrong in all areas since we really don't understand > where the problem is emanating from. > > any ideas are appreciated. > > thanks, > tom The 104 error, assuming it is coming from the engine, does not have anything to do with the physical number of open files. From looking at the code it only comes from 2 places, and it's dealing with an engine internal representation of open files, which has a limit of like 32k. In 1 of the places it's set, it should actually trap with the onmode -I, if that is not working, it would seem to imply it's coming from the other location. It sounds like it could be some sort of internal open table structure leak which is making the engine think you have more then 32k tables open. I would possibly suggest using onstat -g opn to monitor the number of tables each thread hasopen....maybe try to see if any threads have a continuiously increasing open table count, or any open tables counts near 32k. Jacques Renaut IBM Informix Advanced Support APD Team

On 05/11/2010 21:13, jrenaut wrote: > On Nov 5, 3:21 pm, Tom Lehr<tomc...@gmail.com> wrote: >> hello all - >> >> we are on version 11.5.FC7 (from v10.fc5) running HDR on AIX 5.3 >> >> about 5 days after upgrading, the developers started getting an error >> report from their cold fusion servers with several errors in the >> debugging/catch information : >> >> Description: Error Executing Database Query. Cannot open system >> catalog (sysviews). >> >> ErrorCode -311 >> Message Cannot open system catalog (sysviews). >> >> ErrorCode -104 >> Message ISAM error: too many files open. >> >> other times the error will read as : >> >> ErrorCode -104 >> Message ISAM error: too many files open. >> >> -242 >> Could not open database table table-name >> (and the table name is sometimes a system table and other times a >> regular database table >> or will say >> Could not open database table (%s) >> >> the Cold Fusion debugging info shows the data source as usually being >> the Secondary server in the HDR pair. >> >> We have increased the nofiles (in ulimit) from 2000 to unlimited on >> the secondary server >> and also increased the nofiles on the app server as well but that did >> not resolve the error. >> >> On our dev tests, we are able to trap for the -242 error on the db >> server (and send info to support) but the trap will not fire off on >> the db server for the -104 error. >> >> the DB message log does not ever show an error nor does the OS show >> any errors in logging, debugging, etc >> >> Also, we went from AIO to using KAIO about a month before the v11 >> upgrade. >> >> Trying to get support from Adobe for Cold fusion is about as painful >> (and as helpful) as sticking needles in your eye as we are exploring >> what could be wrong in all areas since we really don't understand >> where the problem is emanating from. >> >> any ideas are appreciated. >> >> thanks, >> tom > > The 104 error, assuming it is coming from the engine, does not have > anything to do with the physical number of open files. From looking > at the code it only comes from 2 places, and it's dealing with an > engine internal representation of open files, which has a limit of > like 32k. In 1 of the places it's set, it should actually trap with > the onmode -I, if that is not working, it would seem to imply it's > coming from the other location. It sounds like it could be some sort > of internal open table structure leak which is making the engine think > you have more then 32k tables open. I would possibly suggest using > onstat -g opn to monitor the number of tables each thread has> open....maybe try to see if any threads have a continuiously > increasing open table count, or any open tables counts near 32k. > > Jacques Renaut > IBM Informix Advanced Support > APD Team Could also check: ls -l oninit - i.e. ensure correct permissions and ulimit -a - i.e. environment limits when starting the engine there may be something in errpt -a