Help wanted: Bug w/NCR StarServe SMP using Informix OL 4.00/4.10 (Long)

We have a pair of AT&T/NCR Star Server E platforms, running AT&T NCR UNIX SVR 4.0 3.0 (Patch level B1 or B2) and 4.0 3.1 Processors: 4.0 3.1 64M memory, dual full cache, 2x486/66 4.0 3.0 128 M memory, dual full cache, 2x 486/33 DIsk: Local to each: SCSI SE 1-5 Gbytes Accesible to both: SCSI D 3x1-2 Gbytes per spindle In short, we got lotsa horses. Tunings are vanilla with a full load of tools. Appropriate versions and all knwon and correct patch fixes are installed Hu! Comments on known problems about patch fixes would be appreciated. This includes both standard maintenance diskettes and tapes, as well as individual fixes. RFS, NFS, TCP/IP, and all standard networking packages are installed and operational with correct revs -- RFS/nFS are configured over TCP/IP and active BUT NOT IN USE. We have a STar Server X/C differential SCSI external tower shared between two SS E's SMP (4.0 3.0 B1 and 4.0 3.1) with only entire disks in use on a given processor. restarting one system has, once dumped the other machine. HA resets are bveing clearly sen when system restarts occur. Symptoms we find suspicious: - cron has abended (main daemon) while running SQL scripts. Return code of 377*613 (-117 32 bits as unsigned from cronlog) Since RSTCHOWN is off and MAXUP is high enough (200+) we have no explanation for this. - shmem tunings over 4-8M segment size has troubles bringing up OL engine, We are currently running 2M shmem limit pewr segment. Forced residency, same troubles. We suspect kernel problems rather than Ifx problems. - RFS and NFS function. BUT: = ls on a remote resource is fast, but = 'ls -l' generates a 15 second delay in responding on an ordinary sized directory. This occurs on RFS only, but occurs on both remote resources and the local loopback driver. If this was a mutlitsmallblock delay, we would expect to see high CPU times during access, on both sides throughout the wait. But what we are seeing is a dual pulse of 1-2 seconds twice during the 15 second delay with CPU and IO chane\\nels essentially idle. NUMSP, NUMTIM, NUMTRW are upped to 128/256/512 depedning on what solution e tried to fix. 4.0 3.1 system is vanilla tuning at this point. - TCP/IP flow rates are much more jagged now (pulsing or jerking in transmission during telnet/rlogin sessions). Long remsh's seem to hang more frequently. ulimit is way up -- basically infinite on all resources (save FDs, which are 64 soft/1024 hard). We have been running Informix OL 4.10 (UD2 engine) with various admixtures of the tools from 4.00 UE2 up to 4.10 UG2, SMP versions. KNown problems, please comment. The problems we have encountered appear to occur when multiple parallel access occur on external SCSI disk drives using hardpartitions, with more than one update transaction running and lock mode being page. Secondarily, with lock mode row and the same buffer being involved; all this is very preliminary. It appears that the lock manager goes into an infinite spin loop situation that eats CPU and finally releases (in about50% of the cases). Depending, sometimes the front end and sometimes the back end go CPU crazy; at times the bac end completes, the front end spins into CPU heaven, and transaction commits may or may not be done. OL back end sometimes thinks still in process, sometimes not. We have hada couple of abends leave shared memory around, preventing startup and auto recovery (hand removal works, but is a little nervy (;<>) with multiple OL instances up).. IN short, when the disks are going gangbusters, and heavy update procesing occurs, the OL instances are abending, showing inconsistent row and page IDs in the logs, and on occassion, our processors are panicing (yep, they speack Slavic, too!! (grin)). We thought we had a bad disk controller -- but no HW error symptomology. /etc/.osm and /dev/console clean. No PANICs on disk IO errors, etc. Power supplies, cables, etc, have been replaced. So has parts of cache memory, etc. on one system. Problems continue to occur. We have fallen back to 4.0 Ifx tools with 4.10 OL engine in the mean time. We also think it is possible that the kernel disk IO code is returning incorrect data (wrong data block) to application programs, though we have not been able to prove it. Any comments on this? Does anyone have experience out there with NCR SMP UNIX and 4.00/4.10, and what problems may be encountered in anon-uniform tool environment? Up to now, we have had good past history on Informix, using Star and Net with 4.0 on multiple systems without this type of trouble. Question areas: - KNown differences between SMP and nonSMP (single processor) UNIX and Informix loads? - WHat system tuneables may need adjustment? The loads are vanilla, with some resources hand increased. - Do multiple OL instances cause this type of symptom, as opposed to one per platform. We have had as many as four. We suspect SMP kernel code failure without corresponding kenrel msgs in IPC/locking/sm\\hmem/sem areas. - Is there a problem with the ISQL 4.00 front end to the OL 4.10 engine that causes = multiple parallel scripts to run away = a foreground screen onyl script to work, while a batch job from a pipe SQL source to run away? - Any other strange occurrences you folks might know about NCR SMP kernels? Our suspicion is that we are uncovering something in the kernel IPC facilities that (a) Informix is using correctly, (b) AT&T/NCR promis compatibility on, and (c) goes nuts on occasions. I am basing this tentative conclusion on 10 years of stability from Informix, and 15 years of experience with AT&T multiprocessor and IPC bugs. Most especially around AIO. I am pursuing these questions actively with NCR & Informix tech support, but also want to know of success/failure with these versions in the field as well. If you know any of the following, e-mail me and I will post a sumarized reply: - Later patch levels (or latest you know) for UNIX/ Ifx 4.10 series - Later bug fixes than printed patches for the areas of: RFS, NFS, TCP/IP, streams, kernel, IPC-1600, IPC/shm/sem, SMP, Ifx 4.10 ... Since there is no way to get a comprehensive list of known bugs, and the online services for automated lookup (from the NCR side) have little or no info online as yet, it is difficult to determine a patch exists, until you encounter a problem determine that there _is_ a bug or weirdness lose operational time on it call in, and get to the right person who knows of this describe the symptomology in the same terms and thus land on the fact that a bug fix exists and themn ovtain and install it. It would be nice to have them all online and downloadable. Informix folks are much beter with this, to my experience. Both sets of tech support people try hard, but without a comprehensive list, and auto notification of th