DrWatson: From ifxcollect to Findings
By Oninit · Published 2 November 2026
When an Informix® instance misbehaves, the evidence is already on disk: the onconfig file, the message log, dozens of onstat views and, after a crash, an assertion-failure file. What is usually missing is the time, and the experience, to read all of it against what is known to go wrong.
DrWatson is a rules-based diagnostic platform that does that reading. It takes what you collected, applies a library of expert-written rules, and returns prioritised findings with the evidence behind each one.
What it reads
- ifxcollect collections. The collection tool runs in profiles, and each profile gathers a different mix of artefacts: a general snapshot, performance and CPU samples, connection failures and hangs, backup and restore failures, replication state, and assertion-failure context.
- Assertion-failure (AF) files. The detail Informix writes when the engine fails an internal check.
- onstat -a output. A view of engine internals: memory, locks, sessions and I/O.
Nothing connects to your database. DrWatson does not run on it, does not connect to it remotely, and does not collect anything that is not in the files you submit. There is no agent to install and no schema change anywhere in your estate: you upload the artefacts.
From text to facts
Each artefact has its own parser. A parser turns onstat or onconfig text into ground facts, small precise statements such as the instance's version, a buffer pool's size, or a dbspace's free pages. This is the shape, with illustrative values:
instance('prod1', '14.10.FC10W1', enterprise, 'dbhost', linux, 64).
bufferpool(2048, 50000, 8, 50, 60).
dbspace_usage(3, 410000, 90000, 82).
Three habits keep the facts trustworthy. Parsers are written against real captures, not against a format someone assumed. Every collection gets its own fact module, so two collections never mix. A field that is missing becomes the atom unknown, never a silent zero.
Rules
The rules are written in Prolog and read the facts. They come in five families: configuration audit, health, performance, capacity, and assertion failure. Each rule has a stable ID, so a finding means the same thing from one report to the next, and a severity that reflects operational risk, not certainty. A few examples:
| Rule | What it finds | Severity |
|---|---|---|
| CFG-005 | TAPEDEV is /dev/null with no ON-Bar configured | critical |
| PERF-009 | Buffer-pool starvation (ovbuff above zero) | high |
| PERF-010 | LOCKS exhausted (ovlock above zero) | high |
| CFG-007 | BUFFERPOOL undersized for the workload | medium |
The important design rule is that a rule can fire only when the data it needs is present. A CPU sample contains no backup data, and a naive engine would report "no backup problems" for it. DrWatson guards each rule on the kind of collection or on the specific facts it reads, so absent data produces no finding and is never mistaken for a clean bill of health.
Collections taken more than once give the rules rates instead of single-shot ratios. Repeating the collection and zeroing the counters between runs makes each snapshot a delta, so the rules see reads per second and checkpoints per second, not lifetime totals.
Assertion failures
An AF report is about one incident. If a directory holds more than one AF file, DrWatson asks which to use instead of merging them, because unrelated failures often come from different instances. The report contains:
- a crash fingerprint: server, host, version, source file and line, signal, session and thread;
- known-bug findings, placed above the stack so the diagnosis arrives before the frames;
- the crashing thread's stack, with the suspected fault frame marked;
- an explanation of any operating-system frames in the stack (
semtimedop,futex,polland the like), so you know what the thread was doing in the kernel; - the offending SQL when it is present, and the tail of
online.logfrom just before the failure; - the sections inside the AF (dbspaces, logs, profile counters, transactions) read by the same parsers, kept under their own names so they cannot be confused with facts from a different instance.
What you get back
The result is a PDF report for system administrators, architects or management, with the findings, prioritised recommendations and a business summary. The same engine also runs from the command line, and as a tool surface for an LLM front end, described below.
Traceability is the reason for the design. Every finding traces back to facts, and every fact traces back to a line in the files you supplied, so a finding is something you can check, not something you have to take on trust.
Asking questions in plain English
A report answers the questions DrWatson thought to ask. The next question is usually yours: why did that rule fire, what is the worst thing here, is this the same problem as that crash. You can put those questions in plain English, in two ways.
- In the portal. Open a delivered report and ask a question about it. The assistant is limited to that one report. It works from the report's findings, not from the raw facts, because a busy system produces far more facts than any model can read, and the diagnosis is already distilled into the findings. Some questions never reach a model at all: a request for a raw onstat section, the onconfig, an LRU review, or what a rule means is answered directly from the data, at no model cost. Free-form answers are optional and have to be enabled. The assistant is not shown the underlying facts, other customers' data or earlier reports, and it has no tool for fetching anything beyond your question and the report.
- From your own tools. The analyser also runs as an MCP server, so an LLM front end can use it as a set of tools: load a collection, list findings (by severity if you like), explain a finding, list the rules in a category, ask for tuning recommendations, and query the facts directly. Here the model is a front end to the engine. The engine does the diagnosis, and the model finds and explains. Because explaining a finding returns the rule and the facts behind it, any statement the model makes can be checked.
In both cases the rules decide what is wrong. The model does not diagnose: it finds, explains and summarises what the rules found. Treat an answer as a way into the evidence, and check the finding and its facts before you act on it.
Practical notes
- Pick the profile for the question. A rule can only use what the collection contains, and the report says which sections were and were not analysed.
- Collect more than once for performance questions. Rates need two or more iterations.
- Obfuscate before upload if you need to. Hostnames, table and database names, SQL literals and OS user IDs can be scrubbed first.
- Submissions are confidential. They are kept in storage scoped to your own customer area, for the period agreed in your engagement.
Summary
DrWatson turns the files you already have into checked findings: parsers make facts, rules read the facts, and each result shows the rule and the evidence behind it. You can ask questions of the findings in plain English, and check every answer against that evidence. To see what it covers, visit the DrWatson page.