Inside mttrace: Tracing IBM Informix's Thread Scheduler

What it is

IBM® Informix® Dynamic Server has always kept its most useful diagnostics behind an unlisted door. mttrace is one of them — a support-grade tracer for the engine's MT (MultiThreaded) subsystem, the virtual-processor and thread-scheduler layer that everything else in the server runs on top of. It isn't documented in the public manuals, but once you know its shape, it's a surprisingly direct window into what the engine is actually doing, instruction by instruction, inside a live server.

mttrace runs server-side, as the informix user, attaching directly to the running engine's shared-memory segment — the same mechanism onstat uses. It needs the usual environment in place (INFORMIXDIR, INFORMIXSERVER, sqlhosts, onconfig); without it, you get permission errors, “server not found,” or a shared-memory attach failure, which is itself a decent signal that you're pointed at the wrong instance.

The basic workflow: turn on tracing for one or more internal components at a chosen verbosity level, let the engine run, then view the accumulated trace buffer — live, tailing, or from a dumped shared-memory snapshot.

mttrace on|off|empty|view|session|info|add|reset
        -c <comp> <level>
        -f
        -h
        -i <shfile>
        -n
        -s <sid>
        -t

Components and levels

There are 121 traceable components, each a named subsystem — MT_TRC_BFR_MGR (buffer manager), MT_TRC_BTREE, MT_TRC_CHECKPOINT, MT_TRC_LOCKS, MT_TRC_SQL, and so on down to niche internals like MT_TRC_CDRIF_NIF (Enterprise Replication's network interface). Each has four verbosity levels — off, lite, med, heavy — set independently:

mttrace on -c MT_TRC_BFR_MGR med

You can combine components, and separately scope tracing to up to four specific session IDs with session/-s.

What a trace record actually looks like

This is where it gets interesting for anyone who's ever wanted to understand Informix internals rather than just use them. Every trace event is a fixed 40-byte header followed by a packed, TLV-style argument area:

typedef struct mt_trc_rec {
    uint8_t   narg;       /* number of packed arguments that follow */
    uint8_t   vp;         /* virtual-processor id */
    uint16_t  msgid;      /* index into a 602-entry message catalog */
    uint16_t  resv04;
    uint16_t  reclen;     /* total record length; 0 marks end-of-buffer */
    uint16_t  line;       /* source line number */
    uint16_t  fileid;     /* -> source file name */
    uint16_t  sesid;      /* session id */
    uint16_t  resv0e;
    uint64_t  rstcb;      /* running-session TCB pointer */
    uint64_t  transp;     /* transaction pointer */
    uint64_t  timestamp;
    /* narg packed argument entries follow */
} mt_trc_rec_t;

Each msgid indexes a 602-entry catalog of printf-style format templates, pulled straight from the engine's own internal logging calls — things like buffget(%d %d:%d %x %p %x) or rscommit_complete(current_lsn %d:%x txn %d:%x). The arguments that follow are tagged by type (raw, string, integer, double, two different time encodings, counted string, or a hex-dumped byte buffer) and packed as fixed-width or length-prefixed values. The formatter walks the tag stream and expands it against the matching template.

The effect is that a trace record is self-identifying: because the message template embeds the emitting function's own name, you can often tell exactly which internal routine fired — bfunlink(), physio(), dbcreate() — without even needing the FileName/Line columns that view also prints alongside the timestamp.

Why this matters for forensic work

The mttrace utility provides access to internal diagnostic events recorded by the Informix engine. These records contain information about engine components, internal operations, message identifiers, arguments, and execution context. The diagnostic framework encompasses 121 identified components, supported by defined record structures, argument formats, and a comprehensive message catalog.

By presenting these records in a structured, readable format, mttrace allows engineers to examine recorded activity, establish event sequences, and correlate operations across internal engine subsystems.

For offline diagnostics and recovery, this information is particularly valuable when investigating page corruption, stalled checkpoints, lock conflicts, and other complex engine issues.

It is important to distinguish between recorded events and their underlying causes. A trace may show the operations immediately preceding a failure, the internal state reported at the time, and the engine's subsequent response. However, these observations do not necessarily identify the original cause of the problem.

The value of mttrace lies in providing direct evidence of recorded engine behavior. When combined with other diagnostic information, this evidence can help reconstruct the circumstances surrounding a failure, evaluate possible explanations, and narrow the investigation toward a defensible root-cause determination.

The full reference — all 121 components and the complete 602-entry message catalog — lives at Forensic Tools: mttrace.