Oninit logwalker / Online-log analysis
Talk to an expert →

Oninit® Logwalker — Troubleshooting

Logwalker is intentionally chatty about why a walk didn't start. The pre-flight gates surface the three deployment-shaped failure modes in plain English; this page is the lookup table for what each diagnostic means.

"[CRITICAL] sysmaster connect failed"

The ESQL/C CONNECT TO sysmaster didn't succeed. Common causes, in rough order of likelihood:

  • INFORMIXSERVER not exported, or exported to the wrong DBSERVERNAME. onstat - as the same OS user, with the same env, is the fastest cross-check.
  • INFORMIXSQLHOSTS not exported or pointing at a sqlhosts file that doesn't list your DBSERVERNAME. The binary uses whichever sqlhosts your environment hands the CSDK.
  • Engine is down or in offline / quiescent mode. shared memory not initialized in the engine's online.log is the clearest signal; just bring the engine back up.
  • Running OS user has no CONNECT privilege on sysmaster. Either add the user to the informix group (the standard local form) or pass -u/-p on the command line with credentials that do have CONNECT.

"[CRITICAL] failed to resolve uniqid <N>"

The requested uniqid isn't in the engine's current syslogs view. The engine has either:

  • Recycled past it — the cyclic LOGFILES ring has wrapped and overwritten the uniqid's chunk pages. Logwalker can only walk uniqids the engine still considers resident. Confirm with onstat -l — the table shows the live window of uniqids.
  • Never created it — the uniqid is beyond the current end of the log. Typo in -n is the usual cause.
  • Archived it to a TAPEDEV the engine considers final — logwalker doesn't read archive tapes. For a true archived log on a real backup tape device, use onlog -t against the tape.

"[CRITICAL] chunk path /… not readable"

One of the chunk files in the resolved extent map failed access(R_OK) from the OS user running logwalker. Raw chunk files almost always live with 0660 informix:informix permissions, so the operator must be one of:

  • User informix (the engine's owner).
  • Any user in group informix. Confirm with id.
  • Root — works but heavily discouraged for a diagnostic tool.

There's no partial mode: every chunk in the extent map must pass R_OK or the walk doesn't start. This is intentional — silently skipping a chunk would mean silently skipping records, and the whole point of the tool is to give you a faithful view of what's in the log.

"libifasf.so: cannot open shared object file"

The dynamic linker couldn't find one of the Informix CSDK runtime libraries. The Makefile rpaths $INFORMIXDIR/lib/esql — but on some CSDK install layouts libifasf.so lives one level up in $INFORMIXDIR/lib. The standard fix is to extend LD_LIBRARY_PATH to cover both directories: 

export LD_LIBRARY_PATH=$INFORMIXDIR/lib:$INFORMIXDIR/lib/esql

This also covers the related libthsql.so, libthos.so, libthgen.so, libthxa.so "cannot open" failures from the same family.

"oni_log_cursor_read: short read at offset …"

The raw-log reader couldn't get a full record's worth of bytes from the chunk. Usually this means the extent map's n_pages for the uniqid extends past the file size on disk — in practice, the engine was actively truncating or reorganising the log while logwalker was reading it. Wait for the engine to settle (the next onstat -l shows steady-state uniqids) and rerun.

"catalog: 0 named partitions loaded"

Phase-1 catalog returned no rows from systabnames. The engine accepted the sysmaster CONNECT but the SELECT returned nothing — usually a privilege gap. Sysmaster's pseudo-tables are visible to informix and accounts with explicit RESOURCE; an unprivileged account with bare CONNECT may land in this state. Add the running user to the informix group, or supply -u/-p as informix.

Record count differs from onlog -n

This is the documented residual-record caveat. Logwalker walks raw chunk bytes sequentially; onlog filters by the per-page logserial stamp. For a partially-filled current uniqid, logwalker surfaces extra records left over from prior generations that the engine hadn't yet overwritten. Within any single transaction's BEGIN → COMMIT chain the two tools agree record-for-record; the divergence is at the inter-transaction layer only. See the Output Schema page for the long-form note.

"--decode-rows is empty / hex only"

--decode-rows needs CONNECT privilege on the source database to pull syscolumns. Databases the running user can't reach log SQLCODE -387 / -329 / -23197 warnings at startup and their tables' rows render as hex instead of (col=val, …). The table NAMES still resolve (phase 1 only reads sysmaster), so HUMAN mode is still readable; only the per-row column decode is gated by CONNECT.

"--decrypt output is still ciphertext / [CRITICAL] no valid records"

Almost always the wrong --dbsnum: you passed the chunk number where logwalker wants the dbspace number. The per-dbspace working key is derived from the dbspace number, so a wrong value derives the wrong key — the decrypt succeeds mechanically but yields garbage, and the walk finds no known record types. logwalker flags this with a [CRITICAL] no valid records notice naming the chunk-vs-dbspace trap, rather than failing loudly.

The two numbers are easy to confuse because both come from onstat:

  • chunk number — the chk column of onstat -d (and what onstat -l reports per log). This is not what --dbsnum wants.
  • dbspace number — the number column of onstat -d (the dbsnum in sysmaster:sysdbspaces). This is the --dbsnum value.

Best fix: omit --dbsnum entirely — logwalker auto-detects the dbspace from the log's chunk during the resolve, so the override is rarely needed. Pass it only when the auto-detect is wrong, and take the value from the number column of onstat -d, never the chk column. See Operations for the --decrypt / --keystore / --dbsnum reference.