Oninit® Log Ripper — Schema Safety

The Ripper validates every CDC TABSCHEMA record against the live DESCRIBE snapshot it took at startup. The check is run in three layers, walked in order — the cheap structural test first, deeper checks only when the previous layer matches:

What happens on a mismatch

Schema archive

Set schema_archive_dir in the YAML to point at a directory of <db>_<tab>.sql files. On drift, the Ripper walks through the recovery steps in order:

At startup, when an archive is present the Ripper additionally walks it column-by-column against the live DESCRIBE as a third safety net. Pure rename or type swaps that preserve the (cols, var, fix_bytes) signature and the per-column NAMES are flagged as [ARCHIVE-CMP] divergences and escalated to [CRITICAL] when the snapshot signatures match exactly.

Live release / re-acquire

To ALTER a single table without stopping the ripper, drop sentinel files in control_dir and the per-worker poll picks them up:

# Pause capture on t_customer (ripper drops cdc_endcapture +
# cdc_set_fullrowlogging(0))
oni_logripper_control -d /var/lib/oni_ripper/control release t_customer

# Now ALTER TABLE t_customer ... succeeds because the table is released

# Resume capture under the new schema (re-DESCRIBE + cdc_startcapture)
oni_logripper_control -d /var/lib/oni_ripper/control acquire t_customer

Every other table in the capture set keeps capturing uninterrupted. The ripper itself never restarts.

ALTER TABLE in the captured stream

When the acquire path detects a column-level diff between the pre-release and post-acquire schemas, the ripper emits the corresponding ALTER TABLE as a synthetic single-record transaction (txid=-1) directly into the captured SQL stream before the next DML record under the new schema arrives. The replay target re-applies the same migration in order and remains schema-compatible with the source.

Two text sources, tried in order. (1) If the archive DDL file contains literal ALTER TABLE statements, they are extracted and emitted verbatim — operator-authored ALTERs preserve nuance the auto-differ cannot reconstruct (column rename, table-level constraint changes, index-only changes). (2) Otherwise the auto- differ compares the old and new cdc_column_t arrays and emits column-level ADD / DROP / MODIFY.

Example output for a real release/ALTER/acquire cycle that added location VARCHAR(60) to t_drift_b:

-- Transaction -1 (worker 0)
-- Status:     complete
-- Worker:     0
-- User:       (schema-migration) (uid=0)
-- Started:    (unknown)
-- Committed:  (unknown)
-- Begin LSN:  0
-- Commit LSN: 0
-- Records:    1

BEGIN WORK;

-- LSN: 0
ALTER TABLE t_drift_b ADD location VARCHAR(60);

COMMIT WORK;

Filename convention <ts>_w<worker>_tx_-1.sql makes synthetic schema-migration files easy to grep for separately from real captured transactions. Direct-DB targets (informix / odbc / postgres / mysql / mariadb / db2) execute the ALTER bare — no BEGIN/COMMIT framing — because DDL auto-commits in most engines and rejects framing in some.

Auto-differ limitations: column rename is undetectable (looks like DROP+ADD); table-level constraints (PK/FK/CHECK) are not in cdc_column_t and so are not reproduced; non-default DATETIME / INTERVAL qualifiers fall through to YEAR TO SECOND / DAY TO SECOND. For any of these, place an explicit ALTER TABLE statement in the archive file and the verbatim path runs.

Skip-on-unreconcilable-drift

If neither the live DESCRIBE nor the archive can parse the incoming stream, the ripper logs [CRITICAL], sets tab->skip_capture=1, and drops further row events for that table only. The other tables keep capturing. The ripper exits only when every monitored table is in skip_capture.

Encrypted dbspaces and the raw-log fallback

The Ripper's main CDC capture path is unaffected by Informix's Encryption At Rest (EAR) feature — the engine decrypts pages transparently at the buffer-pool layer before CDC ever sees them. EAR-encrypted dbspaces (V14 EAR; bit 0x10000000 on sysmaster:sysdbspaces.flags) capture exactly like unencrypted dbspaces for every CDC-supported column type.

The exception is the raw-log fallback path used to recover column types that cdc_startcapture rejects (TEXT / BYTE / BLOB / CLOB / SET / MULTISET / LIST / ROW / SQLUDTVAR) by reading the logical-log pages directly via pread(O_RDONLY) against the chunk file. This path bypasses the engine's buffer pool and therefore its transparent decrypt step; on an encrypted chunk pread returns ciphertext that fails every downstream parse. The Ripper supports encrypted log chunks on this path by decrypting them page-by-page itself, gated on the operator configuring the Informix keystore.

Operator opt-in

Point the Ripper at the Informix keystore base path:

security:
  decrypt_keystore: /home/informix/<server>/etc/db_keystore

The path is the base — no .p12 or .sth suffix; the underlying library appends both. Per-dbspace decrypt contexts open lazily on first encrypted-page access, so the keystore-unwrap cost is paid only when an encrypted chunk is actually read.

Runtime requirements

• gsk8capicmd_64 on $PATH. The Ripper invokes it to unwrap the master key from the .p12 file using the passphrase in the matching .sth. On standard Informix hosts this ships in /usr/bin/gsk8capicmd_64 via the GSKit RPM.
• The Ripper process must be able to read both <keystore_base>.p12 and <keystore_base>.sth. The standard Informix install ships them mode 0600 owned by user informix, so the Ripper must run as user informix. Running as any other user surfaces a CRITICAL on the first encrypted-page access. Relaxing the keystore file permissions is NOT a recommended workaround — the master key file is exactly the kind of artefact that should stay 0600.

Startup behaviour

At startup the log-fallback pre-flight scans every log-bearing chunk's dbspace for the EAR bit. Three outcomes:

The main CDC path is untouched by all of this. Even when the fallback refuses to start, an operator who removes recover_unsupported_via_log: true from the affected tables can still capture every CDC-supported column type from the same encrypted dbspaces without configuring a keystore at all.

LSN checkpoint

On graceful shutdown the ripper writes min(last_committed_lsn) across all workers to the configured state_file. The next startup reads it and resumes via cdc_activatesess at that position — at-least-once semantics around the boundary, never data loss. -R on the CLI forces a clean restart for one run while still rewriting the file at exit.