Oninit® Log Ripper — Configuration Reference

Every YAML key the Ripper accepts, grouped by section. Defaults match what config.yml.example ships. Anything not listed in this table is silently ignored.

Precedence

For settings that exist in more than one place, the Ripper resolves them in this order (highest wins):

Source	Notes
CLI flag	-l, -w, -i, -R, the -v / -T / -S / -C / -D / -X debug flags. Each overrides the matching YAML key for the run.
YAML file (-c)	The configuration file is required. Anything set there beats the built-in defaults.
Environment	INFORMIXSERVER is the only env var the Ripper reads directly — used as a fallback for source.server when the YAML omits it. ESQL/C also consults INFORMIXDIR, INFORMIXSQLHOSTS, DBPATH, DB_LOCALE, CLIENT_LOCALE via the CSDK.
Built-in default	The values shown in the Default column of each table below.

source

Key	Type	Default	Effect
database	string	(required)	Source Informix database to capture from.
server	string	(required)	Source INFORMIXSERVER name. Can also be set via the env var.
user	string	(empty)	If set, used as CONNECT TO … USER … USING …. If empty, falls back to OS auth.
password	string	(empty)	Paired with user.
start_lsn	integer	0	0 = start at the current end of the logical log. A non-zero value calls cdc_activatesess at that position. The CLI -l <lsn> overrides.
schema_archive_dir	path	(empty)	Directory holding archived <db>_<tab>.sql DDL files used for drift recovery and the startup [ARCHIVE-CMP] safety net. Time-versioned variants are recognized via the suffix form <db>_<tab>.<YYYYMMDDTHHMMSSZ>.sql (UTC, ISO 8601); the un-suffixed file means "current / effective at +∞". The Ripper picks the file with the largest timestamp ≤ the in-flight transaction's BEGINTX time at acquire-time.
control_dir	path	(empty)	Directory polled by workers for release_<tab> / acquire_<tab> sentinel files. Empty disables live release.
tables	list	all user tables	Glob patterns or extended map entries (see Per-table extended entry below). !pattern excludes.

Per-table extended entry

Key	Type	Default	Effect
name	string	(required)	Table name or glob.
columns	list	all columns	Per-column include / exclude patterns. An all-exclude list (["!body"]) defaults to "include everything else"; any include entry switches to "include only listed".
skip_unsupported_columns	bool	false	When the column DESCRIBE finds a type CDC rejects (TEXT / BYTE / BLOB / CLOB / SET / MULTISET / LIST / ROW / UDT), false aborts the whole table; true silently included = 0's those columns and captures the rest.

target

Key	Type	Default	Effect
mode	string	file	One of file, json, csv, kafka, informix, odbc.
file.directory	path	"."	Where transaction SQL files are written.
json.directory	path	"."	Where transaction JSON files are written. RFC 8259 compliant: UTF-8 only, every byte ≥ 0x80 escaped as \u00XX so the document parses regardless of source DB locale; LSNs emitted as hex strings (not numbers) to dodge IEEE 754 precision loss above 2⁵³.
csv.directory	path	"."	Where transaction CSV files are written. RFC 4180 compliant; long-form (one row per CDC record, every row carries the full transaction context: txid, worker_id, user, started, committed, begin_lsn, commit_lsn, lsn, op, owner, table, status, sql). Bytes pass through verbatim — set the consumer's decoder to match the source DB locale.
csv.delimiter	char	","	Single-character field delimiter. Anything containing the delimiter, double quote, CR, or LF is wrapped in double quotes per RFC 4180; internal double quotes are doubled.
csv.include_header	bool	true	Write the column-name row as the first line of every per-transaction file.
kafka.brokers	string	(required)	Comma-separated list of host:port bootstrap servers for the producer.
kafka.topic	string	(required)	Single destination topic. Per-record messages are keyed by "<owner>.<table>" so all changes for one table go to the same partition (preserves per-table ordering).
kafka.acks	string	"all"	Producer ack mode. all = wait for all in-sync replicas (safest, default); 1 = leader only; 0 = fire-and-forget.
kafka.compression	string	"none"	One of none, gzip, snappy, lz4, zstd.
kafka.client_id	string	"oni_ripper"	Producer client.id; identifies the Ripper in broker logs and metrics.
kafka.flush_timeout_ms	integer	30000	Per-transaction rd_kafka_flush() wait. The worker's last_committed_lsn only advances after the broker acks all of the transaction's messages.
kafka.security_protocol	string	"plaintext"	One of plaintext, ssl, sasl_plaintext, sasl_ssl.
kafka.sasl_mechanism	string	(empty)	PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512. Required when security_protocol is sasl_*.
kafka.sasl_username, kafka.sasl_password	string	(empty)	SASL credentials.
informix.database	string	—	Target Informix DB. Cross-server allowed: db@server.
informix.server	string	(empty)	If set, alternative to the db@server form above.
informix.connection	string	targetconn	Named ESQL/C connection identifier.
informix.user, informix.password	string	(empty)	OS auth if both empty.
odbc.dsn	string	—	DSN name from /etc/odbc.ini.
odbc.user, odbc.password	string	—	Passed to SQLConnect.

threading

Key	Type	Default	Effect
sessions_per_thread	integer	10	Tables per worker thread. Total workers needed = ceil(ntables / sessions_per_thread), capped at max_threads.
max_threads	integer	4	Hard cap on worker threads. Tables that don't fit would otherwise be silently dropped — raise this if a startup log line warns about table coverage.
cdc_timeout	integer (seconds)	5	cdc_opensess timeout argument. 0 means block-forever; finite values let the worker poll the running flag so SIGTERM is honored.
max_recs_per_read	integer	1	Upper bound on records the server packs into a single ifx_lo_read response. Higher amortizes per-read overhead; lower delivers CDC_REC_TIMEOUT sooner.
cleanup_orphans	bool	true	If set, the main thread sweeps syscdcsess at startup and closes any leftover sessions before workers spin up.

logging

Key	Type	Default	Effect
verbose	bool	false	High-level info messages. Same as the -v CLI flag. Reloadable on SIGHUP.
debug_trace	bool	false	[TRACE] function-entry lines. -T. Reloadable on SIGHUP.
debug_sql	bool	false	[SQL] generated statements. -S. Reloadable.
debug_cdc	bool	false	[CDC] API trace. -C. Reloadable.
debug_detail	bool	false	[DET] column-level dump. -D. Reloadable.
debug_hex	bool	false	[HEX] raw byte dump. -X. Reloadable.
status_interval	integer (seconds)	0	[STATUS] + [LSN] cadence. 0 = off. Reloadable on SIGHUP.
logfile	path	(empty)	Daemon-mode stdout/stderr destination. Empty → /dev/null.
pidfile	path	/tmp/oni_ripper.pid	PID file written in daemon mode.
log_maxsize	integer (bytes)	0	Rotate / truncate trigger. 0 = unlimited. Reloadable on SIGHUP.
log_keep	integer	5	Rotated copies retained. 0 = legacy in-place truncate. Reloadable on SIGHUP.
state_file	path	(empty)	LSN checkpoint file. Empty disables persistence.
clean_restart	bool	false	Ignore the saved state_file on read. The CLI -R overrides this for one run.

alerting

Key	Type	Default	Effect
discard_command	shell command	(empty)	Run via /bin/sh -c on each CDC_REC_DISCARD event. Env vars passed: RIPPER_EVENT, RIPPER_WORKER_ID, RIPPER_DISCARDS, RIPPER_LSN, RIPPER_TIMESTAMP.
discard_alert_min_sec	integer (seconds)	60	Minimum gap between alert fires per worker, so a discard storm doesn't fork-bomb the alerter.

buffering

Target-offline buffering. When the configured target returns a connection-class failure, transactions are appended to an in-memory ring; when memory fills, the oldest batch spills to a memory-mapped file under disk_overflow_path. A probe thread retries the target and the buffer drains chronologically when it’s back.

Key	Type	Default	Effect
memory_max_bytes	integer (bytes)	0	Soft cap on the in-memory ring. 0 disables the subsystem entirely — output failures stay fatal. 268435456 (256 MB) is a reasonable starting point for one-target deployments.
disk_overflow_path	directory path	(empty)	Where spill files (<unix_ts>_<seq>.buf, mmap'd on drain) are written. The ripper creates the directory at startup if it doesn’t exist. Empty = memory-only (no disk overflow).
disk_max_bytes	integer (bytes)	0	Cap on total spill bytes on disk. 0 = memory-only. Once both memory_max_bytes and disk_max_bytes are exhausted, the buffer transitions to SUSPENDED: a [CRITICAL] line fires and capture pauses until the operator clears space or the target comes back.
target_check_seconds	integer (seconds)	30	Probe cadence. The probe thread runs a zero-record transaction through the live backend; success transitions the state machine to DRAINING and the backlog replays.
on_shutdown	enum	flush	SIGTERM behavior while memory holds records. flush spills the in-memory tail to disk_overflow_path so the next startup picks it up; drain blocks until the target accepts (falls through to flush if it doesn’t); abandon drops the in-memory tail with a [CRITICAL] audit line.

Connection-class triggers per backend (anything else stays a data-class failure and the legacy ROLLBACK + log path):

Mode	Offline trigger
informix	SQLCODE -908 / -931 / -25555 / -956 / -25588
odbc	SQLSTATE class 08* (ODBC spec "Connection exception")
postgres	PQstatus(conn) == CONNECTION_BAD after BEGIN / exec / COMMIT
mysql / mariadb	mysql_errno 2002 / 2003 / 2006 / 2012 / 2013
db2	SQLSTATE class 08* from SQLGetDiagRec
kafka	librdkafka __TRANSPORT / __ALL_BROKERS_DOWN / __TIMED_OUT / __RESOLVE / __MSG_TIMED_OUT / BROKER_NOT_AVAILABLE

monitoring

Optional embedded HTTP server exposing Prometheus metrics + liveness probe. Default-disabled; the ripper opens no listening sockets unless this section is present and port is non-zero.

Key	Type	Default	Effect
prometheus.port	integer (TCP port)	0	0 disables the endpoint. Any non-zero value starts a single-threaded daemon HTTP server that serves /metrics + /healthz. Bind failures log a WARN line and continue capture — metrics are non-critical.
prometheus.bind	IPv4 address	127.0.0.1	Address the embedded server binds. Loopback by default so a fresh enable carries no external attack surface. Set 0.0.0.0 to allow a Prometheus server on a different host to scrape; wrap with a reverse-proxy if the link crosses an untrusted network — the embedded server is plain HTTP, no TLS, no authentication.

Metric families exposed (v1):

Metric	Type	Labels	Description
oni_logripper_build_info	gauge	version	Constant 1, advertises ripper version.
oni_logripper_records_total	counter	worker, op	Records emitted per worker per op ∈ insert/update/delete/truncate/discard.
oni_logripper_lag_seconds	gauge	worker	Source wall-clock minus last seen transaction timestamp; 0 if no tx has been processed yet.
oni_logripper_recovery_count	counter	worker	Worker session recovery attempts since process start.
oni_logripper_worker_running	gauge	worker	1 if the worker thread is actively running, 0 if stopped.

/healthz returns HTTP 200 when every worker reports error == 0, otherwise HTTP 503. Curated Grafana provisioning files + dashboard JSON ship under share/grafana/; see grafana.html for the drop-in install steps, panel breakdown, and operator playbook.

CLI flags

Flag	Effect
-c <file>	Configuration file (required).
-l <lsn>	Override starting LSN.
-w <num>	Cap worker threads (overrides threading.max_threads).
-i <secs>	Status report interval (overrides logging.status_interval).
-d	Daemonize.
-v / -T / -S / -C / -D / -X	Verbose / Trace / SQL / CDC / Detail / heX dump. Each enables the corresponding logging.debug_* flag for the run.
-R	Clean restart — ignore the saved state file on read. The state file is still rewritten on shutdown.
-t <mode>	Connectivity test. Runs every startup phase except worker spawn (source connect, table resolution, LSN monitor, log-fallback pre-flight, output init) then exits cleanly. <mode> = auto uses target.mode from the YAML; any other mode name overrides for the test. Implies -v. Skips orphan cleanup so no server-side state changes.
-V	Print version and exit.
-h	Show help.

Signals

Signal	Effect
SIGTERM, SIGINT	Graceful shutdown. Drain in-flight transactions to _NO_COMMIT files, persist the LSN checkpoint, close CDC sessions.
SIGHUP	Reload the YAML config and apply the runtime-tunable subset (verbose, debug_, status_interval, log_maxsize, log_keep, cdc_timeout*). Anything else captured at startup is logged as not-reloaded.
SIGCHLD	Ignored (SIG_IGN) so the kernel auto-reaps discard-alert subprocesses.