Oninit® Log Ripper — Output Options

The Ripper supports six output modes selected via the YAML target.mode key: informix, file, json, csv, kafka, and odbc. Each emits the same captured stream of INSERT / UPDATE / DELETE / TRUNCATE events; only the destination format differs.

A seventh option exists for in-process consumers: when the engine is linked as a library (see Embed Ripper), the host application can register a per-transaction callback via oni_config_output_embed(). Captured changes flow directly into application code with no file / socket / Kafka detour. This mode is not selectable from YAML — it requires a C-level callback registration.

Informix mode

target:
  mode: informix
  informix:
    database:   "target_db@target_server"   # cross-server form
    user:       "informix"                  # optional, OS auth otherwise
    password:   "secret"
    connection: "ripper_target"             # named connection (optional)

Direct ESQL/C execution against a target Informix database. Same server or cross-server — database@server just works as the connection target. Each captured statement runs immediately on the target; transactions on the source map 1:1 to transactions on the target, with BEGIN WORK / COMMIT WORK brackets so a crashed source transaction never half-commits on the target.

Errors on the target (duplicate key, constraint violation, lost connection) abort the active transaction and surface in the ripper log; the worker continues with the next captured transaction. Set debug_sql: true under logging: to see every statement before it runs.

File mode

target:
  mode: file
  file:
    directory: "/var/lib/oni_ripper/output"

One SQL file per committed transaction. File naming is collision-free across workers and same-second commits:

<YYYYMMDD>_<HHMMSS>_<microseconds>_w<worker>_tx_<txid>[_status].sql

Status suffix is empty for normal transactions. Incomplete ones get a suffix and a matching tagged log line:

Each file carries a metadata header (User / Started / Committed / Begin LSN / Commit LSN / Records), then the row changes wrapped in BEGIN WORK / COMMIT WORK with one -- LSN: ... comment line per statement so a downstream replay tool can resume mid-file:

-- Transaction 32 (worker 0)
-- Status:     complete
-- Worker:     0
-- User:       informix
-- Started:    2026-04-26 19:32:32
-- Committed:  2026-04-26 19:32:32
-- Begin LSN:  380002e018
-- Commit LSN: 380002e214
-- Records:    1

BEGIN WORK;

-- LSN: 380002e080
INSERT INTO t_customer (cust_id, fname, lname, email, phone, status)
       VALUES (236, 'Mixed_ondyomnpvzscj', 'Test_1', 'mixed_1@test',
               '555-0000', 1);

COMMIT WORK;

JSON mode

target:
  mode: json
  json:
    directory: "/var/lib/oni_ripper/output"

One RFC 8259 compliant JSON document per committed transaction. Naming follows the same pattern as file mode — the _NO_BEGIN / _NO_COMMIT / _FRAGMENT suffixes apply identically; only the extension changes (.json instead of .sql).

RFC 8259 compliance details that matter to operators consuming the stream:

Example (single-record transaction):

{
  "transaction": {
    "txid":      32,
    "worker_id": 0,
    "user":      "informix",
    "uid":       1001,
    "started":   "2026-04-26T19:32:32Z",
    "committed": "2026-04-26T19:32:32Z",
    "begin_lsn": "380002e018",
    "commit_lsn":"380002e214",
    "status":    "complete",
    "records":   [
      {
        "lsn":   "380002e080",
        "op":    "INSERT",
        "owner": "informix",
        "table": "t_customer",
        "sql":   "INSERT INTO t_customer (cust_id, fname, lname, email, phone, status) VALUES (236, 'Mixed_ondyomnpvzscj', 'Test_1', 'mixed_1@test', '555-0000', 1)"
      }
    ],
    "record_count": 1
  }
}

CSV mode

target:
  mode: csv
  csv:
    directory:      "/var/lib/oni_ripper/output"
    delimiter:      ","       # single char; default ","
    include_header: true      # column-name row at top of each file

One RFC 4180 compliant CSV file per committed transaction, long-form — every row carries the full transaction context so the file is self-describing without joins. Naming follows the same pattern as file / JSON modes (.csv extension; _NO_BEGIN / _NO_COMMIT / _FRAGMENT suffixes apply identically).

Encoding: bytes pass through verbatim. If the source DB locale is Latin-1 the file is Latin-1; if it's UTF-8 the file is UTF-8. CSV is byte-stream agnostic so the Ripper does not transcode — set the consumer's decoder to match the source DB locale (the JSON mode is the right pick when locale-independence is required).

Example (header + one record):

txid,worker_id,user,started,committed,begin_lsn,commit_lsn,lsn,op,owner,table,status,sql
32,0,informix,2026-04-26T19:32:32Z,2026-04-26T19:32:32Z,0x380002e018,0x380002e214,0x380002e080,INSERT,informix,t_customer,complete,"INSERT INTO t_customer (cust_id, fname, lname, email, phone, status) VALUES (236, 'Mixed_ondyomnpvzscj', 'Test_1', 'mixed_1@test', '555-0000', 1)"

Kafka mode

target:
  mode: kafka
  kafka:
    brokers: "kafka1:9092,kafka2:9092"
    topic:   "oni_cdc"
    acks:              "all"        # all (default) | 0 | 1 | -1
    compression:       "none"       # none | gzip | snappy | lz4 | zstd
    client_id:         "oni_ripper"
    flush_timeout_ms:  30000
    # security_protocol: "sasl_ssl"
    # sasl_mechanism:    "SCRAM-SHA-512"
    # sasl_username:     "ripper"
    # sasl_password:     "..."

One Kafka message per CDC record produced via librdkafka. The binary always carries the producer; the runtime config picks whether it's used. Build host needs librdkafka-devel / librdkafka-dev.

Reliability: producer runs with acks=all and enable.idempotence=true by default, so retries inside one producer session don't duplicate. After every transaction's records are produced the Ripper calls rd_kafka_flush() with flush_timeout_ms; only when the broker has ack'd everything does the worker's last_committed_lsn advance. If any message fails to deliver, the transaction handler returns an error and the LSN holds — the next run picks up from the same point.

TLS / SASL: set security_protocol to ssl / sasl_plaintext / sasl_ssl and supply sasl_mechanism + sasl_username + sasl_password for the SASL variants. SCRAM-SHA-256 / SCRAM-SHA-512 are the recommended mechanisms for production clusters; PLAIN is supported for testing.

PostgreSQL mode

target:
  mode: postgres
  sql_dialect: postgres
  postgres:
    host:     127.0.0.1
    port:     5432
    database: my_pg_db
    user:     replicator
    password: "secret"

Native PostgreSQL replay via libpq.so.5. The runtime dlopens the library so a binary built without the PG headers still ships and the missing-library failure surfaces only when this mode is actually selected. Per-statement SAVEPOINT in the output path lets one bad record (e.g. a captured embedded-NUL byte that PG text can't store) skip with a logged postgres: stmt failed (skipped via savepoint) line while the rest of the transaction commits. The connector also configures session encoding at startup so captured non-UTF-8 bytes (e.g. from a Latin-1 source DB) land cleanly without tripping the server's UTF-8 validation.

MySQL mode

target:
  mode: mysql
  sql_dialect: mysql
  mysql:
    host:     127.0.0.1
    port:     3306
    database: my_mysql_db
    user:     replicator
    password: "secret"

Native MySQL replay via libmysqlclient.so.21. The connector configures the session at startup so the application byte stream lands opaque in the column's declared charset (Latin-1, UTF-8, etc.) without the server's default validator rejecting captured high-bit bytes. Server-kind is checked at startup — a MySQL config pointed at a MariaDB server is refused with a CRITICAL log line so the operator catches the mismatch before any DML flows.

MariaDB mode

target:
  mode: mariadb
  sql_dialect: mariadb
  mariadb:
    host:     127.0.0.1
    port:     3307
    database: my_mariadb_db
    user:     replicator
    password: "secret"

Same protocol shape as MySQL, routed through libmariadb.so.3 with the matching server-kind check. The two YAML modes are kept distinct so the operator declares which server they're talking to; the connector refuses to run if the running server's mysql_get_server_info() string doesn't match the declared mode.

Db2 LUW mode

target:
  mode: db2
  sql_dialect: db2
  db2:
    host:     127.0.0.1
    port:     50000
    database: TESTDB
    user:     db2inst1
    password: "secret"

Native IBM Db2 LUW replay via the standalone CLI driver (libdb2.so from IBM's Data Server Driver Package). The runtime dlopens the library; if the driver isn't installed, the failure is reported with a one-line install hint. The connector configures the session codepage at init so Latin-1 captured bytes are translated to UTF-8 server-side on insert (parallel to the PostgreSQL connector's encoding step). The dialect rewriter handles Db2's narrow-form constraints automatically: numeric literals over 31 digits are wrapped in DECFLOAT('…') to bypass the parser's literal-precision cap, INTERVAL values fall back to a VARCHAR comment-tagged form (Db2's labeled-duration form is arithmetic-context only), and BLOB / CLOB hex bytes are wrapped in BLOB(X'…').

ODBC mode

target:
  mode: odbc
  odbc:
    dsn:      "my_postgres_dsn"   # name from /etc/odbc.ini
    user:     "dbuser"
    password: "dbpass"

Replay against any ODBC-connected database. Tested with PostgreSQL and MySQL. The binary always carries the ODBC dispatcher; the runtime needs the unixODBC library, the driver package for the target database, and a DSN entry in /etc/odbc.ini. See the Install & Build page for the per-distro package list and DSN setup. Verify with isql -v <dsn> <user> <pass> before pointing the Ripper at it.

If the operator picks target.mode: odbc on a host where the prerequisites are missing or the DSN is wrong, the ODBC connector reports the failure at startup and disables the target rather than running with a half-broken sink. Each diagnostic line carries the SQLSTATE plus a one-line fix hint:

SQL dialect translation is handled automatically when target.sql_dialect is set to a non-Informix value — the rewriter rewrites BOOLEAN literals, INTERVAL syntax, embedded NUL splices, MDY / TODAY built-ins, DATETIME format, and bare hex-literal forms per target. See the Schema Translation page for end-to-end examples and the SQL Mapping page for the per-dialect rewrite forms.