Oninit® Log Ripper — Output Options

The Ripper supports eleven output modes selected via the YAML target.mode key: informix, file, json, csv, kafka, postgres, mysql, mariadb, db2, oracle, and odbc — with Microsoft SQL Server reached through odbc + FreeTDS. Each emits the same captured stream of INSERT / UPDATE / DELETE / TRUNCATE events; only the destination format differs.

A seventh option exists for in-process consumers: when the engine is linked as a library (see Embed Ripper), the host application can register a per-transaction callback via oni_config_output_embed(). Captured changes flow directly into application code with no file / socket / Kafka detour. This mode is not selectable from YAML — it requires a C-level callback registration.

Informix mode

target:
  mode: informix
  informix:
    database:   "target_db@target_server"   # cross-server form
    user:       "informix"                  # optional, OS auth otherwise
    password:   "secret"
    connection: "ripper_target"             # named connection (optional)

Direct ESQL/C execution against a target Informix database. Same server or cross-server — database@server just works as the connection target. Each captured statement runs immediately on the target; transactions on the source map 1:1 to transactions on the target, with BEGIN WORK / COMMIT WORK brackets so a crashed source transaction never half-commits on the target.

Errors on the target (duplicate key, constraint violation, lost connection) abort the active transaction and surface in the ripper log; the worker continues with the next captured transaction. Set debug_sql: true under logging: to see every statement before it runs.

File mode

target:
  mode: file
  file:
    directory: "/var/lib/oni_ripper/output"

One SQL file per committed transaction. File naming is collision-free across workers and same-second commits:

<YYYYMMDD>_<HHMMSS>_<microseconds>_w<worker>_tx_<txid>[_status].sql

Status suffix is empty for normal transactions. Incomplete ones get a suffix and a matching tagged log line:

Each file carries a metadata header (User / Started / Committed / Begin LSN / Commit LSN / Records), then the row changes wrapped in BEGIN WORK / COMMIT WORK with one -- LSN: ... comment line per statement so a downstream replay tool can resume mid-file:

-- Transaction 32 (worker 0)
-- Status:     complete
-- Worker:     0
-- User:       informix
-- Started:    2026-04-26 19:32:32
-- Committed:  2026-04-26 19:32:32
-- Begin LSN:  380002e018
-- Commit LSN: 380002e214
-- Records:    1

BEGIN WORK;

-- LSN: 380002e080
INSERT INTO t_customer (cust_id, fname, lname, email, phone, status)
       VALUES (236, 'Mixed_ondyomnpvzscj', 'Test_1', 'mixed_1@test',
               '555-0000', 1);

COMMIT WORK;

JSON mode

target:
  mode: json
  json:
    directory: "/var/lib/oni_ripper/output"

One RFC 8259 compliant JSON document per committed transaction. Naming follows the same pattern as file mode — the _NO_BEGIN / _NO_COMMIT / _FRAGMENT suffixes apply identically; only the extension changes (.json instead of .sql).

RFC 8259 compliance details that matter to operators consuming the stream:

Example (single-record transaction):

{
  "transaction": {
    "txid":      32,
    "worker_id": 0,
    "user":      "informix",
    "uid":       1001,
    "started":   "2026-04-26T19:32:32Z",
    "committed": "2026-04-26T19:32:32Z",
    "begin_lsn": "380002e018",
    "commit_lsn":"380002e214",
    "status":    "complete",
    "records":   [
      {
        "lsn":   "380002e080",
        "op":    "INSERT",
        "owner": "informix",
        "table": "t_customer",
        "sql":   "INSERT INTO t_customer (cust_id, fname, lname, email, phone, status) VALUES (236, 'Mixed_ondyomnpvzscj', 'Test_1', 'mixed_1@test', '555-0000', 1)"
      }
    ],
    "record_count": 1
  }
}

CSV mode

target:
  mode: csv
  csv:
    directory:      "/var/lib/oni_ripper/output"
    delimiter:      ","       # single char; default ","
    include_header: true      # column-name row at top of each file

One RFC 4180 compliant CSV file per committed transaction, long-form — every row carries the full transaction context so the file is self-describing without joins. Naming follows the same pattern as file / JSON modes (.csv extension; _NO_BEGIN / _NO_COMMIT / _FRAGMENT suffixes apply identically).

Encoding: bytes pass through verbatim. If the source DB locale is Latin-1 the file is Latin-1; if it's UTF-8 the file is UTF-8. CSV is byte-stream agnostic so the Ripper does not transcode — set the consumer's decoder to match the source DB locale (the JSON mode is the right pick when locale-independence is required).

Example (header + one record):

txid,worker_id,user,started,committed,begin_lsn,commit_lsn,lsn,op,owner,table,status,sql
32,0,informix,2026-04-26T19:32:32Z,2026-04-26T19:32:32Z,0x380002e018,0x380002e214,0x380002e080,INSERT,informix,t_customer,complete,"INSERT INTO t_customer (cust_id, fname, lname, email, phone, status) VALUES (236, 'Mixed_ondyomnpvzscj', 'Test_1', 'mixed_1@test', '555-0000', 1)"

Kafka mode

target:
  mode: kafka
  kafka:
    brokers: "kafka1:9092,kafka2:9092"
    topic:   "oni_cdc"
    acks:              "all"        # all (default) | 0 | 1 | -1
    compression:       "none"       # none | gzip | snappy | lz4 | zstd
    client_id:         "oni_ripper"
    flush_timeout_ms:  30000
    # security_protocol: "sasl_ssl"
    # sasl_mechanism:    "SCRAM-SHA-512"
    # sasl_username:     "ripper"
    # sasl_password:     "..."

One Kafka message per CDC record produced via librdkafka. The binary always carries the producer; the runtime config picks whether it's used. Build host needs librdkafka-devel / librdkafka-dev.

Reliability: producer runs with acks=all and enable.idempotence=true by default, so retries inside one producer session don't duplicate. After every transaction's records are produced the Ripper calls rd_kafka_flush() with flush_timeout_ms; only when the broker has ack'd everything does the worker's last_committed_lsn advance. If any message fails to deliver, the transaction handler returns an error and the LSN holds — the next run picks up from the same point.

TLS / SASL: set security_protocol to ssl / sasl_plaintext / sasl_ssl and supply sasl_mechanism + sasl_username + sasl_password for the SASL variants. SCRAM-SHA-256 / SCRAM-SHA-512 are the recommended mechanisms for production clusters; PLAIN is supported for testing.

PostgreSQL mode

target:
  mode: postgres
  sql_dialect: postgres
  postgres:
    host:     127.0.0.1
    port:     5432
    database: my_pg_db
    user:     replicator
    password: "secret"

Native PostgreSQL replay via libpq.so.5. The runtime dlopens the library so a binary built without the PG headers still ships and the missing-library failure surfaces only when this mode is actually selected. Per-statement SAVEPOINT in the output path lets one bad record (e.g. a captured embedded-NUL byte that PG text can't store) skip with a logged postgres: stmt failed (skipped via savepoint) line while the rest of the transaction commits. The connector also configures session encoding at startup so captured non-UTF-8 bytes (e.g. from a Latin-1 source DB) land cleanly without tripping the server's UTF-8 validation.

MySQL mode

target:
  mode: mysql
  sql_dialect: mysql
  mysql:
    host:     127.0.0.1
    port:     3306
    database: my_mysql_db
    user:     replicator
    password: "secret"

Native MySQL replay via libmysqlclient.so.21. The connector configures the session at startup so the application byte stream lands opaque in the column's declared charset (Latin-1, UTF-8, etc.) without the server's default validator rejecting captured high-bit bytes. Server-kind is checked at startup — a MySQL config pointed at a MariaDB server is refused with a CRITICAL log line so the operator catches the mismatch before any DML flows.

MariaDB mode

target:
  mode: mariadb
  sql_dialect: mariadb
  mariadb:
    host:     127.0.0.1
    port:     3307
    database: my_mariadb_db
    user:     replicator
    password: "secret"

Same protocol shape as MySQL, routed through libmariadb.so.3 with the matching server-kind check. The two YAML modes are kept distinct so the operator declares which server they're talking to; the connector refuses to run if the running server's mysql_get_server_info() string doesn't match the declared mode.

Db2 LUW mode

target:
  mode: db2
  sql_dialect: db2
  db2:
    host:     127.0.0.1
    port:     50000
    database: TESTDB
    user:     db2inst1
    password: "secret"

Native IBM Db2 LUW replay via the standalone CLI driver (libdb2.so from IBM's Data Server Driver Package). The runtime dlopens the library; if the driver isn't installed, the failure is reported with a one-line install hint. The connector configures the session codepage at init so Latin-1 captured bytes are translated to UTF-8 server-side on insert (parallel to the PostgreSQL connector's encoding step). The dialect rewriter handles Db2's narrow-form constraints automatically: numeric literals over 31 digits are wrapped in DECFLOAT('…') to bypass the parser's literal-precision cap, INTERVAL values fall back to a VARCHAR comment-tagged form (Db2's labeled-duration form is arithmetic-context only), and BLOB / CLOB hex bytes are wrapped in BLOB(X'…').

Oracle mode

target:
  mode: oracle
  sql_dialect: oracle
  oracle:
    host:     127.0.0.1
    port:     1521
    database: ORCLPDB1            # service name (or SID)
    user:     replicator
    password: "secret"

Native Oracle replay via the Oracle Call Interface (libclntsh.so, from the Instant Client or a full client install). The runtime dlopens the library along its SONAME chain, so a binary built without the Oracle headers still ships and the missing-library failure surfaces only when this mode is selected. The dialect rewriter maps captured Informix SQL to Oracle forms automatically: DECIMAL to NUMBER(p,s), DATETIME to TIMESTAMP(N), BOOLEAN literals to 1 / 0, INTERVAL to Oracle's native interval syntax, MDY() to TO_DATE, TODAY to CURRENT_DATE, and BLOB / CLOB hex bytes to HEXTORAW('…'). Set NLS_LANG (e.g. AMERICAN_AMERICA.WE8ISO8859P1) in the ripper's environment so captured Latin-1 bytes land in the target's NCHAR / NVARCHAR2 columns without transcoding surprises. Oracle rejects an explicit value for an identity column, so the connector reconciles captured inserts into identity tables automatically (parallel to the SQL Server case below).

ODBC mode

target:
  mode: odbc
  odbc:
    dsn:      "my_postgres_dsn"   # name from /etc/odbc.ini
    user:     "dbuser"
    password: "dbpass"

Replay against any ODBC-connected database. Tested with PostgreSQL and MySQL. The binary always carries the ODBC dispatcher; the runtime needs the unixODBC library, the driver package for the target database, and a DSN entry in /etc/odbc.ini. See the Install & Build page for the per-distro package list and DSN setup. Verify with isql -v <dsn> <user> <pass> before pointing the Ripper at it.

If the operator picks target.mode: odbc on a host where the prerequisites are missing or the DSN is wrong, the ODBC connector reports the failure at startup and disables the target rather than running with a half-broken sink. Each diagnostic line carries the SQLSTATE plus a one-line fix hint:

SQL dialect translation is handled automatically when target.sql_dialect is set to a non-Informix value — the rewriter rewrites BOOLEAN literals, INTERVAL syntax, embedded NUL splices, MDY / TODAY built-ins, DATETIME format, and bare hex-literal forms per target. See the Schema Translation page for end-to-end examples and the SQL Mapping page for the per-dialect rewrite forms.

Microsoft SQL Server (via ODBC / FreeTDS)

SQL Server is reached through mode: odbc using the FreeTDS driver — there is no separate mssql mode. Register a FreeTDS DSN in /etc/odbc.ini, set target.sql_dialect: mssql, and point the ODBC mode at the DSN:

target:
  mode: odbc
  sql_dialect: mssql
  odbc:
    dsn:      "my_mssql_dsn"      # FreeTDS DSN in /etc/odbc.ini
    user:     "sa"
    password: "secret"

The dialect rewriter handles SQL Server's forms automatically: DATETIME to DATETIME2(N), BOOLEAN literals to 1 / 0, INTERVAL to a VARCHAR fallback, MDY() to DATEFROMPARTS, and TODAY to CAST(GETDATE() AS DATE) (SQL Server has no CURRENT_DATE); BLOB / CLOB hex bytes go to 0xNN… against VARBINARY(MAX). Set ClientCharset=ISO-8859-1 in the DSN so captured Latin-1 bytes round-trip through the server's NCHAR / NVARCHAR columns. As with Oracle, explicit inserts into an identity column are reconciled by the connector. The FreeTDS prerequisites and DSN setup are the same as the Install & Build page's ODBC section; verify with isql -v <dsn> <user> <pass> before pointing the Ripper at it.