Informix Error -16

Operating-System Meaning

errno 16 is EBUSY — Device or resource busy. The object exists and is accessible, but something else holds it in a way that prevents the requested operation.

Unlike most of this range, EBUSY is rarely about the object being wrong. It is about something else already having it. The productive question is therefore never "what is wrong with this device?" but:

Who is holding it, and are they supposed to be?

That second half matters more than it sounds. On a database server the holder is frequently the engine itself, doing exactly what it should — in which case the fault is in whatever tried to take the device away, not in the thing holding it.

Typical sources: unmounting a filesystem that has open files, opening a device exclusively when something already has it, or removing or resizing storage that is in use.

What This Means in Informix

Nothing protects you from a shared chunk path

Two instances using the same chunk path is the worst thing that can happen to a set of dbspaces, and it is important to understand that this error is not what stands between you and it.

There are two quite different protections, and only one of them exists:

Scope	What happens
Within one instance	Reusing a path already in use is refused. The instance knows its own chunk table and will not add the same path twice.
Between instances	Nothing stops it. A second instance can be configured with the same path, will open it, and will write to it — breaking the first.

The protection inside an instance comes from the engine checking its own catalogue, not from the operating system holding the device exclusively. That check does not extend beyond the instance. Nothing in the operating system prevents two processes opening the same block device or cooked file read-write, and there is no registry of chunk paths spanning the instances on a host.

So the failure mode is not an error at the point it matters. It is two engines writing the same extents, both believing they own them — no message, no refusal, and nothing to stop it starting.

What you eventually see is this. Each chunk carries its own header pages, and the engine validates them. When one instance's writes have overwritten the other's chunk header, the next instance to check it finds the header bad and takes the chunk offline — and depending on which chunk it is, that can take the instance down with it.

The sequence:

both instances write the same extents   ← silent, no error
        ↓
a chunk header no longer validates
        ↓
the chunk is taken offline              ← the first thing anyone sees
        ↓
possibly the instance goes down with it

So the presenting symptom is not "unexplained corruption" — it is a chunk going offline for no apparent reason, or an instance stopping, at some arbitrary later moment. If that happens on a host running more than one instance, an overlapping chunk path belongs near the top of the list, and the check below answers it in two commands.

The damage done before that point is already done. The header check is what notices, not what protects.

This arises more easily than it should:

• An onconfig copied to build a second instance, with the chunk paths left unchanged
• A test instance cloned from production on the same host
• A device presented to two hosts in a cluster with no fencing
• A symlink in one instance resolving to the same device as a differently-named path in another

The check has to be proactive. There is no error to wait for, so the overlap scan below is not a diagnostic step — it is something to run when building a second instance on a host, after any clone, and after any storage change. It is in the Diagnostic Checks section because that is where the commands live, but its real place is a build checklist.

So what does produce a -16 against Informix?

The device-in-use cases, which are genuine: tape drives, filesystems the engine holds, storage layers claiming a disk, and objects still referenced when something tries to remove them. Those are real EBUSY conditions where the operating system does hold something exclusively.

A -16 is still worth treating carefully — identify the holder before forcing anything — but do not read it as evidence that the engine has protected you from a duplicate chunk path, because it has not.

Other contexts

• Unmounting a filesystem holding chunks, logs or backups while the engine is running
• Tape devices — a drive held by another backup process, a stale ontape or onbar session, or a storage-manager daemon
• onspaces -d dropping a chunk or dbspace that still has references
• Storage operations — resizing, removing or reconfiguring a LUN that the engine has open
• A device claimed by another layer — LVM, mdadm or multipath holding the disk, so a chunk path naming the underlying device cannot open it
• Restore operations where the target device is still held by the previous attempt

Common Causes

1. The engine itself holds the device, and something external is trying to take it.
2. A stale process from a previous run still has the device or tape open.
3. A filesystem is mounted on top of, or from, the device.
4. Another storage layer claims the disk — LVM, software RAID, multipath.
5. A tape drive in use by another backup or a storage-manager process.
6. An onspaces operation against an object still in use.

Diagnostic Checks

Find the holder first. Everything else follows from knowing who it is:

fuser -v /path/to/device
fuser -mv /path/to/mountpoint      # everything using that filesystem
lsof /path/to/device
lsof +D /path/to/mountpoint

Then establish whether the holder is an Informix process, and which instance:

ps -o pid,user,cmd -C oninit
ps -ef | grep -E 'oninit|ontape|onbar|bldutil' | grep -v grep
onstat -                            # this instance

Check for chunk-path overlap across every instance on the host. As set out above, no error will tell you about this — so this is a check you run on purpose, when building a second instance, after a clone, and after storage work. Run it now if two instances share a host, whatever brought you here.

The authoritative list comes from the engine, not from onconfig. Only the root chunk paths appear in the configuration file; every chunk added with onspaces is recorded in the instance's reserved pages and is invisible to a grep of onconfig. Scanning configuration files alone will miss almost all of the chunks in a mature instance.

For each instance, with its own environment set, take the list from onstat -d and resolve it:

onstat -d | awk '/^[0-9]/ {print $NF}' | while read -r p; do
    readlink -f "$p"
done | sort > "/tmp/chunks.$INFORMIXSERVER"

Repeat for every instance on the host, then compare across them:

sort /tmp/chunks.* | uniq -d        # any device claimed by more than one instance

Anything that prints is a device referenced by two instances, and needs resolving before either is trusted.

For an instance that is down — which is the usual situation when -16 has just stopped it starting — onstat -d is unavailable, so the root chunk from its configuration is the only path you can read directly:

grep -nE '^(ROOTPATH|ROOTOFFSET|MIRRORPATH)' /path/to/onconfig.<instance>

That is enough to catch the copied-configuration case, which is the common one, but it is not a complete overlap check. A full comparison needs each instance started in isolation, or the reserved pages read with oncheck -pr.

Also confirm the instances are genuinely distinct, since a shared SERVERNUM or shared-memory key is a related class of collision:

grep -nE 'SERVERNUM|DBSERVERNAME|SHMBASE' "$INFORMIXDIR"/etc/onconfig*
ipcs -m | head -20

Check whether another storage layer owns the disk:

lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE
ls -l /sys/block/sdX/holders/          # non-empty means something claims it
dmsetup deps 2>/dev/null
cat /proc/mdstat
pvs ; vgs ; lvs
multipath -ll

A device with entries under holders/ is claimed by device-mapper, RAID or multipath, and a direct path to the underlying disk will not open cleanly.

For a mount that will not release:

mount | grep /path/to/mountpoint
findmnt /path/to/mountpoint
fuser -mv /path/to/mountpoint
lsof +f -- /path/to/mountpoint

For tape:

fuser -v /dev/rmt0 /dev/st0 2>/dev/null
ps -ef | grep -Ei 'ontape|onbar|bar_|nsr|tsm|dsmc' | grep -v grep
mt -f /dev/rmt0 status

Solutions / Resolution

1. Identify the holder before doing anything else. fuser and lsof answer this in one command, and every correct remedy depends on the answer.
2. If the holder turns out to be another Informix instance, stop. Do not kill it, do not force the device open, and do not start the instance that reported the error. Establish which instance legitimately owns the chunk and correct the other configuration — and understand that you were lucky to get an error at all, because the operating system does not normally prevent this.
3. If the holder is this engine and something else wants the device, the external operation is the thing to reschedule — shut the instance down cleanly first if the storage work genuinely has to happen.
4. If it is a stale process, confirm it really is stale before terminating it. A backup that appears hung may be waiting on a tape rather than dead.
5. Never force an unmount (umount -f, umount -l) on a filesystem holding chunks or logs. A lazy unmount detaches the namespace entry while the engine continues writing to a filesystem nobody can see, which converts a clear error into silent damage.
6. If another storage layer claims the disk, point the chunk at the correct layer — the logical volume or multipath device, not the underlying disk.
7. For onspaces operations, find and close the references rather than forcing. A chunk that will not drop usually still holds data or is still open to a session.
8. Retest the specific operation, and re-run the overlap check afterwards if the cause was a duplicate path.

Examples

Two instances configured with the same chunk

This is the case to find before it produces anything, so the example is the proactive check rather than an error transcript.

With each instance's environment set in turn:

$ onstat -d | awk '/^[0-9]/ {print $NF}' | while read -r p; do readlink -f "$p"; done \
      | sort > "/tmp/chunks.$INFORMIXSERVER"

Then across the host:

$ sort /tmp/chunks.* | uniq -d
/dev/mapper/vg_ifx-lv_datadbs

$ grep DBSERVERNAME "$INFORMIXDIR"/etc/onconfig.prod "$INFORMIXDIR"/etc/onconfig.test
onconfig.prod:DBSERVERNAME ol_prod
onconfig.test:DBSERVERNAME ol_test

onconfig.test was copied from production and its chunk paths were never changed. Both instances have that device in their chunk list.

Nothing reports this at the moment it matters. If ol_test has been started at any point while ol_prod was running, both engines have been writing the same extents, and the dbspace should be treated as damaged regardless of what it looks like today — the chunk header check will notice at some later point of its own choosing, take the chunk offline, and possibly stop the instance. If ol_test has not yet been started, the configuration is one oninit away from that.

The fix is in onconfig.test, and it is urgent in a way the absence of an error message does not convey.

Run this comparison the other way round too. If a chunk has recently gone offline on a multi-instance host with no storage fault to explain it, this check is the one to run — it is quick, and it distinguishes a hardware problem from a configuration one before anyone starts replacing disks.

The engine holds a filesystem that someone wants to unmount

$ umount /informix/chunks
umount: /informix/chunks: target is busy.

$ fuser -mv /informix/chunks
                     USER        PID ACCESS COMMAND
/informix/chunks:    ifxprod    3312 F..e. oninit
                     ifxprod    3318 F..e. oninit

Working as designed. The remedy is to shut the instance down cleanly before the storage work, not to force the unmount — umount -l here would leave the engine writing into a detached filesystem.

The disk is claimed by LVM

$ grep -n datadbs "$INFORMIXDIR/etc/$ONCONFIG"
# chunk path /dev/sdc

$ ls -l /sys/block/sdc/holders/
lrwxrwxrwx 1 root root 0 Sep  9 12:02 dm-4 -> ../../../virtual/block/dm-4

$ lsblk /dev/sdc
NAME                     SIZE TYPE
sdc                        2T disk
└─vg_ifx-lv_datadbs        2T lvm

The chunk path names the raw disk, which LVM has claimed as a physical volume. The path should be the logical volume — /dev/mapper/vg_ifx-lv_datadbs — not the disk beneath it.

A tape drive still held

Archive failed: cannot open device /dev/rmt0
system error = 16

$ fuser -v /dev/rmt0
                     USER        PID ACCESS COMMAND
/dev/rmt0:           ifxprod   18442 F....  ontape
$ ps -o pid,etime,cmd -p 18442
   PID     ELAPSED CMD
 18442    19:42:11 ontape -s -L 0

A previous archive has been running for nearly twenty hours. It is holding the drive, and the new one cannot have it. Whether that process is stuck or legitimately slow is the next question — do not kill it without checking mt status and whether it is still making progress.

Platform Note

errno 16 is EBUSY on Linux, AIX, Solaris, HP-UX and the BSD-derived systems — stable, so the number is reliable. The official text's "Mount device busy" is the historic wording; current systems say "Device or resource busy", which better reflects that mounting is only one of the cases.

Finding the holder is platform-specific:

Task	Linux	Solaris	AIX
Who holds a file/device	fuser -v, lsof	fuser -u, pfiles	fuser -V, lsof
Who holds a filesystem	fuser -mv	fuser -cu	fuser -cx
Volume-layer holders	/sys/block/*/holders/, dmsetup deps	zpool status, metastat	lspv, lsvg -l
Open device check	lsof /dev/…	pfiles per pid	fuser -f

On AIX, a raw logical volume already open by one instance is reported this way and fuser -V on the device gives the holder directly. On Solaris with ZFS, a pool imported on another host is the analogous hazard to the duplicate-chunk case above, and zpool import without force will refuse for the same protective reason.

Related Errors / Related Topics

• -6 — No such device or address. The device node exists and nothing is behind it; -16 is the opposite situation, where something very much is.
• -15 — Block device required, the other error that indicates a chunk or tape path naming the wrong object.
• -13 — Permission denied, worth ruling out when nothing appears to hold the device.
• -5 — I/O error, where the device is held correctly and is failing.

A -16 against a chunk is worth treating as an availability and integrity question rather than an error to clear. The duplicate-path check above is cheap, and running it across every instance on a host after any clone, migration or configuration copy prevents the one failure mode here that cannot be undone.