Oninit
Talk to an expert →

Informix Error -28

-28 No space left on device.

An operating-system error code with the meaning shown was unexpectedly returned to the database server. Either a database table or an ASCII output file has probably filled the available disk space. Look for other operating-system error messages that might give more information.

Oninit® Troubleshooting Guidance

This is an operating-system error number, not an Informix diagnosis.

The same errno is returned by many different operations, so on its own it says what the operating system refused, not what Informix was trying to do. Informix will usually have reported a more specific error alongside it — in the message log, in the SQL or ISAM error pair, or in the accompanying assert failure — and that error normally defines the real cause far more precisely than the errno does. Find it before diagnosing from this number alone.

This matters less than it used to. Later versions of the engine trap many of these conditions and report them as specific Informix errors naming the operation, the object and the context, so a bare errno in this range is increasingly a sign of an older version, an unusual code path, or a failure early in startup before the better reporting is available. If you are seeing one on a current version, the more specific error is worth looking for even harder.

Operating-System Meaning

On Unix and Linux, errno 28 is ENOSPC — No space left on device. A write failed because the filesystem could not accommodate it.

"No space" has more than one cause, and df -h only reports the first of them:

Exhausted Detected by Typical presentation
Data blocks df -h The obvious case
Inodes df -i Plenty of free space, writes still fail
Space held by deleted-but-open files lsof +L1 df and du disagree with each other
Reserved blocks tune2fs -l df shows a few percent free, non-root writes fail
Per-file size limit ulimit -f, filesystem type One file stops growing, filesystem has room
Thin-pool backing store lvs, storage array Filesystem looks fine, the volume underneath is full

Checking only df -h, finding free space, and concluding the error is spurious is the most common wrong turn on this error.

What This Means in Informix

Distinguish this from Informix's own space errors immediately:

  • -28 is the operating system reporting a full filesystem.
  • -131 — "no free disk space" — is Informix reporting a full dbspace. There is space on the filesystem; the dbspace has no free pages.

They call for completely different remedies, and the wrong one wastes time. If you are in doubt, the message log will show which layer complained.

Typical origins of -28:

  • Cooked chunks on a filesystem that has filled, or a chunk being extended
  • online.log growth filling $INFORMIXDIR/tmp — often the first casualty, and the one that makes diagnosis harder because the engine can no longer log
  • Logical log backups to disk accumulating faster than they are cleared
  • ontape to a directory, or onbar staging, filling the target
  • Temporary dbspaces and sort space, especially large index builds or hash joins
  • DBSPACETEMP unset, so temporary tables land in the root dbspace or /tmp
  • dbexport, unload, external table writes producing very large output files
  • Core dumps from a previous failure consuming the filesystem
  • /tmp filling from SET EXPLAIN output, sqexplain files, or client tooling
Common Causes
  1. The filesystem is genuinely full — the straightforward case.
  2. Inodes exhausted — very common where logical log backups or sqexplain files accumulate as many small files.
  3. Deleted files still held open — a log or trace file removed with rm while the engine still has the descriptor open. The space is not returned until the process closes it or restarts, and du will no longer see the file while df still counts it.
  4. Reserved blocks — ext filesystems reserve 5% for root by default, so the installation owner hits ENOSPC while df reports free space.
  5. Log backup not running or failing, so logical logs are never freed.
  6. A runaway query consuming temp space — a large sort, hash join, or index build.
  7. Thin-provisioned storage exhausted at the array or LVM layer beneath a filesystem that believes it has room.
  8. Quota reached for the installation owner or its group.
  9. Filesystem maximum file size reached by a single large chunk or unload file.
Diagnostic Checks

Check both space and inodes, always:

df -h
df -i
df -h "$INFORMIXDIR" "$INFORMIXDIR/tmp"

Find the consumer:

du -xh --max-depth=1 /path/to/filesystem 2>/dev/null | sort -h | tail -20
find /path/to/filesystem -xdev -type f -size +500M -exec ls -lh {} \; 2>/dev/null

When df and du disagree, look for deleted-but-open files:

lsof +L1
lsof +L1 | grep -i informix

Check reserved blocks and quotas:

tune2fs -l /dev/<device> | grep -i 'reserved block'
quota -s -u ifxprod
repquota -a 2>/dev/null | head -20

Check the layer beneath the filesystem:

lsblk
lvs                       # look at Data% on thin pools
vgs
dmesg | tail -50

On the Informix side, establish whether this is a filesystem problem or a dbspace problem:

onstat -d                 # chunk free pages — for distinguishing -28 from -131
onstat -l                 # logical log status; are logs backed up and free?
onstat -g ses             # a session consuming temp space
onstat -m
grep -n DBSPACETEMP "$INFORMIXDIR/etc/$ONCONFIG"
Solutions / Resolution

Order matters here, because the instinctive first move — deleting files — can make things worse.

  1. Establish which filesystem is full, and whether it is blocks or inodes.
  2. Do not rm an active log or trace file. Deleting a file the engine holds open returns no space and removes your ability to inspect it. Truncate it instead:
    : > /path/to/large_open_file
    This releases the space immediately while the descriptor stays valid.
  3. Clear the genuinely safe candidates first — old logical log backups already archived elsewhere, previous ontape output, core dumps, old sqexplain files.
  4. If logical logs are the cause, back them up so they can be freed; that is the fix, not deleting them.
  5. If inodes are exhausted, find the directory holding enormous numbers of small files:
    find /path -xdev -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -rn | head
  6. If it is a dbspace rather than a filesystem, add a chunk — you are looking at -131, not -28.
  7. If temp space is the cause, set DBSPACETEMP to dedicated temporary dbspaces rather than letting temporary activity land in the root dbspace or /tmp.
  8. Only then consider extending the filesystem or adding storage.
  9. Confirm the engine is healthy afterwards — a filesystem that filled underneath a running server may have left an incomplete write, so check the message log rather than assuming recovery.
Examples
Free space, and writes still fail
$ df -h /informix
Filesystem      Size  Used Avail Use% Mounted on
/dev/mapper/vg-informix   200G  148G   42G  78% /informix

$ df -i /informix
Filesystem       Inodes   IUsed IFree IUse% Mounted on
/dev/mapper/vg-informix  13107200 13107200     0  100% /informix

42 GB free and not one inode available. In this case the cause was six years of logical log backups written as individual files into one directory.

df and du disagree
$ df -h /informix
/dev/mapper/vg-informix  100G   99G  0.5G  99% /informix

$ du -sh /informix
54G     /informix

$ lsof +L1 | grep informix
oninit  2841 informix  12u  REG  253,2  48318382080  0  /informix/tmp/online.log (deleted)

Someone removed online.log to reclaim space while the engine had it open. The 45 GB was never returned, and the engine has been writing to a file that no longer has a name. Truncating rather than deleting is the correct operation; recovering from this one requires the engine to reopen the file.

Reserved blocks
$ df -h /informix
/dev/sdb1  500G  475G   0G 100% /informix
$ tune2fs -l /dev/sdb1 | grep -i 'reserved block count'
Reserved block count:     6553600

25 GB is reserved for root. The installation owner sees ENOSPC while root can still write — which is why an administrator testing by hand as root may fail to reproduce the fault.

Full underneath the filesystem
$ df -h /informix
/dev/mapper/thin-informix  1.0T  610G  390G  62% /informix
$ lvs
  LV            VG    Attr       LSize  Pool      Data%
  informix      vgthin Vwi-aotz-- 1.00t thinpool  61.00
  thinpool      vgthin twi-aotz-- 800.00g          100.00
$ dmesg | tail -3
device-mapper: thin: 253:4: reached low water mark for data device: sending event.
device-mapper: thin: 253:4: no free data space

The filesystem believes it has 390 GB available; the thin pool beneath it has none. Nothing inside the filesystem will diagnose this, and deleting files may not help either, since thin pools do not reclaim until discard is issued.

Distinguishing -28 from -131
-- filesystem full (errno 28):
09:14:02  Assert Failed: Unable to extend chunk
09:14:02  system error = 28

-- dbspace full (Informix -131):
09:14:02  Space 'datadbs' — no free disk space for the operation

The first needs storage or cleanup; the second needs a chunk added to the dbspace. Reading the surrounding log lines rather than the headline error is what separates them.

Platform Note

errno 28 is ENOSPC on Linux, AIX, Solaris, HP-UX and BSD-derived systems — stable across platforms, so the number itself can be trusted here.

The diagnostic commands are not portable, and the ones above are Linux. Equivalents:

Task Linux Solaris AIX
Space df -h df -h df -g
Inodes df -i df -o i df -o i
Open deleted files lsof +L1 lsof +L1 lsof +L1
Volume layer lvs, lsblk zpool list lsvg, lslv

On ZFS in particular, df output is not a reliable guide — check zpool list and any reservations or quotas on the dataset.

Related Errors / Related Topics
  • -131 — No free disk space. The Informix-level counterpart, and the one people most often reach for by mistake. It is a dbspace problem, not a filesystem problem.
  • -13 — Permission denied. The other errno that routinely interrupts chunk and backup I/O.
  • -12 — Not enough core. Resource exhaustion of a different kind, worth checking if the filesystem turns out to be healthy.

A server that has hit -28 once will hit it again unless the growth is being watched. Filesystem and inode headroom on $INFORMIXDIR, the logical log backup target and any temporary dbspaces are the obvious candidates for monitoring alerts.