Informix Error -23

-23 File table overflow.

An operating-system error code with the meaning shown was unexpectedly returned to the database server. Look for other operating-system error messages that might give more information. If the error recurs, note all circumstances and contact IBM Informix Technical Support.

Oninit® Troubleshooting Guidance

This is an operating-system error number, not an Informix diagnosis.

The same errno is returned by many different operations, so on its own it says what the operating system refused, not what Informix was trying to do. Informix will usually have reported a more specific error alongside it — in the message log, in the SQL or ISAM error pair, or in the accompanying assert failure — and that error normally defines the real cause far more precisely than the errno does. Find it before diagnosing from this number alone.

This matters less than it used to. Later versions of the engine trap many of these conditions and report them as specific Informix errors naming the operation, the object and the context, so a bare errno in this range is increasingly a sign of an older version, an unusual code path, or a failure early in startup before the better reporting is available. If you are seeing one on a current version, the more specific error is worth looking for even harder.

Operating-System Meaning

errno 23 is ENFILE. The official "File table overflow" is the historic wording; current systems say "Too many open files in system", which is clearer about what actually ran out.

The distinction from -24 decides everything that follows, and getting it wrong sends the remedy to the wrong place:

	-23 `ENFILE`	-24 `EMFILE`
Scope	The whole host	One process
Meaning	The system-wide file table is full	This process hit its own descriptor limit
Controlled by	`fs.file-max`, `fs.nr_open`	`ulimit -n` / `RLIMIT_NOFILE`
Raising `ulimit -n` helps?	No	Yes
Who is at fault	Possibly something else entirely	Almost certainly this process

The practical consequence is the one worth internalising:

On -23, Informix may well be the victim rather than the cause. The host ran out of descriptors; whichever process asked next got the error. That can easily be the database while the consumption is somewhere else entirely.

So the investigation is host-wide from the start. Ranking every process by descriptor count is the first move, not a later one.

On modern Linux fs.file-max defaults high enough that -23 has become uncommon, which makes it worth suspecting a leak or an unusual limit rather than assuming normal growth. In containers and on older systems it remains entirely ordinary.

What This Means in Informix

Informix is a substantial descriptor consumer, so it is frequently the largest process on the host even when it is not the cause:

Every chunk the engine has open
Every client connection, and the network structures behind it
Virtual processors and their own open files
The message log, backup devices and temporary files
Multiple instances on one host, each with the full set

Being the biggest consumer is not the same as being the leak. A host that has run comfortably for years and suddenly reports -23 has usually had something added — a new application, a monitoring agent, a backup client, a second instance — rather than having had Informix grow.

Worth checking alongside:

A container's limits, which can be far below the host's and are easily overlooked
Another application leaking descriptors on the same host
A recent instance added to a host already near its ceiling

Common Causes

A descriptor leak in some process on the host — not necessarily Informix.
fs.file-max set low, often a carried-over tuning from an older system.
A container or cgroup limit well below the host's.
More instances or applications added to a host already near its ceiling.
A connection storm consuming descriptors faster than they are released.
Genuine growth in sessions, chunks or instances over time.

Diagnostic Checks

Start with the system totals. /proc/sys/fs/file-nr gives allocated, free and maximum in one line:

cat /proc/sys/fs/file-nr
#  allocated   free   max
#  1048320        0   1048576      <- effectively full

sysctl fs.file-max fs.nr_open

If allocated is close to max, the host is out and the next question is who is holding them.

Rank every process by descriptor count. This is the most useful command here, and frequently names something nobody suspected:

for p in /proc/[0-9]*; do
    n=$(ls "$p/fd" 2>/dev/null | wc -l)
    [ "$n" -gt 0 ] && printf '%7d  %-20s %s\n' "$n" "$(cat "$p/comm" 2>/dev/null)" "${p##*/}"
done | sort -rn | head -20

# aggregate by program name
lsof -n 2>/dev/null | awk '{print $1}' | sort | uniq -c | sort -rn | head -15

Establish Informix's share, so you know whether the database is the cause or a bystander:

ps -o pid= -C oninit | while read -r p; do ls /proc/"$p"/fd 2>/dev/null | wc -l; done \
  | awk '{s+=$1} END {print "oninit descriptors:", s}'
onstat -g ses | wc -l
onstat -d | grep -c '^[0-9]'        # chunks

Compare that against the host total. A database holding a few thousand descriptors on a host with a million-entry table is not the problem.

Look for a leak — a process whose count grows steadily rather than varying with load:

watch -n 30 'cat /proc/sys/fs/file-nr'
for i in 1 2 3; do
    ls /proc/<pid>/fd | wc -l
    sleep 60
done
ls -l /proc/<pid>/fd | awk '{print $NF}' | sort | uniq -c | sort -rn | head

That last line groups a process's descriptors by target — a long list of the same file, socket or pipe is a leak signature.

In a container, the limits that matter are not the host's:

cat /proc/sys/fs/file-max
cat /proc/sys/fs/nr_open
cat /sys/fs/cgroup/pids.max 2>/dev/null
ulimit -n

Solutions / Resolution

Confirm the scope before acting. If file-nr shows the host near its maximum, this is -23 and ulimit -n is irrelevant. If the host has ample headroom and one process is at its own limit, you are looking at -24 instead.
Find the largest consumers with the ranking loop, and establish whether the profile is reasonable. This decides whether the fix is a limit or a leak.
If something is leaking, fix the leak. Raising fs.file-max against a leak buys hours, not a solution, and the next failure will be at a worse moment.
If the limit is genuinely too low, raise it — and persist it:
```
sysctl -w fs.file-max=2097152
echo 'fs.file-max = 2097152' > /etc/sysctl.d/90-files.conf
```
fs.nr_open caps what a single process may be granted and may also need raising if per-process limits are to go higher.
For containers, set the limit in the pod or container specification. Changing it inside a running container does not persist and may not be permitted at all.
Reduce Informix's own footprint only if it is genuinely disproportionate — session counts, the number of chunks, and how many instances share the host are the levers, and all are architectural rather than quick fixes.
Add monitoring on file-nr. This error gives plenty of warning if anyone is watching, and none at all if not.

Examples

The database is the victim

$ cat /proc/sys/fs/file-nr
1048512    0   1048576

$ for p in /proc/[0-9]*; do
>     n=$(ls "$p/fd" 2>/dev/null | wc -l)
>     [ "$n" -gt 0 ] && printf '%7d  %-20s\n' "$n" "$(cat "$p/comm" 2>/dev/null)"
> done | sort -rn | head -5
 982114  java
   4218  oninit
   1902  oninit
    644  filebeat
    118  sshd

A Java application holds 982,000 descriptors against a host limit of 1,048,576. Informix holds around 6,000 across its processes and is entirely reasonable — it simply happened to be the next process to ask. Every minute spent tuning the database here is wasted.

A limit carried from an older system

$ cat /proc/sys/fs/file-nr
65504    0   65536
$ sysctl fs.file-max
fs.file-max = 65536
$ free -g | head -2
               total        used        free
Mem:             256         180          62

64K descriptors on a 256 GB host — a value that made sense on much smaller hardware and was carried forward through rebuilds. Nothing is leaking; the ceiling is simply from another era.

A leak signature

$ ls -l /proc/8842/fd | awk '{print $NF}' | sort | uniq -c | sort -rn | head -3
   4471 /informix/tmp/scratch.tmp
      3 /dev/null
      1 socket:[184422]

One file open 4,471 times by a single process. That is not load — it is a code path that opens and never closes. Raising any limit here only changes when it fails.

Platform Note

errno 23 is ENFILE on Linux, AIX, Solaris, HP-UX and the BSD-derived systems — stable, so the number is reliable. The historic wording "File table overflow" reflects the original fixed-size kernel file table; modern kernels allocate dynamically up to a tunable ceiling, which is why the current wording is "Too many open files in system".

Where that ceiling lives differs:

Task	Linux	Solaris	AIX
System-wide limit	`fs.file-max`	`rlim_fd_max`, project limits	largely dynamic
Per-process ceiling	`fs.nr_open`	`process.max-file-descriptor`	`ulimit -n`, `/etc/security/limits`
Current usage	`/proc/sys/fs/file-nr`	`kstat`, `pfiles`	`lsof`, `procfiles`
Per-process descriptors	`ls /proc/<pid>/fd`	`pfiles <pid>`	`procfiles <pid>`

On AIX the system-wide file table is largely dynamic and -23 is correspondingly rarer; the per-process limit is usually what bites, so check -24 first on that platform. On Solaris the project framework governs these, and a process in an unexpected project may be subject to limits nobody intended.

Related Errors / Related Topics

-24 — Too many open files, for one process. Read the table above before deciding which you have; the remedies do not overlap and applying the wrong one changes nothing.
-9 — Bad file descriptor. Not a resource problem at all, but it appears in the same investigations because both concern descriptors.
-11 — Resource temporarily unavailable, and -12 — cannot allocate memory. A host short of descriptors is often short of other things too, and the three commonly arrive together.

Descriptor consumption is one of the easiest things on a database host to watch and one of the least often watched. /proc/sys/fs/file-nr costs nothing to sample and turns this error from an outage into a ticket raised days earlier.