File Descriptor Limits in Linux: ulimit, Systemd, and EMFILE Fixes

When a Linux process exhausts its allocated file descriptors, system calls like open(), socket(), pipe(), and accept() fail immediately with the EMFILE error code ("Too many open files"). Unlike physical memory exhaustion, which causes memory swapping or invokes the kernel Out-of-Memory (OOM) killer, file descriptor exhaustion leaves the offending process running in a degraded state. The process remains alive in the process table, but it cannot establish incoming network sockets, read configuration files, or write debug logs to disk.

A file descriptor limit is an operating system restriction that caps the maximum number of open files, network sockets, and pipes a single process or the entire system can maintain simultaneously. In Unix-like operating systems, "everything is a file." When a process opens a regular file on an ext4 or XFS filesystem, connects to a PostgreSQL database over a TCP socket, creates a POSIX inter-process pipe, or initializes an epoll event listener, the Linux kernel assigns an unsigned integer known as a file descriptor to track that input and output stream.

The File Descriptor Architecture

Understanding how Linux tracks file descriptors clarifies why limits exist and where bottlenecks emerge. Inside the kernel, file descriptor resolution spans three distinct layers:

1. Per-Process File Descriptor Table: Each running process possesses its own private file descriptor table within its task_struct. The table maps small integer values (0 for standard input, 1 for standard output, 2 for standard error, and ascending integers for newly opened resources) to pointers in the system-wide open file table.
2. System-Wide Open File Table: The kernel maintains a global list of open file objects represented by struct file instances. This table tracks the access mode (read, write, append), current file offset, and reference counts across all running processes.
3. VFS Inode Table: The file object points to an inode representing the physical file on disk or the underlying network socket buffer in memory.

Because file descriptors consume unswappable kernel memory, Linux enforces limits at both the process level and the kernel level to prevent rogue scripts, runaway connection pools, or fork bombs from starving the operating system.

The Three-Tier Limit Hierarchy

Linux implements file descriptor governance across three hierarchical tiers:

• Per-Process Soft Limit (RLIMIT_NOFILE): The current operational threshold enforced by the kernel for an individual process. Most Linux distributions, including Ubuntu, Debian, CentOS, and Red Hat Enterprise Linux, default to a conservative soft limit of 1,024 file descriptors per process. An unprivileged process can dynamically raise its soft limit up to its designated hard limit using the setrlimit() system call.
• Per-Process Hard Limit: The absolute ceiling for the soft limit. Once established, an unprivileged user or process can only lower its hard limit. Raising the hard limit requires root privileges or explicit capability grants (CAP_SYS_RESOURCE).
• System-Wide Kernel Ceiling (fs.file-max): A global limit configured in /proc/sys/fs/file-max that bounds the total number of open file allocations across all running processes on the host. Modern Linux distributions set fs.file-max dynamically during boot based on available physical memory, often establishing limits between 1,048,576 and 65,536,000 handles on modern servers.

EMFILE vs ENFILE Error Codes

When an application fails to allocate a file descriptor, the returned error code reveals which tier blocked the operation:

• EMFILE (Error Maximum Files per process, numeric value 24): Indicates that the calling process reached its own per-process limit (RLIMIT_NOFILE). Adjacent processes running under other system users continue operating normally.
• ENFILE (Error Network or System File table overflow, numeric value 23): Indicates that the host operating system exhausted the system-wide limit set in /proc/sys/fs/file-max. When an ENFILE error occurs, every process across the entire server fails when attempting to open files or accept network handshakes.

Application Ceilings and the AI Ingestion Dilemma

File limits do not exist solely at the operating system level; they interact directly with application boundaries and AI agent workflows. Autonomous agents and Retrieval-Augmented Generation (RAG) indexers attempt to batch thousands of documents or extract code trees across local processes. In AI agent workflows, similar ceilings appear at the application boundary: as documented by the Anthropic Help Center, Claude chat sessions accept up to 20 files at up to 500 MB per file, while Claude Projects accept files up to 30 MB each with no fixed file-count cap, bounded only by the model's context window.

Reaching context ceilings during ingestion forces developers to look for alternative paths, while opening thousands of source files locally trips Linux process file descriptor boundaries before data ever reaches an API.

• CSV File Size Limits: AI Assistant Upload Caps, Parsing Ceilings, and FixesCSV file size limits vary from strict row boundaries in spreadsheets to memory and token constraints in AI chat...
• Google Gemini PDF Limits: File Sizes, Page Caps, and Parsing FixesThe Gemini PDF limit enforces distinct document ceilings across environments: 100MB and 10 files per prompt in web...
• GitHub File Size Limits Explained for Teams and CopilotGitHub hard-blocks any single file larger than 100 MiB in normal Git history, warns above 50 MiB, and caps browser...
• Airtable Attachment Limits: File Sizes, Base Storage, and WorkaroundsThe Airtable attachment limit framework caps individual file uploads at 5 GB across all subscription tiers, while...
• AWS Bedrock File Size Limits: Document Constraints in Knowledge Bases and Converse APIAWS Bedrock Knowledge Bases enforces a 50 MB per-document limit for file ingestion, while the Bedrock Converse API caps...
• AWS Lambda File Size Limits: Payload, Package, and Ephemeral Storage CapsAn AWS Lambda file size limit encompasses the execution constraints of AWS Lambda functions, specifically the 50MB...

More on this subject: Agent File and Document Workflows (325 guides)

Diagnosing file descriptor exhaustion requires inspecting both active limits and real-time consumption across running processes. Because different shells, daemons, and system users run under isolated constraints, querying limits from the correct context is essential.

Checking Process Limits with ulimit

The standard utility for inspecting and modifying resource limits in an interactive shell session is ulimit. By default, ulimit -n inspects the soft file descriptor limit for the current shell:

ulimit -n

To explicitly inspect the soft limit, pass the -Sn flag:

ulimit -Sn

To inspect the hard ceiling for the current user, pass the -Hn flag:

ulimit -Hn

When the command outputs 1024, the current shell session is restricted to that default allocation until you raise the ceiling.

Inspecting Running Processes via /proc

When troubleshooting background daemons or multi-threaded applications, checking ulimit in your personal SSH session can be misleading. System services frequently run with completely different resource boundaries.

To inspect the exact limits assigned to a live process by process ID (PID), query its /proc entry:

cat /proc/<PID>/limits | grep "Max open files"

The output displays three columns: the soft limit, the hard limit, and the unit of measurement:

Limit                     Soft Limit           Hard Limit           Units
Max open files            1024                 4096                 files

To determine how many file descriptors a running process currently holds open, list the entries in its /proc/<PID>/fd virtual directory:

ls -1 /proc/<PID>/fd | wc -l

Comparing the count of entries in /proc/<PID>/fd against the soft limit in /proc/<PID>/limits reveals whether a process is approaching its exhaustion ceiling.

Inspecting System-Wide File Allocations

To evaluate overall host capacity, query the virtual files provided by the kernel under /proc/sys/fs/:

cat /proc/sys/fs/file-max
cat /proc/sys/fs/file-nr

The file-nr virtual file returns three space-delimited integers:

1. Allocated File Handles: The number of file handles the Linux kernel has currently allocated.
2. Free Allocated Handles: The number of allocated file handles that have been released and are ready for reuse (in modern Linux 2.6 and newer kernels, this figure is typically 0 because the kernel frees unused structures immediately).
3. Maximum File Handles: The maximum number of file handles the kernel will permit system-wide (identical to the value in file-max).

For example, an output of 14240 0 2097152 indicates that 14,240 file handles are currently active out of a global capacity of 2,097,152.

Identifying Top File Descriptor Consumers

When a server exhibits high file descriptor counts, run this one-line shell pipeline to identify the top ten processes holding the highest descriptor counts:

for pid in /proc/[0-9]*; do
    p="${pid##*/}"
    count=$(ls -1 "$pid/fd" 2>/dev/null | wc -l)
    if [ "$count" -gt 0 ]; then
        echo "$count $p $(cat "$pid/comm" 2>/dev/null)"
    fi
done | sort -rn | head -n 10

Once you identify the offending PID, inspect the specific resources it holds using lsof:

lsof -p <PID>

This reveals whether the process is legitimately handling client connections or leaking file handles due to unclosed input and output streams.

Diagnostic Command Reference

The following comparison table summarizes essential inspection commands, their operational scope, and output interpretations:

Increasing file descriptor limits requires configuring the specific subsystem that governs the target workload. A common pitfall in Linux systems administration is running ulimit -n 65536 in a terminal and expecting background services to inherit the change.

A temporary ulimit -n invocation modifies only the current shell process and its child processes, resetting the moment the shell session terminates. Establishing permanent, reboot-resilient limits requires configuring PAM limits for user sessions, systemd unit directives for services, and sysctl parameters for the kernel.

Ephemeral Changes for Interactive Shells

To raise the limit for a temporary task, batch script, or immediate debugging session, invoke ulimit followed by the desired integer:

ulimit -n 65536

An unprivileged user can only raise their soft limit up to the value of their hard limit (ulimit -Hn). If an unprivileged user attempts to specify a value higher than the hard limit, the shell rejects the command with an error message. The superuser (root) can set both soft and hard limits to any value up to the kernel ceiling defined in fs.nr_open.

Permanent Limits for User Sessions via PAM

For interactive user logins, SSH sessions, and cron tasks managed by Pluggable Authentication Modules (PAM), persistent limits are configured in /etc/security/limits.conf or drop-in files under /etc/security/limits.d/.

Add configuration directives using the standard format <domain> <type> <item> <value>:

*               soft    nofile          65536
*               hard    nofile          65536
root            soft    nofile          65536
root            hard    nofile          65536
appuser         soft    nofile          1048576
appuser         hard    nofile          1048576

Two critical details must be observed when editing limits.conf:

1. The Wildcard Does Not Cover Root: The asterisk wildcard matches all standard user accounts, but PAM explicitly excludes the root account from wildcard expansion. Always declare separate entries for root.
2. PAM Module Activation: Verify that PAM sessions load the pam_limits.so library. In Debian and Ubuntu systems, verify that /etc/pam.d/common-session and /etc/pam.d/common-session-noninteractive contain:

session required pam_limits.so

Overriding Limits for Systemd Services

The most frequent configuration gap in modern Linux administration involves system services. Systemd does not use PAM for services it spawns. Consequently, background services like Nginx, Docker, PostgreSQL, Redis, and custom Python or Node.js workers completely ignore /etc/security/limits.conf.

Modern systemd releases default DefaultLimitNOFILE to 524288:1048576 (524,288 soft and 1,048,576 hard), but older releases or customized distributions can constrain services to 1,024 descriptors.

To adjust the file descriptor limit for any systemd service, set the property directly using systemctl:

sudo systemctl set-property custom-worker.service LimitNOFILE=65536

Alternatively, create a drop-in configuration file directly on disk:

sudo mkdir -p /etc/systemd/system/custom-worker.service.d
printf '[%%s]
LimitNOFILE=65536
' "Service" | sudo tee /etc/systemd/system/custom-worker.service.d/override.conf

You can specify a single integer to set soft and hard limits equally, or specify two colon-delimited values (LimitNOFILE=65536:524288) to establish separate soft and hard boundaries. To grant a high-throughput network proxy or database the maximum allowable descriptors, use LimitNOFILE=infinity, which sets the limit to the value defined in fs.nr_open.

Apply the override and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart custom-worker.service
cat /proc/$(pgrep -o custom-worker)/limits | grep "Max open files"

To adjust the default file descriptor limit for all systemd services globally, edit /etc/systemd/system.conf and declare:

DefaultLimitNOFILE=65536:524288

Raising Kernel Boundaries with sysctl

The Linux kernel enforces two global parameters under /proc/sys/fs/:

• fs.file-max: The system-wide ceiling on open file handles.
• fs.nr_open: The maximum value any individual process can assign to its soft or hard RLIMIT_NOFILE. If you configure LimitNOFILE=2000000 but fs.nr_open is set to 1048576, systemd or the shell will reject the allocation.

To permanently increase both values across system reboots, create a sysctl configuration file at /etc/sysctl.d/99-file-limits.conf:

fs.file-max = 2097152
fs.nr_open = 2097152

Apply the changes immediately without restarting the host:

sudo sysctl --system

Configuration Matrix Across Subsystems

The following comparison table highlights which configuration file governs each operational environment:

Scale Agent File Ingestion Without Linux File Descriptor Limits

Connect Claude, Cursor, and autonomous agents to Fast.io workspaces over MCP. Store multi-gigabyte corpora, index documents for hybrid search, and eliminate local EMFILE bottlenecks. Monthly plans start with a 14-day free trial.

Start 14-Day Trial

While system administrators often resolve EMFILE errors by raising configuration ceilings, software developers frequently encounter EMFILE due to application architecture defects. In runtime environments like Node.js and Python, EMFILE errors are frequently a symptom of unbounded concurrency or resource leaks rather than insufficient operating system capacity.

Node.js and JavaScript Runtime Failures

In Node.js applications, file operations execute asynchronously. A common antipattern involves reading a large directory tree by launching unthrottled asynchronous promises:

const fs = require('fs/promises');

async function processDirectory(fileList) {
  return Promise.all(fileList.map(filePath => fs.readFile(filePath, 'utf-8')));
}

If fileList contains thousands of files and the default soft limit is 1,024, the Node.js event loop initiates file opening operations until the process exhausts its descriptors, failing with Error: EMFILE: too many open files.

To fix this issue without raising system limits, throttle concurrent operations using batching:

const fs = require('fs/promises');

async function processWithBatching(fileList, concurrency = 64) {
  const results = [];
  for (let i = 0; i < fileList.length; i += concurrency) {
    const chunk = fileList.slice(i, i + concurrency);
    const chunkResults = await Promise.all(
      chunk.map(file => fs.readFile(file, 'utf-8'))
    );
    results.push(...chunkResults);
  }
  return results;
}

Similarly, tools that watch filesystem changes (such as Chokidar or Webpack file watchers) allocate an inotify file descriptor per watched directory. On massive repositories, raising /proc/sys/fs/inotify/max_user_watches alongside LimitNOFILE prevents watcher crashes.

Python Asyncio and Multiprocessing Leaks

In Python, an OSError with errno 24 ("Too many open files") commonly occurs when workers open files or HTTP client sessions without proper context managers:

def load_corpus(paths):
    records = []
    for path in paths:
        f = open(path, 'r')
        records.append(f.read())
    return records

If an exception occurs or if Python's garbage collector delays cleanup, file descriptors remain allocated. Always use context managers (with open(path) as f:) for deterministic closing of descriptors upon block exit.

In high-concurrency asyncio applications, bound simultaneous network requests using asyncio.Semaphore:

import asyncio
import aiohttp

async def fetch_urls(urls, max_concurrency=100):
    semaphore = asyncio.Semaphore(max_concurrency)
    async with aiohttp.ClientSession() as session:
        async def fetch(url):
            async with semaphore:
                async with session.get(url) as response:
                    return await response.text()
        return await asyncio.gather(*(fetch(u) for u in urls))

Docker and Containerized Agent Limits

Containers run in isolated namespaces, but they do not automatically inherit host PAM configurations. By default, a container inherits its file descriptor boundaries from the container engine daemon.

To set file descriptor limits on an individual Docker container, pass the --ulimit flag at runtime:

docker run -d --name agent-worker --ulimit nofile=65536:65536 python:3.11-slim

In Docker Compose configurations, define limits within compose.yaml:

services:
  agent-runner:
    image: custom-agent:latest
    ulimits:
      nofile:
        soft: 65536
        hard: 65536
    restart: on-failure

To configure global container limits across all Docker workloads on a host, define default ulimits in /etc/docker/daemon.json:

{
  "default-ulimits": {
    "nofile": {
      "Name": "nofile",
      "Hard": 65536,
      "Soft": 65536
    }
  }
}

After updating daemon.json, execute sudo systemctl restart docker to apply the baseline limits to all newly initialized containers.

While raising operating system limits and batching local file reads stabilizes single-node services, relying on local filesystem handles presents major architectural limitations when scaling autonomous agents and data extraction pipelines.

When background agents process extensive technical manuals, legal binders, or code repositories, maintaining thousands of open file descriptors on local Linux servers creates memory bloat, high kernel overhead, and file synchronization collisions. Instead of holding thousands of open file handles across ephemeral container disks, modern architectures decouple file storage from compute with cloud workspaces.

Decoupling File I/O with Fast.io Workspaces

In an intelligent workspace architecture, document collections and project assets reside in org-owned Fast.io workspaces. Rather than requiring local agent processes to mirror, unpack, and open local files on an ext4 volume, the workspace is the persistent collaboration layer for both human teams and AI agents.

Teams can ingest documents through several native paths:

• Direct Uploads: Ingest files directly via the browser or API, using chunked upload mechanics that support large file ingestion up to plan limits (25 GB on Starter, 50 GB on Business, and 100 GB on Enterprise).
• Cloud Sync: Cloud Sync connects Google Drive, Dropbox, Box, and OneDrive folders to a workspace, one-way or two-way, on a schedule or on demand.
• URL Import: Ingest files directly from public web addresses and cloud storage links without drawing down local network bandwidth or allocating local server file descriptors.

Native Intelligence Mode and MCP Tooling

Once documents arrive in a workspace, enabling Intelligence Mode indexes the content automatically across full-text, semantic vectors, and metadata values. This eliminates the need for teams to manage independent vector databases, chunking workers, or local file indexing loops.

Autonomous assistants and coding agents interact with the workspace via the remote Model Context Protocol (MCP) server over Streamable HTTP:

• General Assistants and Workflow Agents: Connect Claude Desktop, Claude Cowork, OpenClaw, or custom agents via https://mcp.fast.io/mcp/tools.
• IDE and Coding Agents: Connect Claude Code, Cursor, VS Code, and terminal agents via https://mcp.fast.io/mcp/code.
• ChatGPT and Codex Custom Servers: Connect via https://mcp.fast.io/mcp/operations (or install the Fast.io plugin directly from the ChatGPT plugin directory).

Using the MCP storage search action, an agent queries documents by meaning and retrieves only the precise citations needed to answer a prompt. This completely bypasses local Linux EMFILE bottlenecks: an agent analyzing a repository of 50,000 files requires only a single HTTP connection to the MCP endpoint rather than 50,000 open local file descriptors.

Collaborative Capabilities for Agentic Teams

Intelligent cloud workspaces provide capabilities that local Linux filesystems cannot deliver:

• Metadata Views: Teams can transform unstructured files into queryable spreadsheets with typed schemas at /product/document-data-extraction/. AI automatically extracts fields like contract counterparties, renewal dates, and line item totals from PDFs, images, and office documents without manual OCR rules.
• Collaborative Notes: Notes kept beside the files for people and agents use version-checked saves to prevent conflicting edits between human collaborators and automated workers.
• Per-File Version History: Every modification creates a discrete version, allowing teams to audit changes, compare diffs, and restore prior states across concurrent multi-agent operations.
• Detailed Activity Log: An organization-wide activity log captures changes made across workspaces and shares, recording uploads, edits, moves, and deletions without tracking read-only views or downloads.
• Ownership Transfer: Autonomous agents can create an organization, configure workspaces and branded shares, and transfer full organization ownership to a human stakeholder while retaining administrative access.

Monthly plans start with a 14-day free trial, which requires a credit card. Teams can scale across Starter ($9.99/mo with 3 seats and 250 GB storage), Business ($49.99/mo with 10 seats and 5 TB storage), and Enterprise ($199.99/mo with 30 seats and 25 TB storage) tiers. Explore persistent workspace storage for agentic teams at /storage-for-agents/ and review setup steps at https://mcp.fast.io/docs.