Exit Code 137: Causes, OOM Killer Diagnosis, and Fixes

When a containerized process crashes with exit code 137, the operating system kernel has sent an uncatchable SIGKILL because the container breached its configured memory limit or starved the host machine. The fatal termination leaves no application stack trace, writes nothing to standard error, and drops inflight transactions mid-stream, turning what appears to be an application bug into a silent infrastructure failure.

Under POSIX and Unix standards, fatal signal exit codes follow a uniform calculation:

Exit Code = 128 + Signal Number

Because SIGKILL corresponds to standard Linux signal number 9, the shell reports an exit status of 128 + 9 = 137. Unlike SIGTERM (signal 15, yielding exit code 143), which provides the running application a grace window to intercept the signal, execute shutdown callbacks, flush write buffers to disk, and close network connections, SIGKILL operates directly at the kernel task scheduler layer. The kernel immediately deschedules the process, tears down its virtual address space, frees associated memory pages, closes all open file descriptors, and notifies the parent process. The application code receives zero CPU cycles to log its state or clean up resources.

The Linux kernel relies on the Out-Of-Memory killer (mm/oom_kill.c) to maintain system stability under memory overcommit. When physical RAM is exhausted and the kernel memory allocator cannot satisfy page allocation requests, or when a container exceeds its configured control group (cgroup) memory boundary, the kernel intervenes to prevent a system lockup. The OOM killer selects a victim process by computing a badness score for every active task in the target cgroup or system memory domain.

The badness heuristic evaluates the proportion of physical memory the process occupies, adjusted by a factor stored in /proc/[pid]/oom_score_adj. The adjustment value ranges from -1000 to +1000. Core system daemons often carry an adjustment of -1000, making them completely immune to termination. Conversely, unprivileged application containers default to 0 or positive values, placing them at the front of the line when the kernel needs to reclaim memory pages. Once the kernel selects the victim task with the highest badness score, it transmits SIGKILL directly to the process thread group.

In containerized environments such as Docker and Kubernetes, exit code 137 manifests under two distinct conditions:

1. Container Memory Limit Breaches: The container breaches the memory constraint defined in its deployment manifest, such as the --memory flag detailed in the Docker resource constraints documentation or resources.limits.memory in Kubernetes. The container runtime configures a dedicated Linux cgroup with that limit. When memory allocations inside the container cross that threshold, the cgroup OOM killer terminates the primary process without affecting other workloads on the host node.

2. Host-Level Memory Starvation: The underlying physical host node runs out of physical RAM entirely, forcing the host-level kernel OOM killer to terminate container processes to preserve host operating system stability.

While an OOM event is the predominant trigger for exit code 137, it is not the only source of SIGKILL. For example, executing docker kill explicitly sends SIGKILL, resulting in exit code 137. Similarly, when docker stop is invoked, Docker transmits an initial SIGTERM signal and starts a 10-second countdown timer. If the application process fails to complete its graceful shutdown within those 10 seconds, Docker escalates the termination and sends SIGKILL, yielding exit code 137 with OOMKilled: false.

The table below contrasts standard container termination exit codes, their underlying Linux signals, and their operational root causes:

• HTTP 413 Request Entity Too Large: Causes, Server Limits & WorkaroundsAn HTTP 413 request entity too large error occurs when a client payload exceeds buffer limits configured on a web...
• ChatGPT Conversation Limit: Thread Length, Context Degradation, and Memory FixesA ChatGPT conversation limit is the operational threshold where an individual chat thread begins dropping context,...
• ChatGPT Conversation Too Long: Fixes, Limits, and Context WorkaroundsThe conversation too long error in ChatGPT occurs when an ongoing thread exceeds the model working memory context...
• DeepSeek Context Limit: Reasoning Token Overhead, Output Ceilings & Large-Corpus FixesThe DeepSeek context limit is 128,000 tokens for DeepSeek-V3 and DeepSeek-R1, with newer managed API deployments...
• Gemini Code Assist Limits: Quotas, Daily Caps, and Codebase IndexingGemini Code Assist enforces daily developer quotas, including 6,000 code completions, 240 chat requests, and 1,000 to...
• gRPC Message Size Limit: 4MB Default Cap & RESOURCE_EXHAUSTED FixesThe default gRPC message size limit is 4 MB (4,194,304 bytes) for inbound messages, throwing a...

More on this subject: AI Agents: General Guides (201 guides)

Diagnosing a container exit code 137 termination requires isolating the exact component that transmitted SIGKILL. Relying solely on application log files is ineffective because uncatchable signals prevent processes from writing final log entries before termination. System administrators and platform engineers must query container metadata, kernel diagnostic rings, and cgroup accounting statistics.

Following a systematic four-step diagnostic procedure isolates whether a crash was caused by a cgroup memory breach, host-level starvation, or a graceful shutdown timeout.

Step 1: Checking Container Exit Flags in Docker and Kubernetes

In Docker environments, inspect the termination metadata of the stopped container using the JSON format filter:

docker inspect <container_id> --format '{{json .State}}'

Examine the resulting payload for three critical attributes:

{
  "Status": "exited",
  "Running": false,
  "Paused": false,
  "Restarting": false,
  "OOMKilled": true,
  "Dead": false,
  "Pid": 0,
  "ExitCode": 137,
  "Error": "",
  "StartedAt": "2026-10-26T08:15:30.123456789Z",
  "FinishedAt": "2026-10-26T08:22:15.987654321Z"
}

If OOMKilled evaluates to true, the container kernel cgroup enforced its memory quota and terminated the process. If OOMKilled evaluates to false while ExitCode remains 137, the termination was prompted by an external SIGKILL, such as an explicit docker kill execution or a docker stop timeout.

In Kubernetes clusters, inspect the Pod status using kubectl describe:

kubectl describe pod <pod_name> -n <namespace>

Under the Containers status section, review the Last State block:

State:          Waiting
  Reason:       CrashLoopBackOff
Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      Mon, 26 Oct 2026 08:15:30 +0000
  Finished:     Mon, 26 Oct 2026 08:22:15 +0000

Kubernetes explicitly flags Reason: OOMKilled. To check if the cluster recorded eviction events related to node memory pressure, check the cluster event feed:

kubectl get events -n <namespace> --field-selector reason=Evicted --sort-by='.metadata.creationTimestamp'

Step 2: Inspecting Kernel Ring Buffer Logs via dmesg

When a container runtime terminates a process via OOM, the Linux kernel logs detailed diagnostic records into the kernel ring buffer. Access these records on the container host:

dmesg -T | grep -i -E 'oom[-_]killer|killed process'

A typical kernel diagnostic entry contains the following diagnostic fields:

[ 14285.123456] Task in /docker/8a3f91b7d42e killed as a result of limit of /docker/8a3f91b7d42e
[ 14285.123457] memory: usage 2097152kB, limit 2097152kB, failcnt 14285
[ 14285.123458] memory+swap: usage 2097152kB, limit 2097152kB, failcnt 0
[ 14285.123459] oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[ 14285.123460] Out of memory: Killed process 41285 (python3) total-vm:3842100kB, anon-rss:2045120kB, file-rss:51200kB, shmem-rss:832kB
[ 14285.123461] oom_reaper: reaped process 41285 (python3), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Key indicators to extract from this kernel trace include:

• Cgroup Path: Task in /docker/... killed as a result of limit of /docker/... proves the crash was triggered by the container cgroup boundary rather than global node exhaustion.
• Allocation Metrics: usage 2097152kB, limit 2097152kB indicates that usage hit the exact memory ceiling configured for the container.
• Resident Set Breakdown: anon-rss reflects private heap allocations, while file-rss indicates memory-mapped disk pages and cached buffers. A high anon-rss indicates that heap data structures, objects, or buffers drove the memory spike.

Step 3: Analyzing Control Group Accounting Statistics

Linux systems use cgroups to enforce resource ceilings. In modern environments with cgroup v2 enabled, memory controllers export runtime statistics directly via the pseudo-filesystem:

cat /sys/fs/cgroup/memory.events

This interface surfaces specific counters:

low 0
high 0
max 48
oom 12
oom_kill 12
oom_group_kill 0

Each increment in oom_kill corresponds to an enforced kernel termination event. You can evaluate real-time usage against peak historical consumption by checking:

cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.peak
cat /sys/fs/cgroup/memory.max

On legacy systems operating cgroup v1, inspect /sys/fs/cgroup/memory/memory.oom_control and read memory.max_usage_in_bytes to evaluate how close the workload ran to its configured memory.limit_in_bytes.

Step 4: Isolating Memory Leaks From Transient Allocation Spikes

Distinguishing between persistent memory leaks and transient allocation spikes informs the proper remediation path:

• Memory Leaks: The resident memory profile exhibits a continuous, upward staircase or linear slope across hours of operation. Objects accumulate in memory due to unclosed database connections, long-lived global caches without eviction policies, or circular object references preventing garbage collection.
• Transient Allocation Spikes: Memory consumption remains stable and well within cgroup limits during idle periods, then spikes almost vertically within milliseconds. This profile indicates batch document loading, in-memory unzipping, raster image processing, or large query result buffering.

For containerized Kubernetes workloads, observe the container_memory_working_set_bytes metric rather than simple resident set size. The Kubernetes kubelet compares working set memory against resources.limits.memory when calculating OOM kill conditions, as described in the Kubernetes memory management documentation.

Data processing workers, automated agent pipelines, and background job runners account for a heavy share of exit code 137 incidents in containerized environments. These workloads frequently process dynamic external inputs whose uncompressed memory footprint expands dramatically during ingestion, feature extraction, and model inference.

Four architectural patterns consistently trigger memory limit breaches in containerized pipelines:

1. In-Memory Ingestion and Deserialization of Large Files

A common pitfall in background workers is loading complete source files directly into process memory using operations such as data = file.read() or json.loads(response.content).

When parsing serialized data formats such as JSON, XML, or CSV in languages like Python, Ruby, or JavaScript, the in-memory representation is substantially larger than the raw file size on disk. In Python, every string, integer, dictionary key, and list element incurs object overhead through internal pointer allocations, hash tables, and metadata headers. A 500 megabyte raw JSON export containing millions of nested records frequently expands into 2 to 4 gigabytes of resident heap memory upon deserialization. If the worker container is bounded by a 2 gigabyte limit, the runtime encounters an instant OOM termination during dictionary allocation.

Similarly, document extraction pipelines that process large scanned PDF files or high-resolution images often instantiate complete uncompressed bitmap buffers in memory during OCR parsing. A multi-page document rendered at print resolution expands into hundreds of megabytes of raw pixel arrays per page, quickly blowing past container boundaries.

2. High-Dimensional Vector Embeddings and Tensor Allocations

Containerized AI workflows that generate vector embeddings or execute local model inference run into tight memory constraints. Generating embeddings across large batches of text requires allocating high-dimensional activation tensors in system RAM:

• Batch Tensor Footprint: Processing 500 chunks through an embedding model simultaneously allocates intermediate matrix multiplication buffers, attention weight caches, and output vectors.
• Memory Allocator Fragmentation: Native C++ allocators underlying frameworks such as PyTorch, TensorFlow, and ONNX Runtime allocate large contiguous memory arenas. When tensors are freed, the framework memory pool often retains the allocated physical pages rather than returning them to the operating system kernel immediately, creating artificial heap fragmentation that drives cgroup limits to the threshold.

3. Unbounded Concurrency and Process-Level Forking

Background queues such as Celery, multiprocessing workers, and task consumers frequently rely on multi-process concurrency models. When a parent process forks worker sub-processes, modern operating systems rely on copy-on-write memory page sharing.

Initially, the child processes share physical memory pages with the parent process. However, as worker tasks execute, mutate data, allocate caches, and import modules, the kernel duplicates each modified memory page. In a container configured with 4 worker processes and a 4 gigabyte limit, a single heavy job running in one worker can push cumulative cgroup memory usage past the threshold, causing the OOM killer to terminate either the worker or the parent queue manager.

4. Ephemeral Container Layer Writes and Pagecache Buildup

In Linux containers, files written to the temporary container filesystem (such as /tmp or local working directories) do not bypass memory accounting. The kernel caches disk reads and writes in the pagecache to accelerate subsequent access.

In cgroup v1 and cgroup v2 accounting, active pagecache memory counts directly toward the container memory limit alongside anonymous application memory. If a pipeline worker downloads a 3 gigabyte tar archive to /tmp and extracts its contents, the Linux kernel retains those disk pages in cache. If the application subsequently allocates heap memory while dirty pages remain uncommitted, the kernel cannot reclaim the memory quickly enough, precipitating an exit code 137 crash.

Eliminate Container Exit Code 137 Crashes in Production

Stop scaling container RAM to buffer large files and vector embeddings. Offload heavy documents and datasets to Fastio intelligent workspaces, where files are indexed for semantic search and accessible over MCP without exhausting container memory or triggering exit code 137. Monthly plans start with a 14-day free trial.

Start 14-Day Trial

Eliminating exit code 137 crashes requires addressing both container infrastructure configuration and application data access patterns. Merely restarting crashed containers without structural adjustments guarantees recurring outages whenever workload volumes fluctuate.

Following four concrete operational steps ensures workloads remain within their assigned resource boundaries.

Step 1: Establish Sane Cgroup Boundaries and Headroom

When defining container resource boundaries in Docker and Kubernetes, separate memory requests from memory limits. In Kubernetes, resources.requests.memory dictates pod scheduling on worker nodes, while resources.limits.memory configures the cgroup boundary.

Setting limits strictly equal to average runtime consumption introduces instability. Production applications require approximately 25 to 30 percent memory headroom above average active working set memory to absorb transient spikes caused by runtime garbage collection pauses, transient network buffers, and temporary pagecache caching.

Configure Docker containers with explicit memory reservations and soft limits:

docker run -d \
  --name doc-processor \
  --memory="4g" \
  --memory-reservation="3g" \
  --memory-swap="4g" \
  my-worker-image:latest

In Kubernetes pod manifests, declare realistic requests and limits:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: embedding-worker
spec:
  template:
    spec:
      containers:
        - name: worker
          image: worker-registry/embedding-worker:v2.4
          resources:
            requests:
              memory: "2Gi"
              cpu: "1000m"
            limits:
              memory: "4Gi"
              cpu: "2000m"

Setting --memory-swap equal to --memory in Docker disables swap, ensuring the container fails predictably rather than thrashing swap disks with high latency when physical RAM is exhausted.

Step 2: Implement Chunked Streaming for File Operations

Applications should never read complete files or large API payloads into memory in a single synchronous call. Instead, refactor file processing routines to stream data in discrete chunks.

When reading multi-gigabyte files or handling HTTP downloads, use chunked iterators that keep working memory bounded regardless of total file size:

def process_large_dataset(source_stream, chunk_size=65536):
    """Process data in chunks to maintain bounded memory usage."""
    for chunk in source_stream.iter_lines(chunk_size=chunk_size):
        if chunk:
            parse_record(chunk)

For structured formats such as JSON, adopt iterative streaming parsers (such as ijson in Python or stream parsers in Node.js) rather than monolithic deserializers. Iterative parsing processes elements sequentially from an I/O stream, allowing a container with 256 megabytes of RAM to parse a 10 gigabyte JSON stream without triggering the OOM killer.

Step 3: Bound Worker Concurrency and Memory Pools

Prevent worker explosion by strictly throttling concurrent task execution based on measurable memory constraints:

• Dynamic Concurrency Calculations: Calculate worker concurrency using the formula: Concurrency = floor((Container Memory Limit - Base Runtime Overhead) / Peak Task Memory). If base framework initialization requires 500 megabytes and each task consumes 750 megabytes at peak, a container configured with 2 gigabytes of RAM can safely run only 2 concurrent tasks.
• Worker Lifetime Recycling: In Celery or similar Python background workers, configure --max-tasks-per-child=100. Recycling worker sub-processes after a set number of tasks releases fragmented heap memory back to the operating system, mitigating slow memory accumulation.

Step 4: Profile Memory Curves in Constrained Local Environments

Detect memory leaks and allocation spikes prior to production deployment by running local profiling containers constrained to production memory limits.

Use lightweight memory profilers to capture heap snapshots across synthetic load tests. Observing memory allocation line by line highlights unexpected array duplications, unclosed buffers, and memory-heavy libraries that require refactoring before merging into production branches.

The traditional response to container exit code 137 errors is vertical scaling: increasing container memory limits from 2 gigabytes to 8 gigabytes, 16 gigabytes, or 32 gigabytes. While this workaround temporarily postpones the OOM killer, it is economically inefficient and architecturally fragile. A larger input file, an expanded batch size, or a concurrent traffic surge will inevitably breach the new boundary.

The root design defect is treating container compute memory as an ephemeral storage substrate. Containers are meant to serve as lightweight, stateless compute processes that transform, filter, and route data. Storing multi-gigabyte document corpora, staging large media files, and maintaining in-memory search indexes inside application containers strains infrastructure limits.

Architectural remediation requires decoupling compute logic from data persistence. By offloading heavy document storage, indexing, and extraction to external managed workspaces, container memory footprints remain lean and predictable.

This architectural decoupling is where Fastio Workspaces solves container memory constraints:

• Persistent Workspaces Without Memory Tax: Rather than buffering source archives or embedding datasets in container memory or local ephemeral filesystems, teams store their corpora in centralized, organization-owned Fastio workspaces. Data persists independently of container restarts or pod evictions.
• Remote Semantic Search via Intelligence Mode: In traditional setups, background containers often trigger exit code 137 because they load embedding models and vector collections into RAM to perform similarity matching. When Intelligence Mode is enabled on a Fastio workspace, documents are automatically indexed on arrival for full-text and semantic search. AI agents and containerized workers query the index remotely, retrieving precise excerpts with document citations without maintaining in-memory vector stores.
• Structured Data Extraction With Metadata Views: Instead of containers parsing multi-page scanned PDFs or contracts with heavy OCR libraries, Metadata Views extract typed structured data directly from files. Agents query sortable, filterable schemas over the API rather than parsing raw document structures locally.
• Native MCP Connectivity: Automated agents and containerized worker scripts connect to Fastio using the remote Model Context Protocol (MCP) server. Coding agents and backend pipelines connect over Streamable HTTP at https://mcp.fast.io/mcp/code (or https://mcp.fast.io/mcp/tools for Claude and general clients). Complete setup instructions are available at https://mcp.fast.io/docs. Through MCP tools, agents search, read, and write files remotely without loading multi-gigabyte payloads into local process memory.
• Direct Streaming and Chunked Uploads: Fastio supports direct chunked uploads, enabling containers to stream large files into workspace storage without buffering complete files in container RAM.
• Auditable Collaboration and Versioning: Every workspace maintains complete per-file version history, while changes are tracked in a detailed activity log. Role-based permissions are configured per organization, workspace, and share, ensuring strict security boundaries between distinct background services.

By offloading document indexing and file storage to Fastio AI Storage and Fastio AI, container workloads require only enough memory to manage execution orchestration. This decoupling eliminates exit code 137 crashes while significantly lowering overall cloud infrastructure spend.