Elasticsearch Document Size Limit: http.max_content_length and Bulk Indexing

An Elasticsearch document size limit is determined by the HTTP transport layer setting http.max_content_length (default 100MB) and Lucene's internal 2GB indexing ceiling, although production clusters recommend keeping individual documents below a few megabytes. Elasticsearch enforces a default HTTP request body limit of 100MB on incoming network payloads according to the official Elasticsearch networking documentation. When an application submits a request to Elasticsearch, the payload crosses multiple architectural layers before landing inside an inverted index or doc value column. The outer perimeter is governed by the HTTP transport layer, where Netty handles incoming network frames. Netty evaluates incoming HTTP request sizes against the http.max_content_length threshold before routing requests to cluster worker threads.

If an incoming HTTP request body exceeds http.max_content_length, the node terminates request parsing and returns an HTTP 413 Payload Too Large error. Importantly, when HTTP compression is enabled on the client or server, this 100MB limit applies to the uncompressed HTTP request body. Transmitting a heavily compressed 80MB payload that inflates to 120MB uncompressed triggers an immediate rejection at the transport boundary.

Beneath the HTTP transport layer sits Apache Lucene, the core search library powering each Elasticsearch shard. Lucene structures its internal indexing pipelines around signed 32-bit integers. Because Java arrays and byte buffers index data with signed 32-bit integers, Lucene imposes an absolute maximum ceiling of 2,147,483,519 bytes (approximately 2GB) on any single indexed binary field, stored field, or document structure. In addition, Lucene restricts each individual shard to an absolute maximum of 2,147,483,519 documents.

While these architectural maximums suggest that Elasticsearch can accept massive JSON records, production deployments operate under much stricter constraints:

Treating Elasticsearch as an arbitrary document store or media archive degrades cluster stability. Search engines optimize for fast term matching, scoring, and columnar aggregations, none of which benefit from indexing monolithic multi-megabyte payloads.

• AWS Bedrock File Size Limits: Document Constraints in Knowledge Bases and Converse APIAWS Bedrock Knowledge Bases enforces a 50 MB per-document limit for file ingestion, while the Bedrock Converse API caps...
• ChatGPT Max File Size: The 512 MB Upload Limit Across PlansChatGPT enforces a strict maximum file size limit of 512 MB per uploaded file across all subscription plans, alongside...
• ChatGPT PDF Limit: File Size, Page Count, and Large Document WorkaroundsThe ChatGPT PDF limit enforces a 512MB file size ceiling, a 2 million token extraction threshold, and a restriction of...
• MongoDB Document Size Limits: The 16MB BSON Ceiling and GridFS AlternativesThe MongoDB document size limit is a fixed 16-megabyte ceiling on individual BSON documents, designed to prevent...
• PHP File Upload Limit: upload_max_filesize, post_max_size, and Memory LimitsThe PHP file upload limit is governed by configuration directives in php.ini, primarily upload_max_filesize and...
• GitHub File Size Limits Explained for Teams and CopilotGitHub hard-blocks any single file larger than 100 MiB in normal Git history, warns above 50 MiB, and caps browser...

More on this subject: Agent File and Document Workflows (298 guides)

When a client transmits an HTTP payload that exceeds http.max_content_length, the failure occurs at the Netty transport layer before the document ever reaches the indexing pipeline. Netty decodes incoming HTTP frames sequentially. As soon as the accumulated content length exceeds the configured byte threshold, Netty raises a TooLongFrameException and aborts request processing.

In response, Elasticsearch terminates the connection or returns an HTTP 413 error payload:

{
  "error": {
    "root_cause": [
      {
        "type": "too_long_frame_exception",
        "reason": "An HTTP line is larger than 104857600 bytes."
      }
    ],
    "type": "too_long_frame_exception",
    "reason": "An HTTP line is larger than 104857600 bytes."
  },
  "status": 413
}

This rejection protects cluster memory. Without an HTTP transport ceiling, a small number of concurrent clients streaming 500MB JSON objects could fill the JVM heap, triggering memory exhaustion and crashing active cluster nodes.

If a request bypasses the HTTP transport layer (for example, when an internal plugin or transport client constructs an oversized document in memory), the failure moves downstream into Lucene index writers. If an individual binary field or stored string exceeds the signed 32-bit integer boundary, Lucene throws an IllegalArgumentException stating that the array or string exceeds maximum allowable length.

When oversized payloads bypass network guards, the damage surfaces as severe cluster degradation rather than clean error codes:

1. JVM Garbage Collection Pauses: Parsing multi-megabyte JSON payloads creates millions of transient Java objects. In the HotSpot JVM, large byte arrays and objects allocate directly into Old Generation or Humongous regions, forcing frequent concurrent marking and stop-the-world garbage collection pauses.
2. Indexing Buffer Starvation: Elasticsearch dedicates an indexing memory buffer (governed by indices.memory.index_buffer_size, which defaults to one-tenth of total heap memory) across all active shards on a node. A single oversized document monopolizes this buffer, forcing Lucene to flush partially filled segment buffers to disk prematurely.
3. Small Segment Proliferation: Premature buffer flushes create hundreds of tiny Lucene segments on disk. The node then spends disproportionate CPU and disk I/O merging these small segments in the background, starving search queries of compute capacity.
4. Slow Highlight and Fetch Operations: When users query the cluster, retrieving the _source field for multi-megabyte documents forces Lucene to decompress large stored field blocks, introducing multi-second query latencies.

You can modify http.max_content_length in your elasticsearch.yml configuration file. Because this parameter configures the static network interface, you cannot update it dynamically using the cluster settings API (_cluster/settings).

To adjust the limit, define the parameter on every coordinating and master node across your cluster:

http.max_content_length: 200mb

After modifying the file, you must perform a rolling restart of your nodes for the new limit to take effect. Supported unit notations include mb (megabytes) and gb (gigabytes).

While increasing http.max_content_length to 200MB or 500MB is technically straightforward, doing so in production environments introduces substantial risks to cluster availability. Elasticsearch nodes run inside a Java Virtual Machine with a heap typically configured between 8GB and 31GB. Production administrators cap the heap at 31GB to allow the JVM to use Compressed Object Pointers (Compressed OOPs), which preserves memory efficiency by representing object references as 32-bit pointers.

Allocating heap memory requires careful balancing:

+---------------------------------------------------------------+
|                      Total Physical RAM                       |
+-------------------------------+-------------------------------+
|       JVM Heap (Max 31GB)     |      OS Filesystem Cache      |
|  - Indexing Buffers           |  - Lucene Inverted Indices    |
|  - Search Request Caches      |  - Doc Values Columns         |
|  - Cluster State & Metadata   |  - Stored Fields On Disk      |
+-------------------------------+-------------------------------+

When you raise http.max_content_length to accept large payloads, a burst of concurrent requests can rapidly consume available heap. The write thread pool on an Elasticsearch node defaults to allocating one thread per available CPU core, backed by an execution queue that holds up to 10,000 tasks. If 16 worker threads concurrently process 200MB incoming payloads, those requests alone demand more than 3.2GB of active heap memory solely for raw HTTP parsing and JSON deserialization.

Before raising http.max_content_length, audit the root cause of large payloads:

• Are client applications embedding raw base64-encoded PDF files, images, or audio recordings inside JSON document fields?
• Is an automated ingestion script sending thousands of documents inside a single unsegmented _bulk HTTP request?
• Can documents be split into structured sections or chapters before sending them across the wire?

In almost every production incident involving HTTP 413 rejections, the correct remedy is adjusting the ingestion pipeline rather than increasing http.max_content_length.

The http.max_content_length setting applies to the total HTTP request body, which directly impacts the _bulk API. Sizing bulk requests requires balancing network efficiency against cluster memory pressure. Sending individual documents one at a time introduces massive HTTP overhead, while sending oversized bulk requests risks thread pool exhaustion and memory spikes.

Elastic's official indexing tuning guide advises benchmarking bulk request performance starting from small batches and doubling sizes until throughput plateaus. Elastic explicitly cautions that bulk requests should avoid going beyond a couple tens of megabytes per request even if larger requests appear to perform well, because sending many oversized bulk requests concurrently places the cluster under heavy memory pressure.

Benchmarking Optimal Bulk Payload Volumes

To determine the ideal bulk request volume for your specific document schema, execute a benchmarking routine against a staging cluster using a single primary shard:

1. Start with Baseline Batches: Begin by sending 100 documents per bulk request.
2. Double Batch Sizes Incrementally: Double the batch size to 200, 400, 800, 1,600, and 3,200 documents in successive runs.
3. Record Throughput and Latency: Monitor indexing speed (documents indexed per second and megabytes processed per second) alongside response latency.
4. Identify the Ingestion Plateau: At a certain batch size, indexing speed will plateau. Once throughput levels off, increasing the batch size further only consumes additional memory without improving speed.
5. Enforce Byte-Based Limits: Document sizes often vary across real-world datasets. Setting a fixed document count (such as 5,000 records) can produce a 5MB payload in one request and a 120MB payload in the next, triggering an unexpected HTTP 413 error. Size batches by aggregate byte volume (targeting 5MB to 15MB) rather than fixed document counts.

Handling Thread Pool Rejections (HTTP 429)

When bulk requests arrive faster than Elasticsearch can write them to disk, the node's write thread pool queue fills to capacity. Once the queue is full, the node rejects incoming write tasks with an HTTP 429 Too Many Requests response (EsRejectedExecutionException).

Client applications must implement structured backoff handling rather than failing or retrying immediately. The following Python example demonstrates streaming bulk ingestion configured with an explicit 10MB chunk boundary and randomized exponential backoff:

import time
import random
from elasticsearch import Elasticsearch
from elasticsearch.helpers import streaming_bulk

client = Elasticsearch(
    "https://elasticsearch.internal:9200",
    basic_auth=("elastic", "secure_password"),
    request_timeout=30,
)

def generate_documents(records):
    for record in records:
        yield {
            "_index": "knowledge-chunks",
            "_id": record["id"],
            "_source": {
                "document_id": record["document_id"],
                "chunk_index": record["chunk_index"],
                "content": record["text_content"],
                "metadata": record["metadata"],
            },
        }

def index_with_backoff(client, records):
    max_batch_bytes = 10 * 1024 * 1024
    
    for success, info in streaming_bulk(
        client=client,
        actions=generate_documents(records),
        chunk_size=500,
        max_chunk_bytes=max_batch_bytes,
        raise_on_error=False,
        raise_on_exception=False,
    ):
        if not success:
            error_data = info.get("index", {})
            status = error_data.get("status")
            if status == 429:
                sleep_duration = random.uniform(0.5, 2.0)
                time.sleep(sleep_duration)

Running bulk operations from multiple concurrent client threads allows the cluster to saturate available CPU and storage I/O. However, monitor node metrics to verify that the number of active indexing threads does not exceed available processor cores.

Overcome Elasticsearch Document Size Limits with Persistent Workspaces

Store multi-gigabyte document collections in Fast.io workspaces and query indexed content via the Fast.io MCP server without hitting Elasticsearch payload limits or JVM memory ceilings. Monthly plans start with a 14-day free trial.

Start 14-Day Trial

Attempting to store large multi-megabyte files (such as legal contracts, technical manuals, architectural drawings, or high-resolution media) directly inside Elasticsearch documents confuses search indexing with object persistence. When engineers place large unstructured files in document bodies, search performance degrades while storage costs escalate.

The established production solution is the Claim-Check architectural pattern:

[ Ingestion Source ]
        |
        +-----> 1. Store Raw File ------> [ Persistent Cloud Workspace ]
        |                                        |
        |                                        | (Extract Text & Metadata)
        v                                        v
2. Chunk & Vectorize ------------------> [ Elasticsearch / Vector Index ]
                                                 |
                                         (Indexed Record)
                                         - document_id: "doc-1049"
                                         - chunk_index: 3
                                         - text: "Section summary..."
                                         - storage_url: "https://..."

1. Store Raw Files in Dedicated Storage: Keep binary files, original PDFs, spreadsheets, and archives in persistent cloud workspaces or object storage.
2. Extract Structured Text and Metadata: Extract textual contents and segment them into granular paragraphs, sections, or semantic chunks.
3. Index Lightweight Records in Elasticsearch: Index only the granular text chunks into Elasticsearch, accompanied by structured metadata and a reference link pointing back to the source file.
4. Retrieve Full Context on Demand: When search queries match an indexed chunk, client applications display the relevant snippet and resolve the full source document from persistent storage only when a user or agent requests it.

A similar architectural boundary exists in AI agent workflows. When connecting AI models to corporate data, engineers frequently attempt to feed entire document archives directly into model context windows or embedding pipelines. As documented by Anthropic's Claude Help Center, Claude chat sessions accept up to 20 files at up to 500MB per file, while Claude Projects accept files up to 30MB each with no fixed file-count cap, bounded only by the model's context window. Attempting to attach massive file collections directly to prompt contexts causes immediate token exhaustion and operational failure.

Fast.io workspaces provide the persistent workspace layer designed for these large-corpus workflows. Instead of wrestling with upload caps or inflating Elasticsearch payload limits:

• Store Large File Archives: Teams upload raw files directly into shared org-owned workspaces, supporting individual uploads up to 25 GB on Starter, 50 GB on Business, or 100 GB on Enterprise.
• Automatic Intelligence Mode: Enabling Intelligence Mode on a workspace automatically indexes document contents for hybrid search (combining full-text matching, semantic retrieval, and metadata filtering). Files become queryable without maintaining custom vector databases or building custom ingestion pipelines.
• Model Context Protocol Integration: Autonomous agents connect to Fast.io workspaces through the remote MCP server at https://mcp.fast.io/mcp/tools (for Claude and general personal agents) or https://mcp.fast.io/mcp/code (for coding agents), detailed in the Fast.io MCP documentation and storage for agents guide. Agents search indexed workspaces on demand, retrieving only the precise context required for their tasks.
• Structured Data Extraction: Teams use Metadata Views to define typed schemas in natural language, automatically extracting dates, counterparties, line items, and custom attributes from unstructured documents into queryable tables.
• Enterprise Collaboration: Workspaces support per-file version history, Cloud Sync for Google Drive, Dropbox, Box, and OneDrive (on a schedule or on demand), and role-based permissions configured at the organization, workspace, and share levels.

Monthly plans start with a 14-day free trial that requires a credit card. Paid plans available on the pricing page include Starter at $9.99/mo (3 seats, 250 GB storage, 5 workspaces, 100,000 credits a month), Business at $49.99/mo (10 seats, 5 TB storage, 50 workspaces, 600,000 credits a month), and Enterprise at $199.99/mo (30 seats included, 25 TB storage, 200 workspaces, 3,000,000 credits a month).

Maintaining high-throughput, stable Elasticsearch ingestion requires disciplined data hygiene across your application boundaries. Before modifying server configurations or scaling cluster nodes, review this operational checklist:

• Maintain Default Transport Limits: Keep http.max_content_length set to 100mb unless your infrastructure enforces specific proxy constraints. Never treat increasing this limit as a substitute for payload hygiene.
• Cap Individual Documents: Restrict individual JSON records to a few megabytes (ideally below 5MB). Split longer documents, books, and transcripts into discrete chapters or semantic paragraphs linked by common metadata identifiers.
• Size Bulk Requests by Bytes: Avoid building bulk batches based purely on document counts. Configure client bulk processors with a byte threshold between 5MB and 15MB to prevent memory pressure on cluster worker threads.
• Externalize Binary Data: Never store raw binary files, base64 strings, or uncompressed media inside Elasticsearch documents. Store the primary files in persistent workspace storage and index only search tokens, chunks, and metadata.
• Implement Jittered Exponential Backoff: Catch HTTP 429 Too Many Requests responses in client code and pause writes using randomized exponential backoff to allow node write queues to clear.
• Monitor Garbage Collection Overhead: Track JVM Young and Old Generation collection times. Frequent garbage collection cycles exceeding 200 milliseconds signal that incoming request payloads or indexing buffers are straining available heap memory.